OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / Introduction to Reinforcement Learning / Temporal Difference (TD) Learning
Lesson 116 of 415 6 min read
3.4.2

The TD Error Formula

Understanding the mathematical formula, its components and the TD target.

What is it?

The TD error is calculated using the immediate reward, the discounted estimated value of the next state and the original value estimate of the current state. The formula is δₜ = Rₜ₊₁ + γV(Sₜ₊₁) − V(Sₜ). The first two terms together form the TD Target.

Key points

1. δₜ represents temporal difference error. 2. Rₜ₊₁ is the immediate reward received after taking an action. 3. γ is the discount factor. 4. V(Sₜ₊₁) is the estimated value of the next state. 5. V(Sₜ) is the original estimate of the current state. 6. TD Target = Rₜ₊₁ + γV(Sₜ₊₁). 7. TD Error = TD Target − Old Estimate.

Example: Calculating TD error

Suppose: Immediate reward Rₜ₊₁ = 5 Discount factor γ = 0.9 Next-state value V(Sₜ₊₁) = 20 Current-state value V(Sₜ) = 15 Step 1: Calculate the TD Target. TD Target = 5 + (0.9 × 20) TD Target = 23 Step 2: Calculate TD Error. δₜ = 23 − 15 δₜ = 8 The positive TD error indicates that the updated estimate is higher than the original estimate.

Real-world application

The TD error formula is used by value-learning algorithms to compare predictions with observed rewards and estimated future values.
Key takeaway

TD Error = Immediate Reward + Discounted Next-State Value − Current-State Value.