3.4.2
The TD Error Formula
Understanding the mathematical formula, its components and the TD target.
What is it?
The TD error is calculated using the immediate reward, the discounted estimated value of the next state and the original value estimate of the current state. The formula is δₜ = Rₜ₊₁ + γV(Sₜ₊₁) − V(Sₜ). The first two terms together form the TD Target.
Key points
1. δₜ represents temporal difference error.
2. Rₜ₊₁ is the immediate reward received after taking an action.
3. γ is the discount factor.
4. V(Sₜ₊₁) is the estimated value of the next state.
5. V(Sₜ) is the original estimate of the current state.
6. TD Target = Rₜ₊₁ + γV(Sₜ₊₁).
7. TD Error = TD Target − Old Estimate.
Example: Calculating TD error
Suppose:
Immediate reward Rₜ₊₁ = 5
Discount factor γ = 0.9
Next-state value V(Sₜ₊₁) = 20
Current-state value V(Sₜ) = 15
Step 1: Calculate the TD Target.
TD Target = 5 + (0.9 × 20)
TD Target = 23
Step 2: Calculate TD Error.
δₜ = 23 − 15
δₜ = 8
The positive TD error indicates that the updated estimate is higher than the original estimate.
Real-world application
The TD error formula is used by value-learning algorithms to compare predictions with observed rewards and estimated future values.
Key takeaway
TD Error = Immediate Reward + Discounted Next-State Value − Current-State Value.