3.4.3
How the Agent Uses TD Error
Understanding value updates and how the agent uses TD error during learning.
What is it?
The agent uses TD error to adjust its value function and gradually drive prediction errors toward zero. The update rule is V(Sₜ) ← V(Sₜ) + α · δₜ, where α is the learning rate. TD learning updates after each step instead of waiting for the complete episode to finish.
Key points
1. Value update: V(Sₜ) ← V(Sₜ) + α · δₜ.
2. α represents the learning rate.
3. If δₜ > 0, increase the current state value.
4. If δₜ < 0, decrease the current state value.
5. If δₜ = 0, no update is needed.
6. Monte Carlo waits until the episode ends and uses actual return Gₜ.
7. TD updates after every step using estimated future value.
8. TD(λ) interpolates between TD(0) and Monte Carlo.
9. Multi-step TD uses several rewards before bootstrapping from a future value estimate.
Example: Updating a state value
Suppose:
Current value V(Sₜ) = 10
TD error δₜ = 8
Learning rate α = 0.1
Updated value:
V(Sₜ) = 10 + (0.1 × 8)
V(Sₜ) = 10.8
The agent increases its estimate from 10 to 10.8 because the outcome was better than predicted.
Comparison of learning approaches:
Monte Carlo: Waits until the episode ends and uses actual return.
TD: Updates after every step using estimated future value.
TD(λ): Interpolates between TD(0) and Monte Carlo. λ = 0 represents pure TD; λ = 1 represents pure Monte Carlo.
Multi-step TD: Uses multiple rewards before adding a discounted future value estimate.
Real-world application
TD updates are used in value estimation, Q-learning, Deep Q-Networks and actor-critic methods. TD(λ) also provides the foundation for the bias-variance trade-off used in Generalized Advantage Estimation (GAE).
Key takeaway
TD learning uses prediction error to update value estimates after each step, without waiting for the full episode to end.