OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / Introduction to Reinforcement Learning / Temporal Difference (TD) Learning
Lesson 117 of 415 8 min read
3.4.3

How the Agent Uses TD Error

Understanding value updates and how the agent uses TD error during learning.

What is it?

The agent uses TD error to adjust its value function and gradually drive prediction errors toward zero. The update rule is V(Sₜ) ← V(Sₜ) + α · δₜ, where α is the learning rate. TD learning updates after each step instead of waiting for the complete episode to finish.

Key points

1. Value update: V(Sₜ) ← V(Sₜ) + α · δₜ. 2. α represents the learning rate. 3. If δₜ > 0, increase the current state value. 4. If δₜ < 0, decrease the current state value. 5. If δₜ = 0, no update is needed. 6. Monte Carlo waits until the episode ends and uses actual return Gₜ. 7. TD updates after every step using estimated future value. 8. TD(λ) interpolates between TD(0) and Monte Carlo. 9. Multi-step TD uses several rewards before bootstrapping from a future value estimate.

Example: Updating a state value

Suppose: Current value V(Sₜ) = 10 TD error δₜ = 8 Learning rate α = 0.1 Updated value: V(Sₜ) = 10 + (0.1 × 8) V(Sₜ) = 10.8 The agent increases its estimate from 10 to 10.8 because the outcome was better than predicted. Comparison of learning approaches: Monte Carlo: Waits until the episode ends and uses actual return. TD: Updates after every step using estimated future value. TD(λ): Interpolates between TD(0) and Monte Carlo. λ = 0 represents pure TD; λ = 1 represents pure Monte Carlo. Multi-step TD: Uses multiple rewards before adding a discounted future value estimate.

Real-world application

TD updates are used in value estimation, Q-learning, Deep Q-Networks and actor-critic methods. TD(λ) also provides the foundation for the bias-variance trade-off used in Generalized Advantage Estimation (GAE).
Key takeaway

TD learning uses prediction error to update value estimates after each step, without waiting for the full episode to end.