OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / Introduction to Reinforcement Learning / Actor-Critic Methods
Lesson 126 of 415 7 min read
3.7.3

Critic Update Using TD Error

Understand how the critic learns value estimates by minimizing temporal difference error.

What is it?

The critic learns to estimate state values by minimizing the squared temporal difference error. It compares the observed reward plus discounted next-state value with its current state-value estimate. This trains the critic to provide better feedback to the actor.

Key points

1. Critic value function: Vϕ(s). 2. TD error: δₜ = rₜ + γVϕ(sₜ₊₁) − Vϕ(sₜ). 3. The first two terms form the TD target. 4. The current state value is the original estimate. 5. Critic loss: Lcritic = E[(rₜ + γVϕ(sₜ₊₁) − Vϕ(sₜ))²]. 6. The critic minimizes squared TD error. 7. A better critic provides more useful advantage estimates to the actor.

Example: Training the critic

Suppose: Immediate reward = 4 Discount factor γ = 0.9 Next-state value = 10 Current-state value = 8 TD target = 4 + (0.9 × 10) = 13 TD error = 13 − 8 = 5 Squared TD error = 25 The critic updates its parameters to reduce this prediction error.

Real-world application

The critic loss provides a training objective for value networks used in actor-critic reinforcement learning and PPO-based LLM training.
Key takeaway

The critic learns by minimizing squared TD error so that its value estimates become more accurate.