3.7.3
Critic Update Using TD Error
Understand how the critic learns value estimates by minimizing temporal difference error.
What is it?
The critic learns to estimate state values by minimizing the squared temporal difference error. It compares the observed reward plus discounted next-state value with its current state-value estimate. This trains the critic to provide better feedback to the actor.
Key points
1. Critic value function: Vϕ(s).
2. TD error: δₜ = rₜ + γVϕ(sₜ₊₁) − Vϕ(sₜ).
3. The first two terms form the TD target.
4. The current state value is the original estimate.
5. Critic loss: Lcritic = E[(rₜ + γVϕ(sₜ₊₁) − Vϕ(sₜ))²].
6. The critic minimizes squared TD error.
7. A better critic provides more useful advantage estimates to the actor.
Example: Training the critic
Suppose:
Immediate reward = 4
Discount factor γ = 0.9
Next-state value = 10
Current-state value = 8
TD target = 4 + (0.9 × 10) = 13
TD error = 13 − 8 = 5
Squared TD error = 25
The critic updates its parameters to reduce this prediction error.
Real-world application
The critic loss provides a training objective for value networks used in actor-critic reinforcement learning and PPO-based LLM training.
Key takeaway
The critic learns by minimizing squared TD error so that its value estimates become more accurate.