3.4.1
Understanding TD Error: “Surprise” as a Learning Signal
Understanding how temporal difference error measures the difference between predicted and updated estimates.
What is it?
Temporal Difference (TD) error measures the difference between an agent’s current estimate of future reward and a newly updated estimate after taking one step. It represents the agent’s surprise: the difference between what it expected and what it now expects after observing the outcome.
Key points
1. TD error measures prediction discrepancy.
2. It compares the old estimate with the updated estimate.
3. The updated estimate includes the observed outcome and expected future value.
4. Positive TD error means the outcome was better than predicted.
5. Negative TD error means the outcome was worse than predicted.
6. The error acts as a learning signal for adjusting value estimates.
Example: The unexpected driving delay
Imagine you expect your drive home to take 30 minutes.
Original prediction: 30 minutes total.
After driving for 10 minutes, you encounter unexpected road construction. Your GPS now estimates another 35 minutes.
Updated total estimate = 10 + 35 = 45 minutes.
TD error = 45 − 30 = +15 minutes.
The positive error represents an unexpected increase in the estimated journey time. This surprise can influence your route choice next time.
Real-world application
TD error provides a learning signal in reinforcement learning systems. It helps an agent adjust its estimates when observed outcomes differ from expectations.
Key takeaway
TD error is the agent’s surprise: the difference between its previous prediction and its updated estimate after one step.