5.3.2
Value Function Loss
Understand how the critic learns to predict target returns.
What is it?
The value function loss trains the critic to predict target returns. It measures the squared difference between the critic prediction and the target value. The source specifies a coefficient of 0.1 and notes that the value function is also clipped for stability.
Key points
1. The critic predicts the value of a state.
2. The target represents the return to be predicted.
3. Value loss measures squared prediction error.
4. Formula: L_VF = (Vθ(s_t) − V_target_t)².
5. The source specifies c1 = 0.1.
6. Value loss is also clipped for stability.
Example: Critic Prediction Error
Suppose the critic predicts a value of 6 while the target return is 8.
Value loss = (6 − 8)² = 4.
The squared error measures how far the critic prediction is from the target. Training adjusts the critic to reduce this error.
Real-world application
In PPO, the critic helps estimate returns and supports advantage estimation during policy training.
Key takeaway
Value function loss trains the critic to predict returns more accurately.