7.1.3
Why Value Estimation Is Difficult for Language Models
Understand why predicting expected reward for partial language-model sequences is challenging.
What is it?
Predicting expected reward for a partial language-model sequence is difficult. When the critic estimates value incorrectly, the resulting advantages can also be wrong, sending the policy gradient in an undesirable direction.
Key points
The critic predicts expected reward for partial sequences.
This prediction is difficult for language models.
Incorrect value estimates produce incorrect advantages.
Incorrect advantages can lead to the wrong gradient direction.
How an Inaccurate Critic Affects Learning
Suppose the critic overestimates the expected reward of a partial response. The calculated advantage may then incorrectly indicate that the policy action was poor. The resulting update can move the policy in the wrong direction.
Real-world application
When diagnosing unstable PPO training, examine the quality of value estimates and the advantages calculated from them.
Key takeaway
Inaccurate value estimates can create incorrect advantages and misdirect policy updates.