5.6.2
Reward and KL Feedback
Understand how reward signals and KL constraints guide policy updates.
What is it?
After responses are generated, a reward model evaluates each prompt-response pair and assigns a score. The example uses reward scores ranging from 0.2 to 0.95. These scores represent the reward model assessment of response quality.
Key points
1. The reward model evaluates each prompt-response pair.
2. Each response receives a reward score.
3. Example score range: 0.2–0.95.
4. Higher scores indicate responses preferred by the reward model.
5. These scores are used to calculate the final training reward.
Example: Scoring Generated Responses
Suppose the reward model evaluates three responses:
Response A: 0.95
Response B: 0.62
Response C: 0.20
Response A receives the highest reward-model score, while Response C receives the lowest.
These scores are not yet the final PPO rewards because the KL penalty is applied in the next steps.
Real-world application
A reward model converts qualitative response preferences into numerical signals that can guide policy optimization.
Key takeaway
The reward model provides a numerical quality score for every generated response.