5.6.4
Step 4: Calculate Final Reward
Understand how the reward model score is adjusted using the KL penalty.
What is it?
The final reward combines the reward-model score with a KL penalty. The example uses a KL coefficient of 0.05 and places the resulting sequence-level reward only at the final token.
Key points
1. Reward-model score: r_RM.
2. KL penalty coefficient: 0.05.
3. Final reward: R = r_RM − 0.05 × mean_KL.
4. The mean KL is calculated across generated tokens.
5. The final reward is assigned only at the last token in this example.
6. The KL penalty discourages excessive deviation from the reference policy.
Example: Calculating Final Reward
Assume:
Reward-model score = 0.80
Mean KL = 4
KL coefficient = 0.05
Final reward = 0.80 − (0.05 × 4)
Final reward = 0.60
The resulting reward is placed at the last token of the generated response.
Real-world application
The KL-adjusted reward balances response quality with the requirement to keep the policy close to its reference model.
Key takeaway
The final reward rewards quality while penalizing excessive policy drift.