5.6.3
Repeating the PPO Training Cycle
Understand how rollout collection and policy optimization repeat throughout training.
What is it?
PPO calculates the per-token KL divergence between the current policy and a fixed reference policy. This measures how much the updated policy distribution differs from the reference model distribution.
Key points
1. KL divergence is calculated for each generated token.
2. KL_t = log π_θ(y_t | y_
Example: Measuring Policy Divergence
For a generated token:
Current policy log-probability = −0.4
Reference policy log-probability = −0.6
KL_t = −0.4 − (−0.6) = 0.2
This token has a positive log-probability difference of 0.2 under the stated calculation.
Real-world application
Per-token KL measurements help monitor how far an RLHF policy moves away from its reference model during training.
Key takeaway
KL divergence measures policy deviation from the reference model and provides the basis for a regularization penalty.