5.9.2
KL, Batch and Sampling Parameters
Understand KL settings, batch size and generation temperature.
What is it?
The initial KL penalty coefficient controls how strongly PPO penalizes divergence between the current policy and its reference policy. It helps prevent the model from moving too far from its supervised fine-tuned behaviour.
Key points
1. Typical range: 0.01–0.1.
2. The parameter is init_kl_coef.
3. Too low: the model may exploit weaknesses in the reward signal (reward hacking).
4. Too high: the model may remain stuck close to its SFT behaviour.
5. The coefficient regulates the strength of the KL penalty.
Example: Initial KL Penalty
init_kl_coef = 0.05
This sets the initial strength of the KL penalty in the PPO configuration.
A lower coefficient permits more policy movement, while a higher coefficient applies stronger regularization.
Real-world application
During RLHF training, observe both reward improvements and KL divergence. Increasing reward alone is not sufficient if the model drifts excessively from its reference behaviour.
Key takeaway
The KL penalty balances reward improvement against preserving reference-model behaviour.