5.9.3
The Effects of Incorrect Hyperparameters
Understand how poor parameter choices can cause instability, reward hacking or weak learning.
What is it?
The target KL parameter defines the target divergence used by an adaptive KL controller. It helps regulate how far the trained policy moves from the reference policy.
Key points
1. Typical range: 4–8.
2. The parameter is target_kl.
3. An adaptive controller adjusts the KL penalty based on observed divergence.
4. A lower target produces more conservative updates.
5. The target helps control policy drift during training.
Example: Target KL
target_kl = 6.0
The adaptive controller uses 6.0 as its target KL divergence.
A lower target encourages the policy to remain closer to the reference model.
Real-world application
Track approximate KL divergence during PPO training. Compare observed divergence with the configured target to understand whether policy updates are becoming too aggressive.
Key takeaway
Target KL provides a reference point for controlling policy divergence.