7.2.5
PPO-Style Clipped Policy Update and KL Penalty
Understand how normalized group advantages are used in a PPO-style clipped update with a KL penalty against the reference policy.
What is it?
After calculating group-relative advantages, GRPO applies a PPO-style clipped policy update. The objective uses the policy probability ratio, clipping and a KL penalty against the reference policy.
Key points
The policy update uses normalized advantages.
The PPO-style objective compares the probability ratio rt(θ) with its clipped value.
The clipping parameter ε limits policy updates.
The KL penalty βDKL[πθ∥πref] regularizes the policy against the reference policy.
The source presents this combined objective as Equation 7.1.
From Group Advantages to Policy Updates
A response with positive advantage encourages the policy to increase its likelihood, subject to the clipped update. A response with negative advantage encourages the policy to reduce its likelihood. The KL penalty helps constrain divergence from the reference policy.
Real-world application
In GRPO training, configure the KL coefficient and policy optimization settings carefully. The source TRL example uses beta = 0.04 as its KL penalty coefficient.
Key takeaway
GRPO converts group-relative advantages into policy updates using PPO-style clipping and a reference-policy KL penalty.