OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / Algorithm
Lesson 263 of 415 7 min read
7.2.5

PPO-Style Clipped Policy Update and KL Penalty

Understand how normalized group advantages are used in a PPO-style clipped update with a KL penalty against the reference policy.

What is it?

After calculating group-relative advantages, GRPO applies a PPO-style clipped policy update. The objective uses the policy probability ratio, clipping and a KL penalty against the reference policy.

Key points

The policy update uses normalized advantages. The PPO-style objective compares the probability ratio rt(θ) with its clipped value. The clipping parameter ε limits policy updates. The KL penalty βDKL[πθ∥πref] regularizes the policy against the reference policy. The source presents this combined objective as Equation 7.1.

From Group Advantages to Policy Updates

A response with positive advantage encourages the policy to increase its likelihood, subject to the clipped update. A response with negative advantage encourages the policy to reduce its likelihood. The KL penalty helps constrain divergence from the reference policy.

Real-world application

In GRPO training, configure the KL coefficient and policy optimization settings carefully. The source TRL example uses beta = 0.04 as its KL penalty coefficient.
Key takeaway

GRPO converts group-relative advantages into policy updates using PPO-style clipping and a reference-policy KL penalty.