OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Derivation of the PPO Gradient and Update Rule
Lesson 175 of 415 7 min read
5.4.5

Step 5: PPO’s Clipped Surrogate

Understand how clipping provides a first-order approximation to constrained policy updates.

What is it?

PPO replaces the hard KL constraint with a clipped surrogate objective that uses first-order gradients. It compares the unclipped and clipped advantage-weighted probability ratios and selects the minimum. The gradient becomes zero when the clipped term is the more conservative term.

Key points

1. PPO uses a clipped surrogate objective. 2. Equation (5.8): L_CLIP(θ) = E_t[min(r_t(θ)Â_t, clip(r_t(θ), 1−ε, 1+ε)Â_t)]. 3. The clipped ratio is represented by r̄_t. 4. Equation (5.9) gives the piecewise gradient. 5. Positive advantage: gradient flows when r_t < 1 + ε; it becomes zero at or above the upper boundary. 6. Negative advantage: gradient flows when r_t > 1 − ε; it becomes zero at or below the lower boundary. 7. This limits excessive policy changes using first-order gradients.

Example: When PPO Stops Increasing an Action Probability

Assume ε = 0.2, so the upper clipping boundary is 1.2. For a positive advantage, if the probability ratio reaches 1.3, the clipped term becomes the more conservative term. The gradient for that sample becomes zero. PPO stops pushing the probability higher based on that sample.

Real-world application

Clipping controls policy updates during PPO optimization without requiring the expensive second-order methods used by TRPO.
Key takeaway

PPO uses a clipped surrogate and first-order gradients to limit excessive policy changes.