5.4.5
Step 5: PPO’s Clipped Surrogate
Understand how clipping provides a first-order approximation to constrained policy updates.
What is it?
PPO replaces the hard KL constraint with a clipped surrogate objective that uses first-order gradients. It compares the unclipped and clipped advantage-weighted probability ratios and selects the minimum. The gradient becomes zero when the clipped term is the more conservative term.
Key points
1. PPO uses a clipped surrogate objective.
2. Equation (5.8): L_CLIP(θ) = E_t[min(r_t(θ)Â_t, clip(r_t(θ), 1−ε, 1+ε)Â_t)].
3. The clipped ratio is represented by r̄_t.
4. Equation (5.9) gives the piecewise gradient.
5. Positive advantage: gradient flows when r_t < 1 + ε; it becomes zero at or above the upper boundary.
6. Negative advantage: gradient flows when r_t > 1 − ε; it becomes zero at or below the lower boundary.
7. This limits excessive policy changes using first-order gradients.
Example: When PPO Stops Increasing an Action Probability
Assume ε = 0.2, so the upper clipping boundary is 1.2.
For a positive advantage, if the probability ratio reaches 1.3, the clipped term becomes the more conservative term.
The gradient for that sample becomes zero. PPO stops pushing the probability higher based on that sample.
Real-world application
Clipping controls policy updates during PPO optimization without requiring the expensive second-order methods used by TRPO.
Key takeaway
PPO uses a clipped surrogate and first-order gradients to limit excessive policy changes.