5.2.2
The Clipping Range and Surrogate Objective
Understand how clipping limits the probability ratio within a defined range.
What is it?
PPO uses a clipped surrogate objective to limit excessive policy updates. The probability ratio is clipped between 1 − ε and 1 + ε. The minimum operator selects the more conservative objective value.
Key points
1. Clipping range: [1 − ε, 1 + ε].
2. ε controls the clipping width.
3. With ε = 0.2, the range is [0.8, 1.2].
4. The objective compares clipped and unclipped terms.
5. The minimum operator selects the more conservative term.
6. Clipping helps prevent excessively large policy updates.
Example: Applying a 20% clipping range
Suppose ε = 0.2.
The lower clipping boundary is 1 − 0.2 = 0.8.
The upper clipping boundary is 1 + 0.2 = 1.2.
If the probability ratio is 1.35, its clipped value is 1.2.
If the probability ratio is 0.65, its clipped value is 0.8.
The PPO objective uses the minimum of the clipped and unclipped advantage-weighted terms.
Real-world application
The clipping range is an important PPO training configuration. It helps control how aggressively a policy changes during optimization.
Key takeaway
PPO uses clipping and a minimum operator to create a conservative surrogate objective.