5.2.3
Clipping Intuition for Good and Bad Actions
Understand how clipping limits excessive increases or decreases in action probability.
What is it?
The clipping behaviour depends on whether the estimated advantage is positive or negative. For good actions, PPO limits excessive probability increases. For bad actions, it limits excessive probability decreases.
Key points
1. Positive advantage means the action performed better than expected.
2. PPO encourages increasing the probability of good actions.
3. The benefit is capped at r = 1 + ε.
4. Negative advantage means the action performed worse than expected.
5. PPO encourages decreasing the probability of bad actions.
6. The benefit is capped at r = 1 − ε.
7. The minimum operator selects the more conservative objective.
Example: Good and bad action clipping
Assume ε = 0.2.
Good action: Advantage = +2 and probability ratio = 1.35.
The clipped ratio is 1.2. The objective uses the smaller of 1.35 × 2 and 1.2 × 2, which is 2.4.
Bad action: Advantage = −2 and probability ratio = 0.65.
The clipped ratio is 0.8. The objective uses the smaller of 0.65 × (−2) and 0.8 × (−2), which is −1.6.
This demonstrates how PPO limits the benefit of excessive probability changes.
Real-world application
When diagnosing PPO training, examine positive- and negative-advantage samples separately to understand how clipping affects policy updates.
Key takeaway
PPO prevents excessive increases for good actions and excessive decreases for bad actions through conservative clipping.