7.5.6.1
The Brittleness of Hard Clipping
Understand why PPO-style hard clipping creates discontinuous gradients and sensitivity near clipping boundaries.
What is it?
PPO-style hard clipping creates a discontinuous gradient. The gradient becomes zero outside the clipping band and remains non-zero inside it. This sharp boundary can cause instability and makes training sensitive to the clipping parameter epsilon.
Key points
Hard clipping creates a sharp gradient boundary.|Gradients are zero outside the clip band.|Instability can occur near the boundary.|Training becomes sensitive to epsilon.
A gradient cliff at the clipping boundary
Imagine a policy update whose probability ratio moves just beyond the clipping boundary. The clipped objective can stop providing a gradient for that update, even though the policy may still need controlled adjustment.
Real-world application
Recognise clipping-boundary instability when diagnosing PPO-style policy optimisation.
Key takeaway
Hard clipping can create abrupt changes in gradient behaviour.