5.9.1
Policy Update Hyperparameters
Understand clip range, learning rate and PPO epochs.
What is it?
The PPO clipping range controls how much the probability of an action can change during a policy update. PPO uses clipping to prevent excessively large updates.
Key points
1. Typical value: 0.2.
2. The clipping range is represented by epsilon (ε).
3. Too low: learning may become ineffective.
4. Too high: policy updates may become unstable.
5. A value of 0.2 limits the probability ratio to the interval [0.8, 1.2].
Example: PPO Clipping Range
cliprange = 0.2
The probability ratio is clipped to:
[1 - 0.2, 1 + 0.2]
Therefore:
[0.8, 1.2]
This limits how aggressively the policy can change.
Real-world application
When tuning PPO, monitor policy updates and reward behaviour. If clipping is too restrictive, learning may slow. If it is too permissive, updates can become unstable.
Key takeaway
The clipping range balances learning progress and policy stability.