5.3.1
The Policy Clipped Loss
Understand the policy optimization component of the full PPO objective.
What is it?
The clipped policy loss is the main component of PPO. It improves the policy using the estimated advantage while clipping the probability ratio to prevent excessively large policy updates.
Key points
1. Uses the probability ratio between new and old policies.
2. Uses the estimated advantage to evaluate actions.
3. Clips the probability ratio between 1 − ε and 1 + ε.
4. Uses the minimum operator to select the more conservative objective.
5. Helps maintain stable policy updates.
Example: Clipped Policy Objective
Assume the probability ratio is 1.35, the advantage is +2 and ε = 0.2.
The clipped ratio becomes 1.2.
Unclipped objective = 1.35 × 2 = 2.7.
Clipped objective = 1.2 × 2 = 2.4.
The minimum value is 2.4. This limits the benefit of an excessive probability increase.
Real-world application
During PPO training, the clipped policy loss controls how aggressively the model changes its action probabilities.
Key takeaway
The clipped policy loss improves the policy while limiting excessive updates.