5.4.6
Step 6: The Complete PPO Update Rule
Combine policy loss, value loss and entropy bonus in the complete update rule.
What is it?
The complete PPO update combines the clipped policy objective, value function regression loss and entropy bonus. The optimizer updates the policy parameters using the gradient of this combined objective.
Key points
1. PPO combines three components: clipped policy objective, value loss and entropy bonus.
2. Equation (5.10): θ_{k+1} = θ_k + α · ∇θ[L_CLIP(θ) − c1 L_VF(θ) + c2 H[πθ]].
3. α represents the learning rate.
4. Equation (5.11): L_VF(θ) = (Vθ(s_t) − V_target_t)².
5. Equation (5.12): H[πθ] = −Σ_a πθ(a|s_t) log πθ(a|s_t).
6. Policy gradients provide the improvement direction.
7. Importance sampling supports reuse of old-policy data.
8. Clipping controls extreme importance weights.
9. The minimum operator selects the more conservative objective.
Example: Understanding a Complete PPO Update
Consider a PPO training step.
The clipped policy objective encourages better action selection while limiting excessive policy changes.
The value loss trains the critic to predict target returns.
The entropy bonus encourages exploration.
The optimizer combines these three components and updates the policy parameters using their combined gradient.
Real-world application
A complete PPO training implementation must calculate the clipped policy objective, value function loss and entropy bonus before applying the parameter update.
Key takeaway
PPO combines policy improvement, critic learning and exploration in one parameter update rule.