OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Derivation of the PPO Gradient and Update Rule
Lesson 176 of 415 7 min read
5.4.6

Step 6: The Complete PPO Update Rule

Combine policy loss, value loss and entropy bonus in the complete update rule.

What is it?

The complete PPO update combines the clipped policy objective, value function regression loss and entropy bonus. The optimizer updates the policy parameters using the gradient of this combined objective.

Key points

1. PPO combines three components: clipped policy objective, value loss and entropy bonus. 2. Equation (5.10): θ_{k+1} = θ_k + α · ∇θ[L_CLIP(θ) − c1 L_VF(θ) + c2 H[πθ]]. 3. α represents the learning rate. 4. Equation (5.11): L_VF(θ) = (Vθ(s_t) − V_target_t)². 5. Equation (5.12): H[πθ] = −Σ_a πθ(a|s_t) log πθ(a|s_t). 6. Policy gradients provide the improvement direction. 7. Importance sampling supports reuse of old-policy data. 8. Clipping controls extreme importance weights. 9. The minimum operator selects the more conservative objective.

Example: Understanding a Complete PPO Update

Consider a PPO training step. The clipped policy objective encourages better action selection while limiting excessive policy changes. The value loss trains the critic to predict target returns. The entropy bonus encourages exploration. The optimizer combines these three components and updates the policy parameters using their combined gradient.

Real-world application

A complete PPO training implementation must calculate the clipped policy objective, value function loss and entropy bonus before applying the parameter update.
Key takeaway

PPO combines policy improvement, critic learning and exploration in one parameter update rule.