OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / PPO for RLHF: The Full Loop
Lesson 185 of 415 6 min read
5.6.6

Step 6: Update the Policy

Understand how PPO updates policy weights using multiple epochs, mini-batches and a clipped probability ratio.

What is it?

The final step updates the policy using PPO optimization. The example performs four epochs of stochastic gradient descent using mini-batches of 16, a probability-ratio clipping value of 0.2 and gradient norm clipping at 1.0.

Key points

1. Optimization method: stochastic gradient descent (SGD). 2. PPO epochs: 4. 3. Mini-batch size: 16. 4. Probability-ratio clipping: ε = 0.2. 5. Gradient norm clipping: 1.0. 6. The clipped objective limits excessive policy changes. 7. The updated policy is used to generate responses in the next training cycle. 8. The example reports approximately 0.005 win-rate improvement per step and 5–10% absolute improvement over SFT after 10K steps.

Example: Updating a PPO Policy

The rollout data is divided into mini-batches of 16. The optimizer processes the data for four epochs. During optimization, PPO uses a clipping value of 0.2 to limit excessive changes in action probabilities. Gradient norm clipping is set to 1.0. After the update, the improved policy is used to generate fresh responses for the next training cycle.

Real-world application

These parameters provide a concrete reference configuration for implementing PPO-based RLHF training on a large language model.
Key takeaway

PPO completes the RLHF cycle by updating policy weights using GAE advantages and a clipped objective.