5.6.6
Step 6: Update the Policy
Understand how PPO updates policy weights using multiple epochs, mini-batches and a clipped probability ratio.
What is it?
The final step updates the policy using PPO optimization. The example performs four epochs of stochastic gradient descent using mini-batches of 16, a probability-ratio clipping value of 0.2 and gradient norm clipping at 1.0.
Key points
1. Optimization method: stochastic gradient descent (SGD).
2. PPO epochs: 4.
3. Mini-batch size: 16.
4. Probability-ratio clipping: ε = 0.2.
5. Gradient norm clipping: 1.0.
6. The clipped objective limits excessive policy changes.
7. The updated policy is used to generate responses in the next training cycle.
8. The example reports approximately 0.005 win-rate improvement per step and 5–10% absolute improvement over SFT after 10K steps.
Example: Updating a PPO Policy
The rollout data is divided into mini-batches of 16.
The optimizer processes the data for four epochs.
During optimization, PPO uses a clipping value of 0.2 to limit excessive changes in action probabilities. Gradient norm clipping is set to 1.0.
After the update, the improved policy is used to generate fresh responses for the next training cycle.
Real-world application
These parameters provide a concrete reference configuration for implementing PPO-based RLHF training on a large language model.
Key takeaway
PPO completes the RLHF cycle by updating policy weights using GAE advantages and a clipped objective.