5.9.4
PPO Epochs (ppo_epochs)
Understand how the number of optimization epochs affects reuse of rollout data, overfitting and generation efficiency.
What is it?
PPO epochs determine how many times the trainer reuses the collected rollout batch for optimization. Reusing data improves training efficiency, but excessive reuse can cause overfitting to that batch.
Key points
1. Typical value: 4.
2. The parameter is ppo_epochs.
3. Too many epochs: the policy may overfit to the rollout batch.
4. Too few epochs: generated data and compute may be underutilized.
5. Epoch selection affects the balance between optimization and data reuse.
Example: PPO Epochs
ppo_epochs = 4
The trainer performs four optimization epochs using the collected rollout data.
The goal is to learn effectively without overfitting to the same batch.
Real-world application
If training becomes unstable or performance degrades after updates, examine the number of PPO epochs alongside clipping and learning rate.
Key takeaway
PPO epochs control how extensively each rollout batch is reused.