5.9.5
Learning Rate (learning_rate)
Understand why PPO uses a small learning rate and how an excessive learning rate can cause catastrophic forgetting.
What is it?
The learning rate determines the size of parameter updates during PPO optimization. RLHF typically uses a very small learning rate to avoid destructive changes to the language model.
Key points
1. Typical range: 1–5 × 10⁻⁶.
2. The parameter is learning_rate.
3. Excessively high learning rates can cause catastrophic forgetting.
4. Very small updates support more stable optimization.
5. Learning rate must be considered alongside KL penalty and clipping.
Example: PPO Learning Rate
learning_rate = 1.5e-6
This is the learning rate used in the book’s TRL configuration example.
It is deliberately small to support stable PPO updates.
Real-world application
Monitor reward scores, KL divergence and training loss. If the policy changes too aggressively, reducing the learning rate may help.
Key takeaway
A small learning rate is important for stable language-model reinforcement learning.