OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Critical Hyperparameters
Lesson 200 of 415 5 min read
5.9.5

Learning Rate (learning_rate)

Understand why PPO uses a small learning rate and how an excessive learning rate can cause catastrophic forgetting.

What is it?

The learning rate determines the size of parameter updates during PPO optimization. RLHF typically uses a very small learning rate to avoid destructive changes to the language model.

Key points

1. Typical range: 1–5 × 10⁻⁶. 2. The parameter is learning_rate. 3. Excessively high learning rates can cause catastrophic forgetting. 4. Very small updates support more stable optimization. 5. Learning rate must be considered alongside KL penalty and clipping.

Example: PPO Learning Rate

learning_rate = 1.5e-6 This is the learning rate used in the book’s TRL configuration example. It is deliberately small to support stable PPO updates.

Real-world application

Monitor reward scores, KL divergence and training loss. If the policy changes too aggressively, reducing the learning rate may help.
Key takeaway

A small learning rate is important for stable language-model reinforcement learning.