5.8.3
Running a PPO Training Loop
Understand response generation, reward scoring and PPO updates in an implementation.
What is it?
PPOConfig defines the hyperparameters that control the PPO training process. These settings determine the learning rate, rollout batch size, mini-batch size, number of epochs, discount factor, GAE lambda, clipping, value loss, KL control and gradient accumulation.
Key points
1. learning_rate = 1.5e-6.
2. batch_size = 128 prompts per step.
3. mini_batch_size = 16.
4. ppo_epochs = 4.
5. gamma = 1.0 for single-turn tasks.
6. lam = 0.95 for GAE.
7. cliprange = 0.2 and cliprange_value = 0.2.
8. vf_coef = 0.1.
9. init_kl_coef = 0.05 and target_kl = 6.0.
10. whiten_rewards = True.
11. gradient_accumulation_steps = 4.
12. max_grad_norm = 1.0.
Example: Configuring PPO
from trl import PPOConfig
ppo_config = PPOConfig(
learning_rate=1.5e-6,
batch_size=128,
mini_batch_size=16,
ppo_epochs=4,
gamma=1.0,
lam=0.95,
cliprange=0.2,
cliprange_value=0.2,
vf_coef=0.1,
init_kl_coef=0.05,
target_kl=6.0,
whiten_rewards=True,
gradient_accumulation_steps=4,
max_grad_norm=1.0
)
These values form the example PPO training configuration in the book.
Real-world application
PPO configuration should be selected according to model size, available compute, reward behaviour and training stability. Monitor KL divergence, rewards and gradient norms when tuning these parameters.
Key takeaway
PPOConfig controls the optimization behaviour and stability of the training process.