OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Critical Hyperparameters
Lesson 197 of 415 5 min read
5.9.2

KL, Batch and Sampling Parameters

Understand KL settings, batch size and generation temperature.

What is it?

The initial KL penalty coefficient controls how strongly PPO penalizes divergence between the current policy and its reference policy. It helps prevent the model from moving too far from its supervised fine-tuned behaviour.

Key points

1. Typical range: 0.01–0.1. 2. The parameter is init_kl_coef. 3. Too low: the model may exploit weaknesses in the reward signal (reward hacking). 4. Too high: the model may remain stuck close to its SFT behaviour. 5. The coefficient regulates the strength of the KL penalty.

Example: Initial KL Penalty

init_kl_coef = 0.05 This sets the initial strength of the KL penalty in the PPO configuration. A lower coefficient permits more policy movement, while a higher coefficient applies stronger regularization.

Real-world application

During RLHF training, observe both reward improvements and KL divergence. Increasing reward alone is not sufficient if the model drifts excessively from its reference behaviour.
Key takeaway

The KL penalty balances reward improvement against preserving reference-model behaviour.