OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / PPO for RLHF: The Full Loop
Lesson 182 of 415 5 min read
5.6.3

Repeating the PPO Training Cycle

Understand how rollout collection and policy optimization repeat throughout training.

What is it?

PPO calculates the per-token KL divergence between the current policy and a fixed reference policy. This measures how much the updated policy distribution differs from the reference model distribution.

Key points

1. KL divergence is calculated for each generated token. 2. KL_t = log π_θ(y_t | y_

Example: Measuring Policy Divergence

For a generated token: Current policy log-probability = −0.4 Reference policy log-probability = −0.6 KL_t = −0.4 − (−0.6) = 0.2 This token has a positive log-probability difference of 0.2 under the stated calculation.

Real-world application

Per-token KL measurements help monitor how far an RLHF policy moves away from its reference model during training.
Key takeaway

KL divergence measures policy deviation from the reference model and provides the basis for a regularization penalty.