OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Detailed Mechanics: Logits and Policy Updates
Lesson 187 of 415 7 min read
5.7.2

Phase 2: Optimization Loop (Mini-Batch Updates)

Understand how stored rollout data is used for repeated mini-batch optimization.

What is it?

After the rollout buffer is full, PPO performs multiple optimization epochs over mini-batches. It evaluates the frozen old policy and the live updating policy on stored states. The old-policy log-probability can be reused from the rollout buffer, while the live policy is evaluated again as its weights change.

Key points

1. PPO runs K epochs, typically 3–10. 2. Training operates on mini-batches from the rollout buffer. 3. The old policy remains frozen. 4. The stored old-policy log-probability can be reused. 5. The live policy produces new logits and log-probabilities. 6. Live policy weights change after every mini-batch gradient step. 7. Old-policy outputs remain static throughout the optimization cycle.

Example: Comparing Two Policy Evaluations

A stored state is evaluated during PPO training. Old policy: The old-policy log-probability is retrieved from the rollout buffer. Live policy: The current network performs a forward pass and calculates a new log-probability. The two values are compared to calculate the PPO probability ratio.

Real-world application

Reusing stored old-policy log-probabilities avoids repeating a forward pass through the frozen network for every mini-batch, which is particularly useful for large models.
Key takeaway

During optimization, the old policy stays frozen while the live policy changes.