OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Rollout Buffer and Rollouts
Lesson 179 of 415 6 min read
5.5.3

The Rollout Buffer Lifecycle

Understand how rollout data is collected, stored, consumed and replaced.

What is it?

The rollout buffer follows three phases: Collect, Train and Purge. Fresh trajectories are collected, GAE advantages are calculated and policy weights are updated over multiple mini-batch epochs. The buffer is then completely cleared because PPO is an on-policy algorithm.

Key points

1. Collect: The active policy generates fresh trajectories. 2. Train: Compute GAE advantages across trajectories. 3. Run K epochs, typically 3–10, of mini-batch gradient descent. 4. Use the clipped PPO objective to update policy weights. 5. Purge: Completely clear the buffer. 6. Old-policy data cannot safely be reused in the next update cycle. 7. The book gives an example of up to 65K token-level transitions for a 70B model with batch 128 and maximum 512 tokens. 8. Fresh generation creates a significant training bottleneck.

Example: The Three-Phase Buffer Cycle

Collect: The current policy generates 128 responses, each with a maximum of 512 tokens. Train: The system computes GAE advantages and performs several mini-batch optimization epochs using the clipped PPO objective. Purge: After training, the collected trajectories are discarded. The updated policy must generate fresh trajectories for the next cycle. This process repeats throughout PPO training.

Real-world application

In an RLHF training pipeline, generation workers collect trajectories, training workers optimize the policy, and the rollout buffer is cleared before the next collection cycle.
Key takeaway

PPO uses a temporary on-policy buffer that follows Collect, Train and Purge. Fresh trajectories are required for every update cycle.