5.5.3
The Rollout Buffer Lifecycle
Understand how rollout data is collected, stored, consumed and replaced.
What is it?
The rollout buffer follows three phases: Collect, Train and Purge. Fresh trajectories are collected, GAE advantages are calculated and policy weights are updated over multiple mini-batch epochs. The buffer is then completely cleared because PPO is an on-policy algorithm.
Key points
1. Collect: The active policy generates fresh trajectories.
2. Train: Compute GAE advantages across trajectories.
3. Run K epochs, typically 3–10, of mini-batch gradient descent.
4. Use the clipped PPO objective to update policy weights.
5. Purge: Completely clear the buffer.
6. Old-policy data cannot safely be reused in the next update cycle.
7. The book gives an example of up to 65K token-level transitions for a 70B model with batch 128 and maximum 512 tokens.
8. Fresh generation creates a significant training bottleneck.
Example: The Three-Phase Buffer Cycle
Collect: The current policy generates 128 responses, each with a maximum of 512 tokens.
Train: The system computes GAE advantages and performs several mini-batch optimization epochs using the clipped PPO objective.
Purge: After training, the collected trajectories are discarded. The updated policy must generate fresh trajectories for the next cycle.
This process repeats throughout PPO training.
Real-world application
In an RLHF training pipeline, generation workers collect trajectories, training workers optimize the policy, and the rollout buffer is cleared before the next collection cycle.
Key takeaway
PPO uses a temporary on-policy buffer that follows Collect, Train and Purge. Fresh trajectories are required for every update cycle.