OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Rollout Buffer and Rollouts
Lesson 178 of 415 6 min read
5.5.2

The Rollout Buffer

Understand the temporary storage used to hold PPO training experiences.

What is it?

The rollout buffer is a temporary storage structure that holds data collected during the rollout phase. For every generated token or interaction step, it records the state, action, old-policy log-probability, reward and value-function prediction.

Key points

1. The buffer temporarily stores collected rollout data. 2. B = {(s_t, a_t, log π_θold(a_t|s_t), r_t, V(s_t))} from t = 1 to T. 3. s_t represents the state. 4. a_t represents the action taken. 5. r_t represents the reward. 6. The old-policy log-probability is needed for probability-ratio computation. 7. V(s_t) is the value baseline used for GAE advantage computation.

Example: One Rollout Buffer Record

Consider one generated token in an LLM rollout. State: The prompt and tokens generated so far. Action: The next generated token. Old-policy log-probability: −0.45. Reward: 0.2. Value prediction: 0.6. These values form one record in the rollout buffer. Multiple records are collected across generated tokens and trajectories.

Real-world application

The rollout buffer preserves the exact old-policy log-probabilities and value predictions needed during PPO training.
Key takeaway

The rollout buffer stores the information required to calculate probability ratios and GAE advantages.