5.5.2
The Rollout Buffer
Understand the temporary storage used to hold PPO training experiences.
What is it?
The rollout buffer is a temporary storage structure that holds data collected during the rollout phase. For every generated token or interaction step, it records the state, action, old-policy log-probability, reward and value-function prediction.
Key points
1. The buffer temporarily stores collected rollout data.
2. B = {(s_t, a_t, log π_θold(a_t|s_t), r_t, V(s_t))} from t = 1 to T.
3. s_t represents the state.
4. a_t represents the action taken.
5. r_t represents the reward.
6. The old-policy log-probability is needed for probability-ratio computation.
7. V(s_t) is the value baseline used for GAE advantage computation.
Example: One Rollout Buffer Record
Consider one generated token in an LLM rollout.
State: The prompt and tokens generated so far.
Action: The next generated token.
Old-policy log-probability: −0.45.
Reward: 0.2.
Value prediction: 0.6.
These values form one record in the rollout buffer. Multiple records are collected across generated tokens and trajectories.
Real-world application
The rollout buffer preserves the exact old-policy log-probabilities and value predictions needed during PPO training.
Key takeaway
The rollout buffer stores the information required to calculate probability ratios and GAE advantages.