OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Rollout Buffer and Rollouts
Lesson 177 of 415 5 min read
5.5.1

What Is a Rollout?

Understand a rollout as a trajectory generated by the current policy.

What is it?

A rollout, also called a trajectory, is a sequence of interactions generated by an agent following its current policy. The agent observes a state, selects an action, receives a reward and moves to the next state. In LLM reinforcement learning, a rollout consists of generating tokens from a prompt until an end-of-text marker is reached.

Key points

1. A rollout is a sequence of agent-environment interactions. 2. The agent observes a state and selects an action. 3. The environment returns a reward and next state. 4. A rollout continues for a fixed number of steps or until the episode ends. 5. In LLMs, a prompt is used to generate a sequence of tokens. 6. Each generated token represents one step. 7. Generation continues until an end-of-text marker is reached.

Example: An LLM Rollout

A dataset provides the prompt: Explain how photosynthesis works. The language model generates the response token by token. Token 1: Photosynthesis Token 2: is Token 3: the Token 4: process Token 5: ... The generation continues until the end-of-text marker is reached. The complete generated sequence represents a rollout.

Real-world application

In RLHF training, rollouts provide generated responses that can be evaluated by a reward model and used for policy optimization.
Key takeaway

A rollout is a complete trajectory generated by an agent using its current policy.