5.5.1
What Is a Rollout?
Understand a rollout as a trajectory generated by the current policy.
What is it?
A rollout, also called a trajectory, is a sequence of interactions generated by an agent following its current policy. The agent observes a state, selects an action, receives a reward and moves to the next state. In LLM reinforcement learning, a rollout consists of generating tokens from a prompt until an end-of-text marker is reached.
Key points
1. A rollout is a sequence of agent-environment interactions.
2. The agent observes a state and selects an action.
3. The environment returns a reward and next state.
4. A rollout continues for a fixed number of steps or until the episode ends.
5. In LLMs, a prompt is used to generate a sequence of tokens.
6. Each generated token represents one step.
7. Generation continues until an end-of-text marker is reached.
Example: An LLM Rollout
A dataset provides the prompt: Explain how photosynthesis works.
The language model generates the response token by token.
Token 1: Photosynthesis
Token 2: is
Token 3: the
Token 4: process
Token 5: ...
The generation continues until the end-of-text marker is reached. The complete generated sequence represents a rollout.
Real-world application
In RLHF training, rollouts provide generated responses that can be evaluated by a reward model and used for policy optimization.
Key takeaway
A rollout is a complete trajectory generated by an agent using its current policy.