OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Detailed Mechanics: Logits and Policy Updates
Lesson 186 of 415 7 min read
5.7.1

Phase 1: Rollout (Data Collection)

Understand how the old policy generates responses and records rollout data.

What is it?

During rollout, the frozen old policy interacts with the environment to generate actions. For an LLM, the current state consists of the prompt and tokens generated so far. The network produces logits, converts them into probabilities using Softmax, samples the next token and stores the transition with its old-policy log-probability.

Key points

1. The environment provides the current state s_t. 2. The state is passed through the frozen old policy θ_old. 3. The network produces raw logits z_old. 4. Logits are converted into probabilities using Softmax. 5. The next token is sampled from the probability distribution. 6. The transition tuple and old-policy log-probability are stored in the rollout buffer. 7. The vocabulary size may range from 32K to 128K tokens.

Example: Sampling the Next Token

Suppose an LLM receives the prompt: Artificial intelligence is. The old policy produces logits for its vocabulary. Softmax converts these logits into token probabilities. For illustration: Token A: 0.50 Token B: 0.30 Token C: 0.20 The model samples the next token from this distribution. The selected action, transition and old-policy log-probability are stored in the rollout buffer.

Real-world application

During RLHF data collection, the rollout system generates token sequences and stores the exact log-probabilities needed for subsequent PPO optimization.
Key takeaway

Rollout uses the frozen old policy to generate actions and preserve their log-probabilities for training.