OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / RL Foundations for Language Models / Text Generation as an MDP
Lesson 146 of 415 5 min read
4.2.2

State: The Prompt and Generated Tokens

Understand how the current state consists of the prompt and all tokens generated so far.

What is it?

At generation step t, the state consists of the original prompt x concatenated with all previously generated tokens. It represents the complete context available to the LLM when choosing its next token.

Key points

1. State: sₜ = (x, y₁, ..., yₜ₋₁). 2. x represents the original prompt. 3. y₁ to yₜ₋₁ represent previously generated tokens. 4. The state grows as new tokens are generated. 5. The model uses the complete current state to predict the next token. 6. Each generation step therefore has an updated state.

Example: State changes during generation

Prompt: What is machine learning? Initial state: What is machine learning? After generating the first token: What is machine learning? Machine After generating the next tokens: What is machine learning? Machine learning enables computers... The prompt and all generated tokens together form the current state.

Real-world application

The state representation is important when building token-level reinforcement learning systems, rollout storage and policy optimization.
Key takeaway

The state is the prompt combined with every token generated so far.