4.2.2
State: The Prompt and Generated Tokens
Understand how the current state consists of the prompt and all tokens generated so far.
What is it?
At generation step t, the state consists of the original prompt x concatenated with all previously generated tokens. It represents the complete context available to the LLM when choosing its next token.
Key points
1. State: sₜ = (x, y₁, ..., yₜ₋₁).
2. x represents the original prompt.
3. y₁ to yₜ₋₁ represent previously generated tokens.
4. The state grows as new tokens are generated.
5. The model uses the complete current state to predict the next token.
6. Each generation step therefore has an updated state.
Example: State changes during generation
Prompt: What is machine learning?
Initial state:
What is machine learning?
After generating the first token:
What is machine learning? Machine
After generating the next tokens:
What is machine learning? Machine learning enables computers...
The prompt and all generated tokens together form the current state.
Real-world application
The state representation is important when building token-level reinforcement learning systems, rollout storage and policy optimization.
Key takeaway
The state is the prompt combined with every token generated so far.