4.2.3
Action: Choosing the Next Token
Understand how selecting a token from the vocabulary represents an action.
What is it?
An action in text generation is the selection of the next token from the model vocabulary. The action space contains all possible token IDs, typically around 32K–128K options according to the source.
Key points
1. Action: aₜ ∈ {1, ..., |V|}.
2. V represents the vocabulary.
3. Each possible token ID is an available action.
4. The model selects one token at each generation step.
5. The vocabulary creates a massive action space.
6. The policy assigns probabilities to possible token actions.
Example: Choosing the next token
Current state: The sky is
The model may assign probabilities such as:
blue: 0.60
cloudy: 0.25
clear: 0.10
other tokens: 0.05
If the model selects blue, that token becomes the next action and is appended to the response.
These probabilities are illustrative, not values from the source.
Real-world application
Understanding the token action space is essential for implementing sampling, policy-gradient updates and exploration strategies in LLM training.
Key takeaway
Every next-token selection is an action chosen from the model vocabulary.