OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / RL Foundations for Language Models / Text Generation as an MDP
Lesson 147 of 415 5 min read
4.2.3

Action: Choosing the Next Token

Understand how selecting a token from the vocabulary represents an action.

What is it?

An action in text generation is the selection of the next token from the model vocabulary. The action space contains all possible token IDs, typically around 32K–128K options according to the source.

Key points

1. Action: aₜ ∈ {1, ..., |V|}. 2. V represents the vocabulary. 3. Each possible token ID is an available action. 4. The model selects one token at each generation step. 5. The vocabulary creates a massive action space. 6. The policy assigns probabilities to possible token actions.

Example: Choosing the next token

Current state: The sky is The model may assign probabilities such as: blue: 0.60 cloudy: 0.25 clear: 0.10 other tokens: 0.05 If the model selects blue, that token becomes the next action and is appended to the response. These probabilities are illustrative, not values from the source.

Real-world application

Understanding the token action space is essential for implementing sampling, policy-gradient updates and exploration strategies in LLM training.
Key takeaway

Every next-token selection is an action chosen from the model vocabulary.