4.2.1
The LLM-as-Agent Analogy
Understand how an LLM acts as an agent generating a response one token at a time.
What is it?
An LLM can be viewed as an agent that writes a response one token at a time. At every step, it considers everything written so far, chooses the next token and extends the response. Once generation finishes, a judge evaluates the complete response and provides a reward.
Key points
1. The LLM acts as the agent.
2. It generates one token at a time.
3. Everything written so far represents the current state.
4. Selecting the next token is an action.
5. Appending the token creates the next state.
6. A judge scores the completed response.
7. The goal is to learn a writing strategy that consistently earns higher rewards.
Example: An AI writing an answer
Prompt: Explain artificial intelligence.
Step 1: The LLM reads the prompt.
Step 2: It selects the first response token.
Step 3: It selects another token using the updated context.
Step 4: It continues until the response is complete.
Step 5: A judge evaluates the answer and assigns a score.
The model learns to make better token-selection decisions through reward feedback.
Real-world application
This analogy helps developers understand how reinforcement learning can optimize language generation rather than only training a model to imitate demonstrations.
Key takeaway
An LLM can be treated as an agent that selects tokens sequentially and receives feedback on its completed response.