4.2.4
Transition, Reward and Policy
Understand deterministic token transitions, reward feedback and the LLM next-token policy.
What is it?
The transition in text generation is deterministic: the selected token is simply appended to the current state. Rewards are typically sparse and given at the end of generation. The LLM policy is its next-token probability distribution, already computed by the softmax output.
Key points
1. Transition: P(sₜ₊₁|sₜ,aₜ).
2. The transition is deterministic.
3. The selected token is appended to the existing sequence.
4. No environment stochasticity is involved in this transition.
5. Reward is typically given at the end of generation.
6. RLHF reward: reward model score.
7. RLVR reward: correctness of the final answer.
8. Policy: πθ(aₜ|sₜ), the LLM next-token probability distribution.
Example: From token selection to reward
Prompt: What is 2 + 2?
Current state: What is 2 + 2?
Action: Select token 4.
Next state: What is 2 + 2? 4
The token is appended deterministically.
After the response is complete, a verifier can check whether the answer is correct and provide the reward.
Real-world application
This mapping allows standard reinforcement learning objectives to be applied to autoregressive text generation.
Key takeaway
The transition appends the selected token; the reward evaluates the generated response; the policy supplies next-token probabilities.