4.2.5
Discount Factor and Learning the Policy
Understand why the discount factor is 1.0 for finite responses and how model weights are optimized.
What is it?
The discount factor for text generation is γ = 1.0 because each episode is finite and represents one response. The LLM already acts as the policy through its softmax output, so reinforcement learning adjusts the model weights θ to increase the probability of token sequences that earn higher rewards.
Key points
1. Discount factor: γ = 1.0.
2. An episode is one finite response.
3. No discounting of future rewards is needed in this formulation.
4. The LLM already provides the policy.
5. Softmax output defines πθ(aₜ|sₜ).
6. No separate policy network is required.
7. Training adjusts model weights θ.
8. Higher-reward token sequences should receive higher probability.
Example: Improving a response policy
An LLM generates two possible answers to a question.
Answer A receives a reward of 0.9.
Answer B receives a reward of 0.3.
During policy optimization, the model weights are adjusted so that token sequences associated with better outcomes become more likely.
Because one response is a finite episode, the source uses γ = 1.0.
Real-world application
This is the foundation for PPO and GRPO training: optimize the existing language model policy rather than building a separate policy network.
Key takeaway
The LLM already is the policy. Reinforcement learning adjusts its weights to favour higher-reward token sequences.