3.6.3
Reward-Weighted Maximum Likelihood
Understand how actions from high-reward trajectories receive increased probability.
What is it?
REINFORCE can be understood as reward-weighted maximum likelihood. The policy gradient term increases the probability of a selected action, while its return determines how strongly that action should be reinforced.
Key points
1. ∇θ log πθ(aₜ|sₜ) points in the direction that increases action probability.
2. Multiplication by Gₜ weights the update by return.
3. High-return trajectories increase the probability of their selected actions.
4. Low-return trajectories receive weaker or negative updates when using a baseline.
5. Actions act like training labels.
6. Returns provide the quality weights for those actions.
Example: Learning from successful and unsuccessful responses
Imagine an AI assistant generates two responses.
Response A receives a high reward because it correctly answers the question.
Response B receives a low reward because it contains incorrect information.
The policy update increases the probability of actions associated with the successful response and reduces the relative preference for poorly performing actions.
This resembles supervised learning, but the training actions are weighted by their observed returns.
Real-world application
Reward-weighted policy optimization is useful in reinforcement learning from human feedback and other reward-driven language-model training approaches.
Key takeaway
REINFORCE behaves like maximum-likelihood learning in which the actions are weighted by how well their outcomes performed.