OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / Introduction to Reinforcement Learning / Policy Gradient Methods — REINFORCE
Lesson 122 of 415 5 min read
3.6.3

Reward-Weighted Maximum Likelihood

Understand how actions from high-reward trajectories receive increased probability.

What is it?

REINFORCE can be understood as reward-weighted maximum likelihood. The policy gradient term increases the probability of a selected action, while its return determines how strongly that action should be reinforced.

Key points

1. ∇θ log πθ(aₜ|sₜ) points in the direction that increases action probability. 2. Multiplication by Gₜ weights the update by return. 3. High-return trajectories increase the probability of their selected actions. 4. Low-return trajectories receive weaker or negative updates when using a baseline. 5. Actions act like training labels. 6. Returns provide the quality weights for those actions.

Example: Learning from successful and unsuccessful responses

Imagine an AI assistant generates two responses. Response A receives a high reward because it correctly answers the question. Response B receives a low reward because it contains incorrect information. The policy update increases the probability of actions associated with the successful response and reduces the relative preference for poorly performing actions. This resembles supervised learning, but the training actions are weighted by their observed returns.

Real-world application

Reward-weighted policy optimization is useful in reinforcement learning from human feedback and other reward-driven language-model training approaches.
Key takeaway

REINFORCE behaves like maximum-likelihood learning in which the actions are weighted by how well their outcomes performed.