OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / Introduction to Reinforcement Learning / Policy Gradient Methods — REINFORCE
Lesson 120 of 415 7 min read
3.6.1

Policy Gradient Objective and Theorem

Understand expected-return optimization and the policy gradient theorem.

What is it?

Policy Gradient methods directly optimize the parameters θ of a policy πθ to maximize expected return. Instead of learning a value function and deriving a policy, they improve the probability distribution over actions.

Key points

1. Objective: J(θ) = Eτ~πθ[R(τ)]. 2. The goal is to maximize expected cumulative reward. 3. Policy Gradient Theorem: ∇θJ(θ) = Eπθ[Σₜ ∇θ log πθ(aₜ|sₜ) · Gₜ]. 4. θ represents policy parameters. 5. πθ(aₜ|sₜ) is the probability of selecting action aₜ in state sₜ. 6. Gₜ is the future return from time step t. 7. The gradient indicates how policy parameters should change to improve expected return.

Example: Improving a recommendation policy

An AI recommendation agent suggests products to customers. State: Customer browsing history. Action: Recommend a product. Reward: Customer engagement or purchase. The policy gradient method adjusts the policy parameters so actions associated with higher future returns become more likely. The policy is optimized directly rather than selecting actions through a learned Q-value table.

Real-world application

Policy gradients are useful when the action space is large or continuous. They are particularly relevant to language models, where actions correspond to selecting tokens from a large vocabulary.
Key takeaway

Policy Gradient methods directly optimize policy parameters to maximize expected return.