OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Derivation of the PPO Gradient and Update Rule
Lesson 172 of 415 6 min read
5.4.2

Step 2: Policy Gradient Theorem

Understand how the advantage function contributes to policy gradient updates.

What is it?

The Policy Gradient Theorem expresses the gradient of the expected reward objective using the log-probability of selected actions multiplied by their estimated advantage. The advantage measures how much better an action performed compared with the average action in the same state.

Key points

1. The gradient indicates how policy parameters should change. 2. ∇θ log πθ(a_t|s_t) measures sensitivity of action log-probability to policy parameters. 3. Â_t represents the estimated advantage. 4. Positive advantage indicates a better-than-average action. 5. Negative advantage indicates a worse-than-average action. 6. Using advantage instead of the full return helps reduce variance. 7. Equation (5.4): ∇θJ(θ) = E_{πθ}[Σ ∇θ log πθ(a_t|s_t) · Â_t].

Example: Advantage Guides Policy Improvement

Suppose an action has an estimated advantage of +3. The positive advantage indicates that the action performed better than the average action in that state. The policy gradient encourages increasing its probability. If another action has an advantage of −2, the gradient encourages decreasing its probability.

Real-world application

The policy gradient theorem provides the learning direction used to improve a policy based on action quality.
Key takeaway

Policy gradients use log-probability gradients and advantage estimates to improve action selection.