5.4.2
Step 2: Policy Gradient Theorem
Understand how the advantage function contributes to policy gradient updates.
What is it?
The Policy Gradient Theorem expresses the gradient of the expected reward objective using the log-probability of selected actions multiplied by their estimated advantage. The advantage measures how much better an action performed compared with the average action in the same state.
Key points
1. The gradient indicates how policy parameters should change.
2. ∇θ log πθ(a_t|s_t) measures sensitivity of action log-probability to policy parameters.
3. Â_t represents the estimated advantage.
4. Positive advantage indicates a better-than-average action.
5. Negative advantage indicates a worse-than-average action.
6. Using advantage instead of the full return helps reduce variance.
7. Equation (5.4): ∇θJ(θ) = E_{πθ}[Σ ∇θ log πθ(a_t|s_t) · Â_t].
Example: Advantage Guides Policy Improvement
Suppose an action has an estimated advantage of +3.
The positive advantage indicates that the action performed better than the average action in that state. The policy gradient encourages increasing its probability.
If another action has an advantage of −2, the gradient encourages decreasing its probability.
Real-world application
The policy gradient theorem provides the learning direction used to improve a policy based on action quality.
Key takeaway
Policy gradients use log-probability gradients and advantage estimates to improve action selection.