OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / Introduction to Reinforcement Learning / Policy Gradient Methods — REINFORCE
Lesson 123 of 415 8 min read
3.6.4

Variance Reduction with Baselines and REINFORCE Limitations

Understand state-dependent baselines, advantage estimates and limitations of REINFORCE.

What is it?

A baseline can be subtracted from the return to reduce the variance of policy-gradient estimates without introducing bias, provided the baseline does not depend on the selected action. The state-value function Vπ(s) is a common baseline.

Key points

1. Baseline-adjusted gradient: ∇θJ(θ) = Eπθ[Σₜ ∇θ log πθ(aₜ|sₜ) · (Gₜ − b(sₜ))]. 2. The baseline must not depend on action aₜ. 3. A state-dependent baseline preserves an unbiased gradient estimate. 4. A common choice is b(sₜ) = Vπ(sₜ). 5. Gₜ − Vπ(sₜ) approximates the advantage. 6. REINFORCE has high variance. 7. It must wait for complete episodes and does not bootstrap. 8. It is sample-inefficient because collected data is used once. 9. Unconstrained policy updates can be too large and destabilize training. 10. These limitations motivate Actor-Critic, TRPO and PPO.

Example: Evaluating an action against a baseline

Suppose an action produces a return of 12. The state-value baseline is 8. Advantage estimate = 12 − 8 = 4. The action performed better than the expected baseline, so the policy receives a positive learning signal. If the return were 5, the advantage would be 5 − 8 = −3, indicating performance below the baseline.

Real-world application

Baselines reduce variance in policy-gradient training. Actor-Critic methods introduce a learned critic, while PPO adds constraints to help prevent excessively large policy updates.
Key takeaway

A baseline reduces policy-gradient variance without bias when it is independent of the selected action. REINFORCE limitations motivate more stable policy-optimization methods.