3.6.4
Variance Reduction with Baselines and REINFORCE Limitations
Understand state-dependent baselines, advantage estimates and limitations of REINFORCE.
What is it?
A baseline can be subtracted from the return to reduce the variance of policy-gradient estimates without introducing bias, provided the baseline does not depend on the selected action. The state-value function Vπ(s) is a common baseline.
Key points
1. Baseline-adjusted gradient: ∇θJ(θ) = Eπθ[Σₜ ∇θ log πθ(aₜ|sₜ) · (Gₜ − b(sₜ))].
2. The baseline must not depend on action aₜ.
3. A state-dependent baseline preserves an unbiased gradient estimate.
4. A common choice is b(sₜ) = Vπ(sₜ).
5. Gₜ − Vπ(sₜ) approximates the advantage.
6. REINFORCE has high variance.
7. It must wait for complete episodes and does not bootstrap.
8. It is sample-inefficient because collected data is used once.
9. Unconstrained policy updates can be too large and destabilize training.
10. These limitations motivate Actor-Critic, TRPO and PPO.
Example: Evaluating an action against a baseline
Suppose an action produces a return of 12.
The state-value baseline is 8.
Advantage estimate = 12 − 8 = 4.
The action performed better than the expected baseline, so the policy receives a positive learning signal.
If the return were 5, the advantage would be 5 − 8 = −3, indicating performance below the baseline.
Real-world application
Baselines reduce variance in policy-gradient training. Actor-Critic methods introduce a learned critic, while PPO adds constraints to help prevent excessively large policy updates.
Key takeaway
A baseline reduces policy-gradient variance without bias when it is independent of the selected action. REINFORCE limitations motivate more stable policy-optimization methods.