3.8.3
The Trade-off Matrix
Compare lambda configurations and their statistical properties, practical risks and operating range.
What is it?
GAE uses lambda to balance statistical properties, model reliance and training risk. The source compares three configurations: lambda equal to zero, lambda between 0.95 and 0.99, and lambda equal to one.
Key points
1. Lambda = 0: High bias and low variance.
2. Lambda = 0 relies on model parameters and value predictions.
3. Its practical risk is getting trapped in sub-optimal local minima.
4. Lambda between 0.95 and 0.99: Balanced properties and optimal mean squared error according to the source.
5. This range uses hybrid blending and requires tuning based on environment stochasticity.
6. Lambda = 1: Low bias and high variance.
7. Lambda = 1 relies on empirical environment rollouts.
8. Its practical risk includes destructive gradient updates and training explosions.
Example: Selecting a GAE configuration
Consider three possible training configurations:
Configuration A — Lambda = 0:
Low variance, but the agent depends heavily on its value network.
Configuration B — Lambda = 0.95 to 0.99:
Balances model predictions and actual rollout information.
Configuration C — Lambda = 1:
Uses full Monte Carlo returns, but training updates can become highly unstable.
The source recommends the balanced range as the operational choice, with tuning based on environment stochasticity.
Real-world application
The trade-off matrix helps machine learning engineers select GAE parameters based on training stability, environment characteristics and estimation accuracy.
Key takeaway
The source identifies lambda values between 0.95 and 0.99 as a balanced operating range for minimizing advantage-estimation mean squared error.