5.6.5
Step 5: Compute GAE Advantages
Understand how value-head predictions and GAE produce token-level advantages, followed by advantage whitening.
What is it?
Generalized Advantage Estimation (GAE) calculates an advantage estimate for each token position using value-head predictions. The advantages are then whitened so that they have zero mean and unit variance, helping stabilize policy optimization.
Key points
1. GAE calculates an advantage for each token position.
2. The value head provides predictions used in advantage estimation.
3. Advantages measure whether token choices performed better or worse than expected.
4. Advantage estimates are calculated across the generated sequence.
5. Advantage whitening produces zero mean and unit variance.
6. The whitened advantages are used in the PPO policy update.
Example: Advantage Whitening
Suppose the computed advantages are:
2, 4, 6
Mean = 4.
Subtracting the mean gives:
−2, 0, 2
Whitening also divides the centered values by their standard deviation, producing a normalized set with zero mean and unit variance.
Real-world application
Advantage whitening helps maintain a consistent scale for policy-gradient updates across training batches.
Key takeaway
GAE supplies token-level advantage estimates, and whitening normalizes them before PPO optimization.