3.2.5
The Advantage Function
Learn how the advantage function compares an action value with the average value of its state.
What is it?
The advantage function measures how much better or worse an action performs compared with the expected value of the current state. It is defined as Aᵖⁱ(s,a) = Qᵖⁱ(s,a) − Vᵖⁱ(s).
Key points
1. Advantage compares action value with state value.
2. Aᵖⁱ(s,a) = Qᵖⁱ(s,a) − Vᵖⁱ(s).
3. Positive advantage means better than the state baseline.
4. Negative advantage means worse than the baseline.
5. Zero advantage means both values are equal.
6. Advantage is important in policy-gradient and actor-critic methods.
Example: Comparing two actions
Suppose V(s) = 10.
Action A: Q(s,A) = 14.
A(s,A) = 14 − 10 = 4.
Action B: Q(s,B) = 7.
A(s,B) = 7 − 10 = −3.
Action A performs better than the state baseline.
Real-world application
Advantage estimates help policy-optimization algorithms identify actions that perform better or worse than expected.
Key takeaway
Advantage measures the relative benefit of an action compared with the expected value of its state.