3.7.2
Actor Policy Update Using Advantage
Understand the policy-gradient update using advantage estimates from the critic.
What is it?
The actor updates its policy using the advantage estimate provided by the critic. The advantage indicates whether an action performed better or worse than expected. The actor increases the probability of actions with positive advantage and decreases the probability of actions with negative advantage.
Key points
1. Actor policy: πθ(a|s).
2. Policy gradient: ∇θJ = E[∇θ log πθ(aₜ|sₜ) · Âₜ].
3. Âₜ represents the estimated advantage.
4. Advantage measures action performance relative to the value baseline.
5. Positive advantage encourages the selected action.
6. Negative advantage discourages the selected action.
7. The critic supplies the advantage estimate.
Example: Updating an action using advantage
Suppose an agent selects an action and receives an estimated advantage of +4.
The positive advantage means the action performed better than expected. The actor receives a positive policy-gradient signal and increases the probability of selecting that action in similar states.
If the advantage were −3, the policy update would move in the opposite direction.
Real-world application
Advantage-based actor updates are central to A2C, A3C and PPO. They help improve policies using feedback from learned value estimates.
Key takeaway
The actor uses advantage to learn which actions should become more or less likely.