3.7.1
Understanding Actor-Critic Architecture
Understand how the actor selects actions and the critic evaluates states or actions.
What is it?
Actor-Critic combines policy-gradient learning with a learned value function. The actor represents the policy and proposes actions, while the critic evaluates how good a state or action is. The critic provides a lower-variance baseline that helps improve policy learning.
Key points
1. Actor: πθ(a|s), the policy that proposes actions.
2. Critic: Vϕ(s) or Qϕ(s,a), the value function that evaluates states or actions.
3. The actor learns which actions to select.
4. The critic estimates how good those decisions are.
5. Critic feedback helps reduce policy-gradient variance.
6. Actor-Critic maintains the flexibility of policy optimization.
Example: A robot navigating a warehouse
The actor chooses whether the robot should move left, right or forward.
The critic evaluates the current state and estimates the expected future reward.
If an action leads to a better-than-expected outcome, the critic provides a positive learning signal. The actor then adjusts its policy to make that action more likely in similar situations.
Real-world application
Actor-Critic architectures are used in reinforcement learning systems that need both flexible action selection and value estimation. PPO-based LLM training uses an actor-critic framework.
Key takeaway
The actor proposes actions; the critic evaluates them and provides learning feedback.