3.3.4
Actor-Critic: Combining Policy and Value Learning
Understanding how an actor selects actions while a critic evaluates states or actions.
What is it?
Actor-Critic methods combine policy-gradient learning with a learned value function. The actor proposes actions, while the critic evaluates states or actions and provides feedback for policy improvement.
Key points
1. Actor represents the policy πθ(a|s).
2. Actor proposes actions.
3. Critic represents Vφ(s) or Qφ(s,a).
4. Critic evaluates the quality of states or actions.
5. Critic provides a value estimate or advantage signal.
6. The actor uses this feedback to improve its policy.
7. PPO for LLMs uses an Actor-Critic approach.
Example: An actor and critic working together
A robot is learning to navigate a warehouse.
Actor: Chooses whether to move left, right or forward.
Critic: Estimates how good the current state or selected action is.
Learning: The critic provides feedback that helps the actor improve future decisions.
Real-world application
Actor-Critic architectures are used in policy optimization and reinforcement learning systems. PPO is an important example in LLM training.
Key takeaway
The actor decides what to do; the critic evaluates how good the decision is.