OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / Introduction to Reinforcement Learning / Taxonomy of RL Methods
Lesson 113 of 415 6 min read
3.3.4

Actor-Critic: Combining Policy and Value Learning

Understanding how an actor selects actions while a critic evaluates states or actions.

What is it?

Actor-Critic methods combine policy-gradient learning with a learned value function. The actor proposes actions, while the critic evaluates states or actions and provides feedback for policy improvement.

Key points

1. Actor represents the policy πθ(a|s). 2. Actor proposes actions. 3. Critic represents Vφ(s) or Qφ(s,a). 4. Critic evaluates the quality of states or actions. 5. Critic provides a value estimate or advantage signal. 6. The actor uses this feedback to improve its policy. 7. PPO for LLMs uses an Actor-Critic approach.

Example: An actor and critic working together

A robot is learning to navigate a warehouse. Actor: Chooses whether to move left, right or forward. Critic: Estimates how good the current state or selected action is. Learning: The critic provides feedback that helps the actor improve future decisions.

Real-world application

Actor-Critic architectures are used in policy optimization and reinforcement learning systems. PPO is an important example in LLM training.
Key takeaway

The actor decides what to do; the critic evaluates how good the decision is.