OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / Introduction to Reinforcement Learning / Taxonomy of RL Methods
Lesson 112 of 415 7 min read
3.3.3

Value-Based vs Policy-Based Methods

Understanding methods that learn value functions and methods that directly optimize policies.

What is it?

Value-Based methods learn a value function such as Q(s,a) or V(s) and derive a policy from it. Policy-Based methods directly parameterize and optimize a policy πθ(a|s).

Key points

1. Value-Based methods learn state or action values. 2. A policy can be derived using arg maxₐ Q(s,a). 3. They work well with small, discrete action spaces. 4. Large or continuous action spaces can be difficult. 5. Policy-Based methods directly optimize πθ(a|s). 6. They naturally support continuous and high-dimensional action spaces. 7. Policy-Based methods are essential for LLMs with large vocabulary action spaces.

Example: Choosing actions in a game and an LLM

A game agent has four possible movements. A Value-Based algorithm can estimate the Q-value of each movement and select the highest-value action. An LLM may choose from a vocabulary of 32K–128K possible tokens. Direct policy optimization is more natural for this large action space.

Real-world application

Value-Based learning is useful in discrete decision environments. Policy-Based learning is particularly relevant to continuous control and language-model training.
Key takeaway

Value-Based methods learn how good actions are; Policy-Based methods directly learn how to act.