3.3.3
Value-Based vs Policy-Based Methods
Understanding methods that learn value functions and methods that directly optimize policies.
What is it?
Value-Based methods learn a value function such as Q(s,a) or V(s) and derive a policy from it. Policy-Based methods directly parameterize and optimize a policy πθ(a|s).
Key points
1. Value-Based methods learn state or action values.
2. A policy can be derived using arg maxₐ Q(s,a).
3. They work well with small, discrete action spaces.
4. Large or continuous action spaces can be difficult.
5. Policy-Based methods directly optimize πθ(a|s).
6. They naturally support continuous and high-dimensional action spaces.
7. Policy-Based methods are essential for LLMs with large vocabulary action spaces.
Example: Choosing actions in a game and an LLM
A game agent has four possible movements. A Value-Based algorithm can estimate the Q-value of each movement and select the highest-value action.
An LLM may choose from a vocabulary of 32K–128K possible tokens. Direct policy optimization is more natural for this large action space.
Real-world application
Value-Based learning is useful in discrete decision environments. Policy-Based learning is particularly relevant to continuous control and language-model training.
Key takeaway
Value-Based methods learn how good actions are; Policy-Based methods directly learn how to act.