7.1.1
Why PPO Needs a Value Network
Understand the critic in PPO and its role in estimating expected reward for policy updates.
What is it?
PPO uses a value model called a critic to estimate expected reward. This estimate helps calculate advantages for policy updates. GRPO takes a different approach: it estimates the baseline empirically from multiple responses generated for the same prompt.
Key points
The critic estimates expected reward for a state.
The estimate is used to calculate advantages.
GRPO replaces the learned value estimate with statistics from a group of sampled responses.
This removes the need for a separate value network.
The Critic in PPO
A language model generates an answer to a question. PPO uses a critic to estimate how much reward should be expected from the current state. The policy update uses this estimate to judge whether the generated response performed better or worse than expected.
Real-world application
When comparing PPO and GRPO, identify where each method obtains its baseline: PPO learns it through a value model, while GRPO estimates it from group rewards.
Key takeaway
The PPO critic estimates expected reward; GRPO replaces this learned baseline with group-based reward statistics.