OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / Motivation
Lesson 254 of 415 4 min read
7.1.1

Why PPO Needs a Value Network

Understand the critic in PPO and its role in estimating expected reward for policy updates.

What is it?

PPO uses a value model called a critic to estimate expected reward. This estimate helps calculate advantages for policy updates. GRPO takes a different approach: it estimates the baseline empirically from multiple responses generated for the same prompt.

Key points

The critic estimates expected reward for a state. The estimate is used to calculate advantages. GRPO replaces the learned value estimate with statistics from a group of sampled responses. This removes the need for a separate value network.

The Critic in PPO

A language model generates an answer to a question. PPO uses a critic to estimate how much reward should be expected from the current state. The policy update uses this estimate to judge whether the generated response performed better or worse than expected.

Real-world application

When comparing PPO and GRPO, identify where each method obtains its baseline: PPO learns it through a value model, while GRPO estimates it from group rewards.
Key takeaway

The PPO critic estimates expected reward; GRPO replaces this learned baseline with group-based reward statistics.