7.1.5
GRPO: Estimating Value from a Group of Responses
Understand how GRPO uses group reward statistics as an empirical baseline instead of a learned value function.
What is it?
GRPO avoids learning a separate value function. It generates G responses for the same prompt, calculates their rewards and uses group statistics as an empirical baseline for estimating advantages.
Key points
Generate G responses for the same prompt.
Calculate a reward for each response.
Use group statistics as the baseline.
The group provides an empirical estimate instead of learning V(s).
This removes the separate value network.
Estimating Value from Multiple Responses
For one prompt, generate several candidate responses. Score every response and calculate the group average reward. A response can then be compared with this average to determine whether it performed better or worse than the group.
Real-world application
Use group-based reward comparisons when designing GRPO training. The next topic explains how to calculate and normalise these group-relative advantages.
Key takeaway
GRPO estimates the baseline from multiple responses to the same prompt instead of training a critic.