7.4.3
Why Groups Need Both Successes and Failures
Understand why a group containing only correct or only incorrect responses produces zero advantages and no learning signal.
What is it?
A useful GRPO group must contain both successful and unsuccessful responses. If every response receives the same reward, all group-relative advantages become zero and the policy receives no learning signal.
Key points
All responses correct: ri = 1 for every response; all advantages are zero.
All responses incorrect: the same zero-advantage problem occurs.
Groups need variation in reward outcomes.
Prompt difficulty must match the model capability.
Why a Group Needs Mixed Outcomes
Suppose a group contains eight responses and every answer is correct. Each response receives the same reward, so subtracting the group mean produces zero advantage. The same happens when every response is incorrect. Neither group provides a useful relative learning signal.
Real-world application
When designing a GRPO dataset, check whether prompts produce a mixture of successful and unsuccessful responses for the current model.
Key takeaway
GRPO needs reward variation within a group; identical outcomes produce no learning signal.