7.5.5.1
The It Takes Two Insight
Understand the finding that two completions per prompt can match or exceed larger GRPO groups on many reasoning benchmarks.
What is it?
The It Takes Two paper reports that GRPO with just two completions per prompt can match or exceed GRPO with sixteen completions on most reasoning benchmarks. The finding challenges the assumption that larger groups are always necessary for effective advantage estimation.
Key points
2-GRPO uses G = 2 completions per prompt. The source reports results matching or exceeding G = 16 on most reasoning benchmarks. Its explanation emphasizes an implicit contrastive objective rather than highly accurate advantage estimation.
Two Responses, One Comparison
For a prompt, the model generates two responses. One receives a higher reward and the other a lower reward. Their relative quality creates a direct comparison that can guide the policy update.
Real-world application
Consider two rollouts as a lower-compute alternative when experimenting with GRPO on reasoning tasks. Evaluate performance on the target benchmark rather than assuming the result will generalize to every task.
Key takeaway
The central 2-GRPO insight is that a useful contrast may be obtained from just two responses.