7.5.5.2
The Contrastive Objective Behind 2-GRPO
Learn how two completions create an explicit comparison between a higher-reward and a lower-reward response, structurally resembling DPO.
What is it?
The source explains 2-GRPO through an implicit contrastive objective structurally similar to DPO. With two completions, the higher-reward response is compared directly against the lower-reward response. With larger groups, similar comparisons exist but may be diluted by redundant pairs.
Key points
Higher-reward completion: o+. Lower-reward completion: o−. G = 2 makes the contrastive structure explicit. The objective resembles preference optimization, including a sigmoid-based comparison of policy probability ratios.
Comparing a Preferred and Rejected Response
Suppose response A receives reward 1 and response B receives reward 0. The pair forms a direct comparison: reinforce the higher-reward response relative to the lower-reward one. This is the contrastive structure highlighted by the source.
Real-world application
Think of the two responses as a preference pair when interpreting the learning signal produced by 2-GRPO.
Key takeaway
With two rollouts, the contrast between the better and worse completion becomes explicit.