OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 288 of 415 6 min read
7.5.5.2

The Contrastive Objective Behind 2-GRPO

Learn how two completions create an explicit comparison between a higher-reward and a lower-reward response, structurally resembling DPO.

What is it?

The source explains 2-GRPO through an implicit contrastive objective structurally similar to DPO. With two completions, the higher-reward response is compared directly against the lower-reward response. With larger groups, similar comparisons exist but may be diluted by redundant pairs.

Key points

Higher-reward completion: o+. Lower-reward completion: o−. G = 2 makes the contrastive structure explicit. The objective resembles preference optimization, including a sigmoid-based comparison of policy probability ratios.

Comparing a Preferred and Rejected Response

Suppose response A receives reward 1 and response B receives reward 0. The pair forms a direct comparison: reinforce the higher-reward response relative to the lower-reward one. This is the contrastive structure highlighted by the source.

Real-world application

Think of the two responses as a preference pair when interpreting the learning signal produced by 2-GRPO.
Key takeaway

With two rollouts, the contrast between the better and worse completion becomes explicit.