OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 317 of 415 6 min read
7.5.12.4

GOPO vs GRPO: Magnitudes and Rankings

Compare magnitude-based GRPO advantages with rank-based GOPO advantages.

What is it?

GRPO calculates advantages using reward magnitudes and group statistics. GOPO uses ordinal ranks instead. As a result, GOPO is invariant to monotonic transformations of reward scores, while GRPO can be sensitive to reward scale and calibration.

Key points

GRPO uses reward magnitude and group statistics.|GOPO uses a monotonic function of normalised rank.|GRPO is sensitive to reward scale.|GOPO is invariant to monotonic reward transformations.|GRPO suits verifiable rewards; GOPO suits noisy subjective reward models.

Same ranking, different reward scales

Imagine a reward model gives scores of 0.2, 0.5 and 0.9. A monotonic transformation could change these scores to 10, 20 and 100 without changing their order. GOPO preserves the same ranking-based signal, while GRPO’s magnitude-based advantages can change.

Real-world application

Consider GOPO when reward model calibration is unreliable but its relative ordering is useful.
Key takeaway

GRPO uses reward magnitudes; GOPO uses the order of responses.