7.5.12.4
GOPO vs GRPO: Magnitudes and Rankings
Compare magnitude-based GRPO advantages with rank-based GOPO advantages.
What is it?
GRPO calculates advantages using reward magnitudes and group statistics. GOPO uses ordinal ranks instead. As a result, GOPO is invariant to monotonic transformations of reward scores, while GRPO can be sensitive to reward scale and calibration.
Key points
GRPO uses reward magnitude and group statistics.|GOPO uses a monotonic function of normalised rank.|GRPO is sensitive to reward scale.|GOPO is invariant to monotonic reward transformations.|GRPO suits verifiable rewards; GOPO suits noisy subjective reward models.
Same ranking, different reward scales
Imagine a reward model gives scores of 0.2, 0.5 and 0.9. A monotonic transformation could change these scores to 10, 20 and 100 without changing their order. GOPO preserves the same ranking-based signal, while GRPO’s magnitude-based advantages can change.
Real-world application
Consider GOPO when reward model calibration is unreliable but its relative ordering is useful.
Key takeaway
GRPO uses reward magnitudes; GOPO uses the order of responses.