7.5.5.5
Limitations of Two-Rollout GRPO
Understand why two-sample advantage normalization fixes gradient magnitude and why larger groups may help when reward gaps or partial credit matter.
What is it?
With only two rewards, normalized advantages are always +1 and −1 when the rewards differ, or both zero when rewards are equal. Consequently, gradient magnitude does not reflect how large the reward gap was. Larger groups may be preferable when reward differences or partial credit matter.
Key points
For G = 2, normalized advantages are {+1, −1} when rewards differ, or {0, 0} when rewards are equal. Gradient magnitude is fixed regardless of reward gap. Larger G may be beneficial for tasks where reward magnitude matters, such as partial-credit evaluation.
A Small Reward Gap and a Large Reward Gap
Consider two response pairs. In the first, rewards are 0.51 and 0.50. In the second, rewards are 1.0 and 0.0. With two different rewards, normalization produces the same +1 and −1 pattern in both cases. The normalized advantages therefore do not preserve the size of the reward difference.
Real-world application
Use larger groups when the magnitude of reward differences carries important information, particularly in tasks with partial-credit scoring.
Key takeaway
2-GRPO is compute-efficient, but its two-sample normalization loses information about the magnitude of the reward gap.