OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 321 of 415 5 min read
7.5.5.5

Limitations of Two-Rollout GRPO

Understand why two-sample advantage normalization fixes gradient magnitude and why larger groups may help when reward gaps or partial credit matter.

What is it?

With only two rewards, normalized advantages are always +1 and −1 when the rewards differ, or both zero when rewards are equal. Consequently, gradient magnitude does not reflect how large the reward gap was. Larger groups may be preferable when reward differences or partial credit matter.

Key points

For G = 2, normalized advantages are {+1, −1} when rewards differ, or {0, 0} when rewards are equal. Gradient magnitude is fixed regardless of reward gap. Larger G may be beneficial for tasks where reward magnitude matters, such as partial-credit evaluation.

A Small Reward Gap and a Large Reward Gap

Consider two response pairs. In the first, rewards are 0.51 and 0.50. In the second, rewards are 1.0 and 0.0. With two different rewards, normalization produces the same +1 and −1 pattern in both cases. The normalized advantages therefore do not preserve the size of the reward difference.

Real-world application

Use larger groups when the magnitude of reward differences carries important information, particularly in tasks with partial-credit scoring.
Key takeaway

2-GRPO is compute-efficient, but its two-sample normalization loses information about the magnitude of the reward gap.