7.2.4
Normalizing Group-Relative Advantages
Learn how GRPO calculates advantages by subtracting the group mean and dividing by the group standard deviation.
What is it?
GRPO calculates each response advantage by subtracting the group mean from its reward and dividing by the group standard deviation. This indicates how far the response is above or below the group average.
Key points
Normalized advantage: Âi = (ri − μG) / σG.
Positive advantage: response reward is above the group mean.
Negative advantage: response reward is below the group mean.
Dividing by σG makes advantages scale-invariant across prompts with different reward ranges.
Understanding Positive and Negative Advantages
In the five-response example, the group mean is 0.6. A correct response with reward 1 is above the mean and receives a positive advantage. An incorrect response with reward 0 is below the mean and receives a negative advantage.
Real-world application
Use normalized advantages to identify which responses should be reinforced and which should be suppressed during policy training.
Key takeaway
Group-relative advantages show whether each response performed better or worse than the other responses to the same prompt.