OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / Algorithm
Lesson 262 of 415 6 min read
7.2.4

Normalizing Group-Relative Advantages

Learn how GRPO calculates advantages by subtracting the group mean and dividing by the group standard deviation.

What is it?

GRPO calculates each response advantage by subtracting the group mean from its reward and dividing by the group standard deviation. This indicates how far the response is above or below the group average.

Key points

Normalized advantage: Âi = (ri − μG) / σG. Positive advantage: response reward is above the group mean. Negative advantage: response reward is below the group mean. Dividing by σG makes advantages scale-invariant across prompts with different reward ranges.

Understanding Positive and Negative Advantages

In the five-response example, the group mean is 0.6. A correct response with reward 1 is above the mean and receives a positive advantage. An incorrect response with reward 0 is below the mean and receives a negative advantage.

Real-world application

Use normalized advantages to identify which responses should be reinforced and which should be suppressed during policy training.
Key takeaway

Group-relative advantages show whether each response performed better or worse than the other responses to the same prompt.