OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 293 of 415 6 min read
7.5.10.2

Batch-Level Reward Scaling

Learn how CISPO normalises rewards across the entire training batch instead of separately within each prompt group.

What is it?

Standard GRPO normalises rewards within each prompt group. CISPO instead normalises rewards across the entire training batch using the batch mean and standard deviation. This provides a broader baseline and prevents one prompt from dominating the gradient.

Key points

Standard GRPO uses within-group reward normalisation.|CISPO uses batch-level reward normalisation.|The batch mean and standard deviation are calculated across current training rewards.|Batch scaling helps prevent a single prompt from dominating the gradient.

From group-level to batch-level reward scaling

Suppose a training batch contains responses from several prompts. Rather than calculating a separate reward mean and standard deviation for each prompt group, CISPO calculates them across all rewards in the batch and uses those values to normalise each reward.

Real-world application

Consider batch-level scaling when training across multiple prompts with different reward distributions.
Key takeaway

CISPO normalises rewards across the full batch rather than separately within each prompt group.