7.5.10.2
Batch-Level Reward Scaling
Learn how CISPO normalises rewards across the entire training batch instead of separately within each prompt group.
What is it?
Standard GRPO normalises rewards within each prompt group. CISPO instead normalises rewards across the entire training batch using the batch mean and standard deviation. This provides a broader baseline and prevents one prompt from dominating the gradient.
Key points
Standard GRPO uses within-group reward normalisation.|CISPO uses batch-level reward normalisation.|The batch mean and standard deviation are calculated across current training rewards.|Batch scaling helps prevent a single prompt from dominating the gradient.
From group-level to batch-level reward scaling
Suppose a training batch contains responses from several prompts. Rather than calculating a separate reward mean and standard deviation for each prompt group, CISPO calculates them across all rewards in the batch and uses those values to normalise each reward.
Real-world application
Consider batch-level scaling when training across multiple prompts with different reward distributions.
Key takeaway
CISPO normalises rewards across the full batch rather than separately within each prompt group.