OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 283 of 415 5 min read
7.5.11.1

The Multi-Reward Collapse Problem

Understand how a high-variance reward can dominate combined reward normalisation and suppress the contribution of other objectives.

What is it?

In multi-objective reinforcement learning, standard GRPO normalises the combined reward. If one reward has much higher variance than another, it can dominate the advantage calculation and effectively suppress the lower-variance reward. This is called advantage collapse.

Key points

Multi-objective RL can use correctness and format rewards.|Standard GRPO normalises the combined reward.|A high-variance reward can dominate.|A low-variance reward may contribute near-zero gradient.|GDPO addresses this through independent reward normalisation.

When one reward dominates another

Imagine a model trained with correctness and formatting rewards. If correctness scores vary widely but formatting scores vary very little, combined normalisation may cause correctness to dominate the advantage signal. The formatting objective can become nearly ignored.

Real-world application

Watch for reward-scale and variance imbalance when designing multi-objective RL systems.
Key takeaway

GDPO addresses advantage collapse caused by rewards with very different variances.