7.5.11.1
The Multi-Reward Collapse Problem
Understand how a high-variance reward can dominate combined reward normalisation and suppress the contribution of other objectives.
What is it?
In multi-objective reinforcement learning, standard GRPO normalises the combined reward. If one reward has much higher variance than another, it can dominate the advantage calculation and effectively suppress the lower-variance reward. This is called advantage collapse.
Key points
Multi-objective RL can use correctness and format rewards.|Standard GRPO normalises the combined reward.|A high-variance reward can dominate.|A low-variance reward may contribute near-zero gradient.|GDPO addresses this through independent reward normalisation.
When one reward dominates another
Imagine a model trained with correctness and formatting rewards. If correctness scores vary widely but formatting scores vary very little, combined normalisation may cause correctness to dominate the advantage signal. The formatting objective can become nearly ignored.
Real-world application
Watch for reward-scale and variance imbalance when designing multi-objective RL systems.
Key takeaway
GDPO addresses advantage collapse caused by rewards with very different variances.