7.5.11.4
GDPO vs Standard Multi-Reward GRPO
Compare combined reward normalisation in standard GRPO with GDPO’s independent reward normalisation and weighted aggregation.
What is it?
Standard multi-reward GRPO combines raw rewards before normalisation, allowing high-variance rewards to dominate. GDPO normalises each reward first and then combines them, so each objective contributes in proportion to its assigned weight.
Key points
Standard GRPO normalises the combined weighted reward.|High-variance rewards can dominate standard GRPO.|GDPO normalises each reward separately.|GDPO is useful when reward scales or variances differ substantially.
Standard GRPO versus GDPO
Consider two reward functions with different scales. Standard GRPO combines them before normalisation. GDPO first calculates a normalised advantage for each reward, then adds the weighted advantages. This preserves the intended contribution of each objective.
Real-world application
Choose GDPO when multi-objective reward functions have substantially different scales or variances.
Key takeaway
Independent reward normalisation makes GDPO better suited to imbalanced multi-reward settings.