OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 316 of 415 5 min read
7.5.11.4

GDPO vs Standard Multi-Reward GRPO

Compare combined reward normalisation in standard GRPO with GDPO’s independent reward normalisation and weighted aggregation.

What is it?

Standard multi-reward GRPO combines raw rewards before normalisation, allowing high-variance rewards to dominate. GDPO normalises each reward first and then combines them, so each objective contributes in proportion to its assigned weight.

Key points

Standard GRPO normalises the combined weighted reward.|High-variance rewards can dominate standard GRPO.|GDPO normalises each reward separately.|GDPO is useful when reward scales or variances differ substantially.

Standard GRPO versus GDPO

Consider two reward functions with different scales. Standard GRPO combines them before normalisation. GDPO first calculates a normalised advantage for each reward, then adds the weighted advantages. This preserves the intended contribution of each objective.

Real-world application

Choose GDPO when multi-objective reward functions have substantially different scales or variances.
Key takeaway

Independent reward normalisation makes GDPO better suited to imbalanced multi-reward settings.