OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Multi-Objective Rewards – Combination Strategies
Lesson 372 of 415 6 min read
9.6.3

Normalise Then Sum: GDPO

Normalise each reward within the group before combining the signals with weights.

What is it?

Normalise then sum, used by GDPO, independently normalises each reward within the group to zero mean and unit variance before combining the signals with weights.

Key points

Each reward is normalised independently within the group. Normalisation produces zero mean and unit variance. The normalised rewards are then combined using weights. This approach is scale-invariant compared with a direct weighted sum. The source uses the TRL setting multi_objective_aggregation="normalize_then_sum".

Combining Rewards After Normalisation

Suppose correctness scores vary between 0 and 1, while another reward has values between 0 and 100. A direct weighted sum may allow the second reward to dominate. Normalising each reward within the group first makes their scales comparable before applying weights.

Real-world application

In TRL, configure multi_objective_aggregation="normalize_then_sum" and provide reward_weights for the individual reward functions.
Key takeaway

GDPO normalises reward signals independently before combining them, reducing sensitivity to their original scales.