7.5.11.3
Combining Normalised Rewards with Weights
Understand how independently normalised rewards are aggregated using user-specified weights.
What is it?
After independent normalisation, GDPO combines the reward signals using user-specified weights. Each objective contributes according to its assigned weight rather than being dominated simply by its original reward variance.
Key points
Each reward produces a normalised advantage.|Weights are specified by the user.|The weighted normalised rewards are summed.|Weights control each objective’s contribution.
Combining rewards with weights
The source gives the aggregation as the sum of each independently normalised reward multiplied by its weight. For example, correctness can receive weight 1.0 and format can receive weight 0.5.
Real-world application
Set reward weights to reflect the relative importance of objectives in a training task.
Key takeaway
GDPO combines independently normalised rewards according to user-defined weights.