7.5.12.1
Why Reward Magnitudes Can Be Misleading
Understand why learned reward models may provide unreliable absolute scores even when their relative rankings are useful.
What is it?
Reward models are often trained using pairwise comparisons. Their relative ranking may be useful even when their absolute numeric scores are unreliable. GOPO addresses this mismatch by using rankings instead of raw reward magnitudes.
Key points
Reward models can be trained with pairwise comparisons.|Relative ranking may be more trustworthy than absolute scores.|Raw reward gaps can be misleading.|This matters for summarisation, open-ended chat and instruction following.|GOPO uses ordinal ranking.
Why a reward gap can be misleading
A reward model assigns two responses scores of 0.8 and 0.2. The numerical gap is 0.6. However, that gap does not necessarily represent a consistent amount of quality difference. In another region of the output space, the same numerical gap may mean something different.
Real-world application
Be cautious when using raw learned reward scores for subjective tasks.
Key takeaway
GOPO trusts the order of responses rather than the absolute distance between their reward scores.