9.3.2
Length Bias Correction
Correcting reward models that favour longer responses regardless of quality.
What is it?
Length bias occurs when a reward model assigns higher scores to longer responses regardless of their actual quality. The source describes three ways to correct this behaviour.
Key points
Length normalisation: divide the reward by response length.
Length-controlled training: include length as a feature and train the model to be length-invariant.
Calibration: use post-hoc regression to remove the length effect.
Two Responses with Different Lengths
Imagine two responses that provide equally good answers, but one is much longer. If the reward model consistently gives the longer response a higher score, it may be exhibiting length bias. Length normalisation, length-controlled training, or post-hoc calibration can help correct this effect.
Real-world application
Check whether reward scores systematically increase with response length. Apply one of the three correction approaches described in the source, depending on the training and calibration setup.
Key takeaway
A longer response should not automatically receive a higher reward simply because it contains more tokens.