OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Reward Model Training Tricks
Lesson 364 of 415 5 min read
9.3.2

Length Bias Correction

Correcting reward models that favour longer responses regardless of quality.

What is it?

Length bias occurs when a reward model assigns higher scores to longer responses regardless of their actual quality. The source describes three ways to correct this behaviour.

Key points

Length normalisation: divide the reward by response length. Length-controlled training: include length as a feature and train the model to be length-invariant. Calibration: use post-hoc regression to remove the length effect.

Two Responses with Different Lengths

Imagine two responses that provide equally good answers, but one is much longer. If the reward model consistently gives the longer response a higher score, it may be exhibiting length bias. Length normalisation, length-controlled training, or post-hoc calibration can help correct this effect.

Real-world application

Check whether reward scores systematically increase with response length. Apply one of the three correction approaches described in the source, depending on the training and calibration setup.
Key takeaway

A longer response should not automatically receive a higher reward simply because it contains more tokens.