6.5.5
Per-Token vs. Sequence-Level: Length Normalization
Compare summed sequence log-probabilities with length-normalized per-token log-probabilities and their training effects.
What is it?
Standard DPO sums token log-probabilities. Longer sequences naturally accumulate more negative log-probability terms, which can create a preference for shorter responses.
Key points
1. Every token log-probability is less than or equal to zero.
2. Longer sequences generally have lower summed log-probabilities.
3. Standard DPO uses the raw sequence sum.
4. Length-normalized DPO averages log-probabilities over response tokens.
5. SimPO adopts a length-normalized approach.
6. Length normalization can reduce length gaming but may affect instruction-following quality.
Sum Versus Average
Standard sequence-level log-probability:
log πθ(y|x) = Σ(t=1 to T) log πθ(yt|x, y
Real-world application
Evaluate response lengths and instruction-following quality when choosing between summed and length-normalized objectives.
Key takeaway
Length normalization changes the objective and its implicit preference for response length.