OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / How DPO Works: Full Mechanics
Lesson 228 of 415 6 min read
6.5.5

Per-Token vs. Sequence-Level: Length Normalization

Compare summed sequence log-probabilities with length-normalized per-token log-probabilities and their training effects.

What is it?

Standard DPO sums token log-probabilities. Longer sequences naturally accumulate more negative log-probability terms, which can create a preference for shorter responses.

Key points

1. Every token log-probability is less than or equal to zero. 2. Longer sequences generally have lower summed log-probabilities. 3. Standard DPO uses the raw sequence sum. 4. Length-normalized DPO averages log-probabilities over response tokens. 5. SimPO adopts a length-normalized approach. 6. Length normalization can reduce length gaming but may affect instruction-following quality.

Sum Versus Average

Standard sequence-level log-probability: log πθ(y|x) = Σ(t=1 to T) log πθ(yt|x, y

Real-world application

Evaluate response lengths and instruction-following quality when choosing between summed and length-normalized objectives.
Key takeaway

Length normalization changes the objective and its implicit preference for response length.