6.5.8
Common Pitfalls
Understand implementation mistakes involving prompt masking, sequence summation, reference-model selection and beta.
What is it?
DPO implementations can fail or behave poorly when prompt masking, sequence summation, reference-model selection or beta configuration is incorrect.
Key points
1. Forgetting prompt masking optimizes prompt likelihood instead of response likelihood.
2. Using a mean instead of a sum changes the length penalty.
3. Policy and reference calculations must use consistent conventions.
4. A stale reference model can dominate the KL constraint and cause gradients to vanish.
5. Excessively large beta can saturate sigmoid and produce near-zero gradients.
6. Excessively small beta can make gradients too weak and slow convergence.
7. The source recommends starting with beta = 0.1 and tuning within 0.05–0.5.
Common DPO Implementation Mistakes
Pitfall 1 – Prompt masking: Including prompt tokens in the log-probability sum makes the model optimize prompt likelihood.
Pitfall 2 – Mean versus sum: These produce different implicit length penalties. Keep policy and reference calculations consistent.
Pitfall 3 – Stale reference: A reference model too far from the policy can dominate the KL term. The source recommends using the SFT checkpoint rather than the base model.
Pitfall 4 – Beta too large: Log-probability differences are magnified, sigmoid saturates and gradients approach zero.
Pitfall 5 – Beta too small: The gradient becomes very weak, producing a flat loss landscape and slow convergence.
Starting point in the source: beta = 0.1; tune within [0.05, 0.5].
Real-world application
Validate masking, sequence aggregation, reference checkpoint and beta before launching a large DPO training run.
Key takeaway
Correct implementation details and balanced beta selection are essential for stable DPO training.