6.6.4
Preference Data Quality and Noisy Labels
Understand how incorrect or noisy preference labels can poison DPO training and cause memorization of individual pairs.
What is it?
Noisy or incorrect preference labels can poison DPO training. Because DPO learns from individual preference pairs, it can memorize bad comparisons instead of averaging over many samples as described for PPO.
Key points
1. Preference labels may be noisy or incorrect.
2. DPO directly optimizes individual pairs.
3. Bad labels can teach incorrect preferences.
4. The source contrasts this with PPO averaging over many samples.
5. Dataset quality is critical.
A Mislabelled Preference Pair
Prompt: Explain a safety procedure.
Chosen response: An accurate and safe explanation.
Rejected response: An unsafe explanation.
If the dataset mistakenly labels the unsafe response as chosen, DPO receives a misleading training signal and can reinforce the wrong behaviour.
Real-world application
Review preference labels, remove unreliable comparisons and use clear evaluation guidelines when building datasets.
Key takeaway
Incorrect preference labels can directly teach the model undesirable behaviour.