OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / DPO Variants and When Each Fails
Lesson 235 of 415 5 min read
6.6.4

Preference Data Quality and Noisy Labels

Understand how incorrect or noisy preference labels can poison DPO training and cause memorization of individual pairs.

What is it?

Noisy or incorrect preference labels can poison DPO training. Because DPO learns from individual preference pairs, it can memorize bad comparisons instead of averaging over many samples as described for PPO.

Key points

1. Preference labels may be noisy or incorrect. 2. DPO directly optimizes individual pairs. 3. Bad labels can teach incorrect preferences. 4. The source contrasts this with PPO averaging over many samples. 5. Dataset quality is critical.

A Mislabelled Preference Pair

Prompt: Explain a safety procedure. Chosen response: An accurate and safe explanation. Rejected response: An unsafe explanation. If the dataset mistakenly labels the unsafe response as chosen, DPO receives a misleading training signal and can reinforce the wrong behaviour.

Real-world application

Review preference labels, remove unreliable comparisons and use clear evaluation guidelines when building datasets.
Key takeaway

Incorrect preference labels can directly teach the model undesirable behaviour.