6.3.4
Learning from Confusing Preference Pairs
Understand why confusing preference pairs receive stronger gradient signals.
What is it?
DPO focuses stronger learning signals on confusing pairs where the model currently prefers the rejected response.
Key points
1. Confusing pairs are difficult to distinguish.
2. The rejected response may have a higher implicit reward.
3. The sigmoid weight becomes larger.
4. A stronger corrective update is produced.
5. Already-learned preferences receive smaller updates.
Correcting a Wrong Preference
Suppose the model prefers a technical explanation, but evaluators choose a simpler explanation for a child.
The stronger gradient encourages the model to favour the chosen response.
Real-world application
Review difficult preference examples to identify where the model needs better alignment.
Key takeaway
DPO gives stronger corrective signals to confusing preference pairs.