4.4.2
DPO – Direct Preference Optimization
Understand how preference data can be converted into a contrastive supervised loss without a separate RL loop.
What is it?
Direct Preference Optimization (DPO) bypasses the conventional reinforcement learning loop by converting human preferences into a contrastive supervised loss. It is simpler to train than online RL, but less flexible because it relies on preference data rather than continuous policy interaction.
Key points
1. DPO stands for Direct Preference Optimization.
2. Uses preferred and rejected response pairs.
3. Converts preferences into a contrastive supervised loss.
4. Does not require a separate reward model in its standard formulation.
5. Bypasses the conventional online RL optimization loop.
6. Simpler to implement than PPO-based RLHF.
7. Less flexible than online RL.
Example: Learning from preferred answers
A dataset contains two answers to the same question.
Preferred answer: A clear, accurate and helpful explanation.
Rejected answer: An incomplete or misleading explanation.
DPO directly trains the model to increase the relative likelihood of the preferred answer compared with the rejected answer.
It does this using a preference-based loss rather than a separate reward-model-and-RL pipeline.
Real-world application
DPO is useful when high-quality preference pairs are available and a simpler offline preference-training approach is desired.
Key takeaway
DPO converts preference comparisons into a supervised contrastive loss, bypassing the conventional RL loop.