6.1.3
Learning Directly from Preferences
Understand how chosen and rejected response pairs provide a direct training signal.
What is it?
DPO learns from preference pairs containing a prompt, a chosen response and a rejected response.
Key points
1. x represents the prompt.
2. y_w represents the chosen response.
3. y_l represents the rejected response.
4. Both responses answer the same prompt.
5. The objective encourages a higher relative score for the chosen response.
A Preference Pair
Prompt: Explain photosynthesis to a school student.
Chosen: A clear and accurate explanation using simple language.
Rejected: An unclear explanation containing incorrect information.
The pair provides a direct preference signal.
Real-world application
Collect response comparisons from human evaluators or domain experts to build preference datasets.
Key takeaway
DPO learns from what people prefer rather than requiring manually assigned numerical rewards.