OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / Motivation
Lesson 205 of 415 5 min read
6.1.3

Learning Directly from Preferences

Understand how chosen and rejected response pairs provide a direct training signal.

What is it?

DPO learns from preference pairs containing a prompt, a chosen response and a rejected response.

Key points

1. x represents the prompt. 2. y_w represents the chosen response. 3. y_l represents the rejected response. 4. Both responses answer the same prompt. 5. The objective encourages a higher relative score for the chosen response.

A Preference Pair

Prompt: Explain photosynthesis to a school student. Chosen: A clear and accurate explanation using simple language. Rejected: An unclear explanation containing incorrect information. The pair provides a direct preference signal.

Real-world application

Collect response comparisons from human evaluators or domain experts to build preference datasets.
Key takeaway

DPO learns from what people prefer rather than requiring manually assigned numerical rewards.