OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / Mathematical Derivation
Lesson 213 of 415 7 min read
6.2.6

Deriving the DPO Loss

Understand how substitution into the preference model produces the DPO loss.

What is it?

Substitution into the Bradley-Terry model produces the DPO loss, a binary cross-entropy objective over chosen and rejected responses.

Key points

1. Compare chosen and rejected policy probabilities. 2. Normalize against the reference policy. 3. Apply the sigmoid function. 4. Minimize negative log preference probability. 5. β controls the preference margin.

DPO Loss Function

L_DPO = −E[log σ(β log(π_θ(y_w|x)/π_ref(y_w|x)) − β log(π_θ(y_l|x)/π_ref(y_l|x)))] The loss encourages the chosen response to have a higher implicit reward.

Real-world application

Implement preference learning using a fixed reference model and a trainable policy.
Key takeaway

DPO turns the RLHF objective into a supervised preference loss.