OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / Motivation
Lesson 207 of 415 6 min read
6.1.5

DPO and the RLHF Objective

Understand how DPO converts preference optimization into a supervised learning objective.

What is it?

DPO substitutes the policy-based reward expression into the Bradley-Terry preference model. The normalization term cancels, producing a binary cross-entropy objective.

Key points

1. Define implicit reward using policy and reference probabilities. 2. Compare chosen and rejected responses. 3. The shared normalization term cancels. 4. Apply the sigmoid function. 5. Minimize the negative log preference probability.

DPO Objective

Implicit reward: r_hat(x,y) = β log(π_θ(y|x) / π_ref(y|x)) DPO encourages: r_hat(x,y_w) > r_hat(x,y_l) The reference model acts as a regularizer.

Real-world application

Train a language model using chosen and rejected response pairs without explicitly training a separate reward model.
Key takeaway

DPO converts the RLHF preference objective into a supervised binary cross-entropy loss.