OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / How DPO Works: Full Mechanics
Lesson 225 of 415 7 min read
6.5.2

The DPO Loss Decomposed

Understand the DPO loss, implicit reward margin and token-level expansion of the objective.

What is it?

The DPO loss compares the chosen and rejected responses using their policy-to-reference log-probability differences. Their difference forms the implicit reward margin.

Key points

1. Calculate the chosen response reward proxy. 2. Calculate the rejected response reward proxy. 3. Subtract rejected from chosen to obtain hθ. 4. Multiply the margin by β. 5. Apply sigmoid and negative logarithm. 6. Expand sequence log-probabilities into token-level terms.

Decomposing the DPO Loss

Implicit reward margin: hθ = log(πθ(yw|x) / πref(yw|x)) − log(πθ(yl|x) / πref(yl|x)) DPO loss: L_DPO = −E[log σ(β × hθ)] The chosen and rejected reward proxies are each calculated relative to the reference policy. The loss encourages a positive chosen-versus-rejected margin.

Real-world application

Inspect chosen and rejected reward margins when debugging DPO training or evaluating preference alignment.
Key takeaway

DPO optimizes the difference between chosen and rejected implicit rewards, measured relative to the reference model.