6.5.2
The DPO Loss Decomposed
Understand the DPO loss, implicit reward margin and token-level expansion of the objective.
What is it?
The DPO loss compares the chosen and rejected responses using their policy-to-reference log-probability differences. Their difference forms the implicit reward margin.
Key points
1. Calculate the chosen response reward proxy.
2. Calculate the rejected response reward proxy.
3. Subtract rejected from chosen to obtain hθ.
4. Multiply the margin by β.
5. Apply sigmoid and negative logarithm.
6. Expand sequence log-probabilities into token-level terms.
Decomposing the DPO Loss
Implicit reward margin:
hθ = log(πθ(yw|x) / πref(yw|x)) − log(πθ(yl|x) / πref(yl|x))
DPO loss:
L_DPO = −E[log σ(β × hθ)]
The chosen and rejected reward proxies are each calculated relative to the reference policy. The loss encourages a positive chosen-versus-rejected margin.
Real-world application
Inspect chosen and rejected reward margins when debugging DPO training or evaluating preference alignment.
Key takeaway
DPO optimizes the difference between chosen and rejected implicit rewards, measured relative to the reference model.