6.2.6
Deriving the DPO Loss
Understand how substitution into the preference model produces the DPO loss.
What is it?
Substitution into the Bradley-Terry model produces the DPO loss, a binary cross-entropy objective over chosen and rejected responses.
Key points
1. Compare chosen and rejected policy probabilities.
2. Normalize against the reference policy.
3. Apply the sigmoid function.
4. Minimize negative log preference probability.
5. β controls the preference margin.
DPO Loss Function
L_DPO = −E[log σ(β log(π_θ(y_w|x)/π_ref(y_w|x)) − β log(π_θ(y_l|x)/π_ref(y_l|x)))]
The loss encourages the chosen response to have a higher implicit reward.
Real-world application
Implement preference learning using a fixed reference model and a trainable policy.
Key takeaway
DPO turns the RLHF objective into a supervised preference loss.