6.1.5
DPO and the RLHF Objective
Understand how DPO converts preference optimization into a supervised learning objective.
What is it?
DPO substitutes the policy-based reward expression into the Bradley-Terry preference model. The normalization term cancels, producing a binary cross-entropy objective.
Key points
1. Define implicit reward using policy and reference probabilities.
2. Compare chosen and rejected responses.
3. The shared normalization term cancels.
4. Apply the sigmoid function.
5. Minimize the negative log preference probability.
DPO Objective
Implicit reward:
r_hat(x,y) = β log(π_θ(y|x) / π_ref(y|x))
DPO encourages:
r_hat(x,y_w) > r_hat(x,y_l)
The reference model acts as a regularizer.
Real-world application
Train a language model using chosen and rejected response pairs without explicitly training a separate reward model.
Key takeaway
DPO converts the RLHF preference objective into a supervised binary cross-entropy loss.