6.1.4
The Key Insight Behind DPO
Understand the closed-form relationship between the optimal policy and the RLHF objective.
What is it?
The optimal policy under the KL-regularized RLHF objective has a closed-form solution. This mathematical relationship allows reward to be expressed using policy probabilities.
Key points
1. Start with reward maximization and KL regularization.
2. Derive the optimal policy.
3. Express reward using policy probabilities.
4. Compare chosen and rejected responses.
5. Use the resulting relationship to derive DPO.
The Key Mathematical Insight
π*(y|x) = π_ref(y|x) × exp(r(x,y)/β) / Z(x)
The optimal policy is proportional to the reference policy multiplied by an exponential reward term.
Real-world application
This relationship provides the foundation for replacing an explicit reward model with a policy-based preference objective.
Key takeaway
The closed-form optimal policy is the central mathematical insight behind DPO.