6.3.1
Understanding the DPO Gradient
Understand the mathematical decomposition of the DPO gradient.
What is it?
The DPO gradient combines a sigmoid weighting term with the difference between the chosen and rejected response log-probability gradients.
Key points
1. The gradient is with respect to θ.
2. y_w is the chosen response.
3. y_l is the rejected response.
4. The two log-probability gradients are compared.
5. A weighting term controls update strength.
DPO Gradient
∇_θ L = −β σ(−r_hat_w + r_hat_l) × [∇_θ log π_θ(y_w|x) − ∇_θ log π_θ(y_l|x)]
This is the gradient decomposition in Equation 6.2.
Real-world application
Use gradient decomposition to understand how DPO changes model parameters.
Key takeaway
DPO uses a weighted comparison of chosen and rejected response gradients.