6.3.3
Increasing Chosen and Decreasing Rejected Probabilities
Understand how gradient descent changes chosen and rejected response probabilities.
What is it?
The DPO gradient increases the chosen response probability and decreases the rejected response probability.
Key points
1. Chosen response log-probability is pushed upward.
2. Rejected response log-probability is pushed downward.
3. Both responses contribute to the gradient.
4. The objective increases their relative preference margin.
Two Competing Responses
Prompt: Explain a scientific concept.
Chosen response: Clear and suitable explanation.
Rejected response: Less suitable explanation.
Training increases preference for the chosen response and reduces preference for the rejected response.
Real-world application
This mechanism helps align language-model responses with human preferences.
Key takeaway
DPO learns both which response to prefer and which response to avoid.