6.3.5
Worked DPO Gradient Example
Work through the DPO gradient example using implicit rewards and sigmoid probability.
What is it?
The book demonstrates DPO using a quantum-entanglement explanation for a 10-year-old. The implicit rewards and sigmoid probability show why the model receives a strong gradient.
Key points
1. β = 0.1.
2. Chosen policy log-probability = −15.3.
3. Chosen reference log-probability = −16.1.
4. Rejected policy log-probability = −12.8.
5. Rejected reference log-probability = −12.5.
6. Chosen implicit reward = 0.08.
7. Rejected implicit reward = −0.03.
8. Sigmoid probability = 0.527.
9. Loss ≈ 0.64.
Quantum Entanglement Example
Prompt: Explain quantum entanglement to a 10-year-old.
Chosen response: Imagine you have two magic coins. When you flip one and it is heads, the other one instantly becomes tails, no matter how far apart they are!
Rejected response: Quantum entanglement is a phenomenon where two particles become correlated such that the quantum state of one particle cannot be described independently.
Chosen reward = 0.1 × (−15.3 + 16.1) = 0.08
Rejected reward = 0.1 × (−12.8 + 12.5) = −0.03
Preference probability = σ(0.11) = 0.527
Loss = −log(0.527) ≈ 0.64
Real-world application
Use the example to understand implicit rewards, preference probabilities and the strength of the DPO gradient.
Key takeaway
When the model barely prefers the chosen response, DPO produces a strong corrective learning signal.