OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / Gradient Analysis
Lesson 218 of 415 6 min read
6.3.5

Worked DPO Gradient Example

Work through the DPO gradient example using implicit rewards and sigmoid probability.

What is it?

The book demonstrates DPO using a quantum-entanglement explanation for a 10-year-old. The implicit rewards and sigmoid probability show why the model receives a strong gradient.

Key points

1. β = 0.1. 2. Chosen policy log-probability = −15.3. 3. Chosen reference log-probability = −16.1. 4. Rejected policy log-probability = −12.8. 5. Rejected reference log-probability = −12.5. 6. Chosen implicit reward = 0.08. 7. Rejected implicit reward = −0.03. 8. Sigmoid probability = 0.527. 9. Loss ≈ 0.64.

Quantum Entanglement Example

Prompt: Explain quantum entanglement to a 10-year-old. Chosen response: Imagine you have two magic coins. When you flip one and it is heads, the other one instantly becomes tails, no matter how far apart they are! Rejected response: Quantum entanglement is a phenomenon where two particles become correlated such that the quantum state of one particle cannot be described independently. Chosen reward = 0.1 × (−15.3 + 16.1) = 0.08 Rejected reward = 0.1 × (−12.8 + 12.5) = −0.03 Preference probability = σ(0.11) = 0.527 Loss = −log(0.527) ≈ 0.64

Real-world application

Use the example to understand implicit rewards, preference probabilities and the strength of the DPO gradient.
Key takeaway

When the model barely prefers the chosen response, DPO produces a strong corrective learning signal.