OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / Gradient Analysis
Lesson 214 of 415 5 min read
6.3.1

Understanding the DPO Gradient

Understand the mathematical decomposition of the DPO gradient.

What is it?

The DPO gradient combines a sigmoid weighting term with the difference between the chosen and rejected response log-probability gradients.

Key points

1. The gradient is with respect to θ. 2. y_w is the chosen response. 3. y_l is the rejected response. 4. The two log-probability gradients are compared. 5. A weighting term controls update strength.

DPO Gradient

∇_θ L = −β σ(−r_hat_w + r_hat_l) × [∇_θ log π_θ(y_w|x) − ∇_θ log π_θ(y_l|x)] This is the gradient decomposition in Equation 6.2.

Real-world application

Use gradient decomposition to understand how DPO changes model parameters.
Key takeaway

DPO uses a weighted comparison of chosen and rejected response gradients.