OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / How DPO Works: Full Mechanics
Lesson 227 of 415 6 min read
6.5.4

Token-Level Gradient Analysis

Understand how the DPO gradient reaches individual response tokens and how its shared scaling factor affects learning.

What is it?

Every chosen and rejected response token receives a gradient. A shared sigmoid scaling factor controls the strength of the gradient across both sequences.

Key points

1. Gradients reach every response-token position. 2. The scaling factor is σ(−β × hθ). 3. When hθ is small, the scaling factor is approximately 0.5. 4. When hθ is large, the scaling factor approaches zero. 5. Chosen-token probabilities are pushed upward. 6. Rejected-token probabilities are pushed downward. 7. The comparison is relative to the reference policy.

Adaptive Gradient Strength

The chosen-token gradient contains the scaling factor: −β × σ(−β × hθ) When the model cannot distinguish chosen from rejected responses, hθ is small and the gradient remains strong. When the model already prefers the chosen response, hθ is large and the gradient becomes small. The same scaling factor is shared across token positions in both responses.

Real-world application

Monitor preference margins and gradient behaviour to identify whether training examples still provide useful learning signals.
Key takeaway

DPO adapts gradient strength according to how well the model already distinguishes the preference pair.