6.5.4
Token-Level Gradient Analysis
Understand how the DPO gradient reaches individual response tokens and how its shared scaling factor affects learning.
What is it?
Every chosen and rejected response token receives a gradient. A shared sigmoid scaling factor controls the strength of the gradient across both sequences.
Key points
1. Gradients reach every response-token position.
2. The scaling factor is σ(−β × hθ).
3. When hθ is small, the scaling factor is approximately 0.5.
4. When hθ is large, the scaling factor approaches zero.
5. Chosen-token probabilities are pushed upward.
6. Rejected-token probabilities are pushed downward.
7. The comparison is relative to the reference policy.
Adaptive Gradient Strength
The chosen-token gradient contains the scaling factor:
−β × σ(−β × hθ)
When the model cannot distinguish chosen from rejected responses, hθ is small and the gradient remains strong.
When the model already prefers the chosen response, hθ is large and the gradient becomes small.
The same scaling factor is shared across token positions in both responses.
Real-world application
Monitor preference margins and gradient behaviour to identify whether training examples still provide useful learning signals.
Key takeaway
DPO adapts gradient strength according to how well the model already distinguishes the preference pair.