OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / Gradient Analysis
Lesson 215 of 415 5 min read
6.3.2

The Gradient Weight and Sigmoid Function

Understand how the sigmoid weighting term determines gradient strength.

What is it?

The sigmoid weighting term becomes larger when the model currently prefers the rejected response, creating a stronger corrective gradient.

Key points

1. The weight is β times a sigmoid value. 2. The sigmoid input is rejected reward minus chosen reward. 3. A wrong preference produces a larger weight. 4. A clearly correct preference produces a smaller weight.

Understanding the Gradient Weight

Weight = β × σ(−r_hat_w + r_hat_l) When the rejected response has a higher implicit reward, the weighting term increases.

Real-world application

Monitor preference margins to understand which training examples produce stronger updates.
Key takeaway

The gradient weight focuses learning on difficult preference pairs.