6.3.2
The Gradient Weight and Sigmoid Function
Understand how the sigmoid weighting term determines gradient strength.
What is it?
The sigmoid weighting term becomes larger when the model currently prefers the rejected response, creating a stronger corrective gradient.
Key points
1. The weight is β times a sigmoid value.
2. The sigmoid input is rejected reward minus chosen reward.
3. A wrong preference produces a larger weight.
4. A clearly correct preference produces a smaller weight.
Understanding the Gradient Weight
Weight = β × σ(−r_hat_w + r_hat_l)
When the rejected response has a higher implicit reward, the weighting term increases.
Real-world application
Monitor preference margins to understand which training examples produce stronger updates.
Key takeaway
The gradient weight focuses learning on difficult preference pairs.