6.9.6
SLiC-HF – Sequence Likelihood Calibration
Learn how SLiC-HF replaces the smooth DPO log-sigmoid loss with a hinge loss based on a target margin.
What is it?
DPO uses a smooth log-sigmoid loss, which can converge slowly when the preference margin is large. SLiC-HF uses a hinge loss that becomes zero when the winning response exceeds the losing response by the target margin δ, and is linear otherwise.
Key points
SLiC-HF uses a hinge loss.
δ is the target margin threshold.
When the model reaches the required margin, the loss is zero.
When the margin is below δ, the loss remains active.
The source describes the method as simpler and faster, with competitive performance.
TRL setting: loss_type = "hinge", beta = 0.1.
Understanding the Hinge Loss
Suppose the target margin δ is 1. If the model already assigns a margin of 1.2 between the winning and losing responses, the hinge loss is zero. If the margin is only 0.6, the loss remains positive and training can continue to increase the separation.
Real-world application
Use a hinge-loss configuration when applying the SLiC-HF approach. The target margin determines when the loss becomes zero.
Key takeaway
SLiC-HF uses a margin-based hinge loss instead of DPO’s smooth log-sigmoid loss.