OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / DPO Extensions and Variants
Lesson 251 of 415 7 min read
6.9.6

SLiC-HF – Sequence Likelihood Calibration

Learn how SLiC-HF replaces the smooth DPO log-sigmoid loss with a hinge loss based on a target margin.

What is it?

DPO uses a smooth log-sigmoid loss, which can converge slowly when the preference margin is large. SLiC-HF uses a hinge loss that becomes zero when the winning response exceeds the losing response by the target margin δ, and is linear otherwise.

Key points

SLiC-HF uses a hinge loss. δ is the target margin threshold. When the model reaches the required margin, the loss is zero. When the margin is below δ, the loss remains active. The source describes the method as simpler and faster, with competitive performance. TRL setting: loss_type = "hinge", beta = 0.1.

Understanding the Hinge Loss

Suppose the target margin δ is 1. If the model already assigns a margin of 1.2 between the winning and losing responses, the hinge loss is zero. If the margin is only 0.6, the loss remains positive and training can continue to increase the separation.

Real-world application

Use a hinge-loss configuration when applying the SLiC-HF approach. The target margin determines when the loss becomes zero.
Key takeaway

SLiC-HF uses a margin-based hinge loss instead of DPO’s smooth log-sigmoid loss.