6.9.8
SimPO – Simple Preference Optimisation
Explore reference-free preference learning using length-normalised response log-probabilities and a target reward margin.
What is it?
SimPO is a reference-free preference optimisation method. Instead of using a separate reference model to calculate implicit rewards, it uses the policy’s average response log-probability with length normalisation. A target reward margin γ ensures the winning response scores above the losing response.
Key points
SimPO implicit reward: r_SimPO(y|q) = β/|y| × log π_θ(y|q).
The reward uses average log-probability per response token.
γ is the target reward margin and must be greater than zero.
DPO uses a reference model and ratio-based implicit reward.
ORPO is reference-free and adds an odds-ratio term to the SFT loss.
SimPO is reference-free and uses length-normalised log-probability plus a margin.
TRL example: loss_type = "simpo", simpo_gamma = 0.5, beta = 2.5; no reference model is required.
Comparing DPO, ORPO and SimPO
Suppose a model tends to prefer longer responses because their total log-probability behaves differently from shorter responses. SimPO normalises log-probability by response length and applies a target margin between preferred and rejected responses. Unlike DPO, it does not require a separate reference model.
Real-world application
Use SimPO when reference-free preference learning is desirable. In TRL, the source demonstrates setting loss_type to "simpo", configuring simpo_gamma and beta, and passing ref_model=None.
Key takeaway
SimPO removes the reference model and uses length-normalised response log-probability with a target reward margin.