OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / DPO Extensions and Variants
Lesson 253 of 415 8 min read
6.9.8

SimPO – Simple Preference Optimisation

Explore reference-free preference learning using length-normalised response log-probabilities and a target reward margin.

What is it?

SimPO is a reference-free preference optimisation method. Instead of using a separate reference model to calculate implicit rewards, it uses the policy’s average response log-probability with length normalisation. A target reward margin γ ensures the winning response scores above the losing response.

Key points

SimPO implicit reward: r_SimPO(y|q) = β/|y| × log π_θ(y|q). The reward uses average log-probability per response token. γ is the target reward margin and must be greater than zero. DPO uses a reference model and ratio-based implicit reward. ORPO is reference-free and adds an odds-ratio term to the SFT loss. SimPO is reference-free and uses length-normalised log-probability plus a margin. TRL example: loss_type = "simpo", simpo_gamma = 0.5, beta = 2.5; no reference model is required.

Comparing DPO, ORPO and SimPO

Suppose a model tends to prefer longer responses because their total log-probability behaves differently from shorter responses. SimPO normalises log-probability by response length and applies a target margin between preferred and rejected responses. Unlike DPO, it does not require a separate reference model.

Real-world application

Use SimPO when reference-free preference learning is desirable. In TRL, the source demonstrates setting loss_type to "simpo", configuring simpo_gamma and beta, and passing ref_model=None.
Key takeaway

SimPO removes the reference model and uses length-normalised response log-probability with a target reward margin.