8.4.2
ORPO Loss Function: SFT and Odds Ratio
Understand the supervised fine-tuning term and odds-ratio preference alignment term in the ORPO objective.
What is it?
ORPO combines two loss components. The SFT component applies standard negative log-likelihood to the chosen response. The preference component uses the log odds ratio between chosen and rejected responses to encourage the model to prefer the chosen response.
Key points
L_ORPO = L_SFT − λ × log σ(log(odds(chosen) / odds(rejected))); oddsθ(y|x) = Pθ(y|x) / (1 − Pθ(y|x)); SFT trains the chosen response; odds-ratio term performs preference alignment; λ controls the preference term.
Understanding the Two ORPO Loss Components
The SFT term teaches the model to generate the chosen answer. The odds-ratio term pushes the model towards the chosen response relative to the rejected response. Both are optimized together.
Real-world application
When reviewing ORPO training, inspect both the supervised learning objective and the preference alignment objective.
Key takeaway
The SFT term anchors generation while the odds-ratio term teaches preference.