OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / ORPO - Odds Ratio Preference Optimization
Lesson 343 of 415 7 min read
8.4.2

ORPO Loss Function: SFT and Odds Ratio

Understand the supervised fine-tuning term and odds-ratio preference alignment term in the ORPO objective.

What is it?

ORPO combines two loss components. The SFT component applies standard negative log-likelihood to the chosen response. The preference component uses the log odds ratio between chosen and rejected responses to encourage the model to prefer the chosen response.

Key points

L_ORPO = L_SFT − λ × log σ(log(odds(chosen) / odds(rejected))); oddsθ(y|x) = Pθ(y|x) / (1 − Pθ(y|x)); SFT trains the chosen response; odds-ratio term performs preference alignment; λ controls the preference term.

Understanding the Two ORPO Loss Components

The SFT term teaches the model to generate the chosen answer. The odds-ratio term pushes the model towards the chosen response relative to the rejected response. Both are optimized together.

Real-world application

When reviewing ORPO training, inspect both the supervised learning objective and the preference alignment objective.
Key takeaway

The SFT term anchors generation while the odds-ratio term teaches preference.