OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / ORPO - Odds Ratio Preference Optimization
Lesson 342 of 415 5 min read
8.4.1

Why ORPO Eliminates the Reference Model

Understand how ORPO combines SFT and preference alignment without maintaining a separate reference model.

What is it?

ORPO removes the need for a separate reference model by combining supervised fine-tuning and preference alignment in a single training objective. Earlier methods used a reference model either as a separate copy or implicitly through techniques such as LoRA. ORPO instead uses the chosen response to anchor the model while learning to prefer it over the rejected response.

Key points

Eliminates the separate reference model; combines SFT and preference alignment; uses chosen and rejected responses; SFT anchors generation behaviour; reduces model memory requirements.

Why Remove the Reference Model?

A conventional DPO setup may maintain both a policy model and a reference model. ORPO uses one trainable model and combines the learning objectives, reducing the memory burden of preference training.

Real-world application

Consider ORPO when GPU memory is limited and maintaining a separate reference model is impractical.
Key takeaway

ORPO combines supervised fine-tuning and preference alignment without a separate reference model.