8.4.1
Why ORPO Eliminates the Reference Model
Understand how ORPO combines SFT and preference alignment without maintaining a separate reference model.
What is it?
ORPO removes the need for a separate reference model by combining supervised fine-tuning and preference alignment in a single training objective. Earlier methods used a reference model either as a separate copy or implicitly through techniques such as LoRA. ORPO instead uses the chosen response to anchor the model while learning to prefer it over the rejected response.
Key points
Eliminates the separate reference model; combines SFT and preference alignment; uses chosen and rejected responses; SFT anchors generation behaviour; reduces model memory requirements.
Why Remove the Reference Model?
A conventional DPO setup may maintain both a policy model and a reference model. ORPO uses one trainable model and combines the learning objectives, reducing the memory burden of preference training.
Real-world application
Consider ORPO when GPU memory is limited and maintaining a separate reference model is impractical.
Key takeaway
ORPO combines supervised fine-tuning and preference alignment without a separate reference model.