OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / ORPO - Odds Ratio Preference Optimization
Lesson 344 of 415 7 min read
8.4.3

Implementing ORPO with HuggingFace TRL

Configure ORPOConfig and ORPOTrainer using preference pairs without a reference model.

What is it?

HuggingFace TRL provides ORPOConfig and ORPOTrainer for implementing ORPO. The trainer accepts a model and preference dataset containing prompt, chosen and rejected fields. Unlike DPOTrainer, ORPOTrainer does not require a reference model.

Key points

Import ORPOConfig and ORPOTrainer; configure beta, learning rate and sequence length; provide prompt/chosen/rejected data; use LoRA if required; no ref_model argument is needed; call trainer.train().

A Minimal ORPO Training Configuration

from trl import ORPOConfig, ORPOTrainer orpo_config = ORPOConfig( output_dir="./orpo_output", beta=0.1, learning_rate=5e-7, max_length=2048, per_device_train_batch_size=2, gradient_accumulation_steps=8, bf16=True, num_train_epochs=1, gradient_checkpointing=True, ) trainer = ORPOTrainer( model=model, args=orpo_config, train_dataset=pref_dataset, tokenizer=tokenizer, peft_config=lora_config, ) trainer.train()

Real-world application

Use ORPOTrainer when training a model with preference pairs and you want to avoid loading a separate reference model.
Key takeaway

ORPOTrainer simplifies reference-free preference training using one combined objective.