8.4.3
Implementing ORPO with HuggingFace TRL
Configure ORPOConfig and ORPOTrainer using preference pairs without a reference model.
What is it?
HuggingFace TRL provides ORPOConfig and ORPOTrainer for implementing ORPO. The trainer accepts a model and preference dataset containing prompt, chosen and rejected fields. Unlike DPOTrainer, ORPOTrainer does not require a reference model.
Key points
Import ORPOConfig and ORPOTrainer; configure beta, learning rate and sequence length; provide prompt/chosen/rejected data; use LoRA if required; no ref_model argument is needed; call trainer.train().
A Minimal ORPO Training Configuration
from trl import ORPOConfig, ORPOTrainer
orpo_config = ORPOConfig(
output_dir="./orpo_output",
beta=0.1,
learning_rate=5e-7,
max_length=2048,
per_device_train_batch_size=2,
gradient_accumulation_steps=8,
bf16=True,
num_train_epochs=1,
gradient_checkpointing=True,
)
trainer = ORPOTrainer(
model=model,
args=orpo_config,
train_dataset=pref_dataset,
tokenizer=tokenizer,
peft_config=lora_config,
)
trainer.train()
Real-world application
Use ORPOTrainer when training a model with preference pairs and you want to avoid loading a separate reference model.
Key takeaway
ORPOTrainer simplifies reference-free preference training using one combined objective.