OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / Online DPO
Lesson 332 of 415 8 min read
8.1.3

Online DPO Implementation Using TRL

Explore the OnlineDPOTrainer workflow, reward model, configuration and training process in HuggingFace TRL.

What is it?

HuggingFace TRL provides OnlineDPOConfig and OnlineDPOTrainer for implementing Online DPO. The source example loads a causal language model and a sequence-classification reward model, configures generation and training settings, and starts training with fresh preference generation.

Key points

Import OnlineDPOConfig and OnlineDPOTrainer.|Load the policy model.|Load a sequence-classification reward model.|Configure beta, generations, batch size and temperature.|Create OnlineDPOTrainer and call trainer.train().

Online DPO using HuggingFace TRL

from trl import OnlineDPOConfig, OnlineDPOTrainer from transformers import AutoModelForCausalLM, AutoModelForSequenceClassification model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-3.1-8B-Instruct", torch_dtype=torch.bfloat16 ) reward_model = AutoModelForSequenceClassification.from_pretrained( "RLHFlow/ArmoRM-Llama3-8B-v0.1", torch_dtype=torch.bfloat16 ) online_dpo_config = OnlineDPOConfig( output_dir="./online_dpo_output", learning_rate=5e-7, beta=0.1, num_generations=4, per_device_train_batch_size=4, gradient_accumulation_steps=4, max_new_tokens=512, temperature=0.7, bf16=True, num_train_epochs=1, logging_steps=10, ) trainer = OnlineDPOTrainer( model=model, reward_model=reward_model, args=online_dpo_config, train_dataset=prompt_dataset, tokenizer=tokenizer, ) trainer.train()

Real-world application

Use the source example as a reference implementation and check compatibility with your installed TRL version before running it.
Key takeaway

OnlineDPOTrainer uses a policy model, reward model and fresh prompt generations to train preference pairs.