8.1.3
Online DPO Implementation Using TRL
Explore the OnlineDPOTrainer workflow, reward model, configuration and training process in HuggingFace TRL.
What is it?
HuggingFace TRL provides OnlineDPOConfig and OnlineDPOTrainer for implementing Online DPO. The source example loads a causal language model and a sequence-classification reward model, configures generation and training settings, and starts training with fresh preference generation.
Key points
Import OnlineDPOConfig and OnlineDPOTrainer.|Load the policy model.|Load a sequence-classification reward model.|Configure beta, generations, batch size and temperature.|Create OnlineDPOTrainer and call trainer.train().
Online DPO using HuggingFace TRL
from trl import OnlineDPOConfig, OnlineDPOTrainer
from transformers import AutoModelForCausalLM, AutoModelForSequenceClassification
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B-Instruct",
torch_dtype=torch.bfloat16
)
reward_model = AutoModelForSequenceClassification.from_pretrained(
"RLHFlow/ArmoRM-Llama3-8B-v0.1",
torch_dtype=torch.bfloat16
)
online_dpo_config = OnlineDPOConfig(
output_dir="./online_dpo_output",
learning_rate=5e-7,
beta=0.1,
num_generations=4,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
max_new_tokens=512,
temperature=0.7,
bf16=True,
num_train_epochs=1,
logging_steps=10,
)
trainer = OnlineDPOTrainer(
model=model,
reward_model=reward_model,
args=online_dpo_config,
train_dataset=prompt_dataset,
tokenizer=tokenizer,
)
trainer.train()
Real-world application
Use the source example as a reference implementation and check compatibility with your installed TRL version before running it.
Key takeaway
OnlineDPOTrainer uses a policy model, reward model and fresh prompt generations to train preference pairs.