6.4.5
Initializing DPOTrainer and Monitoring Training
Understand DPOTrainer initialization and training metrics.
What is it?
DPOTrainer connects the policy model, reference model configuration, datasets, tokenizer and LoRA adapters. The source uses ref_model=None with LoRA, where the base model serves as the reference.
Key points
1. DPOTrainer initializes training.
2. model is the trainable policy.
3. ref_model=None in the source example.
4. args receives DPOConfig.
5. Train and test datasets are supplied.
6. tokenizer and peft_config are passed.
7. trainer.train() starts optimization.
8. Monitor chosen reward, rejected reward and reward margins.
Initializing DPOTrainer
trainer = DPOTrainer(
model=model,
ref_model=None,
args=dpo_config,
train_dataset=dataset["train"],
eval_dataset=dataset["test"],
tokenizer=tokenizer,
peft_config=lora_config
)
trainer.train()
# Monitor:
# train/rewards/chosen
# train/rewards/rejected
# train/rewards/margins
Real-world application
Track chosen and rejected reward metrics during training to understand whether preference alignment is improving.
Key takeaway
DPOTrainer brings together the model, preference data and training configuration.