6.4.4
Setting DPO Training Parameters
Understand DPOConfig parameters and the training configuration.
What is it?
DPOConfig defines the optimization, sequence length, batch size, precision, evaluation and checkpoint settings used during training.
Key points
1. beta = 0.1.
2. learning_rate = 5e-7.
3. loss_type = sigmoid.
4. max_length = 2048.
5. max_prompt_length = 1024.
6. Per-device batch size = 2.
7. Gradient accumulation steps = 8.
8. One training epoch.
9. bfloat16 enabled.
DPO Training Configuration
dpo_config = DPOConfig(
output_dir="./dpo_output",
beta=0.1,
learning_rate=5e-7,
loss_type="sigmoid",
max_length=2048,
max_prompt_length=1024,
per_device_train_batch_size=2,
gradient_accumulation_steps=8,
gradient_checkpointing=True,
bf16=True,
num_train_epochs=1,
warmup_ratio=0.1,
logging_steps=10,
eval_strategy="steps",
eval_steps=200,
save_strategy="steps",
save_steps=500
)
Real-world application
Tune training configuration according to available GPU memory, dataset size and observed overfitting.
Key takeaway
DPOConfig controls the training behaviour and resource requirements.