8.2.3
KTO Dataset and TRL Implementation
Learn the KTO data format and configure KTOConfig and KTOTrainer in HuggingFace TRL.
What is it?
KTO uses a simple dataset format containing a prompt, one completion and a Boolean label. HuggingFace TRL provides KTOConfig and KTOTrainer to configure and run the training process.
Key points
Dataset fields: prompt, completion and label.|True indicates a desirable response.|False indicates an undesirable response.|Configure beta and example weights.|Create KTOTrainer with the model, reference model, dataset and tokenizer.|The source example also supports ref_model=None with LoRA.
KTO dataset and TRL example
from trl import KTOConfig, KTOTrainer
kto_dataset = [
{
"prompt": "What is 2+2?",
"completion": "The answer is 4.",
"label": True
},
{
"prompt": "What is 2+2?",
"completion": "It might be 5.",
"label": False
}
]
kto_config = KTOConfig(
output_dir="./kto_output",
beta=0.1,
desirable_weight=1.0,
undesirable_weight=1.0,
learning_rate=5e-7,
max_length=2048,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
num_train_epochs=1,
bf16=True,
)
trainer = KTOTrainer(
model=model,
ref_model=ref_model,
args=kto_config,
train_dataset=kto_dataset,
tokenizer=tokenizer,
)
trainer.train()
Real-world application
Convert individual user ratings into Boolean labels and use them as KTO training examples. Check the installed TRL version for API compatibility.
Key takeaway
KTO uses independent labelled completions rather than chosen-rejected pairs.