OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / KTO - Kahneman-Tversky Optimization
Lesson 336 of 415 8 min read
8.2.3

KTO Dataset and TRL Implementation

Learn the KTO data format and configure KTOConfig and KTOTrainer in HuggingFace TRL.

What is it?

KTO uses a simple dataset format containing a prompt, one completion and a Boolean label. HuggingFace TRL provides KTOConfig and KTOTrainer to configure and run the training process.

Key points

Dataset fields: prompt, completion and label.|True indicates a desirable response.|False indicates an undesirable response.|Configure beta and example weights.|Create KTOTrainer with the model, reference model, dataset and tokenizer.|The source example also supports ref_model=None with LoRA.

KTO dataset and TRL example

from trl import KTOConfig, KTOTrainer kto_dataset = [ { "prompt": "What is 2+2?", "completion": "The answer is 4.", "label": True }, { "prompt": "What is 2+2?", "completion": "It might be 5.", "label": False } ] kto_config = KTOConfig( output_dir="./kto_output", beta=0.1, desirable_weight=1.0, undesirable_weight=1.0, learning_rate=5e-7, max_length=2048, per_device_train_batch_size=4, gradient_accumulation_steps=4, num_train_epochs=1, bf16=True, ) trainer = KTOTrainer( model=model, ref_model=ref_model, args=kto_config, train_dataset=kto_dataset, tokenizer=tokenizer, ) trainer.train()

Real-world application

Convert individual user ratings into Boolean labels and use them as KTO training examples. Check the installed TRL version for API compatibility.
Key takeaway

KTO uses independent labelled completions rather than chosen-rejected pairs.