8.2.4
When to Choose KTO
Identify situations where binary, unpaired and imbalanced feedback make KTO a suitable preference optimisation method.
What is it?
KTO is suitable when feedback is binary but preference pairs are unavailable. The source also highlights production thumbs-up/down data, imbalanced feedback classes and rapid iteration with noisy labels as useful scenarios.
Key points
Use KTO when binary feedback is available without matched pairs.|Production thumbs-up/down data can be used.|KTO handles imbalanced classes, such as 90% positive and 10% negative examples, better than DPO according to the source.|It is useful for rapid iteration with noisy labels.
Choosing KTO for real-world feedback
Imagine a product receives thousands of individual ratings. Most users give positive feedback, while a smaller group gives negative feedback. There may be no matched response pairs. KTO can train on these individual labelled examples.
Real-world application
Consider KTO for user feedback pipelines where collecting paired comparisons is difficult.
Key takeaway
Choose KTO when individual binary feedback is available but matched preference pairs are not.