OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / KTO - Kahneman-Tversky Optimization
Lesson 334 of 415 5 min read
8.2.1

Why KTO Works with Unpaired Feedback

Understand why KTO can learn from individual positive and negative feedback without matched response pairs.

What is it?

DPO requires a chosen and rejected response for the same prompt. In practice, feedback is often unpaired: users may rate individual responses with thumbs-up or thumbs-down without comparing two responses. KTO learns from these independent signals.

Key points

DPO requires matched preference pairs.|Production feedback is often unpaired.|KTO accepts individual positive or negative signals.|KTO is inspired by prospect theory.|The source highlights loss aversion: losses can have stronger impact than gains.

Learning from individual feedback

A user gives a thumbs-up to one assistant response. Another user gives a thumbs-down to a different response. These signals do not need to refer to the same prompt or form a chosen-rejected pair. KTO can use each labelled example independently.

Real-world application

Use KTO when collecting individual user feedback rather than carefully matched preference pairs.
Key takeaway

KTO can train from independent desirable and undesirable examples without matched pairs.