8.2.1
Why KTO Works with Unpaired Feedback
Understand why KTO can learn from individual positive and negative feedback without matched response pairs.
What is it?
DPO requires a chosen and rejected response for the same prompt. In practice, feedback is often unpaired: users may rate individual responses with thumbs-up or thumbs-down without comparing two responses. KTO learns from these independent signals.
Key points
DPO requires matched preference pairs.|Production feedback is often unpaired.|KTO accepts individual positive or negative signals.|KTO is inspired by prospect theory.|The source highlights loss aversion: losses can have stronger impact than gains.
Learning from individual feedback
A user gives a thumbs-up to one assistant response. Another user gives a thumbs-down to a different response. These signals do not need to refer to the same prompt or form a chosen-rejected pair. KTO can use each labelled example independently.
Real-world application
Use KTO when collecting individual user feedback rather than carefully matched preference pairs.
Key takeaway
KTO can train from independent desirable and undesirable examples without matched pairs.