OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / KTO - Kahneman-Tversky Optimization
Lesson 335 of 415 7 min read
8.2.2

KTO Loss Function and Prospect Theory

Understand desirable and undesirable response utility, loss aversion, weighting and the reference-model baseline.

What is it?

KTO uses a loss with separate terms for desirable and undesirable responses. Its utility function compares the policy response likelihood with the reference policy and uses an expected KL divergence baseline. Prospect theory motivates diminishing returns for desirable responses and stronger penalties for undesirable responses.

Key points

The loss has desirable and undesirable response terms.|v(x,y) uses the policy-to-reference log-probability ratio.|β controls the scaling inside the utility function.|zref is the expected KL divergence running baseline.|λw and λl weight desirable and undesirable examples.|The source allows λl to be greater than λw.

Understanding KTO loss and loss aversion

The source gives the loss as: L_KTO = E_yw[λw(1 − v(x, yw))] + E_yl[λl · v(x, yl)] where: v(x, y) = σ(β log(πθ(y|x) / πref(y|x)) − zref) Here, yw is a desirable response, yl is an undesirable response, and zref is the expected KL divergence used as a running baseline. The undesirable response weight can be increased to give negative feedback a stronger influence.

Real-world application

Adjust desirable_weight and undesirable_weight to reflect the importance assigned to positive and negative feedback.
Key takeaway

KTO applies separate utility terms to desirable and undesirable responses, with configurable weights.