8.2.2
KTO Loss Function and Prospect Theory
Understand desirable and undesirable response utility, loss aversion, weighting and the reference-model baseline.
What is it?
KTO uses a loss with separate terms for desirable and undesirable responses. Its utility function compares the policy response likelihood with the reference policy and uses an expected KL divergence baseline. Prospect theory motivates diminishing returns for desirable responses and stronger penalties for undesirable responses.
Key points
The loss has desirable and undesirable response terms.|v(x,y) uses the policy-to-reference log-probability ratio.|β controls the scaling inside the utility function.|zref is the expected KL divergence running baseline.|λw and λl weight desirable and undesirable examples.|The source allows λl to be greater than λw.
Understanding KTO loss and loss aversion
The source gives the loss as:
L_KTO = E_yw[λw(1 − v(x, yw))] + E_yl[λl · v(x, yl)]
where:
v(x, y) = σ(β log(πθ(y|x) / πref(y|x)) − zref)
Here, yw is a desirable response, yl is an undesirable response, and zref is the expected KL divergence used as a running baseline. The undesirable response weight can be increased to give negative feedback a stronger influence.
Real-world application
Adjust desirable_weight and undesirable_weight to reflect the importance assigned to positive and negative feedback.
Key takeaway
KTO applies separate utility terms to desirable and undesirable responses, with configurable weights.