9.1.4
Bradley-Terry Assumptions and Margin Loss
Study the assumptions behind pairwise preference learning and how margin loss encourages a minimum reward gap.
What is it?
The Bradley-Terry model relies on assumptions about how preferences are formed. A margin loss extension can require the preferred response to receive a reward that exceeds the rejected response by a minimum amount.
Key points
The four assumptions are: preferences are transitive; preferences are determined by a scalar reward; preference probability depends only on reward difference; and preferences are independent across pairs with no annotator effects. These assumptions can be violated in practice. A margin m modifies the loss to −(1/N) Σ log σ(rϕ(yw,q) − rϕ(yl,q) − m), encouraging a minimum reward gap.
Adding a Margin to Preference Learning
Suppose the chosen response scores 1.2 and the rejected response scores 1.0. Their gap is only 0.2. A margin of 0.5 requires a larger separation, so the model continues to receive a learning signal until the preferred response is sufficiently ahead.
Real-world application
Use margin loss when the training objective should encourage a minimum separation between chosen and rejected response rewards. Consider richer ranking or multidimensional models when Bradley-Terry assumptions do not fit the data.
Key takeaway
Preference learning depends on modelling assumptions; margin loss encourages a stronger separation between preferred and rejected responses.