OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Bradley-Terry Model: Full Derivation
Lesson 358 of 415 6 min read
9.1.4

Bradley-Terry Assumptions and Margin Loss

Study the assumptions behind pairwise preference learning and how margin loss encourages a minimum reward gap.

What is it?

The Bradley-Terry model relies on assumptions about how preferences are formed. A margin loss extension can require the preferred response to receive a reward that exceeds the rejected response by a minimum amount.

Key points

The four assumptions are: preferences are transitive; preferences are determined by a scalar reward; preference probability depends only on reward difference; and preferences are independent across pairs with no annotator effects. These assumptions can be violated in practice. A margin m modifies the loss to −(1/N) Σ log σ(rϕ(yw,q) − rϕ(yl,q) − m), encouraging a minimum reward gap.

Adding a Margin to Preference Learning

Suppose the chosen response scores 1.2 and the rejected response scores 1.0. Their gap is only 0.2. A margin of 0.5 requires a larger separation, so the model continues to receive a learning signal until the preferred response is sufficiently ahead.

Real-world application

Use margin loss when the training objective should encourage a minimum separation between chosen and rejected response rewards. Consider richer ranking or multidimensional models when Bradley-Terry assumptions do not fit the data.
Key takeaway

Preference learning depends on modelling assumptions; margin loss encourages a stronger separation between preferred and rejected responses.