9.1.1
The Bradley-Terry Preference Model
Understand how a reward difference determines the probability that one response is preferred over another.
What is it?
The Bradley-Terry model is a probabilistic framework for learning preferences between two responses. It assumes that the probability of preferring one response depends on the difference between their scalar reward scores.
Key points
Given prompt q and responses y1 and y2: P(y1 preferred to y2 | q) = σ(r(y1,q) − r(y2,q)). The reward function r maps a prompt-response pair to a scalar value. A higher reward difference means a higher probability of preference.
Comparing Two Responses
For the same prompt, a reward model scores Response A as 2.0 and Response B as 0.5. The reward difference is 1.5. The Bradley-Terry model converts this difference into a probability that Response A is preferred.
Real-world application
Use pairwise response comparisons to train a model that assigns higher reward to responses preferred by human evaluators.
Key takeaway
Bradley-Terry turns scalar reward differences into probabilities of pairwise preference.