OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Bradley-Terry Model: Full Derivation
Lesson 355 of 415 5 min read
9.1.1

The Bradley-Terry Preference Model

Understand how a reward difference determines the probability that one response is preferred over another.

What is it?

The Bradley-Terry model is a probabilistic framework for learning preferences between two responses. It assumes that the probability of preferring one response depends on the difference between their scalar reward scores.

Key points

Given prompt q and responses y1 and y2: P(y1 preferred to y2 | q) = σ(r(y1,q) − r(y2,q)). The reward function r maps a prompt-response pair to a scalar value. A higher reward difference means a higher probability of preference.

Comparing Two Responses

For the same prompt, a reward model scores Response A as 2.0 and Response B as 0.5. The reward difference is 1.5. The Bradley-Terry model converts this difference into a probability that Response A is preferred.

Real-world application

Use pairwise response comparisons to train a model that assigns higher reward to responses preferred by human evaluators.
Key takeaway

Bradley-Terry turns scalar reward differences into probabilities of pairwise preference.