OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Bradley-Terry Model: Full Derivation
Lesson 357 of 415 6 min read
9.1.3

Maximum Likelihood Estimation for Reward Models

Understand how a reward model learns from chosen and rejected response pairs using the Bradley-Terry likelihood objective.

What is it?

Maximum Likelihood Estimation trains the reward model by increasing the probability assigned to observed human preferences. Each training example contains a prompt, a chosen response and a rejected response.

Key points

Dataset D contains preference triples (q, yw, yl). The reward model rϕ is parameterised by neural network parameters ϕ. The Bradley-Terry loss is the negative average log probability of the chosen response being preferred: LBT(ϕ) = −(1/N) Σ log σ(rϕ(yw,q) − rϕ(yl,q)). This is binary cross-entropy with the preferred response as the positive class.

Learning from Chosen and Rejected Responses

For a prompt, an evaluator chooses Response A over Response B. The training example stores A as chosen and B as rejected. The loss encourages the model to assign A a higher reward than B. Repeating this over many preference pairs trains the reward model.

Real-world application

Prepare chosen and rejected response pairs and train the reward model using the Bradley-Terry likelihood objective.
Key takeaway

Maximum Likelihood Estimation teaches the reward model to reproduce observed pairwise preferences.