9.1.3
Maximum Likelihood Estimation for Reward Models
Understand how a reward model learns from chosen and rejected response pairs using the Bradley-Terry likelihood objective.
What is it?
Maximum Likelihood Estimation trains the reward model by increasing the probability assigned to observed human preferences. Each training example contains a prompt, a chosen response and a rejected response.
Key points
Dataset D contains preference triples (q, yw, yl). The reward model rϕ is parameterised by neural network parameters ϕ. The Bradley-Terry loss is the negative average log probability of the chosen response being preferred: LBT(ϕ) = −(1/N) Σ log σ(rϕ(yw,q) − rϕ(yl,q)). This is binary cross-entropy with the preferred response as the positive class.
Learning from Chosen and Rejected Responses
For a prompt, an evaluator chooses Response A over Response B. The training example stores A as chosen and B as rejected. The loss encourages the model to assign A a higher reward than B. Repeating this over many preference pairs trains the reward model.
Real-world application
Prepare chosen and rejected response pairs and train the reward model using the Bradley-Terry likelihood objective.
Key takeaway
Maximum Likelihood Estimation teaches the reward model to reproduce observed pairwise preferences.