4.3.2
Stage 2: Reward Model Training
Understand how human preference comparisons train a reward model using the Bradley-Terry objective.
What is it?
Reward Model Training is the second stage of RLHF. Human evaluators compare responses generated for the same prompt. These preference pairs are used to train a reward model Rφ(x,y) using the Bradley-Terry objective.
Key points
1. Collect human preference comparisons.
2. Each comparison contains a preferred response yw and a rejected response yl.
3. Both responses correspond to the same prompt.
4. The reward model receives the prompt and a response.
5. It produces a scalar reward score Rφ(x,y).
6. The Bradley-Terry objective trains the model to score preferred responses higher.
7. The reward model becomes the feedback source for RL optimization.
Example: Learning from response preferences
Prompt: Explain how to secure an online account.
Response A: Recommends a strong password, two-factor authentication and avoiding suspicious links.
Response B: Recommends using the same password everywhere.
Human evaluators prefer Response A.
The preference pair teaches the reward model to assign a higher score to Response A than Response B.
The reward model learns the quality preferences from many such comparisons.
Real-world application
Reward models are useful when response quality depends on human judgment rather than a simple objectively verifiable answer.
Key takeaway
Human preference comparisons train the reward model to distinguish better responses from rejected responses.