OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / RL Foundations for Language Models / The RLHF Pipeline
Lesson 151 of 415 7 min read
4.3.2

Stage 2: Reward Model Training

Understand how human preference comparisons train a reward model using the Bradley-Terry objective.

What is it?

Reward Model Training is the second stage of RLHF. Human evaluators compare responses generated for the same prompt. These preference pairs are used to train a reward model Rφ(x,y) using the Bradley-Terry objective.

Key points

1. Collect human preference comparisons. 2. Each comparison contains a preferred response yw and a rejected response yl. 3. Both responses correspond to the same prompt. 4. The reward model receives the prompt and a response. 5. It produces a scalar reward score Rφ(x,y). 6. The Bradley-Terry objective trains the model to score preferred responses higher. 7. The reward model becomes the feedback source for RL optimization.

Example: Learning from response preferences

Prompt: Explain how to secure an online account. Response A: Recommends a strong password, two-factor authentication and avoiding suspicious links. Response B: Recommends using the same password everywhere. Human evaluators prefer Response A. The preference pair teaches the reward model to assign a higher score to Response A than Response B. The reward model learns the quality preferences from many such comparisons.

Real-world application

Reward models are useful when response quality depends on human judgment rather than a simple objectively verifiable answer.
Key takeaway

Human preference comparisons train the reward model to distinguish better responses from rejected responses.