4.1.2
Alignment via Human Preferences: RLHF and DPO
Understand how human preference comparisons are used to align language models.
What is it?
Alignment via Human Preferences uses human judgments to teach language models which responses are better. RLHF trains a reward model from pairwise human comparisons and then optimizes the policy to maximize the learned reward. DPO simplifies this process by converting preferences directly into a supervised loss without a separate reward model.
Key points
1. RLHF means Reinforcement Learning from Human Feedback.
2. Human evaluators compare two responses to the same prompt.
3. A reward model learns from these pairwise judgments.
4. The policy is optimized to maximize the learned reward.
5. DPO means Direct Preference Optimization.
6. DPO eliminates the separate reward model.
7. DPO converts preferences directly into a supervised loss.
8. Both approaches aim to produce aligned assistants that follow instructions and respect safety constraints.
Example: Choosing the better assistant response
Prompt: Explain how to protect an online account.
Response A: Provides clear security steps and warns against sharing passwords.
Response B: Gives incomplete and potentially unsafe advice.
Human evaluators prefer Response A.
In RLHF, these comparisons train a reward model that scores responses. The policy is then optimized against that reward.
In DPO, the preferred and rejected responses are used directly to train the policy through a preference-based supervised loss.
Real-world application
RLHF and DPO are useful when human judgment is needed to evaluate qualities such as helpfulness, harmlessness, honesty, instruction following and response style.
Key takeaway
RLHF learns a reward model from human comparisons; DPO directly optimizes the policy from preference pairs.