OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / RL Foundations for Language Models / Two Paradigms for RL in LLMs
Lesson 141 of 415 7 min read
4.1.2

Alignment via Human Preferences: RLHF and DPO

Understand how human preference comparisons are used to align language models.

What is it?

Alignment via Human Preferences uses human judgments to teach language models which responses are better. RLHF trains a reward model from pairwise human comparisons and then optimizes the policy to maximize the learned reward. DPO simplifies this process by converting preferences directly into a supervised loss without a separate reward model.

Key points

1. RLHF means Reinforcement Learning from Human Feedback. 2. Human evaluators compare two responses to the same prompt. 3. A reward model learns from these pairwise judgments. 4. The policy is optimized to maximize the learned reward. 5. DPO means Direct Preference Optimization. 6. DPO eliminates the separate reward model. 7. DPO converts preferences directly into a supervised loss. 8. Both approaches aim to produce aligned assistants that follow instructions and respect safety constraints.

Example: Choosing the better assistant response

Prompt: Explain how to protect an online account. Response A: Provides clear security steps and warns against sharing passwords. Response B: Gives incomplete and potentially unsafe advice. Human evaluators prefer Response A. In RLHF, these comparisons train a reward model that scores responses. The policy is then optimized against that reward. In DPO, the preferred and rejected responses are used directly to train the policy through a preference-based supervised loss.

Real-world application

RLHF and DPO are useful when human judgment is needed to evaluate qualities such as helpfulness, harmlessness, honesty, instruction following and response style.
Key takeaway

RLHF learns a reward model from human comparisons; DPO directly optimizes the policy from preference pairs.