4.1.1
The Two Paradigms of RL in LLMs
Understand the two broad goals of reinforcement learning for language models.
What is it?
Reinforcement learning for language models follows two broad paradigms. The first focuses on aligning models with human preferences. The second focuses on teaching new capabilities through rewards that can be objectively verified.
Key points
1. Paradigm 1: Alignment via Human Preferences (RLHF/DPO).
2. Goal: Make models helpful, harmless and honest.
3. Paradigm 2: Capability Enhancement via Verifiable Rewards (RLVR).
4. Goal: Teach reasoning, mathematics and code generation.
5. RLHF uses human preference judgments.
6. RLVR uses verifiable outcomes.
7. Both use reinforcement-learning techniques to improve model behaviour.
Example: Two different training goals
Consider an AI assistant being trained for two tasks.
Alignment task:
Given two answers, human evaluators identify which answer is more helpful and safe. The model learns to prefer responses that meet those expectations.
Capability task:
The model solves a mathematics problem. A verifier checks whether the final answer is correct.
The first approach improves alignment with human preferences. The second teaches capabilities using objectively checkable results.
Real-world application
These two paradigms help determine whether a language model should be trained primarily for preference alignment or for measurable task performance.
Key takeaway
RLHF/DPO focuses on human-preference alignment; RLVR focuses on capabilities trained through verifiable rewards.