OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / RL Foundations for Language Models / Two Paradigms for RL in LLMs
Lesson 140 of 415 5 min read
4.1.1

The Two Paradigms of RL in LLMs

Understand the two broad goals of reinforcement learning for language models.

What is it?

Reinforcement learning for language models follows two broad paradigms. The first focuses on aligning models with human preferences. The second focuses on teaching new capabilities through rewards that can be objectively verified.

Key points

1. Paradigm 1: Alignment via Human Preferences (RLHF/DPO). 2. Goal: Make models helpful, harmless and honest. 3. Paradigm 2: Capability Enhancement via Verifiable Rewards (RLVR). 4. Goal: Teach reasoning, mathematics and code generation. 5. RLHF uses human preference judgments. 6. RLVR uses verifiable outcomes. 7. Both use reinforcement-learning techniques to improve model behaviour.

Example: Two different training goals

Consider an AI assistant being trained for two tasks. Alignment task: Given two answers, human evaluators identify which answer is more helpful and safe. The model learns to prefer responses that meet those expectations. Capability task: The model solves a mathematics problem. A verifier checks whether the final answer is correct. The first approach improves alignment with human preferences. The second teaches capabilities using objectively checkable results.

Real-world application

These two paradigms help determine whether a language model should be trained primarily for preference alignment or for measurable task performance.
Key takeaway

RLHF/DPO focuses on human-preference alignment; RLVR focuses on capabilities trained through verifiable rewards.