OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / RL Foundations for Language Models / Two Paradigms for RL in LLMs
Lesson 144 of 415 5 min read
4.1.5

Choosing Between Alignment and Capability Training

Understand the different training objectives and reward sources used by RLHF and RLVR.

What is it?

The choice between alignment and capability training depends on the objective and the availability of a suitable reward source. Human preference training evaluates response quality, while verifiable-reward training evaluates measurable task outcomes.

Key points

1. Choose RLHF when human preferences define the desired response quality. 2. RLHF requires preference comparisons and a learned reward model. 3. Choose DPO when preference pairs are available and direct supervised optimization is preferred. 4. Choose RLVR when task outcomes can be objectively verified. 5. Mathematics can use answer-correctness checks. 6. Code can use automated tests. 7. GRPO with rule-based rewards is the example highlighted for DeepSeek-R1. 8. Both paradigms aim to improve the language model, but optimize different objectives.

Example: Selecting a training approach

Use case A — Customer-support assistant: The goal is helpful, safe and clear communication. Human preference comparisons can identify better responses. RLHF or DPO can be appropriate. Use case B — Mathematics assistant: The goal is solving problems correctly. A verifier can check final answers. RLVR can provide the training signal. Use case C — Code-generation assistant: The goal is producing working code. Automated tests can verify whether generated code passes expected checks. RLVR can train toward that outcome.

Real-world application

The distinction helps teams select training data, reward sources and optimization methods according to the capability or alignment objective.
Key takeaway

Select preference-based training for human-defined quality and verifiable-reward training for objectively checkable outcomes.