4.3.5
How the RLVR Pipeline Differs from RLHF
Understand how reasoning traces and verifiers replace the first two stages while retaining RL optimization.
What is it?
For RL from Verifiable Rewards (RLVR), the first two stages of the classic RLHF pipeline are replaced. The SFT model is trained on reasoning traces, and the reward model is replaced by a verifier that checks outcomes such as mathematical correctness. The third stage continues using PPO or GRPO.
Key points
1. RLVR is used for reasoning and agentic training.
2. Stage 1 uses reasoning traces to train the SFT model.
3. Stage 2 replaces the learned reward model with a verifier.
4. A verifier can check mathematical correctness.
5. The verifier provides the reward signal.
6. Stage 3 remains PPO or GRPO optimization.
7. Optimization is performed against the verifiable reward signal.
Example: Mathematics reasoning model
A model is trained on reasoning traces that demonstrate how to solve mathematics problems.
It generates a solution to a new problem.
A verifier checks whether the final answer is mathematically correct.
The verifier supplies the reward signal to PPO or GRPO optimization.
Unlike classic RLHF, this pipeline does not require a human-preference reward model for this training objective.
Real-world application
RLVR is useful for reasoning, mathematics, programming and agentic tasks where results can be objectively checked.
Key takeaway
RLVR replaces the preference-based first two stages with reasoning-trace SFT and a verifier, while retaining PPO or GRPO optimization.