OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / RL Foundations for Language Models / Roadmap of This Part
Lesson 159 of 415 5 min read
4.4.5

Reward Modeling

Study preference-based reward models, process rewards and outcome rewards.

What is it?

Reward modeling converts preference judgments or other feedback into numerical signals that guide policy optimization. The chapter covers Bradley-Terry models, process versus outcome rewards, rule-based rewards for RLVR and multi-objective combinations.

Key points

1. Bradley-Terry models learn from pairwise preferences. 2. Process rewards evaluate intermediate reasoning steps. 3. Outcome rewards evaluate the final result. 4. Rule-based rewards provide verifiable feedback for RLVR. 5. Multiple reward objectives can be combined. 6. Reward design influences the behaviour learned by the policy. 7. Poorly designed rewards can encourage reward hacking.

Example: Rewarding a reasoning model

A mathematics model generates a multi-step solution. A process reward model can evaluate intermediate reasoning steps. An outcome reward can evaluate whether the final answer is correct. A rule-based verifier can check mathematical correctness. A training system may combine different reward signals to guide the model.

Real-world application

Reward modeling is useful when designing feedback signals for preference alignment, reasoning and verifiable task completion.
Key takeaway

Reward models and verifiers convert quality or correctness feedback into signals that guide policy learning.