4.4.5
Reward Modeling
Study preference-based reward models, process rewards and outcome rewards.
What is it?
Reward modeling converts preference judgments or other feedback into numerical signals that guide policy optimization. The chapter covers Bradley-Terry models, process versus outcome rewards, rule-based rewards for RLVR and multi-objective combinations.
Key points
1. Bradley-Terry models learn from pairwise preferences.
2. Process rewards evaluate intermediate reasoning steps.
3. Outcome rewards evaluate the final result.
4. Rule-based rewards provide verifiable feedback for RLVR.
5. Multiple reward objectives can be combined.
6. Reward design influences the behaviour learned by the policy.
7. Poorly designed rewards can encourage reward hacking.
Example: Rewarding a reasoning model
A mathematics model generates a multi-step solution.
A process reward model can evaluate intermediate reasoning steps.
An outcome reward can evaluate whether the final answer is correct.
A rule-based verifier can check mathematical correctness.
A training system may combine different reward signals to guide the model.
Real-world application
Reward modeling is useful when designing feedback signals for preference alignment, reasoning and verifiable task completion.
Key takeaway
Reward models and verifiers convert quality or correctness feedback into signals that guide policy learning.