9.4.1
Outcome Reward Models and Process Reward Models
Compare final-answer rewards with rewards assigned to individual reasoning steps.
What is it?
An Outcome Reward Model (ORM) evaluates only the final answer. A Process Reward Model (PRM) evaluates each reasoning step. They differ in reward signal, training data, annotation cost, credit assignment, reward hacking risk and inference cost.
Key points
ORM: final-answer reward; training data contains prompt, answer and correctness label.
PRM: step-level reward; training data contains prompt, reasoning steps and step labels.
ORM has low annotation and inference cost, but sparse credit assignment.
PRM has higher annotation and inference cost, but dense credit assignment.
The source identifies ORM as easier to hack and PRM as harder to hack.
ORM suits simple tasks; PRM suits multi-step reasoning.
Evaluating a Multi-Step Answer
A model solves a maths problem in five steps and gives the correct final answer. An ORM evaluates the final answer only. A PRM evaluates the individual reasoning steps, allowing the training process to distinguish valid reasoning from incorrect intermediate work.
Real-world application
Use ORM when final-answer evaluation is sufficient and low cost is important. Consider PRM when intermediate reasoning quality matters.
Key takeaway
ORM evaluates the outcome; PRM evaluates the process that produces the outcome.