9.4.2
When to Use Process Reward Models
Identify tasks where step-level reward signals provide more useful feedback than final-answer evaluation.
What is it?
PRMs are especially useful when a task requires multi-step reasoning and the final answer alone does not reveal the quality of the intermediate steps.
Key points
Useful for mathematics, code and logic tasks.
Valuable when the final answer is binary but intermediate steps vary in quality.
Can support search methods such as beam search using step scores.
Requires step-level annotations, either supplied or generated automatically.
For sentiment, toxicity and factuality tasks, the source considers ORMs sufficient and much cheaper.
Choosing a Reward Model for a Task
For a multi-step maths problem, two responses may reach the same final answer through different reasoning paths. A PRM can evaluate the quality of each step. For a simple sentiment classification task, an ORM can evaluate the final output without the added cost of step-level scoring.
Real-world application
Choose PRMs for complex reasoning, step-based search and tasks where intermediate correctness matters. Choose ORMs for simpler tasks where final-output evaluation is sufficient.
Key takeaway
Use PRMs when the quality of the reasoning path matters, not just the final answer.