OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Process Reward Models vs Outcome Reward Models
Lesson 366 of 415 5 min read
9.4.1

Outcome Reward Models and Process Reward Models

Compare final-answer rewards with rewards assigned to individual reasoning steps.

What is it?

An Outcome Reward Model (ORM) evaluates only the final answer. A Process Reward Model (PRM) evaluates each reasoning step. They differ in reward signal, training data, annotation cost, credit assignment, reward hacking risk and inference cost.

Key points

ORM: final-answer reward; training data contains prompt, answer and correctness label. PRM: step-level reward; training data contains prompt, reasoning steps and step labels. ORM has low annotation and inference cost, but sparse credit assignment. PRM has higher annotation and inference cost, but dense credit assignment. The source identifies ORM as easier to hack and PRM as harder to hack. ORM suits simple tasks; PRM suits multi-step reasoning.

Evaluating a Multi-Step Answer

A model solves a maths problem in five steps and gives the correct final answer. An ORM evaluates the final answer only. A PRM evaluates the individual reasoning steps, allowing the training process to distinguish valid reasoning from incorrect intermediate work.

Real-world application

Use ORM when final-answer evaluation is sufficient and low cost is important. Consider PRM when intermediate reasoning quality matters.
Key takeaway

ORM evaluates the outcome; PRM evaluates the process that produces the outcome.