9.4.4
Automatic Process Reward Model Annotation
Explore Monte Carlo rollouts, LLM-as-judge, and formal verification for generating step-level annotations.
What is it?
Process Reward Model annotations identify the quality of individual reasoning steps. The source describes three automatic annotation approaches: Monte Carlo rollouts, LLM-as-judge and formal verification.
Key points
Monte Carlo rollouts: sample multiple completions from an intermediate step and use the fraction reaching the correct answer as its reward.
LLM-as-judge: use a strong language model to evaluate each reasoning step.
Formal verification: use a verifier to check each step, particularly in mathematics and code.
Generating Step-Level Labels
For an intermediate maths step, Monte Carlo annotation can sample several continuations and measure how many reach the correct answer. Alternatively, an LLM judge can assess the step, or a formal verifier can check its mathematical validity.
Real-world application
Use Monte Carlo rollouts when continuation outcomes can estimate step quality, LLM-as-judge for model-based evaluation, and formal verification when deterministic checks are available.
Key takeaway
Automatic PRM annotation can reduce dependence on manually labelled reasoning steps.