7.2.2
Scoring Responses with a Reward Function
Learn how each generated response receives a reward score based on its quality or correctness.
What is it?
After generating responses, GRPO assigns a reward to each completion using a reward function. The reward can measure correctness or other desired response properties.
Key points
Each response yi receives reward ri = R(x, yi).
The reward function evaluates a response in the context of its prompt.
The source demonstrates binary correctness rewards: 1.0 for correct and 0.0 for incorrect.
Multiple reward functions can be combined.
Scoring Mathematical Responses
Suppose five responses are generated for a maths question. A correctness reward function compares each extracted answer with the expected answer. Correct answers receive 1.0 and incorrect answers receive 0.0. A separate format reward can add 0.5 for correctly formatted LaTeX.
Real-world application
Design reward functions that evaluate the intended task. The source TRL example combines a correctness reward function and a format reward function through reward_funcs.
Key takeaway
GRPO uses reward functions to score every generated response before calculating group-relative advantages.