7.2.1
Generating Multiple Responses per Prompt
Understand how GRPO samples G completions from the current policy for each input prompt.
What is it?
For each prompt, GRPO samples G completions from the current policy. These responses form a group that can be compared with one another during training.
Key points
For each prompt x, sample G completions.
The responses are sampled from πθ(·|x).
G represents the group size.
Multiple responses provide a basis for relative reward comparison.
Generating a Group of Responses
For a mathematics prompt, GRPO may generate five candidate solutions. Instead of evaluating only one response, it evaluates all five so their rewards can be compared within the same prompt group.
Real-world application
Configure the number of generations in GRPO training. The source TRL example uses num_generations = 8 and temperature = 1.0 to encourage diversity within the group.
Key takeaway
GRPO begins by generating multiple responses to the same prompt.