9.7.4
Training Listwise Reward Models with GRPO
Use complete response rankings to train a reward model and provide scalar rewards to GRPO.
What is it?
GRPO naturally generates groups of responses that can be ranked. A listwise reward model can learn from these complete rankings and then assign scalar rewards used by GRPO to calculate advantages.
Key points
Generate N = 8 responses per prompt from the policy.
Rank responses using an existing reward model or human annotators.
Train the listwise reward model by optimising Plackett-Luce loss on prompt-ranking tuples.
Use the trained model to assign scalar rewards to responses.
GRPO calculates advantages using A_hat_i = (r_i - μ) / σ.
The model learns the relative quality of all responses in the group.
Listwise Reward Training with GRPO
For each prompt, generate eight responses. Obtain a full ranking from human annotators or an existing reward model. Train the listwise reward model on the prompt and ranking. During GRPO training, use its scalar scores as rewards and calculate group-relative advantages.
Real-world application
Build a pipeline that generates response groups, collects rankings, trains a Plackett-Luce reward model and supplies scalar scores to GRPO.
Key takeaway
Listwise reward models fit naturally into GRPO because both operate on groups of responses.