OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Listwise Rank-Based Rewards
Lesson 378 of 415 6 min read
9.7.4

Training Listwise Reward Models with GRPO

Use complete response rankings to train a reward model and provide scalar rewards to GRPO.

What is it?

GRPO naturally generates groups of responses that can be ranked. A listwise reward model can learn from these complete rankings and then assign scalar rewards used by GRPO to calculate advantages.

Key points

Generate N = 8 responses per prompt from the policy. Rank responses using an existing reward model or human annotators. Train the listwise reward model by optimising Plackett-Luce loss on prompt-ranking tuples. Use the trained model to assign scalar rewards to responses. GRPO calculates advantages using A_hat_i = (r_i - μ) / σ. The model learns the relative quality of all responses in the group.

Listwise Reward Training with GRPO

For each prompt, generate eight responses. Obtain a full ranking from human annotators or an existing reward model. Train the listwise reward model on the prompt and ranking. During GRPO training, use its scalar scores as rewards and calculate group-relative advantages.

Real-world application

Build a pipeline that generates response groups, collects rankings, trains a Plackett-Luce reward model and supplies scalar scores to GRPO.
Key takeaway

Listwise reward models fit naturally into GRPO because both operate on groups of responses.