OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 273 of 415 12 min read
7.5.1

Diversity in GRPO Groups

Understand mode collapse in RL training, group diversity mechanisms, diversity-quality trade-offs and verbalized sampling for diverse response generation.

What is it?

GRPO needs diverse responses within each group to compare different approaches. Without diversity, an RL-trained model can collapse into repetitive, high-reward response patterns instead of learning to reason across alternatives.

Key points

Mode collapse: narrow template responses; entropy decreases and the model becomes deterministic; reward hacking becomes easier; generalization suffers. KL penalty is a primary diversity mechanism, but not sufficient alone. Group diversity: temperature 0.8–1.0; group size 8–16; DAPO No Repeat penalty; identical responses produce zero advantage. Diversity methods: entropy bonus, KL penalty, rejection sampling, Best-of-N, diverse DPO pairs and multiple reward models. Monitor response entropy, unique n-gram ratio and reward distribution width. Verbalized Sampling generates multiple candidate responses with stated probabilities.

Example: Generating Diverse Responses

For one prompt, a model generates eight responses. If all eight use the same approach, their rewards may be identical and the group provides little learning signal. Generating distinct but relevant approaches gives GRPO a more useful comparison. Verbalized Sampling can ask the model to produce five different responses with probabilities such as 0.35, 0.25, 0.20, 0.12 and 0.08, then sample from that distribution.

Real-world application

During GRPO training, monitor response entropy, unique n-gram ratio and reward distribution width. If all three decrease together, investigate possible mode collapse. Adjust sampling temperature and group size to encourage distinct, on-topic responses. Consider methods such as entropy bonuses, rejection sampling or Verbalized Sampling when appropriate.
Key takeaway

The goal is not maximum diversity. It is enough meaningful variation to compare approaches while keeping responses relevant and useful.