OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 318 of 415 6 min read
7.5.2.5

Dynamic Sampling and DAPO Implementation

Understand resampling of zero-variance groups and the key TRL configuration options for DAPO.

What is it?

DAPO dynamically re-samples prompts when every response in a group receives the same reward. Such groups have zero advantage after normalization and provide no useful gradient signal. The source also provides a TRL configuration for enabling DAPO components.

Key points

Dynamic sampling re-samples all-correct or all-incorrect groups. TRL settings shown: epsilon=0.2; epsilon_high=0.28; loss_type="dapo"; mask_truncated_completions=True; max_completion_length=1024; num_generations=8. DAPO loss internally handles zero-variance group filtering.

DAPO Configuration in TRL

A GRPOConfig can enable DAPO using loss_type="dapo", epsilon=0.2, epsilon_high=0.28 and mask_truncated_completions=True. With num_generations=8, the trainer generates groups of eight completions. GRPOTrainer is then initialized with the model, reward function, configuration and training dataset before calling trainer.train().

Real-world application

Consider DAPO for long-form reasoning tasks, frequent length-limit hits, collapsing reward variance, loss spikes or entropy collapse. The source recommends it as a drop-in improvement over base GRPO for most tasks.
Key takeaway

DAPO combines adaptive clipping, token-level aggregation, completion-length handling and dynamic sampling to improve GRPO training stability.