OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 274 of 415 5 min read
7.5.2.1

DAPO: Why Adaptive Policy Optimization?

Understand the limitations of symmetric clipping in base GRPO and the motivation for DAPO.

What is it?

Base GRPO uses symmetric clipping, constraining probability increases and decreases in the same way. DAPO introduces five targeted improvements because exploration and exploitation have different risks. Increasing the probability of a good token can be useful, while suppressing a token from a bad completion can be harmful if that token was neutral.

Key points

DAPO = Dynamic Adaptive Policy Optimization. It introduces five targeted components: asymmetric clipping, token-level loss aggregation, overlong filtering, soft overlong punishment and dynamic sampling. The goal is improved training stability and final performance.

Why Symmetric Clipping Can Be Limiting

Suppose a response contains a useful token in a high-reward completion. Increasing its probability can reinforce useful behavior. A token appearing in a low-reward completion may nevertheless be neutral. Suppressing it too strongly can be undesirable. DAPO therefore treats positive and non-positive advantages differently.

Real-world application

Consider DAPO when base GRPO shows instability or when training requires better handling of exploration, long completions and reward variation.
Key takeaway

DAPO modifies specific parts of GRPO rather than replacing the entire group-relative training approach.