7.5.2.1
DAPO: Why Adaptive Policy Optimization?
Understand the limitations of symmetric clipping in base GRPO and the motivation for DAPO.
What is it?
Base GRPO uses symmetric clipping, constraining probability increases and decreases in the same way. DAPO introduces five targeted improvements because exploration and exploitation have different risks. Increasing the probability of a good token can be useful, while suppressing a token from a bad completion can be harmful if that token was neutral.
Key points
DAPO = Dynamic Adaptive Policy Optimization. It introduces five targeted components: asymmetric clipping, token-level loss aggregation, overlong filtering, soft overlong punishment and dynamic sampling. The goal is improved training stability and final performance.
Why Symmetric Clipping Can Be Limiting
Suppose a response contains a useful token in a high-reward completion. Increasing its probability can reinforce useful behavior. A token appearing in a low-reward completion may nevertheless be neutral. Suppressing it too strongly can be undesirable. DAPO therefore treats positive and non-positive advantages differently.
Real-world application
Consider DAPO when base GRPO shows instability or when training requires better handling of exploration, long completions and reward variation.
Key takeaway
DAPO modifies specific parts of GRPO rather than replacing the entire group-relative training approach.