7.5.2.4
Overlong Filtering and Soft Punishment
Learn how DAPO handles truncated completions through hard filtering and smoothly increasing length penalties.
What is it?
DAPO addresses completions that reach the generation limit without producing an EOS token. Hard overlong filtering masks truncated completions entirely. Soft overlong punishment instead applies a length penalty that increases smoothly as a completion approaches the maximum length.
Key points
Hard filtering: a completion is retained only when EOS is present. Soft punishment: subtract a penalty based on completion length, the safe threshold Lcache and maximum length Lmax. The penalty coefficient is λ.
Hard Filtering vs Soft Punishment
If a completion is cut off before EOS, hard filtering excludes it from the loss. With soft punishment, the reward is reduced according to how far the completion exceeds the safe length threshold. This offers a gradual alternative to excluding the completion entirely.
Real-world application
Use overlong filtering when truncated outputs provide misleading training signals. Consider soft punishment when a gradual length penalty is more appropriate.
Key takeaway
Truncated completions can carry misleading signals; DAPO provides both hard filtering and a softer length-based penalty.