OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 307 of 415 6 min read
7.5.2.4

Overlong Filtering and Soft Punishment

Learn how DAPO handles truncated completions through hard filtering and smoothly increasing length penalties.

What is it?

DAPO addresses completions that reach the generation limit without producing an EOS token. Hard overlong filtering masks truncated completions entirely. Soft overlong punishment instead applies a length penalty that increases smoothly as a completion approaches the maximum length.

Key points

Hard filtering: a completion is retained only when EOS is present. Soft punishment: subtract a penalty based on completion length, the safe threshold Lcache and maximum length Lmax. The penalty coefficient is λ.

Hard Filtering vs Soft Punishment

If a completion is cut off before EOS, hard filtering excludes it from the loss. With soft punishment, the reward is reduced according to how far the completion exceeds the safe length threshold. This offers a gradual alternative to excluding the completion entirely.

Real-world application

Use overlong filtering when truncated outputs provide misleading training signals. Consider soft punishment when a gradual length penalty is more appropriate.
Key takeaway

Truncated completions can carry misleading signals; DAPO provides both hard filtering and a softer length-based penalty.