7.5.2.3
Token-Level Loss Aggregation
Understand how normalizing loss by total tokens prevents long completions from dominating the gradient signal.
What is it?
Base GRPO divides its loss by the number of generated sequences. DAPO instead divides by the total number of tokens across all sequences. This prevents long completions from dominating the gradient simply because they contain more tokens.
Key points
Base GRPO aggregation uses the number of sequences G. DAPO token-level aggregation uses the sum of completion lengths across the group. Each token contributes to the aggregated loss, normalized by the total token count.
Short and Long Completions
Imagine a group containing one completion of 100 tokens and another of 1,000 tokens. Sequence-level averaging can give both sequences equal overall weight. Token-level aggregation normalizes across the 1,100 total tokens, preventing the longer completion from receiving disproportionate influence merely because it is longer.
Real-world application
For variable-length reasoning tasks, consider token-level aggregation when long responses would otherwise dominate training updates.
Key takeaway
Token-level loss aggregation makes the loss normalization depend on total generated tokens rather than the number of sequences.