OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 296 of 415 5 min read
7.5.2.3

Token-Level Loss Aggregation

Understand how normalizing loss by total tokens prevents long completions from dominating the gradient signal.

What is it?

Base GRPO divides its loss by the number of generated sequences. DAPO instead divides by the total number of tokens across all sequences. This prevents long completions from dominating the gradient simply because they contain more tokens.

Key points

Base GRPO aggregation uses the number of sequences G. DAPO token-level aggregation uses the sum of completion lengths across the group. Each token contributes to the aggregated loss, normalized by the total token count.

Short and Long Completions

Imagine a group containing one completion of 100 tokens and another of 1,000 tokens. Sequence-level averaging can give both sequences equal overall weight. Token-level aggregation normalizes across the 1,100 total tokens, preventing the longer completion from receiving disproportionate influence merely because it is longer.

Real-world application

For variable-length reasoning tasks, consider token-level aggregation when long responses would otherwise dominate training updates.
Key takeaway

Token-level loss aggregation makes the loss normalization depend on total generated tokens rather than the number of sequences.