7.5.3.1
The Off-Policy Problem in GRPO
Understand why multiplying per-token importance ratios can create extreme sequence-level ratios during off-policy training.
What is it?
GRPO clips importance ratios independently at each token. During off-policy training, when the same generated batch is reused for multiple gradient steps, the product of these ratios across a long sequence can become extremely large or small. Individual token ratios may remain within their clipping bounds while the overall sequence-level change becomes excessive.
Key points
Per-token importance ratios are clipped in GRPO. A long sequence can accumulate a very large or small product of ratios. The mismatch becomes more significant when training reuses a batch for multiple gradient steps. This is the off-policy problem addressed by GSPO.
Why a Long Sequence Creates a Problem
Imagine a 500-token response. Each token has an importance ratio close to 1, but multiplying hundreds of ratios can produce an extreme sequence-level ratio. Per-token clipping alone does not directly guarantee a bounded sequence-level policy shift.
Real-world application
Pay particular attention to importance-ratio behavior when reusing generated batches for multiple gradient steps. This is where the sequence-level mismatch becomes important.
Key takeaway
GSPO addresses the mismatch between per-token clipping and the accumulated probability change of an entire sequence.