7.5.3.3
GSPO vs GRPO: Clipping at Different Levels
Compare per-token clipping in GRPO with sequence-level clipping in GSPO and understand the policy-shift constraint.
What is it?
GRPO clips each token importance ratio independently. GSPO calculates one sequence-level importance weight and clips that single value. This gives GSPO a direct bound on the sequence-level policy shift.
Key points
GRPO: per-token clipping. GSPO: geometric mean of token ratios, followed by sequence-level clipping. GSPO is described in the source as theoretically correct for off-policy importance sampling, while GRPO is an approximation.
Comparing GRPO and GSPO
For a response containing 500 tokens, GRPO applies clipping to 500 individual ratios. GSPO combines those ratios into one length-normalized sequence weight and clips it once. The sequence-level approach directly constrains the overall policy shift.
Real-world application
Consider GSPO when long sequences and repeated gradient updates make per-token clipping insufficient to control sequence-level changes.
Key takeaway
The central difference is the clipping unit: individual tokens in GRPO versus the whole sequence in GSPO.