OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 297 of 415 5 min read
7.5.3.3

GSPO vs GRPO: Clipping at Different Levels

Compare per-token clipping in GRPO with sequence-level clipping in GSPO and understand the policy-shift constraint.

What is it?

GRPO clips each token importance ratio independently. GSPO calculates one sequence-level importance weight and clips that single value. This gives GSPO a direct bound on the sequence-level policy shift.

Key points

GRPO: per-token clipping. GSPO: geometric mean of token ratios, followed by sequence-level clipping. GSPO is described in the source as theoretically correct for off-policy importance sampling, while GRPO is an approximation.

Comparing GRPO and GSPO

For a response containing 500 tokens, GRPO applies clipping to 500 individual ratios. GSPO combines those ratios into one length-normalized sequence weight and clips it once. The sequence-level approach directly constrains the overall policy shift.

Real-world application

Consider GSPO when long sequences and repeated gradient updates make per-token clipping insufficient to control sequence-level changes.
Key takeaway

The central difference is the clipping unit: individual tokens in GRPO versus the whole sequence in GSPO.