7.5.3.5
When to Use GSPO
Identify when GSPO is useful, particularly when multiple gradient steps reuse generated batches.
What is it?
GSPO is most beneficial when training reuses generated batches for multiple gradient steps. With purely on-policy training and one step per generation, its difference from GRPO is negligible. Off-policy batch reuse can reduce generation costs, which are often the most expensive part of training.
Key points
Most beneficial when steps_per_generation > 1. For on-policy training with steps_per_generation = 1, the difference is negligible. Off-policy training can reduce generation cost. The source identifies generation as the most expensive training step.
Choosing Between GRPO and GSPO
If a training run generates a batch and performs four gradient steps before generating again, it is using off-policy batch reuse. GSPO is particularly relevant in this situation. If the trainer performs only one step per generated batch, the source indicates that the difference is negligible.
Real-world application
Consider GSPO when generation is a major cost and you want to reuse each generated batch for multiple gradient steps. For purely on-policy training, the source gives little reason to expect a significant difference.
Key takeaway
GSPO is especially useful for off-policy training with multiple gradient steps per generation.