OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 308 of 415 5 min read
7.5.3.4

Implementing GSPO in TRL

Understand the TRL configuration for sequence-level importance sampling and off-policy batch reuse.

What is it?

The source demonstrates GSPO using TRL GRPOConfig and GRPOTrainer. Sequence-level importance sampling is enabled through importance_sampling_level="sequence". The example also reuses generated batches for multiple gradient steps.

Key points

TRL configuration: importance_sampling_level="sequence"; steps_per_generation=4; num_generations=8; epsilon=0.2. GRPOTrainer receives the model, reward function, configuration and training dataset.

GSPO Configuration in TRL

Set importance_sampling_level to "sequence" to activate sequence-level importance sampling. Setting steps_per_generation=4 reuses each generated batch for multiple gradient steps. The example uses eight generations and an epsilon value of 0.2.

Real-world application

Use the sequence-level importance sampling configuration when implementing GSPO in a compatible TRL training setup. Verify supported configuration options against the installed TRL version.
Key takeaway

In the source implementation, sequence-level importance sampling is selected through importance_sampling_level="sequence".