7.5.3.4
Implementing GSPO in TRL
Understand the TRL configuration for sequence-level importance sampling and off-policy batch reuse.
What is it?
The source demonstrates GSPO using TRL GRPOConfig and GRPOTrainer. Sequence-level importance sampling is enabled through importance_sampling_level="sequence". The example also reuses generated batches for multiple gradient steps.
Key points
TRL configuration: importance_sampling_level="sequence"; steps_per_generation=4; num_generations=8; epsilon=0.2. GRPOTrainer receives the model, reward function, configuration and training dataset.
GSPO Configuration in TRL
Set importance_sampling_level to "sequence" to activate sequence-level importance sampling. Setting steps_per_generation=4 reuses each generated batch for multiple gradient steps. The example uses eight generations and an epsilon value of 0.2.
Real-world application
Use the sequence-level importance sampling configuration when implementing GSPO in a compatible TRL training setup. Verify supported configuration options against the installed TRL version.
Key takeaway
In the source implementation, sequence-level importance sampling is selected through importance_sampling_level="sequence".