7.5.5.4
Implementing 2-GRPO in TRL
Learn the TRL configuration for two generations per prompt and the associated batch-size requirement.
What is it?
The source implements 2-GRPO through TRL GRPOConfig and GRPOTrainer. The key setting is num_generations=2. It uses the standard GRPO loss and an epsilon value of 0.2.
Key points
TRL settings: num_generations=2; loss_type="grpo"; epsilon=0.2; per_device_train_batch_size=2 in the example. The source notes that batch size must be at least 2 multiplied by the number of prompts per step.
2-GRPO Configuration in TRL
Set num_generations=2 in GRPOConfig to generate two completions per prompt. The example retains the standard GRPO loss with epsilon=0.2 and sets per_device_train_batch_size=2. GRPOTrainer is then initialized with the model, reward function, configuration and dataset.
Real-world application
When implementing two-rollout training, check that the batch-size requirement is satisfied and verify settings against your installed TRL version.
Key takeaway
The defining implementation change is setting num_generations to 2.