7.5.4.4
Implementing Dr. GRPO in TRL
Learn the Dr. GRPO configuration and why the reference model is required for token weighting.
What is it?
The source demonstrates Dr. GRPO in TRL using GRPOConfig and GRPOTrainer. It sets loss_type to "dr_grpo", uses eight generations and a KL coefficient of 0.04. A reference model is supplied because it is required for token weighting.
Key points
TRL configuration: loss_type="dr_grpo"; num_generations=8; beta=0.04. GRPOTrainer receives model, ref_model, reward_funcs, configuration and dataset. The reference model is required for token weighting.
Dr. GRPO Configuration in TRL
The source configuration sets loss_type="dr_grpo", num_generations=8 and beta=0.04. The trainer is initialized with both model and ref_model, along with the reward function and training dataset, and then trained using trainer.train().
Real-world application
When implementing Dr. GRPO, ensure the reference model is supplied and verify the configuration options supported by your installed TRL version.
Key takeaway
The reference model is an essential part of the Dr. GRPO token-weighting implementation shown in the source.