7.3.5
Building and Running the GRPO Trainer
Understand how GRPOTrainer connects the model, configuration, reward functions, dataset and tokenizer before training.
What is it?
GRPOTrainer brings together the model, training configuration, reward functions, dataset and tokenizer. Once the trainer is created, calling train() starts the GRPO training process.
Key points
Create GRPOTrainer with the loaded model.
Pass the GRPOConfig through args.
Provide one or more reward functions through reward_funcs.
Pass the training dataset.
Provide the tokenizer.
Start training with trainer.train().
Creating and Running GRPOTrainer
The source constructs GRPOTrainer using model=model, args=grpo_config, reward_funcs=[reward_fn, format_reward_fn], train_dataset=math_dataset and tokenizer=tokenizer. It then calls trainer.train() to begin training.
Real-world application
Use the trainer structure as a starting point for a GRPO experiment. Ensure that the dataset, reward functions and model are prepared before starting the training run.
Key takeaway
GRPOTrainer connects the model, configuration, rewards and dataset to execute training.