7.3.2
Configuring GRPO Training
Explore GRPOConfig settings including generation count, temperature, completion length, learning rate and batch configuration.
What is it?
GRPOConfig defines how responses are generated and how policy training proceeds. Important settings include the number of generations, sampling temperature, completion length, KL coefficient and learning rate.
Key points
output_dir = "./grpo_output"
num_generations = 8
temperature = 1.0
max_completion_length = 2048
beta = 0.04
learning_rate = 1e-6
per_device_train_batch_size = 2
gradient_accumulation_steps = 8
num_train_epochs = 2.
Configuring GRPO Training
The source configures eight generations per prompt, temperature 1.0 and a maximum completion length of 2048 tokens. It also sets beta to 0.04, learning rate to 1e-6, two prompts per device and eight gradient accumulation steps.
Real-world application
Use GRPOConfig to define generation and training settings. Adjust these values according to the task, model and available compute resources.
Key takeaway
GRPOConfig controls response generation, batch processing and policy training settings.