7.5.11.5
Implementing GDPO in TRL
Learn the TRL configuration for normalise-then-sum aggregation, reward weights, and separate reward functions.
What is it?
The source demonstrates GDPO in TRL using normalize_then_sum aggregation, explicit reward weights, and separate reward functions for correctness and formatting.
Key points
Set multi_objective_aggregation to normalize_then_sum.|Set reward_weights explicitly.|Define separate reward functions.|Pass both functions to GRPOTrainer.|The example uses eight generations.
Configuring GDPO in TRL
from trl import GRPOConfig, GRPOTrainer
config = GRPOConfig(
multi_objective_aggregation="normalize_then_sum",
reward_weights=[1.0, 0.5],
num_generations=8,
)
def correctness_reward(completions, **kwargs):
return [1.0 if is_correct(c) else 0.0 for c in completions]
def format_reward(completions, **kwargs):
return [0.1 if has_good_format(c) else 0.0 for c in completions]
trainer = GRPOTrainer(
model=model,
reward_funcs=[correctness_reward, format_reward],
args=config,
train_dataset=dataset,
)
Real-world application
Use separate reward functions and explicit weights when configuring multi-objective GRPO training in a compatible TRL version.
Key takeaway
GDPO in TRL uses normalize-then-sum aggregation to combine separately defined reward functions.