OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 327 of 415 6 min read
7.5.11.5

Implementing GDPO in TRL

Learn the TRL configuration for normalise-then-sum aggregation, reward weights, and separate reward functions.

What is it?

The source demonstrates GDPO in TRL using normalize_then_sum aggregation, explicit reward weights, and separate reward functions for correctness and formatting.

Key points

Set multi_objective_aggregation to normalize_then_sum.|Set reward_weights explicitly.|Define separate reward functions.|Pass both functions to GRPOTrainer.|The example uses eight generations.

Configuring GDPO in TRL

from trl import GRPOConfig, GRPOTrainer config = GRPOConfig( multi_objective_aggregation="normalize_then_sum", reward_weights=[1.0, 0.5], num_generations=8, ) def correctness_reward(completions, **kwargs): return [1.0 if is_correct(c) else 0.0 for c in completions] def format_reward(completions, **kwargs): return [0.1 if has_good_format(c) else 0.0 for c in completions] trainer = GRPOTrainer( model=model, reward_funcs=[correctness_reward, format_reward], args=config, train_dataset=dataset, )

Real-world application

Use separate reward functions and explicit weights when configuring multi-objective GRPO training in a compatible TRL version.
Key takeaway

GDPO in TRL uses normalize-then-sum aggregation to combine separately defined reward functions.