OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Multi-Objective Rewards – Combination Strategies
Lesson 374 of 415 6 min read
9.6.5

Pareto Optimisation and Multi-Reward Training in TRL

Understand Pareto trade-offs and configure multiple reward functions in TRL.

What is it?

Pareto optimisation maintains a Pareto front of policies and selects among them based on preference. The source also demonstrates multi-reward training in TRL using separate reward functions and configurable weights.

Key points

Pareto optimisation maintains a set of non-dominated policies. The final policy is selected based on preference. TRL example uses GRPOConfig and GRPOTrainer. Reward functions include correctness_reward, format_reward and length_penalty_reward. Example configuration uses multi_objective_aggregation="normalize_then_sum", reward_weights=[1.0, 0.3, 0.1] and num_generations=8.

Configuring Multi-Reward Training in TRL

A GRPO training setup can register three reward functions: correctness, format and length penalty. The configuration combines them using normalize_then_sum and assigns weights of 1.0, 0.3 and 0.1 respectively. Pareto optimisation offers another way to compare policies that make different trade-offs between objectives.

Real-world application

Use GRPOTrainer with multiple reward functions and configure the aggregation strategy and weights in GRPOConfig. Consider Pareto trade-offs when no single policy is best across every objective.
Key takeaway

Choose a reward combination strategy that reflects how the objectives should trade off.