9.6.5
Pareto Optimisation and Multi-Reward Training in TRL
Understand Pareto trade-offs and configure multiple reward functions in TRL.
What is it?
Pareto optimisation maintains a Pareto front of policies and selects among them based on preference. The source also demonstrates multi-reward training in TRL using separate reward functions and configurable weights.
Key points
Pareto optimisation maintains a set of non-dominated policies.
The final policy is selected based on preference.
TRL example uses GRPOConfig and GRPOTrainer.
Reward functions include correctness_reward, format_reward and length_penalty_reward.
Example configuration uses multi_objective_aggregation="normalize_then_sum", reward_weights=[1.0, 0.3, 0.1] and num_generations=8.
Configuring Multi-Reward Training in TRL
A GRPO training setup can register three reward functions: correctness, format and length penalty. The configuration combines them using normalize_then_sum and assigns weights of 1.0, 0.3 and 0.1 respectively. Pareto optimisation offers another way to compare policies that make different trade-offs between objectives.
Real-world application
Use GRPOTrainer with multiple reward functions and configure the aggregation strategy and weights in GRPOConfig. Consider Pareto trade-offs when no single policy is best across every objective.
Key takeaway
Choose a reward combination strategy that reflects how the objectives should trade off.