OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Multi-Objective Rewards – Combination Strategies
Lesson 370 of 415 5 min read
9.6.1

Why Multiple Reward Signals Are Needed

Understand why training may use several reward objectives rather than a single reward signal.

What is it?

A model may need to satisfy several objectives at once. Multi-objective reward training combines multiple reward signals, and the chosen combination strategy significantly affects the final policy.

Key points

Training can use multiple reward signals. Examples in the source include correctness, response format and length. The combination strategy affects the final policy. The source describes five strategies: weighted sum, normalise then sum, lexicographic, constrained and Pareto optimisation.

Balancing Multiple Model Objectives

Consider a model that must provide correct answers, follow a required response format and avoid unnecessarily long responses. These are three separate objectives. A multi-objective reward strategy determines how the model balances them during training.

Real-world application

Identify the reward signals that represent the desired behaviour before selecting a method to combine them.
Key takeaway

Multiple reward signals require an explicit strategy for balancing competing objectives.