9.6.1
Why Multiple Reward Signals Are Needed
Understand why training may use several reward objectives rather than a single reward signal.
What is it?
A model may need to satisfy several objectives at once. Multi-objective reward training combines multiple reward signals, and the chosen combination strategy significantly affects the final policy.
Key points
Training can use multiple reward signals.
Examples in the source include correctness, response format and length.
The combination strategy affects the final policy.
The source describes five strategies: weighted sum, normalise then sum, lexicographic, constrained and Pareto optimisation.
Balancing Multiple Model Objectives
Consider a model that must provide correct answers, follow a required response format and avoid unnecessarily long responses. These are three separate objectives. A multi-objective reward strategy determines how the model balances them during training.
Real-world application
Identify the reward signals that represent the desired behaviour before selecting a method to combine them.
Key takeaway
Multiple reward signals require an explicit strategy for balancing competing objectives.