OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Multi-Objective Rewards – Combination Strategies
Lesson 373 of 415 5 min read
9.6.4

Lexicographic and Constrained Reward Optimisation

Explore priority-based optimisation and maximising a primary reward subject to secondary constraints.

What is it?

Lexicographic optimisation handles objectives in priority order, considering a lower-priority reward only when higher-priority rewards are tied. Constrained optimisation maximises a primary reward subject to constraints on secondary rewards.

Key points

Lexicographic: optimise rewards in priority order. Lower-priority objectives are considered when higher-priority objectives are tied. Constrained optimisation: maximise the primary reward while satisfying constraints on secondary rewards. These approaches express priorities differently from a simple weighted sum.

Priority-Based vs Constrained Optimisation

In lexicographic optimisation, correctness can be the first priority and formatting the second. Formatting is considered only when correctness is tied. In constrained optimisation, the system maximises correctness while requiring the format score to remain above a specified threshold.

Real-world application

Consider lexicographic optimisation when objectives have a strict priority order. Consider constrained optimisation when secondary objectives must meet defined requirements.
Key takeaway

Lexicographic optimisation expresses strict priorities; constrained optimisation protects secondary requirements.