9.6.4
Lexicographic and Constrained Reward Optimisation
Explore priority-based optimisation and maximising a primary reward subject to secondary constraints.
What is it?
Lexicographic optimisation handles objectives in priority order, considering a lower-priority reward only when higher-priority rewards are tied. Constrained optimisation maximises a primary reward subject to constraints on secondary rewards.
Key points
Lexicographic: optimise rewards in priority order.
Lower-priority objectives are considered when higher-priority objectives are tied.
Constrained optimisation: maximise the primary reward while satisfying constraints on secondary rewards.
These approaches express priorities differently from a simple weighted sum.
Priority-Based vs Constrained Optimisation
In lexicographic optimisation, correctness can be the first priority and formatting the second. Formatting is considered only when correctness is tied. In constrained optimisation, the system maximises correctness while requiring the format score to remain above a specified threshold.
Real-world application
Consider lexicographic optimisation when objectives have a strict priority order. Consider constrained optimisation when secondary objectives must meet defined requirements.
Key takeaway
Lexicographic optimisation expresses strict priorities; constrained optimisation protects secondary requirements.