9.7.1
Why Listwise Preference Learning Goes Beyond Pairwise Comparisons
Understand why complete rankings provide richer preference information than independent pairwise comparisons.
What is it?
Listwise preference learning trains on complete rankings of multiple responses rather than separate pairs. It provides richer preference information, captures relative ranking margins and can improve reward calibration.
Key points
Bradley-Terry models handle pairwise preferences.
Listwise models learn from complete response orderings.
A ranking of K responses contains K(K-1)/2 implicit pairwise comparisons.
Listwise rankings also capture relative margins between positions.
They align naturally with GRPO, which generates groups of responses.
Ranking five responses can be more efficient than independently labelling all ten pairs.
Ranking Five Model Responses
Suppose five responses answer the same question. Pairwise annotation would require comparing each possible pair. Listwise annotation asks the evaluator to rank all five responses together. This provides an overall ordering and relative information about their quality.
Real-world application
Use listwise preference data when several responses can be evaluated together and their relative ranking is important.
Key takeaway
Listwise learning captures the overall ordering of responses rather than isolated pairwise preferences.