9.7.5
Practical Challenges in Listwise Reward Training
Understand annotation cost, ties, position bias, list length and annotator agreement.
What is it?
Listwise reward training has practical challenges involving annotation cost, tied responses, position bias, ranking length and disagreement between annotators.
Key points
Annotation cost: full rankings are expensive; top-3 rankings out of eight can reduce cost with minimal quality loss.
Ties: use a Plackett-Luce extension that assigns equal probability mass to tied items.
Position bias: randomise presentation order and train debiasing.
List length: K = 4–8 is typical; lists longer than 16 may add noise without much benefit.
Annotator consistency: use inter-annotator agreement κ > 0.6 as a quality filter.
The source also provides PyTorch code for calculating Plackett-Luce loss.
Preparing Reliable Ranking Data
An annotation task presents eight responses in random order. Annotators rank the responses, but full ranking may be costly. Collecting only the top three can reduce effort. Randomising display order helps limit position bias, while agreement checks help filter inconsistent labels.
Real-world application
Use partial rankings when appropriate, randomise response presentation, handle ties explicitly and filter low-agreement annotations. Keep typical training list sizes around four to eight responses.
Key takeaway
Good listwise training depends on reliable annotations, sensible list sizes and bias control.