7.5.12.2
Ranking Responses Within a Group
Learn how GOPO ranks generated responses from worst to best using their reward scores.
What is it?
GOPO ranks all responses in a generated group according to their reward. The lowest-ranked response receives rank 1, while the highest-ranked response receives rank N, where N is the group size.
Key points
Start with a group of N responses.|Use reward scores to order the responses.|Rank 1 represents the worst response.|Rank N represents the best response.|The algorithm uses ranking rather than original score magnitudes.
Ranking a group of responses
Suppose four responses receive reward scores of 0.2, 0.9, 0.4 and 0.7. Their ordinal ranks are respectively 1, 4, 2 and 3. GOPO uses these ranks as the basis for its advantage calculation.
Real-world application
Use group rankings when relative response quality is more reliable than the reward model’s numeric calibration.
Key takeaway
GOPO converts reward scores into an ordering of responses within each group.