7.5.12.3
Converting Ranks into Advantages
Understand how ordinal ranks are transformed into policy advantages using a monotonic function.
What is it?
After ranking responses, GOPO converts each rank into an advantage score. It uses a monotonic transformation of the normalised rank, allowing the ranking to guide the policy update without relying on raw reward magnitudes.
Key points
Normalise rank by group size N.|Apply a monotonic transformation f.|A linear mapping to the range [-1, 1] is one example.|Quantile normalisation is another example.|Use the resulting scores in the PPO-style clipped objective.
Turning ranks into policy advantages
For a group of N responses, GOPO calculates a score using f(rank divided by N). The function preserves the ordering: a higher-ranked response receives a higher advantage score. The transformed scores replace the raw magnitude-based advantages.
Real-world application
Use a monotonic rank transformation to construct the advantage signal for GOPO.
Key takeaway
GOPO uses transformed ordinal ranks as advantages in its policy update.