7.5.12.5
When to Use GOPO or GRPO
Choose between GOPO and GRPO based on whether rewards are verifiable or derived from subjective learned reward models.
What is it?
Choose GRPO when rewards are exact and verifiable, such as mathematical correctness or code test results. Choose GOPO when rewards come from learned models evaluating subjective qualities such as helpfulness, style or safety.
Key points
Use GRPO for exact correctness and binary signals.|Use GOPO for subjective learned reward models.|GOPO can improve training and held-out reward curves on non-verifiable tasks.|The source reports improved LLM-judged win rates and faster convergence.|Benefits grow with noisier or poorly calibrated reward models.
Selecting the right optimisation method
For a mathematics task with an exact correct-or-incorrect reward, GRPO can use meaningful reward magnitudes. For open-ended response quality evaluated by a learned reward model, GOPO can use the relative ranking without trusting arbitrary score differences.
Real-world application
Match the optimisation method to the reliability and meaning of the reward signal.
Key takeaway
Use GRPO for verifiable rewards and GOPO for subjective, ranking-oriented reward signals.