7.5.4.3
Reference-Model Token Weighting
Understand how the reference model probability is used to down-weight tokens that already have high probability.
What is it?
Dr. GRPO uses the reference model probability in its per-token gradient weight. Tokens already assigned high probability by the reference model are down-weighted, concentrating the gradient on tokens where the policy may need to change.
Key points
Weight from the source: wᵢ,ₜ = Âᵢ × (1 − πref(oᵢ,ₜ | q, oᵢ,
Reference Probability and Token Weighting
Suppose the reference model assigns a probability of 0.9 to a token. Its weighting factor is 1 − 0.9 = 0.1. If another token has reference probability 0.2, its factor is 1 − 0.2 = 0.8. The second token therefore receives a larger weighting factor, assuming the same advantage.
Real-world application
Use the reference model probability to understand which tokens are down-weighted by the Dr. GRPO weighting mechanism.
Key takeaway
The reference model helps distinguish familiar, high-probability tokens from tokens that may need meaningful policy updates.