OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 298 of 415 6 min read
7.5.4.3

Reference-Model Token Weighting

Understand how the reference model probability is used to down-weight tokens that already have high probability.

What is it?

Dr. GRPO uses the reference model probability in its per-token gradient weight. Tokens already assigned high probability by the reference model are down-weighted, concentrating the gradient on tokens where the policy may need to change.

Key points

Weight from the source: wᵢ,ₜ = Âᵢ × (1 − πref(oᵢ,ₜ | q, oᵢ,

Reference Probability and Token Weighting

Suppose the reference model assigns a probability of 0.9 to a token. Its weighting factor is 1 − 0.9 = 0.1. If another token has reference probability 0.2, its factor is 1 − 0.2 = 0.8. The second token therefore receives a larger weighting factor, assuming the same advantage.

Real-world application

Use the reference model probability to understand which tokens are down-weighted by the Dr. GRPO weighting mechanism.
Key takeaway

The reference model helps distinguish familiar, high-probability tokens from tokens that may need meaningful policy updates.