7.5.4.2
The Need for Debiased Token Gradients
Learn why Dr. GRPO focuses gradient updates on tokens that contribute meaningfully to the reward signal.
What is it?
Dr. GRPO modifies per-token gradient weighting to account for a token’s marginal contribution to the reward signal. The goal is to concentrate gradient updates on tokens where the policy genuinely needs to change.
Key points
The method focuses on informative tokens. It reduces the influence of tokens that already have high probability under the reference model. This is described as token efficiency.
Focusing Updates on Informative Tokens
Consider two tokens in a generated response. One is already highly probable under the reference model and contributes little new task information. Another is less expected and more relevant to the reward signal. Dr. GRPO aims to give the latter more meaningful influence in the gradient.
Real-world application
Use token-level gradient analysis to investigate whether updates are focused on task-relevant behavior rather than common patterns.
Key takeaway
Dr. GRPO seeks to make gradient updates more informative at the token level.