7.5.4.5
When to Use Dr. GRPO
Identify situations involving vocabulary mismatch, common filler-token gradients and reference models close to the initial policy.
What is it?
Dr. GRPO is useful when there is a large vocabulary mismatch between pretraining and reinforcement learning, when common filler tokens dominate gradients, or when using a reference model close to the initial policy.
Key points
Use cases from the source: large vocabulary mismatch between pretraining and RL; common filler tokens dominating gradients; reference model close to the initial policy.
Choosing Dr. GRPO
Suppose an RL task uses specialized terminology that differs substantially from the model’s pretraining distribution. If common pretraining tokens dominate gradient updates, Dr. GRPO may help focus training on more task-relevant tokens.
Real-world application
Investigate vocabulary mismatch and gradient patterns before selecting Dr. GRPO. The source also notes that it pairs well with a reference model close to the initial policy.
Key takeaway
Dr. GRPO targets pretraining-related token bias, especially when common tokens dominate gradient updates.