7.5.4.1
Understanding Pretraining Bias in GRPO
Understand how the pretraining distribution can cause common tokens to receive large gradients even when they carry little task-relevant information.
What is it?
Standard GRPO normalizes advantages within a group, but the pretraining distribution can introduce systematic bias. Tokens that are common in pretraining data may receive large gradients even when they carry little task-relevant information. Dr. GRPO identifies this issue and aims to focus updates on informative tokens.
Key points
Pretraining distribution can influence gradient updates. Common tokens may receive large gradients without contributing useful task information. Dr. GRPO is designed to correct this bias and improve token efficiency.
Common Tokens and Training Bias
Imagine a model frequently generates common filler words because they are highly represented in its pretraining data. These tokens may receive large gradients even when they do not help solve the task. Dr. GRPO addresses this type of bias.
Real-world application
Consider pretraining bias when common filler tokens appear to dominate gradient updates during reinforcement learning.
Key takeaway
A token appearing frequently in pretraining data does not necessarily make it useful for the current task.