7.5.7.1
The Silent vLLM Probability Mismatch
Understand why vLLM generation probabilities can differ from training forward-pass probabilities and break the on-policy assumption.
What is it?
When vLLM generates responses, its log-probabilities can differ from those calculated during the training forward pass. Differences in CUDA kernels, floating-point precision, and attention implementations can silently break the on-policy assumption and bias gradient estimates.
Key points
vLLM and training forward passes may calculate different probabilities.|Different CUDA kernels and precision can cause mismatch.|FlashAttention and PagedAttention are examples of different attention implementations.|Incorrect old-policy probabilities can bias gradient estimates.
Why two probability calculations can differ
A response is generated using vLLM and later evaluated during training. Because the two systems may use different kernels or precision, the probability assigned to the same response can differ. Importance ratios based on these mismatched probabilities may therefore be inaccurate.
Real-world application
Consider probability mismatch when using vLLM for generation in GRPO training.
Key takeaway
A vLLM probability mismatch can undermine the on-policy assumption even when generation appears to work normally.