OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 279 of 415 5 min read
7.5.7.1

The Silent vLLM Probability Mismatch

Understand why vLLM generation probabilities can differ from training forward-pass probabilities and break the on-policy assumption.

What is it?

When vLLM generates responses, its log-probabilities can differ from those calculated during the training forward pass. Differences in CUDA kernels, floating-point precision, and attention implementations can silently break the on-policy assumption and bias gradient estimates.

Key points

vLLM and training forward passes may calculate different probabilities.|Different CUDA kernels and precision can cause mismatch.|FlashAttention and PagedAttention are examples of different attention implementations.|Incorrect old-policy probabilities can bias gradient estimates.

Why two probability calculations can differ

A response is generated using vLLM and later evaluated during training. Because the two systems may use different kernels or precision, the probability assigned to the same response can differ. Importance ratios based on these mismatched probabilities may therefore be inaccurate.

Real-world application

Consider probability mismatch when using vLLM for generation in GRPO training.
Key takeaway

A vLLM probability mismatch can undermine the on-policy assumption even when generation appears to work normally.