1.15.6
Integration with vLLM
What is it?
vLLM supports speculative decoding configurations for different drafting methods. These configurations specify the target model, draft model or method, and the number of speculative tokens. The source demonstrates standard draft-model speculation, N-gram speculation and EAGLE-style feature-level drafting.
Key points
1. Standard speculation configures a separate draft model.
2. num_speculative_tokens controls the number of proposed tokens.
3. N-gram speculation uses the ngram method and prompt lookup settings.
4. EAGLE configuration specifies an EAGLE draft model and method.
5. Draft tensor parallelism can be configured separately.
6. Speculation is not beneficial for every workload.
7. High batch sizes, low acceptance rates and very short outputs can reduce its value.
Configuring speculative decoding in vLLM
A standard vLLM configuration can specify a large target model, a smaller draft model and a value for num_speculative_tokens. An N-gram configuration can instead specify the ngram method and prompt_lookup_max. An EAGLE configuration specifies an EAGLE-compatible draft model and method.
Real-world application
Use vLLM speculative decoding when serving latency-sensitive workloads. Benchmark the configuration against standard decoding before deploying it, especially for short outputs or high batch sizes.
Key takeaway
vLLM provides configurable speculative decoding methods, but the performance benefit must be measured for the actual workload.