1.15.5
Methods Comparison
What is it?
Speculative decoding includes several methods with different draft sources and performance characteristics. Standard speculation uses a separate smaller model, Medusa uses additional prediction heads, EAGLE drafts from hidden states, and N-gram lookup reuses repeated text patterns. The best method depends on acceptance rate, computational overhead and serving requirements.
Key points
1. Standard speculative decoding uses a separate draft model.
2. Medusa uses parallel prediction heads.
3. EAGLE uses feature-level drafting.
4. EAGLE-2 uses context-aware dynamic draft trees.
5. N-gram lookup uses cached text patterns.
6. Lookahead uses parallel Jacobi decoding.
7. Multi-token prediction trains a model to predict multiple tokens per step.
8. Speedup depends on the model, workload and implementation.
Comparing speculative decoding approaches
A serving system can use a smaller draft model, Medusa heads, EAGLE features or N-gram lookup to propose tokens. The target model verifies those proposals. Each approach has different requirements for extra models, training, memory and inference support.
Real-world application
Compare methods using measured latency, throughput, acceptance rate, memory overhead and compatibility with the target inference engine.
Key takeaway
There is no universally best speculative decoding method; choose according to the workload and serving environment.