OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Speculative Decoding Methods
Lesson 87 of 415 6 min read
1.15.2

Medusa: Multi-Head Speculative Decoding

What is it?

Medusa accelerates generation by adding multiple prediction heads to an existing language model. These heads share the original model backbone and predict tokens at different future positions. The predictions are generated in parallel and verified using a tree-structured approach. Medusa avoids loading a separate draft model.

Key points

1. Medusa adds extra prediction heads to the target model. 2. The original head predicts the next token. 3. Additional heads predict tokens at later positions. 4. Heads operate in parallel. 5. Tree-structured verification checks candidate sequences. 6. Training can focus on the additional heads while freezing the backbone. 7. Memory overhead is reported as less than 1% in the source.

Predicting several future tokens with Medusa

An ordinary language model predicts the next token. Medusa adds additional heads that predict the token at the next position, the position after that, and further future positions. A verification tree checks possible candidate sequences to determine which tokens can be accepted.

Real-world application

Medusa can improve inference speed when a separate draft model is undesirable and the model architecture supports additional prediction heads.
Key takeaway

Medusa uses parallel prediction heads attached to the target model instead of a separate draft model.