1.15.2
Medusa: Multi-Head Speculative Decoding
What is it?
Medusa accelerates generation by adding multiple prediction heads to an existing language model. These heads share the original model backbone and predict tokens at different future positions. The predictions are generated in parallel and verified using a tree-structured approach. Medusa avoids loading a separate draft model.
Key points
1. Medusa adds extra prediction heads to the target model.
2. The original head predicts the next token.
3. Additional heads predict tokens at later positions.
4. Heads operate in parallel.
5. Tree-structured verification checks candidate sequences.
6. Training can focus on the additional heads while freezing the backbone.
7. Memory overhead is reported as less than 1% in the source.
Predicting several future tokens with Medusa
An ordinary language model predicts the next token. Medusa adds additional heads that predict the token at the next position, the position after that, and further future positions. A verification tree checks possible candidate sequences to determine which tokens can be accepted.
Real-world application
Medusa can improve inference speed when a separate draft model is undesirable and the model architecture supports additional prediction heads.
Key takeaway
Medusa uses parallel prediction heads attached to the target model instead of a separate draft model.