1.15.4
N-gram Speculative Decoding
What is it?
N-gram speculative decoding uses repeated token patterns from the prompt and previously generated text to propose future tokens. It maintains an n-gram cache and searches for matching context. When a match is found, the corresponding continuation becomes a draft proposal. It requires no additional draft model or training.
Key points
1. N-grams are sequences of n tokens.
2. The system maintains an n-gram cache.
3. It matches the current context against cached patterns.
4. A matching continuation becomes a draft proposal.
5. The target model verifies the proposed tokens.
6. No additional model training is required.
7. It is particularly useful for repetitive code and structured outputs.
Using repeated patterns to predict continuations
Suppose a prompt contains a repeated code pattern. The system finds that the current context matches an earlier sequence in its n-gram cache. It proposes the continuation from that earlier occurrence, then asks the target model to verify the proposal.
Real-world application
N-gram speculation is useful for code generation, JSON or XML output, and prompts containing repeated sequences.
Key takeaway
N-gram speculation reuses existing text patterns to draft tokens without requiring another model.