Large language models (LLMs) like GPT are powerful, but they’re notoriously slow at inference time. Every token (word or piece of text) is generated sequentially, which creates a bottleneck in performance. Speculative decoding is a recent breakthrough that accelerates this process, delivering up to 3x faster inference without compromising text quality. Here’s how it works.
The Core Idea
Speculative decoding splits the work between two models:
- Draft model (speculator): A smaller, faster model that makes quick guesses for the next several tokens.
- Target model: The large, accurate model that ultimately determines the output.
Instead of having the big model generate tokens one by one, the draft model proposes multiple tokens in advance, and the target model verifies them efficiently in parallel.
Step-by-Step Workflow
1. Drafting
- The draft model looks at the current context (the tokens generated so far) and predicts the next K tokens quickly.
- Example: If the sequence so far is x₁ … xₜ, the draft outputs K candidate tokens.
- Drafting strategies include:
- Greedy decoding: Always pick the most likely token.
- Sampling: Randomly pick based on probability.
- Beam search: Explore multiple high-probability options.
2. Verification
- The target model evaluates the batch of draft tokens in a single forward pass.
- It accepts the longest prefix of tokens that align with its own probability distribution.
- If tokens diverge, the unmatched ones are discarded. The target model then generates one safe token itself to keep the sequence consistent.
This process is known as rejection sampling, ensuring the output is statistically identical to what the target model would have produced on its own.
3. Continuation
- Accepted tokens are appended to the output.
- The draft model proposes the next batch based on this updated context.
- The cycle repeats until the sequence is complete.
Why It Works
- Efficiency: The draft model makes fast guesses, while the target model only confirms them in parallel instead of working sequentially.
- Lossless guarantee: The output distribution remains the same as if the target model alone had generated the text.
- Speed: Because the target model verifies multiple tokens at once, inference is significantly faster, especially when the acceptance rate (the percentage of draft tokens verified) is high.
Key Concepts
- Acceptance rate: Higher acceptance means fewer rejections and greater speedups.
- Parallelization: Multiple draft tokens are verified at once, taking advantage of GPU acceleration.
- Autoregressive models: LLMs generate tokens one after another; speculative decoding cleverly bypasses this bottleneck.
Practical Impact
- Real-world deployments have shown 2–3x improvements in tokens per second.
- Works best when the draft model is a smaller version of the target (or a distilled/quantized variant).
- Adjustable parameters like speculation length (K) allow fine-tuning the balance between speed and accuracy.
Conclusion
Speculative decoding is a powerful optimization for LLM serving. By introducing a fast draft model to speculate future tokens and letting the main model verify them in parallel, we can dramatically reduce latency without altering the quality of the generated text. As LLMs continue to scale, speculative decoding is set to become a standard in efficient AI deployment.