How does "draft-then-verify" double inference speed?
10 min read
LLM inference is autoregressive: it generates one token at a time, and each token requires loading all model weights from memory. There is no shortcut -- every single token pays the full cost of a forward pass.
Consider a 9B-parameter model running on hardware with 800 GB/s memory bandwidth. Loading 9 billion parameters (in FP16, ~18 GB) takes roughly ~18 ms per token -- just for the weight transfer alone. The actual compute finishes well before the data arrives.
Each token must load ALL weights. Watch how time accumulates:
What if a small, fast draft model (say 0.8B parameters) could guess several candidate tokens ahead of time? Then the large target model (say 35B parameters) could verify all those guesses at once in a single forward pass.
This is speculative decoding. The draft model proposes γ candidate tokens cheaply and quickly. The target model then checks them all in one shot. Tokens are accepted sequentially until the first mismatch -- at that point, the target model's own prediction replaces the wrong guess, and the cycle restarts.
The small model drafts tokens (orange), then the large model verifies (purple = accepted, red = rejected):
Mixture-of-Experts (MoE) models are deceptively efficient. A 35B total parameter MoE might only activate 3B parameters per token. Sounds fast, right? The problem: all 35B parameters must reside in memory, because you never know which experts will be needed next.
This means MoE inference speed is limited by memory bandwidth, not compute. The GPU or NPU spends most of its time waiting for weights to be loaded from memory, not doing math.
All 128 experts loaded
Only 2 experts active
Here is the core discovery from AtomGradient's research: even when the draft model's acceptance rate is less than 1%, speculative decoding still achieves 1.26-1.30x speedup on MoE models. How?
The key insight: when the target model verifies γ draft tokens, it loads the weights once for all γ+1 positions, instead of loading them γ+1 separate times. The bandwidth cost is amortized across the entire batch, regardless of how many tokens are accepted.
MoE Q8 model -- baseline vs. speculative decoding with varying draft lengths:
| Draft Length (γ) | Baseline (tok/s) | Speculative (tok/s) | Speedup | Acceptance |
|---|---|---|---|---|
| γ = 4 | 49.9 | 58.9 | 1.18x | 0.2% |
| γ = 8 | 49.9 | 60.9 | 1.22x | 0.2% |
| γ = 16 | 49.9 | 64.8 | 1.30x | 0.2% |
Source: Data from AtomGradient's 306 experiments
Longer drafts = more weight-loading amortization, even at near-zero acceptance:
The speedup from speculative decoding scales with the model's total parameter count, not the number of active parameters. This is because the bottleneck is memory bandwidth -- loading weights -- and bigger models have more weights to amortize over.
A smaller draft model also tends to be better. Using a 0.8B draft outperforms a 2B draft because the tiny model spends negligible time generating candidates, leaving more bandwidth budget for the main verification pass.
Each token loads all model weights. Memory bandwidth, not compute, is the bottleneck for inference speed.
A small draft model guesses tokens; the large model verifies them all in one forward pass.
Loading weights once for multiple positions amortizes the bandwidth cost, even at <1% acceptance.
The larger the model, the greater the speedup -- from 1.12x (4B) to 2.03x (9B Dense).
"Speculative decoding proves: even when 99% of guesses are wrong, the efficiency of batch verification persists."
Next, explore how personal AI changes everything
Next: Personal AI →