Fast Inference from Transformers via Speculative Decoding

Key Idea

Autoregressive LLM inference is slow because tokens are generated one at a time; each token requires a full forward pass through the model. Speculative decoding exploits the fact that many tokens are "easy" and predictable. A small, fast draft model generates several candidate tokens cheaply, then the large target model verifies all of them in a single parallel forward pass. Accepted tokens are free; rejected tokens trigger a fallback to the target model's distribution.

How It Works

1. Draft phase: A small model (e.g., a 125M parameter model) autoregressively generates K candidate tokens (typically K=4-8)
2. Verify phase: The large target model processes all K draft tokens in a single forward pass (parallel, not sequential), producing probabilities for each position
3. Accept/reject: For each draft token, compare the draft model's probability with the target model's probability:
   • If the target model agrees (high probability), accept the token
   • If the target model disagrees, reject with probability proportional to the gap, and resample from an adjusted distribution
4. Guarantee: The accept/reject scheme ensures the final output distribution is mathematically identical to running the target model alone, this is lossless, not approximate

Why It's Lossless

• The rejection sampling scheme is carefully designed so that accepted tokens follow exactly the target model's distribution
• When a token is rejected, the algorithm samples from a residual distribution (target minus draft, normalized), this corrects for the draft model's bias
• Net result: you get the exact same output as the target model, just faster

Why It Matters

• 2-3× speedup on T5-XXL with no quality loss, pure win
• No retraining needed: Works with any existing model pair (small draft + large target)
• Widely adopted: Used in production by Google, Meta, and most major LLM serving frameworks (vLLM, TensorRT-LLM, TGI)
• Spawned a family of variants: Medusa (multiple draft heads), EAGLE (feature-level drafting), staged speculation, and self-speculative decoding
• Changes the economics of serving: you can serve a large model at nearly the latency of a smaller one

Key Takeaways for Interviews

• Speculative decoding = small draft model generates candidates, large model verifies in one parallel forward pass
• Lossless: output distribution is mathematically identical to the target model, this is not an approximation
• Speedup depends on the acceptance rate, how often the draft model agrees with the target model. Higher agreement = more tokens accepted per step
• The draft model should be much faster than the target but share similar distribution (e.g., same model family, different size)
• In system design: speculative decoding reduces latency (time to generate), while batching/PagedAttention improve throughput (requests per second), they're complementary