Speculative decoding and MTP: speed for free

Local AI has a well-known bottleneck: autoregressive decoding. Every token in the response goes through a full model pass, and even on serious hardware that means dozens of tokens per second — enough to read, often not enough to work. There is, however, a technique that works around this limit for nearly free: speculative decoding. This article explains how it works, why it is lossless, what the MTP acronym in Qwen models means, and how to enable it today with llama.cpp, vLLM and SGLang.

How draft-verify works

The original idea comes from the paper Fast Inference from Transformers via Speculative Decoding by Leviathan, Kalman and Matias (arXiv:2211.17192): hard language tasks often contain easier subtasks that a small model can approximate well. The mechanism is a two-step loop:

1. Draft — a small model (or a lightweight head of the same model) proposes K consecutive tokens in rapid sequence.
2. Verify — the big model runs one forward pass over those K tokens and evaluates them in parallel. If the first j tokens are correct, they are accepted; the first rejected token is replaced by the big model's true answer.

The crucial part is the sampling: with the right method (speculative sampling), the final token distribution is identical to that of the big model used alone. No fine-tuning, no quality approximation: if a proposed token is accepted, it is exactly as if the big model had produced it.

Why does it pay off? The draft cost is negligible compared to the big model's forward pass, and that single pass verifies up to K tokens instead of one. If the small model is often right — and on real text it very often is — speed is multiplied by a factor close to the average accepted block length.

MTP: the multiple heads inside Qwen checkpoints

In the latest checkpoints of the Qwen family, such as Qwen3.8-27B, the model already carries a Multi-Token Prediction component: during training, the model learns to predict not only the next token but also the next two or three, through MTP heads trained with multiple steps. The official model documentation confirms it: "MTP (Multi-Token Prediction): trained with multiple steps" (Qwen3.8 model card).

In practice this means you do not need a second model: the MTP heads act as the drafter inside the same checkpoint. In llama.cpp, it all switches on with one flag, no extra file:

llama-server -m Qwen3.8-27B.Q4_K_M.gguf --spec-type draft-mtp

The --spec-type draft-mtp flag is documented in the official llama.cpp documentation as "Multi Token Prediction (MTP) heads from the main model". For models with a separate draft model, the same document lists the other types (draft, draft-eagle3, ngram-simple and more).

EAGLE: a draft that reads the model's thoughts

The EAGLE family — from the paper EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (arXiv:2401.15077) — moves drafting to a more informative level: instead of letting a separate model predict tokens from text alone, a small draft model reads the big model's hidden states and predicts the next tokens from there. EAGLE is lossless like classical speculative sampling: no fine-tuning of the target model and an unchanged output distribution.

Mistral adopted this route for its flagship: alongside Mistral Medium 3.5, a dense 128B model with a 256k context window, it ships a dedicated EAGLE draft head, mistralai/Mistral-Medium-3.5-128B-EAGLE (model card). The draft is a 2-layer body quantized to FP8, about 4 GB, aimed at low-concurrency serving where latency matters.

The recommended starting point is (num_steps, eagle_topk, num_draft_tokens) = (3, 1, 4), and it is worth remembering the principle: EAGLE shines at low concurrency; at heavy batch load, throughput is dominated by the target model's pass and the draft's contribution shrinks.

How much speed, really

The numbers depend on hardware, model and text type, but the orders of magnitude are stable:

• SGLang reports for Mistral Medium 3.5 on 4× H200 a single-stream of 69.07 tok/s without EAGLE and 97.10 tok/s with EAGLE, with an accept length of 1.72 tokens per step (SGLang cookbook): a ~40% streaming-latency gain for a single user.
• EAGLE-3 drafts on other models typically reach 3-4 accepted tokens per step, with speedups climbing toward 3×.
• On laptops and consumer GPUs, percentage gains tend to be higher: the draft is cheap in RAM and the parallelized verify amortizes the prefill cost well.

The practical rule: if you use local AI for long answers (writing, code, chat), speculative decoding pays for itself; if you only generate a few tokens at a time, prefill dominates and the gain thins out.

How to enable it: three concrete scenarios

1. Model with MTP heads (Qwen3.8, llama.cpp). The flag is enough, as shown above. No extra file, no download: the MTP heads travel inside the model's GGUF.

2. Official EAGLE draft (Mistral Medium 3.5, vLLM). Mistral's EAGLE checkpoint is served natively by vLLM:

vllm serve mistralai/Mistral-Medium-3.5-128B-EAGLE

with speculative token count = 3 and method eagle, as documented in the model card. Same day-zero support in SGLang, with dedicated docker tags.

3. High-throughput serving (SGLang). SGLang is an open-source serving engine (Apache-2.0 license) that made speculative decoding a first-class feature: MTP, EAGLE, draft models, ngram — all behind the same --speculative-algorithm. It is the natural choice when serving a model to multiple users, because it combines continuous batching, RadixAttention and speculative decoding in one stack.

When it pays off, and when it does not

Speculative decoding is not magic: the gain is proportional to accepted block length, which depends on how predictable the text is. Short or highly creative answers accept less; structured text, code and templated answers accept more. For everyday use on consumer hardware — the territory where most people run local AI on their own machines — it is one of the biggest zero-cost improvements available: you enable it with a flag, quality does not change, and the weight of perceived latency drops. The model stays yours, the tokens stay yours, and speed is no longer a compromise.

References

• Leviathan, Kalman, Matias — Fast Inference from Transformers via Speculative Decoding (arXiv:2211.17192)
• Li et al. — EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (arXiv:2401.15077)
• llama.cpp speculative decoding documentation
• Qwen3.8-27B model card
• Mistral-Medium-3.5-128B-EAGLE model card
• SGLang cookbook — Mistral Medium 3.5

Article generated with AI