Speculative Decoding¶

Speculative decoding is a family of algorithms that lets a cheap drafter propose future tokens and asks the authoritative target model to verify several proposals together. It is not a smaller model replacing a larger one: no proposed token becomes output unless the target accepts it under the selected verification rule.

This page develops the common algorithm, correctness conditions, economics, and ecosystem taxonomy. The specialized MTP and DFlash page covers mlxcel's learned drafter implementations and operating controls.

Why Autoregressive Decoding Is Difficult to Accelerate¶

An autoregressive language model factorizes a sequence from left to right:

p(x_1:n) = product from t=1 to n of p(x_t | x_<t)

After prefill, ordinary decoding therefore runs one target forward pass, samples one token, appends it to the KV cache, and repeats. At batch size one, each step often moves a large fraction of the model's weights to produce only one useful token. The next target step cannot begin until the current token is known, so simply launching more identical target steps in parallel would violate the dependency.

Speculation moves the serial dependency to a much cheaper model. If a drafter can cheaply predict likely future tokens, the target can score all candidate positions in a wider verification pass. Modern accelerators usually process this short verification block more efficiently than the same number of separate target decode steps.

The Basic Round¶

Let the current accepted prefix be x, proposal width be K, drafter distribution be q, and target distribution be p.

1. The drafter predicts candidates y_1 ... y_K, normally in a short sequential chain or a structured block.
2. The target evaluates the candidate prefix in one verification pass and returns target logits for all relevant positions.
3. The verifier accepts the longest valid prefix of candidates.
4. At the first rejection it samples or selects a target-authorized replacement, discards invalid speculative cache state, and starts another round.
5. If all candidates are accepted, some algorithms also emit a target-predicted bonus token from the last verification position.

For greedy decoding, validity normally means that a candidate equals the target's argmax at that position. For sampling, correctness needs an acceptance-and-correction rule; comparing only top-1 tokens does not preserve the target distribution.

Key Metrics¶

Per-request acceptance on the response¶

mlxcel-server reports these counters for one request on that request's own response, so a client can tell whether the drafter helped without reading server logs. The block appears only when a drafter executed at least one verify round. On the native /completion and /completions routes the counters ride the existing timings object; /v1/chat/completions carries the whole timings object at the top level of the non-streaming body and of the streaming chunk that carries finish_reason.

"timings": {"prompt_n": 12, "predicted_n": 64, "draft_n": 72, "draft_n_accepted": 55, "draft_rounds": 9, "draft_kind": "dflash"}

draft_n and draft_n_accepted are llama-server's optional pair; draft_rounds and draft_kind (mtp, dflash, full-model, or prompt-lookup) are mlxcel additions; for prompt lookup the counters are the request's drafter totals, and the mean accepted length per round is draft_n_accepted / draft_rounds. Mid-stream chunks never carry the draft counters because they are totals for the run. The default-off batched burst (B > 1) and the disaggregated router report nothing.

Why It Can Be Faster¶

Suppose ordinary decoding spends t_T per target token. Producing K tokens costs about K * t_T. A speculative round instead costs approximately

t_round(K) = t_D(K) + t_V(K) + t_sync + t_rollback

and emits an expected E[A] useful tokens. The useful-token rate improves only when

E[A] / t_round(K) > 1 / t_T

This inequality explains the practical rules:

• The drafter must be much cheaper than the target.
• Its proposals must resemble the target closely enough to produce long accepted prefixes.
• Verification must exploit a favorable wider shape; K ordinary target decodes disguised as one API call do not help.
• Cache snapshot, rollback, host readback, synchronization, and scheduling costs must remain small.
• Larger K is not automatically faster. Acceptance usually falls with depth while verification and wasted draft work rise.

The target remains the expensive authority, but it is invoked fewer times per emitted token. Speculation does not reduce prompt prefill in the general case, so TTFT can stay unchanged or even rise while ITL improves.

Correctness Contracts¶

“Lossless” needs a precise definition.

Greedy Equivalence¶

At temperature 0, an implementation can aim for the exact token sequence of ordinary target decoding. That requires more than target verification in principle: block and single-token target forwards must agree, penalty state must be applied at the same positions, and cache rollback must restore the exact accepted prefix. Floating-point reduction order or a different quantized matrix kernel can change a near-tied argmax.

Distribution Preservation¶

For stochastic decoding, the classic speculative sampling construction accepts a draft token with probability related to p(y) / q(y). At the first rejection, it samples from a residual distribution proportional to [p - q]+. With compatible tokenization and the complete rejection rule, the final samples have the target distribution even though the drafter proposed them.

MTP and DFlash have different round loops, qualification gates, and penalty limitations; do not extend this distribution-preservation claim to every mode.

Task-Quality Equivalence¶

Some research systems trade exact output equivalence for speed and report downstream quality instead. That can be a valid service choice, but it is a different contract. Record it explicitly and do not describe it as exact target decoding.

Drafter Families¶

Community-published “speculative models” are therefore not interchangeable. A small causal LM may need only shared token IDs; an assistant can require a particular target size and hidden-state interface; a Medusa or EAGLE checkpoint needs a runtime that implements its tree or feature protocol; a DFlash checkpoint needs a diffusion draft loop. Check the model card, target revision, tokenizer, quantization, and runtime support rather than relying on the word draft in the repository name.

Full-Model Drafter in mlxcel¶

An ordinary smaller causal language model drafts for a larger target when both share a tokenizer. The drafter runs autoregressively from its own KV caches, which hold a prefix of the context that stops short of the current token, and proposes the argmax at each step; it reads nothing from the target. The target verifies each round's proposals in one token-only verify round, the round it shares with prompt lookup, and rejected proposals are trimmed from the drafter's caches alongside the target's. Only the target decides what is emitted.

mlxcel generate \
  --model ./models/Llama-3.1-70B-4bit \
  --draft-model ./models/Llama-3.2-1B \
  --draft-kind full-model \
  --prompt "Summarize the key ideas in the Rust ownership model." \
  --max-tokens 400 \
  --profile

--draft-kind full-model selects the path. When the kind is unset and a drafter path is given, any readable model directory that is neither MTP nor DFlash is detected as a full model. --draft-model takes a local path or an owner/name repo id on generate, run, serve, and mlxcel-server. --draft-block-size is the only width control and defaults to 4. The target and draft checkpoints must agree on tokenizer vocabulary and token IDs; a smaller same-family model is normally the safest candidate. --spec-type draft-simple, llama.cpp's name for ordinary draft-model speculation, maps onto this kind. --draft-block-size is the one width control for every drafter kind; mlxcel generate has no separate --num-draft-tokens flag. A --draft-model whose config.json is missing or malformed fails startup with the config error.

The drafter is the engine's token-only FullModelDrafter, verified by the same Engine::verify_round that prompt lookup uses, so the offline and server paths run one implementation and the library has no SpeculativeGenerator type. Startup refuses the pairing when the target's or the drafter's cache cannot be rolled back by a trim, when the drafter's vocabulary is larger than the target's or either config declares no vocab_size, and when a DFlash-family directory is passed as a full model.

On mlxcel serve and mlxcel-server the drafter runs the same per-request verify rounds as prompt lookup, under the same --prompt-lookup-max-batch gate (default 2): a wider active batch decodes plainly. It is also not served for --no-batch, for rows adopted through a disaggregated KV handoff, under a unified KV budget (--kv-unified), or for requests with images, audio, video, structured output, logprobs, or the extended sampler chain; those decode plainly.

This is a capability, not a measured speedup. On an idle M1 Ultra at batch size one, 256 output tokens, and temperature zero, Qwen3-8B 4-bit drafted by Qwen3-0.6B 4-bit runs 93.5 tok/s plainly against 56.1 tok/s drafted, and Llama 3.1 8B bf16 drafted by a 1B checkpoint runs 40.3 tok/s both ways. Baseline the target without --draft-model and measure the real pair before enabling it.

Prompt Lookup in mlxcel¶

Prompt lookup is n-gram speculation without a drafter checkpoint. The drafter matches the last few tokens of the sequence against earlier positions in the prompt and the generated text, proposes the tokens that followed that earlier occurrence, and the target verifies them in one forward. It pays off on replies that copy the prompt (edits, rewrites, reformatting, quoting); on free prose, proposals are shortened or paused so decoding stays near plain speed.

Acceptance is sampler-match: a proposed token is accepted only when it equals a fresh draw from the target's sampler at that position. For a one-hot proposal this rule is lossless and acceptance-optimal. Greedy output can still differ from plain decoding at logit near-ties, because a verify block and single-token forwards can reduce in a different order.

Prompt lookup needs a target whose state a cache trim can roll back. Recurrent and hybrid models, and families that own their caches internally (Qwen 3.5, Gemma 3 and Gemma 4, Llama 4), are refused.

Offline generate¶

mlxcel generate \
  --model ./models/Qwen3-1.7B-4bit \
  --prompt-lookup \
  --prompt "Fix the typos in this paragraph and return it otherwise unchanged: ..." \
  --max-tokens 400

The tuning flags require --prompt-lookup. mlxcel generate --draft-kind prompt-lookup is a parse error that points to --prompt-lookup. The CLI refuses prompt lookup for the interactive chat (no -p), multimodal prompts (--image, --audio, --video), pipeline parallelism, and --draft-model. Before the timed run it warms every verify width so the measured generation does not pay first-use kernel compilation. Measured on GB10, edit-style replies run 1.10x to 1.94x faster than plain decoding, and free prose runs at 0.98x to 1.00x with the gated policy.

Server --draft-kind prompt-lookup¶

mlxcel-server \
  --model ./models/Qwen3-1.7B-4bit \
  --draft-kind prompt-lookup \
  --port 8080

mlxcel serve and mlxcel-server accept --draft-kind prompt-lookup with no drafter checkpoint; a --draft-model alongside it fails startup. Decoding without the flag is unchanged. At prefill completion each eligible request gets its own drafter. On each decode tick a row that has a proposal runs one verify round on its own while the other rows take the regular batched step, and ticks without proposals stay on the lookahead pipeline.

The draft policy is gated on CUDA builds and graded elsewhere, and the startup speculative=prompt-lookup line logs max_draft, policy, max_batch, min_tokens_per_round, shadow_confirm_tokens, and plain_rounds_before_pipeline. Verify rounds stay off under a unified KV budget (--kv-unified), and on paged KV storage a sequence stops verifying once its total context passes 4608 tokens; dense storage has no such limit. On CUDA a break-even governor pauses a row after two drafted rounds once its moving average of emitted tokens per verify round falls below four, and seven consecutive proposed tokens that come true unverified resume it.

Requests with images, audio, or video, structured output, per-token logprobs, mirostat, adaptive-p, or the extended sampler chain (dynamic temperature, min_keep), and models whose state a cache trim cannot roll back, decode plainly with a debug log naming the reason. llama-server's own n-gram --spec-type values still fail startup; --draft-kind prompt-lookup is the mlxcel spelling. The first eligible prefill, normally the startup warmup request, warms every verify width from 2 to --draft-block-size + 1. A finished request reports its rounds in the timings block with draft_kind: "prompt-lookup".

Measured on GB10 with Qwen3-1.7B 4-bit, the server configuration changes chat throughput by -0.275%, short-copy workloads by +149% to +159%, and long-copy workloads by 1.52x to 2.82x. Continuous Batching covers how verify rounds share ticks with other rows.

Known Issue: Temperature 0 Block Verification¶

At temperature 0, a block verification path (--prompt-lookup or a full-model --draft-model) can leave the plain-decode token stream, so speculative output is not guaranteed to match plain decode (#2778). The root cause is not known. Where byte-identical output matters, compare a speculative run against the same prompt without a drafter, and decode plainly if they differ. The acceptance rule is sampler-match; the acceptance-optimal rejection rule and MLXCEL_SPECULATIVE_STOCHASTIC_ACCEPT are not available.

Tree Drafting¶

A chain has one candidate at each future depth. A tree spends some verification positions on alternatives: if the drafter's first and second choices are close, both can become branches. Tree attention must prevent one branch from seeing tokens on another branch, while positional encodings must reflect each node's depth.

mlxcel's MTP tree drafting is experimental and off by default. MLXCEL_MTP_TREE_BRANCHES controls how many runner-up candidates may consume verification positions; 0 is the linear behavior. MLXCEL_MTP_TREE_MARGIN can limit branching to a chosen top-two logit gap. Branching may increase the chance of finding an accepted path, but widens verification and weakens the straightforward chain-equivalence argument for the bonus token. Measure end-to-end gain before using it.

Choosing a Proposal Width¶

Start with the checkpoint or mode default. Then sweep a small set of widths using production-like prompts.

1. Record target-only latency and memory.
2. Record drafter and verification time separately.
3. Plot mean accepted prefix and useful tokens per verification, not acceptance percentage alone.
4. Stop increasing width when verification grows faster than useful accepted work.
5. Repeat for the actual batch and concurrency regime.

mlxcel defaults to four proposals for a full-model drafter and to seven proposed tokens (a verify block of eight) for prompt lookup. For MTP and DFlash drafters, a width the drafter checkpoint declares wins (for example the block_size of a qwen3_5_mtp checkpoint), then a measured per-device default (DFlash on GB10 with an affine-quantized target resolves to four), then four for MTP and sixteen for DFlash; see Draft Block Width. A round near the remaining output or cache limit can use a narrower effective width.

Pairing and Operational Checklist¶

• Verify that target and drafter model cards explicitly name a compatible family, size, and revision.
• Keep tokenizer files with the intended pair. Matching vocabulary size alone does not prove token-ID compatibility.
• Include both models in memory estimates and leave room for both caches and verification temporaries.
• Confirm startup logs say speculative=full-model, speculative=mtp, speculative=dflash, or speculative=prompt-lookup as intended.
• Test stop sequences, EOS, penalties, temperature, long prompts, and cache boundaries, not only a short greedy prompt.
• Treat text-only qualification as distinct from a multimodal checkpoint's ability to accept images. Specialized speculative tails have modality restrictions.
• Monitor fallback and decline logs. A healthy baseline response can conceal an acceleration path that never engaged.
• Rebenchmark after checkpoint, quantization, mlxcel, MLX, macOS, or hardware changes.

Common Failure Modes¶

Further Reading¶

• Leviathan, Kalman, and Matias, Fast Inference from Transformers via Speculative Decoding
• Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling
• Cai et al., Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
• Li et al., EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test