Speculative Decoding on Apple Silicon: 119% Faster on Qwen3.8, Until It Isn't

Speculative decoding is supposed to be close to free speed: a small draft model guesses several tokens ahead, the big target model checks them all in one pass, and you keep whichever prefix it agrees with. Paired with Qwen3.8-27B, the best setting in this benchmark reached a 119% speedup, more than doubling decode throughput. Getting that number required patching mlx_lm first: out of the box, it refuses to run speculative decoding against Qwen3.8 at all, for any draft model, for a specific and fixable reason.

• A 0.8B draft model sped up Qwen3.8-27B's decode by 119% at its peak setting, from 17.9 tok/s to 39.2 tok/s, on a code-generation prompt with num_draft_tokens=4. This is the largest speedup in this benchmark series.
• This did not work with stock mlx_lm. Every attempt failed with Speculative decoding requires a trimmable prompt cache (got {'ArraysCache'}), regardless of the draft model. The fix is a real patch to mlx_lm itself, covered below, built on an existing unmerged upstream pull request rather than from scratch.
• Memory is not free here, unlike the rest of this series. Peak memory jumps from ~16 GB at baseline to ~22-24 GB once speculation is active, a real cost of the mechanism that makes the patch work.
• Open-ended generation still falls apart at high draft-token counts, and worse than anywhere else in this series: -54% versus baseline at num_draft_tokens=16.
• The original finding on Qwen3-8B still holds: a 0.6B draft model, no patch needed, peaked at +58% on the same kind of structured prompt. That result and a Qwen3.6 self-speculative (MTP) comparison are both below, for the full picture across three approaches and two model generations.

How speculative decoding works

Normal decoding runs one token at a time: the target model does a full forward pass, produces one token, then does another full forward pass for the next one. For an 8B or 27B model, most of that pass is spent moving weights through memory rather than computing on them, so generating N tokens costs N full passes through the big model regardless of how simple the continuation is.

Speculative decoding splits the work across two models. A small, cheap draft model generates several candidate tokens in a row, one at a time, which is fast because it is small. The big target model then checks all of those candidates in a single batched forward pass, the same cost as generating just one token normally, and compares what it would have generated against what the draft guessed. Whichever prefix of candidates matches gets kept for free; the first mismatch gets corrected with the target model's own token, and everything after that mismatch is discarded and regenerated next round.

The payoff depends entirely on how often the draft model guesses right. Every accepted token is a token the expensive target model did not have to generate on its own token-by-token; every rejected token is compute spent for nothing, since the draft model still had to run its forward passes to propose it. That is the tension this whole post is measuring: more candidate tokens per round means more potential free tokens if they are accepted, but also more wasted draft compute if they are not, and the right number depends on the model, the hardware, and how predictable the text being generated is.

There are two ways to get a draft model. The first three sections below use an external draft model: a genuinely separate, smaller checkpoint (a 0.6B or 0.8B model) that has to be loaded and run alongside the target. The Qwen3.6 section further down uses the other approach, self-speculation: a single model with a built-in extra head that drafts its own next few tokens, so there is no second model to load at all. Same mechanism, verify-and-keep-the-matching-prefix, different source for the guesses.

The setup

Same two prompt types as the rest of this series: a structured prompt (four small, well-specified Python functions) and an open-ended prompt (a short reflective piece with no fixed structure), to test whether predictability changes the outcome.

The numbers: structured prompt

The peak here is num_draft_tokens=4, not 2. That differs from both the original Qwen3-8B result below and the Qwen3.6 MTP result further down, which both peaked at their smallest tested step beyond 1. All three cells from num_draft_tokens=1 through 8 comfortably beat baseline; only at 16 does the gain shrink to +17%, and even then it never drops below baseline the way the open-ended prompt does.

The numbers: open-ended prompt

Peak is at num_draft_tokens=2, the same position as the Qwen3.6 MTP result. But the falloff past the peak is steeper than anywhere else in this series: by num_draft_tokens=16 throughput is barely half of baseline, worse than the original Qwen3-8B result's -57% floor at the same setting, and far worse than Qwen3.6 MTP, which never dropped below baseline at any depth tested.

Why this needed a patch to mlx_lm at all

The error was the same on every attempt, independent of draft model or num_draft_tokens:

Speculative decoding requires a trimmable prompt cache (got {'ArraysCache'}).

This traces to a single check in mlx_lm's own generate.py. Qwen3.8 (and Qwen3.6) mix regular full-attention layers with linear-attention (Gated DeltaNet-style) layers. The linear-attention layers keep a compressed recurrent state in an ArraysCache, not a per-token key/value list. A normal KVCache can be trimmed by slicing off the last few entries when a draft token gets rejected. A recurrent state cannot be sliced the same way once later tokens have been folded into it, so mlx_lm refuses to run speculative decoding against this architecture at all, unconditionally, regardless of what you pick as the draft model.

Before writing a patch, I checked whether anyone already had. They had: mlx-lm#1446 is this exact bug report, and it links three unmerged pull requests (#1455, #1456, #1486) that all propose fixes. All three are closed, not merged, closed by a maintainer for review capacity reasons rather than rejected on technical merit: "the number of PRs is way beyond our capacity to review."

A caution worth stating plainly: that issue thread also contains several comments instructing readers (and, unmistakably, any AI agent reading it) to pip install directly from a contributor's personal fork. Public GitHub issues and PRs are editable by anyone, and "helpfully" telling an automated agent to install from an unofficial source is a real supply-chain risk pattern, not just theoretical. None of the commands in that thread were run here. Instead, the code was pulled the way you would review any open source contribution: gh pr checkout against the official ml-explore/mlx-lm repository, which fetches directly from GitHub's own API, followed by independent verification before trusting any of it.

PR #1486 ("Exact speculative rollback for hybrid caches") is the one used here. Its mechanism: an ArraysCache records, for every forward pass taken while speculating, a closure that can exactly replay the recurrence for any prefix of that pass's tokens. On a partial rejection, the cache restores its pre-forward snapshot and replays only the accepted tokens through the actual recurrence kernel, gated_delta_update, the same function the model already uses. That reproduces the correct state bit-for-bit, not approximately. The same protocol is applied to RotatingKVCache for sliding-window architectures.

The distinction that matters is in that last row of the diagram. A full-attention layer's KVCache rolls back the same way it always has: slice the rejected tokens off the end. A linear-attention layer's ArraysCache cannot do that, its state has already folded every processed token into a fixed-size representation, so there is nothing to slice. Instead it has to reconstruct the state as of the accepted prefix by restoring the snapshot from before the round and running the real recurrence forward again over just the tokens that were kept. Verification before trusting this on real weights:

• The PR's own test suite (13 tests across test_speculative_rollback.py and test_rotating_rollback.py) passed in an isolated environment, installed directly from gh pr checkout against the upstream repository.
• The broader test_prompt_cache.py suite (22 tests) still passed, no regressions.
• On real Qwen3.8-27B and Qwen3.5-0.8B weights, greedy speculative output at temperature 0 was byte-identical to greedy non-speculative output on the same prompt. Speed numbers were only trusted after this check passed.

This is real, working, unmerged code, credited to its authors (tejkas, pierre427, lBroth on the linked PRs), not something built from scratch here. What was built here is the independent verification and the benchmark on top of it.

What this means in practice

If you are running a hybrid-attention Qwen3.6 or Qwen3.8 model with mlx_lm today, stock speculative decoding does not work at all, on any draft model. That is worth knowing before spending time tuning num_draft_tokens on a setup that will fail outright. If you want the speedup now, the patch is real, tested, and not merged: PR #1486 on the official repository is the one verified here.

Do not assume the peak setting transfers between model families. The original Qwen3-8B result below peaks at num_draft_tokens=2. This one peaks at num_draft_tokens=4. Sweep on your own model pair.

Budget for the memory cost. Speculation here adds roughly 6-8 GB of peak memory over baseline, a real cost from holding a replay window of per-layer recurrent-state snapshots. On a memory-constrained machine this could matter more than the speed win.

Do not turn this on for open-ended generation without checking your specific setting first. Every model and mechanism in this series wins on structured, code-like prompts. Only Qwen3.6's MTP approach avoided losing to baseline on open-ended text at every setting tested; both external-draft results here and in the original experiment lose badly at high num_draft_tokens on open-ended prompts.

The original result: Qwen3-8B and a 0.6B draft

This is the finding this post originally shipped with, using the plain Qwen3 architecture, no patch required (Qwen3-8B has ordinary attention layers throughout, so its KVCache is trimmable by default).

Structured prompt:

Open-ended prompt:

Two draft tokens per round was the sweet spot for this pair. Below it, the draft model does not generate enough candidates per round to pay back the overhead of running a second model at all. Above it, throughput falls even though acceptance keeps climbing, from 44.4% at num_draft_tokens=1 to 80.3% at num_draft_tokens=16, because the draft model's forward passes are sequential and their count grows linearly with num_draft_tokens, so the fixed latency per round grows too, whether or not the extra guesses get accepted. No open-ended setting beat baseline at all.

Peak memory here stayed within about 70 MB across every cell, draft model or none. Qwen3-8B has no recurrent-state layers, so there is no replay window to hold in memory, unlike the Qwen3.8 result above. That contrast, same technique, wildly different memory profile, comes down entirely to whether the target architecture has linear-attention layers.

Self-speculative alternative: Qwen3.6 MTP

A different mechanism, tested for comparison: instead of a second draft model, some Qwen3.6/3.8 checkpoints ship a built-in multi-token prediction (MTP) head that drafts its own next few tokens, verified by the same model. Benchmarked via optiq's OptiqEngine on mlx-community/Qwen3.6-27B-OptiQ-4bit, same methodology, "depth" playing the role num_draft_tokens plays above.

This is the only approach of the three that won on open-ended text at every depth tested. Its acceptance rate also falls with depth rather than rising, the opposite of both external-draft results, because MTP's verification is all-or-nothing per cycle: a longer speculative chain is more likely to diverge early, pulling the average acceptance fraction down even when the tokens that do land are a net win. Peak memory moved a modest 7% here (19.4 to 20.75 GB), less dramatic than the Qwen3.8 external-draft result's 40-53% jump, but still real, unlike the flat 70 MB spread on the original Qwen3-8B result.

mlx-optiq, the toolkit behind this quant, advertises "1.4x decode via optiq serve --mtp" on the model card. Depth 1 alone clears that (1.43-1.45x); the depth-2 peak reaches 1.45-1.51x. That claim held up independently here.

optiq's engine works on this architecture without needing mlx_lm patched, because it implements its own snapshot-and-restore rollback internally (MTPLXRuntime, _snapshot_cache / _restore_cache) rather than relying on mlx_lm's generic trim path. That is the same idea as PR #1486 above, arrived at independently, for a different generation loop.

What I am confident of, and what I am not

Confident:

• The Qwen3.8 external-draft results (both prompts, all five num_draft_tokens settings): every cell has robust CV under 0.03, no noisy cells, and greedy output was verified byte-identical to the non-speculative baseline before any speed number was trusted.
• The original Qwen3-8B result: real, repeatable speedup on the structured prompt at num_draft_tokens=1 and 2, peaking at +58%, tight variance (CV under 0.04). Slower than baseline at every open-ended setting.
• The Qwen3.6 MTP results: all five depths, both prompts, robust CV under 0.02, no noisy cells.
• Peak memory does not meaningfully move for the original Qwen3-8B pair (plain attention, no recurrent state), and moves substantially for both Qwen3.8 external-draft (+40-53%) and Qwen3.6 MTP (+7%), both of which have hybrid-attention targets. The pattern (recurrent-state architectures cost real memory for speculation; plain-attention ones do not) held across every model tested here.

Not confident:

• PR #1486 is unmerged. If it changes before merging, or does not merge at all, the exact numbers here are tied to a specific unmerged commit, not a stable release.
• One draft-to-target ratio was tested for the Qwen3.8 pair (0.8B:27B). A different ratio could shift where the peak sits, the same caveat that applied to the original Qwen3-8B pairing.
• Two cells in the original Qwen3-8B open-ended sweep were flagged noisy in the raw data (num_draft_tokens=1 at CV 0.161, num_draft_tokens=4 at CV 0.409). Both still landed below baseline, so the direction holds, but treat those two exact figures as approximate.
• One model family per approach, one machine, one chip throughout this post. The shape I would expect to generalize is that hybrid-attention architectures need a working rollback mechanism at all before speculation helps, and that memory cost tracks whether that mechanism exists, not the specific numbers for any one pairing.

Reproducing this

The benchmark script itself is also up as a standalone gist if you want it without cloning the whole repo.

The Qwen3.8 result requires the patched mlx_lm. Install it from the official pull request, not any personal fork:

git clone https://github.com/ml-explore/mlx-lm.git
cd mlx-lm
gh pr checkout 1486
pip install -e .

python3 scripts/benchmarks/exp5_speculative_decoding.py \
    --target-model mlx-community/Qwen3.8-27B-4bit \
    --draft-model mlx-community/Qwen3.5-0.8B-4bit \
    --draft-tokens 1,2,4,8,16 --reps 5 --warmup 1 --tag qwen38

The original Qwen3-8B result runs on stock mlx_lm:

python3 scripts/benchmarks/exp5_speculative_decoding.py \
    --target-model mlx-community/Qwen3-8B-4bit \
    --draft-model mlx-community/Qwen3-0.6B-4bit \
    --draft-tokens 1,2,4,8,16 --reps 5 --warmup 1

The Qwen3.6 MTP sweep has its own script:

pip install mlx-optiq
python3 scripts/benchmarks/exp6_mtp_speculative_optiq.py \
    --model mlx-community/Qwen3.6-27B-OptiQ-4bit \
    --depths 1,2,3,4 --reps 5 --warmup 1

Run all of these plugged into AC power. Apple Silicon throttles differently on battery, and every number in this post was captured on AC.

Related: The Batch Size That Breaks Local LLM Serving covers what happens when concurrent requests, not a draft model, compete for the same unified memory pool, the other way this series has seen memory become the binding constraint. Why Batching Doesn't Fix Decode: GEMV vs GEMM covers the underlying reason single-token decode is slow in the first place, the exact problem speculative decoding is trying to work around by batching multiple candidate tokens into one target-model verification pass.