One llama.cpp flag unlocks +33% to +145% decode speed for Qwen3.8-27B on consumer GPUs, depending on the card, and the knobs the community mapped push further still. No new files, no conversion, no custom build. The MTP head already ships inside the GGUF you downloaded on launch night.
Opened hours after the Aug 14 2026 release. Within four days the community grew it into a living record: 53 configurations, 40 contributors, a decade of silicon from 2016 Pascal to Blackwell workstation cards plus Apple Silicon and an unlocked mining card, three quant makers, and seven tuning rules nobody knew at launch, see Community numbers.
--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1
llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
-c 131072 -ngl 999 -fa 1 \
--cache-type-k q4_0 --cache-type-v q4_0 \
--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1The KV cache flags matter on their own: without them, context creation fails past roughly 90K next to 17GB of weights. With them, the full 262K window fits a 24GB card at 22.2GB (drop -c to 262144 and remove the spec flags if you want maximum window instead of maximum speed).
Weights: unsloth/Qwen3.8-27B-GGUF. Official model: Qwen/Qwen3.8-27B.
Start with the launch command above, --spec-draft-n-max 2. Then sweep n-max 2-4 for your card, and on bandwidth-poor rigs try --spec-draft-p-min 0.60-0.75, per rules 1 and 2 below. Every contributor's full sweep lives in sweeps/.
Discovered by contributors in the table across the first three days, detailed in the sections and footnotes below:
- The n-max sweet spot is card- and topology-dependent. 24GB cards peak at n-max 2, bigger or faster cards at 3-4, and switching split mode moves it too, so re-sweep after any config change (@lingster, @Jackwwg83).
--spec-draft-p-minhelps starved cards and hurts fast ones. The ~0.60 confidence gate makes deep drafting nearly free on bandwidth-poor rigs (@tomertec), and inverts on desktop Blackwell, where three independent RTX 5090s ran fastest ungated. Acceptance is a vanity metric there: gating raised it and lowered throughput (@taco-devs, @paulomcg, @jcr211). Sweep it, don't adopt it.- The gain scales with generation length where overhead dominates. Short generations can pay more than they win (@Spadav); on rigs whose baseline is already bandwidth-bound the full gain shows at 400 tokens and length adds nothing (@Jackwwg83).
- On multi-GPU boxes, fix the split mode before touching spec flags. The default
--split-mode layerserializes decode;tensorwas +68% on its own on a 5060 Ti pair, and the two levers stack to 3.14x (@Jackwwg83). - Speculative decode is a single-stream optimization. The advantage is gone by
--parallel 4, and a--parallel 2BASELINE reads ~20% low, which inflates your gain claim, so measure both arms at--parallel 1(@Jackwwg83, @paulomcg). - Rebuild llama.cpp before you tune anything, and re-derive whole flag stacks per card class. A config that pays on a bandwidth-starved APU inverted below baseline on a 960 GB/s card (@Splizard). Upstream is optimizing this arch weekly: a current build was +10-15% on every quant before any flag (@taco-devs), and a fresh 3090 baseline now equals the day-one with-flag number (@hauntedhost).
- A shared desktop halves everything, silently. A live compositor and browser spilled 3.5 GB of weights to host RAM over PCIe while
/healthstayed green; decode halved with no error anywhere. Bench headless, or verify the weights are actually resident (mem_info_gtt_usedor your vendor's equivalent) before trusting a number (@Splizard).
Qwen trained multi-token-prediction (nextn) layers into Qwen3.8. The quantizers kept them: unsloth's GGUFs carry the blk.*.nextn.* tensors, which llama.cpp loads and, without the flag, ignores. llama.cpp added draft-mtp speculative decoding in PR #22673 (July 2026): the server drafts tokens with the built-in head and verifies them with the main model, so accepted drafts cost a fraction of a full forward pass. Everything was in place on release night. The flag connects it.
--parallel 1for measurement, always: some hosts do serve with higher parallel, but the spec advantage is gone by 4 concurrent streams, and a parallel-2 baseline corrupts the A/B (see rule 5)- prompt processing takes a small hit from device-to-host embedding transfers
- these are day-one llama.cpp speeds through the qwen3_5 code path, the hybrid attention kernels are young and the floor should rise with upstream work
- your absolute numbers will differ with hardware, drivers, and thermals, the deltas are the durable part
probe.py is the streaming client behind every number here. It clocks every generated token (reasoning and content deltas both) against a live server and prints per-prompt medians.
python3 probe.py # defaults to http://127.0.0.1:8080
python3 probe.py http://127.0.0.1:8090Run it once against a baseline serve and once with the flag, same everything otherwise. That pairing is the whole method.
Ran the A/B on your card? Open a PR and add a row.
| Card | Baseline | With flag | n-max | Acceptance | Contributor |
|---|---|---|---|---|---|
| RTX 3090 24GB | 31.0 | 41.3 | 2 | 0.78 | @sudoingX |
| RTX 5090 mobile 24GB | 36.7 | 50.9 | 2 | 0.79 | @sudoingX |
| RTX 4090 24GB | 47.7 | 76.3 | 2 | 0.56 | @Spadav_ |
| RTX A6000 48GB (Ada) | 26.7 | 52.5 | 2 | 0.54-0.98 | @lingster |
| RX 7900 XTX 24GB | 30.7 | 43.9 | 2 | 0.60-0.95 | @Jqianggu |
| 2× RTX 3090 + 3090 Ti 24GB (TP) | 49.1 | 81.1 | 2 | 0.52-0.96 | @guilhermedemelocabral |
| RTX 4090 24GB (UD-Q4_K_XL) | 36.1 | 74.8 | 2 | 0.56-0.94 | @rkvhtd |
| 2x RX 9070 16GB (Vulkan) | 22.1 | 41.6 | 2 | 0.73 | @tomertec |
| AMD Radeon AI PRO R9700 32GB | 27.0 | 43.3 | 2 | 0.60-0.94 | @ajnytebot |
| Ryzen AI Max+ 395 / Radeon 8060S | 11.5 | 23.7 | 2 | 0.52-0.94 | @shiwuxiu |
| AMD Radeon 890M iGPU (Strix Point) 48GB UMA | 2.7 | 5.7 | 2 | 0.59-0.91 | @davidglogan |
| RTX PRO 6000 Blackwell 96GB | 63.8 | 91.5 | 2 | 0.72-0.81 | @commdata2338 |
| RTX 5090 32GB (desktop) | 61.4 | 135.0 | 4 | 0.38-0.87 | @taco-devs |
| RTX 5090 32GB (Q6_K, 128K) | 61.9 | 130.0 | 2 | 0.52-0.95 | @hypertectonic |
| RTX 5090 32GB (Q6_K, 256K) | 62.0 | 121.7 | 2 | 0.50-0.95 | @hypertectonic |
| RTX 3090 Ti 24GB (Q4_K_M, 128K) | 42.0 | 60.9 | 2 | 0.47-0.93 | @hypertectonic |
| RTX 3090 Ti 24GB (Q4_K_M, 256K) | 41.2 | 61.4 | 2 | 0.55-0.94 | @hypertectonic |
| RTX 5090 32GB desktop (UD-Q5_K_XL) | 66.3 | 144.2 | 4 | 0.32-0.89 | @TrickRiggin |
| 3× RTX 3060 12GB (layer split) | 17.3 | 24.5 | 8 | 0.87 | @EamonMcKiernan05 |
| AMD Radeon 9060 XT 16 GB | 15.2 | 28.7 | 2 | 0.62-0.93 | @kdrapel |
| RTX 3090 24GB (turboquant, n-max 6) | 39.8 | 61.5 | 6 | 0.61-0.90 | @NicholaiVogel |
| RTX 3090 24GB (b10450) | 41.3 | 63.5 | 2 | 0.69-0.87 | @hauntedhost |
| GMK EVO-X2, Ryzen AI Max+ 395 (64GB unified, ROCm/HIP) | 10.5-11.1 | 21.4-22.2 | 12 | 0.95-1.0 | @KyaniteLabs |
2× RTX 5060 Ti 16GB (PP, default -sm layer) |
22.1 | 42.8 | 2 | 0.53-0.94 | @Jackwwg83 |
2× RTX 5060 Ti 16GB (TP, -sm tensor) |
37.1 | 65.9 | 2 | 0.51-0.88 | @Jackwwg83 |
| RTX 5090 32GB (desktop) | 62.7 | 108.7 | 3 | 0.72 | @jcr211 |
| RTX 5090 32GB (desktop) | 69.3 | 129.1 | 4 | 0.55 | @paulomcg |
| RX 7900 GRE 16GB (Vulkan, packed) | 28.7 * | 47.8 avg (36.6–53.8) | 3 | 0.87–0.96 (avg ~0.93) | @lsunay (Hermes agent on PC-12) |
| RTX 5090 32GB (Q4_K_M, 131K) | 76.9 | 155.5 | 2 | 0.50-0.95 | @anstaendig |
| RTX 5090 32GB (UD_Q4_K_XL, 262K) | 74.3 | 179.7 | 4 | 0.35–0.91 | @anstaendig |
| RTX PRO 6000 Blackwell Max-Q 96GB | 45.7 | 97.1 | 2 | 0.52-0.95 | @awilliamson |
| 2× Tesla P40 24GB (tensor split) | 11.7 | 22.6 | 4 | 0.68-0.87 | @lyesrock |
| RTX 5090 32GB (UD-Q5_K_XL, 262K) | 69.6 | 147.5 | 6 | 0.69-0.87 | @lyesrock |
| 2× RTX 5060 Ti 16GB (TP, Q4_K_M) | 38.3 | 76.0 | 3 | 0.43-0.93 | @mgoswick |
| RTX 5090 32GB (UD-Q4_K_XL, 262K, q4_0 KV) | 76.3 | 171.7 | 4 | 0.69-0.87 | @lyesrock |
| 2× Tesla P40 24GB (UD-Q4_K_XL, 262K, q4_0 KV) | 13.2 | 23.4 | 4 | 0.82 | @lyesrock |
| RX 7900 XTX 24GB (Vulkan/RADV) | 28.8 | 70.7 | 3 | 0.43-0.95 | @Splizard |
| RTX 3090 24GB (UD-Q2_K_XL, OC) | 52.4 | 85.6 | 2 | 0.76 | @dcrey7 |
| RTX 3090 24GB (UD-Q4_K_XL, OC) | 43.9 | 79.6 | 2 | 0.78 | @dcrey7 |
RTX 3060 Ti 8GB + RTX 3080 Ti 12GB (TP, -sm tensor) |
39.4 | 49.2 | 2 | 0.895 | @randyjohnson11 |
| Apple M4 24GB (Metal) | 5.8 | 5.8 | 2 | 0.48-0.95 | @sternryan |
| Mac Studio (M3 Ultra, 96GB UMA, Q6_K, Metal) | 22.8 | 24.2 | 2 | 0.53-0.95 | @adityavsingh |
| RTX 3090 24GB (Q4_K_M, 131K, 250W) | 31.9 | 52.8 | 4 | 0.65 | @ctaylor83 |
| RTX 5090 32GB (UD-Q4_K_XL, 131K, q4_0 KV) | 74.4 | 182.0 | 4 | 0.38-0.90 | @cmoro-deusto |
| RTX 5080 16GB + RTX 3090 24GB (TP) | 55.6 | 92.9 | 2 | 0.51-0.95 | @plyra |
| RTX 5060 Laptop 8GB | 4.2 | 7.3 | 2 | 0.75-0.97 (0.90 aggregate) | @ENY66n |
| RTX PRO 3000 Blackwell Laptop 12GB | 5.3 | 10.8 | 2 | 0.67-0.93 | @ENY66n |
| RTX 3080 10GB (UD-IQ2_XXS, 16K, headless) | 45.1 | 64.4 | 2 | 0.44-0.93 | @ntindle |
| RTX 3080 Ti Laptop 16GB (UD-Q3_K_XL) | 14.8 | 28.6 | 2 | 0.76 | @salibam99 |
| NVIDIA CMP 170HX 64GB (unlocked HBM2e) | 33.0 | 46.7 | 2 | 0.53-0.94 | @shiwuxiu |
| RTX 4090 24GB | 47.0 | 68.8 | 2 | — | @snakeyjay63-png |
| RTX 4090 24GB (Purusha, LoRA merge) | 47.0 | 83.4 | 2 | — | @snakeyjay63-png |
| Tesla V100 32GB (Volta) | 33.3 | 48.0 | 2 | 0.75 | @cameron |
| 2× Tesla V100-SXM2-16GB (Volta, NVLink, layer split) | 35.7 | 59.1 | 4 | 0.33-0.92 | @ATIVX928 |
| RTX 5060 Ti 16GB (Q4-XYZ-v2, 32K) | 26.3 | 59.5 | 4 | 0.34-0.69 | @jaisusx |
| RX 7900 XTX 24GB (Windows/Vulkan) | 41.0 | 85.4 | 3 | 0.47-0.90 (0.73) | @pparuzel |
| 2× RTX A5000 24GB (NVLink, TP) | 51.9 | 81.1 | 2 | 0.56-0.96 | @TheRiotCoder |
| Ryzen AI Max+ 395 / Radeon 8060S 64GB (Linux, Vulkan/RADV) | 11.9 | 28.7 | 4 | 0.30-0.92 | @Nathanw1014 |
| RTX 5090 32GB (UD-Q4_K_XL Dynamic 3.0, 192K) | 74.7 | 160.8 | 4 | 0.86-0.93 | @paulomcg |
| RTX 5080 16GB | 53.4 | 101.3 | 2 | 0.53-0.95 | @ChumBoxBaron |
| RTX 4060 Ti 16GB (Q4-XYZ-v2, 32K) | 17.6 | 40.0 | 3 | 0.41-0.94 | @CeIest2 |
| RX 7900 XTX 24GB (Linux/ROCm 10, UD-Q4_K_M, 131K) | 36.3 | 62.6 | 2 | 0.56-0.94 (0.80) | @vijay-14 |
| RTX 5090 32GB (UD-Q4_K_XL Dynamic 3.0, 131K, b10680) | 71.9 | 170.3 | 7 | 0.19-0.85 (0.48 aggregate) | @hagope |
| RTX 2070 8GB + RTX 5060 Ti 16GB (mismatched Turing+Blackwell, PCIe x1 riser, TP)* | 25.4 | 41.5 | 2 | 0.81 (0.50-0.96) | @Lucas12807 |
| AMD Radeon 780M iGPU (Phoenix, 32GB UMA) | 4.1 | 8.4 | 2 | 0.50-0.92 (0.77 aggregate) | @ob7282 |
| AMD Radeon AI PRO R9700 32GB (Windows/Vulkan) | 29.7 | 56.3 | 3 | 0.39-0.97 (0.72) | @misterkerns |
| 2× RTX 5090 32GB (TP, UD-Q4_K_M) | 107.5 | 191.9 | 3 | 0.46-0.93 (0.72 aggregate) | @Enigmatic331 |
| AMD Instinct MI210 64GB (CDNA2/gfx90a, Q8_0, 32K, f16 KV, custom build) | 30.4 | 46.8 | 8 | 0.375 aggregate | @pestopoppa |
* A6000 row: unsloth Q8_K_XL, 256K context, q8_0 KV cache — 40.0 GB VRAM baseline, 41.4 GB with spec (rows above: Q4_K_M, 131K, q4_0 KV).
* RX 7900 XTX row: unsloth Q4_K_M, 131K context, q4_0 KV cache — 18.9 GB VRAM baseline, 19.7 GB with spec.
* 3×24GB TP row: two RTX 3090 + one 3090 Ti, Unsloth UD-Q6_K_XL, tensor-parallel --split-mode tensor, --parallel 4, 500K unified KV pool, q8_0 KV, f16 draft KV, mmproj Q8, temp 1.0. VRAM ~16.6 GB/GPU baseline, ~18.7 GB/GPU with spec (tightest card). Method: stock probe.py. --parallel 1 was not required on this host.
* RTX 4090 row: unsloth UD-Q4_K_XL, 160K context, q4_0 KV cache, llama.cpp b10360, Windows/CUDA, 275W power limit — 20.6 GB VRAM baseline, 21.9 GB with spec.
* RX 9070 row: two 16GB cards on one 31.84 GiB pool, Vulkan build b10426, unsloth UD-Q4_K_XL, 262K context, q8_0 KV cache — 28.0 GiB across the pool with spec. Method differs from the rows above and is spelled out under the sweep below.
* R9700 row: unsloth UD-Q4_K_XL, 262K context, q4_0 KV cache, llama.cpp b10433, Vulkan/RADV — 22.53 GB VRAM baseline, 24.55 GB with n-max 2. Method: unchanged probe.py at commit 67c20536, three runs x three prompts, thinking off.
* Ryzen AI Max+ 395 row: 64GB unified memory, unsloth UD-Q4_K_XL, 32K context, q8_0 KV cache, llama.cpp b10437, Windows build 26200, Vulkan with AMD driver 32.0.31035.1003. Method: unchanged probe.py at commit 67c2053, three runs x three prompts, thinking off.
* RTX 4090 Spadav_ row: unsloth Q4_K_M, 200K context, q4_0 KV cache, q8_0 draft KV, mmproj loaded (888MB on GPU) — method: stock probe.py + 4096-token curl (MTP crossover: overhead dominates at ≤400 tokens, +60% at 4096 tokens).
* Radeon 890M row: Ryzen AI 9 HX 370 (Strix Point, gfx1150), 48 GB UMA carve of 96 GB DDR5, unsloth UD-Q4_K_XL, 131K context, q4_0 KV cache, llama.cpp Vulkan/RADV, Ubuntu 26.04 — 19.15 GB VRAM baseline, 20.13 GB with n-max 2. Method: unchanged probe.py, three runs x three prompts, thinking off. Note this is a different APU class from the Ryzen AI Max+ 395 row above: Strix Point 890M is 16 CUs on a 128-bit bus, Strix Halo 8060S is 40 CUs on 256-bit, and this row holds 131K context resident against that row's 32K — both differences push this baseline down.
* RTX PRO 6000 row: unsloth Q4_K_M, 131K context, q4_0 KV cache, llama.cpp b10335, CUDA — method: stock probe.py, thinking off, --parallel 1 both sides. n-max 4 on this card: 85.7 overall (code 105.7 up, prose 58.8 down, acceptance 0.65-0.77) — overall peaks at n-max 2 here, same code-up/prose-down shape as the A6000 and 3×3090 sweeps. Cross-engine bonus on the same card: vLLM 0.27.1 with unsloth's NVFP4 build and MTP (--speculative-config '{"method":"mtp","num_speculative_tokens":N}') gives 63.3 -> 96.1 at n=2 and 116.3 (+84%) at n=4 — vLLM keeps climbing where llama.cpp has peaked. (An earlier revision carried a streamed-word-drop caveat for vLLM MTP; it was traced to the benchmark harness's own SSE parsing — a grep regex truncating multi-token deltas at escaped quotes — not to vLLM. Retracted with verification at vllm-project/vllm#52469, closed; streamed output is byte-identical to non-streamed with MTP on.)
* RTX 5090 desktop row: unsloth UD-Q4_K_XL, 192K context, q8_0 KV cache, llama-server self-built from the PR #26704 branch (master-equivalent for this path, CUDA arch 120), Windows 11 native — ~26.3 GB VRAM baseline, ~28.2 GB at n-max 4. Method differs from probe.py: server timings.predicted_per_second over 900-token generations, 2 runs x the same 3 prompts, thinking off, warmup discarded. Acceptance range is per-request (prose low end, Python high end), 0.60 aggregate.
* RTX 5090 32GB rows: unsloth Q6_K, 128K/256K context, q4_0 KV cache, llama.cpp 62bf73d, Windows/CUDA. Loaded VRAM was 24,441/27,385 MiB baseline and 25,797/29,381 MiB with spec. MTP gained 110.0% at 128K and 96.3% at 256K.
* RTX 3090 Ti rows: Q4_K_M, 128K/256K context, q4_0 KV cache, llama.cpp 62bf73d, Linux/CUDA. Loaded VRAM was 18,842/21,786 MiB baseline and 20,134/23,718 MiB with spec. MTP gained 45.0% at 128K and 49.0% at 256K. The 256K MTP arm had 846 MiB free at load.
* RTX 5090 32GB and RTX 3090 Ti method: unchanged probe.py at commit b299c0f, three runs x three prompts, thinking off, warmup discarded. Baseline and MTP used the same model and serving config at each context. Only the MTP arm added --spec-type draft-mtp --spec-draft-n-max 2.
* RTX 5090 desktop UD-Q5_K_XL row: 131K context, q4_0 KV cache, llama.cpp b10448 (ad1de39e0), Windows/CUDA 13.3. Loaded VRAM was 22,449 MiB baseline and 24,083 MiB at n-max 4. Method: unchanged probe.py at commit b299c0f, three runs x three prompts, thinking off, warmup discarded. N-max 2 reached 129.4 tok/s with 80.9% aggregate acceptance; n-max 4 reached 144.2 tok/s with 63.8% aggregate acceptance, a 117.5% gain over baseline. Only the MTP flags changed within each pair.
* 3×3060 row: unsloth UD-Q4_K_XL, 262K context, q8_0 KV cache, layer split --split-mode layer --tensor-split 35,37,28 (no NVLink), llama.cpp b10068, --parallel 1, --spec-draft-n-max 8 --spec-draft-p-min 0.85. Method differs from the rows above: serve-timing averages from llama-server logs, not probe.py — baseline = weighted mean of 126K decoded tokens with MTP off (17.3 t/s); with flag = weighted mean over 23.4M decoded tokens of live traffic (24.5 t/s, short bursts 28-39 t/s, long-context drops to 14-16 t/s). Acceptance = aggregate 0.87 (176,446/202,620 drafted tokens, per-task median 0.884, range 0.70-1.00) from draft acceptance log lines.
* Radeon 9060 XT 16 row: AtomicChat AD-IQ3_S_IQ3_XXS, 128k context, q4_0 KV cache — method: stock probe.py. Windows, ROCM
* RTX 3090 b10450 row: unsloth Q4_K_M, 131K context, q8_0 KV cache, q8_0 draft KV, llama.cpp b10450 (master ece963f41), CUDA/Linux (CachyOS), froggeric fixed chat template, --reasoning-format deepseek — method: unchanged probe.py, three runs x three prompts, thinking off. Worth noting: the b10450 baseline (41.3) equals the original day-one with-flag number for this card — the young hybrid-attention kernels caught up upstream, and the flag now stacks on top of that (+54%).
* GMK EVO-X2 row: Ryzen AI Max+ 395 (Strix Halo), 64GB unified memory, Linux, ROCm/HIP llama.cpp (gfx1151, ROCm 7.2.4), unsloth UD-Q4_K_XL, 96K context, f16 KV cache, --parallel 1, thinking off — 10.5-11.1 tok/s spec-off baseline; 21.4-22.2 tok/s with --spec-draft-n-max 12 at 0.95-1.0 acceptance on the bench prompt (novel-traffic acceptance 0.345); stacking --spec-type draft-mtp,ngram-mod --spec-ngram-mod-n-min 24 takes the streamed count bench to 59.7-64.0 cold and 148-163 warm on back-to-back repeats — the warm figure is an ngram repetition artifact on that prompt, not a general speedup (real novel traffic: prose 11-24, code 30-40 tok/s); production metric is time-per-task, 7.6-14.3 s per correct task across the thermal band. Method: streamed HTTP bench against a live llama-server (not probe.py), medians of 3+ runs; one-command reproducer: bench.sh, full writeup: one week with Qwen3.8-27B on Strix Halo.
* 2× RTX 5060 Ti rows: unsloth UD-Q4_K_XL (sha256 bee238bb…1372), 131K context, q4_0 KV cache, llama.cpp built from source at commit ece963f4 with -DCMAKE_CUDA_ARCHITECTURES=120, CUDA 13.0 / driver 580.173.02, PCIe 3.0 x8, PHB topology (no P2P). Method: unchanged probe.py, three runs x three prompts, thinking off. VRAM: PP 21.7 GB baseline / 23.0 GB with spec; TP 20.9 GB / 22.1 GB. The 16.68 GiB of weights do not fit one 16GB card, so two cards is the floor on this box — the split-mode choice is not optional here, which is what makes the two rows worth reading side by side. Both rows verified at n_ctx_slot = 131072. Details under the sweep below.
* RTX 5090 desktop row: unsloth Qwen3.8 Dynamic NVFP4 (FP8-as-Q8 unified-mtp), 163,840 context, q8_0 KV cache, llama.cpp b10430, Windows/CUDA, driver 610.88 — first NVFP4 quant in the table. Full n-max sweep: 2 → 98.4, 3 → 108.7 (+73%), 4 + p-min 0.60 → 103.9, 8 → 92.6 — deep-draft optimum consistent with the A6000 48GB pattern; n-max 8 confirmed worst spec setting. Method: unchanged probe.py, three runs x three prompts, thinking at template default (xhigh). Acceptance 0.721 aggregate (1845/2558 from server logs).
* RTX 5090 32GB row: unsloth UD-Q4_K_XL, 192K context, q8_0 KV cache, mmproj loaded (vision, --image-min-tokens 1024), llama-swap unified-cuda-2026-08-14, Linux/CUDA — 26.5 GB VRAM baseline, 28.5 GB with spec. Ungated — see the p-min A/B below. Both arms at --parallel 1 so only the spec flags differ. Method: stock probe.py, medians of three runs x three prompts, thinking off. n-max sweep at p-min 0.60 below.
* RTX 3090 turboquant row: unsloth Q4_K_M, 131K context, q4_0 KV cache, custom turboquant llama.cpp at commit 95b18c0, NVIDIA driver 610.43.03, --parallel 1, all layers on GPU, thinking off. Method: unchanged probe.py, medians of three runs x three prompts, same setup both arms. MTP at n-max 6 with p-min 0.75.
* RX 7900 GRE row: the only packed-16GB row in the table (96% VRAM with MTP; 86% spec-off). spec-off baseline back-filled 2026-08-16 on the same card (MTP removed as the only variable, live agent traffic to ~45K) — 28.5–28.8 tok/s at 35–45K context. custom AtomicChat IQ3_XXS quant (not unsloth Q4_K_M), 90K context, turbo3/turboquant KV cache, llama.cpp 1655 (2168b0cd8) in a custom llama-cpp-turboquant Docker image, Debian 13 trixie, kernel 6.12.101, Vulkan/AMD Navi 31, --parallel 1, --reasoning-budget 512, flash-attn on. Both arms live Hermes agent traffic (not probe.py), context bands differ by ~1.5K tokens. Full study under the section below.
* RTX 3090 turbo3 section: same host (RTX 3090 24GB, Debian 12, Ryzen 7 9700X, CPB disabled), unsloth Q4_K_M, both arms MTP-on (--spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75 --spec-default, -np 1) — this row compares two KV cache configurations, not spec-off vs spec-on. Baseline arm: q4_0 KV, 185K context, llama27b-mtp:cuda image (mtp-clean fork), live Hermes agent traffic (14 completed turns, 100–2700 output tokens, avg 37.2 tok/s, acceptance 0.57). Flag arm: turbo3 KV, 200K (204800) context (262K exceeded 24 GB at model load), llama-cpp-turboquant:cuda-latest (v10465, fca3093c9), clean probe 3×400 tokens, medians of 3, avg 54.2 tok/s (48.06–58.34), acceptance 0.64–0.84. VRAM 23.0/23.3 GB of 24.6 GB (~94–95%), 52–63°C. Confound disclosed: the two arms run different llama.cpp builds (mtp-clean vs turboquant v10465), so part of the delta is the newer build, not only the KV cache. Method differs from the rows above and is spelled out below.
* RTX 5090 32GB (Q4_K_M, 131K): Unsloth Q4_K_M, 131K context, q4_0 KV cache - 19.54GB baseline, 20.38GB with spec.
* RTX 5090 32GB (UD_Q4_K_XL, 262K): Unsloth Q4_K_XL, 262K context, q4_0 KV cache - 23.27GB baseline, 24.75GB with spec.
* RTX PRO 6000 Blackwell Max-Q row: lmstudio-community/Qwen3.8-27B-GGUF Q8_0, 131K context, q4_0 K/V cache, llama.cpp 0.1.0-dev build 10454 (4df29be4f), built with GNU 16.1.1 for Linux x86_64 / CUDA 13.3.1-1, RTX PRO 6000 Blackwell Max-Q Workstation Edition 96GB at 300W, --parallel 1. VRAM: 32.072 GiB baseline / 33.540 GiB with spec (+1468 MiB). Method: unchanged probe.py, three runs x three prompts, thinking off; qwen38-mtp (28527fc55) commit used for probe.py.
* 2× Tesla P40 row: unsloth UD-Q5_K_XL, 131K context, q4_0 KV cache, llama.cpp b10453 (3cb7ffb1a), Ubuntu 25.04 / CUDA 12.9, driver 580.178.04, --tensor-split 1,1 with numactl --interleave=all, P2P disabled (cards on separate NUMA nodes, SYS path), --parallel 1, thinking off. ~11 GB/GPU baseline, ~12 GB/GPU with spec. Method: probe.py unchanged, three runs x three prompts, both arms otherwise identical. MTP arm: --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75; prefill 354 → 295 tok/s (20K tokens). Full n-max sweep and a multi-model companion sweep below.
* RTX 5090 32GB UD-Q5_K_XL 262K row: unsloth UD-Q5_K_XL, 262K context, q4_0 KV cache, llama.cpp b10453 (3cb7ffb1a), Linux/CUDA, driver 610.57.04. Loaded VRAM ~32.1 GB. Method: unchanged probe.py, three runs x three prompts, thinking off, warmup discarded. Both arms at --parallel 1. Baseline 69.6 tok/s; with --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.75 147.5 tok/s (+112%). Acceptance 0.69-0.87 (avg 0.78). N-max 4 ungated reached 155.8 tok/s (0.31-0.87 acceptance) — ungated is faster but the gated config is the safer daily bet.
* 2× RTX 5060 Ti Q4_K_M row: unsloth Q4_K_M, 131K context, q4_0 KV cache, --split-mode tensor, llama.cpp b10450 (ece963f), Windows 11 / CUDA 13.2 native sm_120a, driver 596.36, --parallel 1. VRAM per GPU: 9,808 MiB baseline / 10,684 MiB at n-max 3. Method: unchanged probe.py at commit b299c0f, three runs x three prompts, thinking off, warmup discarded. Only the MTP arm added --spec-type draft-mtp --spec-draft-n-max 3.
* RTX 5090 32GB UD-Q4_K_XL 262K q4_0 KV row: unsloth UD-Q4_K_XL, 262K context, q4_0 KV cache (both K and V), llama.cpp b10453 (3cb7ffb1a), Linux/CUDA, driver 610.57.04. Loaded VRAM ~25.8 GB. Method: unchanged probe.py, three runs x three prompts, thinking off, warmup discarded. Both arms at --parallel 1. Baseline 76.3 tok/s; with --spec-type draft-mtp --spec-draft-n-max 4 (ungated, no p-min) 171.7 tok/s (+125%). Acceptance 0.69-0.87 (avg 0.69). N-max 6 gated (p-min 0.75) reached 154.5 tok/s (0.77-0.87 acceptance) — ungated is faster, gated is the safer daily bet.
* 2× Tesla P40 UD-Q4_K_XL 262K q4_0 KV row: unsloth UD-Q4_K_XL, 262K context, q4_0 KV cache (both K and V), llama.cpp b10453 (3cb7ffb1a), Linux/CUDA, --tensor-split 1,1 (no NVLink), --parallel 1. Method: unchanged probe.py, three runs x three prompts, thinking off, warmup discarded. Baseline 13.2 tok/s; with --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75 23.4 tok/s (+77%). Acceptance 0.82 avg. N-max 4 ungated reached 23.6 tok/s (0.68 acceptance) — only +0.8% faster (noise) with much lower acceptance; gated is the clear winner on bandwidth-starved Pascal. N-max 6 gated: 22.5 tok/s (0.80 acceptance) — n-max 4 is the sweet spot. KV f16 was within 2% of q4_0 on all arms (13.4 / 23.6 / 22.4 / 22.6) and also fits 262K ctx on 2×24GB with this quant — q4_0 KV buys headroom, not speed; f16 is the default choice when VRAM allows.
* RX 7900 XTX Vulkan row: unsloth Q4_K_M, 131K context, q4_0 KV cache, llama.cpp master 4695f00 (2026-08-17), Vulkan backend (RADV, Mesa 26.1.5) on Void Linux musl, kernel 7.0.14, ReBAR enabled, Ryzen 9 5900XT host — no ROCm installed; this is mainline Mesa only. 0.6 GB VRAM before serve, 18.6 GB serving baseline, 19.9 GB with spec. Method: unchanged probe.py at dc18736, three runs x three prompts, thinking off, both arms --parallel 1. Row is the n-max 3 arm (aggregate acceptance 0.72); n-max 2 on the same config measured 57.9 at 0.80 aggregate (1652/2055), the range's low end is the prose prompt in both arms. Full sweep and gate A/B in the section below. Relative to the earlier XTX row: this baseline reads lower (28.8 vs 30.7) while the flag arms read far higher — different backend and a 2026-08-17 build (see rule 6); the deltas are the durable part.
* RTX 3090 UD-Q2_K_XL row: unsloth UD-Q2_K_XL (10.7 GB), 32K context, f16 KV cache, llama.cpp master source snapshot of 2026-08-14 (built from archive, so no commit hash; it contains the ssm_scan state-rollback that makes deep MTP drafts work on the DeltaNet layers), CUDA 13 self-built, CachyOS/Linux, driver 610.43.03. VRAM 13.9 GB baseline, 14.7 GB with spec. The card is overclocked: memory +1200, core +180, power limit 430 W against a 390 W default. Method: unmodified probe.py at a4c3028da1, three full passes per arm, the table shows the median of the three pass medians; passes were 52.5/52.4/52.3 and 85.6/85.1/87.2. Both arms --parallel 1, thinking off.
* RTX 3090 UD-Q4_K_XL row: same machine, same build, same method, same overclock; unsloth UD-Q4_K_XL (17.9 GB), 32K context, f16 KV. VRAM 20.5 GB baseline, 21.4 GB with spec. Passes were 43.9/43.9/43.9 and 79.8/79.6/78.2. This row is the one to compare against the other 3090 rows, since those use ~17 GB Q4 files; the UD-Q2_K_XL row above is faster mainly because it reads 38% fewer weight bytes per token, not because of the overclock. See the section below.
* 3060 Ti + 3080 Ti row: Unsloth Dynamic IQ3_XXS (Qwen38_27B-UD-IQ3_XXS.gguf) + mmproj-BF16, tensor-parallel across an 8GB 3060 Ti and a 12GB 3080 Ti (--split-mode tensor, --tensor-split 11,9), 131K context, q4_0 KV cache, llama.cpp local build @ f9779dd, Linux/CUDA (driver 595.84). VRAM: ~0.2/0.9 GB pre-load baseline (desktop residual) vs 7.5/11.3 GB with spec. Method: stock probe.py @ dc18736, both arms --parallel 1, medians of 3 runs; with-flag arm run three times at the exact spec-on config (49.2 / 46.4 / 52.4) — reported value is the median. Prose prompt runs ~3 tok/s slower with the flag in every arm; the gain is carried by code prompts.
* Apple M4 row: first Apple Silicon row. M4 base (10-core GPU, ~120 GB/s), 24GB unified memory, macOS 15 / Metal, iogpu.wired_limit_mb=21504. unsloth UD-Q3_K_XL (12.5 GiB), 32K context, q4_0 KV cache, llama.cpp b10450 (ece963f41, Homebrew bottle), -b 512 -ub 512 — the default -b 2048 OOMs the spec arm mid-generation and takes the server down, see the section. VRAM 13.2 GB baseline, 13.9 GB with spec. Method: unmodified probe.py at a4c3028da1 (which is itself medians of three runs per prompt), run as two independent full passes per arm — six samples per prompt, --parallel 1, thinking off. Runs were 5.8/5.8 baseline and 5.6/5.9 with the flag, so the two runs straddle parity and the overall number is a wash within noise (band ~±0.2 tok/s on ~6). The reproducible part is the per-prompt split: code +9-10%, prose -22-24%. Batch-8 decode amortizes only 1.2x on this backend against 3.3x on CUDA, which is why; a matched RTX 3090 control on the same GGUF, flags and commit is in the sweep rather than this table, since it is a method check and not a new card. Full analysis in sweeps/apple-silicon.md.
* Mac Studio M3 Ultra row: Apple M3 Ultra with 96 GB unified memory, macOS/Metal; unsloth Q6_K (21 GB file), 131K context, Flash Attention, q4_0 K/V main and draft KV caches, --parallel 1, LM Studio llama.cpp Metal backend 2.29.0 (llama-server version: 1 (dd1ea52)). Process RSS was 23.20 GiB baseline / 24.34 GiB at MTP n-max 2; Apple has no discrete VRAM, so these are process-RSS measurements, not VRAM allocations. Method: unmodified probe.py, warmup discarded, three runs x three prompts, thinking off, and Qwen3.8 was the only resident model. The table uses the overall probe median. Only the MTP arm enabled --spec-type draft-mtp --spec-draft-n-max 2 and its q4_0 draft KV cache. Aggregate acceptance was 0.798 (1681/2106); the n-max and p-min sweep is below.
* RTX 3090 Q4_K_M 131K 250W row: Unsloth Qwen3.8-27B Q4_K_M, 131072 context, q4_0 K/V cache, llama.cpp b10473 (b75ecd197), Fedora 44/CUDA 13.3, NVIDIA 610.43.03, RTX 3090 24GB capped at 250W. Loaded VRAM was 18,856 MiB baseline and 20,486 MiB at n-max 4. Method: unchanged probe.py at commit 70a699e4, --parallel 1 both arms, thinking off, three complete probe passes per arm; reported values are the median of the three overall medians. Only the MTP arm added --spec-type draft-mtp --spec-draft-n-max 4. Aggregate n-max 4 acceptance: 5420/8370 = 0.648.
* RTX 5090 32GB UD-Q4_K_XL 131K row: ASUS TUF RTX 5090 32GB, unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL, 131K context, q4_0 KV cache (both K and V), llama.cpp build 10451 (10bf611e5), Arch Linux kernel 7.1.8, CUDA. Serving config is stock serve_mtp.sh (-c 131072 -ngl 999 -fa 1 --cache-type-k q4_0 --cache-type-v q4_0 --parallel 1); the spec arm changed only --spec-draft-n-max, 2 to 4. VRAM 20,366 MiB baseline / 21,500 MiB with spec (+1,134 MiB). Method: unchanged probe.py at 70a699e, three runs x three prompts, thinking off, warmup discarded. Table figures are the OVERALL medians; pooled means were 74.5 baseline and 169.1 with spec, so the gain reads +145% on medians and +127% on means. Acceptance aggregate 0.69 (1860/2696 from server logs); the quoted range is per-run and the spread is unusually wide — code 0.87, bash 0.74, prose 0.40 aggregate, with mean draft length falling 4.60 to 2.52 along the same axis. The baseline is exceptionally flat (74.3-74.8 across all nine runs), i.e. fully bandwidth-bound before the flag. Compare the 262K UD-Q4_K_XL row above: 74.3 → 179.7 at the same n-max 4 on a different host, so halving context to 131K moved this pair very little.
* RTX 5080 + 3090 row: mixed Blackwell + Ampere, no NVLink. unsloth UD-Q4_K_XL (sha256 bee238bb…1372), 131K context, q4_0 KV cache (K and V), llama.cpp b10470 (34af94c, official win-cuda-13.3 zip), Windows 11 / CUDA 13.3, driver 610.88, --split-mode tensor, --parallel 1, flash-attn on. VRAM 12,048+10,334 MiB baseline, 12,827+11,130 MiB with spec. Method: unchanged probe.py, three runs x three prompts, thinking off, warmup discarded. Both arms otherwise identical. Only the MTP arm added --spec-type draft-mtp --spec-draft-n-max 2 (ungated; 5080 is Blackwell). Server log: n_ctx_slot = 131072. Note: backend sampling not supported with SPLIT_MODE_TENSOR; using CPU sampler — MTP still 55.6 → 92.9 (+67%). Per-prompt medians: code 55.7→105.2, prose 54.0→79.2, bash 55.8→92.9. Acceptance 0.51-0.95 (agg ~0.79 from draft acceptance lines, warmup excluded). No n-max sweep yet.
* RTX 5060 Laptop 8GB row: unsloth/Qwen3.8-27B-GGUF IQ4_XS (15,705,861,088 bytes, sha256 9fd40d70...ce666), 64K context, q4_0 K/V cache, 25 GPU layers with the remaining layers on CPU, llama.cpp b10453 (3cb7ffb1a), Windows build 26200 / CUDA 12.4, driver 582.05, Ryzen 7 260, 31.31 GiB RAM, --parallel 1, Flash Attention on, batch 2048 / ubatch 512. Loaded VRAM: 6,796 MiB baseline / 7,540 MiB with MTP (6,868 / 7,614 MiB after the probe). Method: unchanged probe.py at commit 70a699e, three runs x three prompts, thinking off, warmup discarded. Both arms were otherwise identical; MTP added --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.70. Acceptance 0.75-0.97, 0.90 aggregate (1,594 / 1,770). This is intentional hybrid CPU/GPU offload; the full model weights are not resident in 8GB VRAM.
* RTX PRO 3000 Blackwell Laptop 12GB row: unsloth Qwen3.8-27B IQ4_XS (SHA-256 9fd40d70...ace666), 64K context, 38/66 layers on GPU, with the remaining layers on CPU, q4_0 K/V cache, batches 2048/512, 24/24 threads, Flash Attention, llama.cpp b10456 (f275595dd) official Windows CUDA 12.4 build, Windows 11 Enterprise, NVIDIA driver 573.71 (CUDA 12.8 compatibility), --parallel 1, thinking off. Peak monitored dedicated VRAM was 10,112 MiB baseline / 10,922 MiB MTP. Method: unchanged probe.py at commit 70a699e4, three runs x three prompts, warmup discarded. Overall nine-run median was 5.3 tok/s baseline / 10.8 tok/s with --spec-type draft-mtp --spec-draft-n-max 2 (+104%); only those two MTP options changed. Draft acceptance was 1,747/2,047 aggregate (0.85), per-request range 0.67-0.93. Laptop was on AC; peak temperature 67/68 C baseline/MTP.
* RTX 3080 10GB row: the smallest-VRAM card in the merged table, and the only one holding the weights fully resident under 12 GB. unsloth UD-IQ2_XXS (Qwen3.8-27B-UD-IQ2_XXS.gguf, 9,010,048,064 B, sha256 8d1b3729…c35b8524, verified byte-exact against the HF repo), 16K context, q8_0 KV cache, llama.cpp b10472 (60eeeb608), Windows 11 build 26200 / CUDA 12.4, driver 591.86, RTX 3080 10GB GA102 SM86 at stock 370 W on PCIe 4.0 x16, Ryzen 9 7900X + 32 GB DDR5-6000, full offload (-ngl 99, every layer), --parallel 1, --jinja, thinking off, both arms otherwise identical. VRAM 9,156 MiB baseline / 9,734 MiB with spec (+578 MiB) of 10,240 MiB. llama-bench -ngl 99 -fa 1 on the same file: pp512 1132.46 ± 15.80, tg128 45.86 ± 0.04. Method: unchanged probe.py at 70a699e, medians of three complete passes per @dcrey7's note (baseline 45.8/45.1/44.8; n-max 2 63.2/65.3/64.4). Aggregate acceptance 0.74 (4,543/6,179 from server logs). n-max 3 measured 62.9 at 0.67 — n-max 2 is the peak, matching rule 1. The card must be headless and that is load-bearing: with both monitors on the 7900X iGPU the desktop's footprint on the 3080 is 64 MiB; with them attached dwm alone holds 1,846 MiB, full offload does not fit, and the same box baselines 33.0 at ngl 62. The MTP arm's extra ~580 MiB, not the baseline's fit, sets the context ceiling: 32K inverts to 45.4 → 24.7 (−46%) while acceptance stays at 0.73 and nvidia-smi still reports 446 MiB free. Measured at the bus, the inverted arm moves 11.3 GB/s rx / 13.6 GB/s tx against 0.66 / 1.34 healthy, and pulls 252 W against 366 W — starved on PCIe, not throttling. Full study in sweeps/rtx-3090.md.
* RTX 3080 Ti Laptop row: unsloth UD-Q3_K_XL (13.4 GB), 16K context, q4_0 KV cache, llama.cpp on Debian Linux / CUDA 12.4, driver 550.163, 115W TGP. Method: stock probe.py, three prompts x three runs, --parallel 1, thinking off. Baseline 14.8 tok/s median [15.3, 14.8, 14.0]; with --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.60 reached 28.6 tok/s overall median (code 32.9, prose 24.8, bash 28.6) (+93% overall speedup, 1.93x). Full weight offloaded into 16GB VRAM.
* CMP 170HX row: unsloth Q4_K_M (15.93 GiB), 131K context, q4_0 KV cache, llama.cpp 0.1.0-dev commit f275595dd (reported as build 1), Linux/CUDA, driver 610.43.02, headless Ubuntu VM. The card is a GA100 CMP 170HX (70 SMs) with Hynix HBM2e unlocked from 8GB to 64GB, 200W cap, PCIe Gen1 x4. VRAM 18,864 MiB baseline, 20,152 MiB with spec. Method: unchanged probe.py at 70a699e, three runs x three prompts, thinking off, both arms --parallel 1. Only the MTP arm added --spec-type draft-mtp --spec-draft-n-max 2. Per-prompt medians: baseline 33.1 / 33.0 / 32.7, with flag 51.4 / 38.3 / 46.7 (code / prose / bash). Overall mean 32.9 -> 45.5, median 33.0 -> 46.7 (+41.5%). Draft acceptance from slot print_timing 0.53-0.94 (prose low end, Python high end). No n-max sweep in this PR; 64GB has headroom to try 3-4 later. See sweeps/cmp-170hx.md.
* RTX 4090 snakeyjay63-png rows: four configs measured on same hardware, Ryzen 7 9800X3D, Linux/CUDA, 500W power limit (raised from 450W stock default — likely pushes tokens/s a touch higher than stock-limit rows). Row 1 (Q4_K_M): unsloth Q4_K_M, 131K context, q4_0 KV cache, ngl 999, flash attention on, MTP n-max 2 — overall 68.8, per-prompt: Python 47.0→91.4 (+94%), Prose 47.0→64.6 (+38%), Bash 47.3→67.4 (+43%). Row 2 (Purusha ★): LoRA merge (18 safetensors on stock Qwen3.8-27B), three variants tested — (b) 65K/ngl 79/p-split 0.05 → 81.5, (c) 131K/ngl 999 → 83.1, (d) 65K/ngl 79/p-split 0.06 → 83.4, overall +78%. Simplest config = highest score. First row with GPU metrics: baseline 353W/64°C/97% mem util vs MTP 403W/69°C/88% mem util — +50W, +5°C, +519 MiB VRAM, but GPU util drops 3% and mem util drops 9% (fewer forward passes). Method: stock probe.py, 3 runs × 3 prompts, thinking off.
* Tesla V100 32GB row: one V100 32GB in a Dell PowerEdge R720, unsloth Q4_K_M, 4K context, f16 KV cache, flash attention on, llama.cpp b10358 (030ebb558), NixOS / CUDA driver 580.142, both arms at --parallel 1 with only --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0 added. VRAM 18,022 MiB spec-off against 18,864 MiB with the head, measured at 32K context rather than at this row's 4K. Method differs from probe.py: non-streaming requests scored on the server's own timings.predicted_per_second, greedy, n_predict 400 with ignore_eos, cache_prompt false, medians of five samples after a discarded warmup — prose 33.6 → 47.9, code 33.0 → 48.1, each arm inside 2.2% of its own minimum. Depth and build sweeps in sweeps/workstation.md.
* RTX 5060 Ti 16GB row: quimmedes/Qwen3.8-27B-XYZ Q4-XYZ-v2 (15.06 GB — the largest Q4-tier file that fits one 16 GB card; unsloth Q4_K_M 17.11 GB, IQ4_NL 16.34 GB and UD-Q4_K_XL 17.92 GB do not fit a single card), 32K context, q4_0 KV cache, llama.cpp b10472 (60eeeb6, 2026-08-17) built from source with CUDA 12.8 (-DCMAKE_CUDA_ARCHITECTURES=120), Ubuntu 24.04 / CUDA, driver 595.84, Ryzen 5 3600. Method: unchanged probe.py at 70a699e, three runs x three prompts, thinking off, --parallel 1 both arms. VRAM 14,514 MiB baseline / 15,704 MiB at n-max 4. Full n-max sweep (2/3/4/6 + p-min) and an unsloth IQ4-family study in sweeps/rtx-5060-ti.md: draft depth keeps paying on this card (n2→n4: 50.0→59.3), n-max 6 OOMs at load (99.1% VRAM at n4), p-min 0.65 gating costs 11.5% throughput (52.5 vs 59.3) despite raising acceptance, and a June-22 vs b10472 build A/B shows only +1.5% baseline / +0.3% at n4 (rule 6's +10-15% did not reproduce on this card).
* RX 7900 XTX Windows/Vulkan row: unsloth UD-Q4_K_XL (via -hf), 131K context, q4_0 KV cache (K and V), flash attention on, official ggml.llamacpp WinGet build 9553 (9e3b928fd, Clang 19.1.5) on Windows 11, Vulkan backend (AMD driver, no ROCm — not RADV), Ryzen 7 9800X3D, --parallel 1 all arms, thinking off. VRAM 19.9 GiB baseline / 20.9 GiB at n-max 2 / 21.1 GiB at n-max 3 (811 MiB pre-serve desktop). Method: unchanged probe.py at c7bc415, three runs x three prompts, warmup discarded. Only the MTP arm added --spec-type draft-mtp (ungated). Row is the n-max 3 arm: 41.0 → 85.4 (+108%), per-prompt 40.9/41.1/40.8 baseline vs 97.9/63.6/85.4 with flag (code/prose/bash), acceptance 0.73 aggregate (1740/2369), range 0.47-0.90, code 0.90 / prose 0.48 / bash 0.75. n-max 2 on the same config: 83.6/65.4/76.9, overall 76.9 (+88%), 0.79 aggregate acceptance (1681/2130; a re-run reproduced 82.2/67.1/73.9). n-max 4 is past the optimum: 89.8/45.1/70.8, overall 70.8 (+73%), 0.61 aggregate (1709/2791). Full sweep in the section below.
* 2× RTX A5000 NVLink row: the table's first NVLink pair — two RTX A5000 24GB on an NV4 bridge (~112 GB/s), Threadripper PRO 5965WX, 128 GB RAM, Ubuntu 24.04 headless, driver 595-open, CUDA 13.3, both cards ECC-disabled. unsloth/Qwen3.8-27B-GGUF Q4_K_M (17,106,775,008 B, sha256 7e78da5d…fe169 — upstream replaced this file on 2026-08-19: that filename now 404s, superseded by the Dynamic 3.0 UD-Q4_K_M 16,464,440,224 B 322e194f…, nearest by size UD-Q4_K_XL 17,559,178,144 B 3f227079…. The numbers below are from the original file. The MTP head is still inline in UD-Q4_K_M / UD-Q4_K_XL / UD-Q2_K_XL, verified via qwen35.nextn_predict_layers), 131K context, q4_0 K/V cache, llama.cpp 0.1.0-dev build 10454 (4df29be4f, official server-cuda image, digest sha256:f0b55651…950e7), Linux/CUDA, --split-mode tensor on both arms, --parallel 1, flash attention on, thinking off. VRAM 20,188 MiB baseline / 21,896 MiB with spec (+1,708 MiB), 22,868 MiB at n-max 8. Method: unchanged probe.py at a4c3028, three complete passes per arm, the row is the median of the three pass medians — baseline 52.0/51.6/51.9, n-max 2 81.1/80.5/82.9. Only the MTP arm added --spec-type draft-mtp --spec-draft-n-max 2. Aggregate acceptance 0.792 (1,522/1,921 from draft acceptance lines, warmup excluded), per-task 0.56-0.96. Split mode is the bigger lever here: on the default --split-mode layer the same box baselines 35.1 and reaches only 54.8 with the flag, so tensor split is +47% before any spec flag and the two stack to 2.7x. n-max 8 is faster still (96.1 median) but its spread across passes was 88.9/96.1/107.9 against 0.6% for the baseline, so the row uses the comparable n-max 2 arm; sweep, per-prompt split, and a quant that has no MTP head at all are in sweeps/multi-gpu.md.
* Ryzen AI Max+ 395 Linux/RADV row: the first Linux Vulkan row for this APU; the two existing 395 rows are Windows Vulkan and Linux ROCm/HIP, so this is the same silicon on a third stack. Framework Desktop, 64GB unified memory, headless (no compositor, so rule 7's desktop tax does not apply here). unsloth UD-Q4_K_XL (Qwen3.8-27B-UD-Q4_K_XL.gguf, 17,923,394,624 B, sha256 bee238bb...1372, the same file the 2x RTX 5060 Ti and RTX 5080 + 3090 rows cite), 131K context (n_ctx_slot = 131072 confirmed in both arms), q4_0 K and V cache, --parallel 1 both arms, thinking off. Ubuntu, kernel 6.17.0-35-generic, amd_iommu=off, Vulkan/RADV on Mesa 26.3.0-devel (git-d18d598e27). Unified memory, so these are GTT figures rather than discrete VRAM: 18.90 GiB serving baseline against 20.61 GiB at n-max 4, so spec costs +1.71 GiB; idle GTT is 0.02 GiB on this box. The build is not upstream and that is load-bearing for the absolute numbers: llama-server from Nathanw1014/llama.cpp branch strix-halo-vulkan at baf6360be, a gfx1151 tuning fork whose Vulkan perf flags are compiled defaults-on, with no GGML_VK_* environment overrides set in either arm; both arms are the same binary and the same serving config so the pairing is clean, but stock master will not reproduce the absolute figures and may not reproduce the delta. Method: unchanged probe.py at a4c3028da, three complete passes per arm (each pass being probe.py's own 3 prompts x 3 runs), warmup discarded, and the table figures are the median of the three pass medians; only the MTP arm added --spec-type draft-mtp --spec-draft-n-max 4. Acceptance is aggregate 6004/9345 = 0.642 from draft acceptance log lines with the per-pass warmup request excluded; the quoted range is per-request. Every arm held an flock GPU mutex so nothing else shared the device, and the session opened and closed with a baseline arm reading 11.9 both times (per-prompt 11.8/11.9/11.8 against 11.9/12.0/11.8), so nothing drifted across the run. n-max sweep, overall medians: 2 -> 24.6 (2.07x, acceptance 0.797), 3 -> 27.7 (2.33x, 0.732), 4 -> 28.7 (2.41x, 0.642), 12 -> 13.9 (1.17x, 0.274). Peak is n-max 3 to 4: a repeat launch of one config moved 2.8% on this box, so 27.7 against 28.7 is not a resolved difference and the row quotes the arm that measured highest. Either way this APU sits with rule 1's "bigger or faster cards at 3-4" rather than the 24GB cards' n-max 2. Full detail, including why n-max 12 collapses and the measured launch-to-launch variance floor, in sweeps/apu-igpu.md.
* 2× Tesla V100-SXM2-16GB row: two V100-SXM2-16GB (Volta, sm_70) with NVLink + P2P, --split-mode layer, unsloth Q4_K_M, 262144 context, q4_0 KV cache (both K and V), llama.cpp 0.1.1-dev commit 25ae3a9, Ubuntu 24.04 / CUDA 12.8, driver 580.173.02. VRAM 12,302 + 12,868 MiB baseline against 12,618 + 15,692 MiB at n-max 4. Method: unchanged probe.py, three runs x three prompts, thinking off, both arms --parallel 1, only the spec flags changed. Baseline 35.7 tok/s; n-max 4 59.1 tok/s (+65.5%). Full n-max sweep (2-6): 2 → 57.5, 3 → 57.8, 4 → 59.1, 5 → 52.3, 6 → 45.6 — the optimum is shallow, matching the single-V100 pattern, and acceptance decays monotonically (0.78 → 0.73 → 0.66 → 0.57 → 0.51). Per-prompt medians at n-max 4: code 69.4/70.0/68.8, prose 39.3/39.4/35.8, bash 60.0/59.1/56.3. Draft acceptance from slot print_timing 0.33-0.92 (prose low end, code high end).
* RTX 5090 Dynamic 3.0 row: unsloth UD-Q4_K_XL Dynamic 3.0 (sha256 3f227079003add2511437e5b1e94812e363385225bf6a9b47b0054a72bc8b01e, re-quantized and re-uploaded 2026-08-19), 192K context, q8_0 KV cache, mmproj-BF16 loaded, llama.cpp 3d93885 as shipped in the llama-swap unified-cuda-2026-08-14 image, Linux/CUDA, headless (desktop moved off this card to the iGPU). VRAM 25,199 MiB baseline / 27,169 MiB at n-max 4. Method: unchanged probe.py at b299c0f, three runs x three prompts, thinking off, --parallel 1 both arms, ungated (no --spec-draft-p-min); each cell run twice in opposite arm order for 18 runs total, medians over all 18. Every arm gated on 60s of sustained idle GPU first — a contended first attempt gave a 2x spread inside one prompt and was discarded. Acceptance is two single-request samples (226/244, 151/176), not a distribution. Different rig, OS and method from my Windows row above; the two are not comparable to each other. Dynamic 2.0 vs 3.0 A/B in sweeps/rtx-5090.md.
* RTX 5080 16GB: unsloth Qwen3.8-27B Q3_K_M, ctx 65536, KV q4_0+q4_0, llama.cpp b10488, Windows 11 Pro / CUDA 13.3, VRAM 48 MiB idle → 14,445 MiB baseline / 15,359 MiB MTP, probe.py unchanged @master (both arms), --parallel 1, flash-attn on, thinking off.
* RTX 4060 Ti 16GB row: quimmedes/Qwen3.8-27B-XYZ Q4-XYZ-v2 (15,064,569,440 B, sha256 ab58f29fa81dd604… — same file as the 5060 Ti row above, making the two a controlled cross-card pair: 288 vs 448 GB/s), 32K context, q4_0 KV cache (both K and V), llama.cpp master 9a286ac (2026-08-21) built from source with CUDA 12.8 (-DCMAKE_CUDA_ARCHITECTURES=89), Ubuntu 24.04 / CUDA, driver 580.173.02, desktop nearly idle during bench (~280 MiB GPU footprint). Method: unchanged probe.py at c7bc415, three runs x three prompts, thinking off, --parallel 1 both arms. VRAM 14,778 MiB baseline / 15,830 MiB at n-max 3. Full n-max sweep (2/3/4 + p-min 0.70) and a Q4-XYZ v1-vs-v2 study in sweeps/rtx-4060-ti.md: depth pays to n-max 3 (+11% over n2), n-max 4 OOMs at load (same 130 MiB shortfall as the 5060 Ti's n-max 6, also with -b 512 -ub 512), p-min 0.70 gating is a wash at the ceiling (−0.5%, acceptance 0.72-0.98), and the quant version alone moves the depth optimum by a full step on identical silicon and build.
* RTX 2070 + RTX 5060 Ti row: unsloth Qwen3.8-27B-UD-Q4_K_XL (17,559,178,144 bytes), 65536 context, q8_0 KV cache, llama.cpp b10662 (CUDA 13.3), Windows 11, driver 610.47. Mismatched pair on one board: RTX 2070 8GB (Turing) sits outside the case on a powered USB x1 riser (PCIe 2.0 x1, ~500 MB/s — the motherboard, an AM4 ASUS PRIME B450M-A, has only one physical x16 slot), RTX 5060 Ti 16GB (Blackwell) in the primary x16 slot and driving the desktop. --split-mode tensor --tensor-split 5,2 (5060 Ti,2070 order), --parallel 1. Method: stock probe.py unchanged, three runs x three prompts both arms (rerun at the maintainer's request — supersedes an earlier best-of-2 /completion pass on the same config, which had reported 25.5/37.7). Baseline 25.4 tok/s, MTP n-max 2 41.5 tok/s median (mean 39.6; per-prompt spread 32.6-45.0, one prompt runs noticeably slower under spec than the other two). Acceptance aggregate 0.81 (1,588/1,951 drafted tokens across the 9 with-flag runs, warmup excluded), per-run range 0.50-0.96 — the same prompt that decodes slowest also has the lowest acceptance. VRAM logged at this exact 5,2 split (earlier revision only had it for the production 13,7 ratio): 2070 6.0→6.5 GB/8.0 GB baseline→MTP, 5060 Ti 14.2→14.9 GB/15.9 GB baseline→MTP. Full sweep, the riser-bandwidth discussion, and a KV-quant-vs-MTP-buffer finding in sweeps/multi-gpu.md.
* RX 7900 XTX Linux/ROCm 10 row: unsloth Qwen3.8-27B-UD-Q4_K_M.gguf (16,464,440,224 B, sha256 322e194f…23482), 131K context, q4_0 K/V cache, flash attention on, all 66 layers on the GPU, and --parallel 1 both arms. RX 7900 XTX (gfx1100), Ryzen 7 7700X, Ubuntu 24.04.4 / kernel 6.8.0-138, AMDGPU DKMS 7.1.3, ROCm 10.0; repo-local HIP llama.cpp 62acc89. Method: unchanged probe.py at 668cb10, one warmup then three runs × three prompts, thinking off; table figure is the overall median of the nine measured requests. Baseline 36.3 tok/s; MTP adds only --spec-type draft-mtp --spec-draft-n-max 2 (no p-min flag) and reaches 62.6 tok/s (+72.5%). VRAM 18.06 GiB baseline / 19.41 GiB N=2; GTT stayed 45 MiB. Acceptance 1,543/1,920 = 0.804, range 0.56–0.94. Full N=2/3/4 sweep below. A separate fixed-seed local check produced deterministic answers within each arm but different baseline/MTP wording, so this is a throughput observation, not an identical-output speed claim.
* RTX 5090 b10680 row: unsloth UD-Q4_K_XL Dynamic 3.0 (Qwen3.8-27B-UD-Q4_K_XL.gguf, 17,559,178,144 B, sha256 3f227079003add2511437e5b1e94812e363385225bf6a9b47b0054a72bc8b01e), 131K context, q4_0 K/V cache, full GPU offload, flash attention, and --parallel 1 on both arms. llama.cpp CUDA build 10680 (d7bd3bfca, image digest sha256:952424b09abc…90c7), Ubuntu 24.04.4 / kernel 6.8.0-138, NVIDIA driver 595.84, RTX 5090 at a 575 W limit. Loaded VRAM was 19,496 MiB baseline / 21,594 MiB at n-max 7. Method: unchanged probe.py at 668cb10f, three complete passes per arm, each pass three runs x three prompts, thinking off; the row is the median of the three pass medians. Baseline passes: 71.9/71.8/71.9; n-max 7 passes: 170.3/158.4/180.3 (+136.9%, 2.37x). Only the MTP arm added --spec-type draft-mtp --spec-draft-n-max 7. Acceptance excludes warmups: 5,899/12,386 aggregate (0.476), per-request 0.19-0.85; code 0.769, prose 0.230, bash 0.479. Full depth sweep in sweeps/rtx-5090.md: n-max 6 and 7 are tied within 0.5%, and n-max 8 turns down.
* Radeon 780M row: GMKtec NucBox K12, AMD Ryzen 7 / Radeon 780M Graphics (Phoenix, 12 CUs RDNA3), 32 GB UMA, unsloth UD-Q4_K_XL (17.9 GB), 131K context, q4_0 KV cache, llama.cpp b10354 (d2f83055d), Windows 11, official AMD Vulkan driver 32.0.31041.1004, --parallel 1, flash-attn on. Method: unchanged probe.py at commit 431bf8a, three runs x three prompts, thinking off, warmup discarded. Baseline 4.1 tok/s (4.1/4.1/4.1); MTP n-max 2 reaches 8.4 tok/s (+105%, code 9.1 / prose 6.5 / bash 8.4). Acceptance 0.50-0.92 (0.77 aggregate, 1,599/2,076). Full n-max sweep (2/3/4) in sweeps/apu-igpu.md.
* R9700 Windows/Vulkan row: first Windows stack for this card; the existing @ajnytebot row is Linux Vulkan/RADV. AMD Radeon AI PRO R9700 32GB (DEV_7551) on Windows 11 Pro build 26200, AMD driver 32.0.22042.14002, host Ryzen 9 5900XT 16C/32T + 32 GB RAM. unsloth UD-Q4_K_XL Dynamic 3.0 (Qwen3.8-27B-UD-Q4_K_XL.gguf, 17,559,178,144 B, sha256 3f227079…8b01e), 262K context, q4_0 K/V cache, flash attention on, --parallel 1, -t 16, official ggml.llamacpp WinGet b10711 (9723942ad, Clang 20.1.8, win-vulkan-x64). Method: unchanged probe.py at 431bf8a, three runs x three prompts, thinking off, warmup discarded. Harbor (llamaswap ROCm gfx1201, ollama ROCm, hermes, webui) and snap Ollama were stopped for the run; Firefox/Discord closed. Pre-serve adapter dedicated ~0.90 GiB; DWM remained at ~1.24 GiB. Dedicated process VRAM: 20.50 GiB baseline / 22.75 GiB at n-max 4. Row is the n-max 3 arm: 29.7 → 56.3 (+90%), per-prompt 29.8/29.8/29.7 baseline vs 67.2/41.5/56.3 with flag (code/prose/bash), acceptance 0.72 aggregate (1863/2593), range 0.39-0.97. --spec-draft-p-min 0.60 at n-max 3 is a wash on speed (still 56.3) while acceptance rises to 0.84 — rule 2's fast-card shape; it does not promote n-max 4 the way it did on the 2×9070 pool. Full n-max and p-min sweep in sweeps/radeon.md.
* 2× RTX 5090 UD-Q4_K_M row: two RTX 5090 32GB cards, Unsloth Qwen3.8-27B UD-Q4_K_M (Qwen3.8-27B-UD-Q4_K_M.gguf, 16,464,440,224 B, sha256 322e194f…23482), 131K context (n_ctx_slot = 131072), q4_0 K/V cache, full offload, flash attention, --split-mode tensor --tensor-split 1,1, and --parallel 1 on both arms. llama.cpp master 95ef7fc built for SM120 with CUDA/NCCL, Ubuntu 24.04.4 / kernel 7.0.0-30, NVIDIA 610.43.02; the cards are PCIe 4.0 x16 with NODE topology and NCCL logged P2P/direct pointer. Peak monitored VRAM was 10,258 MiB/GPU baseline and 11,200 MiB/GPU at n-max 3. Method: unchanged probe.py at b21edce, three runs x three prompts, thinking off, warmup discarded. Baseline was 107.5 tok/s; n-max 2/3/4 reached 168.2/191.9/180.7 tok/s, so the row uses n-max 3 (+78.5%, 1.79x). Only the MTP arm added --spec-type draft-mtp --spec-draft-n-max 3. Acceptance was 1,705/2,374 = 0.718 aggregate, range 0.46-0.93. Deterministic arithmetic, executable Python, and 16K retrieval fixtures were byte-identical across both arms.
* AMD Instinct MI210 row: the table's first AMD datacenter / CDNA entry — every other AMD row here is RDNA or an RDNA APU. One MI210 64 GB HBM2e (gfx90a, CDNA2), ROCm 6.2.0-66, headless server with no compositor, so rule 7's desktop tax does not apply. unsloth Qwen3.8-27B-Q8_0.gguf (29,047,086,048 B, sha256 a680f44a06920e5d689774823782006aa3acc8db95750323373b24139b67e348), 32768 context, f16 K/V cache (64 GB has the headroom, so no quantized-KV confound), -ngl 99, -fa on, -b/-ub 2048, -t 8 -tb 8, --parallel 1 both arms, thinking off. VRAM 28.65 GiB baseline / 30.90 GiB at n-max 8. Not a stock build, and that is load-bearing for the absolute numbers: a private llama.cpp fork ("frozen-v9 + champion" lineage) at champion tip 9e18beb0. Both arms are the same binary and the same serving config so the delta is clean, but master will not reproduce these figures (rule 6). Build note for other AMD users: -funsafe-math-optimizations was removed from this build on 2026-09-01 after it was demonstrated to flip greedy argmax on gfx90a on 6 of 8 test prompts (upstream #26696) — these are correct-math numbers, and anyone compiling llama.cpp for CDNA with fast-math may be getting silently different outputs. Method: unchanged probe.py at repo HEAD 431bf8a821, one warmup discarded then three runs x three prompts, thinking off; the row is probe.py's overall median of the nine measured requests (mean 52.0). Only the MTP arm adds --spec-type draft-mtp --spec-draft-n-max 8. Acceptance 299/797 = 0.375 aggregate, mean accepted draft length 3.99. n-max 2 on the same config measured 40.3 (acceptance 0.880, mean len 2.76), so n-max 8 wins by 16% here — against this table's trend, where n-max 8 is a loser nearly everywhere it has been swept. Prompt dependence is extreme at that depth: python 74.6 (+145%), bash 46.8 (+54%), prose 30.5 (+0%, dead even with baseline). Per-run spread at n-max 8 was P1 [74.6, 79.5, 72.0], P2 [31.1, 30.5, 29.7], P3 [58.2, 46.8, 45.9] — P3 carries one ~25% excursion, disclosed because the A5000 row declined n-max 8 on spread grounds; the baseline arm was tight across all nine runs (30.0-30.6). A second, non-MTP drafting path in the same fork (a block drafter with a separate draft model, not this repo's flag) reached 61.1 on the identical config and instrument. Carried caveat: on this platform a greedy-vs-baseline divergence exists in the shared speculative verify path — it affects all speculation modes including plain n-gram, is not attributable to that drafter, and is under investigation — so every figure here is an honest throughput measurement, not an identical-output claim. Full sweep in sweeps/instinct-cdna.md.
Every contributor sweep and study is in sweeps/, grouped by card family and PR-able like the table: RTX 5090 · RTX 3090 · Radeon · APUs and iGPUs · Multi-GPU · Workstation · RTX 5060 Ti · RTX 4060 Ti · Apple Silicon · CMP 170HX · CMP 50HX · Instinct (CDNA)
Apache-2.0. The numbers and verdicts are real, the conclusions are mine.