docs(local-llm): add speculative decoding and MTP page

[JacobPEvans-personal]

What

Multi-token prediction was undocumented on the site, despite the Nix serving stack carrying a full option surface for it.

Adds local-llm/speculative-decoding.mdx plus a nav entry, and extends two existing pages.

The new page

• Why decode is bandwidth-bound, and the only two levers that follow: read fewer bytes per token, or get more tokens per read.
• What a trained MTP head changes versus a separate drafter — the draft comes from the target's own learned distribution, so acceptance is structurally higher than a small model trained on different data.
• Drafter/target quantization independence. A 6-bit drafter serves 2/3/4/5/6/8-bit targets, so adopting MTP doesn't mean re-downloading weights. Includes the non-obvious failure mode: over-quantizing a draft head costs acceptance, not answer quality, so the model keeps answering correctly while the speedup quietly disappears — invisible from output alone.
• Draft depth as the tunable, swept per workload, with the code-vs-prose acceptance difference called out.

Two corrections to common misreadings

Concurrency is a coupling, not a cap. Worker batch width and the proxy admission gate must be equal; both accept 1–4; nothing pins MTP to 1. The default of 1 is precisely what makes it look otherwise, so the page says so explicitly in a <Warning>.

A ratio of two runs cannot identify a setting. New section in verifying-the-instrument, which is already the page about procedures that produce answers of the right shape rather than correct ones. The concrete number: the same model serving byte-identical output measured 17.4–27.3 tok/s across runs here — a spread that swallows the effect being attributed. This is why the catalog entry for that model records no tok/s figure at all. Also covers the quiet one: most serving options are read at startup only, so an edited config without a restart benchmarks the old setting and returns a clean null result that looks like evidence.

Also

models-and-quantization gains a short subsection framing MTP as an axis distinct from bit depth, since the two combine freely and the page otherwise implies quantization is the only dimension.

Content boundaries

No internal hostnames, IPs, credentials, or references to private repos. Model IDs cited are public Hugging Face repos. Hardware measurements are described as "this estate" without identifying infrastructure.

Verification

⚠️ Not run — no Node toolchain in this environment. Please run before merge:

nix develop
mint dev            # visual check, incl. the <Steps> and <Warning> blocks
mint broken-links

docs.json was validated as parseable JSON, and all four internal link targets in the new page were confirmed to exist as files. Longest prose line is well under the 160-char MD013 limit.

Provenance

Triage of a Reddit thread about running Qwen3.8-27B faster. Most of its advice was already implemented in the estate — often with better data — but the gaps it exposed were worth writing down. The companion code fix is dryvist/nix-ai#1835, which found that the MTP option surface was unreachable.

---

Generated by Claude Code

---

Note

Low Risk
Documentation-only changes with no runtime, auth, or data-path impact.

Overview
Documents multi-token prediction (MTP) and speculative decoding for the local LLM / Nix serving stack, which previously had options but no site coverage.

Adds local-llm/speculative-decoding.mdx (nav entry in docs.json): bandwidth-bound decode, MTP vs a separate drafter, independent drafter/target quants (including the “over-quant the draft → silent loss of speedup” trap), draft-depth tuning, and that maxNumSeqs and proxy concurrency must match (1–4), not that MTP is stuck at 1. Includes a short evaluation workflow and warns against inferring settings from a two-run throughput ratio.

models-and-quantization gets a subsection framing MTP as an axis separate from bit depth. verifying-the-instrument adds the same ratio fallacy (report spreads, read config/CMD line, restart between arms) and a related card linking to the new page.

^{Reviewed by Cursor Bugbot for commit b3b4e39. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_a6152e84-8896-4ab4-8f94-6c64864a0c61)

[gemini-code-assist]

Code Review

This pull request introduces a new documentation page on speculative decoding and multi-token prediction (MTP), detailing its performance characteristics, configuration, and evaluation. It also updates existing documentation to link to this new page and adds a section warning against benchmarking pitfalls. The review feedback correctly identifies potential typos in the model names and Hugging Face repository identifiers (e.g., 'Qwen 3.8' and 'Qwen3.8-27B') that should be verified and corrected.