docs(vllm): Qwen3.8-27B DFlash drafter recipe — 72.2 tok/s on Arc Pro B70 - #620
docs(vllm): Qwen3.8-27B DFlash drafter recipe — 72.2 tok/s on Arc Pro B70#620rmacy wants to merge 3 commits into
Conversation
Adds the complete recipe for achieving 72.2 tok/s median (85.9 peak) on isolated C1 with Qwen3.8-27B FP8 + DSpark speculative decoding on 2x Intel Arc Pro B70 (TP=2). Includes: - Patched vLLM dflash files (kernel readout fix for SpecForge drafters) - Dockerfile for qwen38-fp8-dspark:v8 image - Reference serving script - Full README with quick start, kernel fix explanation, and quality proof Published artifacts: - Drafter: rwmacy/qwen3.8-27b-dflash-drafter-fp8-b70 (HF Hub) - Image: ghcr.io/rmacy/qwen38-fp8-dspark:v8 - Kernel fix PR: inteliodev/vllm#1 - SpecForge XPU port PR: sgl-project/SpecForge#769
|
Seems your container image isn't accessible. I' can't seem to get my own container build to load with similar config to what you suggested, it OOM's when I have speculative decode enabled. Loads fine without. |
|
Ah never-mind found it here - docker pull ghcr.io/rmacy/qwen38-fp8-dspark:v12 |
SpecForge-trained DSpark drafters (e.g. drafter-fp8-v5 for Qwen3.8-27B-FP8) declare architectures: ["DSparkDraftModel"]: the DFlash backbone plus a low-rank Markov bigram-bias head and an optional per-position acceptance-confidence head. Serving one failed at startup: pydantic ValidationError for SpeculativeConfig: Model architectures ['DSparkDraftModel'] are not supported for now. The drafter's auto_map remote code (dspark.py) imports specforge, which is not installed in the image, so vLLM falls back to the config architectures and the registry lookup fails. The missing support cannot be taken from rmacy/vllm main: there DSpark landed as upstream plumbing (Qwen3DSparkModel / qwen3_dspark.py, "dspark" speculative method) on a much newer base than the v0.21.0 patch tree. PR intel#620 (intel/llm-scaler) hot-patches the same support into a running image but references qwen3_dflash.py / registry.py snapshots it does not actually ship, so this ports the full support into the multi-arc patch instead. Changes (new patch hunks, base v0.21.0): - qwen3_dflash.py: VanillaMarkov (markov_w1 embedding + markov_w2 projection), AcceptRatePredictor (confidence head), and DSparkDraftModel(DFlashQwen3ForCausalLM) constructing the heads from the draft config (markov_rank, enable_confidence_head, confidence_head_with_markov), exposing compute_markov_bias(), and loading markov_head.*/confidence_head.* checkpoint keys alongside the DFlash backbone path. - registry.py: register "DSparkDraftModel" so ModelConfig creation and SpeculativeConfig.update_arch_ resolve it. - eagle.py: the dflash arch rewrite now passes DSpark*-prefixed architectures through unchanged (otherwise the EAGLEConfig wrap mangles "DSparkDraftModel" into "DFlashDSparkDraftModel"). - dflash.py: DSpark proposer logic from PR intel#620 (verified compatible with our llm_base_proposer signatures): propose() captures the accepted bonus token as the Markov previous-token context; _greedy_sample applies the full-vocab bigram bias semi-autoregressively across block positions (position i conditioned on position i-1's biased prediction) and optionally evaluates the confidence head; _truncate_by_confidence enables adaptive draft block length via DSPARK_ADAPTIVE_BLOCK=1 (off by default). Not ported from PR intel#620's snapshots: the gpu_model_runner.py MTP long-prompt-skip tweak (orthogonal MTP feature) and a comment-only utils.py change. The list-type draft_token_ids padding the adaptive path needs is already present (1a03357). Verified: git apply --check passes on pristine v0.21.0; patch grew 22859 -> 23249 lines, diff limited to the four files above. The "Unable to load dspark.DSparkDraftModel ... No module named 'specforge'" startup warnings are expected and harmless: remote-code loading fails, then the in-tree class resolves via the registry. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Mikko Pesonen <mikko.pesonen@kotiverkko.net>
Independent replication evidence: dual Arc Pro B70I reproduced this recipe on a bare-metal host with 2× Intel Arc Pro B70 GPUs. Configuration
PR-style short benchmarkThe probe used temperature 0, an approximately zero-prefill prompt (
This closely reproduces the PR's reported 72.2 tok/s median. Sustained acceptance-friendly decodeI generated 512 tokens per request from the same deterministic sequence prompt.
The three runs produced 1,536 target tokens. vLLM reported 1,233 accepted draft tokens from 1,248 proposed draft tokens, or 98.8% draft-token acceptance. Acceptance-dependent natural textThree 512-token technical prompts measured 38.43, 40.98, and 53.00 tok/s. Therefore, the 71–112 tok/s results are valid for acceptance-friendly decoding. They are not representative of every natural-text request. Correctness and stability
Host topology noteThe published The working deployment required the no-P2P oneCCL path: All model, drafter, batching, dtype, and speculative-decoding settings remained aligned with the recipe. BF16 short-prompt edge caseA raw completion containing only one prompt token entered the ESIMD decode kernel before drafting and failed with: Multi-token completion prompts and chat-template prompts passed. Setting the target dtype to FP16 avoided this edge case, but BF16 was retained for the reported recipe-aligned benchmark. |
4 participants
Summary
Adds the complete recipe for achieving 72.2 tok/s median (85.9 peak) on isolated C1 with Qwen3.8-27B FP8 + DSpark speculative decoding on 2× Intel Arc Pro B70 (TP=2).
Contents
vllm/patches/qwen38-dflash/: patched vLLM dflash files (kernel readout fix), Dockerfile, serve script, READMEqwen38-fp8-dspark:v8(also published atghcr.io/rmacy/qwen38-fp8-dspark:v8)Published artifacts
ghcr.io/rmacy/qwen38-fp8-dspark:v8Quality
Greedy spec decode is lossless by construction. Verified 4/5 byte-identical outputs vs target-only baseline; cross-checked against independent bf16 reference endpoint. Zero drafter-induced quality degradation.