Support Nemotron 3.5 Lightning Q8 GGUF conversion - #497
Closed
justinchuby wants to merge 1 commit into
Closed
Conversation
6 tasks
Performance Comparison
|
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
justinchuby
force-pushed
the
justinchuby-add-nemotron-35-lightning
branch
from
August 15, 2026 05:38
cfbbc7a to
ae4fd70
Compare
justinchuby
force-pushed
the
justinchuby-support-nemotron-gguf
branch
2 times, most recently
from
August 15, 2026 06:55
9e6ffbc to
2e31b58
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a GGUF architecture-adapter framework and a strict Nemotron 3.5 Lightning (nemotron_h_moe) adapter to enable exact Q8_0 preservation during GGUF → ONNX conversion, along with tokenizer reconstruction and direct-ORT validation support.
Changes:
- Introduces a reusable GGUF architecture adapter interface/registry and implements a strict Nemotron adapter that validates the pinned tensor table, schedules, qtypes, and mapping completeness.
- Extends NemotronH model construction to support quantized projections/embeddings (MatMulNBits + GatherBlockQuantized) and adds repacking support for stacked expert tensors.
- Adds strict GPT-2/Pixtral tokenizer reconstruction, broader test coverage for the adapter/tokenizer/repacker paths, and updates docs + an acceptance runner script.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/mobius/models/nemotron_h.py | Adds quantized linear/embedding wiring for NemotronH graphs. |
| src/mobius/models/nemotron_h_test.py | Adds unit coverage for float vs quantized NemotronH graph contracts and initializer coverage validation. |
| src/mobius/integrations/gguf/_tokenizer.py | Adds strict GPT-2/Pixtral tokenizer reconstruction for Nemotron GGUF metadata. |
| src/mobius/integrations/gguf/_tokenizer_test.py | Adds tests for strict Pixtral/GPT-2 reconstruction and pinned-artifact parity hooks. |
| src/mobius/integrations/gguf/_repacker.py | Adds repacking for leading-axis stacked 3D quantized tensors (expert stacks). |
| src/mobius/integrations/gguf/_repacker_test.py | Adds execution-level test validating stacked-expert repack output matches dequantized matmul. |
| src/mobius/integrations/gguf/_nemotron_h_moe.py | New strict Nemotron 3.5 Lightning adapter: validation, config extraction, mapping, and transforms. |
| src/mobius/integrations/gguf/_mmproj_test.py | Updates multimodal preflight expectation to reflect adapter validation errors. |
| src/mobius/integrations/gguf/_config_mapping.py | Allows config extraction via adapter (validated) before generic mapping. |
| src/mobius/integrations/gguf/_builder.py | Routes GGUF build flow through adapters (validation, quant config, mapping-based quant detection/state_dict load). |
| src/mobius/integrations/gguf/_builder_test.py | Updates preflight guard tests now that Nemotron adapter is supported. |
| src/mobius/integrations/gguf/_architecture.py | New adapter registry + mapping audit + package state-dict coverage checker. |
| src/mobius/integrations/gguf/_architecture_test.py | Adds synthetic + optional real-artifact integration validation for the Nemotron adapter. |
| src/mobius/components/_mamba_block.py | Adds linear_class injection point so Mamba blocks can use quantized linears. |
| examples/olive/nemotron-3_5-lightning-30b/validate_gguf_q8.py | Adds a reproducible build/save/reload/direct-ORT acceptance runner for pinned Q8_0. |
| docs/cli_reference.md | Updates CLI docs to reflect supported pinned Nemotron Q8_0 GGUF import. |
| docs/api/build_from_gguf.md | Replaces the prior waiver section with pinned Q8_0 support + reproducibility details. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
justinchuby
force-pushed
the
justinchuby-support-nemotron-gguf
branch
from
August 15, 2026 11:11
2e31b58 to
4e9ab2e
Compare
justinchuby
force-pushed
the
justinchuby-add-nemotron-35-lightning
branch
from
August 20, 2026 01:01
0722ddb to
8ccfc25
Compare
Add a reusable GGUF architecture-adapter seam and a strict nemotron_h_moe adapter that validates the pinned 52-layer backbone, excludes the auxiliary MTP block, and enforces complete source-to-initializer mapping. Preserve Q8_0 weights in MatMulNBits, including stacked routed experts, reconstruct the pinned Pixtral tokenizer contract, and add a fresh-process direct-ORT acceptance runner with exact llama.cpp generation evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
justinchuby
force-pushed
the
justinchuby-support-nemotron-gguf
branch
from
August 20, 2026 01:36
4e9ab2e to
b0b94f9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nemotron_h_moeowns its schedule, tensor names, transforms, qtype policy, tokenizer contract, and MTP semanticsMatMulNBits(bits=8, block_size=32), including leading-axis routed-expert expansion without dequantizationStack and lineage
This PR is linearly restacked on exact #487 head
8ccfc25977d9551c7dfc07088eea1827a5fcb8b8(itself based on current maincf77d0f7fb28adebbeb3aaecb25fe2e51ee0ddd0). Exact PR head isb0b94f92cd39aea883bdbc008a3ff3ff31b56121;8ccfc259...HEADcontains exactly one commit,Support Nemotron 3.5 Q8 GGUF conversion.The parent inherits merged #500 skills once and retains #498 once. This PR has no
.agents/or.github/delta and does not duplicate either dependency. The only rebase adaptation updates a test import to the currentmobius.integrations.transformerslocation introduced by the ecosystem refactor.Exact architecture and mapping contract
The adapter validates the complete pinned tensor table before graph construction:
ssm_a -> log(-A), A/D/grouped-norm squeeze/flatten, conv-axis expansion, inverse llama.cpp Q/K row permutation, and direct expert-axis slicing are shape-validatedAll Nemotron constants and error prose live in
src/mobius/integrations/gguf/_nemotron_h_moe.py; model/component factory changes are source-format agnostic and the existing float/Safetensors graph remains unchanged. Every registered adapter must implement source validation, and generic validation dispatch is explicitly typed.Pinned real-artifact acceptance
Source:
unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF@f2d3fe3694501008786e81e5f20360cbf715496aNVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q8_0.ggufdc5276dd0619c04e277504d2358a793e31ccbe39e894d767d0d14f2a221e2ca4The full build/save/reload/prefill/cached-decode acceptance was rerun from exact head
b0b94f92cd39with this worktree first onPYTHONPATHand an isolated approved-feed ONNX Runtime 1.28.0 install:MatMulNBits+ 1GatherBlockQuantizedQuantizeLinearorDequantizeLinearA fresh process loaded the saved external-data package in ONNX Runtime 1.28.0 (
45de2a8b06) onCPUExecutionProvider, ran nonzero unpadded prefill, independently ran right-padded prefill with an explicit mask, and executed every cached decode step:0.0(atol=1e-5)Independent llama.cpp reference: commit
9d57ce456c94d241dde672b2db9cf18879766568, MSVC 19.44.35228.0, CPU only:[1784, 8961, 1307, 5498, 1395][6993, 1046, 1256, 1010, 1784, 8961, 1307, 10787]Paris. \nThe capital of GermanyThe exact-head direct-ORT run matched every generated ID and the decoded text exactly.
Tokenizer and special-token contract
The GGUF tokenizer is selected only for exact
model=gpt2, pre=pixtralmetadata and matches the pinned official tokenizer atnvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16@d468880b6ad3c6e0d21377ce7242adaea4cc884dacross vocabulary, added-token flags, pre-tokenizer, decoder, post-processor, encode IDs, and decode text.4999709474e3c967358c1f1199b6be65fb9055d3eb59e0cd387f9e7077fc40edb1b0165185b1925118c2f7b1e978439b02010c3a420ebfec5c19a093a0d9b4cb623c34567aebb18582765289fbe23d901c62704d6518d71866e0e58db892b5b71/11/9991/2/0[2, 11]<|im_end|>ID 11PAD 999 is rejected because it names
<SPECIAL_999>. The GGUF-embedded chat template differs from the pinned official asset, so the recipe verifies immutable official sidecar hashes rather than silently treating it as authoritative. Refreshed #487 sentinel filtering and EOS/EOT/EOM union semantics remain intact for the generic path.Explicit limits
Validation on exact head
1,692 passed, 47 skipped2 passed4,130 passed, 64 skipped, plus four failures in untouched upstream-asset patch fixtures8ccfc259; excluding only them gives4,130 passed, 64 skippedlintrunner -a: cleangit diff --checkis clean8ccfc259...b0b94f92: no significant findingsWaivers
8ccfc259validates the official model's CPU/CUDA cached generation and fresh Olive Q4 package; this PR specifically validates the preserved source-Q8 ONNX package through direct ORT.