Add NVIDIA Nemotron 3.5 Lightning export support - #487
Open
justinchuby wants to merge 13 commits into
Open
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Performance Comparison
|
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
6 tasks
Contributor
There was a problem hiding this comment.
Pull request overview
Adds export + validation support for the official NVIDIA Nemotron 3.5 Lightning (nemotron_h) BF16 checkpoint by pinning the upstream revision, normalizing config/weight handling for the current Transformers vocabulary, and introducing reduced-real-weight L4/L5 evidence plus runtime/ORT GenAI guardrails.
Changes:
- Normalize NemotronH config + cache dtype behavior (layer-type vocabularies, SSM cache dtype, pinned revisions/dtypes in validation paths).
- Update NemotronH weight alignment and MoE routing numerics (preserve fp32 router bias / recurrent state precision; ignore only
mtp.*auxiliary heads). - Add reduced-real-weight fixtures + tests, plus an Olive/ORT-direct example pipeline; reject unsupported ORT GenAI mixed-cache export earlier (CLI + config writer).
Reviewed changes
Copilot reviewed 34 out of 34 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/weight_alignment_test.py | Adds NemotronH-specific assertions for mtp.* filtering and per-expert weight name mapping. |
| tests/synthetic_parity_test.py | Updates NemotronH HF layer-type vocabulary mapping and removes the prior NemotronH xfail. |
| tests/nemotron_h_real_weight_test.py | New reduced-real-weight L4/L5 integration tests comparing ONNX vs HF plus golden checks. |
| tests/model_coverage_test.py | Removes NemotronH “no test model id” skip now that a checkpoint is pinned. |
| tests/cli_test.py | Ensures --runtime ort-genai rejects NemotronH before build/weight download. |
| tests/build_graph_test.py | Adds tests for fp32 preservation (router bias + SSM cache) and BF16 rejection. |
| tests/arch_validation_test.py | Forwards YAML-pinned revisions into config load and applies YAML dtype for NemotronH. |
| tests/_test_configs.py | Updates NemotronH test configs (adds explicit rms_norm_eps). |
| testdata/golden/causal-lm/nemotron-3_5-lightning-30b-reduced.json | New reduced-real L4 golden reference (top-k IDs/logits + summary). |
| testdata/golden/causal-lm/nemotron-3_5-lightning-30b-reduced_generation.json | New reduced-real L5 greedy generation golden reference. |
| testdata/cases/causal-lm/nemotron-3_5-lightning-30b.yaml | New pinned YAML case documenting revision/dtype and CI skip rationale. |
| src/mobius/tasks/_cache_utils.py | Allows NemotronH SSM cache inputs to use explicit cache dtype (fp32 by default). |
| src/mobius/models/nemotron_h.py | Implements mtp.* filtering, bf16 rejection, and fp32-sensitive MoE routing details. |
| src/mobius/integrations/ort_genai/auto_export.py | Adds early model-type rejection for known unsupported ORT GenAI contracts (NemotronH). |
| src/mobius/integrations/ort_genai/auto_export_test.py | Verifies NemotronH is rejected before writing ORT GenAI artifacts. |
| src/mobius/_registry.py | Updates NemotronH test model id to the official 3.5 Lightning BF16 checkpoint. |
| src/mobius/_configs/_base.py | Adds NemotronH layer-type normalization, stricter validation, and SSM cache dtype config. |
| src/mobius/_configs_test.py | Adds coverage for NemotronH layer-type vocabulary normalization and unknown-type rejection. |
| src/mobius/main.py | Adds CLI preflight rejection for ORT GenAI runtime before build/weight download. |
| examples/olive/nemotron-3_5-lightning-30b/validate_reduced_checkpoint.py | New script to build/validate reduced-real-weight packages (incl. range-fetch). |
| examples/olive/nemotron-3_5-lightning-30b/requirements.txt | Example-only deps for Olive workflow + range-reading. |
| examples/olive/nemotron-3_5-lightning-30b/README.md | Documents Option A flow, constraints, reduced-real evidence, and usage. |
| examples/olive/nemotron-3_5-lightning-30b/optimize.py | Exports pinned FP16 ONNX and runs CPU-isolated Olive INT4 quantization + smoke test. |
| examples/olive/nemotron-3_5-lightning-30b/olive_q4.json | Example Olive workflow JSON for Q4 K-quant. |
| examples/olive/nemotron-3_5-lightning-30b/inference.py | Direct ORT token-by-token generation loop for hybrid caches. |
| examples/olive/nemotron-3_5-lightning-30b/.gitignore | Ignores example cache/output directories. |
| .github/workflows/gpu_l5_generation_e2e.yml | Includes new NemotronH reduced-real-weight generation test in GPU L5 job. |
| .github/workflows/gpu_l4_golden_parity.yml | Includes new NemotronH reduced-real-weight golden test in GPU L4 job. |
| .agents/skills/writing-tests/SKILL.md | Codifies reduced-real-weight fixture guidance and warns about stale layer-type mappings. |
| .agents/skills/ort-genai-config/SKILL.md | Documents NemotronH mixed-cache ORT GenAI incompatibility and required early rejection. |
| .agents/skills/onnx-export-quantization/SKILL.md | Adds guidance on isolating Olive from unrelated EP DLL registration. |
| .agents/skills/moe-models/SKILL.md | Adds NemotronH config vocabulary + reduced-precision routing and mtp.* filtering guidance. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
🏗️ Architecture Diff
qwen3_5_vl (hybrid-qwen-vl) / embedding — 18 change(s)Op summary: 32 → 47 nodes --- base
+++ head
@@ -1,14 +1,29 @@
Gather
Constant
Equal
+Equal
+Not
+Or
Unsqueeze
+Reshape
+Cast
+Reshape
Cast
Constant
CumSum
Constant
Sub
Constant
+CumSum
+Constant
+Sub
+ReduceSum
+Add
+Where
+Constant
Clip
+Shape
+Reshape
CastLike
Constant
ShapeAdded nodes:
Modified attributes:
Connectivity changes:
qwen3_5_vl (hybrid-qwen-vl) / vision_encoder — 435 change(s)Op summary: 247 → 271 nodes --- base
+++ head
@@ -1,134 +1,233 @@
+Cast
Reshape
Conv
Reshape
+Constant
+Gather
+Constant
+Gather
+Constant
+Gather
+Mul
+Mul
+Constant
+CumSum
+Constant
+Constant
+Pad
+ReduceSum
+Constant
+Constant
+Range
Slice
-Squeeze
-Slice
-Squeeze
-Slice
-Squeeze
-Mul
-Mul
-ReduceMax
-Scan
Shape
-Constant
-Gather
-Constant
-Squeeze
+ConstantOfShape
+Constant
+Mul
+ScatterElements
+Constant
+CumSum
+Gather
+Sub
+Gather
+Gather
+Mul
+Mod
+Constant
+Div
+Mod
+Constant
+Div
+Div
+Mod
+Constant
+Div
+Constant
+Mod
+Constant
+Mul
+Add
+Constant
+Mul
+Add
+Cast
+Cast
+Cast
+Cast
+Mul
+Sub
+Div
+Mul
+Sub
+Div
+Floor
+Floor
+Cast
+Cast
+Constant
+Constant
+Add
+Min
+Constant
+Add
+Min
+Sub
+Unsqueeze
+Sub
+Unsqueeze
+Constant
+Mul
+Constant
+Mul
+Add
+Add
+Add
+Add
+Cast
+Gather
+Gather
+Gather
+Gather
+Sub
+Mul
+Add
+Sub
+Mul
+Add
+Sub
+Mul
+Add
+CastLike
+Add
+Unsqueeze
+Unsqueeze
+Concat
+Gather
+Gather
+Squeeze
+Squeeze
+Gather
+Gather
+Gather
+Gather
+Concat
+Concat
+Constant
+Equal
+Compress
+Unsqueeze
+Concat
+LayerNormalization
+Transpose
+MatMul
+Add
+Split
+Reshape
+Reshape
+CastLike
+CastLike
+Unsqueeze
+Unsqueeze
+Split
+Split
+Mul
+Mul
+Sub
+Mul
+Mul
+Add
+Concat
+Mul
+Mul
+Sub
+Mul
+Mul
+Add
+Concat
+Reshape
+Reshape
+Shape
+Squeeze
+Constant
Constant
Range
Unsqueeze
Unsqueeze
-Less
+GreaterOrEqual
+Cast
+ReduceSum
+Constant
+Sub
+Unsqueeze
+Unsqueeze
+Equal
+CastLike
+CastLike
+Where
+Unsqueeze
+Unsqueeze
+Unsqueeze
+Unsqueeze
+Attention
+Squeeze
+Transpose
+MatMul
+Add
+Add
+LayerNormalization
+Transpose
+MatMul
+Add
+Gelu
+Transpose
+MatMul
+Add
+Add
+Reshape
+LayerNormalization
+Transpose
+MatMul
+Add
+Gelu
+Transpose
+MatMul
+Add
+LayerNormalization
+Transpose
+MatMul
+Add
+Split
+Reshape
+Reshape
+CastLike
+CastLike
+Unsqueeze
+Unsqueeze
+Split
+Split
+Mul
+Mul
+Sub
+Mul
+Mul
+Add
+Concat
+Mul
+Mul
+Sub
+Mul
+Mul
+Add
+Concat
+Reshape
Reshape
Shape
-Slice
-Constant
-Concat
-Reshape
-Compress
-CastLike
-Add
-Slice
-Squeeze
-Slice
-Squeeze
-Slice
-Squeeze
-Mul
-Mul
-ReduceMax
-Scan
-Shape
-Constant
-Gather
-Constant
-Squeeze
+Squeeze
+Constant
Constant
Range
Unsqueeze
Unsqueeze
-Less
-Reshape
-Shape
-Slice
-Constant
-Concat
-Reshape
-Compress
-Gather
-Gather
-Squeeze
-Squeeze
-Gather
-Gather
-Gather
-Gather
-Concat
-Concat
-Slice
-Squeeze
-ReduceMax
-Scan
-Shape
-Constant
-Gather
-Constant
-Squeeze
-Constant
-Range
-Unsqueeze
-Unsqueeze
-Less
-Reshape
-Shape
-Slice
-Constant
-Concat
-Reshape
-Compress
-Constant
-CumSum
-Constant
-Constant
-Pad
-LayerNormalization
-Transpose
-MatMul
-Add
-Split
-Reshape
-Reshape
-CastLike
-CastLike
-Unsqueeze
-Unsqueeze
-Split
-Split
-Mul
-Mul
-Sub
-Mul
-Mul
-Add
-Concat
-Mul
-Mul
-Sub
-Mul
-Mul
-Add
-Concat
-Reshape
-Reshape
-Shape
-Squeeze
-Constant
-Constant
-Range
-Unsqueeze
-Unsqueeze
GreaterOrEqual
Cast
ReduceSum
@@ -159,89 +258,14 @@
MatMul
Add
Add
-Reshape
-LayerNormalization
-Transpose
-MatMul
-Add
-Gelu
-Transpose
-MatMul
-Add
-LayerNormalization
-Transpose
-MatMul
-Add
-Split
-Reshape
-Reshape
-CastLike
-CastLike
-Unsqueeze
-Unsqueeze
-Split
-Split
-Mul
-Mul
-Sub
-Mul
-Mul
-Add
-Concat
-Mul
-Mul
-Sub
-Mul
-Mul
-Add
-Concat
-Reshape
-Reshape
-Shape
-Squeeze
-Constant
-Constant
-Range
-Unsqueeze
-Unsqueeze
-GreaterOrEqual
-Cast
-ReduceSum
-Constant
-Sub
-Unsqueeze
-Unsqueeze
-Equal
-CastLike
-CastLike
-Where
-Unsqueeze
-Unsqueeze
-Unsqueeze
-Unsqueeze
-Attention
-Squeeze
-Transpose
-MatMul
-Add
-Add
-LayerNormalization
-Transpose
-MatMul
-Add
-Gelu
-Transpose
-MatMul
-Add
-Add
-LayerNormalization
-Reshape
-Transpose
-MatMul
-Add
-Gelu
-Transpose
-MatMul
-Add
-Unsqueeze
-Concat
+LayerNormalization
+Reshape
+Transpose
+MatMul
+Add
+Gelu
+Transpose
+MatMul
+Add
+Unsqueeze
+ConcatAdded nodes:
Removed nodes:
Modified attributes:
Connectivity changes:
Initializer changes:
Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
justinchuby
commented
Aug 14, 2026
justinchuby
commented
Aug 14, 2026
justinchuby
commented
Aug 14, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 34 out of 34 changed files in this pull request and generated no new comments.
Suppressed comments (3)
examples/olive/nemotron-3_5-lightning-30b/inference.py:207
OnnxKQuantQuantizationcan rename the graph output tologits_Q4(the repository already handles this inexamples/qwen3_6_35b_a3b_ort_genai.py:574-581andexamples/gemma4_unified_ort_genai.py:222-239). Becausequantize_package()passes that Olive output directly to this function, indexing only"logits"can raiseValueError, preventing the advertised post-quantization smoke test and reduced Q4 generation from running. Accept both output names or normalize the graph output after quantization.
logits = _as_numpy(outputs[output_names.index("logits")])[0, -1].astype(np.float32)
src/mobius/main.py:345
- This direct
AutoConfigcall bypasses the supported raw-config.jsonfallback inbuild()(src/mobius/_builder.py:542-552). Registered models whose config type is not recognized by the installed Transformers version will now fail only when--runtime ort-genaiis requested, before either the compatibility validator or the normal builder runs. Resolve the model type through the same fallback path (ideally a shared helper) so the new preflight does not regress those models.
runtime_config = transformers.AutoConfig.from_pretrained(
model_id_or_path,
trust_remote_code=trust_remote_code,
)
src/mobius/integrations/ort_genai/auto_export.py:1187
- This actionable error points Nemotron 3.5 users to the older Nano-specific script, whose default checkpoint and compatibility patches differ from the package introduced by this PR. Point to the new direct-runtime example so users follow the pinned 3.5 export and inference path described here.
"Runtime (see examples/nemotron_3_nano_text_generation.py)."
justinchuby
force-pushed
the
justinchuby-add-nemotron-35-lightning
branch
3 times, most recently
from
August 15, 2026 02:29
eee55c8 to
fbffb25
Compare
This was referenced Aug 15, 2026
justinchuby
force-pushed
the
justinchuby-add-nemotron-35-lightning
branch
from
August 15, 2026 05:38
cfbbc7a to
ae4fd70
Compare
justinchuby
added a commit
that referenced
this pull request
Aug 15, 2026
## Summary - extract the model-conversion lessons from #487 into a standalone skills-only change - require faithful graph-derived metadata without gating Mobius artifacts on the current ORT GenAI registry, version, topology, or cache executor - clarify cached-decode diagnosis, standard-ONNX controls before blaming Olive, and provider isolation for weight-only quantization - preserve the quantized-by-default and Direct GGUF guidance merged in #496 ## Scope Only the seven intended `SKILL.md` files are changed. Model code, tests, workflows, examples, and `.agents/skills/moe-models/SKILL.md` are intentionally excluded. ## Validation - `python -m pytest tests/yaml_schema_test.py -q --tb=short` (246 passed) - all 19 skill frontmatter blocks parsed and matched their directory names - `lintrunner -a --output oneline` - `git diff --check` - independent code review completed; its optional-runtime waiver contradiction was resolved --------- Signed-off-by: Justin Chu <justinchu@microsoft.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 19, 2026
## Summary - add production support for pinned `Qwen/Qwen3.8-27B@1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0` as a checkpoint alias of the existing dense Qwen3.5 hybrid vision-language architecture - preserve the exact 64-layer 48-DeltaNet/16-full-attention text schedule, 27-layer vision encoder, image/video processor contracts, mixed-batch scatter order, dtype boundaries, and optional separately packaged MTP drafter - preserve canonical `qwen3_5` registry regression coverage on `Qwen/Qwen3.5-2B`; Qwen3.8 uses its own pinned case/config assertions instead of replacing that mapping - emit topology-faithful ORT GenAI metadata without a runtime capability gate: decoder-only packages keep `qwen3_5_text`, while multimodal Qwen3.5/Qwen3.8 packages emit `qwen3_5` - remove the standalone Qwen3.8 Olive example, its coupled reduced-real test module, and its private reduced golden fixtures; core model, processor, metadata, and parity coverage remain unchanged Stacked on #487 final skill-split head `a8cd77570bca980861eeadab9e6fa077464f0138`. Exact PR head: `59af1a51f1a162e9444a9be1ea5785ca4ec0ef0b`. Tracks #483. ## Validation - the real processor from the pinned official revision drives deterministic tiny-model HF/ONNX pipeline parity for image-only, video-only, and a mixed two-row batch with opposite placeholder order: image max abs `0.00697723`, video `0.00411959`, mixed `0.00677243`; cosine is above `0.99999` and final argmax matches - exact raw-config assertions cover the 48/16 hybrid schedule, text/vision dimensions, processor token contracts, and intentional separate MTP packaging - packed-vision coordinates are emitted once and shared by interpolation, rotary, and frame-boundary consumers; media ownership uses boundary `ScatterElements` + `CumSum` in O(patches + media) - deterministic Qwen3.5-VL benchmark: `428 -> 467` top-level nodes (`+9.1%`, model size unchanged), below the unchanged 10% blocker threshold - topology-specific metadata regression coverage generates both package shapes: decoder-only `qwen3_5_text` and multimodal `qwen3_5` - post-removal YAML/coverage/focused suite: `987 passed, 227 skipped`; diff lintrunner clean; independent review found no stale reduced/Olive claims or dangling references - canonical non-integration suite before the deletion-only follow-up: `3962 passed, 61 skipped` ## L4/L5 waiver The pinned official checkpoint is 55.6 GB and exceeds hosted CI storage and GPU memory; no reduced or quantized private fixture is committed. The Qwen3.8 case records this exact `ci_skip_reason` rather than claiming retained executable evidence. Earlier reduced-real and Olive development artifacts were removed and are not part of the final PR validation surface. ## CI Current head `59af1a51f1a162e9444a9be1ea5785ca4ec0ef0b`: - Benchmark run `32314325378` is complete and successful (base, head, and comparison). - CI run `32314325553` is still active. Lint, affected-model detection, L1, L3, and lintrunner have completed successfully; the Linux/Windows test matrices are in progress, while Integration (fast), L4, and L5 are queued. - Architecture Diff run `32314325423` is in progress. Historical run `31870831200` belongs to pre-removal head `067cb489599687b74f96574de20133cb96e016a7`, not the final head. It is retained only as historical context: its three red jobs reproduced on the exact #487 base run `31869640946`, but it is not cited as current-head CI evidence. PR #498 remains ready for maintainer review; auto-merge is off. --------- Signed-off-by: Justin Chu <justinchu@microsoft.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Normalize current and legacy NemotronH configs, preserve routing and recurrent-state precision, and filter only auxiliary MTP weights. Add pinned reduced-real L4/L5 evidence, CUDA/ORT GenAI guards, and an executable Olive Q4 direct-runtime recipe. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Apply the Linux formatter/import order to the Olive scripts and include the pending ModelPackage formatting fixes required by the repository-wide lint job. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Separate unsupported BF16 evidence from the supported validation matrix, add automated FP16 CUDA cached-logit parity, and verify fresh Olive Q4 quantization through final-package CUDA reload and generation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Fail reduced-real validation whenever the maximum absolute prefill or cached-step error exceeds the advertised 1e-2 gate, independent of relative tolerance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Cache test-case discovery, use generic reduced-real GPU workflow discovery, and emit an honest NemotronH GenAI config without a runtime-version-specific hard rejection. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Teach affected-model detection about reduced-real tests and example assets, and stop direct generation on configured EOS without an unused final decoder invocation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Profile only the cached-generation session used for placement evidence so prefill session teardown does not leave untracked ORT profile files. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Reject structurally unrepresentable SSM cache graphs before artifact creation and persist the pinned reduced-real fixture with validated range retries, atomic schema-versioned writes, and shared L4/L5 CI caching. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Set the runner-temp fixture path on the L4/L5 execution steps, where the runner context is available, while retaining the shared cache action path and key. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Remove downstream ORT GenAI capability gates and derive decoder cache/output metadata directly from exported graphs. Use portable standard-ONNX cache operators for validated Nemotron CUDA and Olive generation, including quantized logits handling. Preserve GGUF special-token metadata and update model-development guidance to keep runtime acceptance downstream. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Drop serialized null-token sentinels and out-of-vocabulary special-token IDs while preserving valid BOS and padding metadata. Combine embedded EOS, EOT, and EOM IDs into the generation stop-token contract. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
- add production support for pinned `Qwen/Qwen3.8-27B@1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0` as a checkpoint alias of the existing dense Qwen3.5 hybrid vision-language architecture - preserve the exact 64-layer 48-DeltaNet/16-full-attention text schedule, 27-layer vision encoder, image/video processor contracts, mixed-batch scatter order, dtype boundaries, and optional separately packaged MTP drafter - preserve canonical `qwen3_5` registry regression coverage on `Qwen/Qwen3.5-2B`; Qwen3.8 uses its own pinned case/config assertions instead of replacing that mapping - emit topology-faithful ORT GenAI metadata without a runtime capability gate: decoder-only packages keep `qwen3_5_text`, while multimodal Qwen3.5/Qwen3.8 packages emit `qwen3_5` - remove the standalone Qwen3.8 Olive example, its coupled reduced-real test module, and its private reduced golden fixtures; core model, processor, metadata, and parity coverage remain unchanged Stacked on #487 final skill-split head `a8cd77570bca980861eeadab9e6fa077464f0138`. Exact PR head: `59af1a51f1a162e9444a9be1ea5785ca4ec0ef0b`. Tracks #483. - the real processor from the pinned official revision drives deterministic tiny-model HF/ONNX pipeline parity for image-only, video-only, and a mixed two-row batch with opposite placeholder order: image max abs `0.00697723`, video `0.00411959`, mixed `0.00677243`; cosine is above `0.99999` and final argmax matches - exact raw-config assertions cover the 48/16 hybrid schedule, text/vision dimensions, processor token contracts, and intentional separate MTP packaging - packed-vision coordinates are emitted once and shared by interpolation, rotary, and frame-boundary consumers; media ownership uses boundary `ScatterElements` + `CumSum` in O(patches + media) - deterministic Qwen3.5-VL benchmark: `428 -> 467` top-level nodes (`+9.1%`, model size unchanged), below the unchanged 10% blocker threshold - topology-specific metadata regression coverage generates both package shapes: decoder-only `qwen3_5_text` and multimodal `qwen3_5` - post-removal YAML/coverage/focused suite: `987 passed, 227 skipped`; diff lintrunner clean; independent review found no stale reduced/Olive claims or dangling references - canonical non-integration suite before the deletion-only follow-up: `3962 passed, 61 skipped` The pinned official checkpoint is 55.6 GB and exceeds hosted CI storage and GPU memory; no reduced or quantized private fixture is committed. The Qwen3.8 case records this exact `ci_skip_reason` rather than claiming retained executable evidence. Earlier reduced-real and Olive development artifacts were removed and are not part of the final PR validation surface. Current head `59af1a51f1a162e9444a9be1ea5785ca4ec0ef0b`: - Benchmark run `32314325378` is complete and successful (base, head, and comparison). - CI run `32314325553` is still active. Lint, affected-model detection, L1, L3, and lintrunner have completed successfully; the Linux/Windows test matrices are in progress, while Integration (fast), L4, and L5 are queued. - Architecture Diff run `32314325423` is in progress. Historical run `31870831200` belongs to pre-removal head `067cb489599687b74f96574de20133cb96e016a7`, not the final head. It is retained only as historical context: its three red jobs reproduced on the exact #487 base run `31869640946`, but it is not cited as current-head CI evidence. PR #498 remains ready for maintainer review; auto-merge is off. --------- Signed-off-by: Justin Chu <justinchu@microsoft.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use the integrations namespace introduced on main so the rebased Qwen3.8 parity test remains executable after the ecosystem refactor. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
justinchuby
force-pushed
the
justinchuby-add-nemotron-35-lightning
branch
from
August 20, 2026 01:01
0722ddb to
8ccfc25
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16@d468880b6ad3c6e0d21377ce7242adaea4cc884dusing the realnemotron_harchitecture, not an aliasmtp.*tensors--dequantizeopt-out unchanged8143a7cc06a26bb32a6f4e99f0000b06e5ba3ae0) exactly once; this PR retains only 19 lines of Nemotron-specificmoe-modelsguidancemainExact-head evidence
cf77d0f7fb28adebbeb3aaecb25fe2e51ee0ddd08ccfc25977d9551c7dfc07088eea1827a5fcb8b8lintrunner;lintrunner -a -r origin/mainandgit diff --check: cleanmain;moe-models: 19 Nemotron-specific added linesThe nine exact-base failures are: GGUF Q4_0 zero-point dequantization; two Attention softcap decomposition selectors; Qwen Image FP16 and BF16 CUDA low-precision selectors; and four newly merged upstream-patch asset tests. All reproduce on detached
cf77d0f; this PR does not hide or modify them.CUDA and Olive
0.0078125, cosine0.9999970790.00292969,0.0078125,0.00390625,0.003906250.999999642,0.999999821,0.999999702,0.999999702[12, 13, 12, 12]Attention,Scan, andConv, with noGroupQueryAttention,LinearAttention, orCausalConvWithStateMatMulNBits, smaller package, CUDA reload, finite logits, exact generation0.855469max abs; supported Option A converts source BF16 weights to FP16The old A10
max_abs=0.309412cached-step failure was caused by provider-dependent fused hybrid cache kernels, not Olive: it occurred identically before and after quantization. A fully standard ONNX cache graph restores every reused-state step below1e-2; no tolerance or golden was changed.ORT GenAI metadata policy
Mobius emits faithful graph-derived filenames, semantic inputs/outputs, representable KV/cross/conv templates, actual
logits/logits_Q4names, and the global cache-slot count. It does not reject export based on the current runtime registry, version, topology support, or cache executor capability. NemotronHssm_statehas no current config template, so that unrepresentable field is omitted without blocking artifacts; downstream load/generation remains optional evidence.GGUF config extraction also validates special-token IDs against vocabulary bounds, drops serialized null/out-of-range sentinels, and emits a deduplicated EOS/EOT/EOM stop-token union. Current-main
generation_config.jsontoken-ID precedence and upstream asset corrections remain intact.Resource/runtime waivers
Relates to #483.
Cross-cutting skill guidance merged in #500.