Add conventional MoE GGUF architecture cohort - #575
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Performance Comparison
|
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
justinchuby
force-pushed
the
justinchuby-gguf-dense-transformer-cohort
branch
from
August 25, 2026 14:54
fbdc807 to
29d33a9
Compare
Base automatically changed from
justinchuby-gguf-dense-transformer-cohort
to
main
August 25, 2026 14:55
Implement pinned llama.cpp metadata, tensor closure, routing, and quantized expert import for OLMoE, PhiMoE, Qwen2MoE, Qwen3MoE, and GraniteMoE. Preserve tied quantized heads, split fused Phi QKV tensors, add synthetic and pinned Granite runtime validation, and keep unsupported runtime claims deferred. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Reverse llama.cpp's GraniteMoE Q/K row permutation in float and packed quantized imports. Make pinned expert scale/input_scale suffixes visible and reject unsupported scale2 sidecars before graph construction instead of dropping them. Downgrade GraniteMoE runtime to deferred until independent cross-runtime parity is available. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Route nextn projection scale/input_scale suffixes through the MTP mapper and reject them in the shared GGUF preflight before either backbone or MTP graph construction. Cover dense and MoE Qwen3.5 variants, float and quantized files, and preserve ordinary MTP weight/bias mapping. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Register coverage metadata for the GGUF-only PhiMoE routing variant and make main's synthetic native-block MoE fixture use block-aligned logical dimensions so strict expert-shape validation remains meaningful after the replay. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Recognize the complete affine expert-major repack route added by the conventional MoE cohort, keep native and decoder rejection semantics intact, and make GGUF MoE output heads quantization-aware. Refresh generated policy documentation and coverage pins for the supported cohort. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby-gguf-moe-transformer-cohort
branch
from
August 25, 2026 15:07
72b4f42 to
4d6efb5
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new “conventional MoE” GGUF architecture cohort (olmoe, phimoe, qwen2moe, qwen3moe, granitemoe) to Mobius’ GGUF importer, including pinned upstream tensor-family/spec data, config/metadata extraction, quantized import semantics, and expanded test coverage (synthetic + a pinned real GraniteMoE artifact).
Changes:
- Extend GGUF architecture registry/specs to support the new MoE cohort, including module-type overrides (PhiMoE GGUF variant), pinned tensor-family closure, and alias handling.
- Implement/adjust MoE-specific config postprocessing, tensor-name mapping (including expert suffix sidecars), quantized import routes (expert-major affine repack), and preflight rejection for unsupported auxiliary scale sidecars.
- Add/extend tests: synthetic GGUF save/load/ORT execution for MoE architectures, pinned GraniteMoE real-artifact integration coverage, and closure/shape/negative validation tests.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/model_coverage_test.py | Exempts phimoe_gguf from HF checkpoint coverage expectations. |
| tests/gguf_moe_cohort_integration_test.py | Adds pinned real GraniteMoE GGUF integration test (download, hash/size, closure, ORT determinism). |
| tests/_test_configs.py | Adds a tiny phimoe_gguf test config variant (LongRoPE fields). |
| src/mobius/models/moe.py | Adds GGUF-specific PhiMoE model variant; GGUF-specific quantization preprocessing; routing scaling support; embedding factory use; fused-QKV splitting in Phi3MoE. |
| src/mobius/models/granite.py | Uses embedding_for_config, preserves ties via _replace_text_model, and routes GraniteMoE preprocessing through shared MoE quantization path. |
| src/mobius/models/_models_test.py | Adds unit test asserting PhiMoE GGUF norm class matches checkpoint format. |
| src/mobius/models/init.py | Exports PhiMoEGGUFCausalLMModel. |
| src/mobius/integrations/gguf/_upstream.py | Extends upstream architecture payload model with expert_tensor_suffixes. |
| src/mobius/integrations/gguf/_upstream_data/llamacpp_pin.json | Pins tensor families + expert suffix sets for MoE GGUF architectures. |
| src/mobius/integrations/gguf/_tensor_processors_test.py | Extends Q/K permutation expectations to GraniteMoE. |
| src/mobius/integrations/gguf/_tensor_mapping.py | Adds MoE Q/K norm mapping extras; recognizes rope factor tensor skips; includes .scale/.input_scale in suffix splitting. |
| src/mobius/integrations/gguf/_tensor_mapping_test.py | Adds mapping tests for MoE expert/shared tensors, sidecar suffixes, and aliases. |
| src/mobius/integrations/gguf/_spec.py | Adds module_type to support GGUF-only internal graph variants. |
| src/mobius/integrations/gguf/_quant_registry.py | Allows expert-major affine repack route (no longer hard-rejected). |
| src/mobius/integrations/gguf/_quant_registry_test.py | Updates tests to reflect expert-major affine repack and improved rejection reason when decoder is absent. |
| src/mobius/integrations/gguf/_mtp.py | Makes MTP mapping suffix-aware for .scale/.input_scale. |
| src/mobius/integrations/gguf/_mtp_test.py | Adds tests rejecting MTP auxiliary scale sidecars before graph build; expands MTP writer helper. |
| src/mobius/integrations/gguf/_config_mapping.py | Adds conventional MoE postprocessors; supports routed scaling / norm / group metadata; GraniteMoE dense-vs-MoE dispatch; PhiMoE LongRoPE tensor-backed factors. |
| src/mobius/integrations/gguf/_config_mapping_test.py | Adds config extraction tests for MoE architectures, GraniteMoE scaling/dense dispatch, and PhiMoE LongRoPE. |
| src/mobius/integrations/gguf/_builder.py | Adds preflight rejection for unsupported auxiliary quant sidecars; routes module selection via module_type; adds fused-QKV split targeting; validates MoE tensor shapes; supports affine repack expert-major quantized imports. |
| src/mobius/integrations/gguf/_builder_test.py | Adds synthetic MoE GGUF writer + build/save/load/ORT tests; validates Granite Q/K reverse permutation by value; tests fused Phi QKV splitting and tied quantized embedding/head behavior; adds auxiliary scale rejection tests. |
| src/mobius/integrations/gguf/_block_quantized_moe_builder_test.py | Adjusts E2E dims to align with native IQ block constraints. |
| src/mobius/integrations/gguf/_arch_registry.py | Registers new MoE GGUF architectures (required metadata, postprocessors, tensor recipes, module override for PhiMoE). |
| src/mobius/integrations/gguf/_arch_registry_test.py | Updates supported-architecture count and pins closure tests for the new MoE cohort (including expert suffix closure). |
| src/mobius/components/_rms_norm.py | Adds RMSNormBias for RMSNorm-with-bias checkpoint formats. |
| src/mobius/components/_moe.py | Adds routed_scaling_factor handling to gate implementations (forward + QMoE metadata path). |
| src/mobius/components/init.py | Exports RMSNormBias. |
| src/mobius/_registry.py | Registers phimoe_gguf internal module key and default model id mapping. |
| docs/api/build_from_gguf.md | Updates GGUF support matrix and quantization matrix documentation for MoE cohort and expert-major routing behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+330
to
+333
| self.config = config | ||
| self.model = MoETextModel(config, norm_class=RMSNormBias) | ||
| linear_class = _quantized_linear_class(config) or Linear | ||
| self.lm_head = linear_class(config.hidden_size, config.vocab_size, bias=True) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
8d9af256337d1a501250f9bbf4c0859a654bddd6) GGUF specs forolmoe,phimoe,qwen2moe,qwen3moe, andgranitemoe.scale/.input_scalesuffixes and reject affected files before graph construction because Mobius cannot safely represent NVFP4 scale2/activation-scale sidecarsnextn.eh_projscale sidecars before either backbone or MTP graph constructionStacks on #572 (
justinchuby-gguf-dense-transformer-cohort).Real artifact import evidence (runtime deferred)
GraniteMoE
Validated with:
bartowski/granite-3.0-1b-a400m-instruct-GGUF0e1c3cecaa6e49ac0721be91ef441ec72eae62d4granite-3.0-1b-a400m-instruct-Q4_K_M.gguf821,845,024bytes074f09e13484e54e73c93830d34e9fa9917a6319fb8bae762a22594b9b4da0dcibm-granite/granite-3.0-1b-a400m-instruct@ffec3c35bdfd97a06f0b4cd5fcc92cd9b1584445The pinned integration test verifies exact tensor closure, 24 routers, 2,304 routed expert projections, mixed-qtype routing, 2,401
MatMulNBitsnodes including the tied quantized head, zeroQMoEnodes, ONNX save/load, full 49,155-token logits, and repeatable three-token cached decoding. This is deterministic Mobius execution evidence, not independent llama.cpp/HuggingFace parity, so GraniteMoE runtime remains deferred.Runtime-DEFERRED
All five entries—
olmoe,phimoe,qwen2moe,qwen3moe, andgranitemoe—have exact registry/config/tensor support and synthetic float/Q4 ORT execution, but no qualifying independent real-weight cross-runtime evidence.The only compatible Phi representative found was
tripathyShaswata/Phi-tiny-MoE-instruct-GGUF@873ccb08cd3380ee2c08573d45267fac9a6cc81b, filePhi-tiny-MoE-instruct-Q8_0.gguf, size3,999,171,104bytes, SHA-256297fa09e906e18aaf03850e77d6de8d9ee8e246e00916ac09787ae2cf4bb6019, sourced frommicrosoft/Phi-tiny-MoE-instruct@2fe50e88d0e2a5a132563815686ea0dcc8e252b5. It was not downloaded because it is not a reasonably sized representative.Validation
504 passedfull focused GGUF/model tests after the Granite/expert-scale fixes160 passedfocused MTP/builder/tensor-mapping tests after the MTP scale preflight fix1 passedrevised pinned Granite real-artifact integration test5585 passed, 52 skippedbroad non-integration suite withglm_moe_dsaexcludedlintrunner f --output oneline --all-filesThe two excluded
glm_moe_dsashape-inference/checker failures reproduce unchanged on the stack base and are unrelated to this cohort.