Introduce a GGUF support registry and fix five architecture drift bugs - #571
Open
justinchuby wants to merge 9 commits into
Open
Introduce a GGUF support registry and fix five architecture drift bugs#571justinchuby wants to merge 9 commits into
justinchuby wants to merge 9 commits into
Conversation
GGUF support is currently asserted implicitly by nine hand-maintained containers spread across five modules, keyed inconsistently on either the GGUF general.architecture string or the mobius model_type. Nothing cross-checks them, so they have drifted apart silently. Introduce the vocabulary a single source of truth needs, as pure dataclasses with no dependencies on the rest of the package: - GGUFArchitectureSpec splits support into four independent verdicts — config extraction, tensor mapping, graph construction, and runtime packaging — because they genuinely differ per architecture. Anything short of SUPPORTED must carry a reason, so being listed is never itself a support claim. - GGUFQuantSpec does the same for stored quantization: parse-layer readability, dequantization, native block preservation, and affine repack are answered separately. - Validation rejects incoherent registrations outright: a buildable architecture with no model_type, a mapping recipe attached to an unsupported verdict, a type that is both natively preserved and repacked, or an unreadable slot claiming it can be dequantized. Behavior is referenced by name rather than by callable so the registry modules can stay import leaves; the owning module resolves the name. The new exceptions subclass the types the corresponding paths already raised, so existing callers and tests keep working. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Add a trimmed, mechanically extracted census of llama.cpp at commit 8d9af256337d1a501250f9bbf4c0859a654bddd6: all 147 real llm_arch entries and all 43 ggml_type slots. Block geometry was read from a compiled libggml-base via ggml_get_type_traits(), so it is exact. This exists to make coverage measurable, not to claim it. Nothing here implies mobius supports an architecture. It is used only to - validate that every architecture mobius registers is a real upstream architecture string rather than a mobius model_type that leaked into the architecture namespace, - keep quantization block geometry from being typed by hand, and - turn an unrecognized architecture into a message that names the upstream cohort it belongs to. The loader is an import leaf and the payload is data only. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
_repacker held four literal dicts (_BLOCK_BYTES, _GGUF_BLOCK_ELEMENTS, _SUPPORTED_TYPES, _REPACK_PARAMS), a fifth for native block layouts, and sixteen hand-typed ggml type ids. _builder held four more overlapping tables. Nothing checked them against each other; a comment in _builder documented the consequence outright — a type repackable in one table but missing from another raised KeyError partway through a build, after the multi-gigabyte download. Add _quant_registry as the single source of truth for all 43 ggml slots and rewrite the _repacker tables as derivations of it. Block geometry now comes from the pinned census rather than being typed by hand, so it cannot drift from upstream, and the type ids resolve by name. Only three facts are declared by mobius: which types the runtime reads byte-for-byte, which are repacked into MatMulNBits, and which an untied lm_head may stay quantized in. Types with no gguf-py dequantizer (Q1_0, Q2_0) are DEFERRED with that reason rather than failing obscurely, and the compute-only and removed slots are REJECTED with an explanation instead of a KeyError. This is behavior-preserving. _quant_registry_test pins every literal the refactor replaced, verbatim, and asserts the derived tables equal them. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Nine containers answered the question "is this architecture supported?", and they disagreed: - bloom and t5 resolved to real model types but had no tensor mapping, so config extraction succeeded and the build then died with a message contradicting the config map. - gemma and internlm2 had tensor mappings but no config entry, so they worked only by an unmapped .get(arch, arch) fall-through. - qwen2_moe, qwen3_moe, hunyuan_v1_dense, mistral and muse_glimmer sat in architecture-keyed containers although llama.cpp emits none of them. The two that matter are qwen2moe/qwen3moe: a real Qwen-MoE GGUF passed tensor mapping, then fell through to model_type "qwen2moe", which is not registered, and failed. The reachable keys were dead and the dead keys were reachable. Add _arch_registry as the single source of truth and derive every architecture-keyed container from it: GGUF_ARCH_TO_MODEL_TYPE, _ARCH_KEY_MAPS and _CONFIG_POSTPROCESSORS in _config_mapping; the five family frozensets and the _build_mapping if/elif chain in _tensor_mapping; LLAMA_QK_PERMUTE_MODEL_TYPES and the processor table in _tensor_processors; _OFFSET_NORM_GGUF_ARCHS, _V_HEAD_REORDER_GGUF_ARCHS and the disabled-architecture guard in _builder; and the VLM dispatch in _mmproj. The remaining quantization tables in _builder are derived from _quant_registry. Canonical names are validated against the pinned census, so a mobius model_type can no longer masquerade as an architecture; defensive spellings are declared as aliases. qwen2moe/qwen3moe therefore become canonical and the old spellings become aliases, which is what makes real Qwen-MoE GGUFs importable. gemma and internlm2 are declared explicitly rather than working by accident. bloom and t5 are declared configurable but unmappable, so they now fail before config extraction with a reason instead of a contradiction; they never imported successfully. Specs name their behavior instead of holding callables, so the registry stays an import leaf and both a typo and an orphaned implementation become test failures. The tests are written to fail: capability closure, registry resolution, bidirectional name closure, namespace hygiene, alias determinism, a pinned supported count, and an actionable rejection for every one of the 147 upstream architectures mobius does not import. Tensor mappings are byte-identical for all 28 previously mappable architecture spellings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
llama.cpp bakes the 1 + of a centered RMSNorm into the stored weight so its generic kernel can use the tensor directly, writing w_gguf = w_hf + 1. models/gemma3_text.py normalizes with OffsetRMSNorm, which re-applies the 1 + at runtime, so the import path has to subtract it back out. It did not. The weight processors were keyed by mobius model_type and registered under "gemma3", but GGUF gemma3 resolves to model_type "gemma3_text", so _process_gemma never ran. No other code compensated: _OFFSET_NORM_GGUF_ARCHS covers only qwen35 and qwen35moe. Every norm in an imported Gemma 3 model was therefore offset twice. Measured on the tree before this change, a unit norm weight came out unchanged for gemma3 while gemma2 correctly came out at 0. Now that dispatch keys on the architecture spec rather than on model_type, declaring the processor on the gemma3 spec fixes it. Gemma 4 is deliberately left with no processor: models/gemma4.py normalizes with plain RMSNorm, not OffsetRMSNorm, so its GGUF weights are already in the form the graph consumes and applying the un-offset would corrupt them. A test guards that inverse too. The invariant is mechanical rather than a hand-written list: every importable architecture whose mobius module instantiates OffsetRMSNorm must have either the gemma processor or the offset_norm hook. Gemma 3 was the only architecture failing it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
docs/api/build_from_gguf.md said "Most decoder-only LLM architectures are supported", which cannot be checked and was not true in either direction: bloom and t5 were documented as mapping to model classes but could not be imported, while gemma, internlm2 and the real qwen2moe / qwen3moe spellings went unmentioned. Publish the registry instead: one row per architecture with its accepted aliases, its mobius model_type, and its four capability verdicts. Add the supported stored-quantization types with the same separation between repacked, natively preserved, and rejected, including why the eight retired ggml slots and the two compute-only types can never be read. A test asserts the table equals the registry, so the documentation cannot drift from the code. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Three defects found reviewing the registry change. _gguf_arch was dropped by the dtype and quantization dataclasses.replace calls in _builder and never re-attached, so the registry-keyed dispatch the refactor introduced was dead on every non-float32 and every quantized import; those silently fell back to the model_type table. Re-attach it beside the MTP metadata that block already restores, and in the Muse Glimmer VLM path. _build_mapping raised NotImplementedError for architectures the registry marks REJECTED. That gate has always reported every unmapped architecture as a ValueError, and clip reaches it: the mmproj sidecar is deliberately exempted from the earlier guard, so build_from_gguf on an mmproj file changed base exception type. Translate at the gate; the deliberately-disabled architectures still raise NotImplementedError from _validate_gguf_model exactly as before. _process_nemotron added one to every norm weight while citing HF's NemotronTensorProcessor, which subtracts one. llama.cpp's conversion/nemotron.py:191 writes w_gguf = w_hf + 1 to fold layernorm1p into its generic kernel, and models/nemotron.py normalizes with OffsetLayerNorm, which re-applies the 1 +. Adding instead of subtracting made the effective scale w_hf + 3: for a stored weight of 1.25 the graph scaled by 3.25. Corrected, the transform is byte-identical to Gemma's, so the two are consolidated into one unoffset_norm processor. The offset-norm invariant missed this because it only matched OffsetRMSNorm, not OffsetLayerNorm, and only checked which processor was declared. It now matches both classes and pins the direction by value, which is what actually catches an inverted sign. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
_gguf_arch is a plain instance attribute, so every dataclasses.replace in the builder drops it, and it is the key the weight-processor dispatch is built on. Losing it silently demotes dispatch to the model_type fallback — the indirection the registry exists to remove — and would stay invisible until a spec's processor stopped agreeing with its model_type. Assert it survives to process_tensors on all three build paths. Removing the re-attachment fails the dtype-override and quantized cases while the plain float path still passes, which is exactly the asymmetry that makes this worth pinning. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
The pinned payload carried all 24 survey columns per architecture and 10 per ggml slot, at one JSON line per field. That was 1994 lines — 42% of the change — for data nothing consumed: rope_type, topology, recurrent, hybrid and converter had no reader at all, and status and has_to_float were likewise unused. Keep only what the registry or its tests actually read (cohort and cpp_loader for rejection messages, block geometry and dequantizer coverage for the quantization registry) and reduce moe_mode to the dual_moe boolean the one assertion needs. Emit one record per line so each architecture and each slot is still readable in a diff. 1994 lines to 198, 46 KB to 21 KB, with no change in behavior. The full survey stays where it was produced rather than in the repository. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Performance Comparison
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a single-source-of-truth registry for GGUF architecture and stored-quantization support, backed by a pinned llama.cpp “census” dataset, and refactors the GGUF import pipeline to derive all previously hand-maintained mappings/tables from those immutable specs. It also includes targeted fixes for previously identified architecture drift bugs by moving dispatch to GGUF-architecture-keyed specs and enforcing invariants via tests.
Changes:
- Add immutable spec vocab (
GGUFArchitectureSpec,GGUFQuantSpec) plus registries (_arch_registry,_quant_registry) and derive legacy tables/views from them. - Vendor a pinned llama.cpp support census (architectures + ggml types) and add integrity/gap tests to keep coverage measurable and error messages actionable.
- Refactor GGUF config/tensor mapping, tensor processors, builder, and mmproj routing to dispatch via registry specs; add regression tests ensuring
_gguf_archsurvives config replacements.
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/mobius/integrations/gguf/_upstream.py | Loader for pinned llama.cpp census payload (architectures + ggml types). |
| src/mobius/integrations/gguf/_upstream_gap_test.py | Tests ensuring upstream coverage gaps are explicit and refusal messages are actionable. |
| src/mobius/integrations/gguf/_upstream_data/llamacpp_pin.json | Vendored upstream census data (pinned commit). |
| src/mobius/integrations/gguf/_upstream_data/init.py | Marks vendored census data as a package resource. |
| src/mobius/integrations/gguf/_tensor_processors.py | Derives Q/K permute set and tensor-processor dispatch from the arch registry; consolidates norm un-offset logic. |
| src/mobius/integrations/gguf/_tensor_mapping.py | Moves tensor-map selection from local frozensets/if-elif to registry recipes + named mapping tables. |
| src/mobius/integrations/gguf/_spec.py | Introduces immutable capability spec dataclasses/enums with validation rules. |
| src/mobius/integrations/gguf/_spec_test.py | Unit tests for spec validation invariants. |
| src/mobius/integrations/gguf/_repacker.py | Derives repack/native-block tables from the quant registry instead of literals. |
| src/mobius/integrations/gguf/_quant_registry.py | New single source of truth for ggml type capabilities, built from the pinned census. |
| src/mobius/integrations/gguf/_quant_registry_test.py | Pins pre-refactor literals and asserts derived behavior/tables remain identical. |
| src/mobius/integrations/gguf/_mtp.py | Propagates _gguf_arch into derived MTP configs for correct downstream dispatch. |
| src/mobius/integrations/gguf/_mmproj.py | Switches VLM builder routing to spec-selected builders and centralizes mmproj architecture constant. |
| src/mobius/integrations/gguf/_errors.py | Adds GGUF-specific exception types while preserving legacy base classes. |
| src/mobius/integrations/gguf/_config_mapping.py | Derives GGUF_ARCH_TO_MODEL_TYPE from the arch registry; keys config postprocessors by spec name and persists _gguf_arch. |
| src/mobius/integrations/gguf/_builder.py | Uses registry/specs for disabled-arch gating and derived arch sets; preserves _gguf_arch across config replacements; uses quant registry for float/quant logic. |
| src/mobius/integrations/gguf/_builder_test.py | Adds regression tests asserting _gguf_arch reaches process_tensors on all build paths. |
| src/mobius/integrations/gguf/_arch_registry.py | New single source of truth for GGUF architecture support (capability verdicts + named behaviors). |
| src/mobius/integrations/gguf/_arch_registry_test.py | Registry invariant tests (name hygiene, closure, doc sync, offset-norm compensation, etc.). |
| pyproject.toml | Ensures the vendored census JSON is packaged as distribution data. |
| docs/api/build_from_gguf.md | Replaces vague support claims with a registry-synchronized support matrix and quantization summary. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| #: Per-architecture key maps expanded over every canonical name and alias. | ||
| #: Derived from the registry rather than declared, so a spelling accepted by the | ||
| #: tensor mapping cannot be missing here. | ||
| _ARCH_KEY_MAPS: MappingProxyType[str, dict[str, str]] = MappingProxyType( |
| # than hand-typed, so they cannot drift from ``gguf.GGMLQuantizationType``. | ||
| _GGUF_Q4_0 = _type_id("Q4_0") | ||
| _GGUF_Q4_1 = _type_id("Q4_1") | ||
| _GGUF_Q8_0 = _type_id("Q8_0") |
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
This was referenced Aug 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Foundation for supporting every GGUF architecture and stored quantization type llama.cpp supports. This PR is structure and drift-proofing only — no new architecture cohorts, no sharded GGUF, no new dequantizers.
Problem
GGUF support was asserted implicitly by nine hand-maintained containers spread across five modules, keyed inconsistently on either the GGUF
general.architecturestring or the mobiusmodel_type. Nothing cross-checked them, and they had drifted apart: 25 architectures had a config mapping, 28 had a tensor mapping, and the two sets disagreed in both directions.Design
One immutable source of truth per question, with everything else derived.
GGUFArchitectureSpecanswers four capability questions independently —config,tensor_map,graph,runtime— instead of one boolean, because they genuinely differ per architecture. Anything short ofSUPPORTEDmust carry a reason, so listing an architecture is never itself a support claim.GGUFQuantSpecdoes the same for stored quantization: parse-layerreadable,dequantize,native_preserve,affine_repack._arch_registryan import leaf and makes both a typo and an unreferenced implementation a test failure rather than dead weight.Derived from the registry:
GGUF_ARCH_TO_MODEL_TYPE,_ARCH_KEY_MAPS,_CONFIG_POSTPROCESSORS, the five_*_FAMILYfrozensets and the_build_mappingif/elif chain,LLAMA_QK_PERMUTE_MODEL_TYPESand the processor table,_OFFSET_NORM_GGUF_ARCHS,_V_HEAD_REORDER_GGUF_ARCHS, the disabled-architecture guard, the mmproj VLM dispatch, and nine quantization tables across_repackerand_builder.Upstream pin
llama.cpp
8d9af256337d1a501250f9bbf4c0859a654bddd6— 147 realllm_archentries and all 43ggml_typeslots, extracted mechanically (block geometry read from a compiledlibggml-baseviaggml_get_type_traits()).Vendored as data, trimmed to the fields the registry actually reads (198 lines). It exists to make coverage measurable, not to claim it: canonical names are validated against it so a mobius
model_typecan no longer masquerade as an architecture, and an unrecognized architecture gets a message naming its upstream cohort.Five proven bugs
Each was reproduced and root-caused before being touched.
model_typeand registered undergemma3, but GGUFgemma3resolves togemma3_text, so the Gemma un-offset never ran.models/gemma3_text.pyusesOffsetRMSNorm, so llama.cpp's baked-in+1was applied twice on every norm._process_nemotronadded one while citing HF'sNemotronTensorProcessor, which subtracts. llama.cppconversion/nemotron.py:191writesw_gguf = w_hf + 1andmodels/nemotron.pyusesOffsetLayerNorm, so the effective scale wasw_hf + 3— a stored weight of 1.25 scaled by 3.25. Corrected, it is byte-identical to Gemma's, so the two consolidated into one processor.qwen2moe/qwen3moeunreachable. mobius keyed onqwen2_moe/qwen3_moe, which are mobiusmodel_typestrings no GGUF carries. A real Qwen-MoE GGUF passed tensor mapping and then failed registry lookup: the reachable keys were dead and the dead keys were reachable. Now canonical, with the old spellings as declared aliases._gguf_archdropped bydataclasses.replace, leaving the new registry dispatch dead on every non-float32 and every quantized import._build_mappingexception base changed forclip, which is reachable viabuild_from_ggufon an mmproj file.gemmaandinternlm2previously worked only by an unmapped.get(arch, arch)fall-through and are now declared explicitly.bloomandt5are declared configurable but unmappable, so they fail before config extraction with a reason instead of a contradiction — they never imported successfully.Behavior boundaries
Behavior-preserving except for the five fixes above.
if/elifchain and diffing._quant_registry_test.pyand asserted equal to the derived table.__all__untouched,mobius.build_from_ggufand__main__untouched,GGUF_ARCH_TO_MODEL_TYPEretained (now a derived read-only proxy).except ValueError/except NotImplementedErrorcallers keep working. The tensor-mapping gate still reports every unmapped architecture as aValueError;_validate_gguf_modelstill raisesNotImplementedErrorfor deliberately disabled ones.Validation
glm_moe_dsafailures are pre-existing, confirmed by stashing onto cleanmain.lintrunnerclean.model_type-as-architecture, adding an orphan processor, claiming tensor mapping without config, re-inverting the norm sign, and deleting the_gguf_archre-attachment each failed exactly the intended test and nothing else.docs/api/build_from_gguf.mdreplaces "Most decoder-only LLM architectures are supported" with a support matrix that a test asserts equals the registry.Follow-up
The
OffsetRMSNormAdd(weight, 1.0)is not constant-folded and ships in the graph. A separate PR will propose folding the offset at weight load on both the HF and GGUF paths; out of scope here.