Add dense GGUF architecture cohort - #572
Open
justinchuby wants to merge 1 commit into
Open
Conversation
Add exact config and tensor mappings for OLMo, OLMo2, Cohere2, Arcee, SmolLM3, and Exaone against the pinned llama.cpp architecture census. Preserve architecture-specific normalization, RoPE, attention schedules, quantized module wiring, and InternLM2 Q/K permutation semantics. Validate strict tensor closure, synthetic float and quantized graph execution, and pinned real OLMo-family GGUF artifacts while deferring runtime claims that lack real-weight evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Performance Comparison
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the first reviewable C01 dense-transformer GGUF cohort on top of #571, using llama.cpp commit
8d9af256337d1a501250f9bbf4c0859a654bddd6as the tensor-family source of truth.olmoOLMoCausalLMModelolmo2OLMo2CausalLMModelcohere2CohereCausalLMModelarceeArceeCausalLMModelsmollm3SmolLM3CausalLMModelexaoneCausalLMModelArchitecture correctness
olmo2files carrying OLMo3 sliding-window patterns until the graph supports per-layer sliding attention.Real GGUF evidence
Pinned revisions, exact filenames, sizes, SHA-256 values, and per-tensor qtype tables are encoded in
tests/gguf_dense_cohort_integration_test.pyfor:QuantFactory/AMD-OLMo-1B-GGUF@5f34243a42dbae2141b8f5286320bf63d51eeefballenai/OLMo-2-0425-1B-Instruct-GGUF@62f8c199538474c3e33ed5d7e0580abd66686a27ggml-org/SmolLM3-3B-GGUF@4965cb60b150737b68a0408c36aeefb65078f894All three packages build, save, open in ORT CPU, and produce deterministic finite logits and pinned greedy outputs.
The mixed Q4_K_M files do not use native preservation. Q4_K and Q6_K projections are dequantized and affine-requantized to explicit-zero-point 4-bit/block-32
MatMulNBits; compatible embeddings useGatherBlockQuantized; float norm tensors remain float.Explicit exclusions
The PR does not infer compatibility from common names. Deferred candidates include GPT-NeoX and Phi-2 (fused/interleaved or dual-form QKV), Baichuan and MPT (ALiBi), StarCoder (learned positions), and unproven Qwen/Command-R aliases. Large Command-R/Seed artifacts were not downloaded.
Validation
The existing
glm_moe_dsasymbolic shape-inference/checker failures reproduce independently and are excluded from the final broad run.