feat(models): add Muse Glimmer 30B - #848
Conversation
There was a problem hiding this comment.
Pull request overview
Adds the CPU-only Muse Glimmer 30B model with vision and DFlash support.
Changes:
- Adds the model definition and publishing workflow entry.
- Pins CPU llama.cpp backends while retaining the CUDA release tag.
- Documents the published image and updates backend tests.
Show a summary per file
| File | Description |
|---|---|
models/muse-glimmer-30b.yaml |
Defines model, projector, and DFlash artifacts. |
pkg/aikit2llb/inference/convert.go |
Splits CPU and CUDA backend versions. |
pkg/aikit2llb/inference/backend.go |
Selects backend versions by runtime. |
pkg/aikit2llb/inference/backend_test.go |
Updates backend-version tests. |
.github/workflows/update-models.yaml |
Registers the model and excludes Apple Silicon. |
README.md |
Documents the CPU image. |
website/docs/premade-models.md |
Adds the model to website documentation. |
Review details
Tip
Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 7/7 changed files
- Comments generated: 0
- Review effort level: Balanced
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
fa97097 to
0dd07bd
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0dd07bd1d5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| #syntax=ghcr.io/kaito-project/aikit/aikit:latest | ||
| apiVersion: v1alpha1 | ||
| debug: true | ||
| models: |
There was a problem hiding this comment.
When this aikitfile is built with the frontend from this tree, the omitted runtime defaults to CPU (requestedRuntime in pkg/aikit2llb/inference/catalog.go), but both default CPU llama-cpp entries in pkg/backendcatalog/catalog.lock.json still resolve to v4.8.2 rather than the Muse-capable revision. That backend lacks this model architecture, so the newly advertised CPU image can build while completion requests fail at model loading; update the locked amd64 and arm64 CPU artifacts to the validated Muse-capable backend.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Review details
Suppressed comments (1)
models/muse-glimmer-30b.yaml:16
- The CPU backend pin promised by the PR is missing. Because this file omits
runtime, the checked-in resolver selects the existing CPU llama.cpp entries, which still point to the v4.8.2 artifacts (pkg/backendcatalog/catalog.lock.json:5916-5921and5953-5958). The PR description identifies that backend generation as lacking Muse support, so builds from this repository are not guaranteed to run the newly advertised model. Add the Muse-capable CPU backend/catalog changes (with both AMD64 and ARM64 coverage) before publishing this definition.
backend: llama-cpp
- Files reviewed: 4/4 changed files
- Comments generated: 0 new
- Review effort level: Balanced
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
There was a problem hiding this comment.
Review details
Suppressed comments (2)
README.md:105
- This only documents the CPU invocation. The model is built with the CUDA runtime, and the other CUDA-capable pre-made models are mirrored in the NVIDIA CUDA table; without that row, GPU users following this documentation omit
--gpus alland run the CPU fallback. Add the corresponding CUDA row with--gpus all.
| ✨ Muse Glimmer | Vision + DFlash | 30B | `docker run -d --rm -p 8080:8080 ghcr.io/kaito-project/aikit/muse-glimmer:30b` | `muse-glimmer-30b` | [Apache 2.0](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF/blob/main/LICENSE) |
website/docs/premade-models.md:29
- This only documents the CPU invocation. The model is built with the CUDA runtime, and the other CUDA-capable pre-made models are mirrored in the NVIDIA CUDA table; without that row, GPU users following this documentation omit
--gpus alland run the CPU fallback. Add the corresponding CUDA row with--gpus all.
| ✨ Muse Glimmer | Vision + DFlash | 30B | `docker run -d --rm -p 8080:8080 ghcr.io/kaito-project/aikit/muse-glimmer:30b` | `muse-glimmer-30b` | [Apache 2.0](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF/blob/main/LICENSE) |
- Files reviewed: 4/4 changed files
- Comments generated: 1
- Review effort level: Balanced
| sha256: "27d9a805fa29b943cfb6ad4843367cd4eaaaf06bd452d8cc3e00a2cd18a677bc" | ||
| config: | | ||
| - name: muse-glimmer-30b | ||
| backend: llama-cpp |
Summary
muse-glimmer-30bmodel using the pinned 17 GB K-quant model, vision projector, and DFlash draft model.draft-dflashspeculative decoding.Compatibility status
Muse Glimmer requires llama.cpp b10353 or newer. Support landed in ggml-org/llama.cpp#26841 at commit
62bf73d.AIKit's current signed backend catalog pins LocalAI v4.8.2 for both CPU and CUDA llama.cpp backends. That release embeds llama.cpp b10235 (
221f0f6), which rejects the model with:This PR intentionally does not introduce a CPU-only backend upgrade or asymmetric CPU/CUDA version pins. It should not be merged until Muse-capable CPU and CUDA backend artifacts can be promoted together in the backend catalog.
Validation
make testpasses after rebasing ontomain.sha-7cfccdc-cpu-llama-cppbackend.sha-7cfccdc-gpu-nvidia-cuda-12-llama-cppartifact is not currently published.