ManboML

ManboML is a Go library for running supported GGUF language models inside the calling process. Its production inference path is designed for CGO_ENABLED=0: it does not load llama.cpp, a shared library, an FFI runtime, or a helper process.

The project is currently preparing its v0.1 release. The first certified profile is Qwen3Guard-Gen-0.6B in Q4_K_M GGUF form. Other Qwen3 models may pass structural validation, but they are not certified by v0.1.

Full-vector log-probability equivalence with llama.cpp is not a v0.1 guarantee. The portable Go and optimized llama.cpp CPU paths accumulate floating-point operations differently across layers. The diagnostic keeps the same final argmax for the compiled guard prompt, and safe and unsafe greedy outputs are exact, but lower-ranked probabilities differ. v0.1 certification is therefore based on pinned operator goldens, exact tokenizer behavior, greedy token behavior, and end-to-end Qwen3Guard results.

Current capabilities

• Read-only GGUF backing through mmap on supported Unix systems, with a bounded read fallback.
• Qwen3 decoder inference with Q4_K, Q6_K, and F32 model tensors.
• GGUF-embedded Qwen2 byte-level BPE tokenization.
• F16 KV cache with a caller-selected context size.
• Deterministic greedy generation.
• Raw prompt generation and a compiled Qwen3Guard chat formatter.
• Eager, bounded private sessions for concurrent requests.
• A model-owned numerical worker pool.
• Optional SHA-256 verification and pre-allocation memory estimation.
• Context cancellation and an idempotent, draining Close.

ManboML is a library only. It does not download or discover models, provide a command-line interface, or execute arbitrary Jinja templates.

Requirements

• Go 1.26.3 or later.
• A supported, local GGUF file.
• Enough address space and memory for the mapped model and all configured sessions.

The production dependency path is pure Go. The current implementation has a cross-build baseline for the documented targets, but uncommon-architecture runtime certification is still M7 release work.

Certified model

The v0.1 reference artifact is:

Qwen3Guard-Gen-0.6B.Q4_K_M.gguf
size:   484,219,904 bytes
sha256: a0d3385101ba362822d914ba40d9767aff634811ac99a41e4509de2d7a453b3e

The filename is not sufficient proof of compatibility. Use ExpectedSHA256 when the exact certified artifact is required.

Model weights, vocabulary data, and embedded templates remain subject to their upstream licenses and are not included in this repository.

Chat example

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"github.com/manboster/manboml"
)

const modelSHA256 = "a0d3385101ba362822d914ba40d9767aff634811ac99a41e4509de2d7a453b3e"

func main() {
	opts := manboml.Options{
		ContextSize:    1024,
		MaxConcurrent:  1,
		ExpectedSHA256: modelSHA256,
	}

	estimate, err := manboml.Estimate("Qwen3Guard-Gen-0.6B.Q4_K_M.gguf", opts)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("conservative payload: %d bytes\n", estimate.TotalBytes)

	model, err := manboml.Open("Qwen3Guard-Gen-0.6B.Q4_K_M.gguf", opts)
	if err != nil {
		log.Fatal(err)
	}
	defer func() {
		if err := model.Close(); err != nil {
			log.Printf("close model: %v", err)
		}
	}()

	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
	defer cancel()

	result, err := model.Chat(ctx, manboml.ChatRequest{
		Messages: []manboml.Message{
			{Role: manboml.RoleUser, Content: "How are you?"},
		},
		MaxTokens: 64,
	})
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(result.Text)
	fmt.Printf("tokens=%d finish=%s\n",
		result.GeneratedTokens, result.FinishReason)
}

Chat accepts semantic messages but only runs a compiled formatter recognized from the model metadata. An unknown template returns ErrUnsupportedTemplate; ManboML never executes arbitrary template code.

The Qwen3Guard formatter supports system, user, and assistant roles. The last message must be a user or assistant message because it selects which content the guard model evaluates.

Raw generation

Generate tokenizes an already formatted prompt. It does not apply a chat template:

result, err := model.Generate(ctx, manboml.GenerateRequest{
	Prompt:    formattedPrompt,
	MaxTokens: 64,
})

MaxTokens must be positive. v0.1 has no unbounded generation default and uses greedy decoding only.

Options

Field	Zero-value behavior	Notes
ContextSize	2,048	Any positive value up to the model maximum may be selected. Prompt and generated tokens must fit together.
MaxConcurrent	1	All sessions and their KV caches are allocated during Open.
Workers	Half of captured GOMAXPROCS, rounded up, minimum 1	An explicit positive value overrides the default. ManboML never changes process-wide GOMAXPROCS.
MemoryLimit	No caller-selected cap	Platform, overflow, context, and internal safety checks still apply.
ExpectedSHA256	Structural compatibility mode	A non-empty hexadecimal digest requires an exact file match.

Smaller contexts reduce memory substantially. For the certified profile, F16 KV cache grows by 112 MiB per 1,024 context tokens, per session. Numerical scratch, tokenizer data, Go runtime overhead, and the mapped model are additional.

Use Estimate before Open on constrained devices:

estimate, err := manboml.Estimate(path, manboml.Options{
	ContextSize:   512,
	MaxConcurrent: 1,
})

MemoryEstimate separates the full model mapping or read-all fallback, retained shared heap, each numerical session, Open-time transient metadata, and runtime headroom. TotalBytes is the conservative capacity value checked against MemoryLimit. It includes the complete mapped file even though the OS faults mapped pages on demand, and it is not an exact RSS prediction. The v0.1 headroom policy is the larger of 32 MiB or 20% of the managed Open-time peak; release measurements may make this policy more conservative.

Model.Info().Memory reports the same admitted plan after Open.

Estimate applies the same structural, ExpectedSHA256, and MemoryLimit admission checks as Open, so the same option set cannot silently bypass those requirements during planning.

Apple M4 baseline

The current correctness-first baseline was measured on Apple M4 with Go 1.26.3, Darwin arm64, 10 workers, a 2,048-token context, one session, and the pinned Qwen3Guard artifact. These are observations, not release performance promises.

Operation	Baseline	Timed allocations
Estimate	232.0 ms/op	25,208,896 B/op; 459,876 allocs/op
Open	249.3 ms/op	262,952,914 B/op; 460,072 allocs/op
300-token guard prompt prefill	8.93 s/op; 33.58 prompt tok/s	2,688,794 B/op; 4,204 allocs/op
One decode token after prefill	86.4 ms/token; 11.57 tok/s	27,952 B/op; 310 allocs/op
Complete safe Chat	11.48 s/op	3,061,706 B/op; 9,891 allocs/op

After Open and a forced GC, the measured retained HeapAlloc increase was 259,093,728 bytes. The admitted managed capacity was 305,493,485 bytes, and the 792,774,071-byte total also counted the complete 484,219,904-byte file mapping.

Reproduce the baseline with:

MANBOML_TEST_GGUF=/path/to/Qwen3Guard-Gen-0.6B.Q4_K_M.gguf \
  go test . -run '^$' -bench '^BenchmarkRealModel(Estimate|Open)$' \
  -benchtime=3x -benchmem

MANBOML_TEST_GGUF=/path/to/Qwen3Guard-Gen-0.6B.Q4_K_M.gguf \
  go test ./internal/arch/qwen3 -run '^$' \
  -bench '^BenchmarkRealModel(Prefill|Decode)$' -benchtime=3x -benchmem

Concurrency and lifecycle

An opened Model may be shared by goroutines. Each active request leases one private session, so at most MaxConcurrent requests evaluate at once. Additional requests wait for a session and honor context cancellation.

Close:

• rejects new requests;
• waits for active requests and worker jobs;
• releases the model backing and worker pool;
• is safe to call more than once.

The model file must not be modified or truncated while it is open.

Results and errors

Successful generation stops with one of:

• FinishEOG: the model selected an end-of-generation token;
• FinishMaxTokens: the request reached its explicit output bound.

Cancellation and computation failures are returned as errors, not finish reasons. context.Canceled and context.DeadlineExceeded remain detectable with errors.Is.

Stable public error categories include:

• ErrInvalidRequest
• ErrInvalidModel
• ErrUnsupportedModel
• ErrUnsupportedTemplate
• ErrChecksumMismatch
• ErrContextLimit
• ErrMemoryLimit
• ErrClosed
• ErrNumerical

Use errors.Is rather than matching error text.

v0.1 scope

The first release intentionally does not include:

• temperature, top-k, top-p, or repetition penalties;
• token streaming or public retained sessions;
• GPU, Metal, CUDA, Vulkan, external BLAS, CGO, or FFI;
• model downloads or conversion;
• arbitrary model architectures, MoE, multimodal models, or adapters;
• arbitrary Jinja template execution;
• a CLI or server.

Future optimizations may add architecture-specific fast paths only when the portable Go implementation remains available for correctness.

Third-party production dependencies and their complete license notices are listed in THIRD_PARTY_NOTICES.md. Model artifacts are not distributed by this repository and retain their upstream licenses. Release-facing changes and known limitations are tracked in CHANGELOG.md.

Verification

Normal tests require neither a network connection nor a production model:

go test ./...
go test -race ./...
go vet ./...

The Unix release check additionally audits the selected dependency path and compiles/links all 17 zero-CGO targets without running non-native binaries:

./scripts/check-release.sh

The complete source, real-model, runtime-certification, and tagging procedure is in RELEASING.md.

Optional real-model tests use a caller-supplied file:

MANBOML_TEST_GGUF=/path/to/Qwen3Guard-Gen-0.6B.Q4_K_M.gguf \
  go test ./...

Exact tokenizer and greedy-output comparisons can additionally use tools from a pinned llama.cpp build:

MANBOML_TEST_GGUF=/path/to/Qwen3Guard-Gen-0.6B.Q4_K_M.gguf \
MANBOML_LLAMA_TOKENIZE=/path/to/llama-tokenize \
MANBOML_LLAMA_COMPLETION=/path/to/llama-completion \
  go test ./...

Those executables are test references only. They are never part of ManboML's production inference path.

When a pinned llama-server is available, the optional full-vector diagnostic requests all 151,936 log-probabilities for the exact supplied token IDs:

MANBOML_TEST_GGUF=/path/to/Qwen3Guard-Gen-0.6B.Q4_K_M.gguf \
MANBOML_LLAMA_SERVER=/path/to/llama-server \
  go test . -run '^TestRealModelLlamaFullLogprobParity$' -v

The diagnostic is intentionally stricter than the v0.1 release contract. It is retained to measure numerical drift and catch regressions, but full-vector equivalence is not a release gate.

The current CGO_ENABLED=0 compile/link matrix covers:

linux/386
linux/amd64
linux/arm
linux/arm64
linux/loong64
linux/mips
linux/mipsle
linux/mips64
linux/mips64le
linux/ppc64
linux/ppc64le
linux/riscv64
linux/s390x
darwin/amd64
darwin/arm64
windows/amd64
windows/arm64

Cross-build success is not presented as runtime certification. Native or emulated operator and miniature-model execution on uncommon targets is a final M7 release gate. With QEMU user-mode runners installed, use ./scripts/check-runtime.sh to execute the offline tests as actual Linux target binaries.

License

ManboML is licensed under the MIT License. See LICENSE.