🔥  e m b e r

A from-scratch LLM inference engine in Rust.
Run a real language model on your CPU — no framework, no GPU, no server.

Hand-written transformer · RMSNorm · RoPE · GQA + KV-cache · SwiGLU   •   INT8 / INT4 quantization   •   no TensorFlow / PyTorch / candle

What it is  ·  Metrics  ·  How it works  ·  Quickstart  ·  Verified

What it is

Most "LLM in Rust" projects are a wrapper around candle or llama.cpp.
ember isn't. The transformer forward pass — attention, the KV cache, RoPE, quantization —
is written by hand. The only dependencies are plumbing: weight loading, the tokenizer, threads.
_{~1,300 lines of Rust. No ML framework. No GPU. Runs a real Qwen2.5 model on a laptop CPU.}

🧮 Hand-written Every kernel from scratch — matvec, RMSNorm, RoPE, grouped-query attention + KV cache, SwiGLU, softmax. No candle, burn, tch, or ndarray in the core.

🗜 Quantized Custom INT8 and group-wise INT4 with a fused dequant mat-vec — 4× / 7× smaller, and INT8 is quality-lossless (measured, not claimed).

✅ Verified Green CI (build + 13 tests + a live run), a NumPy oracle that mirrors the math, and perplexity numbers — this is evaluated, not just "it runs".

📊 Metrics — measured on the real model

Real numbers, Qwen2.5-0.5B, through the compiled engine — not estimates.

Quality & memory — perplexity over a fixed passage (--perplexity, lower is better):

scheme	weights	vs f32	perplexity ↓	verdict
f32	1976 MB	1.0×	6.49	baseline
int8	496 MB	4.0×	6.38	quality-lossless — within noise of f32
int4 (group-64)	278 MB	7.1×	7.75	+19% — a real, quantified trade

Latency — greedy, single CPU (--prompt … -n 20):

scheme	prefill	decode
f32	8.3 tok/s	8.6 tok/s · 116 ms/tok
int8	6.6 tok/s	6.6 tok/s · 152 ms/tok

The headline: INT8 gives 4× less memory for free (perplexity 6.38 vs 6.49). INT4 buys 7× less memory for ~19% perplexity. Quantization is a memory win here; the scalar dequant trades a little throughput, so SIMD dequant is the natural next step.

🧠 How it works

One decode step is a stack of hand-written kernels. The ego of the whole project is that every box below is code in this repo, not a library call.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#161b22','primaryTextColor':'#e6edf3','primaryBorderColor':'#f0883e','lineColor':'#f0883e','fontSize':'14px'}}}%%
flowchart LR
  TOK["token id"] --> EMB["Embedding<br/>(dequantized row)"]
  EMB --> BLK
  subgraph BLK["× 24 transformer blocks"]
    direction TB
    N1["RMSNorm"] --> ATT["Grouped-Query Attention<br/>RoPE · causal KV cache"]
    ATT --> R1(("+"))
    R1 --> N2["RMSNorm"]
    N2 --> MLP["SwiGLU MLP"]
    MLP --> R2(("+"))
  end
  BLK --> FN["RMSNorm"]
  FN --> LM["LM head<br/>(tied embedding)"]
  LM --> LOG["logits"] --> SMP["sample"] --> TOK

  classDef s fill:#161b22,stroke:#30363d,color:#e6edf3;
  class TOK,EMB,N1,ATT,R1,N2,MLP,R2,FN,LM,LOG,SMP s;

Loading

Module map

Module	Responsibility
config.rs	Parse the Qwen2.5 config.json (GQA-aware)
tensor.rs	The rayon-parallel mat-vec hot loop
ops.rs	RMSNorm · RoPE (rotate-half) · SwiGLU · softmax
attention.rs	Grouped-query attention + rolling KV cache
quant.rs	INT8 / group-wise INT4 quantization + fused dequant mat-vec
sample.rs	Greedy / temperature / top-p sampling
model.rs	safetensors loading (bf16→f32) + the forward pass
chat.rs	Qwen ChatML formatting
main.rs	CLI, generation loop, --bench, --perplexity

Quickstart

# 1. Rust toolchain (https://rustup.rs)
rustup default stable

# 2. Fetch a small model  (needs: pip install huggingface_hub)
huggingface-cli download Qwen/Qwen2.5-0.5B-Instruct \
  model.safetensors config.json tokenizer.json --local-dir ./weights

# 3. Chat, or complete, or quantize
cargo run --release -- --chat --prompt "Write a haiku about Rust programming."
cargo run --release -- --prompt "The capital of France is" --quant int8
cargo run --release -- --perplexity --quant int4     # measure quality
cargo run --release -- --bench --quant int8          # measure throughput

_{On Windows, the tokenizer needs a C compiler at build time — use WSL, or install WinLibs / MSVC Build Tools. Linux & macOS work out of the box.}

How it's verified

Nothing here is "trust me":

• CI — every push runs cargo build --release, 13 unit tests, and a live --bench on the runner (status).
• NumPy oracle — scripts/reference_forward.py reimplements the exact math; the Rust output matches it token-for-token.
• Perplexity — the quantization quality numbers above are measured, not asserted.

$ ember --chat --prompt "What is the capital of France? Answer in one sentence."
The capital of France is Paris.

Roadmap

See PHASES.md.

• Phase 1 — Correctness — hand-written forward pass; generates coherent text
• Phase 2 — Performance — INT8 / INT4 quantization (4× / 7× memory) + --bench
• Phase 3 — Polish — chat mode, unit tests, perplexity + latency metrics, CI
• Next — SIMD dequant (throughput), a static-linked binary, a perf PR upstream

_{Built by Ali Zulfiqar — a transformer, from the gradients up.}