Skip to content

quantize: default to trellis, and to the variant that has a fused kernel - #66

Merged
cnygaard merged 1 commit into
mainfrom
feat/trellis-default-codebook
Aug 18, 2026
Merged

quantize: default to trellis, and to the variant that has a fused kernel#66
cnygaard merged 1 commit into
mainfrom
feat/trellis-default-codebook

Conversation

@cnygaard

Copy link
Copy Markdown
Owner

--codebook defaulted to e8_shell while the README recommended trellis and the checkpoint table marked a trellis build as "fastest GLQ decode". Someone following the docs and running glq-quantize got a lattice checkpoint.

Trellis (QTIP TCQ) has been the recommended path since v0.7 and wins where the bits are scarcest — SmolLM3-3B at 2 bpw is PPL 11.94 vs 13.79.

Two defaults, not one

GLQ_TRELLIS_VARIANT defaulted to hyb, but the fused CUDA kernels consume a 3INST checkpoint — a hyb one falls back to the pure-torch decode, "correct but materially slower". Defaulting to trellis alone would have shipped checkpoints with no fast path, which is worse than the lattice default it replaced. Quality is a wash: 3INST-4 matches HYB-4 on SQNR and PPL.

Consequences handled, not discovered later

scripts/batch_quantization.py omitted --codebook when it equalled e8_shell, relying on the CLI default matching its own dataclass default now always explicit — otherwise a job declaring e8_shell silently becomes trellis, and any fractional-bpw job fails outright
The resume run-key hashed "" for the variant now hashes the real one. This invalidates existing trellis resume stores, which is correct: they hold hyb work that must never be resumed into a 3inst run
The mixed-precision refusal didn't say what to use instead now names --codebook e8_shell. --bpw 3.5 used to work with no flag; without the pointer the new default makes it a dead end
README claimed "fractional like 2.5 also works" corrected — uniform integers only by default

build_parser() is split out of main() so CLI defaults can be asserted without running a quantization; there was no test surface for them before.

Also in this PR

The Blackwell/FlashInfer caveat next to the vllm serve examples: on sm_120 (RTX 5090, RTX PRO 6000) with no CUDA Toolkit, vLLM's sampler ships no prebuilt kernel, compiles one at startup, and takes the engine down before the first token — GLQ's own kernels load fine. glq-chat detects and falls back; vllm serve and LLM(...) do not, and the README now says so.

Verification

  • 6 new tests pinning the defaults, the still-reachable lattice codebooks, that the default bpw is one trellis accepts, the variant default, the env override, and that the refusal names e8_shell.
  • Full local suite: 1085 passed, 882 skipped, 0 failed.
  • Lint: the files touched are clean; the repo's pre-existing 70 findings are unchanged in count.

Note for whoever cuts the next release

This changes what glq-quantize produces for anyone who does not pass --codebook. That is the intent, but it is a behaviour change worth a release-note line rather than a footnote.

`--codebook` defaulted to `e8_shell`, the original 8-D lattice, while the README recommended
trellis and the checkpoint table marked a trellis build as the fastest GLQ decode. Someone
following the docs and running `glq-quantize` got a lattice checkpoint. Trellis (QTIP TCQ)
has been the recommended path since v0.7 and wins where bits are scarcest — SmolLM3-3B at
2 bpw is PPL 11.94 against 13.79.

Two defaults had to move together. `GLQ_TRELLIS_VARIANT` defaulted to `hyb`, but the fused
CUDA kernels consume a **3INST** checkpoint; a hyb one falls back to the pure-torch decode,
"correct but materially slower". Defaulting to trellis alone would have shipped checkpoints
with no fast path — worse than the lattice default it replaced. Quality is a wash: 3INST-4
matches HYB-4 on SQNR and PPL.

Consequences handled rather than discovered later:

- **`scripts/batch_quantization.py` omitted `--codebook` when it equalled `e8_shell`**,
  relying on the CLI default matching its own. It now always passes the flag: a job that
  declared e8_shell would otherwise have silently become trellis, and any fractional-bpw job
  would then fail, since trellis takes uniform integers only.
- **The resume run-key** now hashes the real variant instead of `""`. This invalidates
  existing trellis resume stores, which is correct: they hold *hyb* work that must never be
  resumed into a 3inst run.
- **The mixed-precision refusal now names `--codebook e8_shell`.** `--bpw 3.5` used to work
  with no flag at all; without that pointer the new default turns it into a dead end.
- README's "fractional like 2.5 also works" is no longer true by default, and says so.

`build_parser()` is split out of main() so CLI defaults can be asserted without running a
quantization — there was no test surface for them before.

Also documents the Blackwell/FlashInfer caveat next to the `vllm serve` examples: on sm_120
with no CUDA Toolkit, vLLM's sampler compiles at startup and takes the engine down before
the first token. glq-chat falls back on its own; `vllm serve` and `LLM(...)` do not.

Full local suite: 1085 passed, 882 skipped.
@cnygaard
cnygaard merged commit ea35a75 into main Aug 18, 2026
3 checks passed
@cnygaard
cnygaard deleted the feat/trellis-default-codebook branch August 18, 2026 22:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant