quantize: default to trellis, and to the variant that has a fused kernel - #66
Merged
Conversation
`--codebook` defaulted to `e8_shell`, the original 8-D lattice, while the README recommended trellis and the checkpoint table marked a trellis build as the fastest GLQ decode. Someone following the docs and running `glq-quantize` got a lattice checkpoint. Trellis (QTIP TCQ) has been the recommended path since v0.7 and wins where bits are scarcest — SmolLM3-3B at 2 bpw is PPL 11.94 against 13.79. Two defaults had to move together. `GLQ_TRELLIS_VARIANT` defaulted to `hyb`, but the fused CUDA kernels consume a **3INST** checkpoint; a hyb one falls back to the pure-torch decode, "correct but materially slower". Defaulting to trellis alone would have shipped checkpoints with no fast path — worse than the lattice default it replaced. Quality is a wash: 3INST-4 matches HYB-4 on SQNR and PPL. Consequences handled rather than discovered later: - **`scripts/batch_quantization.py` omitted `--codebook` when it equalled `e8_shell`**, relying on the CLI default matching its own. It now always passes the flag: a job that declared e8_shell would otherwise have silently become trellis, and any fractional-bpw job would then fail, since trellis takes uniform integers only. - **The resume run-key** now hashes the real variant instead of `""`. This invalidates existing trellis resume stores, which is correct: they hold *hyb* work that must never be resumed into a 3inst run. - **The mixed-precision refusal now names `--codebook e8_shell`.** `--bpw 3.5` used to work with no flag at all; without that pointer the new default turns it into a dead end. - README's "fractional like 2.5 also works" is no longer true by default, and says so. `build_parser()` is split out of main() so CLI defaults can be asserted without running a quantization — there was no test surface for them before. Also documents the Blackwell/FlashInfer caveat next to the `vllm serve` examples: on sm_120 with no CUDA Toolkit, vLLM's sampler compiles at startup and takes the engine down before the first token. glq-chat falls back on its own; `vllm serve` and `LLM(...)` do not. Full local suite: 1085 passed, 882 skipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
--codebookdefaulted toe8_shellwhile the README recommended trellis and the checkpoint table marked a trellis build as "fastest GLQ decode". Someone following the docs and runningglq-quantizegot a lattice checkpoint.Trellis (QTIP TCQ) has been the recommended path since v0.7 and wins where the bits are scarcest — SmolLM3-3B at 2 bpw is PPL 11.94 vs 13.79.
Two defaults, not one
GLQ_TRELLIS_VARIANTdefaulted tohyb, but the fused CUDA kernels consume a 3INST checkpoint — a hyb one falls back to the pure-torch decode, "correct but materially slower". Defaulting to trellis alone would have shipped checkpoints with no fast path, which is worse than the lattice default it replaced. Quality is a wash: 3INST-4 matches HYB-4 on SQNR and PPL.Consequences handled, not discovered later
scripts/batch_quantization.pyomitted--codebookwhen it equallede8_shell, relying on the CLI default matching its own dataclass default""for the variant--codebook e8_shell.--bpw 3.5used to work with no flag; without the pointer the new default makes it a dead endbuild_parser()is split out ofmain()so CLI defaults can be asserted without running a quantization; there was no test surface for them before.Also in this PR
The Blackwell/FlashInfer caveat next to the
vllm serveexamples: on sm_120 (RTX 5090, RTX PRO 6000) with no CUDA Toolkit, vLLM's sampler ships no prebuilt kernel, compiles one at startup, and takes the engine down before the first token — GLQ's own kernels load fine.glq-chatdetects and falls back;vllm serveandLLM(...)do not, and the README now says so.Verification
Note for whoever cuts the next release
This changes what
glq-quantizeproduces for anyone who does not pass--codebook. That is the intent, but it is a behaviour change worth a release-note line rather than a footnote.