Split out of #182, which added q4_K and q6_K dequantisation and closed there. That issue said this was the interesting half and it was never filed.
The problem
q8_0 goes to the device still quantised and is dequantised inside the matmul (#164, and the tiling and SIMD-reduction work in #189 and #191 all live in that kernel). K-quants have no such path: every tensor is materialised as float on the host and narrowed on the way in.
So the smaller file costs more:
file load peak RSS
Gemma 2 q8_0 2.8 GB 7.7s 3.7 GB
Gemma 2 Q4_K_M 1.7 GB 18.8s 6.4 GB
40% less on disk, 70% more resident, two and a half times the load. Reading K-quants bought reach — models that could not be opened at all now can — and none of the economy that is the point of quantising.
Why it was left
It is the same design decision as the dequantise-in-the-matmul work, and doing it twice would be wrong. That work is now done and the shape is known: a qweight holds the file's bytes on the device, and k_q8_gemv unpacks a block into registers inside the loop.
What it needs
A K-quant block is two-tier where q8_0 is flat — 256 weights sharing one f16 super-block scale, with sub-block scales quantised against it, and q4_K additionally carrying quantised mins so its reconstruction is affine rather than symmetric. The unpacking is already written and tested against the reference dequantiser bit for bit (tests/kquant_values.hh); what does not exist is doing it inside a matmul kernel rather than at load.
Q4_K_M is a mixture — some tensors q4_K, some q6_K — so one file needs both kernels, or one kernel that branches on a per-tensor constant. The function-constant specialisation added in #191 is the mechanism for that: it builds one source twice with different compile-time values, precisely so a uniform branch costs nothing.
And there is a warning from #191 worth reading first
An attempt to repack q8_0 on the device for aligned wide loads — committed locally on ai-q8-layout, deliberately not merged — measured exactly zero benefit at a 14-28% cost in load time. Its device-side repack machinery is the thing to reuse here; its conclusion about layout is not evidence about K-quants, but the general lesson is: measure the kernel, not the bytes.
Current baseline
Decode 92.6 tok/s, 82% of the device's measured bandwidth, with q8_0. The five q8 matmuls are 83% of a layer's kernel time and run at 155 GB/s. A K-quant path would be competing with that, on half the bytes.
Split out of #182, which added q4_K and q6_K dequantisation and closed there. That issue said this was the interesting half and it was never filed.
The problem
q8_0 goes to the device still quantised and is dequantised inside the matmul (#164, and the tiling and SIMD-reduction work in #189 and #191 all live in that kernel). K-quants have no such path: every tensor is materialised as float on the host and narrowed on the way in.
So the smaller file costs more:
40% less on disk, 70% more resident, two and a half times the load. Reading K-quants bought reach — models that could not be opened at all now can — and none of the economy that is the point of quantising.
Why it was left
It is the same design decision as the dequantise-in-the-matmul work, and doing it twice would be wrong. That work is now done and the shape is known: a
qweightholds the file's bytes on the device, andk_q8_gemvunpacks a block into registers inside the loop.What it needs
A K-quant block is two-tier where q8_0 is flat — 256 weights sharing one f16 super-block scale, with sub-block scales quantised against it, and q4_K additionally carrying quantised mins so its reconstruction is affine rather than symmetric. The unpacking is already written and tested against the reference dequantiser bit for bit (
tests/kquant_values.hh); what does not exist is doing it inside a matmul kernel rather than at load.Q4_K_M is a mixture — some tensors q4_K, some q6_K — so one file needs both kernels, or one kernel that branches on a per-tensor constant. The function-constant specialisation added in #191 is the mechanism for that: it builds one source twice with different compile-time values, precisely so a uniform branch costs nothing.
And there is a warning from #191 worth reading first
An attempt to repack q8_0 on the device for aligned wide loads — committed locally on
ai-q8-layout, deliberately not merged — measured exactly zero benefit at a 14-28% cost in load time. Its device-side repack machinery is the thing to reuse here; its conclusion about layout is not evidence about K-quants, but the general lesson is: measure the kernel, not the bytes.Current baseline
Decode 92.6 tok/s, 82% of the device's measured bandwidth, with q8_0. The five q8 matmuls are 83% of a layer's kernel time and run at 155 GB/s. A K-quant path would be competing with that, on half the bytes.