kv-cache: per-layer KV types, measured bit allocation

[Pascal-SAPUI5]

What

LLAMA_KV_TYPE_PER_LAYER — a comma-separated ggml type name per KV layer, so different layers can use different KV quantization. Unset, nothing changes.

The cache already created tensors per layer and every consumer reads the type off the tensor, so the change is small: pick a type per layer, and check the TurboQuant head_dim guard against the effective type rather than the global one. That guard was the one real correctness issue — a per-layer override would have bypassed it.

Why, and does it actually help

Measured on Qwen3.5-9B, which has 8 attention layers (the other 24 are linear attention with no KV cache).

Upgrading exactly one layer to turbo4, against a 1.5337 all-turbo3 baseline:

layer	PPL	delta
0	1.5341	+0.0004
1	1.5338	+0.0001
2	1.5324	-0.0013
3	1.5325	-0.0012
4	1.5321	-0.0016
5	1.5330	-0.0007
6	1.5329	-0.0008
7	1.5326	-0.0011

Layer 0 gets worse with more bits, which is impossible — that puts the noise floor around ±0.0005 and makes every individual row too weak to act on.

The ranking is a different matter. Same memory, different allocation:

allocation	bpw	PPL
all turbo3	3.500	1.5337
worst four (0,1,5,6)	4.000	1.5322
best four (2,3,4,7)	4.000	1.5292
all turbo4	4.500	1.5284

0.0030 between two configurations costing exactly the same, six times the noise floor. Sorting cancels part of the per-measurement error.

Walking the ranking gives the curve:

upgraded	bpw	PPL	share of turbo4 gain
0	3.500	1.5337	0%
2	3.750	1.5311	49%
3	3.875	1.5303	64%
4	4.000	1.5292	85%
6	4.250	1.5279	109%
8	4.500	1.5284	100%

Six beating eight is 0.0005 — exactly the noise floor, so that ordering means nothing on its own. But the two layers left out are 0 and 1, the only two whose single-layer measurement came out positive. Two independent measurements agreeing is worth more than either.

Verified

• correct list applied per layer (log confirms layers 3,7,11…31 — the eight attention layers)
• fewer entries than layers: warns, last entry covers the remainder
• unknown type name: hard error at construction

Caveats

• The ranking is model-specific and was measured on one model. It should not be assumed to transfer.
• Env var rather than a CLI flag: this is an experiment, and it does not belong in the command-line surface until it earns a place there.
• K and V get the same type per layer. Splitting them was not measured and is not exposed.
• No automated test — the parser is inline in the constructor and CI has no GPU. Verification was manual, as listed above.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q5jMcvdevae8j4T36C9h2z

[Pascal-SAPUI5]

Added a measurement of the other axis a bit budget could go to, since it decides whether this PR is aimed at the right one.

--cache-type-k and --cache-type-v have always allowed K and V to use different types, and KV-quantization work generally finds K the more sensitive of the two. At the same 4.0 bpw:

allocation	PPL
K=turbo4, V=turbo3	1.5319
K=turbo3, V=turbo4	1.5317
per-layer, best four	1.5292

The two K/V directions are 0.0002 apart — noise. No asymmetry to exploit, which is what the Hadamard transform is for: every output is a signed sum of all 128 inputs, so an outlier channel gets spread across the vector instead of dominating one dimension. The asymmetry other methods rely on has already been flattened here.

That makes the layer axis the productive one: 0.0026 better than either K/V split at identical cost, five times the noise floor.