Sizing the KV-cache block pool vs. batch size with paged attention?

[HumbleBee14]

When serving an LLM with paged attention (vLLM-style), how should I think about sizing the KV-cache block pool vs. batch size?

I keep hitting CUDA out of memory under bursty load. A few specific questions:

1. Is it better to cap max_num_seqs or lower gpu_memory_utilization?
2. How do block size (e.g. 16 vs 32 tokens) and fragmentation interact?
3. Any rule of thumb for headroom to avoid preemption/recompute?

Numbers or war stories welcome.

[HumbleBee14]

Rule of thumb that has worked for me on A100/H100:

• Lower gpu_memory_utilization first (e.g. 0.90 → 0.85) before capping max_num_seqs. Capping sequences throttles throughput harder than giving the block pool a little headroom.
• Block size 16 is a good default. Larger blocks (32) reduce metadata overhead but waste more on short sequences (internal fragmentation). Only go bigger if your median sequence is long.
• Keep ~10-15% KV headroom so bursty arrivals don't trigger preemption + recompute — recompute is what usually shows up as a latency cliff, not OOM.

If you are still OOMing under bursts, it is almost always the peak concurrent sequence count, not steady state. Log num_preemptions — if it is climbing, you are over-subscribed.

[HumbleBee14]

Nice!