Skip to content

vLLM 0.34 trades decode speed for a much bigger KV cache

The inference server's new default KV cache mode grows context capacity by half on some GPUs, but adds up to 18% decode latency.

Source: vLLM Project

vLLM 0.34.0 shipped on 16 November, and two default changes affect anyone running the inference server on their own GPUs. The deprecated Model Runner V1 is gone for good: the VLLM_USE_V2_MODEL_RUNNER=0 escape hatch now fails at startup, so anyone still on it has to port custom logits processors to the V2 interface before upgrading.

The bigger change is the KV cache. Extensible KV cache is now on by default, and gpu_memory_utilization defaults to 1.0 when it’s active, sizing the cache from whatever memory is actually free after warmup rather than a fixed fraction of it. The default block size also moves from 16 to 64. vLLM’s own release notes show the upside with a concrete example: MiniMax-M3 on an H200 goes from a 433k to a 672k token cache. The cost is decode latency, which the notes say can rise 6 to 18% on some H200 models at a concurrency of 128.

--block-size 16                    # keep the old block size
--no-enable-extensible-kv-cache    # keep the old cache behaviour entirely

Why it matters: upgrade without reading the release notes and you get a bigger context window for free, plus a latency regression you didn’t ask for, on the same hardware. Anyone with a tight time-to-first-token or per-token SLA under concurrent load should benchmark before taking the new default, not after.

Share

Related reading

More from The Wire

All briefs