ModelFitRun fit check→
Home/Calculators/KV cache
Context × concurrency × architecture

KV cache
calculator.

Isolate the serving-memory component that grows with active tokens. Model weights are deliberately excluded from this result.

Workload calculatorKV CACHEFormula v1.0.0
KV cache per GPU16.0 GiB16.0 GiB total across 4 active sequences
Per sequence
4.00 GiB
Cache shards
1×
Per 1K tokens
0.12 GiB

KV cache sharding stops at the model's 8 KV heads. This result excludes weights, runtime workspace, prefix cache and allocator fragmentation.

Compare GPU profilesCheck live GPU capacity ↗
Formula

Cache memory follows architecture, not parameter count.

ModelFit uses the verified layer count, KV-head count and head dimension condensed into a per-1K-token coefficient. It then applies effective context, concurrent sequences and the selected cache precision.

KV GiB = coefficient × effective tokens ÷ 1,000 × sequences × precision
Sharding limit

More GPUs do not split cache forever.

Tensor parallelism can shard cache across ranks, but the useful shard count is capped by the model's KV-head count. The tool displays that cap instead of assuming perfect division.

Read the context-window guide →
Known omissions

Keep allocator and prefix-cache margins visible.

Runtime fragmentation, CUDA graphs, prefix caching, speculative decoding and multimodal inputs are stack-dependent. Add them during load testing rather than hiding them in the cache formula.

Calculate the full VRAM envelope →