Context length is not only a model capability. During autoregressive inference, every active sequence creates a growing memory record used by attention. That record—the KV cache—can become the difference between a comfortable fit and an out-of-memory failure.
The important number is active tokens across the workload, not simply the maximum context printed on a model card. A 128K-capable model serving 4K requests has a different memory profile from the same model serving multiple 64K conversations.
What the KV cache stores.
Transformer attention calculates key and value tensors for each processed token. During generation, retaining those tensors avoids recomputing the entire prefix for every new token. The performance win costs memory: cache grows as tokens remain active.
Weights are shared across requests, but each active sequence needs its own cache. That distinction matters when moving from a single-user local test to a batched or concurrent service.
Architecture determines bytes per token.
KV bytes ≈ 2 × layers × KV heads × head dimension × active tokens × bytes per element × concurrent sequencesThe factor of two represents keys and values. Models using grouped-query attention (GQA) or multi-query attention (MQA) have fewer KV heads than standard multi-head attention, so they can cache each token more efficiently. This is why parameter count alone cannot predict context memory.
Global-attention tokens add memory linearly.
Within a fixed full-attention architecture and cache precision, doubling active context approximately doubles KV-cache memory. It does not double weight memory. Local or sliding-attention layers can plateau after their configured window while global layers continue to grow, so hybrid architectures need a layer-aware estimate.
Relative full-attention illustration for one fixed model and cache format; hybrid/sliding models can scale differently.
A model’s advertised context window describes its supported input envelope, not a recommendation to allocate the maximum for every deployment. Set the runtime maximum to match product requirements and protect it with request limits.
Count active tokens, not users.
Concurrency multiplies cache when multiple sequences are active at once. For a full-attention model, two sequences at 16K tokens have approximately the same raw cache footprint as one 32K sequence at the same precision. Hybrid local/global attention, prefix reuse, mixed prompt lengths, generated tokens, and block allocation can change that comparison, so average and peak active sequences are more useful than a simple registered-user count.
Start with one active sequence and a realistic personal context.
Weights remain shared; cache grows across four active sequences.
Similar full-attention token count, but latency and prefill behavior differ.
Five levers before adding GPUs.
- 01Cap context deliberately
Do not expose the model’s maximum when the product only needs a smaller window.
- 02Control concurrent active tokens
Use scheduling and admission limits to protect the peak cache budget.
- 03Use a supported lower-precision cache
FP8 can reduce cache memory, but confirm runtime and hardware support.
- 04Choose an efficient attention architecture
GQA and MQA models often use less cache per token.
- 05Test prefix caching carefully
Shared prefixes can improve efficiency in suitable workloads, but behavior is runtime-specific.
Combine weights, cache, workspace, and operating headroom.
Context window and VRAM questions.
Does a larger context window require more VRAM?+
Yes. During inference, the KV cache retains attention keys and values for active tokens. Longer active sequences therefore consume more GPU memory, although the amount per token varies by model architecture and cache precision.
Does maximum context use memory even when the prompt is short?+
Not always. Paged-attention runtimes can allocate cache in blocks as sequences grow, while other configurations may reserve larger pools up front. Configure the maximum sensibly and inspect the behavior of your chosen runtime.
What happens when concurrency doubles?+
If two sequences have the same active length, their KV cache requirement is approximately additive. Doubling concurrent active tokens can roughly double the cache component, though weights are shared.
Does FP8 KV cache cut total VRAM in half?+
It can approximately halve the KV-cache component compared with FP16, not the model weights or all runtime memory. Support, accuracy, and performance depend on model, GPU, and runtime.
Why do GQA models use less KV cache?+
Grouped-query attention uses fewer key/value heads than query heads, reducing the cached key and value tensors per token compared with standard multi-head attention.
Turn the cache budget into a fit plan.
Set context, concurrency, and cache precision in the worksheet, then compare the resulting memory envelope with the hardware profiles.