LLM VRAM Calculator: From Model Weights to a GPU Plan

LLM VRAM Calculator: From Model Weights to a GPU Plan

An llm vram calculator is useful only when its inputs describe your workload. Parameter count tells you the weight payload; it does not tell you whether an inference server can sustain your context length and concurrent requests. Start with weights, add KV cache and runtime memory, then reserve operating headroom.

Use the ModelFit calculator to explore those inputs. This worksheet shows how to audit the estimate before choosing hardware. It covers inference, not training or fine-tuning.

01 / Set the assumptions first

Record the exact checkpoint, weight format, KV cache dtype, maximum total tokens per sequence, concurrent resident sequences, GPU count, and runtime version. Total tokens means prompt plus generated tokens. A 4,096-token prompt followed by 1,024 generated tokens needs a 5,120-token budget.

This article's example uses Qwen2.5-7B-Instruct, one GPU, BF16 weights, a two-byte KV cache, two resident sequences, and 8,192 total tokens per sequence. There is no prefix-sharing discount, CPU offload, speculative decoder, or multimodal encoder. Runtime memory is an explicit assumption rather than a measurement. Sources were checked on October 14, 2026.

The Qwen model card reports approximately 7.61 billion parameters. Its configuration supplies 28 layers, four KV heads, 28 query heads, and hidden size 3,584. Head dimension is therefore 3,584 / 28 = 128. Use KV heads, not query heads, when sizing this grouped-query attention cache.

02 / Read an LLM VRAM calculator as a memory ledger

For a single GPU, use the following planning model, with all components expressed in bytes:

M = W + Q + K + R
Required capacity = M / (1 - h)

Dividing by 0.85 leaves 15% of capacity unused. Multiplying the subtotal by 1.15 adds 15% of the subtotal, which is a different policy. State which convention your tool uses. For broader background, read how to calculate LLM VRAM.

03 / Quantization changes more than the bit count

BF16 and FP16 payloads use two bytes per parameter. An ideal four-bit payload uses half a byte. That ideal excludes scales, zero points where used, block metadata, padding, and tensors retained at higher precision.

For an illustrative group format with one two-byte scale and one two-byte zero point per 128 weights, metadata adds 4 / 128 = 0.03125 bytes per quantized weight. This is a hypothetical format calculation, not a claim about every AWQ, GPTQ, GGUF, or bitsandbytes checkpoint. Some formats use different groups or omit zero points.

The Hugging Face quantization documentation distinguishes storage and computation settings. Selecting four-bit memory in a calculator does not create a compatible quantized checkpoint. Check the model repository and runtime's supported loader before launch. Quantizing weights also does not automatically quantize the KV cache.

04 / Calculate KV cache from resident tokens

For conventional full-attention transformer layers with equal KV dimensions:

K = 2 × L × Hkv × D × Bkv × sum(Ti)

L is the layer count, Hkv the KV-head count, D the head dimension, Bkv bytes per cached element, and Ti the resident token count of sequence i. The leading two accounts for keys and values. If S sequences each reach T tokens, sum(Ti) becomes S × T.

For this architecture, two-byte caching costs 2 × 28 × 4 × 128 × 2 = 57,344 bytes per token across all layers. Two sequences at 8,192 tokens require 939,524,096 bytes, or 0.875 GiB. This is a derived tensor-size estimate; block allocation and implementation details can increase the actual reservation.

Doubling resident tokens doubles this cache estimate. Weight memory stays unchanged. Sliding-window attention, hybrid architectures, latent attention, cache sharing, and offloading require a different accounting model.

05 / Reproduce the worked example

Use a rounded parameter count, so the result is a planning estimate rather than an exact checkpoint allocation:

gib = 2**30
weights = 7.61e9 * 2
kv = 2 * 28 * 4 * 128 * 2 * 8192 * 2
runtime = 2 * gib  # assumed allowance; replace with a measurement
subtotal = weights + kv + runtime
capacity = subtotal / 0.85
for label, value in [("weights", weights), ("KV", kv),
                     ("subtotal", subtotal), ("capacity", capacity)]:
    print(label, round(value / gib, 3), "GiB")

The two-GiB allowance covers a budget for activations, temporary workspaces, allocator behavior, and graph-related buffers. It is not a measured vLLM requirement. Re-run the arithmetic with the actual profiled allowance.

06 / Keep GB and GiB separate

One GB is 1,000,000,000 bytes; one GiB is 1,073,741,824 bytes. A hypothetical capacity of exactly 24 decimal GB equals 22.352 GiB. Do not compare a GiB estimate directly against a GB number without checking units.

NVIDIA specifies 24 GB for the L4. That makes it a candidate capacity class for the example, not a fit guarantee. Read device-reported total and available bytes on the actual instance; reserved memory and other processes matter. Compare candidates in the ModelFit hardware directory.

07 / Validate the server before reserving production capacity

vLLM's engine arguments expose model-length, sequence-count, dtype, and memory-budget controls. A starting template for this workload is:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --max-num-seqs 2 \
  --gpu-memory-utilization 0.85

Verify these flags against your installed release and inspect the resolved KV cache dtype. The utilization flag controls the engine's memory budget; it does not prove that the worksheet's runtime assumption is sufficient. vLLM may reserve more cache than the two sequences actually consume.

1. Record startup memory profiling and available KV capacity.
2. Send representative requests with the intended prompt and output limits.
3. Test simultaneous long requests and prefill peaks, not only short decode loops.
4. Check peak memory, OOMs, preemption, latency, and throughput.
5. Change one input at a time and repeat the fit check.

The vLLM tuning guide discusses cache pressure and preemption. Memory fit does not establish acceptable performance. Tensor parallelism also needs per-device accounting: replicated buffers, KV-head placement, and communication allocations mean total memory cannot always be divided evenly by GPU count.

Once your llm vram calculator worksheet and load test support a configuration, check capacity on Flatkey Compute. Confirm the GPU configuration and availability before reserving it.

08 / Questions to settle before deployment

Can a model fit at four-bit but fail at long context? Yes. The weight payload shrinks, while cache growth remains tied to resident tokens and cache precision.

Does enough VRAM mean the deployment is fast enough? No. Memory bandwidth, compute, interconnects, batching, and the workload still determine performance. This worksheet makes no benchmark prediction.

Can this formula size training? No. Training needs separate accounting for gradients, optimizer states, retained activations, and the training strategy.

What should I save with the result? Checkpoint revision, runtime version, all dtypes, context and concurrency limits, measured overhead, device capacity, and load-test results. A number without those assumptions is not a deployment plan.

Source note: ModelFit provides planning estimates. Flatkey is the linked deployment destination. Hardware facts and architecture inputs above come from NVIDIA and Qwen; calculated values and runtime allowances are explicitly identified.