Plan multi-GPU LLM inference with tensor parallelism, per-device VRAM, KV-cache sharding, runtime workspace, interconnect, and validation steps.
Use the estimates as a transparent planning envelope, then validate them on the exact checkpoint, runtime, GPU topology, and provider configuration.
Calculate per-device memory
A first tensor-parallel estimate divides weight memory by the GPU count. KV cache can also shard, but only up to the model's KV-head count, while runtime workspace keeps a per-device floor. ModelFit exposes these buckets separately so the division is not treated as perfect.
Compare the resulting subtotal against 85% of each GPU's advertised capacity. The remaining 15% is operating headroom, not unused waste: it protects the deployment from allocators, kernels, graph capture, fragmentation, and workload variance.
Interconnect can dominate
Tensor-parallel layers exchange data during inference. The more devices involved, the more the deployment depends on NVLink, NVSwitch, PCIe topology, and collective communication behavior. A configuration that fits in memory can still miss latency or throughput targets.
Provider pages may list a GPU count without describing whether the devices share a host or a high-bandwidth fabric. Confirm the actual topology before treating an 8-GPU offer as equivalent to another 8-GPU offer.
Do not scale cache past the architecture
Grouped-query and multi-query attention use fewer KV heads than attention heads. Cache sharding cannot usefully extend beyond the number of KV heads represented by the architecture. ModelFit caps the cache shard count at that boundary.
Schedulers, prefix caching, speculative decoding, and multimodal components can create additional allocations. Include the exact serving features used in production when validating the plan.
- Record the model revision and KV-head count.
- Test the intended cache precision and context length.
- Measure peak memory on every device, not only device zero.
Validate failure behavior
Run cold startup, steady generation, peak concurrency, and long-context tests. Observe load time, peak memory, tokens per second, latency percentiles, collective timeouts, and out-of-memory recovery.
Use the smallest topology that meets both the memory envelope and service target. Additional GPUs are not automatically safer or cheaper once communication and idle capacity are included.
Use the matching calculator.
Carry the model, precision, context, concurrency, and topology assumptions into an explicit worksheet.
Open the topology calculator →Common planning questions.
Does adding two GPUs double usable VRAM?+
It can nearly double the sharded weight capacity, but cache and runtime allocations do not always divide perfectly, and communication overhead increases.
What is tensor parallelism?+
Tensor parallelism splits model operations across GPUs within each layer, allowing large weights to be served across multiple devices.
Do I need NVLink for multi-GPU inference?+
Not every workload requires it, but high-bandwidth interconnect often matters for latency and throughput as tensor-parallel communication grows.
Turn the estimate into a GPU shortlist.
Compare memory profiles, then check the current price index and live capacity before renting.