What GPU does a 70B LLM need?

A dense 70B model stores roughly 130 GiB of raw weights at BF16, about 65 GiB at 8-bit, and about 32.6 GiB at a theoretical 4-bit floor. Loaded memory is usually higher because quantized formats add scales and metadata, while some modules may remain at higher precision.

That is why a file-size comparison is not enough. ModelFit applies format overhead before adding cache and runtime workspace, then reserves operating headroom instead of planning against every advertised gigabyte.

KV cache holds attention keys and values for active tokens. Its size depends on layer count, KV heads, head dimension, cache precision, context length, and the number of active sequences. Moving from one 8K request to four 32K requests can make cache memory a first-order part of the deployment.

Use the context accepted by the service and the peak active sequences maintained by the scheduler. The model card's maximum context is a limit, not a sensible default.

An 80 GB GPU may carry many 70B 4-bit workloads, but the fit can become tight after cache and runtime allocations. BF16 normally requires multiple high-memory GPUs. Tensor parallelism shards most weights, but cache sharding stops at the model's KV-head count and runtime allocations do not divide perfectly.

Memory fit is only the first gate. Interconnect bandwidth, collective communication, kernel support, and host topology determine whether a valid memory plan delivers useful throughput.

• Use one 80 GB GPU only after validating the exact quantized checkpoint.
• Use 2× or 4× high-memory GPUs when precision, cache, or concurrency exceeds the single-device envelope.
• Benchmark latency and throughput on the provider topology before committing capacity.

Choose the actual checkpoint and runtime, set the intended context and concurrency, calculate the per-device allocation, and keep headroom. Then run a cold load, sustained generation, and peak-concurrency test while recording device memory and failure behavior.

Treat a calculator result as a shortlist. The final answer belongs to the exact model revision, quantization backend, runtime version, launch flags, GPU form factor, and provider configuration.