How quantization changes LLM VRAM

BF16 and FP16 use two bytes per parameter. INT8 begins around one byte, while a 4-bit representation begins around half a byte. Dividing parameter count by bit width gives a useful first estimate, but not the loaded footprint.

Quantized files store scales, zero points, group metadata, and sometimes higher-precision output or normalization layers. ModelFit therefore uses an explicit packing coefficient rather than promising that Q4 is exactly one quarter of BF16 memory.

Quantization primarily changes model weights. KV cache precision is a separate decision, and runtime workspace, graph capture, adapters, multimodal encoders, and allocator fragmentation may remain largely unchanged.

For small models or long-context workloads, the non-weight portion can be large enough that moving from 8-bit to 4-bit saves less total VRAM than the bit ratio suggests.

A Q4 label can refer to different packing layouts and kernels. A checkpoint must match the backend expected by vLLM, llama.cpp, TensorRT-LLM, or another runtime. Unsupported combinations may fail to load, silently dequantize, or deliver poor performance.

Use quality benchmarks that match the workload. Weight-only quantization may preserve general quality while affecting coding, reasoning, multilingual, or tool-use behavior differently.

• Confirm the checkpoint format and quantization backend.
• Verify that the target GPU has supported kernels.
• Measure output quality and throughput, not only successful loading.

Use BF16 when quality, compatibility, and predictable kernels matter more than capacity cost. Use 8-bit when the platform supports efficient kernels and a moderate memory reduction is sufficient. Use 4-bit when fitting a larger model is the primary constraint and the quality/runtime trade-off has been validated.

Keep the calculation auditable: record the checkpoint, format, runtime, device, context, concurrency, and measured peak memory alongside the estimate.