High throughput. This profile uses an 85% planning ceiling, leaving approximately 12.0 GB outside the recommended workload envelope.
These estimates use one 8K sequence and include runtime workspace. “Multi-GPU” indicates the minimum memory-equivalent device count, not a guarantee that the model or runtime supports that topology efficiently.
NVIDIA H100 is positioned as high throughput hardware with 80 GB of memory. It is a practical candidate when the full serving envelope stays below 68.0 GB per device and the runtime supports the selected precision. For latency or throughput planning, memory fit is only the first gate: benchmark the exact model, batch shape, framework, and provider topology.
Advertised capacity is not always fully available to model serving. Drivers, runtime kernels, CUDA or ROCm graphs, fragmentation, adapters, multimodal encoders, speculative decoding, and provider reservations can all move the real boundary. Treat the matrix as a shortlist, then validate on the target stack.
The sample rate on this page is a planning input, not a live offer. Recheck context, precision, concurrency, and GPU count before committing capacity.