Compare NVIDIA A100 and H100 for LLM inference by VRAM capacity, model fit, precision support, throughput, topology, and GPU-hour price.
Use the estimates as a transparent planning envelope, then validate them on the exact checkpoint, runtime, GPU topology, and provider configuration.
Equal capacity does not mean equal performance
An 80 GB A100 and an 80 GB H100 expose a similar advertised memory capacity, so a pure fit calculation may return the same verdict. ModelFit still applies the same 85% planning ceiling because drivers, kernels, graph capture, and workload variance need room on both devices.
H100 generally offers newer tensor cores, stronger low-precision support, and higher memory bandwidth. A100 remains widely available and can be cost-effective when the workload already meets latency and throughput targets.
Start with the workload envelope
Calculate weights, KV cache, workspace, and headroom before comparing speed. If the workload exceeds the single-device envelope, the decision becomes a topology question rather than an A100-versus-H100 benchmark.
For workloads that fit, record input length, output length, batch or concurrency, cache precision, and service-level target. Tokens per second without the same workload shape is not a useful comparison.
Price must be tied to time and source
GPU-hour prices vary by region, provider, availability class, commitment term, and host topology. ModelFit uses the Flatkey public price index where a comparable model is covered, showing its timestamp, range, sample count, and methodology.
A lower hourly rate can still be more expensive per generated token if the device delivers materially lower throughput. Compare cost per successful workload or cost per million generated tokens after benchmarking.
- Check whether the quote is on-demand, spot, or reserved.
- Confirm PCIe versus SXM form factor and multi-GPU interconnect.
- Use the same runtime and request mix in the benchmark.
A practical decision rule
Choose A100 when it meets the service target with a meaningful cost advantage and the required precision/runtime is mature. Choose H100 when lower latency, higher concurrency, FP8 support, or faster batch processing compensates for the higher rate.
Do not decide from peak specifications alone. Fit the workload, benchmark both profiles, and make the commercial decision from measured throughput, error rate, and total cost.
Use the matching calculator.
Carry the model, precision, context, concurrency, and topology assumptions into an explicit worksheet.
Test the workload on both profiles →Common planning questions.
Can A100 and H100 fit the same models?+
Comparable 80 GB configurations have similar capacity, so many memory-fit results are the same. Performance and precision support can differ substantially.
Is H100 always cheaper per token?+
No. It may deliver more throughput, but the hourly premium, utilization, workload shape, and provider configuration determine cost per token.
Should I compare PCIe and SXM as the same GPU?+
No. Power limits, bandwidth, and interconnect topology can materially affect performance, especially for multi-GPU inference.
Turn the estimate into a GPU shortlist.
Compare memory profiles, then check the current price index and live capacity before renting.