LLM Inference Latency: Why Your 7B Model Gets 15 tok/s on a T4 but 3,500 tok/s on an H100

`NVIDIA’s spec sheet says the H100 delivers 989 TFLOPS of FP16 compute. The A100: 312. The T4: 65. Simple arithmetic says the H100 is 15× faster.

So a 7-billion-parameter LLM should be 15× faster on an H100, right?

It’s 150× faster.

The T4 struggles at ~15 tok/s. The H100 cruises at ~2,200 tok/s — and with continuous batching, north of 3,500.

The 15× TFLOPS gap doesn’t explain the 150× throughput gap. The missing variable is the one thing NVIDIA’s marketing pages bury on line three of the spec table:

Memory bandwidth.

The T4 has 300 GB/s. The H100 has 3,350 GB/s. That’s an 11× gap in raw bandwidth, and closer to 100–150× in effective throughput once cache size and clock speed are factored in.

This post traces the arithmetic from first principles: why memory bandwidth is the bottleneck, how quantization turns it into a lever, and what tok/s you should actually expect across 12 real GPU × model combinations.

1. Memory Bandwidth: The Bottleneck Nobody Talks About

Autoregressive LLM inference has two phases:

Phase What Happens Bottleneck Prefill Process the input prompt (all tokens in parallel) Compute Decode Generate output tokens (one at a time) Memory bandwidth

The decode phase dominates total latency for any response longer than a few tokens. And in decode, the GPU spends 98%+ of its time doing nothing — waiting for the next chunk of model weights to arrive from VRAM.

The Arithmetic, Step by Step

Each generated token requires reading every single parameter from VRAM. For a 7B model at FP16:

`plaintext
Weights: 7,000,000,000 params × 2 bytes = 14 GB per token

H100: 14 GB ÷ 3,350 GB/s = 4.18 ms → 239 tok/s (theoretical)
T4: 14 GB ÷ 300 GB/s = 46.7 ms → 21 tok/s (theoretical)
`

Now consider the compute cost: ~14 TFLOPs per token. The H100’s 989 TFLOPS could execute that 70 times in 4.18 ms. The compute units finish their work and sit idle, waiting for the next 14 GB of weights to trickle in.

You cannot compute faster than you can read.

Every meaningful LLM optimization — quantization, KV cache compression, FlashAttention, speculative decoding — is fundamentally about reducing bytes moved per token.

The Full GPU Landscape

GPU Bandwidth 7B FP16 7B INT4 70B INT4 VRAM NVIDIA B200 8,000 GB/s 571 2,286 229 192 GB NVIDIA H200 4,800 GB/s 343 1,371 137 141 GB NVIDIA H100 3,350 GB/s 239 957 96 80 GB NVIDIA A100-80GB 2,039 GB/s 146 583 58 80 GB NVIDIA RTX 4090 1,008 GB/s 72 288 28.8 24 GB NVIDIA L40S 864 GB/s 62 247 — 48 GB NVIDIA T4 300 GB/s 21 86 — 16 GB

Theoretical tok/s at 100% bandwidth utilization. Real-world: multiply by 0.6–0.85. “—” = doesn’t fit in VRAM.

2. Quantization: 4× Faster Without Changing GPUs

If your bottleneck is bytes-per-token, and you can’t change the GPU’s memory bandwidth, there’s exactly one lever left: reduce bytes per parameter.

Precision Bytes/Param 7B Model 70B Model Speedup Quality FP16 2.0 14 GB 140 GB 1× Reference INT8 1.0 7 GB 70 GB ~1.8× Negligible INT4 (GPTQ/AWQ) 0.5 3.5 GB 35 GB ~3.5× 1–3% perplexity INT3 0.375 2.6 GB 26 GB ~4.5× 3–8% loss

The rule of thumb: the larger the model, the harder it is to break with quantization. A 405B model at INT2 often outperforms a 70B model at FP16 on knowledge tasks — despite using fewer bytes per token.

What quantization doesn’t improve: time-to-first-token (prefill latency). Prefill is compute-bound. Quantization can even make it slower due to dequantization overhead. The gains apply almost entirely to the decode phase.

3. Real Benchmarks

Theoretical ceilings are clean math. Reality includes framework overhead, attention kernel efficiency, and KV cache management. These numbers are from vLLM 0.6.x, continuous batching, bare-metal H100 instances — mid-2026.

Model Precision GPU(s) Batch 1 Batch 8 Batch 32 Llama 4 Scout (8B) FP16 1× H100 185 1,200 3,200 Llama 4 Scout (8B) INT4 1× H100 620 3,800 8,500 Llama 4 Scout (8B) INT4 1× RTX 4090 95 310 — Mistral Small 3 (7B) FP16 1× H100 195 1,250 3,500 Llama 4 Maverick (70B) FP16 2× H100 35 190 480 Llama 4 Maverick (70B) INT4 1× H100 78 440 1,050 Mistral Large 2 (123B) FP16 4× H100 15 70 140 DeepSeek-V3 (671B MoE) INT8 8× H100 18 85 180 Qwen 2.5 (72B) INT4 1× H100 72 410 960 Phi-4 (14B) INT4 1× RTX 4090 78 — — Gemma 3 (27B) INT4 1× RTX 4090 55 — —

“TP=N” = Tensor Parallelism across N GPUs. “—” = VRAM insufficient at listed context length. Output token decode only.

The headline: INT4 turns a 70B model from “needs 2 GPUs and is still slow” into “runs on 1 GPU and is faster than FP16 on 2.”

4. Batch Size: The Latency–Throughput Tradeoff

Batching fills the idle time between memory reads. While one request waits for weights, another’s compute can run. But per-request latency climbs with batch size.

Workload Batch Latency Target Real-time chat / copilot 1–4 <200ms TTFT Code completion 1–2 <20ms/tok Support chatbot 4–16 <1s TTFT Summarization (async) 16–32 Throughput priority Dataset labeling 32–128 Throughput only

Continuous batching (vLLM, TensorRT-LLM, SGLang) is the critical innovation. Traditional static batching waits for all requests to finish. Continuous batching evicts completed requests and admits new ones at every decode step — eliminating the “one slow request holds up 31 others” problem.

If you deploy without continuous batching, you’re leaving 50–70% of your GPU throughput on the table.

5. Multi-GPU: The Scaling Efficiency Problem

When one GPU isn’t enough, you split the model with tensor parallelism. But all-reduce communication between GPUs eats into your gains:

Configuration Batch 32 tok/s Per-GPU Efficiency NCCL Tax 2× A100 (TP=2) 380 92% ~8% 4× A100 (TP=4) 610 80% ~20% 8× A100 (TP=8) 840 63% ~37%

At TP=8, nearly 40% of interconnect bandwidth goes to synchronization.

This is why expert-parallel inference for MoE models is revolutionary: each expert lives on a dedicated GPU, and only the active experts participate in each forward pass. DeepSeek-V3 has 671B total parameters but only 37B active — so it’s faster than a 70B dense model despite having 10× more total weights.

The One-Sentence Takeaway

LLM inference throughput = GPU memory bandwidth ÷ bytes per token. Everything else is overhead management.

Try the interactive calculator at jslet.com/llm-inference-latency — plug in your model, GPU, and quantization level. All 100% client-side, no signup.
`