Here is the short version: the model weights are the smallest GPU-memory surprise you’ll hit. A 7B model in FP16 needs about 14GB just for weights, but the KV cache — the per-request memory that grows with context length and batch size — is what actually decides whether your setup survives real traffic. Most “run an LLM on your GPU” tutorials load the weights, run one short prompt, and declare victory. Then you send a 6,000-token document at a batch of eight and it OOMs.
I’ve set this up enough times to know the failure isn’t random. It’s arithmetic you can do before you rent the GPU. This post is that arithmetic.
Why does a model that “fits” still run out of memory?
GPU memory for inference is four separate buckets, and tutorials only mention the first:
- Model weights — fixed, predictable, load once.
-
KV cache — scales with
context_length × batch_size. This is the one that bites. - Activations and CUDA overhead — the runtime itself, plus temporary tensors during a forward pass.
- Fragmentation — memory that’s technically free but unusable because it’s in the wrong-sized holes.
The weights number is the one everyone quotes because it’s easy. Parameter count × bytes-per-parameter:
Precision Bytes/param 7B model 13B model 70B model FP16/BF16 2 ~14 GB ~26 GB ~140 GB INT8 1 ~7 GB ~13 GB ~70 GB INT4 (GPTQ/AWQ/GGUF Q4) ~0.5 ~4 GB ~7 GB ~38 GBSo a 7B model in 4-bit “fits” on a 24GB card with 20GB to spare, and the tutorial ends there. That leftover 20GB is not spare — it’s your working budget for everything in buckets 2 through 4, and it disappears faster than you’d guess.
Takeaway: Weights tell you if the model loads; they tell you nothing about whether it serves traffic.
How big is the KV cache, really?
The KV cache stores the key and value tensors for every token already in the context so the model doesn’t recompute them each step. Its size, per request, is roughly:
kv_bytes = 2 × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_element
Enter fullscreen mode Exit fullscreen mode
The 2 is for keys and values. Note num_kv_heads, not the full attention-head count — modern models use grouped-query attention (GQA), which is the single biggest reason KV cache is smaller than older formulas suggest. Worth checking your model’s config, because it swings the number by 4-8×.
Worked example, Llama-3-8B-class architecture (32 layers, 8 KV heads, head_dim 128, FP16):
per_token = 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes ≈ 128 KB/token
Enter fullscreen mode Exit fullscreen mode
That’s about 1 GB for a single 8,192-token request. Now the part tutorials skip: this is per concurrent request. Serve a batch of 16 at that context length and you’ve spent ~16 GB on KV cache alone — more than the weights. On a 24GB card holding a 4-bit 8B model (~5 GB with overhead), you have maybe 17-18GB left, and you just watched a modest batch eat all of it.
Two levers shrink this:
-
Shorter
max_model_len. If you don’t need 32k context, don’t provision for it. Serving frameworks pre-reserve cache based on this ceiling. - KV-cache quantization (FP8 or INT8 KV). Roughly halves the cache at a small quality cost. vLLM and TGI both support it; it’s off by default.
Takeaway: Budget KV cache as per_token_KB × max_context × expected_concurrency before you pick a card — it often dwarfs the weights.
What does the runtime itself cost?
Even before a single token, loading CUDA and your inference framework claims memory. The CUDA context alone is typically a few hundred MB to over a gigabyte depending on driver and GPU. PyTorch’s caching allocator reserves more. Frameworks like vLLM deliberately grab a large fraction of remaining VRAM up front (controlled by gpu_memory_utilization, default 0.9) to manage the KV cache themselves — which is great for throughput but means “nvidia-smi shows 90% used” is expected, not a leak.
Then there’s fragmentation. Naive allocators hand out one contiguous block per request’s KV cache. When requests of varying lengths come and go, you get Swiss-cheese memory: 4GB free, but no single 2GB hole. This is exactly the problem vLLM’s PagedAttention solves — it pages the KV cache like an OS pages RAM, so non-contiguous free memory is usable. If you’re comparing serving frameworks, this is the practical reason vLLM sustains higher concurrency than a plain transformers loop on the same card.
Takeaway: Reserve 1-2GB for runtime overhead as a floor, and prefer a paged-attention serving stack the moment you have concurrent requests.
Which framework for which situation?
These are the four I actually reach for, with honest limitations:
Tool Best for Real drawback Ollama Local dev, single user, “just run it” Not built for high-concurrency serving; batching is limited llama.cpp CPU/GPU hybrid, low VRAM, edge boxes GGUF quant setup is fiddly; peak throughput trails GPU-native servers vLLM Production serving, high concurrency Heavier setup; needs a proper CUDA GPU; startup VRAM grab surprises people TGI (Text Generation Inference) Production serving with HF ecosystem Tighter model-support window; also GPU-hungry at startFor a first self-host on a single 24GB consumer card (RTX 3090/4090), a 7-8B model quantized to 4-bit via Ollama or llama.cpp is the reliable starting point. When you move past one user, switch to vLLM and set max_model_len and gpu_memory_utilization deliberately rather than accepting defaults.
Takeaway: Match the framework to concurrency, not to model size — the model fits on the card either way; only one of them survives real traffic.
A quick sizing pass before you commit
Rather than guess, do this arithmetic with your actual model config (config.json has num_hidden_layers, num_key_value_heads, head_dim or hidden_size / num_attention_heads):
def vram_estimate_gb(params_b, bytes_per_param,
num_layers, num_kv_heads, head_dim,
max_ctx, concurrency,
kv_bytes=2, overhead_gb=2.0):
weights = params_b * 1e9 * bytes_per_param / 1e9
per_tok = 2 * num_layers * num_kv_heads * head_dim * kv_bytes
kv = per_tok * max_ctx * concurrency / 1e9
return round(weights + kv + overhead_gb, 1)
# Llama-3-8B, 4-bit weights, FP16 KV, 8k context, 8 concurrent
print(vram_estimate_gb(8, 0.5, 32, 8, 128, 8192, 8)) # ~14.5 GB
Enter fullscreen mode Exit fullscreen mode
Bump concurrency to 24 and that same setup crosses 30GB — past a 24GB card. This ten-line function has saved me more grief than any benchmark, because it turns “will it work?” into a number you check before spending money on a bigger GPU or a cloud instance. Treat it as an estimate with ±15% slack for allocator behavior, not a guarantee.
Bottom line
If you’re self-hosting your first LLM, size for the KV cache and overhead, not just the weights — the weights are the part that always fits. For a single user on one consumer GPU, a 4-bit 7-8B model under Ollama or llama.cpp is the safe first step. The moment you have concurrent requests, move to vLLM or TGI, cap max_model_len to what you actually need, and consider FP8 KV cache. And before you upgrade to a pricier card because you hit OOM, run the arithmetic above — nine times out of ten the fix is a smaller context ceiling or KV quantization, not more VRAM.
답글 남기기