Originally published at mrsaynothing.dev.
Last week a reader asked why their GGUF file 404s inside vLLM on a CPU box while the same file runs fine in Ollama. Short answer, before anything else: yes, vLLM runs GGUF — through an official plugin, on GPU only. The plugin is vllm-gguf-plugin, the syntax is repo:quant_type, and the moment you try it on a CPU you are outside the supported hardware table. Everything below comes from the vLLM documentation and the vllm-project/vllm repo record (92,000+ stars since February 2023) — the docs, not my benchmark rig.
How do you serve a GGUF model with vLLM?
Two steps: install the plugin, then point vLLM at the model. GGUF support no longer ships inside core vLLM — the docs note it “has migrated to OOT vllm-gguf-plugin“, so a plain pip install vllm is not enough:
uv pip install vllm-gguf-plugin
# Directly from Hugging Face, repo_id:quant_type format:
vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M \\
--tokenizer Qwen/Qwen3-0.6B
# Or a local file you already downloaded:
vllm serve ./Qwen3-0.6B-Q4_K_M.gguf \\
--tokenizer Qwen/Qwen3-0.6B
Enter fullscreen mode Exit fullscreen mode
The --tokenizer flag is not decoration. The official docs recommend the base model’s tokenizer because GGUF tokenizer conversion is “time-consuming and unstable, especially for some models with large vocabularies”. Skip it and you trade a one-line flag for a long, flaky startup.
Two GPUs, one model: add --tensor-parallel-size 2 to shard the same GGUF across both cards — tensor parallelism works with GGUF the same as any other format.
Why does vLLM refuse GGUF on CPU?
This is the part that surprises people. GGUF’s reputation is CPU-first — it is the format llama.cpp built its name on, running models on laptops and Raspberry Pis. Inside vLLM the situation inverts. The official quantization hardware compatibility table marks GGUF:
Hardware GGUF in vLLM NVIDIA Volta / Turing / Ampere / Ada / Hopper supported AMD GPU supported Intel GPU not supported x86 CPU not supported Arm CPU not supportedSource: vLLM quantization docs. The reason is architectural: vLLM’s GGUF path dequantizes blocks into GPU kernels built for batched serving. There is no CPU kernel behind it, because vLLM is a serving engine, not a laptop toy. If your machine has no GPU, no flag will help — use llama.cpp.
What breaks: the honest limits list
The same docs page carries a warning worth quoting verbatim: “GGUF support in vLLM is highly experimental and under-optimized.” Concretely:
- Quant coverage is narrower than llama.cpp. The K-quants people actually download (Q4_K_M and friends) work; exotic schemes may not. llama.cpp remains the reference implementation for the format — we mapped the whole quant family in our GGUF quantization guide.
- Architecture support trails. New model families land in llama.cpp first; the plugin follows later.
- No mmap-style lazy loading. llama.cpp memory-maps the file; vLLM loads it like any other checkpoint.
- It is a memory-footprint feature first. The docs frame GGUF as a way to shrink VRAM use, not a throughput play.
None of this is hidden. It is all in the first paragraph of the official GGUF page — which is more than most experimental features get.
vLLM or llama.cpp for GGUF: which wins?
Different tools that happen to read the same file:
vLLM + GGUF llama.cpp CPU inference no yes, first-class Concurrent users continuous batching, built for it limited Quant coverage subset, experimental the reference Setup vLLM + plugin one binary Best for one GPU, many users one user, any hardwareIf you serve a model to a team from a single GPU, vLLM + GGUF lets you reuse the same Q4_K_M files the rest of the local-LLM world shares. For the full engine comparison, see llama.cpp vs Ollama and how to run GGUF models locally.
The one-line rule: same file, opposite instincts — llama.cpp treats GGUF as the format, vLLM treats it as an option.
More field notes at mrsaynothing.dev · code at GitHub · say hi: [email protected]
This ships daily at mrsaynothing.dev — the full archive, every piece in 21 languages, zero missed days. New posts land in the newsletter the moment they ship: join it here. Code at GitHub.