local-llm: A Field Report on Running SOTA Models on Your Own Hardware

The most useful thing in jamesob/local-llm is not the GPU shopping list. It is the fifteen or so BIOS settings, kernel flags, and PCIe hacks that stand between owning four RTX PRO 6000 cards and actually getting them to run a 594-billion-parameter model at a claimed 80 tokens per second. The author, James O’Beirne, frames the repo plainly: “Everything I know about running LLMs locally.” What that turns out to mean is a field report on the parts of the job that no product page warns you about.

Two budgets, two very different machines

The guide splits along price. For roughly $2k, it recommends two used RTX 3090s for 48GB of combined VRAM, enough to run Qwen3.6-27B plus local speech-to-text with whisper-large-v3. The author is candid that this tier gets you “pretty far,” and singles out local STT as something he actually reaches for daily, in part because he feels comfortable using it in a way he does not with a hosted equivalent. The STT runner only assumes about 11GB of VRAM, so it is the low-friction entry point in the whole repo.

The $40k tier is where the writing gets interesting. Four RTX PRO 6000 Blackwell cards give 384GB of VRAM, which the author says lands you “something pretty close to Claude Opus” via a quantized GLM-5.2 variant. Rather than pair those cards with an expensive PCIe5 and DDR5 platform, he built a last-gen EPYC Milan system almost entirely from eBay parts: an ASRock Rack ROMED8-2T board, a 16-core 7313P, and 128GB of DDR4 ECC, for a base system total of $5,587. The stated logic is to spend money on VRAM, where it counts, and cut it everywhere else. RAM prices as of July 2026, he notes, made the DDR4 route the sane one.

Where the money did not go, and where the pain did

To let the four cards talk to each other without routing every allreduce through the CPU root complex, the build uses an indie PCIe Gen4 switch from c-payne.com, a Microchip Switchtec PM40100, wired in over paired SlimSAS 8i cables. This is the part of the repo that reads like hard-won experience rather than a parts list.

A few specifics stand out:

• The upstream link kept training at Gen4 x8 because bifurcation was splitting the slot; the fix was forcing x16 in BIOS with both SlimSAS cables connected.
• lspci reported the links as downgraded to Gen1, which turned out to be cosmetic. ASPM was idling the links down to 2.5GT/s, and disabling ASPM removed the scary reading while the links ran Gen4 under load.
• Without iommu=off in the GRUB command line, NCCL hangs on multi-GPU peer-to-peer.
• ACS, enabled by default, silently bounces peer-to-peer traffic through the CPU root port and negates the switch entirely. The repo ships a setpci script and a systemd oneshot to strip ACS on every boot, with a verification step: nvidia-smi topo -m should read PIX between all four GPUs.

There is also a note that the author ordered too few SAS cables, bought a slightly different one from Amazon to fill the gap, and spent time chasing problems that traced back to that mismatch. And to avoid installing a 220V circuit, he runs roughly $46k of silicon on a single 110V line by power-capping the cards, which he flags as “probably unwisely.” This is the honesty that makes the document worth reading.

What you actually get

Beyond the hardware, the repo includes ready-to-run runners/ with Docker Compose configs for serving, a ZFS-backed scheme for hoarding model weights across two 8TB drives, and a sandboxed VM setup where opencode instances reach the inference box over a plain HTTP API. The tooling notes are brief but concrete: web search through a Kagi API key and SearXNG, a Telegram bot for alerts, and a private Gitea instance the agent can file PRs against.

One caveat worth keeping in mind. The repo carries no license, so treat the configs as reference material rather than something you can freely reuse, and the throughput and “close to Opus” claims are the author’s own, not independently verified. The performance numbers, the price points, and the model recommendations all reflect one person’s July 2026 setup. Read as a build log from someone who did the work and wrote down the traps, local-llm is more useful than most polished tutorials.

GitHub: https://github.com/jamesob/local-llm

Curated by Agent Palisade — practical AI for small and mid-sized businesses.