Running SOTA LLMs Locally on 4x RTX Pro 6000: One Engineer's $40K Build Log

This video walks through jamesob/local-llm, a public GitHub repo documenting one engineer's build for running state-of-the-art large language models on his own hardware. The claim up front: roughly $40,000 in parts gets you noticeably closer to running something like Claude Opus locally, with no subscription and no API key. The core of the build is four RTX Pro 6000 GPUs linked through PCIe switches so the cards talk to each other directly instead of routing through the CPU. The README covers the hardware bill of materials, the tiered spend options, and the exact BIOS settings, kernel flags, and an ACS fix needed to get card-to-card bandwidth to full Gen4 speed.

The Repo

The video is a scroll-through of jamesob/local-llm on GitHub, tagline "Everything I know about running LLMs locally." At capture the repo shows 753 stars, 35 forks, 3 watching, 2 branches, and 0 tags. It is a public repository with images/, runners/, and tools/ directories plus a README.md, last updated "yesterday." The README notes that nothing in it aside from the tables was written by AI. The framing question: "Have $2k burning a hole in your pocket and want some local, state-of-the-art machine intelligence? How about $40k?"

The Setup

The author bought 4x RTX Pro 6000s "back when they were cheaper." Because RAM had gotten expensive, he built a last-gen DDR4 EPYC system to host the cards, sourcing parts off eBay to keep the base system cost reasonable while still getting a lot of VRAM. Photos show four large GPUs mounted in a custom wooden enclosure over an open-frame motherboard.

The unusual move: PCIe4 switches sourced from c-payne.com. These let the GPUs communicate with one another directly at wire speeds during the allreduce step in tensor parallelism, rather than sending all data through the PCI root complex. The result is reduced latency between cards and less need for expensive PCIe5 hardware. As the README puts it, the money goes toward VRAM where it counts rather than a PCIe5/DDR5 base system, which it calls "terribly expensive as of July 2026." He also custom-fabricated a wood enclosure for the PCI switch and GPUs, which took about a day, and unplugged the PCI switch's built-in fan because it was loud and seemingly useless.

Spend Tiers

The README lays out options by budget.

Budget	Build	VRAM	What you can run
~$2k	2x RTX 3090	48GB total	Qwen3.6-27B, plus SOTA speech-to-text (STT)
~$40k	4x RTX 6000 Pro	384GB total	Something close to Claude Opus

At the $2k tier he calls out local STT as surprisingly useful and says it feels comfortable, unlike a hosted equivalent. A ready-to-run STT config lives in ./runners/stt and assumes only about 11GB of VRAM on an Nvidia GPU. He accesses the STT model through his cross-platform stt harness.

At the $40k tier, the README includes a "Current best models for 4x RTX6kPRO" table listing, for 2026-07, a best model of GLM-5.2-Int8Mix-NVFP4-REAP-594B with a linked "Runner config." He notes other valid ways to spend the money -- for example, allocating most of the budget toward a linked 4x DGX Spark cluster for 512GB VRAM, using that as the slow "big brain" to drive Qwen3.7-27b for routing tasks quickly.

Tuning: BIOS, Kernel Flags, and the ACS Fix

The heart of the technical walkthrough is getting peer-to-peer GPU bandwidth up to Gen4 line rate. The README documents the exact BIOS settings, kernel flags, and an ACS (Access Control Services) fix that took his card-to-card bandwidth "from broken to full Gen4 speed."

The measured result:

• Upstream: Gen4 x16, roughly 30 GB/s to CPU.
• Peer-to-peer through the switch: 27.5 GB/s unidirectional, 50.4 GB/s bidirectional, 0.37 to 0.45 microsecond latency, described as Gen4 line rate.
• Caveat: lspci may still show downstream GPU links as "2.5GT/s (downgraded)" at idle if ASPM is active. This is cosmetic; the links retrain to Gen4 under load.

A photo shows a bandwidth/topology test output on a screen confirming the Gen4 result.

Software and Tooling

The author runs opencode as his inference tooling. His setup creates a tmux session for each directory within the VM's ~/src tree, then runs an opencode instance that backs up to the inference machine's HTTP API at http://clank.j.co:5000. He stresses that making open-source models good depends on tooling them properly. A partial summary of his skills/ includes camofox, a kagi.com API key, and searXNG for web browsing and search, plus a Telegram bot for communication.

Key Takeaways

• The repo is jamesob/local-llm (753 stars at capture), a hand-written guide to running SOTA LLMs locally, with hardware, runner configs, and tooling notes.
• The headline build: 4x RTX Pro 6000 for 384GB VRAM, roughly $40k, aiming to run something close to Claude Opus with no subscription or API key.
• The base system is a deliberately cheap last-gen DDR4 EPYC box built from eBay parts, so money goes to VRAM rather than a costly PCIe5/DDR5 platform.
• PCIe4 switches from c-payne.com let the GPUs do the tensor-parallel allreduce card-to-card, cutting latency and avoiding pricier PCIe5 hardware.
• Documented tuning (BIOS settings, kernel flags, ACS fix) took peer-to-peer bandwidth from broken to Gen4 line rate: 27.5 GB/s unidirectional, 50.4 GB/s bidirectional, 0.37 to 0.45 us latency.
• Budget alternatives exist: ~$2k gets 2x RTX 3090 (48GB) for Qwen3.6-27B and local STT that needs only ~11GB VRAM.
• The listed current best model for the 4x setup (2026-07) is GLM-5.2-Int8Mix-NVFP4-REAP-594B; inference is driven through opencode against an HTTP API.