Jamesob's guide to running SOTA LLMs locally
Points and comments are a snapshot, not live.
A detailed guide to building a $40k local LLM rig with 384GB of VRAM.
The guide details two tiers: $2k for 2x RTX 3090s (48GB VRAM) running Qwen3.6-27B and whisper-large-v3 STT, and $40k for 4x RTX PRO 6000s (384GB VRAM) running GLM-5.2-594B, nearly matching Claude Opus. The author's rig uses a last-gen EPYC system with eBay DDR4, a c-payne PCIe Gen4 switch for direct GPU P2P communication, and a custom wood enclosure. Configuration details include BIOS settings (ASPM disabled, 4x bifurcation), ACS disable for switch P2P, power limiting to 350W per GPU to run on a 110V circuit, and Docker-based model serving. Achieved P2P bandwidth: 27.5/50.4 GB/s, 0.4 µs latency.
What commenters are saying
Commenters debate the cost-effectiveness of a $40k custom rig vs. alternatives. Several recommend Apple M-series Macs for unified memory, noting a 128GB M4 Max Studio at $6.8k offers more VRAM per dollar but has lower memory bandwidth (614 GB/s vs. 1.87 TB/s for dual 3090s). Others suggest 2x RTX 3090s are sufficient for Qwen 27B at 68 tok/s. A correction notes the article's $40k GLM setup runs a quantized 594B model, not the full Opus-class GLM-5.2. Practical advice includes using the DRY penalty in llama.cpp to fix looping issues in reasoning models, and that 8x RTX6000s can achieve 1M context on the NVFP4 quant.