AMD Strix Halo RDMA Cluster Setup Guide
Points and comments are a snapshot, not live.
A guide for setting up a two-node AMD Strix Halo cluster with RDMA for vLLM inference.
The guide details configuring two AMD Strix Halo nodes (Framework Desktop mainboards, 128GB unified memory) linked via Intel E810 100GbE NICs (RoCE v2) for distributed vLLM inference. It covers Fedora 43 host setup, kernel parameters (iommu=pt, pci=realloc, amdgpu.gttsize=126976), and a toolbox container with a custom librccl.so for Strix Halo RDMA support. Achieved ~5µs latency and ~50 Gbps bandwidth via RDMA, enabling tensor parallelism across nodes. A TUI utility manages Ray cluster and vLLM launch.
What commenters are saying
Commenters are enthusiastic about using Strix Halo clusters for local AI, citing kyuz0's containers as enabling. Some note price volatility: 128GB units that were ~$1,800-2,500 are now often $4,000+, making them less attractive. There is debate on laptop vs. desktop for AI workloads, with some arguing desktops offer better thermal and value. Others mention alternative approaches like antirez's ds4 for Deepseek or used enterprise GPU setups. Several commenters see the current hardware market as overpriced and suggest waiting for improvements.