Mesh LLM: distributed AI computing on iroh
Points and comments are a snapshot, not live.
Mesh LLM pools GPUs across machines into one OpenAI-compatible API using iroh.
Mesh LLM distributes model inference across a mesh of iroh endpoints, letting users pool GPUs they already own. Requests can run locally, route to a peer, or split a model across nodes via "Skippy" pipeline partitioning. The system exposes localhost:9337/v1 as an OpenAI-compatible API. Plugins declare capabilities in manifests; the catalog ships with 40+ models. iroh handles NAT-traversing QUIC connections authenticated by public keys, with no central server. Two iroh relays provide fallback paths. An 18 MB binary lets users join the public mesh or deploy privately.
What commenters are saying
Commenters debated performance feasibility. Some argued network latency kills interactive use, especially over WAN; others countered that split inference transfers only kilobytes per token, so latency is the main bottleneck, not throughput. A contributor reported 16 tok/s for Qwen 235B across two nodes on a lab setup with 5ms simulated latency, and 10 tok/s for GLM 5.2 on two Mac Studios over 1Gbit Ethernet. Privacy concerns were raised about clear-text model computations. The contributor noted no good answer yet for protecting activations from malicious nodes, recommending private meshes instead.