Mesh LLM Runs Distributed AI Across Iroh Networks Without Central Servers

Mesh LLM pools GPUs from separate machines through iroh peer-to-peer links and presents a single OpenAI compatible endpoint.

The project removes the usual requirement for a central scheduler or shared cluster. Instead each participating node connects directly to others and decides locally where to run each request.

Users start the 18 MB binary on any machine that has spare GPU capacity. The service listens at localhost:9337/v1 and accepts the same chat and completion requests as commercial providers.

Requests can execute on the local GPU, move to another node that already holds the model weights, or split a large model layer by layer across several machines using the internal Skippy scheduler.

The software supports more than 40 models ranging from 500 million parameters up to 235 billion parameter mixture-of-experts checkpoints.

Direct connections replace cluster managers

Iroh supplies the networking layer that lets nodes discover one another and exchange tensors without opening ports or running a VPN. Each machine registers with a small set of relay nodes only long enough to establish an encrypted direct path.

Once the path exists, inference traffic flows between the original request origin and the chosen compute node. No persistent central record of which node holds which weights is kept.

This design removes single points of failure that appear in traditional orchestration tools. When any node leaves the mesh the remaining nodes continue to serve requests they can still reach.

How request routing actually works

The runtime maintains a lightweight registry of loaded models on every peer. When a request arrives the local node first checks whether it can satisfy the call itself.

If the model is absent locally it looks for another peer that already holds the weights. Only when no such peer exists does the system fall back to layer-wise partitioning through Skippy.

Skippy divides transformer layers into pipeline stages that run on different GPUs in sequence. Intermediate activations move over the iroh link between stages.

The approach lets a 70 billion parameter model run across four consumer GPUs that individually lack enough memory to load the full checkpoint.

What this changes for small teams

A researcher who owns two laptops and one desktop can now combine their GPUs into a single endpoint. The same laptop that receives the API call can also decide to offload the heaviest layers to the desktop overnight.

Because the binary is small and requires no additional database or message queue, setup time stays under five minutes on a fresh Linux box.

Larger organizations can add spare capacity from existing workstations without carving out a dedicated rack or negotiating with IT for new VLANs.

Limits that remain

The current implementation reports no built-in authentication or usage quotas. Operators must add their own proxy layer if they need access control.

Network latency between nodes directly affects time to first token when a request crosses machines. Local area networks see acceptable overhead while wide area links require further tuning.

Model parallelism through Skippy still demands that each stage fits on a single GPU. Models larger than the biggest available card cannot run until further sharding is added.

What to watch next

The project publishes its source and build scripts on a public repository. Watch the frequency of new model additions and any commits that improve cross-node bandwidth measurement.

Independent tests that measure end-to-end latency on mixed consumer hardware will show whether the claimed 18 MB footprint holds once dependencies are included.

Adoption inside existing local LLM tools such as Ollama or LM Studio would indicate whether the mesh approach solves a practical gap or remains a niche experiment.