+
+When people picture running a large language model, they picture a data center. Racks of GPUs that belong to someone else, a metered API, and a bill that grows every month you succeed. You send your prompts off to a black box and hope the price, the model, and the privacy policy all stay the way they were when you signed up.
+
+For a lot of teams that is a bad trade. You give up control over when models change, where your data goes, and what hardware runs your workloads. And as usage grows, so does the bill, with no lever to pull except "pay more."
+
+[Mesh LLM](https://meshllm.cloud) is a different shape. It pools the GPUs and memory you already have, across as many machines as you want to add, and exposes the whole thing as one OpenAI-compatible API. Start one node. Add more later. Let the mesh decide whether a model runs on the box in front of you, routes to a peer, or splits across several machines.
+
+## The problem: AI is expensive, and it is somebody else's
+
+The popular models are monoliths. Most people reach them through a UI or an API key and pay a large provider to run everything. That is convenient, and it is also a surrender. You do not control when the model gets updated, what memory it runs in, or what hardware sits underneath.
+
+Plenty of businesses and services that depend on these models want the opposite: more control, more pluggability, lower cost. They have GPUs sitting in offices, in closets, under desks. What they are missing is a way to make those machines act like one.
+
+## Mesh LLM: run the models yourself
+
+The pitch is simple. Run bigger models without buying bigger GPUs. Share compute privately with your team, or publicly with the world, to power agents and chat. Point any OpenAI client at http://localhost:9337/v1 and stop caring where the work actually happens.
+
+Under the hood, Mesh LLM distributes model compute across a mesh of iroh endpoints. A request can be served three ways:
+- Run it locally, on this machine's GPU.
+- Route it to a peer that already has the model loaded.
+- Split a model too big for any single box across several machines, as a pipeline.
+
+## How it works
+
+The architecture is pluggable. Plugins declare what they provide in a manifest, the runtime starts them, routes calls, and exposes their capabilities over MCP, HTTP, inference, and mesh events. The catalog ships with 40+ models, from half-a-billion-parameter models that fit on a laptop to 235B mixture-of-experts giants.
+
+For the giants, Mesh LLM has a split mode (internally, "Skippy"). A model gets partitioned by layer ranges into stages: layers 0 to 15 on one node, 16 to 31 on the next, and so on down the pipeline. Activations flow from one stage to the next, so several modest machines can run a model none of them could hold alone. The OpenAI client never sees any of this. It still just talks to localhost.
+
+## How it uses iroh
+
+Every node, whether it serves models or only sends requests, boots an iroh endpoint. That endpoint is the node's identity, a public key, and its only network surface. There is no central server. iroh handles the hole-punching, NAT traversal, and relay fallback needed to open a direct, authenticated QUIC connection between any two nodes, wherever they sit.
+
+To keep that working across the open internet, Mesh LLM runs two iroh relays in different regions, so nodes that cannot reach each other directly always have a fallback path nearby.
+
+The whole protocol rides on QUIC's ALPN negotiation. There are three:
+| ALPN | What it carries |
+|------|----------------|
+| mesh-llm/1 | Main mesh: gossip, routing, HTTP tunnels, plugin channels |
+| mesh-llm-control/1 | Owner control plane (config sync, ownership attestation) |
+| skippy-stage/2 | Latency-sensitive activation transport for split models |
+
+
+Inside the main `mesh-llm/1` connection, everything is a bidirectional QUIC stream tagged with a single leading byte that says what kind of stream it is. One connection carries gossip, inference, route queries, and peer-lifecycle events, all demuxed by that first byte:
+
+| Byte | Stream type | Description |
+|------|-------------|------------|
+| 0x01 | GOSSIP | peer announcements (models, GPU, RTT, capabilities) |
+| 0x04 | TUNNEL_HTTP | inference requests proxied to a peer |
+| 0x05 | ROUTE_REQUEST | "which models do you host?" |
+| 0x06 | PEER_DOWN | dead-peer notification |
+| 0x07 | PEER_LEAVING | graceful shutdown |
+| 0x08 | PLUGIN_CHANNEL | plugin RPC |
+| 0x0e | DIRECT_PATH_REQUEST | share direct addresses for NAT traversal |
+
+
+The neat part is what this buys you. iroh gives authenticated, NAT-traversing QUIC between any two machines, addressed by public key. So "route to a peer" and "stream activations to the next pipeline stage" become the same primitive as "talk to localhost," just with a different endpoint ID. The networking stops being something you have to think about.
+
+iroh provides the secure transport. Mesh LLM builds its own gossip layer on top, so it controls exactly who gets admitted to the mesh, which versions are compatible, and which peers to trust.
+
+
+## Getting started
+
+Users can install the lightweight software (about 18 MB) and either join the
+public mesh or configure private deployments. The system presents itself as
+`localhost:9337/v1` to any standard OpenAI client.
+
+A mobile app is coming, built on iroh's Swift SDK. The plan is to speak ACP, the
+emerging agent standard, so other clients can join the mesh too. The throughline
+is the same one that motivated the whole project: more peer to peer, fewer
+closed servers, and no lock-in.
+
+- [See the code](https://github.com/Mesh-LLM/mesh-llm)
+- [Mesh LLM Website](https://meshllm.cloud)
diff --git a/src/app/page.jsx b/src/app/page.jsx
index 25128721..18c3e1d6 100644
--- a/src/app/page.jsx
+++ b/src/app/page.jsx
@@ -123,7 +123,7 @@ export default function Page() {
Distributed AI Training
Train foundation LLMs with compute distributed around the world, across AWS, GCP, Azure, and self-hosted infrastructure.
diff --git a/src/app/services/enterprise/page.tsx b/src/app/services/enterprise/page.tsx index 701d41c9..87c88730 100644 --- a/src/app/services/enterprise/page.tsx +++ b/src/app/services/enterprise/page.tsx @@ -221,10 +221,10 @@ export default function EnterprisePage() {Real-time video for millions of concurrent connections.
Read the case study →
- +
- Distributed AI Training
-Train foundation LLMs across AWS, GCP, Azure, and self-hosted compute.
+Distributed AI Training & Inference
+Train foundation LLMs across AWS, GCP, Azure, and self-hosted compute, or pool GPUs into a single inference API.
Read the case study →
diff --git a/src/app/solutions/nous/page.jsx b/src/app/solutions/distributed-ai/page.jsx similarity index 73% rename from src/app/solutions/nous/page.jsx rename to src/app/solutions/distributed-ai/page.jsx index 58d41f79..60cd804e 100644 --- a/src/app/solutions/nous/page.jsx +++ b/src/app/solutions/distributed-ai/page.jsx @@ -3,14 +3,14 @@ import { HeaderSparse } from '@/components/HeaderSparse' import { FooterMarketing } from "@/components/FooterMarketing" import { OpenSourceIllustration } from "@/components/OpenSourceIllustration" import Link from "next/link" -import { Server, Zap, Globe, Shield, Cpu, MessageSquare, BarChart3, Network } from "lucide-react" +import { Server, Zap, Globe, Shield, Cpu, Network } from "lucide-react" export const metadata = { - title: 'Nous Research - Use Case | Iroh', - description: 'How Nous uses iroh to train foundation LLMs with compute distributed around the world.', + title: 'Distributed AI - Use Case | Iroh', + description: 'How Nous and Mesh LLM use iroh to train foundation LLMs and pool GPUs across machines, with compute distributed around the world instead of centralized in one data center.', } -export default function NousUseCasePage() { +export default function DistributedAIUseCasePage() { return (
- Use Case: AI/ML
- Distributed AI Training at Global Scale + Distributed AI, Training to Inference
- Nous Research uses iroh to train foundation LLMs with compute distributed around the world, - across AWS, GCP, Azure, and self-hosted infrastructure. + Nous Research uses + iroh to train foundation LLMs with compute distributed around the world, across AWS, GCP, + Azure, and self-hosted infrastructure. Mesh LLM uses + it to pool GPUs across machines into a single inference API. Neither needed a data center to do it.
Nous Research: Training
10x
@@ -171,19 +170,75 @@ export default function NousUseCasePage() {Mesh LLM: Inference
+Pool GPUs Into One API
++ Mesh LLM pools + existing GPU resources across multiple machines and exposes them through a single + OpenAI-compatible API, with no central server coordinating the mesh. Every node runs an + iroh endpoint, so nodes connect directly to each other over authenticated QUIC instead of + routing through a coordinator. +
++ “Control over when models change, where your data goes, and what hardware runs + your workloads.” +
+Local Execution
++ A request runs on the requesting machine's GPU, if it's big enough. +
+Peer Routing
++ Otherwise, it routes directly to a peer node that already has the model loaded. +
+Pipeline Splitting
++ Models too large for any single machine get partitioned by layer ranges across + multiple nodes, so modest machines can collectively run what none of them could load alone. +
+Why Relays Matter
Nous runs 5 iroh relays to ensure reliable connectivity across their distributed training network. - The key insight: when things deteriorate, they can't break. + Mesh LLM runs two, in different regions, for the same reason. The key insight: when things + deteriorate, they can't break.
Iroh automatically establishes direct connections when possible for maximum throughput. When direct connections aren't possible—due to NATs, firewalls, or network conditions—traffic - flows through relays. This fallback mechanism means training runs continue even when network - conditions change. + flows through relays. This fallback mechanism means training and inference keep running even when + network conditions change.