feat(distributed): run distributed mode without a message broker

[localai-org-maint-bot]

Distributed mode without a message broker

Running LocalAI distributed used to mean running and managing a NATS cluster, and giving every worker a routable address with inbound ports open back to it. Both are gone.

A worker now dials the frontend load balancer over ordinary HTTP and holds one multiplexed yamux tunnel that carries gRPC, HTTP and websocket traffic together. That tunnel lands on exactly one frontend replica; every other replica reaches the worker by relaying through the owner. Workers bind loopback only and advertise nothing. A deployment needs PostgreSQL and the frontends' own HTTP listener, and nothing else.

github.com/nats-io/... is out of go.mod and go.sum.

What a worker needs now

One outbound HTTPS connection to the same load balancer a browser would use. No routable address, no inbound ports, no broker credentials.

How it works

• The worker dials GET /api/cluster/connect and holds a yamux session. Control RPCs, file staging and inference all ride it.
• Tunnel ownership is fenced by a PostgreSQL sequence-backed epoch, and ownership resolution joins live replicas so a dead one can never be named as owner.
• A replica that does not hold a worker relays to the one that does.
• Workers authenticate with a per-node credential minted at registration and stored only as a hash.
• Fan-out runs on PostgreSQL LISTEN/NOTIFY. Payloads above the 8000-byte notification cap become a row plus a notify-with-id, never a truncated payload.
• Competing consumers use a SELECT ... FOR UPDATE SKIP LOCKED claim queue. A claim is released when the replica holding it stops being live, with no age term, so a slow claimant is never robbed.
• Absence is read from PostgreSQL rather than inferred from a bus timeout.

The invariant this rests on

Four conditions must never be reported as each other: a routing fact, a connection absent within the reconnect grace, an unreachable peer, and the worker's own answer. Only the first and last may be acted on. Absence makes the scheduler reap rows and evict models, and one of those paths runs during inference, so a collapse between them is not a cosmetic bug.

Review found and fixed fourteen instances of that collapse, including one created by the fix for another, one where an expired caller deadline was blamed on the peer under contention, and one where a worker's own refusal was read as "no route", which left permanently unreapable ghost rows.

Operator impact

• Upgrade frontends first, then restart workers one at a time. During the window, not-yet-restarted workers show healthy while their models return "no route", and it clears on restart. The reverse order fails: an old frontend rejects a new worker's registration and the worker exits.
• LOCALAI_ADVERTISE_ADDR and LOCALAI_ADVERTISE_HTTP_ADDR are no longer used.
• --nats-url is accepted and ignored so existing command lines keep starting. --nats-require-auth no longer gates agent-worker approval.
• Rolling a frontend back requires restarting every worker, because registration is the only writer of the address columns.

Testing

A real-process end-to-end suite runs a two-frontend, two-worker cluster of live local-ai binaries, which is the configuration where the owner path and the relay path are live at once:

• Both workers serve inference from both replicas. The relay case sends the first request for a model to the replica that does not own the worker, so the backend install, the model-file staging over the tunnel and the gRPC load and predict all cross the relay. Warming it at its owner first would have left only the predict on the relayed path.
• Which worker served a request is read back from the database, and the relay is proven by resolving ownership through the production query before the request and re-reading it after. Deleting the relay reddens only the request sent to the non-owner.
• Killing the replica that owns a worker's tunnel: nothing acts on the absence inside the grace, the worker re-homes to the survivor and serves again, and the other worker keeps serving throughout. Setting the grace to a nanosecond reddens exactly that spec.
• Cross-replica fan-out is proven between two live replicas sharing one database, including a payload above the 8000-byte notification cap making the round trip byte-identical through the spill row, with the size asserted absolutely so moving the cap in either direction reddens it.
• A suite-wide negative control blocks both tunnel dials while registration and heartbeats continue: every request must be refused naming the routing fact rather than absence, both workers must stay healthy, and restoring one tunnel must recover exactly that worker.

The suite previously could not fail honestly: with no binary present it ran zero specs, exited zero and printed "Test Suite Passed". It now builds the binary and refuses to run against one older than the tree. That guard has since caught three real mistakes.

The race detector runs clean across 98 suites and 7095 specs with no known-failure allowance.

Bugs found in existing code and fixed

• Unloading a model reported success when the free failed, so a failed eviction told the scheduler VRAM had been freed.
• Status readers were handed a live pointer that broadcast subscribers mutate, a real race on the operations endpoint.
• A progress tick delivered after a cancel cleared the cancelled flag.
• Two end-to-end specs had been red for an entire phase because a label filter excluded them.

Follow-ups closed before merge

The gaps found during the work were closed rather than deferred:

• Peer-link identity. /api/cluster/peer took a self-declared replica id, so anything holding the shared registration token could relay to every worker a replica owned and repeatedly evict its inbound link. Each replica now mints its own secret at startup and publishes only the hash; the plaintext never leaves the process that minted it. A mixed-version rollout refuses rather than falling back, because an old replica and an attacker send byte-identical requests.
• Port allocation now checks a port is actually free instead of trusting bookkeeping, and no longer hands out ports from the kernel's ephemeral range.
• Two unbounded maps bounded: health-monitor miss streaks, and a departed node's cached HTTP clients.
• A failed claim backs off instead of respinning.
• Three pre-existing data races fixed, so the race detector runs with no allowance.
• Skills and collection caches are documented as replica-local rather than given invalidation, because every read verb hits local files: telling a peer to invalidate would only make it re-derive from a directory that lacks the change. The two consequences are documented.

Known remaining

• DistributedManager.List truncates skill content to 500 bytes for agents, a separate pre-existing defect.
• MCP prompts and resources have no distributed path, pre-existing and already broken before this work.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y2TjpXdY7SszRrM5PhSp1e

[mudler]

@localai-org-maint-bot fix tests

[localai-org-maint-bot]

Pushed two test fixes on top of the latest master:

• 994ec9dde: put the existing macOS Docker skip in sharedPostgres, so the maintenance-connection test cannot bypass it.
• 4e3224922: stop the general test-e2e target from recursively collecting distributed/cluster suites. Those retain their dedicated targets, which provision PostgreSQL and build the required host binaries.

Verification: the maintenance test fails before the change and skips afterward with a Go overlay selecting the macOS path; four FakeBus specs and go vet ./core/services/testutil pass. The e2e dry run selects the intended 118 specs in one suite. The full CLI suite passes, as do 31 focused runs without coverage and 101 with coverage. git diff --check passes.

Still unresolved: the original Linux timeout in the mcp ci run wiring test did not reproduce in those runs. The VibeVoice job failed while downloading its Hugging Face model. Docker is unavailable here, so I could not execute the database/container suites. This is progress on the request, not a claim that every test is now green; leaving it in progress for the next invocation to check the new results.

Update 2026-09-08: resumed this request and pushed 4e4597dfc after merging master 561d62397. The remaining macOS worker test failed because the staging allow-list resolved symlinks in the requested file but not in the allowed directory. Both paths now resolve before comparison; the distributed-mode guide documents this behavior.

Reproduced the exact failure on Linux with a symlinked TMPDIR before the fix. Afterward, all 13 file-stager round-trip specs and four new containment cases pass with that TMPDIR, including rejection of sibling paths and symlinks escaping the allowed root. The focused worker file-staging tests, incoming virtual-model companion tests, go vet ./core/services/worker, and diff checks pass. Full lint could not run: this environment has neither make nor golangci-lint.

The existing Linux, distributed, cluster, and VibeVoice test runs were cancelled, so full verification remains incomplete. No CI jobs were restarted or polled, and this request remains in progress.

[localai-org-maint-bot]

This is a large PR (100 files) with tests-linux and tests-apple CANCELLED (required checks), plus a gosec failure and backend build failures (magpie-tts-cpp, face-detect). Could you check if the cancelled tests are due to a cascading failure or a real issue? The gosec failure also needs investigation.