Repository navigation

feat(distributed): run distributed mode without a message broker - #11812

Open

localai-org-maint-bot wants to merge 158 commits into

masterfrom

test/distributed-e2e-ci

Open

feat(distributed): run distributed mode without a message broker#11812
localai-org-maint-bot wants to merge 158 commits into
masterfrom
test/distributed-e2e-ci

Commits on Sep 27, 2026
- test(distributed): share suite containers, isolate specs by database
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  c837f16
- test(distributed): stop leaking admin pools when database setup fails
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  d013619
- test(distributed): wait for the log subscriber instead of racing it
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  4c8b8c3
- test(distributed): scope the log-subscriber wait and mark the race it works around
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  1dc7998
- ci(distributed): run the distributed e2e suite on PRs
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  05bc519
- ci(distributed): widen the trigger and drop the mid-suite image pull
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  f5e90f2
- test(distributed): add a process-level cluster harness
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  8533a10
- test(distributed): make the cluster harness survive a restart
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  0cf3981
- test(distributed): add admin session helper to the cluster harness
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  84ac07d
- test(distributed): give each frontend its own data dir and one pinned secret
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  1d74675
- test(distributed): add kill and restart primitives to the cluster harness
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  941150a
- test(distributed): correct the failure-primitive comments and guard the wipe
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  6e78ea9
- test(distributed): prove the cluster harness with a two-replica baseline
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  3d2a96d
- test(distributed): fail rather than skip the cluster specs in CI
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  9862d9e
- test(distributed): require the cluster binaries by default under CI
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  9b0d382
- test(distributed): cover replica death, rolling restart and racing replicas
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  8990c51
- test(distributed): prove health checking was alive during the failover windows
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  4e079c9
- ci(distributed): run the process-level cluster suite
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  c3458e9
- ci(distributed): fail the cluster job when it selects no specs
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  8e8a5d3
- docs: document the distributed and cluster e2e suites
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  1dae0ce
- docs: correct the flake-attempts semantics and fill the review gaps
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  235057b
- test(distributed): correct the claims the e2e comments make
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  1e5f5db
- docs(e2e): correct three claims in the distributed e2e comments
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  15e325c
- feat(cluster): record frontend replicas in a shared instances table
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  c72baa4
- fix(cluster): stamp liveness on the database clock and refuse undialable addresses
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  42be69d
- feat(cluster): add the bidirectional splice used by the relay and tunnel
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  6caceb4
- fix(cluster): treat a yamux teardown in Splice as a normal ending
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  c05246a
- fix(cluster): report a dead yamux session instead of swallowing it
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  a0deca6
- feat(cluster): accept authenticated peer links on /api/cluster/peer
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  c2162f7
- fix(cluster): stop reading a bare EOF as a clean ending
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  73da808
- fix(cluster): own the peer auth prefix in auth, and pin what the specs claimed
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  9fdd0be
- feat(cluster): dial and pool yamux links to peer replicas
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  c7cb7e1
- refactor(cluster): make the cluster service a leaf and blame the caller's deadline
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  393797e
- feat(cluster): fence worker-connection ownership with a monotonic epoch
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  edc7313
- fix(cluster): draw connection epochs from a sequence so none is ever reused
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  6f53701
- fix(cluster): keep the connection schema migratable on SQLite
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  e951433
- feat(cluster): give phase 1 a call site, and prove it against real replicas
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  dca8ce7
- fix(cluster): hold the guarantees phase 1's comments were claiming
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  bb4719e
- fix(cluster): share one lock order, and correct the phase 1 comments
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  3daf889
- feat(cluster): resolve tunnel ownership against a live owner
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  a275147
- fix(cluster): make the ownership comments say only what holds
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  df7cebf
- feat(cluster): hold worker tunnels, and re-claim them after a sweep
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  032725c
- fix(cluster): make a claim and its record indivisible per node
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  2466e42
- fix(cluster): keep a session close out of the per-node claim gate
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  9e4df19
- feat(cluster): open the door a worker dials its tunnel through
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  9702e01
- fix(cluster): harden the worker tunnel, and stop starting a database per spec
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  7deda4a
- fix(testutil): stop the shared-database change from disarming two regressions
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  0762fc0
- fix(testutil): clear the maintenance timeouts at connection startup
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  7746bcf
- feat(worker): dial, hold and serve the tunnel, on a credential of its own
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  148fff1
- fix(worker): spec the tunnel's routing table, which was the SSRF boundary
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  9bf2013
- fix(worker): make the tunnel credential's node-type gate actually structural
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  9095480
- feat(cluster): relay a peer's stream onto a worker tunnel held here
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  08aa37b
- fix(cluster): express the splice policy once, and pin the relay's budgets
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  1f3dc70
- feat(cluster): reach every worker through its tunnel, never its address
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  b1dc6c4
- fix(cluster): make "no route" a condition of its own, and let it out of the package
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  d26d8b9
- fix(grpc): let the transport answer through the wrappers, not only past gRPC
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  b0a51f5
- test(nodes): make the joiner spec deterministic, and gate the wrapper shape in lint
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  050e585
- docs(grpc): withdraw the lint-cost claim, and make the one nolint tidy-proof
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  21f1fcc
- feat(distributed): stop workers listening, and stop them advertising
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  256d4ee
- fix(distributed): refuse a worker that cannot tunnel, and say why it was refused
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  4ce7efd
- test(distributed): prove the worker tunnel end to end, under real inference
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  d508964
- fix(cluster): stop blaming a peer for the caller's own expired deadline
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  5f5baf9
- fix(distributed): let a worker's own refusal be evidence about its backend
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  92e1927
- fix(cluster): stop a late request frame reading as the worker's verdict
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  00a2645
- fix(cluster): keep the refusal vocabulary in one table
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  e209492
- feat(cluster): record a departure instead of erasing the connection
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  db31cf5
- fix(cluster): pin the departure retention to the loop that applies it
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  f8bece3
- feat(cluster): answer presence with four values on the database clock
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  c4e5587
- fix(cluster): pin the derived retention to the sweep that applies it
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  29a0fd3
- feat(worker): serve the control plane over the tunnel, not over NATS
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  9b611f0
- fix(worker): pin the rune cut, answer unload honestly, drop the dead publisher
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  3b836bc
- feat(distributed): call the worker's control routes instead of the bus
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  96e259f
- test(distributed): pin the no-demotion rule at every call site it is stated
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  0494202
- feat(distributed): serve file staging over the worker tunnel
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  dccc536
- test(distributed): pin the unreadable-request rule at all three file verbs
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  fd1eb34
- feat(distributed): read worker absence from the database, not from a bus timeout
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  1b12d87
- fix(distributed): let absence be decided by something, at all three call sites
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  c0b3246
- feat(distributed): take the backend worker off NATS entirely
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  52c099e
- test(distributed): prove phase 3 end to end, under real processes
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  1ed2222
- refactor(distributed): one subject matcher, shared by the carrier and the doubles
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  b35e1b4
- feat(distributed): carry fan-out on PostgreSQL LISTEN/NOTIFY
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  e7d322c
- fix(distributed): order LISTEN and UNLISTEN on one lock
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  43956a9
- feat(distributed): keep the listener draining and let it come back
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  b8d514d
- feat(distributed): give agent workers a tunnel of their own
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  7629804
- feat(distributed): make MCP execution and discovery a selection
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  7c7324b
- test(distributed): name the replica endpoint the router now requires
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  4c2d696
- feat(distributed): stop a backend on one route, whatever the worker is
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  0d61cc1
- test(e2e): build the cluster binary and drop a load-sensitive budget
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  8c6c434
- feat(distributed): let a worker name a broadcast the frontend decides on
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  ae3ff21
- feat(distributed): dispatch queued work as a claim queue
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  dc0c50f
- feat(distributed): give responses.metadata something to re-hydrate from
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  236c3a0
- fix(distributed): give one tenant's agent tasks a subject of their own
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  a3443c3
- feat(distributed): carry the state.*.delta families on PostgreSQL
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  27e2ead
- refactor(distributed): move job and agent fan-out onto the PostgreSQL carrier
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  0d2d4f0
- feat(distributed): move the last nine fan-out families onto PostgreSQL
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  000545b
- refactor(distributed): delete MessagingClient and shrink the NATS client to fan-out
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  eaf4eb6
- feat(distributed): carry an agent cancel on the worker's own tunnel
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  dc34e0c
- fix(distributed): read a departed agent tunnel as the routing fact it is
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  60221f0
- chore(distributed): stop telling an operator to run a NATS cluster
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  b91ed7a
- chore(distributed): take the nats-io modules out of the build
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  646fd69
- test(distributed): prove the busless cluster on two frontends and two workers
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  cf62a72
- test(distributed): prove the fan-out carrier between two real replicas
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  276e38f
- feat(cluster): make a peer prove which replica it is
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  fbfbb85
- fix(distributed): say that skills and collections are replica-local
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  9f1939c
- fix(worker): check a gRPC port is free before handing it out
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  c56fcec
- fix(distributed): bound the health monitor's miss streaks to live rows
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  a5f54b7
- fix(distributed): drop a departed node's cached HTTP clients
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  d42d70e
- fix(distributed): back a failed claim off instead of respinning it
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  8f5f303
- test: fix the three data races that made -race runs noisy
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  55bc99c
- test(worker): script the kernel out of the port bookkeeping specs
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  22ef89c
- test(openai): drop the audio snapshot nothing reads
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  3aab79f
- test: guard shared PostgreSQL setup on macOS
  localai-org-maint-bot
  committed
  3cce3e3
- test: keep distributed specs in their own targets
  localai-org-maint-bot
  committed
  595dfb9
- fix(worker): resolve staging directory symlinks
  localai-org-maint-bot
  committed
  553d11a
- test(worker): canonicalize missing upload fixture
  localai-org-maint-bot
  committed
  06aaed9
- fix(distributed): preserve master protocol extensions
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  5612dd1
- test(distributed): use renamed worker address
  localai-org-maint-bot
  committed
  bc8e843
- test(nodes): match the frontend setup command
  localai-org-maint-bot
  committed
  4f6648b
- docs(distributed): define binary conformance
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  b06149b
- docs(distributed): test machine authentication
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  58d0bfa
- fix(distributed): offline stale unhealthy nodes
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  a626c9c
- fix(distributed): accept legacy empty stop replies
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  c06dd46
- test(distributed): migrate before restricting DB
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  6f1f0aa
- test(distributed): pin cluster health timings
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  189439f
- fix(ui): complete node operation journeys (#12070)
  
  localai-org-maint-bot
  and
  mudler
  committed
  9bd9139
- test(distributed): add deterministic backend fixtures
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  57736e5
- fix(mock-backend): harden deterministic fixtures
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  5b308fe
- test(distributed): cover binary backend routes
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  42e525d
- test(distributed): complete binary backend conformance
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  f746c3c
- fix(distributed): harden binary conformance
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  1cebf6a
- fix(distributed): stage directory outputs
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  6dc6f8d
- fix(distributed): preserve async staging state
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  7fafe3c
- fix(distributed): close staging conformance gaps
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  c1f102e
- fix(distributed): harden staged resource ownership
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  ed5ab1b
- test(distributed): guard backend protocol coverage
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  5822ec1
- test(distributed): scan staging overrides package-wide
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  8892940
- test(distributed): verify binary machine authentication
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  d14f93a
- fix(distributed): prove agent worker credentials
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  9cf56f4
- test(distributed): integrate binary CI gates
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  08f3b4d
- test(distributed): align conformance specs with Ginkgo
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  14fc9d5
- test(distributed): reap binary backend children
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  0a81ac4
- test(distributed): correct cluster suite docs
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  85d1e61
- fix(vram): bound remote GGUF metadata lengths
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  2637573
- fix(deps): remove stale GGUF parser sums
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  3a87afb
- fix(security): clear the gosec findings this branch introduced
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  9e944ab
- fix(security): justify the last two gosec findings in new code
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  8c2e324
- fix(distributed): follow current backend protocol
  
  mudler
  authored and
  localai-org-maint-bot
  committed
  6cb1a2f
- test(worker): canonicalize shared staging paths
  localai-org-maint-bot
  committed
  90876e3
- test(ui): open node setup before checking commands
  localai-org-maint-bot
  committed
  4d336db
- test(ui): wait for the automatic worker drawer
  localai-org-maint-bot
  committed
  7890e5f
Commits on Sep 28, 2026
Commits on Oct 1, 2026
Commits on Oct 2, 2026
- chore: merge master into distributed test branch
  localai-org-maint-bot
  committed
  b941fc4

Commits

Commits on Sep 27, 2026