Repository navigation
feat(distributed): switch the cluster transport between NATS and the database at runtime - #12549
Draft
mudler-agent wants to merge 168 commits into
Draft
mudler-agent wants to merge 168 commits into
mudler-agent wants to merge 168 commits into
Conversation
Add table cluster_carrier and a store for it. The row names the carrier that the whole cluster uses, an epoch that grows with every change, and the fields a later change of carrier needs: state, target and the carrier that is draining. Seed inserts the row only when it is absent, so replicas that start together agree on one writer. Transition is a compare-and-set on the epoch, so two writers cannot interleave. A check constraint keeps the table at one row. Nothing reads the row yet. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The backend and model managers and the replica reconciler took the NATS sender by its concrete type. Add NodeControl, the interface of every control call they make, and take it instead. The sender implements it, so nothing changes at run time. NodeControl names the three unload interfaces that the model loader reads by type assertion. A sender that lacked one would still compile and would silently lose that behaviour. Export the install-with-force fallback so a sender in another package can implement it. Add a ClientFactory option to the reconciler, so its default probe dials the way the router does. Add NewTokenClientFactory for the direct factory. Narrow the agent event bridge from MessagingClient to Broadcaster. It only publishes and subscribes. Pin that the NATS client returns a client for a server that is not up yet, which startup relies on. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A carrier.Set is the implementation of every seam for one carrier. The holders implement the same interfaces as the seams (fan-out, work queue, control verbs, file staging, backend clients, worker dialer, agent RPC) and forward each call to the set that an atomic pointer names. They take no lock and add no allocation. The fan-out holder keeps a registry of its subscriptions. Listen attaches every subscription to a second set, so each handler listens on both carriers while a swap runs. Release drops the first set again. Publish always uses the set the pointer names. The holder also forwards the reconnect hook that syncstate and the failover sync register. NewNATSSet builds the set for the NATS carrier. A spec fails when any other non-test file imports the NATS libraries. Nothing stores a second set yet, so no behaviour changes. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…lders Read the cluster carrier row at startup, or insert it when it is absent, and build the set it names. An existing deployment has a NATS URL, so it seeds nats and runs as before. A row that names a carrier this build does not run stops the start with a message, and does not fall back. DistributedServices no longer holds the NATS client. The router, the health monitor, the reconciler, the job dispatcher, the agent bridge, the prefix cache and the other users of the bus take holders over the active set. The file manager is built before the set because the NATS file stager needs it. The NATS URL keeps its meaning. A URL the client cannot use still stops the start. A server that is not up does not: the client retries, as it did before. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
make test-e2e-distributed was not run by any workflow. Add one, with continue-on-error so a failure is visible on the pull request and does not block a merge until the job has a track record. The suite starts its own PostgreSQL and NATS servers, so the job pre-pulls both images. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a section on the holders, the carrier set, the subscription registry and the cluster_carrier row. Record the import boundary of the NATS libraries and the startup rule for an unreachable NATS server. Remove the open item for a reconciler client factory, which now exists. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The fan-out holder is a Broadcaster, so it has to pass the suite that every carrier passes. Run it on a holder as built, and on a holder whose carrier was swapped. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…nnections Each frontend replica records itself in the instances table and refreshes the row from a membership loop. The database clock stamps every row, so no replica decides on the clock of another. The row also carries the version and the epoch of the carrier that the replica has built, with a reason when it could not. The node_connections table records which live replica holds the connection of a worker. Every claim takes a fresh epoch from a sequence, so a replica that lost a worker cannot clear the claim of the replica that won it. A released or orphaned row is kept as a departure, and Presence tells a worker that is reconnecting from one that is gone only after a grace period. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A worker that follows a change of carrier is attached to two carriers for a while. Add the columns that hold what the worker reports and the epoch it saw when it reported. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…istry Run the membership loop in distributed mode, on every carrier, and remove the row when the replica stops. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…yloads Move the subject matcher out of the test helper into messaging, so a carrier and the in-memory doubles use one rule. Add MaxBroadcastBytes, checked by the NATS client and by the fake bus, so a message that one carrier accepts is not refused by another. The conformance suite now takes two ends of a carrier and runs every broadcast root, the sizes around the notify cap and the bound. It also asserts what a carrier does with the control roots, and reads the drop counter when the carrier has one. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
… string The postgres image declares its data directory as a volume. A removed container leaves that anonymous volume behind, so every spec that started a database left tens of megabytes on the disk of the host. Put the data directory on a tmpfs. Also return the connection string, for code that needs a connection of its own, such as a LISTEN connection that cannot come from a pool. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…t and reply A carrier that has fan-out and no request and reply serves the broadcast roots only. ValidateBroadcastSubject applies the shared rules and then refuses the nodes and mcp roots with the same class of error as a root that is not served. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The second implementation of messaging.Broadcaster. A message goes out with pg_notify. When its encoded notification is 8000 bytes or more, the payload is written to the bus_messages table and the notification carries the row id. Every replica deletes the rows that are older than the retention, by the clock of the database. Delivery is at-most-once, as with NATS. The listener copies each notification into a bounded queue and never blocks, so the server is not held up by a slow consumer. A lost connection is dialled again and every channel is listened again, then the reconnect callbacks run. The subject rules are the shared ones, and the carrier refuses the control roots. It checks the shared payload bound. Counters for the messages published by path, the messages lost by stage, the spilled rows deleted and the reconnections go to OpenTelemetry, because a lost broadcast is otherwise visible only in a log. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
… order A broadcast of 64 KiB is written to a row, and every replica reads that row with a SELECT. One reader at a time limited a replica to about a thousand such broadcasts per second, and a burst above that was dropped. Read up to eight rows at the same time and hand the notifications to the handlers in the order they arrived, so the order of one subject does not change. In a burst of 4000 broadcasts of 64 KiB from 16 publishers, the replica with one reader delivered 1860, and the replica with eight readers delivered all of them. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
NewPgbusFanout opens the LISTEN connection and returns the broadcaster, the reconnect hook, a ready check and the close that a set needs. It opens the connection when it is called, so a deployment on NATS holds no LISTEN session until a change of carrier asks for one. Run the broadcaster conformance suite on holders that were swapped from one carrier to this one on a real database, and check that a message published after the swap reaches a subscriber made before it. The import-boundary spec now also keeps the PostgreSQL driver in the pgbus package, and the pgbus package behind the carrier builder. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A worker holds one outbound websocket to a frontend and a yamux session runs on it. The frontend opens a stream for every request. The first frame of a stream names the local service of the worker, and the worker always answers with a reply frame before the carried protocol starts. The four refusals stay apart, so that a caller acts on the three that are evidence about a backend and does nothing about the one that says the worker learned nothing. The upgrader and the dialer come from one constructor each, with 64 KiB buffers. With the 4 KiB default of gorilla, a stream that sends from the worker to the frontend ran at 560 to 600 MB/s on a loopback socket and the other direction at 3,200 to 3,800 MB/s. With 64 KiB the first direction runs at 2,900 to 3,000 MB/s. BenchmarkTransfer repeats the measurement. A session has a lane. The inference lane keeps the yamux windows. The bulk lane uses 4 MiB and 32 MiB, which a transfer needs on a link with a long round trip, and which would let more data queue ahead of a small call on a session that is shared. Only the files of the tunnel package import yamux. The import boundary spec lists it. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Splice used to close both streams as soon as one direction ended. A protocol that closes its sending side and then waits for the reply lost the reply that was still in flight. Now a direction that ends clean passes the end on with CloseWrite and the other direction goes on until it ends too. When the destination cannot half-close, or a direction fails or ends because a stream was closed, both streams are closed at once, as before. This is what wakes a copy that is parked in a read or in a write. A copy into a TCP connection wraps the error in a net.OpError. The classifier of the endings of yamux now removes that layer, so a stream that this side reset is not reported as a failure of the transport. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A proxy in the spec works as one link of 1 Gbit/s with 10 ms of delay in each direction and one queue that every connection shares. A transfer of 256 MiB runs from the worker to the frontend while a small call goes every 5 ms in the other direction, and its reply has to wait in the queue of the loaded direction. The spec runs three set-ups: a probe on its own TCP connection, a probe on the session of the transfer, and a probe on the inference session with the transfer on the bulk session. It requires the bulk lane to stay within 10 ms of the TCP baseline at the 99th percentile, and the shared session to be worse than the bulk lane. A build with the race detector moves 128 MiB. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The registry keeps the sessions that workers opened to this replica and keeps the node_connections table in agreement with them. Attach writes the claim first and stores the session second, so a failed claim leaves the replica as it was, and the claim and its record are one step for each node. Detach matches the token of the attachment by equality, so a session that was replaced cannot remove the one that replaced it. Open returns ErrNotOwner only when the node is not held here. A session that ended under a held entry, and a caller whose budget ran out, are reported as themselves. A node has two lanes. The inference lane owns the claim. The bulk lane is stored under the claim of the inference lane, is refused with ErrNotOwner on a replica that does not hold it, and is closed with it. Open on the bulk lane uses the inference lane when the node has no bulk session, so a transfer on an older worker or during a reconnect works. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
When a peer sweeps a replica that stalled, it deletes the instance row of that replica and records a departure for every connection it owned. The replica registers again and rebuilds the instance row, but the sockets are still open on it. The other replicas then answer "not connected" for workers that are connected. The membership loop now takes a Reclaimer and calls it after a successful registration that follows a sweep. The interface keeps the package free of the code that holds the sockets. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A node gets its own secret for the tunnel. Registration mints a new random token with 128 bits or more, returns it once, and stores only its SHA-256 in a new column. The registration token of the deployment never opens a tunnel, so a leak of it does not let its holder be any worker. Every registration mints a new token, and the worker reads it each time it dials. A node waits for approval with a token that the connect route refuses with 403 until the approval. A node type that holds no credential has the column cleared. The answer to a registration now names the active carrier and its epoch when the endpoint has a carrier reader, and hands over the credential for that carrier only: the tunnel token when the tunnel is active, the NATS user JWT otherwise. Without a reader, or when the row cannot be read, the answer is the one that a deployment with only NATS always got. NATS still requires the address of a backend worker. With the tunnel active the address is optional, and a worker that says that its address is not routable has it dropped and cleared. When the tunnel is active and registration is open, the frontend logs one warning. It does not refuse. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
GET /api/cluster/connect takes the dial of a worker. It reads the bearer token before it checks anything else, so an anonymous dial gets 401. A frontend with no registry answers 503 to a caller that has a token. An unknown node gets 401 and not 404, a lookup that fails gets 500, and a node that waits for approval gets 403. The token is compared in constant time with the hash on the row of the node. The registration token of the deployment is never accepted. The upgrade uses the shared upgrader with 64 KiB buffers. The query parameter lane picks the session. A bulk dial to a replica that does not hold the inference session of the node gets 409 before the upgrade, so the worker can dial again at once and land on the owner. The route is registered on every replica. The global auth middleware lets exactly this path through to its own check. The prefix /api/cluster/ is not delegated. The route coverage test walks the route and a new spec fails if the route is removed. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The proxy that works as a link with a rate and a delay is now a package, so that the specs of the worker can use it on the real endpoint. It also passes the end of a connection on behind the data that is still queued. Before, a connection that ended was closed at once and the data in the queue was lost. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A worker that is attached to the tunnel is ready when it holds a tunnel session. The probe is the counterpart of the one for NATS, and its error is a different one, so that /readyz says which link is down. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…amed The credential manager is no longer only for NATS. The answer to a registration names the carrier and its epoch. When the carrier is the tunnel, the manager waits for the tunnel token and not for the NATS JWT, and it does not wait for the approval of the node: the frontend mints the token at registration and refuses the dial of a node that is still pending, so the tunnel client waits for the approval itself. The token of the latest registration is the one that the client presents, because every registration mints a new one. For NATS, or for a frontend that names no carrier, the manager works as before. RegisterWithRetry now uses RegisterFullWithRetry, which returns the whole answer. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The worker dials the frontend and serves the streams that the frontend opens. It holds two sessions. The inference lane carries model calls and health. The bulk lane carries file transfers and lives only as long as the inference lane. A dial of the bulk lane that lands on a replica which does not hold the inference lane gets 409, and the worker tries again soon. A frontend that refuses the bulk lane does not cost the worker its inference lane. A stream is routed by the tag in its first frame. The host that the frontend names is dropped, and the port must be in the range of the allocator of the worker, so the tunnel cannot reach the LAN of the worker. A refusal says which of four things is true, and a late frame, a timeout and a cancelled context are never evidence about a backend. The wait between two attempts grows from half a second to thirty seconds with jitter, and it goes back to the floor only after a session that lasted. The token is read at every dial. The specs run the client against the real connect endpoint and a real PostgreSQL. Behind a link of 1 Gbit/s with 10 ms of delay in each direction, a transfer of 256 MiB delays a small call on the same session to a p99 of 88 ms, and on the bulk lane to 34 ms, against 31 ms for a call on its own TCP connection. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A spec reads the files of the connect endpoint and of the worker client and fails if one of them builds a gorilla upgrader or dialer by hand. Such a websocket gets the 4 KiB buffers, and a transfer from the worker runs five times slower than in the other direction. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The worker registers first and learns the carrier from the answer. For the tunnel it starts the tunnel client and needs no NATS URL. For NATS, or for a frontend that names no carrier, it connects to NATS as before, and it stops with a message that says so if it has no URL. The flag is no longer required, because for a cluster on the tunnel nothing needs it. The heartbeat and the readiness probe look at the link of the carrier that the worker is attached to. A worker whose link is down does not send its heartbeat, so the registry does not show a node that looks healthy and cannot be reached. A worker that is reached only through its tunnel and was started with no address binds its backends and its file-transfer server to loopback, and tells the frontend that its address is not routable. A worker with an address, or any worker on NATS, binds all interfaces as before. The lifecycle verbs have no route on the tunnel yet, and the worker says so when it starts. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…nd purge what moved MigratePending marks the rows as migrated before they are published. A row that the new carrier refused stayed migrated and was lost. It goes back to pending now, so the next sweep takes it, and only the refused rows go back. Rows that moved stay for a day as evidence and are then deleted with their payloads, in the handoff and on the poll of a consumer. The seams document now says what the claim queue stores: the payload of a row is the whole request of the work, including the configuration of the MCP servers of a job, in plain text until the work completes or the row is purged. It also states the release rules, the failure cap and the cancel path. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…rol token A worker may ask the frontend to publish a progress line on a subject. The allow list of the node type was the only limit, so a worker that held one job could publish progress, results or agent events of any other job or user, and a subject with an empty token or white space reached the filters. - The driver now allows a run only the subjects of the run: the progress and the result of its job, or the events of its agent and user. A run that names no scope keeps the allow list of its node type. - A subject with an empty token, white space or a control character is refused before the filters see it. - agentworker.Start refuses an empty control token. The control server listens on loopback, and an empty token left it open to every process of the host. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…indow gap, re-attach a drain Review of the swap found six robustness defects. - retire waited for the work of the old set without a bound, so one stuck handler held the hand-over of the queue and the close of the set. The wait is now bounded (StopGrace, 30 s), logged, and left behind. Close stops the sets in parallel under the same bound. - Close could block for two minutes in the hand-over, which ran on a context that nothing cancelled. The hand-over now runs on a context that Close cancels after CloseGrace (10 s). - swapTo stored the new set before it opened the window, and a call to a worker attached only to the old carrier failed in the gap. The window opens first. - The holder kept a reconnect hook entry for every set it had ever hooked. It drops the entry when the set is released, on abort and on close too. - A replica that started or restarted during a drain never attached the draining carrier, so workers attached only to it were not reached until the drain ended. It now builds that carrier again, opens the window and starts its work, and tries again at the next poll if it cannot. - A failure to start the work of the active carrier was logged and not reported. The replica now reports not ready with the reason, and ready again once the work starts. The fakes of the swap specs answer with an error once their set is closed, so that the spec "never reaches a closed set" can fail. It now also calls a worker through the window during the drain. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The ready report of a replica never consumed the check, and only its own spec called it. The LISTEN connection is checked when the carrier is built, and a lost connection is redialled by the bus. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…re storing anything The endpoint of the cluster settings checked that a wait parsed, and the store refused one that was not positive only after it had stored the settings before it. A request with a valid address and a wait of 0s stored the address, answered 500, and left half of the request in place. A value such as 1ns passed both checks, and it aborted every later change before a replica could see it. The endpoint now runs the same check as the store on every part of the request before it writes. A wait must be at least five seconds. The timings that a replica reads ignore a stored value below that. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The end of a drain replaced the note of the change with the word "drained", so an admin who looked at the status after a forced change no longer saw that it was forced or which replica it left behind. The drain ending is recorded by the log and the counter. The note stays. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…tabase The switch of the carrier counted a worker as alive when its last heartbeat was younger than the stale limit by the clock of the replica that read it. The heartbeat is stamped by whichever replica handled it, so a skew between two replicas made a live worker look stale, or a dead one look alive. The age is now read on the clock of the database, which every replica shares, and falls back to the local clock only when the database does not answer. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
gosec flagged the plain conversion of the replica index in four requests (G115). The index is a small count, so the conversion now clamps a value that does not fit and does not wrap it, which could have named another replica. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…k was asked A dry run or a request asks every replica what it can build, waits up to five seconds, and then reads the reports. The preflight accepted a report up to 30 seconds old, so a replica that never heard the question passed on the strength of an old answer. The endpoint now measures how long the wait took, and the preflight counts a report only if it is younger than that. A replica that did not answer shows as a blocker that can be forced. A caller that asked nothing keeps the old age. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…is closed The swapper closed the released set at once after the hand-over, and only then waited for the producers that were slow to flip. A call that had taken the draining set from the window, or the old set from the pointer, a moment before they moved could meet a closed set, and a slow producer's work on the old carrier met a closed queue. Under CPU load a spec that calls through the window during the drain failed in 1 run of 10. The set now stays open for the settle time, the second hand-over runs, and the set is closed after it. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The assistant can now read which transport a distributed cluster uses, the state and epoch of a change, the live frontends and the workers that cannot follow. It reads the same report as GET /api/cluster/carrier, in process or over HTTP. No tool changes the carrier. A change moves every frontend and worker and needs a dry run and a check of the frontend list by an admin, so the skill tells the assistant to point the user to the dashboard or the CLI. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…gger Add swagger blocks to the four admin routes under /api/cluster/ and regenerate the spec. Export the request and settings types so the spec can name them, and mark the duration fields as integers (nanoseconds) so the generator can read them. The JSON that the routes send does not change. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
The new command group calls the admin API of a running frontend: local-ai cluster carrier status local-ai cluster carrier switch --to nats|tunnel [--dry-run] [--force] [--yes] [--wait] local-ai cluster carrier abort local-ai cluster settings get local-ai cluster settings set [--nats-url ...] [--nats-worker-url ...] [--max-drain ...] A change runs the dry run first and prints the preflight with the list of live replicas. It then asks whether the list is complete and whether all frontends run this release, because a frontend of an older release cannot be detected. A command with no terminal needs --yes. --force is needed for blockers, and --wait follows the change until it commits or is aborted. Saving a NATS address does not change the carrier, and the command says so. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The panel shows the active carrier, the epoch, the state of a change (stable, prepare, commit, draining), the readiness of each frontend replica, and each worker with its profile, the carriers it is attached to and its follow error. It shows only when the cluster routes answer, so it does not appear on a frontend that is not distributed. The switch dialog runs a dry run and lists the blockers and the live replica list. The admin must confirm that the list is complete and that all frontends are upgraded. A forced change needs a second confirmation. An admin can abort a change during prepare, and a drain shows a countdown. The NATS settings form saves the address and reports whether this frontend reaches it. Saving does not switch the cluster. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Rewrite the distributed mode page for carriers. A cluster runs on one carrier at a time: a NATS broker, or the database with a websocket tunnel, the pgbus and a claim queue. A new deployment with only PostgreSQL starts on the tunnel, and one with a NATS URL stays on NATS. An admin changes the carrier while the cluster runs. The page covers the dry run and its blockers, the confirmation of the frontend list, force, abort, the drain time and what is cut after it, worker profiles and how workers follow, the credential handover, the peer link, security, rolling upgrades and rollback, metrics and logs, the limits that were measured, troubleshooting and a short FAQ. The worker no longer needs a NATS URL, and the page says so. Redraw the architecture diagram, name the peer CA flag --peer-tls-ca as documented, and list get_cluster_carrier for the assistant. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A deployment with only PostgreSQL runs on the tunnel carrier, so the flag no longer needs NATS. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a Ginkgo scenario that starts two frontends and two workers as child processes, with PostgreSQL and NATS in containers. One worker has an address and one is tunnel-only. The scenario runs a dry run, a change from NATS to the tunnel while both frontends answer chat, a change back that the tunnel-only worker blocks, a forced change that leaves that worker unroutable but registered, a frontend that dies during prepare, a worker that dies while it follows, and an abort. It asserts that the backend process of a worker keeps its PID across a change. The spec has the label CarrierSwitch and runs with make test-e2e-distributed-switch. It skips without docker or a Go toolchain. The Distributed label filter of test-e2e-distributed does not select it. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a step that runs make test-e2e-distributed-switch after the distributed suite. It runs with real processes, PostgreSQL and NATS in containers, and it needs no secret, so a pull request from a fork runs it too. The job stays advisory like the suite. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Release removed the set from the hooked map, so the next Listen of the same set registered NotifyReconnect a second time. A set has no way to drop a callback, and the first one stayed live. A set that was released and attached again, as in a drain that re-attaches, ran every reconnect hook twice. Give each registration a live flag. Release switches it off, and the map still does not keep the set alive. The existing spec for one run per reconnect is the regression test. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Adopt ignored the error of the first start of the work. The first poll reports it, but on a stable row the report was never cleared when a later poll started the work, so the replica stayed "not ready" until the next change. Log the error in Adopt, and report ready on a stable row when the last report carried a reason. A spec covers a replica whose first start fails. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…lag from the worker hint The workers table showed "ok" for a worker that predates carrier switching, although it cannot follow a change to the tunnel. It now says that the worker predates carrier switching, and it names the carrier that a worker with limited capabilities cannot follow. A reason that the worker reported is shown as before. The worker hint printed --nats-url and LOCALAI_NATS_URL. A worker learns the carrier from the frontend, so the flag is an optional override. The hint no longer shows it in the commands and says so in one line. The commands also lost their line breaks, because a backslash at the end of a line in a template literal joins the lines; they now keep one flag on each line. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The "active carrier" label of the architecture diagram sat on the edge of the Worker #2 box. Move the workers to the right, so the gap between the control plane and the workers holds the label, and put the label on the middle dashed line. The "embeddings" chip of Worker #3 was wider than its box, so it now reads "embed". The PNG is rendered again from the HTML. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…tive carrier After a forced change back to NATS, a worker that holds only the tunnel stays registered, and its replicas keep the address that they were loaded with. The backend of such a worker listens on its own loopback address, so a request that picked one of its replicas failed with a refused connection and a 500. The guard of the direct client factory did not help, because the recorded address is a host name and not an IP literal. SmartRouter now takes the active carrier. It leaves out of the candidate set every node that is not attached to that carrier and cannot follow it, both when it looks for a loaded replica and when it places a new one, so the replica falls through to another node. When no node is left the request ends with ErrNoAvailableNodes and says why, which is a 503. No row is changed. The set stays "any healthy node" while every node can be reached, so a deployment on NATS routes as before. The list of unreachable nodes is cached for two seconds. The switch e2e no longer tolerates a failed chat during the forced change. Before the fix that spec failed in one of three runs; it now passes in three of three. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…n the status is read A read of the carrier state ran the preflight without asking the replicas to look again. A replica writes what it can build only every few minutes, so for most of that time its report was older than the age that a change accepts. The state then listed it as a forceable blocker for a change back to NATS, although the dry run, which asks and waits, found nothing wrong. Add Switch.PreflightForStatus. It keeps the age in the report and does not list an old report as a blocker. A replica that says it cannot use the carrier, and one that never reported, still are. The status endpoint uses it, and it asks the replicas to look again, at most once in 15 seconds, because the page polls it. The dry run is unchanged: it asks the replicas and waits for them, bounded. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
… interrupts The spec for a frontend that dies during prepare only waited until the change left prepare. The change is forced, so the product commits it on the tunnel without the dead frontend. The spec now asserts that the row is stable on the tunnel and not aborted, that both workers follow, that the frontend that is left answers chat all along with no failure, and that the restarted frontend follows the row. The spec for a worker that dies while it follows now asserts that the row commits on the tunnel, that the dead worker stays registered, that both workers are attached to the tunnel after the restart, and that chat through both frontends then runs with no failure. Requests that need the worker while it is down can fail, so the spec asserts nothing about that window. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
… write The spec for streams that the worker opens wrote to each stream and expected the write to work. The session resets such a stream as soon as it is opened, and under load the reset can arrive before the write, so the write failed with a reset by the remote. A reset is the behaviour under test. The spec now accepts a write that fails with a reset from the remote and fails on any other write error. Under 40 busy loops, 8 of 48 runs failed before and none of 48 fail now. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
mudler-agent
changed the title
mudler-agent
changed the base branch from
feat/distributed-live-carrier
to
master
October 13, 2026 02:06
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Distributed mode can now switch its transport between NATS and a database-backed carrier at runtime, with no restart and no boot-time mode flag. Exactly one carrier is active cluster-wide. The active carrier and its epoch live in a
cluster_carrierrow.The database-backed carrier has three parts:
SKIP LOCKED, a 2 s poll plus a NOTIFY wake hint.NATS keeps working as before. Saving a NATS URL only stores it and checks reachability. The switch is an explicit step with a dry-run preflight.
Switching
POST /api/cluster/carrierwithdry_run,forceandabort;GET /api/cluster/carrierfor status;/api/cluster/settingsfor the NATS URL and timings.max_drain(default 15 min).forceoverrides; those workers report afollow_errorand are not routed to until they can follow.--addr).Index of the change (commit series)
cluster_carrierstoreinstances, membership, node connection claims)Credentials are per node; open registration is allowed but logs a warning. The peer link supports
--peer-address,--peer-tlsand--peer-tls-ca.Testing
make test-e2e-distributed-switch: real processes (frontend, workers, Postgres, NATS). It switches NATS to database and back with zero chat failures through a forced flip, and covers interrupted frontend and worker changes.make test-e2e-distributed(existing NATS suite): 237 passed, 0 failed.golangci-lintclean with and without theauthtag;go vet, and Darwin and Windows vet of the touched packages.Known follow-ups
MaxFailureshas no CLI flag.-raceload; it did not reproduce in isolation.This supersedes #11812, which was used as a reference only.
🤖 Generated with Claude Code