fix(orchestrator): heartbeat + restart recovery without false disconnects - #105
Merged
Conversation
WorkerRegistry.UpdateHeartbeat had no caller anywhere in the service, so last_heartbeat was written once by RegisterWorker and never again. A worker that had been connected and serving WhatsApp traffic for fourteen hours still reported a fourteen-hour-old heartbeat, which is indistinguishable from an abandoned row. The column could not answer the one question it exists for, and anything alerting on heartbeat age fired permanently and so told an operator nothing. The health check already proves liveness every tick by signalling the process with 0, so it is the right place to record it. Wired through a field rather than calling the registry directly, so the behaviour is testable without a database. A failed write is logged, not fatal. Stopping a worker that is running perfectly well and holding a live WhatsApp session, because a bookkeeping UPDATE failed, would be a far worse outcome than the stale row it guards against. This is bookkeeping only. It changes no connection status, publishes no event, and starts or stops nothing.
Replacing the orchestrator container always leaves every worker's registry row pointing at a PID that no longer exists, because the workers are its child processes. Only the durable status separates a planned restart from a crash, and shutdown was not recording it reliably. Shutdown now marks every worker for recovery in one statement before signalling any process, so a SIGKILL partway through cannot leave later workers unmarked. It stops them concurrently: each stop costs up to 5s of grace plus a 2s kill wait, so stopping in turn fitted only about four workers into the 30s budget while GLOBAL_MAX_ACTIVE_CONNECTIONS permits far more, and the rest died with the container instead of closing their WhatsApp sessions. Failures are collected and returned rather than swallowed. Recovery then announces a marked worker as connecting rather than as an error. Announcing a planned restart as a failure raised a WhatsApp disconnected alert at every operator seconds before the same worker reconnected, which teaches people to ignore the alert that matters. A surviving worker is now left alone entirely. worker_registry is written once at spawn and never advanced as the session comes up, so a worker connected for hours still carries "connecting"; republishing that on recovery pushed a connection the API held as connected back to connecting, where nothing would correct it because the process survived and had no reason to re-announce itself. The orchestrator cannot observe the WhatsApp session, so it says nothing and leaves the API holding the last status the worker itself reported. Claiming connected would be inventing state; that is the trade this makes deliberately. compose.production.yml raises the orchestrator stop_grace_period to 40s. That is not cosmetic: Docker's 10s default SIGKILLs the process partway through the shutdown above, which is the failure this whole change exists to prevent. The guard test reads the compose file and fails without it. Scoped to the orchestrator. The API shutdown work, its compose grace period, and the deployment-doc rewrite are deliberately not included.
Both sit in the shutdown and recovery path this branch exists to harden, and both were introduced by it. A surviving worker is now never announced, whatever its record says. The previous revision still published "connecting" when the record read "recovering", on the reasoning that this orchestrator's own shutdown wrote that marker so it could be trusted. But survivorAnnouncement is only consulted when the process is alive, which means the stop it recorded never took effect: the worker never left and its WhatsApp session is still up. Announcing "connecting" downgraded a live connection exactly as the spawn-time default did, and nothing corrected it, because a survivor has no reason to re-announce itself. A record that contradicts the observed process is not evidence about the session. The argument is kept so the rule stays one auditable decision and a future status has to be considered here rather than silently acquiring a meaning. The pre-stop registry write is now bounded at 5s instead of running on the caller's full shutdown budget. It sits ahead of every SIGTERM, so a slow or unreachable PostgreSQL spent the time meant for closing WhatsApp sessions and left the container to SIGKILL the workers mid-session — the exact outcome the marking exists to prevent, reached by way of a database problem. On timeout it gives up and goes to stop the workers; stopWorkerInternal marks each record again as it goes, which is what that second marking is for. Both are pinned by tests that fail when the behaviour is reverted. The budget test is the useful one: restoring the caller's context makes it run for the whole 25s budget and leaves both workers reporting "did not exit: context deadline exceeded", which is the production failure rather than a code shape.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The affected worker is healthy right now. It is connected, serving WhatsApp
traffic, and its child process has been alive as long as its parent. Nothing is
broken in production today.
Merging changes the orchestrator image and its Compose stop grace period, so
the next
deploy.shwill replace the orchestrator container — which owns theWhatsApp worker as a child process, dropping the live session. Because the
Compose change alters container configuration, the orchestrator is recreated
even if the image were byte-identical; there is no config-only path. Land this
at a moment you are willing to reconnect WhatsApp.
CI here runs checks only; it does not deploy. Deployment is the manual
deploy.shon the Droplet.What this fixes
1. A heartbeat that never beat (
302b165)WorkerRegistry.UpdateHeartbeatexisted but had no caller anywhere in theservice.
last_heartbeatwas written once byRegisterWorkerand neveragain, so a worker connected and serving traffic for fourteen hours still
reported a fourteen-hour-old heartbeat — indistinguishable from an abandoned
row. Anything alerting on heartbeat age fired permanently and told an operator
nothing.
The health check already proves liveness every tick by signalling the process
with 0, so it records the heartbeat there. A failed write is logged, not fatal:
stopping a worker that is running fine and holding a live WhatsApp session
because a bookkeeping
UPDATEfailed would be far worse than the stale row.2. Restart reported as a crash (
df09745)Replacing the orchestrator always leaves every worker's registry row pointing at
a dead PID, because the workers are its children. Only the durable status
separates a planned restart from a crash, and shutdown was not recording it
reliably.
signalling any process, so a SIGKILL partway through cannot leave later
workers unmarked.
stopping in turn fitted only ~4 workers into the 30s budget while
GLOBAL_MAX_ACTIVE_CONNECTIONSpermits far more; the rest died with thecontainer instead of closing their sessions.
connecting, not as an error.Announcing a planned restart as a failure raised a "WhatsApp disconnected"
alert seconds before the same worker reconnected, which teaches people to
ignore the alert that matters.
stop_grace_period: 40son the orchestrator. Not cosmetic — productioncurrently has
StopTimeout: unset → Docker's 10s default, which SIGKILLs theprocess partway through the shutdown above. The guard test reads the Compose
file and fails without it.
3. Two holes found reviewing this branch (
77e0535)Both were introduced by this branch, and both sit in the path it exists to
harden.
A surviving worker is never announced, whatever its record says. The
previous revision still published
connectingwhen the record readrecovering, reasoning that this orchestrator's own shutdown wrote that markerso it could be trusted. But that path is only reached when the process is
alive, which means the stop it recorded never took effect: the worker never
left and its session is still up. Announcing
connectingdowngraded a liveconnection exactly as the spawn-time default did, and nothing corrected it,
because a survivor has no reason to re-announce itself. A record that
contradicts the observed process is not evidence about the session.
Scenario: orchestrator SIGKILLed between marking and the child's death —
docker kill, host OOM, grace exhaustion. The child is orphaned but alive.The replacement sees alive +
recoveringand pushes a connected workspace toconnecting, permanently, while messages keep flowing.The pre-stop registry write is bounded at 5s, instead of running on the
caller's full 30s shutdown budget. It sits ahead of every SIGTERM, so a slow or
unreachable PostgreSQL spent the time meant for closing WhatsApp sessions and
left the container to SIGKILL workers mid-session — the exact outcome the
marking prevents, reached by way of a database problem. On timeout it gives up
and stops the workers;
stopWorkerInternalmarks each record again as it goes,which is what that second marking is for.
Validation
internal/api,internal/manager,internal/nats,internal/typesmanagerpackagego build,go vet,gofmtcleanMutation evidence
Every guard in this PR was verified to fail when its behaviour is reverted:
TestSurvivorAnnouncement_RecoveringIsAlsoDeclinedrecoveringwas announced asconnecting; the stop never took effect, so the session is still up"TestStop_MarkingCannotConsumeTheShutdownBudgetTestProductionComposeGivesOrchestratorRoomToShutDownTestStop_StopsWorkersConcurrentlymeasures wall clock and asserts< serial/2and< shutdownBudget, so a regression to serial stops iscaught rather than assumed.
Scope — deliberately excluded
Orchestrator only. Not included:
apps/api/src/index.ts,src/lib/shutdown.tsand itstests) and the API
stop_grace_period— separate changedocs/deployment.mdrewrite: its anchor section does not exist onmainyet, so the paragraph would land in unrelated context and conflict when the
API work arrives. The Compose file carries a full inline comment pointing at
the guard test instead.
Known, not addressed here
stopWorkerInternalreadshealthCancel,PID,cmdandcancelFuncfromthe live map pointer after releasing the mutex. Safe today — concurrent stops
touch distinct workers and
-racepasses on real processes — but it holds byargument rather than by construction.
MarkWorkersRecoveringandUpdateStatusboth setlast_heartbeat = now(),so heartbeat freshness alone cannot distinguish "running" from "just marked
for recovery". Anything alerting on it should read
statustoo.worker_registry.statusstill never advances past its spawn-time value. Makingit accurate needs the orchestrator to learn connection status from worker
events — a real design change, not a bug fix.
Rollback
APP_IMAGE_TAG=<previous release> docker compose ... up -d --no-build --wait.The Compose stop grace period is not covered by the image tag, so a rollback
must revert the checkout as well. A tag-only rollback leaves 40s grace running
old code — harmless, but not a true revert.
Post-deploy checks (read-only)
last_heartbeatadvances within oneHEALTH_CHECK_INTERVAL(30s)whatsapp-workerchild process is alive under the orchestratorconnecting