Skip to content

fix(orchestrator): heartbeat + restart recovery without false disconnects - #105

Merged
setkyar merged 3 commits into
mainfrom
hotfix/worker-heartbeat
Aug 11, 2026
Merged

fix(orchestrator): heartbeat + restart recovery without false disconnects#105
setkyar merged 3 commits into
mainfrom
hotfix/worker-heartbeat

Conversation

@setkyar

@setkyar setkyar commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

⚠️ Do not auto-merge or deploy

The affected worker is healthy right now. It is connected, serving WhatsApp
traffic, and its child process has been alive as long as its parent. Nothing is
broken in production today.

Merging changes the orchestrator image and its Compose stop grace period, so
the next deploy.sh will replace the orchestrator container — which owns the
WhatsApp worker as a child process, dropping the live session.
Because the
Compose change alters container configuration, the orchestrator is recreated
even if the image were byte-identical; there is no config-only path. Land this
at a moment you are willing to reconnect WhatsApp.

CI here runs checks only; it does not deploy. Deployment is the manual
deploy.sh on the Droplet.


What this fixes

1. A heartbeat that never beat (302b165)

WorkerRegistry.UpdateHeartbeat existed but had no caller anywhere in the
service
. last_heartbeat was written once by RegisterWorker and never
again, so a worker connected and serving traffic for fourteen hours still
reported a fourteen-hour-old heartbeat — indistinguishable from an abandoned
row. Anything alerting on heartbeat age fired permanently and told an operator
nothing.

The health check already proves liveness every tick by signalling the process
with 0, so it records the heartbeat there. A failed write is logged, not fatal:
stopping a worker that is running fine and holding a live WhatsApp session
because a bookkeeping UPDATE failed would be far worse than the stale row.

2. Restart reported as a crash (df09745)

Replacing the orchestrator always leaves every worker's registry row pointing at
a dead PID, because the workers are its children. Only the durable status
separates a planned restart from a crash, and shutdown was not recording it
reliably.

  • Shutdown marks every worker for recovery in one statement before
    signalling any process, so a SIGKILL partway through cannot leave later
    workers unmarked.
  • Stops run concurrently. Each costs up to 5s grace plus a 2s kill wait, so
    stopping in turn fitted only ~4 workers into the 30s budget while
    GLOBAL_MAX_ACTIVE_CONNECTIONS permits far more; the rest died with the
    container instead of closing their sessions.
  • Recovery announces a dead marked worker as connecting, not as an error.
    Announcing a planned restart as a failure raised a "WhatsApp disconnected"
    alert seconds before the same worker reconnected, which teaches people to
    ignore the alert that matters.
  • stop_grace_period: 40s on the orchestrator. Not cosmetic — production
    currently has StopTimeout: unset → Docker's 10s default, which SIGKILLs the
    process partway through the shutdown above. The guard test reads the Compose
    file and fails without it.

3. Two holes found reviewing this branch (77e0535)

Both were introduced by this branch, and both sit in the path it exists to
harden.

A surviving worker is never announced, whatever its record says. The
previous revision still published connecting when the record read
recovering, reasoning that this orchestrator's own shutdown wrote that marker
so it could be trusted. But that path is only reached when the process is
alive, which means the stop it recorded never took effect: the worker never
left and its session is still up. Announcing connecting downgraded a live
connection exactly as the spawn-time default did, and nothing corrected it,
because a survivor has no reason to re-announce itself. A record that
contradicts the observed process is not evidence about the session.

Scenario: orchestrator SIGKILLed between marking and the child's death —
docker kill, host OOM, grace exhaustion. The child is orphaned but alive.
The replacement sees alive + recovering and pushes a connected workspace to
connecting, permanently, while messages keep flowing.

The pre-stop registry write is bounded at 5s, instead of running on the
caller's full 30s shutdown budget. It sits ahead of every SIGTERM, so a slow or
unreachable PostgreSQL spent the time meant for closing WhatsApp sessions and
left the container to SIGKILL workers mid-session — the exact outcome the
marking prevents, reached by way of a database problem. On timeout it gives up
and stops the workers; stopWorkerInternal marks each record again as it goes,
which is what that second marking is for.


Validation

  • Full orchestrator suite green: internal/api, internal/manager,
    internal/nats, internal/types
  • Race detector clean across the whole manager package
  • go build, go vet, gofmt clean

Mutation evidence

Every guard in this PR was verified to fail when its behaviour is reverted:

Guard Reverted behaviour produces
TestSurvivorAnnouncement_RecoveringIsAlsoDeclined "a surviving worker marked recovering was announced as connecting; the stop never took effect, so the session is still up"
TestStop_MarkingCannotConsumeTheShutdownBudget runs the full 25s budget, and both workers fail with "did not exit: context deadline exceeded" — the production failure, not a code shape
TestProductionComposeGivesOrchestratorRoomToShutDown "Docker's 10s default SIGKILLs it partway through stopping workers"

TestStop_StopsWorkersConcurrently measures wall clock and asserts
< serial/2 and < shutdownBudget, so a regression to serial stops is
caught rather than assumed.

Scope — deliberately excluded

Orchestrator only. Not included:

  • the API shutdown work (apps/api/src/index.ts, src/lib/shutdown.ts and its
    tests) and the API stop_grace_period — separate change
  • the docs/deployment.md rewrite: its anchor section does not exist on main
    yet, so the paragraph would land in unrelated context and conflict when the
    API work arrives. The Compose file carries a full inline comment pointing at
    the guard test instead.
  • all unrelated in-flight work (contacts/tags, web, cloud-control)

Known, not addressed here

  • stopWorkerInternal reads healthCancel, PID, cmd and cancelFunc from
    the live map pointer after releasing the mutex. Safe today — concurrent stops
    touch distinct workers and -race passes on real processes — but it holds by
    argument rather than by construction.
  • MarkWorkersRecovering and UpdateStatus both set last_heartbeat = now(),
    so heartbeat freshness alone cannot distinguish "running" from "just marked
    for recovery". Anything alerting on it should read status too.
  • worker_registry.status still never advances past its spawn-time value. Making
    it accurate needs the orchestrator to learn connection status from worker
    events — a real design change, not a bug fix.

Rollback

APP_IMAGE_TAG=<previous release> docker compose ... up -d --no-build --wait.

The Compose stop grace period is not covered by the image tag, so a rollback
must revert the checkout as well. A tag-only rollback leaves 40s grace running
old code — harmless, but not a true revert.

Post-deploy checks (read-only)

  1. last_heartbeat advances within one HEALTH_CHECK_INTERVAL (30s)
  2. the whatsapp-worker child process is alive under the orchestrator
  3. the tenant connection did not flip to connecting

setkyar added 3 commits August 9, 2026 18:28
WorkerRegistry.UpdateHeartbeat had no caller anywhere in the service, so
last_heartbeat was written once by RegisterWorker and never again. A worker
that had been connected and serving WhatsApp traffic for fourteen hours still
reported a fourteen-hour-old heartbeat, which is indistinguishable from an
abandoned row. The column could not answer the one question it exists for, and
anything alerting on heartbeat age fired permanently and so told an operator
nothing.

The health check already proves liveness every tick by signalling the process
with 0, so it is the right place to record it. Wired through a field rather
than calling the registry directly, so the behaviour is testable without a
database.

A failed write is logged, not fatal. Stopping a worker that is running
perfectly well and holding a live WhatsApp session, because a bookkeeping
UPDATE failed, would be a far worse outcome than the stale row it guards
against.

This is bookkeeping only. It changes no connection status, publishes no event,
and starts or stops nothing.
Replacing the orchestrator container always leaves every worker's registry row
pointing at a PID that no longer exists, because the workers are its child
processes. Only the durable status separates a planned restart from a crash,
and shutdown was not recording it reliably.

Shutdown now marks every worker for recovery in one statement before signalling
any process, so a SIGKILL partway through cannot leave later workers unmarked.
It stops them concurrently: each stop costs up to 5s of grace plus a 2s kill
wait, so stopping in turn fitted only about four workers into the 30s budget
while GLOBAL_MAX_ACTIVE_CONNECTIONS permits far more, and the rest died with the
container instead of closing their WhatsApp sessions. Failures are collected and
returned rather than swallowed.

Recovery then announces a marked worker as connecting rather than as an error.
Announcing a planned restart as a failure raised a WhatsApp disconnected alert
at every operator seconds before the same worker reconnected, which teaches
people to ignore the alert that matters.

A surviving worker is now left alone entirely. worker_registry is written once
at spawn and never advanced as the session comes up, so a worker connected for
hours still carries "connecting"; republishing that on recovery pushed a
connection the API held as connected back to connecting, where nothing would
correct it because the process survived and had no reason to re-announce
itself. The orchestrator cannot observe the WhatsApp session, so it says
nothing and leaves the API holding the last status the worker itself reported.
Claiming connected would be inventing state; that is the trade this makes
deliberately.

compose.production.yml raises the orchestrator stop_grace_period to 40s. That
is not cosmetic: Docker's 10s default SIGKILLs the process partway through the
shutdown above, which is the failure this whole change exists to prevent. The
guard test reads the compose file and fails without it.

Scoped to the orchestrator. The API shutdown work, its compose grace period,
and the deployment-doc rewrite are deliberately not included.
Both sit in the shutdown and recovery path this branch exists to harden, and
both were introduced by it.

A surviving worker is now never announced, whatever its record says. The
previous revision still published "connecting" when the record read
"recovering", on the reasoning that this orchestrator's own shutdown wrote that
marker so it could be trusted. But survivorAnnouncement is only consulted when
the process is alive, which means the stop it recorded never took effect: the
worker never left and its WhatsApp session is still up. Announcing "connecting"
downgraded a live connection exactly as the spawn-time default did, and nothing
corrected it, because a survivor has no reason to re-announce itself. A record
that contradicts the observed process is not evidence about the session. The
argument is kept so the rule stays one auditable decision and a future status
has to be considered here rather than silently acquiring a meaning.

The pre-stop registry write is now bounded at 5s instead of running on the
caller's full shutdown budget. It sits ahead of every SIGTERM, so a slow or
unreachable PostgreSQL spent the time meant for closing WhatsApp sessions and
left the container to SIGKILL the workers mid-session — the exact outcome the
marking exists to prevent, reached by way of a database problem. On timeout it
gives up and goes to stop the workers; stopWorkerInternal marks each record
again as it goes, which is what that second marking is for.

Both are pinned by tests that fail when the behaviour is reverted. The budget
test is the useful one: restoring the caller's context makes it run for the
whole 25s budget and leaves both workers reporting "did not exit: context
deadline exceeded", which is the production failure rather than a code shape.
@setkyar
setkyar merged commit 53e2807 into main Aug 11, 2026
1 check passed
@setkyar
setkyar deleted the hotfix/worker-heartbeat branch August 11, 2026 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant