From 81cb101b8510357d6d09feed99457472c0515e74 Mon Sep 17 00:00:00 2001 From: SamPlvs Date: Sun, 5 Jul 2026 15:53:35 +0530 Subject: [PATCH] docs(spec): add watchdog/heartbeat anti-stall design spec Design spec (RFC) for an orchestrator-owned watchdog that closes the idle-stall hole: an event-driven lead receives no wake signal when every teammate goes idle or a background job dies silently, so a run can flat-line unnoticed. Spec covers a scheduled heartbeat tick, evidence-based liveness (pgrep + artifact deltas, not absence-of-bad-news), a long-job failure-aware monitor, remediation escalation (nudge -> respawn -> reroute), config, and integration points. Implementation follows in a separate PR. Co-Authored-By: Claude Opus 4.8 --- specs/watchdog.md | 93 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 specs/watchdog.md diff --git a/specs/watchdog.md b/specs/watchdog.md new file mode 100644 index 0000000..765b40b --- /dev/null +++ b/specs/watchdog.md @@ -0,0 +1,93 @@ +# Watchdog / Heartbeat — anti-stall for long-running autonomous runs + +**Status:** design spec (RFC). Implementation tracked as the follow-up to this PR. +**Owner:** core / orchestrator. +**Motivation:** a real failure mode observed in a long autonomous run — the whole team stalled *silently for ~38 hours* and nobody noticed until the human asked "where are we?". + +--- + +## 1. The failure mode + +ZO's lead-orchestrator is **event-driven**: it acts when it receives a message (from the human or from a teammate agent). This is efficient, but it has a silent-death hole: + +- When every teammate agent finishes its immediate task and goes **idle** (or dies silently), no teammate message is emitted. +- The orchestrator therefore receives **no wake signal**, so it is never re-invoked. +- A long **background job** (a training run, a sweep, an external CI/deploy) emits no agent message while it runs — and if its launcher dies, nothing reports it. +- Net: "wait for the next message" silently degrades into **"do nothing, indefinitely."** The run looks alive (no error) but has flat-lined. + +Contributing factors seen in the wild: +- A dedicated monitor agent (a "context-warden") is itself a single point of failure — it can die silently too, taking the only liveness check with it. +- No-news is read as progress. Absence of an error is not evidence of forward motion. + +This is distinct from context-window saturation (already handled by checkpoint→respawn). This is **idle-stall**: healthy-looking agents that have simply stopped moving the work forward. + +--- + +## 2. Requirements + +1. **Active heartbeat, not passive wait.** The run must have a wake source that fires on a wall-clock schedule independent of teammate messages, so an all-idle team cannot go unnoticed. +2. **Liveness by evidence, not by silence.** Detect progress from observable state (process is running, output files/heartbeats advancing), never from "no bad message arrived." +3. **Auto-remediation.** On a detected stall, re-mobilize (nudge/respawn the responsible agent, re-issue its task) rather than only alerting. +4. **No single point of failure.** The heartbeat is owned by the orchestrator/run itself, not by one killable monitor agent. +5. **Survives session boundaries.** Session-scoped timers (cron/wakeup) are re-armed at the start of every session; the *policy* lives in durable config so a fresh session re-establishes the watch automatically. +6. **Cheap + quiet.** Low token/compute cost; only speaks when it detects a stall or a state change, not on every tick. + +--- + +## 3. Design + +### 3.1 Heartbeat tick (scheduled self-invoke) +A recurring, off-minute schedule (default `*/17 * * * *` — ~every 17 min, jittered, never on `:00`) enqueues a **watchdog tick** prompt to the orchestrator. Session-scoped by nature, so it is re-armed on session start from `project_config`. This is the wake source that closes the "all-idle → never re-invoked" hole. + +### 3.2 Liveness probe (evidence-based) +Each tick evaluates, per active agent / critical-path job: +- **Agent liveness:** heartbeat file mtime fresh (< `stall_threshold`, default 20 min) AND/OR a live owned process. +- **Job liveness:** the real process PID is running (not just a wrapper/launcher that already exited); output artifacts advancing (new/growing files since last tick). +- **Progress delta:** did any tracked artifact change since the previous tick? + +A **stall** = no progress delta AND no live critical-path process for ≥ N consecutive ticks (default N=1 for a hard stall, N=2 for a soft one). + +### 3.3 Long-job monitor (event stream) +For a specific long-running job, attach a streaming monitor on its log with a filter covering **both** progress and failure signatures (`epoch|elapsed|wrote|saved|Traceback|Error|FAILED|Killed|OOM|nan`). Silence-is-not-success: a filter that matches only the happy path stays quiet through a crashloop. This gives immediate notification on crash/completion between heartbeat ticks. + +### 3.4 Remediation policy +On stall detection the watchdog escalates by policy: +1. **Nudge** the responsible agent (re-send its outstanding task). +2. If still dead next tick → **respawn** it from its on-disk resume/checkpoint and re-issue. +3. Prefer a **reliable executor**: if an agent role has failed to engage repeatedly (e.g., two silent dead spawns), route its critical-path task to a known-reliable agent instead of retrying the same role. Log the reroute. +4. **Never silently truncate**: every stall, reroute, and dropped item is logged and surfaced to the human at the next checkpoint. + +### 3.5 Configuration (`project_config`) +``` +watchdog: + enabled: true + tick_cron: "*/17 * * * *" # off-minute; re-armed each session + stall_threshold_min: 20 # heartbeat/artifact staleness => suspect + stall_ticks_hard: 1 # no-process + no-progress => stall now + stall_ticks_soft: 2 # progress-but-slow => stall after N ticks + remediation: [nudge, respawn, reroute] + monitor_failure_signatures: ["Traceback","Error","FAILED","Killed","OOM","nan"] +``` + +--- + +## 4. Integration points +- **`orchestrator.py`** — arm the heartbeat cron on run/session start (re-arm if a prior schedule is gone); own the watchdog-tick handler. +- **`comms.py`** — the watchdog reads agent heartbeats + emits nudge/respawn messages through the existing comms bus. +- **`.claude/agents/lead-orchestrator.md`** — add the standing rule: never end a turn in passive "hold and wait" while critical-path work is outstanding; a heartbeat must be armed. Verify liveness by evidence (pgrep the real PID, artifact deltas), not by absence of bad news. +- **`project_config.py`** — the `watchdog:` block above; default-on. +- A dedicated monitor agent (if any) becomes a *helper*, not the sole mechanism — the orchestrator-owned heartbeat is the backstop. + +--- + +## 5. Acceptance / tests +- Unit: stall-detection predicate (fresh vs stale heartbeat; live vs dead PID; progress delta present/absent) → correct hard/soft/no-stall classification. +- Unit: remediation escalation (nudge → respawn → reroute) state machine. +- Unit: config parse + defaults + re-arm idempotency (arming twice doesn't double-schedule). +- Integration: simulate an all-idle team with no progress across ticks → watchdog fires remediation; simulate a live-but-slow job → no false stall. +- Regression: the failure-signature filter matches a crashloop/OOM sample (silence-is-not-success guard). + +--- + +## 6. Out of scope (this PR is spec-only) +Code, tests, and orchestrator wiring land in the follow-up implementation PR against this spec. Context-window saturation handling (checkpoint→respawn) already exists and is unchanged — this feature covers **idle-stall**, the orthogonal hole.