Skip to content

fix(watch): preserve declared-pause stale suppression#720

Open
raywest wants to merge 3 commits into
kunchenguid:mainfrom
raywest:fm/fm-watch-mergewait-stale-b1
Open

fix(watch): preserve declared-pause stale suppression#720
raywest wants to merge 3 commits into
kunchenguid:mainfrom
raywest:fm/fm-watch-mergewait-stale-b1

Conversation

@raywest

@raywest raywest commented Jul 19, 2026

Copy link
Copy Markdown

Intent

Fix a live, reproducible watcher bug in bin/fm-watch.sh: a done-then-declared-paused crewmate (e.g. a green-CI PR sitting idle awaiting the captain's merge decision) refired a bare 'stale: ' wake on EVERY freshly-armed watcher, even though the .stale- hash suppressor was already correctly set to the current pane hash. Root cause: once a stale hash is already classified for a task whose status log's last line is 'paused:', the watcher re-derives the absorb verdict via crew_absorb_class/pause_state_class on every recheck. A stale/misattributed no-mistakes run-step (e.g. an aborted run whose terminal outcome predates the crew's LATER paused: declaration) can report a non-working, non-paused verdict from fm-crew-state.sh; the old code treated ANY such verdict the same as a fresh 'crew has stopped' first-sight event, firing the bare stale wake and also deleting the .paused- marker as a side effect - which guaranteed the identical misfire on every subsequent poll or watcher restart forever, since crew_absorb_class kept re-reading the same stale aborted-run record. Fix: at all three call sites in bin/fm-watch.sh that consume pause_state_class/crew_absorb_class for a crew whose current status-log line is 'paused:', only a definitive 'working' verdict now breaks the paused treatment; any other verdict (including this misattributed non-working/non-paused case) stays in the existing paused/long-recheck-cadence treatment (handle_paused_stale, anchored on PAUSE_RESURFACE_SECS / the status file's own mtime) instead of surfacing immediately. Genuine wedge detection for a plain (non-paused) stale pane is completely untouched - the escalation timer path (wedge_timer_check, STALE_ESCALATE_SECS) still works exactly as before. Deliberately scoped narrow per the task brief: this fixes only the restart-suppression defect in fm-watch.sh; it does NOT attempt the larger redesign discussed in upstream issues #703 ('Ship task awaiting captain merge is indistinguishable from a wedged crewmate') and #694 ('Watcher cannot hold a wait when the sole in-flight task is a correctly-paused pane') - that boundary is intentional and should be preserved in review. Verification: reproduced the exact live bug in an isolated fixture with a fake fm-crew-state.sh (bare stale refiring on every fresh watcher invocation despite the suppressor already matching the pane hash), added a colocated regression test (tests/fm-watch-triage.test.sh: test_nonterminal_stale_paused_survives_misattributed_run_step_across_restarts) that fails against the pre-fix code and passes post-fix, ran it and the full existing fm-watch-triage.test.sh suite (all green, including all pre-existing declared-pause/resurface/wedge tests, which still validate promptly-detected resumption and untouched wedge escalation), and cross-checked the adjacent watcher/daemon/wake-queue/crew-state/pi-extension suites (all green except one confirmed pre-existing flake in fm-watcher-lock.test.sh that reproduces identically against unmodified HEAD, unrelated to this change). bin/fm-lint.sh (shellcheck 0.11.0, the pinned CI version) is clean.

What Changed

  • Keep stale panes on declared-pause handling unless the watcher has definitive working evidence, preventing stale/misattributed run-state reads from repeatedly surfacing bare stale wakes.
  • Preserve and throttle paused stale rechecks across watcher restarts while leaving working-pane wedge escalation unchanged.
  • Add regression coverage for the restart case and document the declared-pause precedence rule.

Risk Assessment

✅ Low: Captain, the follow-up correctly throttles non-working paused rechecks while preserving the intended narrow stale-suppression behavior.

Testing

Completed 1 recorded test check.

  • Outcome: ⏭️ skipped across 5 runs (1h57m55s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed ✅
  • ⚠️ bin/fm-watch.sh:973 - For the persistent none verdict this now absorbs, pause_state_class removes .paused-rechecked-*, so every subsequent poll re-runs crew_absorb_class instead of using its paused recheck throttle. That can invoke the bounded no-mistakes state read every FM_POLL (up to 10 seconds each) for the exact misattributed-run condition. Retain/rearm the recheck marker for non-working paused verdicts while preserving the periodic working-state check.

🔧 Fix: Throttle paused stale rechecks
✅ Re-checked - no issues remain.

⏭️ **Test** - skipped
  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"

🔧 Fix: Preserve declared-pause stale suppression
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"

🔧 Fix: Validate watcher stale suppression
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"

🔧 Fix: Confirm watcher tests pass
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"

🔧 Fix: Preserve declared-pause stale suppression
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

Ray West added 3 commits July 18, 2026 19:11
A done-then-paused crew (a green-CI PR awaiting the captain's merge
decision) refired a bare "stale: <window>" wake on every freshly-armed
watcher, even though .stale-<key> already held the current pane hash.

Root cause: once a stale hash is already classified and the crew's own
status log still says paused:, the watcher re-derives the verdict via
crew_absorb_class on every recheck. A stale/misattributed no-mistakes
run-step (e.g. an aborted run whose terminal outcome predates the later
paused: line) can report a non-working, non-paused verdict; the old code
treated any such verdict as a first-sight surface, which also deleted the
.paused-<key> flag - guaranteeing the identical misfire on the next poll
or restart forever.

Fixed by making only a definitive working verdict break a declared
pause; any other verdict (including a misattributed run-step outcome)
stays in the paused/long-recheck-cadence treatment, at all three sites
that read pause_state_class/crew_absorb_class. Genuine wedge detection
for a non-paused stale pane is untouched.

This fixes the restart-suppression defect only. The larger question of
how firstmate should distinguish a done-awaiting-merge crew from a
wedged one (kunchenguid#703, kunchenguid#694) is out of scope here.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant