fix(watch): preserve declared-pause stale suppression#720
Open
raywest wants to merge 3 commits into
Open
Conversation
added 3 commits
July 18, 2026 19:11
A done-then-paused crew (a green-CI PR awaiting the captain's merge decision) refired a bare "stale: <window>" wake on every freshly-armed watcher, even though .stale-<key> already held the current pane hash. Root cause: once a stale hash is already classified and the crew's own status log still says paused:, the watcher re-derives the verdict via crew_absorb_class on every recheck. A stale/misattributed no-mistakes run-step (e.g. an aborted run whose terminal outcome predates the later paused: line) can report a non-working, non-paused verdict; the old code treated any such verdict as a first-sight surface, which also deleted the .paused-<key> flag - guaranteeing the identical misfire on the next poll or restart forever. Fixed by making only a definitive working verdict break a declared pause; any other verdict (including a misattributed run-step outcome) stays in the paused/long-recheck-cadence treatment, at all three sites that read pause_state_class/crew_absorb_class. Genuine wedge detection for a non-paused stale pane is untouched. This fixes the restart-suppression defect only. The larger question of how firstmate should distinguish a done-awaiting-merge crew from a wedged one (kunchenguid#703, kunchenguid#694) is out of scope here.
Open
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Fix a live, reproducible watcher bug in bin/fm-watch.sh: a done-then-declared-paused crewmate (e.g. a green-CI PR sitting idle awaiting the captain's merge decision) refired a bare 'stale: ' wake on EVERY freshly-armed watcher, even though the .stale- hash suppressor was already correctly set to the current pane hash. Root cause: once a stale hash is already classified for a task whose status log's last line is 'paused:', the watcher re-derives the absorb verdict via crew_absorb_class/pause_state_class on every recheck. A stale/misattributed no-mistakes run-step (e.g. an aborted run whose terminal outcome predates the crew's LATER paused: declaration) can report a non-working, non-paused verdict from fm-crew-state.sh; the old code treated ANY such verdict the same as a fresh 'crew has stopped' first-sight event, firing the bare stale wake and also deleting the .paused- marker as a side effect - which guaranteed the identical misfire on every subsequent poll or watcher restart forever, since crew_absorb_class kept re-reading the same stale aborted-run record. Fix: at all three call sites in bin/fm-watch.sh that consume pause_state_class/crew_absorb_class for a crew whose current status-log line is 'paused:', only a definitive 'working' verdict now breaks the paused treatment; any other verdict (including this misattributed non-working/non-paused case) stays in the existing paused/long-recheck-cadence treatment (handle_paused_stale, anchored on PAUSE_RESURFACE_SECS / the status file's own mtime) instead of surfacing immediately. Genuine wedge detection for a plain (non-paused) stale pane is completely untouched - the escalation timer path (wedge_timer_check, STALE_ESCALATE_SECS) still works exactly as before. Deliberately scoped narrow per the task brief: this fixes only the restart-suppression defect in fm-watch.sh; it does NOT attempt the larger redesign discussed in upstream issues #703 ('Ship task awaiting captain merge is indistinguishable from a wedged crewmate') and #694 ('Watcher cannot hold a wait when the sole in-flight task is a correctly-paused pane') - that boundary is intentional and should be preserved in review. Verification: reproduced the exact live bug in an isolated fixture with a fake fm-crew-state.sh (bare stale refiring on every fresh watcher invocation despite the suppressor already matching the pane hash), added a colocated regression test (tests/fm-watch-triage.test.sh: test_nonterminal_stale_paused_survives_misattributed_run_step_across_restarts) that fails against the pre-fix code and passes post-fix, ran it and the full existing fm-watch-triage.test.sh suite (all green, including all pre-existing declared-pause/resurface/wedge tests, which still validate promptly-detected resumption and untouched wedge escalation), and cross-checked the adjacent watcher/daemon/wake-queue/crew-state/pi-extension suites (all green except one confirmed pre-existing flake in fm-watcher-lock.test.sh that reproduces identically against unmodified HEAD, unrelated to this change). bin/fm-lint.sh (shellcheck 0.11.0, the pinned CI version) is clean.
What Changed
Risk Assessment
✅ Low: Captain, the follow-up correctly throttles non-working paused rechecks while preserving the intended narrow stale-suppression behavior.
Testing
Completed 1 recorded test check.
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 1 issue found → auto-fixed ✅
bin/fm-watch.sh:973- For the persistentnoneverdict this now absorbs,pause_state_classremoves.paused-rechecked-*, so every subsequent poll re-runscrew_absorb_classinstead of using its paused recheck throttle. That can invoke the boundedno-mistakesstate read everyFM_POLL(up to 10 seconds each) for the exact misattributed-run condition. Retain/rearm the recheck marker for non-working paused verdicts while preserving the periodic working-state check.🔧 Fix: Throttle paused stale rechecks
✅ Re-checked - no issues remain.
⏭️ **Test** - skipped
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"🔧 Fix: Preserve declared-pause stale suppression
1 error still open:
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"🔧 Fix: Validate watcher stale suppression
1 error still open:
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"🔧 Fix: Confirm watcher tests pass
1 error still open:
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"🔧 Fix: Preserve declared-pause stale suppression
1 error still open:
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.