feat(bin): add NAS deploy sync and dispatch quality guardrails#671
Open
inivosdigital wants to merge 66 commits into
Open
feat(bin): add NAS deploy sync and dispatch quality guardrails#671inivosdigital wants to merge 66 commits into
inivosdigital wants to merge 66 commits into
Conversation
Two treehouse worktrees of the same project share a leaf directory basename, so Compose's default project-naming collides across them on container/volume/network names. fm-spawn.sh now derives a stable name from the worktree's own pool-slot path and drops it at .treehouse-compose-project (git-excluded, like the other turn-end pointer files), recorded in meta as compose_project=. fm-brief.sh tells ship/scout crewmates to consume it on every docker compose invocation and to avoid fixed host ports as a second collision vector.
…against backgrounded server call
…r-compose note to fm-brief.sh header comment
…compose isolation test
Claude crewmates and secondmates are separate claude processes launched by fm-spawn.sh; they never inherit the primary's .claude/settings.local.json, so they only compact at the hard context wall instead of proactively. Add CLAUDE_CODE_AUTO_COMPACT_WINDOW=200000 to the claude launch template. The var is capped at the model's real context window, so a fixed 200000 lands auto-compaction around 200k regardless of whether the spawned model's window is 200k or (as verified live for --model sonnet on this account, which carries the 1M-context beta) ~1M. The primary's own two-var form (WINDOW=1000000 + PCT_OVERRIDE=20) assumes a 1M window and is not safe to copy blindly for a smaller-window model/account combination. grok, codex, opencode, and pi launches are left untouched: grok's binary is not verified to honor Claude Code env vars, and the other three ignore them harmlessly but should not carry a var meant for a harness they aren't.
Bump CLAUDE_CODE_AUTO_COMPACT_WINDOW from 200000 to 300000 for claude crewmates and secondmates, matching the primary session's own updated target. The override is capped at the model's real context window, so it is a no-op below 300000 and only takes effect above it; today's standard --model sonnet crewmate spawn resolves to the ~1M-context beta, well above the new threshold. Updated the fm-spawn.sh header comment, launch_template() comment, and the harness-adapters SKILL.md fact to describe this conditional behavior instead of the "regardless of window" framing that only held at the old 200000 value.
content_in_default() only ever diffed a branch against the fetched refs/remotes/origin/$name, so a no-mistakes-mode task whose branch was merged into the LOCAL default branch by hand (e.g. after its upstream PR was closed unmerged) was never recognized as landed and teardown refused it. It now also checks the local refs/heads/$name ref.
…tests content_in_default()'s diff-based comparison (from local main's own prior rewrite) no longer depends on git merge-tree --write-tree, so the two tests that used to skip themselves on older git now run and pass here.
* Add quota-balanced dispatch selection * no-mistakes(document): Document dispatch selector guidance
Extends config/crew-dispatch.json's schema with model/effort/ultracode axes per the resource-tiering design: docs/examples/crew-dispatch.json gains rules for the firstmate-repo supervision-backbone carve-out, safety-critical/security/migration work (opus/xhigh, ultracode-flagged), and architectural/product tradeoffs (opus/xhigh). Adds the four hardened guardrails as mechanical checks rather than prose aspirations: - bin/fm-tier-guard.sh: firstmate-observable escalation triggers (diff size and elapsed time vs. the trivial tier's envelope, plus a general heavy-scale ceiling for any tier). - bin/fm-risk-tripwire.sh: a keyword/path trip-wire against a task's brief and diff, independent of the natural-language dispatch match. - bin/fm-ultracode-guard.sh: flag/reviewed/check subcommands that mechanically confirm a genuinely independent second pass ran before an ultracode-flagged task reaches PR-ready. - A documented model-verification gate in AGENTS.md section 4, extending the existing unverified-harness discipline to model names. Verification tiering (light/standard/heavy no-mistakes runs) is deliberately out of scope here and left for a follow-up change.
… distinct exit codes
…d distinct exit codes
… guardrail coverage
…herdr server-race
The read-tree portability rewrite restructured content_in_default() into an if/elif that checks exactly one candidate ref: the fetched origin ref when a remote exists, falling back to the local ref only when there is no remote at all. That silently dropped the case a no-mistakes-mode project with a closed, unmerged PR but work fast-forwarded into local main by hand - the fetched origin ref exists but does not contain the work, so the local ref is never even checked. Restore checking both candidates in sequence via a shared content_matches_ref helper, matching the pre-rewrite behavior tests/fm-teardown.test.sh's local-land regression test expects.
A merged PR does not, by itself, reach the live site: deployed apps run from a separate checkout under /mnt/nas/experiments/<name>/, managed by pm2, independent from the projects/<name> clone used for development. This gap required a manual pull+restart to notice and fix. bin/fm-nas-deploy-sync.sh fast-forwards a project's live NAS checkout and restarts its pm2 process(es), looked up from a captain-private data/nas-deployments.md mapping (best-effort, non-fatal, mirrors fm-fleet-sync.sh's dirty/diverged safety exactly). bin/fm-teardown.sh now calls it automatically after a landed ship task, including local-only merges.
added 29 commits
July 14, 2026 00:12
…late Post-swap origin is the captain's fork, so /updatefirstmate only converges the fleet onto that fork (a no-op on the primary source of truth) and never pulls the read-only upstream template. Folding upstream-template improvements into local main is the separate reviewed reconciliation task surfaced by the UPSTREAM_DRIFT bootstrap line, not a self-update. Note it inline in AGENTS.md section 12 and in the updatefirstmate skill.
The turn-end guard blocked essentially every turn during a legitimate watcher re-arm, not just a genuine supervision gap. fm_watcher_healthy requires a live, identity-matched watcher lock with zero grace, but a backgrounded re-arm needs a brief moment, roughly 0.25-0.3s measured, more under load, to fork the new watcher and register its lock after the old one released it. The Stop hook fired inside that gap on essentially every re-arm, so it blocked with no real supervision lapse. The pull-based warning banner tolerates this same gap because it checks only beacon freshness with a 300s grace. The turn-end guard's sharper live-lock check had no equivalent tolerance. Replace the single fm_watcher_healthy check with a bounded poll: retry every 0.2s, mirroring the watcher-arm script's own confirm loop, and block only once a deadline passes still unhealthy. The deadline defaults to FM_ARM_CONFIRM_TIMEOUT (10s), the same budget the arm gives itself to confirm, and is independently overridable via FM_TURNEND_ARM_WAIT. A watcher that is genuinely missing, dead, or identity-mismatched for the whole window still blocks; it now takes up to that deadline instead of blocking on the first check. This intentionally slows the truly-blind path (no arm in flight at all), which is the documented tradeoff. Update docs/turnend-guard.md's Shared Predicate section, the contract's one authoritative statement, to describe the bounded poll instead of an immediate block. Testing: tests/fm-turnend-guard.test.sh's test_hook_runs_fast now pins the bounded-poll latency instead of asserting a near-instant return, and a new test_hook_allows_watcher_that_registers_mid_poll confirms a lock that registers partway through the window is let through rather than blocked. Other block-path tests default to a short override so the suite stays fast; ran the lint gate and the full behavior suite (41 tests, all passing).
…ndoff tasks-axi gate
…unch signal-race fix
…ard.md with turn-end-guard poll fix
Crew session for this task crashed (tmux/captain core dump). This commit preserves in-flight work from the orphaned worktree so it is not lost. A tar backup also exists under /home/orangepi/fm-recovery-backups/.
fm_test_tmproot installed its EXIT trap and appended to FM_TEST_CLEANUP_DIRS from inside a command-substitution subshell, so neither ever reached the calling script: the trap fired (and deleted the just-created dir) the instant the subshell exited, and the array mutation never propagated back. Every run of the secondmate suites therefore leaked its temp root, growing worse under SIGKILL (e.g. the OOM killer) since no trap can run for that signal regardless. Move registration to an on-disk manifest, which survives the subshell boundary, and install the EXIT trap once at source time in the real top-level shell. Add a startup sweep for leftover <prefix>.<pid>.XXXXXX roots whose owning PID is no longer alive, as the actual safety net for the SIGKILL case, and call it from both secondmate suites before they claim their own root.
Claude draws its cleared composer prompt as the glyph "❯" followed by a non-breaking space (U+00A0), not an ASCII space. Under a glibc UTF-8 locale, [[:space:]] does not classify U+00A0 as whitespace, so the composer classifier's ASCII-only trimming left the U+00A0 behind as phantom content and read an otherwise-empty composer as pending. fm-send's submit-verify loop then exhausted its retries and returned the false "Enter swallowed" / exit 1 the caller saw against genuinely submitted steers to a claude target. Normalize U+00A0 and its narrow sibling U+202F to an ASCII space before classifying, so a composer empty but for no-break padding reads empty on every backend while real typed text separated by a no-break space still classifies pending. Add tests/fm-composer-nbsp.test.sh pinning the fix at four layers: the shared classifier, the tmux composer reader on the real captured bytes, the submit loop, and the full fm-send exit code.
fm-merge-local.sh pushed to origin unconditionally whenever a local-only project happened to have one, contradicting the documented local-only contract of no remote, no PR. Auto-push now applies only when PROJ resolves to firstmate's own repo (the deliberate fork-sync case); every other project needs an explicit --push-origin. Also bundles three small pre-existing fixes touched during this review: fm-teardown.sh no longer runs the NAS deploy sync hook on a forced teardown, fm-ultracode-guard.sh validates the role argument before writing it into the line-based .ultracode marker file, and fm-bootstrap.sh's upstream fetch now falls back to gtimeout or a background-kill bound when timeout is unavailable, instead of fetching unbounded.
…kip and added missing FM_UPSTREAM_FETCH_TIMEOUT var
Add a config/autodeploy-logs sweep to fm-watch.sh's heartbeat fleet-scan. Each configured status log's last line is read on every heartbeat, and a failure (an ALERT rollup or a STUCK:/FAILED: line in fleet-sync's convention) surfaces once as a check wake carrying that line until it clears; a later healthy run re-arms the alarm. Mirrors bootstrap's critical_services_check but fires periodically, so a deploy failure between sessions is caught instead of waiting for the next session start. Absent config, a healthy "ok" last line, or a missing/unreadable log (a NAS hiccup) are all silent and non-fatal, so the check stays cheap enough to run every heartbeat without spurious alarms. Config is gitignored and absent by default, so it is a no-op for fleets that do not opt in.
Factor the failure predicate into the shared bin/fm-autodeploy-lib.sh (fm_autodeploy_line_failed), sourced by both the fm-watch.sh sweep and a new fm-bootstrap.sh autodeploy_logs_check, so an idle fleet catches a deploy failure at the next session start exactly as critical-services already does. One owner for the match set; docs and colocated tests updated to match.
On a hung NAS mount the tail read of an autodeploy status log could block the watcher heartbeat (and the bootstrap check). Wrap the read in the shared fm-autodeploy-lib.sh behind a bounded timeout and treat a timeout as unreadable, preserving the existing fail-quiet behavior for missing or unreadable logs. Both read sites go through the one shared helper.
…b and sweep When config/autodeploy-logs names a log but no read-timeout mechanism (timeout, gtimeout, or perl) is on PATH, fm_autodeploy_read_last_line fails every read and the autodeploy-alert feature silently never fires. Surface a one-time AUTODEPLOY_INERT bootstrap diagnostic in that case so the captain learns alerting is disabled rather than assuming a healthy deploy; the transient missing or unreadable log path stays fail-quiet. Document the FM_AUTODEPLOY_LOG_READ_TIMEOUT knob with the other timeout overrides and describe the heartbeat autodeploy sweep in the event-driven supervision architecture.
Independent review found the detected failure format attributed to firstmate's fleet-sync convention (bin/fm-fleet-sync.sh) in five places, including the primary ok/ALERT rollup example. fm-fleet-sync.sh emits the STUCK:/needs-attention alert line but never an ok or ALERT rollup; that rollup is an external per-project autodeploy tool's own convention (e.g. beamanalyzer's deploy job). Rephrase all five sites to name the external tool as the rollup's source, keep firstmate's own STUCK:/FAILED:/ needs-attention format, and point at the one-owner match set (bin/fm-autodeploy-lib.sh's fm_autodeploy_line_failed) instead of implying bin/fm-fleet-sync.sh produces the format.
…ard-timeout env vars
…pstream-q8 # Conflicts: # .agents/skills/bootstrap-diagnostics/SKILL.md # AGENTS.md # bin/fm-bootstrap.sh # bin/fm-teardown.sh # bin/fm-watch.sh # docs/scripts.md
…keys exist A scout task with no captain decision holds had no way to pass command_verify() without a tasks-axi build exposing the captain-hold contract, unlike every other tasks-axi touchpoint in this repo, which falls back to manual backlog handling when tasks-axi is incompatible. Move the decisions_reviewed/decision_keys read ahead of require_tasks_axi and only call it once decision_keys is non-empty.
fm_lock_is_provably_stale's [ -e ], lsof, and stat probes ran without this script's own bounded-timeout wrapper, unlike every other filesystem/git touch of the NAS checkout. A degraded NAS mount - the exact failure mode this script exists to tolerate - could hang the probe indefinitely, blocking fm-teardown.sh's caller instead of exiting 0 as documented. timeout/gtimeout only bound a real subprocess, not an in-process shell function, so bounded_is_provably_stale execs the check (and its fm-lock-lib.sh dependency chain, exported for the child shell) in a fresh bash -c under the existing bounded() wrapper.
…against a hung NAS
…h as lock-lib consumer
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Fix an unbounded-hang bug in bin/fm-nas-deploy-sync.sh found by review gate ship-firstmate-fmsend-verify on 2026-07-15: fm_lock_is_provably_stale's probe/removal path for the packed-refs.lock (the [ -e ] existence check, the lsof live-holder check, and the stat mtime check) ran without the script's own bounded-timeout wrapper (the 'bounded' helper using timeout/gtimeout, FM_NAS_SYNC_TIMEOUT) that every other filesystem/git touch of the NAS deploy checkout already uses. On a degraded/unreachable NAS mount - the exact failure mode this script exists to be resilient against - those calls could block indefinitely, hanging fm-teardown.sh's post-teardown call instead of honoring the script's documented best-effort, always-exit-0, bounded-timeout contract. Fix: added a bounded_is_provably_stale() wrapper in bin/fm-nas-deploy-sync.sh that execs fm_lock_is_provably_stale (from bin/fm-lock-lib.sh, left untouched since it is a shared lib also used unbounded by fm-teardown.sh and fm-fleet-sync.sh for non-NAS checkouts where this hang risk does not apply) inside a fresh bash -c under the existing bounded() timeout helper, since timeout/gtimeout can only bound a real subprocess, not an in-process shell function - the function chain (fm_lock_is_provably_stale, fm_lock_has_live_holder, fm_lock_lsof_holder, fm_lock_age, fm_lock_path_mtime, fm_lock_log) is exported via export -f for the child bash to see. Also bounded the adjacent [ -e ] existence check and the rm -f lock removal in the same code path for full removal-path coverage, reusing the same bounded() helper rather than inventing a new mechanism. Extended tests/fm-nas-deploy-sync.test.sh with test_stale_lock_probe_lsof_hang_is_bounded: fakebin shims make every git fetch fail with the packed-refs.lock signature (forcing the retries-exhausted stale-probe path) and make lsof hang for 20s; verified this reproduces a 21s+ hang against the pre-fix code and completes in ~1s (bounded by FM_NAS_SYNC_TIMEOUT) with the fix. Both changed files pass shellcheck cleanly (one intentional SC2016 suppressed with a disable comment, since the bash -c script is deliberately single-quoted so the timeout'd child shell expands its own positional args, not the parent's). Note: a prior run of this same fix hit an unrelated pipeline infra failure (auto-fix-tests agent crashed with fork/exec argument list too long) after the rebase gate was approved to bundle 62 unrelated unpushed local-main commits (71 files) - now confirmed reproducible across sibling tasks when that bundling happens. This run intentionally SKIPS the rebase-bundling gate to keep the diff small and avoid re-triggering that crash.
What Changed
bin/fm-nas-deploy-sync.shto auto-sync live NAS deployments after a landed ship task, then hardened it end-to-end: the packed-refs lock path resolution, the stale-lock existence/lsof/mtime probe, and the lock removal all now run inside the script'sbounded()timeout wrapper (via abounded_is_provably_stale()shim around the sharedfm_lock_is_provably_stale), so a degraded or unreachable NAS mount can no longer hangfm-teardown.shindefinitely.bin/fm-tier-guard.sh, abin/fm-risk-tripwire.shsafety floor, and anbin/fm-ultracode-guard.shindependent-review gate, along with.agents/skills/verify-trivialfor the trivial-tier verification path.Risk Assessment
✅ Low: The round-1 finding (unbounded cd/pwd -P in packed_refs_lock_path against a hung NAS mount) is correctly fixed by wrapping the resolution in bounded bash -c, mirroring the existing bounded_is_provably_stale pattern exactly (same timeout mechanism, same positional-arg-passing to avoid injection, same shellcheck disable rationale); a new regression test (test_packed_refs_lock_path_resolution_hang_is_bounded) exercises the exact hang scenario via an exported cd() shim scoped to a subshell and verifies both the 1s bound and the resulting skip message. No other unbounded filesystem/git touch of $NAS_PATH remains in the file, and the change is minimal (4 lines in bin/, a self-contained new test).
Testing
Ran the targeted regression suite (tests/fm-nas-deploy-sync.test.sh, 11/11 passing) against the fix, and additionally reproduced the pre-fix bug in an isolated /tmp copy of the repo by substituting each historical (pre-fix) script revision — confirming the new tests genuinely hang for 21s and 40s respectively without each of the two fix commits, and complete in ~1s with both applied, which directly demonstrates the unbounded-hang bug was real and is now bounded as intended.
Evidence: fm-nas-deploy-sync.test.sh full run at target commit (all pass)
Evidence: Manual pre-fix vs post-fix reproduction summary
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
⏭️ **Rebase** - skipped
.agents/skills/bootstrap-diagnostics/SKILL.md- branch carries 62 commit(s) that exist on your local main branch but were never pushed to origin/main; rebasing would bundle this unrelated work (75 file(s)) into the PR:Push main to origin, or rebase your branch onto origin/main, before gating.
🔧 **Review** - 1 issue found → auto-fixed ✅
bin/fm-nas-deploy-sync.sh:132- The commit's stated goal is 'full removal-path coverage' for the stale-lock probe/removal sequence, and it correctly boundstest -e,fm_lock_is_provably_stale, andrm -f. Butpacked_refs_lock_path()(called at lines 149 and 164, immediately upstream of those newly-bounded calls, to compute the very$lockvalue they operate on) still doesabs=$(cd "$NAS_PATH" && pwd -P)unbounded.git -C "$NAS_PATH" rev-parse --git-path packed-refs.locknormally returns a path relative to$NAS_PATH(e.g..git/packed-refs.lock) for a plain deployment clone with no external git-dir, so thecase ... /*) ... *) cd/pwd -Pfallback branch is the common path taken on essentially every call, not a rare corner case. On the exact degraded/unreachable NAS mount this fix targets,cd "$NAS_PATH"orpwd -Pcan itself block indefinitely (a chdir/getcwd syscall against a hung mount), hanging the caller before the script ever reaches the newly-bounded calls - reproducing the same unbounded-hang bug this commit fixes, just one function call earlier. Wrap this resolution the same waybounded_is_provably_stalewrapsfm_lock_is_provably_stale(e.g.abs=$(bounded bash -c 'cd "$1" && pwd -P' bash "$NAS_PATH") || return 1).🔧 Fix: bound packed-refs lock path resolution's cd/pwd against a hung NAS
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"bash tests/fm-nas-deploy-sync.test.shat target commit 9f1a3f7 — all 11 tests pass in ~8.4s, including the two new hang-probe regression testsFull configured baseline suite (for t in tests/*.test.sh; do bash "$t"; done) already ran green per task setupManual repro: swapped bin/fm-nas-deploy-sync.sh with the pre-fix revision (git show eaef5c1:...) in an isolated /tmp copy and re-rantest_stale_lock_probe_lsof_hang_is_bounded— failed with a genuine 21s hang, proving the lsof/stat probe really was unbounded before the fixManual repro: swapped in the version with only the first fix commit applied (git show 4164f9b:...) and re-rantest_packed_refs_lock_path_resolution_hang_is_bounded— failed with a genuine 40s hang, proving the second commit's cd/pwd fix was independently necessaryConfirmed both hang-probe tests pass in ~1s each against the full fix, and confirmed the actual worktree file was never modified (git status clean throughout)✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.