Skip to content

fix(omp): recover wedged primary wake delivery - #157

Merged
dnth merged 7 commits into
mainfrom
fm/fm-omp-wake-consume-fix
Sep 21, 2026
Merged

dnth merged 7 commits into
mainfrom
fm/fm-omp-wake-consume-fix

Conversation

@dnth

@dnth dnth commented Sep 21, 2026

Copy link
Copy Markdown
Owner

Intent

Fix the stuck OMP primary wake delivery in the firstmate repo: an idle-main injection sets mainFallbackWakeInFlight and never clears it, suppressing every later main-bound wake. Diagnosis lives at data/fm-inbox-delay-fable-investigate/report.md in the live home.

Requirements:

  • Fix consumeWake/episode bookkeeping in bin/fm-primary-watch-core.ts and .omp/extensions/fm-primary-omp.ts: treat the first turn_start after an accepted idle-main send as consumption (OMP emits turn_start and the extension already listens); acknowledge message_start for role custom with customType firstmate-watcher-wake and matching text.
  • Bound the in-flight marker: at turnEnd, if mainFallbackWakeInFlight is still set but a drain ran since the send (or the main-owned row count is zero), clear it instead of returning early; let the rows-outlived-every-close successor fire when in-flight outlives one full turn boundary.
  • Persist episode state (mainFallbackEpisode, in-flight token, last send/consume timestamps, turn-boundary bookkeeping) under state/extensions/omp-primary-watch/ so the next incident is diagnosable from disk and fm-guard.sh can warn when an in-flight wake is older than a turn.
  • Fix tests/fm-omp-primary.test.sh:1284 to model idle-main sends as emitting no before_agent_start; add a regression that sends one idle-main wake, ends a turn, then asserts the next check close is injected; cover the stale in-flight bound; cover persisted episode state and the guard warning via executable behavior or guard-output proof.
  • Align docs/omp-supervision-branch.md:67 with the real runtime contract (idle main does not consume at before_agent_start).
  • Scope is confined to bin/fm-primary-watch-core.ts, .omp/extensions/fm-primary-omp.ts, tests/fm-omp-primary.test.sh, docs/omp-supervision-branch.md, plus the minimal bin/fm-guard.sh touch for the persisted-state warning. Nothing else may change.
  • Acceptance criteria: AC1 idle-main wake clears in-flight on turn_start or custom-role ack with a regression test that fails pre-fix; AC2 stale in-flight cannot wedge delivery forever with test coverage of the bound; AC3 persisted episode state plus fm-guard.sh stale warning with proof; AC4 docs contract and mock corrected with changes confined to the listed files.

Firstmate-Validation-Generation: 7f9be95550dc0171245c9acf05c907a8

What Changed

  • Fixed OMP primary wake consumption and turn-boundary bookkeeping so idle-main wakes no longer wedge later main-bound delivery.
  • Persisted primary wake episode and in-flight state under state/extensions/omp-primary-watch/, with stale-marker warnings in fm-guard.sh.
  • Updated supervision documentation and regression coverage for idle-main sends, stale in-flight bounds, persistence, and guard behavior.

Risk Assessment

✅ Low: The changed wake-consumption, stale-marker bounding, episode persistence, guard warning, mock behavior, and documentation are internally consistent with the stated acceptance criteria; no concrete reachable defect was substantiated.

Testing

The prior payload reports focused OMP tests using a simulated OMP API and explicitly marks them non-live; therefore all scenarios remain untested under the live-validation contract.

  • Live validation: ⚠️ inconclusive - 0 of 4 scenarios driven live against the product
Scenario Result Live Evidence
An idle-main wake is consumed by turn_start/custom-role acknowledgement and later wakes are delivered ⏸️ untested no The prior payload cites a simulated OMP integration test but explicitly records live=false, so it does not establish a result against the live product.
A stale in-flight marker cannot suppress delivery forever across turn boundaries ⏸️ untested no The prior payload cites a simulated OMP integration test but explicitly records live=false, so it does not establish a result against the live product.
Persisted episode state records send/consume/boundary data and fm-guard warns only for stale in-flight wakes ⏸️ untested no The prior payload cites a simulated fm-guard test but explicitly records live=false, so it does not establish a result against the live product.
OMP primary lifecycle and regression suite remains green after the wake bookkeeping changes ⏸️ untested no The prior payload records only a non-live targeted test execution, so it does not establish a live product result.
Evidence: OMP primary targeted test log

Targeted OMP integration test completed successfully; final checks include idle-main consumption, stale in-flight bounding, persisted episode state, and fm-guard warning behavior.

ok - OMP path resolution stays canonical when readlink -f is unavailable
ok - OMP primary identity requires launch-bound Bun and OMP realpaths plus the exact argv boundary
ok - standalone OMP identity requires the launch-bound PID executable
ok - exact-OMP ancestry stops at the innermost foreign harness ancestor
ok - OMP fresh primary lifecycle creates canonical state and atomically replaces a marker symlink without following it
ok - OMP native identity supports physical Bun scripts and known compiled virtual entrypoints
ok - native OMP alone admits a fresh plain checkout and delivers one startup instruction
ok - OMP primary refuses whitespace-bearing identity before marker publication
ok - OMP primary extension binds secondmate doorbells after session readiness
ok - OMP turn_start re-arms a missing watcher and keeps the guard silent through a long turn
ok - OMP confirms the recovery handling handshake after delivering its hidden next-turn wake
ok - OMP surfaces a refused handling handshake as one typed wake
ok - OMP /new /resume /fork and reload session_switch paths auto-arm and carry an in-flight actionable close
ok - OMP unacknowledged wake delivery keeps the successor chain and delivers once per close
ok - OMP re-notifies durable wakes once on session start and switch
ok - OMP suppresses session notifications when the durable queue is empty
ok - OMP core handoff suppresses duplicate durable queue notification
ok - OMP coalesces fallback wakes into one in-flight notification per handling episode
ok - OMP idle-main wakes consume at turn_start and a stale in-flight marker is bounded
ok - OMP ignores already-delivered handoffs when notifying queued wakes
ok - fm-guard warns only when an OMP in-flight wake outlives a turn boundary
- Outcome: ⚠️ 1 warning across 1 run (4m51s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed ✅
  • ⚠️ bin/fm-primary-watch-core.ts:742 - sendWake persists the old generation's episode after awaiting sendFollowUp without checking whether that generation was replaced (lines 731-742). If a session switch occurs while the send is pending, the old owner can write in_flight=1 after shutdown, and the new session has no corresponding clear; fm-guard.sh then emits a stale suppression warning until another turn happens. The rejection path also clears the in-memory marker without persisting the cleared state. Guard persistence by generation liveness and write a terminal/cleared episode state during replacement or send failure.

🔧 Fix applied.
✅ Re-checked - no issues remain.

⚠️ **Test** - 1 warning
  • ⚠️ live validation verdict: inconclusive (0 of 4 scenarios were driven live against the product); untested: An idle-main wake is consumed by turn_start/custom-role acknowledgement and later wakes are delivered, A stale in-flight marker cannot suppress delivery forever across turn boundaries, Persisted episode state records send/consume/boundary data and fm-guard warns only for stale in-flight wakes, OMP primary lifecycle and regression suite remains green after the wake bookkeeping changes
  • Live validation: ⚠️ inconclusive - 0 of 4 scenarios driven live against the product
Scenario Result Live Evidence
An idle-main wake is consumed by turn_start/custom-role acknowledgement and later wakes are delivered ⏸️ untested no The prior payload cites a simulated OMP integration test but explicitly records live=false, so it does not establish a result against the live product.
A stale in-flight marker cannot suppress delivery forever across turn boundaries ⏸️ untested no The prior payload cites a simulated OMP integration test but explicitly records live=false, so it does not establish a result against the live product.
Persisted episode state records send/consume/boundary data and fm-guard warns only for stale in-flight wakes ⏸️ untested no The prior payload cites a simulated fm-guard test but explicitly records live=false, so it does not establish a result against the live product.
OMP primary lifecycle and regression suite remains green after the wake bookkeeping changes ⏸️ untested no The prior payload records only a non-live targeted test execution, so it does not establish a live product result.
  • bash tests/fm-omp-primary.test.sh
  • git diff --check a4fa70111b2f20ebc1fd20e97f4dda9078a94dad..a82e39379dce75967ad1e9d0cd2cf4099397eb9f
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

An idle-main wake injection drives agent.prompt() directly, which emits no
before_agent_start and arrives as a custom-role message, so the existing
consumeWake evidence (before_agent_start or a user message_start matching the
wake text) never fired. mainFallbackWakeInFlight stayed set forever and every
later main-bound wake was suppressed behind it.

Treat the first turn_start after an accepted send as consumption, acknowledge
the wake's own custom-role message_start (customType firstmate-watcher-wake),
and bound the marker at turn_end: clear it when every send-time queue row has
been drained or it has outlived a full turn boundary, so the rows-outlived
successor still fires. The episode mirror persisted at
state/extensions/omp-primary-watch/main-fallback-episode.state records the
marker plus send/consume/boundary timestamps, and fm-guard.sh warns when an
in-flight wake is older than a turn.
@dnth dnth changed the title fix(omp): bound main-fallback in-flight wake marker at turn boundaries fix(omp): recover wedged primary wake delivery Sep 21, 2026
…eplacing the new quoted heredoc fixture state with portable printf output. Verified with bash -n and the full tests/fm-omp-primary.test.sh suite
…-omp-primary.test.sh by replacing non-portable in-place sed edits with a portable awk-and-mv field updater. Verified Bash parsing, the full OMP primary test suite, and git diff checks successfully
…ving the problematic single-quoted awk program in tests/fm-omp-primary.test.sh and using an equivalent portable double-quoted program. Verified the full shell parse sweep with /bin/bash -n and git diff --check
…mments that desynchronized stock Bash 3.2 parsing. Verified with `bash -n`, `git diff --check`, and the full `tests/fm-omp-primary.test.sh` suite
@dnth
dnth merged commit b5c17b1 into main Sep 21, 2026
15 checks passed
@dnth
dnth deleted the fm/fm-omp-wake-consume-fix branch September 21, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant