Workers wedge with delivered-no-turn despite truthful receipts; need custody-aware auto-recovery

[dnth]

Problem

Long-lived OMP worker sessions can stop starting turns while transport still acknowledges durable delivery. PR #153 fixed the false-receipt side: fallback now requires bounded turn proof and unproven rings remain queued, so wedges are correctly reported as omp-native-queued / delivered-no-turn. Nothing yet performs a custody-safe automatic recovery.

This issue is the narrow deterministic recovery adapter for that specific incident class. The broader task-completion control loop is tracked in #156.

Evidence (2026-09-20)

Three workers wedged the same way in one day and recovered only after a manual same-worktree relaunch through fm-control with a progress note:

• ic-prod-outage-astra-fix — /fresh returned delivered-no-turn; the four-hour session remained alive but swallowed input.
• fm-omp-compact-adviser-adapter — three durable inbox decisions remained unhandled while the pane idled.
• ic-ingest-auto-readiness — QA-blocking decisions remained unhandled for about 30 minutes; releasing its supervision lease also required watcher-cycle repair through fm_watch_arm_omp.

All three manual recoveries preserved their worktrees and existing work. The Astra root-cause scout at data/fm-stuck-crew-astra-debug/report.md attributed the initiating runtime trigger upstream of FirstMate. These incidents establish usefulness, not a complete automatic-safety proof or a calibrated retry threshold.

Scope

Define the adapter input as a generation-bound, persistent instruction-delivery incident and its output as a structured recovery disposition plus evidence.

In scope:

• persistent supported delivered-no-turn / pending durable instruction incidents;
• existing bounded re-ring and escalation ownership;
• deterministic custody/preservation checks;
• same-task, same-worktree relaunch through the existing actuator;
• durable incident, attempt and result records;
• post-relaunch target-instruction and downstream-progress verification;
• task-lifetime retry cap and loud escalation.

Out of scope:

• a second task lifecycle or delivery orchestrator;
• semantic task-completion reconciliation (tracked in Recovery Reconciler: reconcile accepted tasks to verified terminal outcomes #156);
• Jev on the deterministic fast path;
• generic provider-loop diagnosis, supervisor-health repair or secondmate recovery;
• replaying a typed command when no matching durable instruction exists;
• merge, deployment or destructive cleanup authority.

Corrected safety contract

The earlier statement “any active run or unlanded work blocks auto-action” is too broad and unsafe in both directions.

Unlanded or dirty work does not itself block same-worktree recovery. It is precisely what recovery must preserve. However, preserved work is not permission to terminate a live writer.

An automatic replacement requires all of the following:

1. Authority — current FirstMate mutation authority for the exact home/task and a supported ordinary-task recovery capability.
2. Preservation binding — canonical existing worktree and project/lease binding; branch, HEAD, pending instructions, staged/untracked content and commits remain accounted for. No fresh allocation, reset, stash, cleanup or implicit migration.
3. Fresh identity — task generation and process/start identity still match the incident. Acquire existing control/spawn exclusion in documented lock order.
4. Atomic custody — a positive authoritative validation/branch-custody verdict, with query failures and freshness retained, plus a reservation preventing new custody across the effect boundary. Unknown, conflicting, failed-query or pipeline-owned cases are ineligible.
5. Owner transition — for a dead owner, recovery-grade death/absence proof. For a live-but-wedged owner, a separately supported quiesce-and-stop transition with fresh turn/pending checks. Prove old writers stopped before the successor receives write authority.
6. Journal then mutate — recheck identity/custody under reservation, preserve all bytes and commits, durably reserve the attempt, then execute the existing relaunch transaction.
7. Reconcile partial effects — after crash or ambiguous failure, reconcile the journal and current process/generation before another attempt. Release reservation only after durable successor binding or an explicit preserved failure state.

A preceding read-only state query is not sufficient mutation authority. A finite run search or a helper that suppresses query failures cannot prove global owner absence.

Candidate and action lifecycle

1. Existing inbox/watcher owner detects a persistent supported incident after its normal grace/re-ring ladder.
2. Publish exactly one durable candidate identity; duplicate/concurrent detectors reconcile the same candidate.
3. FirstMate rereads current task, instruction, endpoint, completion and custody evidence.
4. If applicable completion already passes, route normal delivery—do not relaunch.
5. If safe deterministic predicate and task-wide budget pass, reserve custody and attempt identity.
6. Recheck for a late turn, handled instruction, new custody, generation change or completion at the effect boundary.
7. Journal intent and relaunch through existing fm-control/fm-spawn ownership.
8. Bind the successor generation and verify target-message consumption plus expected downstream progress.
9. Record recovered, retry eligible, or escalated with work preserved.

A process launch or alive endpoint is not recovery.

Progress and retry policy

Use at most two automatic relaunches per task lifetime, spanning generations and incident identities. This is a conservative starting policy, not an empirically optimized threshold.

Reserve an attempt durably before the first lifecycle effect. A positively proven pre-effect refusal may release it; a crash with uncertain effects does not.

Strong recovery evidence:

• the target durable instruction is handled by the successor and its expected downstream effect becomes visible; or
• the applicable completion gate passes.

Useful activity that may cancel/settle an incident but does not prove task success:

• a request-correlated turn starts;
• attributed validation advances;
• a new criterion becomes successfully evidenced;
• exact PR/check state advances for the current head.

Do not reset or replenish retry authority for a new PID/generation, turn counter, enqueue, duplicate receipt, status/log timestamp, pane animation, dirty-flag toggle, arbitrary HEAD change, commit restamping, repeated validation loop, or Jev confidence.

After the second unsuccessful attempt, preserve all work and escalate with the failed instruction, custody evidence, attempts and observed consequences. Do not call the underlying implementation irrecoverable merely because automatic recovery is exhausted.

Jev boundary

No Jev call is required for this adapter's supported deterministic predicate. Jev may later advise #156 about semantic ambiguity, but it cannot establish custody, grant completion, authorize restart, replenish retries or override merge authority.

Known actuator gaps to resolve

The 2026-09-20 Astra medium review identified two source-level gaps that must be verified and repaired before automatic recovery:

1. Relaunch transaction publication identity — fm-control.sh passes FM_CONTROL_RELAUNCH_TX and expects control_relaunch_tx to identify an ambiguously published successor, but the inspected fm-spawn.sh metadata writer did not publish that field. Add the handoff identity and a regression for failure/crash after successor publication.
2. Task evidence versus process metadata — successor metadata replacement changes process identity but can discard task-scoped validation, implementation-completion and PR bindings. Define which task evidence survives, which process fields rotate, and which stale bindings require authoritative revalidation.

Extend the existing transaction rather than wrapping it in a check-then-relaunch script.

Acceptance criteria

Trigger and idempotency

• A persistent pending native instruction reaches exactly one generation-bound candidate.
• Transient queueing and positively busy/productive workers do not become candidates.
• Typed no-turn without a matching durable instruction does not invent or replay instructions.
• Duplicate notifications, concurrent reconciliation and old-generation events cause at most one lifecycle effect.
• Stable incident, snapshot and attempt identities are separate; timestamps/log churn do not create new incidents.

Effect-boundary safety

• A late turn or handled acknowledgement before stop cancels replacement.
• Custody acquired between observation and effect prevents stop.
• Query timeout, malformed output, incomplete coverage or unknown ownership is not treated as no owner.
• Missing worktree, foreign lease, unsupported backend or unresolved endpoint identity preserves state and escalates.
• Live-owner recovery uses an explicit quiesce-and-stop contract; old writers are proven stopped before successor write authority.

Preservation and crash recovery

• Eligible dirty tracked, staged and untracked content plus unpushed commits survive byte-for-byte in the same worktree.
• No reset, stash, cleanup, fresh worktree allocation or implicit migration occurs.
• Still-valid task-scoped validation, implementation and PR evidence survives process replacement; stale evidence is explicitly reconciled.
• Crashes after reservation, stop, launch and successor publication reconcile the same journal transaction without duplicate execution or budget evasion.
• Spawn failure after metadata publication is attributable to the exact relaunch transaction.

Outcome and authority

• An alive successor without target-message/downstream progress remains unrecovered.
• Two unsuccessful relaunches exhaust the task-lifetime allowance despite generation, HEAD, turns or receipt changes.
• Applicable completion routes normal delivery without restart.
• PR-ready work does not merge without explicit authority, and unrelated/new-head CI does not satisfy stale evidence.
• Parked or active pipeline custody remains intact unless a future explicit run-preserving handoff contract exists.
• Exhaustion escalates loudly with work preserved and complete evidence.

Likely implementation areas

• bin/fm-watch.sh and bin/fm-task-inbox-lib.sh — integrate candidates with the existing pending/re-ring owner.
• Existing durable wake reconciliation — FirstMate-owned decision and audit path.
• bin/fm-control.sh, bin/fm-spawn.sh and shared custody/lease owner — in-transaction authorization, generation exclusion and handoff.
• bin/fm-nm-run-lib.sh or the authoritative validation owner — checked positive custody observations without turning permissive display queries into mutation authority.
• Existing behavioral tests including relaunch, OMP guard, send/turn-start, watcher recovery, task inbox, crew state and receipt suites.
• .agents/skills/stuck-crewmate-recovery/SKILL.md and supervision protocol/configuration docs — one canonical supported path, budget and exclusions.

Do not replace existing completion or PR logic. In particular, pipeline-accounted head advancement/restamping may be valid; a new literal HEAD-equality rule would be incorrect.

Relationship to other work

• Recovery Reconciler: reconcile accepted tasks to verified terminal outcomes #156 tracks the broader Recovery Reconciler and optional later semantic/Jev layer.
• PR fix(omp): prevent false-success fallback receipts #153 provides truthful no-turn evidence on which this adapter depends.
• feat: return inbox results to the originating Hermes session #152 / PR feat: return durable inbox results to Hermes sessions #154 concern trusted-local Hermes result round trips, not recovery authorization.
• Supervisor-health and renderer defects remain separate fault domains and must not spend a worker's retry budget.

Review provenance

A read-only Astra medium design review was completed 2026-09-20 against a4fa70111b2f20ebc1fd20e97f4dda9078a94dad.

Verdict: retain #155 as the deterministic first adapter, but revise its safety contract before implementation. The review did not reproduce a new wedge, run validation, modify source, or authorize implementation.