[roodboi]

Refs HACK-1281

Summary

native-config-networks fails when a healthy storage-bearing hack exec (or up/restart) is refused by the storage carrier after its Docker effect. The refusal leaves carrier intent uncertain. By design, down --recover then stays incomplete, so the fixture reports "Cleanup also failed".

The carrier used two fixed bounds that do not survive a host stall of the CLI process:

• a 1-second process identity read (ps);
• a 3-second post-exit settlement budget.

A stalled process resumes with its timer already due, so the command times out even though the child finished in time. This PR keeps those bounds finite but stall-tolerant. It also fixes the diagnostics gaps the verifier found.

Root cause and evidence

• Mechanism, replayed on hosted CI with real Docker. A temporary commit (fbf5cf6, reverted in 055ecce) stopped the CLI process for 1.2 s of every 2 s during 20 saved exec commands on shards 1 and 2.
  • On the pre-fix carrier (run 39643040488), both stalled shards failed with the original signature. Shard 1 (job 119371288508) failed on the first stalled exec: hack:exec exit 1, E_COMPOSE_FAILED, stage storage-helper-cleanup, reason probe-timeout. Shard 2 (job 119371288299) failed on the 13th: stage storage-helper-create, reason probe-timeout. In both, saved state showed storageWitnessesPending, cleanup down --recover exited 1 with stop-incomplete, and the scenario printed "Cleanup also failed" and "Fixture command must exit 0".
  • On this fix (fc4095b, run 39644645624), both stalled shards passed all 20 stalled execs and the whole scenario. All four shards passed.
• The same replay in a local Linux container. Running the real recorded transport 600 times under the same 1.2 s / 2 s stalls gave 5 probe-timeout refusals before the fix and 0 after. With 3.5 s stalls over 300 runs, there were 2 refusals before (one probe-timeout, one probe-budget) and 0 after.
• The historical failures match this signature. I found eight natural native-config-networks failures on shard 1 since 2026-10-12, not four. They are 7 natural failures plus the load-sampled run, across 331 completed runs (about 2%).
  • Each one is "Fixture command must exit 0" followed by "Cleanup also failed", which is the replayed signature.
  • They happen at varied points in the run (88 s to 33 min into phase 2, and inside the ingress phase), which fits a per-command event rather than one phase.
  • The rate rises with CI concurrency: about 1.0% in quiet hours (fewer than 8 runs per hour), 1.7% in middle hours and 4.5% in busy hours (20 or more runs per hour). That fits host contention.
• What is not proven. The original failing commands cannot be recovered, because those runs uploaded no captures. A natural stall monitor ran on all six sampled shards (about 4 runner-hours and 7,700 product-shaped ps reads). It saw no stall of 250 ms or more in a read, and an event-loop lag of at most 250 ms. Those runs were in a quiet CI period. Stall-induced refusal is therefore the mechanism that reproduces the signature on CI and that this PR removes. It is not a direct observation of the historical cause. If a failure with this signature recurs on this head, the diagnostics below will name the step, stage and reason.

What changed

• Carrier fix (src/lib/native-compose-storage-witness-docker.ts). Identity reads now share the remaining handshake or settlement budget instead of a fixed 1 s bound. The post-exit settlement budget is a finite 15 s (SETTLEMENT_BUDGET_MS). No check, record, retention, recovery rule or authority changes. An exhausted budget still refuses as probe-budget. bounded() now observes abandoned work, so a rejection after the budget has run out cannot escape as an unhandled process error. The over-budget test exposed this bug.
• Docs (docs/reference/native-compose-storage-xattr-carrier.md). Documents the new budgets and why they must outlast a multi-second host stall.
• Diagnostics (tests/e2e/native-config-networks-acceptance.ts). These close the gaps from the review:
  • An incomplete saved stop issues no refusal boundary, so its stage and reason stay null. The command diagnostic now adds a closed message label for exact fixed product sentences: stop-incomplete, operation-failed, and native-state, native-busy, native-uncertain and native-stale. Any other text, or a changed sentence, gives null.
  • A command the runner killed (timeout) or could not return now logs its class, role, sequence and duration, plus a closed runner outcome. Before this, it logged nothing.
  • No output, argv or path is printed.
• Earlier commits on this branch are kept: the closed refusal boundary in E_COMPOSE_FAILED detail and the human summary line, and the classified probe-timeout, probe-cancel and probe-budget carrier causes.

Validation

• New regression tests (tests/native-compose-storage-command-record.test.ts), each proven by a mutation:
  • A child identity read stalled for 1.2 s still settles. Fails with the 1 s bound restored.
  • A 3.5 s post-exit stall still settles. Fails with the 3 s budget restored.
  • A settlement past the budget still refuses as probe-budget and stays unsettled, using a monotonic clock jump with no real wait. Fails with the budget raised to 60 s, and fails without the abandoned-work catch.
• Fixture tests (tests/native-networks-fixture.test.ts), each proven by a mutation:
  • An incomplete stop is named stop-incomplete, from both JSON and human output, and changed text gives null. Fails with no fixed-message classification, and fails with no human-line classification.
  • A real runner timeout is labelled runner: "timeout". Fails with the outcome always set to other.
  • The wiring of that timeout diagnostic inside the scenario's execute is only exercised by the Docker scenario.
• Focused suites: bun test on the fixture, closed-error-output, effect-diagnostics, command-record, docker-io, tests/cli-spawn-env-isolation.test.ts and e2e-shard tests: 135 pass, 0 fail.
• bun run --cwd packages/cli typecheck: clean. bun run --cwd packages/cli check: 0 errors and 34 warnings. The two complexity warnings in the carrier file existed before this PR.
• Full suite: hosted test passed on fc4095b. A local macOS bun run test (Bun 1.4.2, load average about 30) failed in files unrelated to this change, including native-authored-project-start, global-command (in the combined run) and native-compose-storage-fence-cost. native-authored-project-start and native-compose-storage-fence-cost fail the same way with the pre-fix carrier source. global-command passes when run alone. These are local environment failures, not regressions.
• gitleaks with .gitleaks.toml on the full branch diff: no leaks.
• Hosted e2e on the fixed code (run 39644645624, head fc4095b): all four shards passed native-config-networks. The two stalled shards took 3,200 s and 2,675 s, and the two unstalled shards took 2,190 s and 2,430 s. Run 39643040488 shards 3 and 4 (pre-fix, unstalled) also passed.
• Final head 055ecce (run 39654287962): all 16 checks passed, including test and docker-e2e shards 1-4.

Open gates

• The historical failures are attributed by matching signature and correlation, not by a recovered command. See "What is not proven" above.
• The stability evidence is the stall A/B plus a few clean hosted runs. It is not a long natural sample.
• HACK-1281 in Linear has not been updated from this unit.

🤖 Generated with Claude Code

[linear-code]

HACK-1281