[roodboi]

Why

down, recover and host-filesystem inspection prove the provider released its disks by running lsof over storage.raw and overlay.raw. That scan was bounded at 5 s (STOP_HANDLE_WAIT). lsof walks every process's descriptors, and on a loaded host it takes longer than that. down then ended as provider_timeout_uncertain, wrote no clean-stop marker, and the next restart fell back to the fresh disk budget. The start path's release check already used 30 s. Neither bound had a test, because lsof is a fixed host path.

What changed

• New lifecycle/disk_handles.rs. The scan now goes through an injectable HandleProbe and a Clock. Production uses the host lsof and the monotonic clock. The answer is interpreted the same way for every probe: only exit 1 with no stdout and no stderr proves the disks closed. The ownership checks are unchanged (VM lock, lsof, sockets, disk identity).
• Stop bound raised from 5 s to 30 s, the same as RELEASE_HANDLE_WAIT. Both constants now live together with their measurements documented. 30 s is more than four times the slowest scan measured. With down's other bounded waits (15 s guest shutdown acknowledgement, 10 s provider exit), a stop still ends within about a minute.
• process::capture split into spawn, Pending::poll/expire and wait_within(pending, timeout, clock). The bounded wait can now run on a fake clock. Production behavior is the same: the bound covers exit plus stream close; on timeout only the direct child is killed and reaped; a helper that keeps output open is reported as provider_output_uncertain.
• Deterministic lsof tests. In test builds only, real lsof scans take a per-user flock, so every test process (isolated children included) runs one whole-host scan at a time. Each scan's bound starts once it holds the lock. Concurrent scans under load are what timed these tests out. The lock is released explicitly, so a forked copy of the descriptor cannot keep it held.
• scripts/measure-lsof-scan.ts times the exact scan (sequential and concurrent rounds, optional CPU burners) and prints JSON with no paths or hostnames.
• README: one sentence on the bound next to the clean-stop marker description.

Measurements

These are from a Linux host with 20 CPUs (bun scripts/measure-lsof-scan.ts). Every scan returned the clean no-holder answer.

processes	load (1 min)	round	median	p95	max
~790	57	sequential	0.38 s	0.46 s	0.51 s
~810	62	8 concurrent	2.41 s	2.86 s	2.87 s
~840	84	sequential, 40 burners	0.66 s	1.82 s	4.88 s
~890	121	16 concurrent, 80 burners	4.16 s	4.99 s	5.34 s (above the old 5 s bound)

The earlier macOS figure, 5.4–6.5 s at about 1,850 processes and load 30–80, came from the previous RELEASE_HANDLE_WAIT comment. No new macOS timings have been taken yet. To take them, run bun scripts/measure-lsof-scan.ts --runs 32 --concurrency 8 (add --burn N for CPU contention) on a loaded Mac.

Tests

disk_handles::tests are not platform-gated, so they run on Linux and macOS:

• both_bounds_outlast_a_slow_loaded_host_scan: a scan four times the slowest measured one (26 s on the fake clock) passes under both bounds.
• both_bounds_expire_a_hung_scan_as_uncertain_at_the_bound: a scan that never finishes is expired at the bound as provider_timeout_uncertain. The test also checks that each bound is at most 45 s.
• a_scan_finishing_after_the_bound_is_uncertain: a "no holder" answer that arrives late is not accepted.
• only_a_silent_no_holder_answer_proves_the_disks_closed: a listed PID, a diagnostic on stderr, exit 0 and exit 2 all refuse as stop_uncertain.
• host_lsof_finds_a_held_disk and host_lsof_proves_closed_disks_closed (isolated child): the same checks against real lsof. They skip on Linux hosts without lsof; macOS must have it.

process::tests add exit status and both streams, kill and reap on timeout, and a helper that retains output reported as provider_output_uncertain.

Mutation testing. All 11 mutations were killed:

• stop bound set to 5 s, and to 600 s
• release bound set to 5 s, and to 600 s
• lsof exit status ignored; lsof stderr ignored
• wait doubled at the call site
• wait never expires
• expiry returns without signalling the child
• no kill on timeout
• output retention reported as a timeout

Where it ran

• Linux (this host): cargo fmt --check, cargo clippy --all-targets -D warnings, the same with --all-features, and the focused tests (provider::lifecycle::disk_handles, provider::process) all pass. The full cargo test run was still queued behind other work on this machine's shared job limiter when I pushed; hosted CI is the full-suite signal.
• macOS: I only type-checked, via cargo clippy --target aarch64-apple-darwin --all-targets (default and all features, clean), cross-compiled from Linux. The macOS-only lifecycle tests (admission_pool, short_home, host_filesystem, interrupted) were not run locally. The CI Candidate core (macos-latest) job runs them.

Remaining gaps

• No macOS scan timings measured yet (see above).
• Call-site wiring (lock_absent_disks → stop bound, released → release bound) is covered by the macOS lifecycle tests only. Both bounds are 30 s, so swapping them would change nothing.

Linear: HACK-1328

Review fixes (after fee8ad8)

• process::wait_within reads the clock after every poll. A result first seen at or after the bound is refused as provider_timeout_uncertain, so a waiter that oversleeps past the deadline can no longer accept a late scan. New fake-clock tests oversleep across the bound (refused) and within it (accepted). Restoring the old order makes the first test fail.
• scripts/measure-lsof-scan.ts owns every child it starts:
  • Burners and scans are registered as they are spawned, inside the cleanup scope.
  • Each scan is bounded across its exit and both streams (--scan-timeout, default 120 s).
  • Abort and failure cancel and await in-flight scans, then kill and await every child.
  • An already-aborted signal starts nothing. --lsof lets you time a different lsof.
  • tests/measure-lsof-scan.test.ts has 8 tests: a clean run, a hung scan, a held pipe, a partial launch failure, abort, an already-aborted signal, a concurrent spawn failure, and SIGINT to the CLI with a held pipe.
• The merge of next keeps test(runtime-core): size release-check scans for load and pin the drop, kill-anchor and retired-format guards #501's test-only 60 s LOADED_HOST_SCAN_WAIT for the lock_absent_disks fixtures, as STOP_SCAN_WAIT. Production stays at the 30 s disk_handles::STOP_HANDLE_WAIT.
• Independent review (GPT-6-Astra, codex-dio): CLEAN at fe29abef.

🤖 Generated with Claude Code

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

[linear-code]

HACK-1328

[roodboi]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-11-02T23:16:27.004796Z We couldn’t complete this review. Comment @codex review to try again.	fee8ad8	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[roodboi]

Independent review (GPT-6-Astra, codex-dio): VERDICT: CLEAN at fe29abe. Reviewer thread mcp:1c082903-bbd9-4cbe-a2c3-42090cee6afe.

Report-only review over three rounds. Round 1 at fee8ad8 raised two P2s: a late scan result accepted after the deadline, and benchmark children escaping cleanup. Both were fixed in 11b4215. Round 2 raised one P2 (abort not cancelling in-flight scans), fixed in fe29abe. Round 3: no remaining findings. Exact-head CI and fresh macOS lsof timings remain separate gates. No PR merge performed.

[roodboi]

Re the Codex connector comments above (summary and usage-limit notice at fee8ad8): they hold no findings; the bot review didn't run because of its usage limit. The independent GPT-6-Astra (codex-dio) review stood in for it and is CLEAN at fe29abe (comment above). A @codex review can be re-requested once the limit resets.

[chatgpt-codex-connector]

You have reached your Codex usage limits. You can see your limits in the Codex usage dashboard.