[roodboi]

Current state (head 67400987)

Head 6740098790762bed610b990f21ee14c9e27caa22 merges origin/next at 8621d907 and fixes the should-fix from the independent review of 619fb8e1. Sections below this one describe older heads.

Change: partial_hello_has_fixed_deadline_and_never_connects now admits with a 4 s budget, asserts that this is shorter than the fixture's 5 s connect timeout, and brackets the observed deadline against the 4 s budget. It still expires the deadline by hand. Test-only; no production code changed.

Evidence (local, macOS, Rust 1.97.1, Zig 0.15.2, on the final test source unless noted):

• Changed test: 12 of 12 runs passed on the final source, each confirmed to run the test. A further 35 passed before a formatting-only edit.
• Mutation: with RelayLoop::admit using self.limits.connect_timeout instead of its budget, 5 of 5 runs failed with E_TEST_HANDSHAKE_DEADLINE_NOT_FROM_BUDGET. Reverted.
• cargo fmt --check and cargo clippy --all-targets --all-features -- -D warnings pass.
• Lib test binary, run once before the formatting-only edit: 1071 passed, 0 failed, 103 ignored.
• Gitleaks over 619fb8e1..67400987: no leaks.

Not verified: hosted CI at this head, the integration test binaries and the full suite, and the areas the review did not read (including whether the supervisor also signals the single process as well as the group).

---

Current state (head 619fb8e1)

Head 619fb8e1641b1b7971babc7b5091a3de3c70e1d0 merges origin/next at ae032e79 and reworks the two defects from the failed re-verification of 6508480b. Everything below this section is earlier history and describes older heads.

Changes in this rework

• buffered_hello_keeps_original_deadline_and_wrong_proof_never_connects: replaced the fixed 20 ms deadline with MAX_CONNECT_BUDGET. It still asserts that the exact deadline survives admission. It now also asserts that handshake progress does not renew a deadline, and it expires a deadline that the test sets itself.
• cpu_counter_advances_in_nanoseconds_under_work: now measures in an isolated single-thread child (cpu_counter_child), so the upper bound is tight. The bounds are 2/3 and 3/2, and the false comment is corrected.
• The other fixed bounds that the verdict listed as non-blocking are unchanged.

Verified locally on macOS at load average about 11-22, with no load generators, using cargo +1.97.1 in packages/runtime-core

Check	Result
Buffered hello test, repeated	20/20 passed
CPU counter test, repeated	20/20 passed
Child ratios over 30 runs	native/inner 1.0002-1.0009, native/outer 0.9998-0.9999
Mutation: cpu_nanoseconds x2	CPU test failed 10/10 (E_TEST_CPU_COUNTER_UNITS)
Mutation: raw Mach ticks	CPU test failed 10/10 (E_TEST_CPU_COUNTER_UNITS)
Mutation: admit_hello renews the deadline	Buffered hello test failed 5/5
Mutation: handshake progress renews the deadline	Buffered hello test failed 5/5
Lib test binary, one run	1070 passed, 0 failed, 102 ignored
cargo fmt --check	Passed
cargo clippy --all-targets --all-features -- -D warnings (Zig 0.15.2)	Passed
gitleaks over 6508480b..619fb8e1	No leaks found

All mutations were reverted, confirmed by file hash.

Not verified

• Hosted CI at 619fb8e1 had not finished when this section was written.
• These tests have not been re-run under the heavy load that exposed the original failure. No loaded lib runs were done at this head, and an independent re-verification is still open.
• The full test suite and bun run test:local were not run.

Remaining limit: admit_hello refuses a deadline more than 5 s away, so a stall of 5 s or more between computing the deadline and that call would still fail the test.

---

Follow-ups from the #484, #485 and #486 verifiers. This changes test code, with one exception: Reservation::drop is refactored behind a named budget, and its behavior is unchanged. Refs HACK-1457

Root cause and fix

1. Load-sensitive release-check tests

Cause. Every failing test reaches lifecycle::lock_absent_disks, which runs a whole-host lsof over the owned disks with the production STOP_HANDLE_WAIT of 5 s. lsof walks every process on the host. Under load, and with other tests' isolated children scanning at the same time, one scan can take longer than 5 s. The fixture then fails as provider_timeout_uncertain before it reaches its result under test.

I reproduced this on next (246f1b4) by adding 20 CPU spinners and 20 /bin/sh -c true spawn loops (load ~105-120). Run 3 of 3 failed 7 tests, all with provider_timeout_uncertain or the isolated parent of such a child:

• interrupted::unadopted_unclean_stop_child (interrupted.rs:300, the one in the issue)
• admission_pool::uncertain_stop_rewrite_child
• host_filesystem::device_migration_child and substituted_lock_child
• short_home::explicit_recovery_child, vm_lock_release_child and active_socket_retry_child

The G105 verifier logs show the same set at load 78-137.

Fix. The unit-test build gets a named budget instead of the fixed 5 s. STOP_HANDLE_WAIT is test_executable::LOADED_HOST_SCAN_WAIT (60 s) under cfg(test). The production value (5 s) is unchanged. These fixtures prove what the scan reports (status 1 and no output), not how fast it reports it, so what they prove is unchanged. Tests of the timeout path itself already pass their own bound to lock_absent_disks_within. 60 s is still well inside ISOLATED_HANG_GUARD (120 s), so a scan that hangs is still reported.

The loaded runs found one more fixed bound, a 2 s wait for a spawned /bin/sh in private_child tests. It is a hang guard, so it is now FIXTURE_SHELL_HANG_GUARD (60 s), with a closed E_TEST_FIXTURE_SHELL_HANG message.

I also tightened unadopted_unclean_stop_child's second check. It used to accept any error (is_err()), so a scan timeout also satisfied it. It now requires the identification error itself (provider_state) and requires that no disk was recorded.

2. Reservation::drop's unnamed 5 s

The wait is now RELEASE_LEASE_WAIT, and the drop logic moved into Reservation::release_within(wait) -> Release. The fallback is documented. If the lease stays busy, or the lease, home or receipt does not verify, the receipt is left as written and its slots stay reserved. It is then released by recover_orphan (never-admitted runs) or by dead-owner graph cleanup (recover_cleaned). Both require the recorded owner to be dead. The behavior is unchanged.

New test a_release_that_outlasts_a_busy_lease_leaves_a_recoverable_receipt checks four things:

• a free lease removes the receipt
• a held lease returns LeaseBusy only after the wait has elapsed
• the receipt bytes and its reserved slot are unchanged after that
• recover_record, the shared core of both recovery paths, then releases the receipt under the lease, and a late retry does not touch the receipt again

3. Claim-writer failure attribution

Cause. A writer that panicked inside the lease never decremented the shared "inside" counter. Every later writer then failed with E_TEST_CLAIM_LEASE_OVERLAP, and join().unwrap() hid the original message.

Fix. An InsideLease drop guard is declared after the lease guard, so it drops first and leaves the count before the lease is released. Joins now collect every writer's panic message in writer order (E_TEST_CLAIM_WRITER_PANICKED (writer N: …)). New test a_writer_panic_inside_the_lease_names_itself_not_an_overlap injects a panic in writer 0 and requires a failure that names only that panic.

4. #486 gaps (src/node/)

• Kill anchor. New test repeated_kills_of_a_child_that_never_exits_keep_the_first_kill_anchor. The child is not a group leader, but its recorded identity says it is, so every group signal is authorized and finds no group (ESRCH, tolerated). The child outlives each SIGKILL and is re-signalled on every later pass. The test requires Child termination deadline exceeded., at least three signal checks, and the child still alive afterwards. A named 60 s hang guard turns a regression into a failure instead of a hang.
• Retired-format fixture. The upgraded-node fixture is now the live child's own identity (observe(pid)) with only boot in the retired seconds:micros form. The old fixture also had start: 1. Note that schedule never calls verify: the "never signals" result comes from the structure. So the test now checks the only signal authority directly. verify(legacy, live) and verify(legacy, legacy) must both refuse with the retired-format message.

5. IPv6 probe branch

New test peer probe fetches an IPv6-only answer through a bracketed host. The stubbed Bun.dns.lookup returns only {address: "::1", family: 6}, and the listener is on ::1. The test requires exit 0 and that both requests arrive with Host: [::1]:port. runStubbedPeerProbe gained an optional answer for this.

Mutation proofs (each reverted)

#	Mutation	Result
M1	Base binary (5 s scan budget) under the same churn load	1 of 3 full runs failed, 7 tests, all provider_timeout_uncertain
M2a	Remove the receipt when the lease wait expires	E_TEST_RECEIPT_CHANGED
M2b	Stop waiting for the lease (single attempt)	E_TEST_RELEASE_WAIT_NOT_HONORED
M2c	Never remove the receipt under a free lease	E_TEST_RELEASE_NOT_REMOVED
M3	InsideLease skips its decrement while panicking (old behavior)	E_TEST_CLAIM_FAILURE_MISATTRIBUTED; the message lists writers 1-7 as E_TEST_CLAIM_LEASE_OVERLAP
M4a	killed.get_or_insert_with(Instant::now) changed to killed = Some(Instant::now())	E_TEST_KILL_GRACE_REANCHORED after the 60 s guard; the 7 existing supervisor tests still pass
M4b	Remove the retired-format guard in ownership::verify	E_TEST_RETIRED_FORMAT_NOT_REFUSED
M5	target.hostname=x.address (no brackets)	IPv6 test fails, exit 26 (rejected)

Evidence

• Under load. I ran full lib runs (1,068 tests) with 20 CPU spinners and 20 spawn-churn loops, plus whatever else was running on the host. A trap stops the generator, and none was left running.
  • Batch 1 (commit a3b87e8, load 101-242): 29 of 30 passed. The one failure, at load 228, was a different fixed bound: private_child::independently_absent_guest_allows_reaping_failed_or_live_outer_transport waited 2 s for a spawned /bin/sh to exit. Commit 8d591db turns that wait and its sibling into FIXTURE_SHELL_HANG_GUARD (60 s).
  • Batch 2 (final head 8d591db, load 75-249): 29 of 30 passed. The one failure was https_recovery::full_legacy_command_child, which got https_recovery_refused after 1.3 s. That is not a wall-clock bound; see Gaps.
  • Across all 60 runs, no release-check test failed and no provider_timeout_uncertain appeared. The base commit under the same generator had failed 7 such tests in 1 of 3 runs.
• Focused. private_child tests pass: 8. dependency_slots::tests, node:: and interrupted tests pass: 75 passed. tests/native-networks-fixture.test.ts passes: 67 tests.
• Gates.
  • bun run check:local (fmt and clippy -D warnings, cargo 1.97.1) passes.
  • All-feature clippy (--all-targets --all-features, with Zig 0.15.2) passes.
  • Root bun run check (with privacy:check) and bun run --cwd packages/cli typecheck pass.
  • gitleaks 8.30.1 over origin/next..HEAD finds no leaks.
• Full suite. bun run test:local (cargo 1.97.1, no added load) at 8d591db: all 21 test binaries passed, including the lib (1,068 passed). Hosted CI at 8d591db is all green: test shards 1-4, Candidate core (ubuntu and macos), docker-e2e shards 1-4, Config compiler on 4 platforms, Runtime state models, linux-process-lifetime, runtime-images and secret-scan.

Gaps

• https_recovery::full_legacy_command_child failed once in 30 loaded runs, with https_recovery_refused after 1.3 s. The fixture binds an ephemeral port, drops it, and later requires port_absent on that port, so another socket on the host can take the port in between. This is a port-reservation race, not a timing bound, so it is reported as a finding and not changed here. The refusal is closed (no stage is reported), so I have not confirmed which check refused.
• Other short fixed deadlines remain in the lib tests (host_endpoint, relay_loop and https_recovery tests use 1-3 s waits). None of them failed in the 60 loaded runs, so they were left as they are.
• The production STOP_HANDLE_WAIT (5 s) is unchanged. admission_pool.rs documents lsof taking 5.4-6.5 s at load 30-80, so down and recover on a loaded host can still fail as provider_timeout_uncertain. This is reported as a finding and not changed here, because it changes product behavior.
• network_update::stopped still has a fixed, unnamed 5 s lsof budget. No lib test reaches it after the scan, so it is not load-sensitive in the suite. It is left as is.
• The kill-anchor test models a SIGKILL that never takes effect with a non-leader child and an injected leader identity. It does not use a real unkillable process.
• The Docker e2e suite was not run locally, per policy.

🤖 Generated with Claude Code

[linear-code]

HACK-1457