test(e2e): prove network probe budget and resolved-address fetch (HACK-1428) by roodboi · Pull Request #484 · hack-dance/hack

roodboi · 2026-10-24T15:16:33Z

Follow-ups after #470 (e08f971). Test and fixture-harness changes only; no product code changes.

Root cause

Shared-budget subtraction untested. Local DNS resolution of 127.0.0.1 or localhost takes about 0 ms, and the hung-probe bound (< 2000 ms for a 300 ms budget) was loose. So Math.max(1,${timeout}-d) → ${timeout} still passed.
Resolved-address fetch untested. Every probe test used names that the system resolver also resolves to the listener. A fetch to the original hostname reached the same server, so removing target.hostname = …x.address still passed.
Create shim oracle flake (Expected: 71, Received: 128). The test makes nine shim invocations in sequence, and each one starts one or two Bun processes. On a loaded host this took longer than bun test's 5 s default timeout. When the timeout fires, bun test SIGTERMs the still-running shim ("killed 1 dangling process"). The runner mapped the signaled exit (code === null) to a bare 128. The timed-out test then keeps running while later tests run, and its expect(…).toBe(71) fails. Bun reports that as "Unhandled error between tests", so the visible failure is Expected 71 / Received 128 and not the timeout. A minimal repro outside the repo showed the same thing: a test whose child outlives the timeout gets exit code=null signal=SIGTERM, then Expected: 71 Received: 128 as an unhandled error between tests. The same mechanism also hit probe scripts exit with a closed code… (Received: 143, from Bun.spawn's 128+15).

Fix

tests/native-networks-fixture.test.ts
- New peer probe fetches the address it just resolved: replaces Bun.dns.lookup in the probe child with a stub that resolves RFC 6761 .invalid names to 127.0.0.1. The probe must pass, and the server must see Host: 127.0.0.1:<port> on both requests. A second, real resolution of a .invalid name cannot succeed.
- New peer probe charges resolution time against the shared per-URL budget: the stub delays resolution by 1200 ms with a 2000 ms budget against a hung endpoint. It asserts dnsMs ≥ 1150, elapsedMs ≥ 1950, and elapsedMs < 2600 (a fresh full budget ends near 3200). It also checks that a 500 ms budget with a 1200 ms resolution exits dnsTimeout (31) with phase: dns at 450–1200 ms.
- New capture reports a signaled command as 128 plus its signal number (SIGTERM → 143, SIGKILL → 137).
- setDefaultTimeout(60_000) for this file, following tests/native-compose-ownership.test.ts. Every fixture command and probe keeps its own, tighter timeout.
tests/e2e/native-config-networks-acceptance.ts: runNativeNetworkFixtureCommand reports a signaled command as 128 + signal number, as a shell does, so a future kill names its signal. The timeout and oversize paths still throw before any exit code is returned. Nothing compared the exit code with 128.

Evidence (local, macOS, Bun 1.4.2; host load average 45–110 from other work)

Mutations (each reverted afterwards):

Mutation	Before this PR (`-t probe`)	With this PR
`Math.max(1,${timeout}-d)` → `${timeout}`	5/5 pass	budget test fails: `Expected: < 2600, Received: 3211`
drop only `target.hostname=…x.address;` (keep `let ok=false;`)	5/5 pass	address test fails (`code 26`, not 0); budget test fails (`Expected 27, Received 26`)
runner `code ?? 128` (signal ignored)	n/a	signal test fails: `Expected: 143, Received: 128`

Flake loops (bun test tests/native-networks-fixture.test.ts, private HOME/HACK_HOME/DOCKER_CONFIG, HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK=1, no Docker):

Run	Before fix	After fix
shim test alone (`-t "strict argv oracle"`), 40 runs	21 pass / 19 fail, all at the 5 s timeout	39 pass / 1 fail (the failure was not this test; see gaps)
whole file, 10 / 30 runs	8 / 10: both failures were the shim test timing out with `Expected: 98, Received: 128` plus the closed-code probe test with `Received: 143`	30 / 30 pass

Gates: bun run check (includes privacy:check) passes, bun run --cwd packages/cli typecheck reports 0 errors (it includes tests/), commitlint passes, and gitleaks 8.30.1 finds no leaks over origin/next..HEAD. The CLI surface and agent guidance did not change. Hosted CI at the PR head is the full-suite evidence.

Gaps

Hosted test and docker-e2e shards at this head are the full-suite proof; Docker e2e was not run locally.
In 1 of the 40 post-fix runs of the single test, the shim test itself passed, but the checkout write guard's afterAll git status timed out at 5 s under load (tests/helpers/checkout-write-guard.ts). That is outside this unit and is reported separately.
The IPv6 branch of the address rewrite ([${x.address}]) still has no test.
Stopping a loop mid-run (killing its bun test) left one Bun SIGABRT crash report from an orphaned child. The loops that ran to completion added none.

Refs HACK-1428

🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}

…K-1428) The peer probe charges name resolution against one per-URL budget and fetches the address it just resolved, but no test failed when either behavior was removed. Two tests now run the probe with a stubbed Bun.dns.lookup: one resolves RFC 6761 .invalid names to 127.0.0.1 and checks both requests reached that address, and one delays resolution and checks the HTTP phase only gets the remaining budget. The create shim oracle flake (Expected 71, Received 128) was bun test's 5 s default timeout firing on a loaded host: bun test SIGTERMs the running shim and the runner mapped the signaled exit to a bare 128. The fixture test file now sets a 60 s default, and the runner reports a signaled command as 128 plus its signal number. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

linear-code · 2026-10-24T15:16:40Z

HACK-1428

…drain isolated test output (#482) ## Summary Two follow-ups from the #399 and #467 verifiers (HACK-1427). ### 1. `run_isolated` no longer hides a noisy child's failure behind the hang guard `packages/runtime-core/src/provider/test_executable.rs`: `run_isolated` (from #387, extended by #399) used to pipe the child's stdout and read it only after the child exited. A failing child that wrote more than a 64 KiB pipe buffer blocked on its write until the 120 s `ISOLATED_HANG_GUARD`. The guard then reported a watchdog panic instead of the real failure. - stdout and stderr are both piped and drained on reader threads while the parent waits. Each reader keeps a bounded tail (16 KiB) for the failure message and drains the rest, so the child never blocks on a full pipe. - The bound covers the child's exit and the readers reaching end of stream. A descendant that keeps a copy of a pipe open is reported, not waited on forever. - A failure message now includes both streams' tails. A passing child must still print at most 4096 bytes on stdout and must show the exact `test <name> ... ok` / `1 passed` lines, as before. - The second helper, `provider/graph/startup/runtime/rebind/tests.rs:219`, only delegates to this shared helper, so this change fixes it too. No other `run_isolated` exists. New test `isolated_child_failure_with_large_output_is_reported` launches the ignored child `noisy_failure_child`. That child writes 160 KiB to each stream and then panics. The test requires the parent to fail well within the guard with the child's own failure, not "watchdog". The message must hold the panic text and the stderr end marker, and must be bounded. ### 2. The busy refusal over an interrupted rollback names the command that works there `src/commands/config-adopt.ts`: once a switch's rollback is journaled (`publication.phase: rolling-back`), plain `hack config adopt --recover` refuses whatever the override holds, because `completePublication` accepts only `switching`. The generic `E_LEGACY_ADOPTION_BUSY` step still named it first whenever the override was not literal. The busy step for `rolling-back` is now `PENDING_ROLLBACK_BUSY_NEXT_STEP`, which names only `hack config adopt --recover --rollback`. The `switching` wording (HACK-1414) is unchanged. The text is fixed and value-free. `docs/reference/native-compose-adoption.md` is updated. `src/lib/native-compose-adoption-marker.ts`: the `classify()` comment now matches `pendingSelectionMatches`: a pending stop is also admitted over a `rolled-back` publication. `LegacyComposeAdoptionSelectionDetail` gains `switchPhase` so the CLI can tell the two pending-switch phases apart. ## Validation All at head `2adbd9dc`: - Rust (cargo 1.97.1): `bun run check:local` passes (fmt and clippy `-D warnings`). The focused `provider::test_executable` test passes. - `bun run test:local` was run twice on a heavily loaded local host (load average 50 to 130). Each run had 1051 passing tests and one different failure: `dependency_slots::concurrent_claim_writers_serialize_without_overlapping_slots` (its 3 s lock deadline) in the first run, and `relay_owner::publication::dead_publication_recovery_requires_unchanged_receipt_and_socket` in the second. Neither test touches `run_isolated` or any changed file, and both modules pass when rerun alone (`dependency_slots` 20/20, `relay_owner::publication` 11/11 three times). Hosted Candidate core on macOS and ubuntu is green. - Adoption: both HACK-1427 tests pass, run focused and serially with `--timeout 60000` (2 pass, 146 s), because these tests are load-sensitive (HACK-1367). - gitleaks 8.30.1 over `origin/next..HEAD`: no leaks. `bun run check` (includes `privacy:check`): exit 0. `bun run --cwd packages/cli typecheck`: 0 errors. - Hosted CI at `2adbd9dc`: all checks green, including Candidate core (macos-latest, ubuntu-latest), `test` shards 1-4/4, docker-e2e shards 1-4/4, Config compiler (4 targets), Runtime state models, linux-process-lifetime, runtime-images and secret-scan. Refs HACK-1427 ## Round 2 (rework after verifier: no PR existed) The first implementation round committed `ad916597` locally but never pushed it, so the verifier found no branch and no PR. This round reviews that commit, adds the report's exact repro, reruns every gate, pushes the branch and opens this PR. **Added test.** `CLI busy refusal over an interrupted rollback of a literal-bound switch does not name plain recovery once the override is moved aside (HACK-1427)` sets up the exact reported state. It starts from the HACK-1318 literal fixture, journals the switch with a first `--recover --stop`, rewrites the receipt phase to `rolling-back`, and moves the bound override aside. `--recover --stop --json`, plain `--json` and `--stop --json` must each return the store's BUSY message plus `PENDING_ROLLBACK_BUSY_NEXT_STEP`, and must never name plain `--recover`. Plain `--recover` must refuse and leave the receipt unchanged. The existing round-1 test, `...names only the recovery rollback that finishes it (HACK-1427)`, covers an active adoption with a reference-form override, rewritten to `rolling-back`. It shows that the named `--recover --rollback` actually completes the rollback and restores the originals. Plain `--recover` refuses there. **Mutation proof.** - TS: with the `rolling-back` branch of `pendingSwitchBusyNextStep` disabled, both HACK-1427 adoption tests fail (the busy message is the generic step). Restored, both pass. - Rust: with the reader threads started only after the child exits (the old ordering), `isolated_child_failure_with_large_output_is_reported` fails after 120 s with `exceeded its watchdog`. Restored, it passes. **Known limit (reported separately).** In the reported state, the bound override has been moved aside. `--recover --rollback` then refuses as `E_LEGACY_ADOPTION_STATE`, because the saved generated-input proof pins the file, including its ctime. Moving the file back does not satisfy the proof, because a rename changes ctime. The message now names the only command that can finish a `rolling-back` publication, and that command works whenever the bound inputs are intact. With the bound file moved away, though, no `hack config adopt` command finishes the rollback. That is outside this unit and is reported as a follow-up finding. (Resolved in round 3 below, HACK-1437.) ## Round 3 (HACK-1437: a supported way out of an interrupted rollback with a moved override) The round-2 verifier found that the known limit above was a real dead end. In the HACK-1318 literal fixture, journal the switch with `--recover --stop`, then move `.hack/.internal/compose.env.override.yml` aside. `--recover --rollback` journaled `rolling-back` and then refused with `E_CONFIG_INVALID`. Plain `--recover` refuses a rollback. The busy message named `--recover --rollback` again, and moving the file back did not help, because a rename changes ctime. This round fixes that path. It merges `origin/next` twice, first `1281184b` and then `7dac4a03` after next moved again. Both merged cleanly and neither touches the files this round changes. The fix commit is `0e17a563` and the head is `80c9b1fe`. **What the proof protects.** A saved read binds each generated override by its bytes and file identity, ctime included. This protects adopted execution and switch completion, which must run exactly the inputs that preparation qualified and whose Compose config hash the retained containers carry. The binding also keeps `hack doctor --fix` and the env scrubs from removing a pinned file. Rollback restores the held legacy originals, archives the candidate and verifies the original resources. It never reads, runs, rewrites or removes the env override, which is derived from the managed env layers and rewritten by the next `hack up`. No data depends on it. **Change.** - `src/lib/native-compose-adoption-projection.ts`: `readSavedLegacyComposeAdoptionProjection` takes `envOverride: "bound" | "released"`, with `bound` as the default. In `released` mode the env override is neither read nor compared. Every other generated proof stays exact: the `.hack/.internal` directory identity and the runtime override's bytes and identity. The saved `config_files` path list keeps the override's path, so the containers are still matched against the label Compose recorded. The new value-free `savedLegacyComposeEnvOverrideChanged` reports a moved or changed override when the other generated inputs still match. - `src/lib/native-compose-adoption-generation.ts`: rollback reads in `released` mode in `rollback()` (from `active`) and in `completeRollback` (`--recover --rollback` from `switching` or `rolling-back`). Execution, leases, publication and completion still use `bound`. The new read-only store method `pendingSwitchEnvOverrideChanged()` backs the CLI hint. - `src/commands/config-adopt.ts`: every pending-switch step names only a command that the state does not refuse for a known reason. The new `PENDING_CHANGED_SWITCH_BUSY_NEXT_STEP` covers a `switching` publication whose bound override was moved or changed, and names only `--recover --rollback`. A refused plain `--recover` now adds the step when its cause is known (`rolling-back`, or a moved or changed override). The text is fixed and value-free. - `docs/reference/native-compose-adoption.md`: a new "Rollback releases the generated env override (HACK-1437)" section gives the written rationale. It covers what the binding protects, why rollback can release it, what rollback still checks, and the step for each pending state. The version 3 paragraph notes the exception. **Tests** (`tests/native-compose-adoption-generation.test.ts`; long tests use the named budget `ROLLBACK_RECOVERY_BUDGET_MS`, HACK-1367): - State 2, reachable without hand edits: `CLI recovery rollback finishes a pending switch whose bound literal override was moved aside and back, without hand edits (HACK-1437)`. With the override moved aside, BUSY names only `--recover --rollback` (`PENDING_CHANGED_SWITCH_BUSY_NEXT_STEP`). Plain `--recover` refuses, journals nothing and names the same step. After the file is moved back (same inode, new ctime), BUSY names the literal step. The test then parses the message and runs the one command it names. That command must finish: the originals are restored, the selection is `rolled-back`, and the literal file is untouched and no longer pinned. - State 1, `rolling-back` with the override moved aside: the round-2 test (renamed `..., and that rollback finishes (HACK-1427, HACK-1437)`) now also checks that plain `--recover` names `PENDING_ROLLBACK_BUSY_NEXT_STEP`, then runs the named command to completion. The override stays absent and the moved file is untouched. - `CLI rollback of an active adoption releases a replaced generated env override while plain recovery stays refused (HACK-1437)`: `--rollback` finishes with a different file at the bound path and does not rewrite it. - Store level: `rollback releases a moved-aside generated env override that execution still refuses` checks that `loadActive` still refuses while rollback succeeds. `rollback still binds the generated runtime override when the env override is released` rewrites the runtime override with identical bytes (a new ctime) and checks that rollback still refuses before anything moves. **Mutation proof** (each reverted afterwards): - M1, rollback reads with the full binding (`released` disabled): store-level release, state 1, state 2 and active-rollback tests fail at the rollback step. The strictness test still passes. - M2, the moved/changed check removed from the busy step: state 2 fails at the BUSY message (it gets the generic step naming plain `--recover`). - M3, the hint on a refused plain `--recover` disabled: state 1 and state 2 fail at the completion message. - M4, the released path dropped from `config_files`: the store-level release, state 1 and active-rollback tests fail (rollback refuses on the binding). - M5, the released generated-proof comparison skipped: the runtime-override strictness test fails with "unexpected adoption success". **Gates** (fix commit `0e17a563`; rerun at head `80c9b1fe` after the second merge): - At both commits: gitleaks 8.30.1 (`gitleaks dir` over a `git log -p origin/next..HEAD` export): no leaks. `bun run check` (includes `privacy:check`): exit 0, 34 existing complexity warnings. `bun run --cwd packages/cli typecheck`: 0 errors. CLI surface and agent guidance are unchanged, so no regeneration was needed. - Focused (local host at load 20 to 170): all 5 HACK-1437 tests pass together at `0e17a563` (714 s) and again at `80c9b1fe` (555 s). The HACK-1318/1414/1427/1273 group passes 8/10. The two failures, `...keeps it loadable until rollback frees it (HACK-1273)` and `v3 value-free env override binds the engine config hash... (HACK-1273)`, time out at their fixed 20 s budget inside `store.prepare`, before any changed path. They fail the same way on the unchanged PR head `2adbd9dc` on this host, so they are load-sensitive (HACK-1367), not a regression. For local runs only, the named budget was raised so that the load could not end a test. The committed value is 240 s. - Hosted CI at `0e17a563` (run 40124835555): all 20 checks green, including `test` shards 1-4/4, docker-e2e shards 1-4/4, Candidate core (macos-latest, ubuntu-latest), Config compiler (4 targets), Runtime state models, linux-process-lifetime, runtime-images and secret-scan. - Hosted CI at `80c9b1fe`: all 20 checks green (run 40128748032), with the same set as above. After that run, `next` moved three more commits to `246f1b4e` (#484, #485 and #486: runtime-core and e2e tests, none in this change's files). `git merge-tree` merges them cleanly. They are not merged here, so GitHub reports BEHIND again. **Remaining limits.** Rollback still refuses when something it checks has changed: a running original, an edited held original or candidate, changed resources, changed managed env layers, the runtime override, or conflicting bytes at an original path. That refusal keeps the journal and evidence. Rollback keeps the managed-revision proof. Hack's own `hack env set` refuses while a switch is pending: a temporary probe, not committed, showed it fails with the adoption selection refusal. A managed env layer file edited outside Hack during a pending switch may still leave a similar dead end, because `--recover --rollback` journals `rolling-back` before it reads its inputs. That case is not reproduced here and is reported as a lower-severity follow-up finding. Refs HACK-1437 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…p, kill-anchor and retired-format guards (#501) ## Current state (head `67400987`) Head `6740098790762bed610b990f21ee14c9e27caa22` merges `origin/next` at `8621d907` and fixes the should-fix from the independent review of `619fb8e1`. Sections below this one describe older heads. **Change:** `partial_hello_has_fixed_deadline_and_never_connects` now admits with a 4 s budget, asserts that this is shorter than the fixture's 5 s connect timeout, and brackets the observed deadline against the 4 s budget. It still expires the deadline by hand. Test-only; no production code changed. **Evidence (local, macOS, Rust 1.97.1, Zig 0.15.2, on the final test source unless noted):** - Changed test: 12 of 12 runs passed on the final source, each confirmed to run the test. A further 35 passed before a formatting-only edit. - Mutation: with `RelayLoop::admit` using `self.limits.connect_timeout` instead of its budget, 5 of 5 runs failed with `E_TEST_HANDSHAKE_DEADLINE_NOT_FROM_BUDGET`. Reverted. - `cargo fmt --check` and `cargo clippy --all-targets --all-features -- -D warnings` pass. - Lib test binary, run once before the formatting-only edit: 1071 passed, 0 failed, 103 ignored. - Gitleaks over `619fb8e1..6740098`: no leaks. **Not verified:** hosted CI at this head, the integration test binaries and the full suite, and the areas the review did not read (including whether the supervisor also signals the single process as well as the group). --- ## Current state (head `619fb8e1`) Head `619fb8e1641b1b7971babc7b5091a3de3c70e1d0` merges `origin/next` at `ae032e79` and reworks the two defects from the failed re-verification of `6508480b`. Everything below this section is earlier history and describes older heads. **Changes in this rework** - `buffered_hello_keeps_original_deadline_and_wrong_proof_never_connects`: replaced the fixed 20 ms deadline with `MAX_CONNECT_BUDGET`. It still asserts that the exact deadline survives admission. It now also asserts that handshake progress does not renew a deadline, and it expires a deadline that the test sets itself. - `cpu_counter_advances_in_nanoseconds_under_work`: now measures in an isolated single-thread child (`cpu_counter_child`), so the upper bound is tight. The bounds are 2/3 and 3/2, and the false comment is corrected. - The other fixed bounds that the verdict listed as non-blocking are unchanged. **Verified locally on macOS at load average about 11-22, with no load generators, using `cargo +1.97.1` in `packages/runtime-core`** | Check | Result | | --- | --- | | Buffered hello test, repeated | 20/20 passed | | CPU counter test, repeated | 20/20 passed | | Child ratios over 30 runs | native/inner 1.0002-1.0009, native/outer 0.9998-0.9999 | | Mutation: `cpu_nanoseconds` x2 | CPU test failed 10/10 (`E_TEST_CPU_COUNTER_UNITS`) | | Mutation: raw Mach ticks | CPU test failed 10/10 (`E_TEST_CPU_COUNTER_UNITS`) | | Mutation: `admit_hello` renews the deadline | Buffered hello test failed 5/5 | | Mutation: handshake progress renews the deadline | Buffered hello test failed 5/5 | | Lib test binary, one run | 1070 passed, 0 failed, 102 ignored | | `cargo fmt --check` | Passed | | `cargo clippy --all-targets --all-features -- -D warnings` (Zig 0.15.2) | Passed | | `gitleaks` over `6508480b..619fb8e` | No leaks found | All mutations were reverted, confirmed by file hash. **Not verified** - Hosted CI at `619fb8e1` had not finished when this section was written. - These tests have not been re-run under the heavy load that exposed the original failure. No loaded lib runs were done at this head, and an independent re-verification is still open. - The full test suite and `bun run test:local` were not run. **Remaining limit:** `admit_hello` refuses a deadline more than 5 s away, so a stall of 5 s or more between computing the deadline and that call would still fail the test. --- Follow-ups from the #484, #485 and #486 verifiers. This changes test code, with one exception: `Reservation::drop` is refactored behind a named budget, and its behavior is unchanged. Refs HACK-1457 ## Root cause and fix ### 1. Load-sensitive release-check tests **Cause.** Every failing test reaches `lifecycle::lock_absent_disks`, which runs a whole-host `lsof` over the owned disks with the production `STOP_HANDLE_WAIT` of 5 s. `lsof` walks every process on the host. Under load, and with other tests' isolated children scanning at the same time, one scan can take longer than 5 s. The fixture then fails as `provider_timeout_uncertain` before it reaches its result under test. I reproduced this on `next` (246f1b4) by adding 20 CPU spinners and 20 `/bin/sh -c true` spawn loops (load ~105-120). Run 3 of 3 failed 7 tests, all with `provider_timeout_uncertain` or the isolated parent of such a child: - `interrupted::unadopted_unclean_stop_child` (interrupted.rs:300, the one in the issue) - `admission_pool::uncertain_stop_rewrite_child` - `host_filesystem::device_migration_child` and `substituted_lock_child` - `short_home::explicit_recovery_child`, `vm_lock_release_child` and `active_socket_retry_child` The G105 verifier logs show the same set at load 78-137. **Fix.** The unit-test build gets a named budget instead of the fixed 5 s. `STOP_HANDLE_WAIT` is `test_executable::LOADED_HOST_SCAN_WAIT` (60 s) under `cfg(test)`. The production value (5 s) is unchanged. These fixtures prove what the scan reports (status 1 and no output), not how fast it reports it, so what they prove is unchanged. Tests of the timeout path itself already pass their own bound to `lock_absent_disks_within`. 60 s is still well inside `ISOLATED_HANG_GUARD` (120 s), so a scan that hangs is still reported. The loaded runs found one more fixed bound, a 2 s wait for a spawned `/bin/sh` in `private_child` tests. It is a hang guard, so it is now `FIXTURE_SHELL_HANG_GUARD` (60 s), with a closed `E_TEST_FIXTURE_SHELL_HANG` message. I also tightened `unadopted_unclean_stop_child`'s second check. It used to accept any error (`is_err()`), so a scan timeout also satisfied it. It now requires the identification error itself (`provider_state`) and requires that no disk was recorded. ### 2. `Reservation::drop`'s unnamed 5 s The wait is now `RELEASE_LEASE_WAIT`, and the drop logic moved into `Reservation::release_within(wait) -> Release`. The fallback is documented. If the lease stays busy, or the lease, home or receipt does not verify, the receipt is left as written and its slots stay reserved. It is then released by `recover_orphan` (never-admitted runs) or by dead-owner graph cleanup (`recover_cleaned`). Both require the recorded owner to be dead. The behavior is unchanged. New test `a_release_that_outlasts_a_busy_lease_leaves_a_recoverable_receipt` checks four things: - a free lease removes the receipt - a held lease returns `LeaseBusy` only after the wait has elapsed - the receipt bytes and its reserved slot are unchanged after that - `recover_record`, the shared core of both recovery paths, then releases the receipt under the lease, and a late retry does not touch the receipt again ### 3. Claim-writer failure attribution **Cause.** A writer that panicked inside the lease never decremented the shared "inside" counter. Every later writer then failed with `E_TEST_CLAIM_LEASE_OVERLAP`, and `join().unwrap()` hid the original message. **Fix.** An `InsideLease` drop guard is declared after the lease guard, so it drops first and leaves the count before the lease is released. Joins now collect every writer's panic message in writer order (`E_TEST_CLAIM_WRITER_PANICKED (writer N: …)`). New test `a_writer_panic_inside_the_lease_names_itself_not_an_overlap` injects a panic in writer 0 and requires a failure that names only that panic. ### 4. #486 gaps (`src/node/`) - **Kill anchor.** New test `repeated_kills_of_a_child_that_never_exits_keep_the_first_kill_anchor`. The child is not a group leader, but its recorded identity says it is, so every group signal is authorized and finds no group (`ESRCH`, tolerated). The child outlives each SIGKILL and is re-signalled on every later pass. The test requires `Child termination deadline exceeded.`, at least three signal checks, and the child still alive afterwards. A named 60 s hang guard turns a regression into a failure instead of a hang. - **Retired-format fixture.** The upgraded-node fixture is now the live child's own identity (`observe(pid)`) with only `boot` in the retired `seconds:micros` form. The old fixture also had `start: 1`. Note that `schedule` never calls `verify`: the "never signals" result comes from the structure. So the test now checks the only signal authority directly. `verify(legacy, live)` and `verify(legacy, legacy)` must both refuse with the retired-format message. ### 5. IPv6 probe branch New test `peer probe fetches an IPv6-only answer through a bracketed host`. The stubbed `Bun.dns.lookup` returns only `{address: "::1", family: 6}`, and the listener is on `::1`. The test requires exit 0 and that both requests arrive with `Host: [::1]:port`. `runStubbedPeerProbe` gained an optional `answer` for this. ## Mutation proofs (each reverted) | # | Mutation | Result | |---|---|---| | M1 | Base binary (5 s scan budget) under the same churn load | 1 of 3 full runs failed, 7 tests, all `provider_timeout_uncertain` | | M2a | Remove the receipt when the lease wait expires | `E_TEST_RECEIPT_CHANGED` | | M2b | Stop waiting for the lease (single attempt) | `E_TEST_RELEASE_WAIT_NOT_HONORED` | | M2c | Never remove the receipt under a free lease | `E_TEST_RELEASE_NOT_REMOVED` | | M3 | `InsideLease` skips its decrement while panicking (old behavior) | `E_TEST_CLAIM_FAILURE_MISATTRIBUTED`; the message lists writers 1-7 as `E_TEST_CLAIM_LEASE_OVERLAP` | | M4a | `killed.get_or_insert_with(Instant::now)` changed to `killed = Some(Instant::now())` | `E_TEST_KILL_GRACE_REANCHORED` after the 60 s guard; the 7 existing supervisor tests still pass | | M4b | Remove the retired-format guard in `ownership::verify` | `E_TEST_RETIRED_FORMAT_NOT_REFUSED` | | M5 | `target.hostname=x.address` (no brackets) | IPv6 test fails, exit 26 (`rejected`) | ## Evidence - **Under load.** I ran full lib runs (1,068 tests) with 20 CPU spinners and 20 spawn-churn loops, plus whatever else was running on the host. A trap stops the generator, and none was left running. - **Batch 1** (commit a3b87e8, load 101-242): 29 of 30 passed. The one failure, at load 228, was a different fixed bound: `private_child::independently_absent_guest_allows_reaping_failed_or_live_outer_transport` waited 2 s for a spawned `/bin/sh` to exit. Commit 8d591db turns that wait and its sibling into `FIXTURE_SHELL_HANG_GUARD` (60 s). - **Batch 2** (final head 8d591db, load 75-249): 29 of 30 passed. The one failure was `https_recovery::full_legacy_command_child`, which got `https_recovery_refused` after 1.3 s. That is not a wall-clock bound; see Gaps. - Across all 60 runs, no release-check test failed and no `provider_timeout_uncertain` appeared. The base commit under the same generator had failed 7 such tests in 1 of 3 runs. - **Focused.** `private_child` tests pass: 8. `dependency_slots::tests`, `node::` and `interrupted` tests pass: 75 passed. `tests/native-networks-fixture.test.ts` passes: 67 tests. - **Gates.** - `bun run check:local` (fmt and clippy `-D warnings`, cargo 1.97.1) passes. - All-feature clippy (`--all-targets --all-features`, with Zig 0.15.2) passes. - Root `bun run check` (with privacy:check) and `bun run --cwd packages/cli typecheck` pass. - gitleaks 8.30.1 over `origin/next..HEAD` finds no leaks. - **Full suite.** `bun run test:local` (cargo 1.97.1, no added load) at 8d591db: all 21 test binaries passed, including the lib (1,068 passed). Hosted CI at 8d591db is all green: test shards 1-4, Candidate core (ubuntu and macos), docker-e2e shards 1-4, Config compiler on 4 platforms, Runtime state models, linux-process-lifetime, runtime-images and secret-scan. ## Gaps - `https_recovery::full_legacy_command_child` failed once in 30 loaded runs, with `https_recovery_refused` after 1.3 s. The fixture binds an ephemeral port, drops it, and later requires `port_absent` on that port, so another socket on the host can take the port in between. This is a port-reservation race, not a timing bound, so it is reported as a finding and not changed here. The refusal is closed (no stage is reported), so I have not confirmed which check refused. - Other short fixed deadlines remain in the lib tests (`host_endpoint`, `relay_loop` and `https_recovery` tests use 1-3 s waits). None of them failed in the 60 loaded runs, so they were left as they are. - The production `STOP_HANDLE_WAIT` (5 s) is unchanged. `admission_pool.rs` documents `lsof` taking 5.4-6.5 s at load 30-80, so `down` and `recover` on a loaded host can still fail as `provider_timeout_uncertain`. This is reported as a finding and not changed here, because it changes product behavior. - `network_update::stopped` still has a fixed, unnamed 5 s `lsof` budget. No lib test reaches it after the scan, so it is not load-sensitive in the suite. It is left as is. - The kill-anchor test models a SIGKILL that never takes effect with a non-leader child and an injected leader identity. It does not use a real unkillable process. - The Docker e2e suite was not run locally, per policy. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Dimitri Kennedy <344842+roodboi@users.noreply.github.com>

roodboi mentioned this pull request Oct 24, 2026

train b58 integration gate (do not merge) #493

Closed

roodboi marked this pull request as ready for review October 24, 2026 20:31

roodboi merged commit 926cbdd into next Oct 24, 2026
21 checks passed

roodboi deleted the wf/g93-network-probe-test-gaps branch October 24, 2026 20:31

This was referenced Oct 24, 2026

fix(adoption): name the recovery an interrupted rollback accepts and drain isolated test output #482

Merged

test(runtime-core): size release-check scans for load and pin the drop, kill-anchor and retired-format guards #501

Merged

Navigation Menu

Repository navigation

test(e2e): prove network probe budget and resolved-address fetch (HACK-1428) - #484

test(e2e): prove network probe budget and resolved-address fetch (HACK-1428)#484
roodboi merged 1 commit into
nextfrom
wf/g93-network-probe-test-gaps

roodboi commented Oct 24, 2026 •

edited by blacksmith-sh Bot

Loading

Uh oh!

linear-code Bot commented Oct 24, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

roodboi commented Oct 24, 2026 • edited by blacksmith-sh Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Root cause

Fix

Evidence (local, macOS, Bun 1.4.2; host load average 45–110 from other work)

Gaps

Uh oh!

linear-code Bot commented Oct 24, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

roodboi commented Oct 24, 2026 •

edited by blacksmith-sh Bot

Loading