Repository navigation
fix: report a stopped shared host helper only after its exit is observed - #389
Merged
Merged
Conversation
Release now reports `stopped` only after a census, taken after the signals, shows every member of the captured tree exited: PID gone or a zombie. The wait is bounded and named (TERM grace, then SHARED_HOST_KILL_CONFIRM_MS after KILL). A PID still running, even with a drifted birth time, is never an exit and ends as `stop-unconfirmed` with the claim kept. A blind census mid-wait keeps polling instead of escalating early. The product and the fixtures now share one liveness predicate, isSharedHostProcessRunning, so a Linux zombie waiting for a slow reaper reads as not running everywhere. The zombie-blind fixture check caused the Linux-only failure of the lost-starter release test. Refs HACK-1336 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
roodboi
marked this pull request as ready for review
October 22, 2026 23:41
roodboi
added a commit
that referenced
this pull request
Oct 23, 2026
…ding; test shared host stop waits (#432) ## Summary Follow-ups to #392 (doctor in adopted projects, symlinked `.hack`) and #389 (shared host stop waits), merged into `next` at `4e0bfa55`. - **Deep start directory.** `hack doctor` no longer throws `E_NATIVE_PROJECT_UNSUPPORTED` when it starts more than 64 directories deep. It now uses ordinary project discovery. - **Adopted-project skips.** The 12 checks that doctor does not run for an adopted project now use doctor's skip wording. They keep status `ok`. - **New tests.** A pending adoption, `doctor --fix` with a symlinked `.hack`, a census that fails during the stop wait, and the length of the KILL-confirm wait. - **Docs.** The shared-host test row for a running PID with a new birth now describes what production does. ## Root cause 1. **Skip convention.** #392 reported the 12 checks as `ok` with `Not checked: …`. Every other doctor skip uses `Skipped (…)`. 2. **Deep start directory.** `preflightDoctorProject` called `selectLegacyComposeAdoptedRoot` without a catch. With no `--project`, that selector walks up at most 64 directories. If it reaches no root in that walk, it throws `unsupported()` (`E_NATIVE_PROJECT_UNSUPPORTED`). From a start directory more than 64 levels deep, every doctor run and every `--fix` step that looks for an adoption threw. Ordinary discovery has no such limit. 3. **Test gaps.** No test covered the pending-adoption warning or `--fix` in a pending adoption. There was a CLI regression for a symlinked `.hack/.internal`, but none for a symlinked `.hack`. In #389, mutants M7 (the old loop, which stops waiting on a failed census) and M10 (`awaitExit(0)` after KILL) survived. 4. **Docs.** The row "running PID with a new birth" said that release sends KILL. That is true only at the simulated signal boundary. In production, `signalVerifiedProcessTree` checks the birth time again before it sends a signal, so a PID with a different birth time gets no signal. ## Fix - **Skip convention: keep `ok` and use the `Skipped (…)` wording.** The new message is `Skipped (adopted Compose project: its inputs belong to 'hack config adopt')`. Doctor already uses two kinds of skip: - `warn` when the skip points at something to fix, such as `Skipped (no .hack/ found)` or `Skipped (docker runtime unavailable)`; - `ok` when the skip is expected in a healthy setup, such as `Skipped (not a git checkout)`. These 12 checks never apply to an adopted project, so `warn` would add the same 12 warnings to every healthy adoption in the summary and in `counts.warn`, and real warnings would be harder to see. Problems that belong to the adoption still show up in the `project` check (`warn` for a pending adoption) or in the adopted commands. The reasoning is in a comment in `src/lib/doctor-adopted-project.ts`, and in `docs/reference/native-compose-adoption.md`. - **Deep start directory: a closed result.** `findDoctorAdoptedProject` catches only the selector's search-limit refusal and returns `null`, meaning no adoption was found. Ordinary discovery then runs. With no `--project`, `E_NATIVE_PROJECT_UNSUPPORTED` from this selector can only mean the search limit. Unsafe adoption state throws `E_CONFIG_INVALID`, and that still stops doctor. `preflightDoctorProject` moved from `src/commands/doctor.ts` to `src/lib/doctor-adopted-project.ts` so the tests can call it. Its behavior did not change. - **Docs.** In `docs/reference/shared-host-lifecycle.md`, the row now says that release passes the member to the KILL step and waits the whole `SHARED_HOST_KILL_CONFIRM_MS`. The simulated boundary records the request, but in production no signal reaches that PID. There are new rows for the two new tests. ## Tests (every new test proven by mutation) The private env is set by a runner script: `HOME`, `HACK_HOME` and `DOCKER_CONFIG` point to a temp directory, `HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK=1`, and `DOCKER_HOST` points to a socket that does not exist. `doctor-fix-noninteractive` runs without `HACK_HOME`, because its existing daemon test finds `~/.hack` through its own private `HOME`. The pending-adoption tests use a new fixture helper, `leaveRetainedSourceBindOperationPending`. It runs a retained `start` whose callback reports failure, so the store keeps the pending journal. No engine command runs, and nothing sleeps. | Test | Mutation | Result | | --- | --- | --- | | `doctor-adopted-project`: the existing adopted `doctor --json` test now also checks the exact skip text and that there are 12 skipped checks | D1: status `warn` | killed | | same | D2: the old `Not checked: …` text | killed | | `doctor-adopted-project`: "a start directory deeper than the adoption search bound gets ordinary discovery instead of a throw". It uses 70 nested directories, first checks that the selector itself throws, then expects `findDoctorAdoptedProject`, `preflightDoctorProject` and the `--fix` plan to return without an error | D3: no catch (the #392 code) | killed: `E_NATIVE_PROJECT_UNSUPPORTED` | | `doctor-adopted-project`: "unsafe adoption state still refuses doctor". The state root is a regular file, and the test expects `E_CONFIG_INVALID` with no canary text and no path in it | D4: catch every error | killed | | `doctor-adopted-project`: pending adoption. `doctor --json` gives a `warn` `project` check with the exact message and `ok: true`. The 12 checks are skipped, the `--fix` plan is checked from the root and from a subdirectory, the adoption is still pending, the receipt is unchanged, and no container or project command runs | D5: pending reported like active | killed | | same, plus `doctor-fix-adopted-project`: real `runCli doctor --fix --no-interactive --migrate-env-config` with a pending adoption and a committed `.hack/.env.state.json` (real `git init`). It checks exit 0, the plan line, the untrack note, the env-migration skip, the closing note, an empty `git ls-files`, no `.hack/.gitignore`, the adoption still pending, the receipt unchanged, and no engine command or project `run` call | D6: pending no longer selected | killed (both) | | `doctor-fix-adopted-project` pending case | D6b: the untrack step treats a pending adoption as legacy | killed: exit 1 from the legacy writer's refusal | | `doctor-fix-noninteractive`: "doctor --fix leaves an override behind a symlinked .hack untouched". The whole `.hack` links to an outside directory that holds a file with the override's name. The test expects that file unchanged, both directory listings unchanged, `.hack` still a link, and the notes do not mention the file | D7: the ownership check covers only `.internal` | killed: ENOENT (the outside file was deleted) | | `shared-host-lifecycle`: "a census that fails during the TERM wait keeps the wait going". The first three polls fail and poll 4 shows the helper gone. The test expects `stopped`, 4 polls and TERM only | M7: the old loop | killed: `stop-unconfirmed` | | `shared-host-lifecycle`: "after KILL, release keeps polling for the whole kill-confirm wait". A 100 ms grace; the helper leaves the census only 80% of the way into `SHARED_HOST_KILL_CONFIRM_MS` after KILL. The test expects `stopped`, TERM then KILL, and a return at or after 80% of that wait | M10: `awaitExit(0)` | killed | | Existing "running with another birth time" test, which now also expects the return at or after `SHARED_HOST_KILL_CONFIRM_MS` after KILL | M10, and M10b: `awaitExit(SHARED_HOST_KILL_CONFIRM_MS / 2)` | both killed (both tests) | For each mutation, the test was run alone and failed for the reason shown. The file was restored afterwards. ## Evidence - **Toolchain.** Bun 1.4.2 (`mise x bun@1.4.2`), `bun install --frozen-lockfile`. `bun.lock` is unchanged. - **Local checks (M5):** - `bun run --cwd packages/cli typecheck`: 0 errors. - Root `bun run check`: exit 0, including `privacy-check: ok`. - Biome on the 7 changed source and test files: clean. - gitleaks 8.30.1, `gitleaks dir` over a `git log -p origin/next..HEAD` export: no leaks. - **Focused tests:** 179 pass, 0 fail, across 10 files: `doctor-adopted-project`, `doctor-fix-adopted-project`, `shared-host-lifecycle`, `compose-env-at-rest`, `doctor- command`, `doctor-json`, `doctor-generated-files`, `doctor-env-key`, `native-compose-adoption-source-bind-generation` and `native-compose-adoption-source-bind-fixture`. `doctor-fix-noninteractive` passed 11 of 11. `shared-host-lifecycle` also passed 3 more times in a row, 38 of 38 each. - **Hosted CI at `1345ce5a`:** all 16 checks pass (run 39876007802). These include the full-suite `test` job (42 min, ubuntu), `linux-process-lifetime` (which runs `tests/shared-host-lifecycle.test.ts`), all four `docker-e2e` shards and `secret-scan`. - **Full suite on the M3: not run.** I queued a runner for `1345ce5a` at 20:04. It would take the shared CI lock and start only with at least 60 GiB free. It waited about 52 minutes. During that time, someone else had held the lock directory since 17:39, two other units were queued ahead of mine, and free space was 55.1 to 56.6 GiB, under the 58 GiB floor. I stopped my runner before it took the lock. It created no worktree, ref or lock, and I did not touch the existing lock. ## Gaps - **No M3 full suite.** The shared lock was blocked and free space was under the floor (see Evidence). The hosted full suite ran on ubuntu, not on macOS. The focused tests ran on macOS. - **Search limit is unchanged.** Doctor does not search past the selector's 64 parent directories. An adopted root that is more than 64 levels above the start directory falls through to ordinary discovery, which refuses it with the existing closed native-input error. - **Other commands still throw.** The adopted-command selector has the same throw for other commands, for example the native Compose command path and `native-compose-open`. This PR does not change them; it is reported as a finding. - **No CLI or tooling regeneration.** There are no CLI surface, agent guidance or `package.json` script changes, so nothing needed to be regenerated or re-pinned. Refs HACK-1374 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
On Linux CI (
linux-process-lifetime, PR #381), the test "a starter lost after readiness leaves a claimed helper that another client reuses and releases" failed at the last assertion.releaseSharedHostHelper({ holder: "branch-b" })returned{ state: "stopped" }, and then the fixture'sisAlive(owned)returnedtrue.The product and the fixture used different liveness rules:
readCensus) already counts a zombie as exited. Release returnsstoppedonly after a census shows every tree member gone or a zombie.isAlivematched anypsrow with the same PID and birth time. That includes a zombie.In this test the starter is killed with SIGKILL, so the helper is orphaned. On Linux, init or a subreaper reaps it, and that can lag behind the stop. During that window the helper is a zombie. Release correctly reported
stopped, but the zombie-blind fixture still saw the helper as alive. On macOS, launchd reaps orphans almost at once, so the window is rarely seen. The same mismatch applies to the time-zone test, whose first starter also exits and orphans the helper.The only other way to get that result is
lstartdrift that flips back within milliseconds. A zombie is the simple explanation that fits. The new assertions would still fail, and show the problem, if a non-zombie row were ever left afterstopped.Contract (decided and documented)
stoppedmeans a census, taken after the signals, observed every member of the captured tree exited within one bounded, named wait:graceMs(SHARED_HOST_RELEASE_GRACE_MS= 2000 ms by default; a failed start uses 1000 ms).SHARED_HOST_KILL_CONFIRM_MS(1000 ms).stop-unconfirmed(the existing closed state for an uncertain stop), and the claim is kept.Exit is observed when the PID is gone or is a zombie. A PID that is still running never counts as exited, even with another birth time. Linux
lstartcan drift, so that row may be the same helper. Before this change such a row read as "exited", so a live process could be reportedstopped. A census that fails during the wait now keeps polling until the deadline instead of escalating to KILL early.Fix
src/lib/shared-host-lifecycle.tsisSharedHostProcessRunning: exact PID and birth, not a zombie. The product uses it internally.exitObservedfor the stop wait.awaitExit.tests/shared-host-lifecycle.test.tsisAlivenow uses the product predicate.isPresentandwaitReapedfor the one check that really is about reaping.waitGonenow means "exited".docs/reference/shared-host-lifecycle.md: new "Whatstoppedmeans" section, plus new fixture rows.Tests (each new assertion proven by mutation)
slow-termfixture), 10 s grace. Release returnsstoppedonly after the exit, the helper is not running at return, the TERM-path marker exists, and elapsed time is at least about 600 ms.stoppedright after TERMstoppedon poll 4, TERM only.stop-unconfirmed, TERM then KILL, claim kept with no holders.isAliveis false while the row is still a zombie. This is the deterministic local repro of the HACK-1336 failure mode.isAlive(the old helper)M6 (dropping the
!row.zombieclause insideexitObserved) is an equivalent mutant, becausereadCensusalready removes zombie rows. The clause is kept as defense in depth.Evidence
mise x bun@1.4.2),bun install --frozen-lockfile;bun.lockunchanged.bun test tests/shared-host-lifecycle.test.tspassed 5 out of 5 runs, 36 tests each, 0 failures. All mutations above were run locally.bun run check(includesprivacy:check: ok) andbun run --cwd packages/cli typecheck: exit 0. The Biome warnings shown are pre-existing in other files.origin/next..HEAD(patch export): no leaks.89b6221d(detached worktree, Bun 1.4.2): 7678 pass, 126 skip, 5 fail, 548 files, 3421 s.tests/shared-host-lifecycle.test.tspassed there. None of the 5 failures is in a file this PR changes, and the changed module is imported only by the shared-host tests and fixtures:repository catalog > the checked-in .hack/tasks.json recipes are all reviewed: fails in detached worktrees (HACK-1358). It passes in this checkout (18/0) and in the hostedtestjob.new no-host death uses real read-only quiescence facts…andnew no-host recovery retains original bindings on last-await(tests/native-authored-project-recovery.test.ts): these also fail when rerun serially with--timeout 60000on this Mac (16/2). The same two failures appear in full-suite logs for other branches, including a baseline run, so they predate this PR. Reported separately.original capture uses its live session…(native-authored-process-incarnation): rerun serially on this Mac, 10/0.legacy linked admission shares all six managed layers…(native-compose-adoption-env-inputs, 15 s timeout): rerun serially with--timeout 60000on this Mac, 35/0.89b6221d: all pass, includinglinux-process-lifetime(the step withtests/shared-host-lifecycle.test.ts: 67 pass, 0 fail) andtest.Gaps
linux-process-lifetimejob on this PR. One green hosted run does not prove the intermittent failure is gone. The deterministic proof is the zombie test plus mutation M3.lstartdrift outside a stop (a release that reads the claim as stale) is still the documented gap under "Not covered here". This PR only makes drift during a stop fail closed.package.jsonscripts changed, so there is nothing to regenerate or re-pin.Refs HACK-1336
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.