Skip to content

fix: report a stopped shared host helper only after its exit is observed - #389

Merged
roodboi merged 1 commit into
nextfrom
wf/g34-shared-host-stopped-contract
Oct 22, 2026
Merged

roodboi merged 1 commit into
nextfrom
wf/g34-shared-host-stopped-contract

Conversation

@roodboi

@roodboi roodboi commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Root cause

On Linux CI (linux-process-lifetime, PR #381), the test "a starter lost after readiness leaves a claimed helper that another client reuses and releases" failed at the last assertion. releaseSharedHostHelper({ holder: "branch-b" }) returned { state: "stopped" }, and then the fixture's isAlive(owned) returned true.

The product and the fixture used different liveness rules:

  • The product (readCensus) already counts a zombie as exited. Release returns stopped only after a census shows every tree member gone or a zombie.
  • The fixture's isAlive matched any ps row with the same PID and birth time. That includes a zombie.

In this test the starter is killed with SIGKILL, so the helper is orphaned. On Linux, init or a subreaper reaps it, and that can lag behind the stop. During that window the helper is a zombie. Release correctly reported stopped, but the zombie-blind fixture still saw the helper as alive. On macOS, launchd reaps orphans almost at once, so the window is rarely seen. The same mismatch applies to the time-zone test, whose first starter also exits and orphans the helper.

The only other way to get that result is lstart drift that flips back within milliseconds. A zombie is the simple explanation that fits. The new assertions would still fail, and show the problem, if a non-zombie row were ever left after stopped.

Contract (decided and documented)

stopped means a census, taken after the signals, observed every member of the captured tree exited within one bounded, named wait:

  1. TERM, then poll for graceMs (SHARED_HOST_RELEASE_GRACE_MS = 2000 ms by default; a failed start uses 1000 ms).
  2. KILL to the members still running, then poll for up to SHARED_HOST_KILL_CONFIRM_MS (1000 ms).
  3. Otherwise the result is stop-unconfirmed (the existing closed state for an uncertain stop), and the claim is kept.

Exit is observed when the PID is gone or is a zombie. A PID that is still running never counts as exited, even with another birth time. Linux lstart can drift, so that row may be the same helper. Before this change such a row read as "exited", so a live process could be reported stopped. A census that fails during the wait now keeps polling until the deadline instead of escalating to KILL early.

Fix

  • src/lib/shared-host-lifecycle.ts
    • Exported one liveness predicate, isSharedHostProcessRunning: exact PID and birth, not a zombie. The product uses it internally.
    • Added exitObserved for the stop wait.
    • Named the waits, and folded the two poll loops into one bounded awaitExit.
  • tests/shared-host-lifecycle.test.ts
    • isAlive now uses the product predicate.
    • Added isPresent and waitReaped for the one check that really is about reaping. waitGone now means "exited".
  • docs/reference/shared-host-lifecycle.md: new "What stopped means" section, plus new fixture rows.

Tests (each new assertion proven by mutation)

New test / assertion Mutation Result
Helper ignores TERM for 600 ms (slow-term fixture), 10 s grace. Release returns stopped only after the exit, the helper is not running at return, the TERM-path marker exists, and elapsed time is at least about 600 ms. M1: return stopped right after TERM fails (plus the census test below)
Same M4: TERM grace forced to 0 (straight to KILL) fails (no marker)
Simulated census: helper runs for 3 polls after TERM, then shows as a zombie. Expect stopped on poll 4, TERM only. M1, M4 fail
Simulated census: PID keeps running with a drifted birth. Expect stop-unconfirmed, TERM then KILL, claim kept with no holders. M2: exit observed by birth match (the old rule) fails
Existing unreaped-zombie test now also asserts the fixture isAlive is false while the row is still a zombie. This is the deterministic local repro of the HACK-1336 failure mode. M3: zombie-blind fixture isAlive (the old helper) fails
Same M5: product predicate counts zombies as running fails

M6 (dropping the !row.zombie clause inside exitObserved) is an equivalent mutant, because readCensus already removes zombie rows. The clause is kept as defense in depth.

Evidence

  • Bun 1.4.2 (mise x bun@1.4.2), bun install --frozen-lockfile; bun.lock unchanged.
  • Local (macOS, loaded host, load average about 28 on 18 cores, plus 4 extra CPU spinners): bun test tests/shared-host-lifecycle.test.ts passed 5 out of 5 runs, 36 tests each, 0 failures. All mutations above were run locally.
  • Root bun run check (includes privacy:check: ok) and bun run --cwd packages/cli typecheck: exit 0. The Biome warnings shown are pre-existing in other files.
  • gitleaks 8.30.1 over origin/next..HEAD (patch export): no leaks.
  • Full suite on the M3 at the exact pushed commit 89b6221d (detached worktree, Bun 1.4.2): 7678 pass, 126 skip, 5 fail, 548 files, 3421 s. tests/shared-host-lifecycle.test.ts passed there. None of the 5 failures is in a file this PR changes, and the changed module is imported only by the shared-host tests and fixtures:
    • repository catalog > the checked-in .hack/tasks.json recipes are all reviewed: fails in detached worktrees (HACK-1358). It passes in this checkout (18/0) and in the hosted test job.
    • new no-host death uses real read-only quiescence facts… and new no-host recovery retains original bindings on last-await (tests/native-authored-project-recovery.test.ts): these also fail when rerun serially with --timeout 60000 on this Mac (16/2). The same two failures appear in full-suite logs for other branches, including a baseline run, so they predate this PR. Reported separately.
    • original capture uses its live session… (native-authored-process-incarnation): rerun serially on this Mac, 10/0.
    • legacy linked admission shares all six managed layers… (native-compose-adoption-env-inputs, 15 s timeout): rerun serially with --timeout 60000 on this Mac, 35/0.
  • Hosted checks on this PR at 89b6221d: all pass, including linux-process-lifetime (the step with tests/shared-host-lifecycle.test.ts: 67 pass, 0 fail) and test.

Gaps

  • The Linux zombie window cannot be forced on macOS, where launchd reaps orphans at once. The Linux receipt is the hosted linux-process-lifetime job on this PR. One green hosted run does not prove the intermittent failure is gone. The deterministic proof is the zombie test plus mutation M3.
  • lstart drift outside a stop (a release that reads the claim as stale) is still the documented gap under "Not covered here". This PR only makes drift during a stop fail closed.
  • The serial reruns of the M3 failures ran on this Mac, not on the M3. A queued M3 rerun waited almost three hours behind other units for the shared M3 lock, so I stopped it without touching the lock.
  • No CLI surface, agent guidance, or package.json scripts changed, so there is nothing to regenerate or re-pin.

Refs HACK-1336

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Release now reports `stopped` only after a census, taken after the
signals, shows every member of the captured tree exited: PID gone or a
zombie. The wait is bounded and named (TERM grace, then
SHARED_HOST_KILL_CONFIRM_MS after KILL). A PID still running, even with a
drifted birth time, is never an exit and ends as `stop-unconfirmed` with
the claim kept. A blind census mid-wait keeps polling instead of
escalating early.

The product and the fixtures now share one liveness predicate,
isSharedHostProcessRunning, so a Linux zombie waiting for a slow reaper
reads as not running everywhere. The zombie-blind fixture check caused
the Linux-only failure of the lost-starter release test.

Refs HACK-1336

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Oct 22, 2026

Copy link
Copy Markdown

HACK-1336

@roodboi
roodboi marked this pull request as ready for review October 22, 2026 23:41
@roodboi
roodboi merged commit 4e0bfa5 into next Oct 22, 2026
16 checks passed
@roodboi
roodboi deleted the wf/g34-shared-host-stopped-contract branch October 22, 2026 23:41
roodboi added a commit that referenced this pull request Oct 23, 2026
…ding; test shared host stop waits (#432)

## Summary

Follow-ups to #392 (doctor in adopted projects, symlinked `.hack`) and #389 (shared host stop
waits), merged into `next` at `4e0bfa55`.

- **Deep start directory.** `hack doctor` no longer throws `E_NATIVE_PROJECT_UNSUPPORTED` when
  it starts more than 64 directories deep. It now uses ordinary project discovery.
- **Adopted-project skips.** The 12 checks that doctor does not run for an adopted project now
  use doctor's skip wording. They keep status `ok`.
- **New tests.** A pending adoption, `doctor --fix` with a symlinked `.hack`, a census that
  fails
  during the stop wait, and the length of the KILL-confirm wait.
- **Docs.** The shared-host test row for a running PID with a new birth now describes what
  production does.

## Root cause

1. **Skip convention.** #392 reported the 12 checks as `ok` with `Not checked: …`. Every other
doctor skip uses `Skipped (…)`. 2. **Deep start directory.** `preflightDoctorProject` called
`selectLegacyComposeAdoptedRoot`    without a catch. With no `--project`, that selector walks up
at most 64 directories. If it    reaches no root in that walk, it throws `unsupported()`
(`E_NATIVE_PROJECT_UNSUPPORTED`). From    a start directory more than 64 levels deep, every
doctor run and every `--fix` step that looks    for an adoption threw. Ordinary discovery has no
such limit. 3. **Test gaps.** No test covered the pending-adoption warning or `--fix` in a
pending    adoption. There was a CLI regression for a symlinked `.hack/.internal`, but none for
a    symlinked `.hack`. In #389, mutants M7 (the old loop, which stops waiting on a failed
census)    and M10 (`awaitExit(0)` after KILL) survived. 4. **Docs.** The row "running PID with
a new birth" said that release sends KILL. That is true    only at the simulated signal
boundary. In production, `signalVerifiedProcessTree` checks the    birth time again before it
sends a signal, so a PID with a different birth time gets no    signal.

## Fix

- **Skip convention: keep `ok` and use the `Skipped (…)` wording.** The new message is `Skipped
  (adopted Compose project: its inputs belong to 'hack config adopt')`. Doctor already uses two
  kinds of skip:
  - `warn` when the skip points at something to fix, such as `Skipped (no .hack/ found)` or
    `Skipped (docker runtime unavailable)`;
  - `ok` when the skip is expected in a healthy setup, such as `Skipped (not a git checkout)`.

These 12 checks never apply to an adopted project, so `warn` would add the same 12 warnings to
  every healthy adoption in the summary and in `counts.warn`, and real warnings would be harder
  to
  see. Problems that belong to the adoption still show up in the `project` check (`warn` for a
  pending adoption) or in the adopted commands. The reasoning is in a comment in
  `src/lib/doctor-adopted-project.ts`, and in `docs/reference/native-compose-adoption.md`.
- **Deep start directory: a closed result.** `findDoctorAdoptedProject` catches only the
  selector's search-limit refusal and returns `null`, meaning no adoption was found. Ordinary
  discovery then runs. With no `--project`, `E_NATIVE_PROJECT_UNSUPPORTED` from this selector
  can
  only mean the search limit. Unsafe adoption state throws `E_CONFIG_INVALID`, and that still
  stops doctor. `preflightDoctorProject` moved from `src/commands/doctor.ts` to
  `src/lib/doctor-adopted-project.ts` so the tests can call it. Its behavior did not change.
- **Docs.** In `docs/reference/shared-host-lifecycle.md`, the row now says that release passes
  the
  member to the KILL step and waits the whole `SHARED_HOST_KILL_CONFIRM_MS`. The simulated
  boundary records the request, but in production no signal reaches that PID. There are new rows
  for the two new tests.

## Tests (every new test proven by mutation)

The private env is set by a runner script: `HOME`, `HACK_HOME` and `DOCKER_CONFIG` point to a
temp directory, `HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK=1`, and `DOCKER_HOST` points to a socket
that does not exist. `doctor-fix-noninteractive` runs without `HACK_HOME`, because its existing
daemon test finds `~/.hack` through its own private `HOME`. The pending-adoption tests use a new
fixture helper, `leaveRetainedSourceBindOperationPending`. It runs a retained `start` whose
callback reports failure, so the store keeps the pending journal. No engine command runs, and
nothing sleeps.

| Test | Mutation | Result |
| --- | --- | --- |
| `doctor-adopted-project`: the existing adopted `doctor --json` test now also checks the exact skip text and that there are 12 skipped checks | D1: status `warn` | killed |
| same | D2: the old `Not checked: …` text | killed |
| `doctor-adopted-project`: "a start directory deeper than the adoption search bound gets ordinary discovery instead of a throw". It uses 70 nested directories, first checks that the selector itself throws, then expects `findDoctorAdoptedProject`, `preflightDoctorProject` and the `--fix` plan to return without an error | D3: no catch (the #392 code) | killed: `E_NATIVE_PROJECT_UNSUPPORTED` |
| `doctor-adopted-project`: "unsafe adoption state still refuses doctor". The state root is a regular file, and the test expects `E_CONFIG_INVALID` with no canary text and no path in it | D4: catch every error | killed |
| `doctor-adopted-project`: pending adoption. `doctor --json` gives a `warn` `project` check with the exact message and `ok: true`. The 12 checks are skipped, the `--fix` plan is checked from the root and from a subdirectory, the adoption is still pending, the receipt is unchanged, and no container or project command runs | D5: pending reported like active | killed |
| same, plus `doctor-fix-adopted-project`: real `runCli doctor --fix --no-interactive --migrate-env-config` with a pending adoption and a committed `.hack/.env.state.json` (real `git init`). It checks exit 0, the plan line, the untrack note, the env-migration skip, the closing note, an empty `git ls-files`, no `.hack/.gitignore`, the adoption still pending, the receipt unchanged, and no engine command or project `run` call | D6: pending no longer selected | killed (both) |
| `doctor-fix-adopted-project` pending case | D6b: the untrack step treats a pending adoption as legacy | killed: exit 1 from the legacy writer's refusal |
| `doctor-fix-noninteractive`: "doctor --fix leaves an override behind a symlinked .hack untouched". The whole `.hack` links to an outside directory that holds a file with the override's name. The test expects that file unchanged, both directory listings unchanged, `.hack` still a link, and the notes do not mention the file | D7: the ownership check covers only `.internal` | killed: ENOENT (the outside file was deleted) |
| `shared-host-lifecycle`: "a census that fails during the TERM wait keeps the wait going". The first three polls fail and poll 4 shows the helper gone. The test expects `stopped`, 4 polls and TERM only | M7: the old loop | killed: `stop-unconfirmed` |
| `shared-host-lifecycle`: "after KILL, release keeps polling for the whole kill-confirm wait". A 100 ms grace; the helper leaves the census only 80% of the way into `SHARED_HOST_KILL_CONFIRM_MS` after KILL. The test expects `stopped`, TERM then KILL, and a return at or after 80% of that wait | M10: `awaitExit(0)` | killed |
| Existing "running with another birth time" test, which now also expects the return at or after `SHARED_HOST_KILL_CONFIRM_MS` after KILL | M10, and M10b: `awaitExit(SHARED_HOST_KILL_CONFIRM_MS / 2)` | both killed (both tests) |

For each mutation, the test was run alone and failed for the reason shown. The file was restored
afterwards.

## Evidence

- **Toolchain.** Bun 1.4.2 (`mise x bun@1.4.2`), `bun install --frozen-lockfile`. `bun.lock` is
  unchanged.
- **Local checks (M5):**
  - `bun run --cwd packages/cli typecheck`: 0 errors.
  - Root `bun run check`: exit 0, including `privacy-check: ok`.
  - Biome on the 7 changed source and test files: clean.
  - gitleaks 8.30.1, `gitleaks dir` over a `git log -p origin/next..HEAD` export: no leaks.
- **Focused tests:** 179 pass, 0 fail, across 10 files: `doctor-adopted-project`,
  `doctor-fix-adopted-project`, `shared-host-lifecycle`, `compose-env-at-rest`, `doctor-
  command`,
  `doctor-json`, `doctor-generated-files`, `doctor-env-key`,
  `native-compose-adoption-source-bind-generation` and
  `native-compose-adoption-source-bind-fixture`. `doctor-fix-noninteractive` passed 11 of 11.
  `shared-host-lifecycle` also passed 3 more times in a row, 38 of 38 each.
- **Hosted CI at `1345ce5a`:** all 16 checks pass (run 39876007802). These include the
  full-suite `test` job (42 min, ubuntu), `linux-process-lifetime` (which runs
  `tests/shared-host-lifecycle.test.ts`), all four `docker-e2e` shards and `secret-scan`.
- **Full suite on the M3: not run.** I queued a runner for `1345ce5a` at 20:04. It would take
  the shared CI lock and start only with at least 60 GiB free. It waited about 52 minutes.
  During that time, someone else had held the lock directory since 17:39, two other units were
  queued ahead of mine, and free space was 55.1 to 56.6 GiB, under the 58 GiB floor. I stopped
  my runner before it took the lock. It created no worktree, ref or lock, and I did not touch
  the existing lock.

## Gaps

- **No M3 full suite.** The shared lock was blocked and free space was under the floor (see
  Evidence). The hosted full suite ran on ubuntu, not on macOS. The focused tests ran on macOS.
- **Search limit is unchanged.** Doctor does not search past the selector's 64 parent
  directories. An adopted root that is more than 64 levels above the start directory falls
  through to ordinary discovery, which refuses it with the existing closed native-input error.
- **Other commands still throw.** The adopted-command selector has the same throw for other
  commands, for example the native Compose command path and `native-compose-open`. This PR does
  not change them; it is reported as a finding.
- **No CLI or tooling regeneration.** There are no CLI surface, agent guidance or `package.json`
  script changes, so nothing needed to be regenerated or re-pinned.

Refs HACK-1374

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant