Repository navigation
fix: fail local gates on a Bun pin mismatch and run native plan CLI checks locally - #381
Merged
Merged
Conversation
…hecks locally Root test, check and typecheck, and the CLI package's test, check and typecheck, now run `bun run check:bun-version` first. It fails when the running Bun differs from the package.json packageManager pin that CI installs, unless HACK_ALLOW_BUN_VERSION_MISMATCH=1 is set deliberately. Turbo passes that variable through so package tasks honor it. `bun run check:native-plan-cli` runs the compiled-bundle CLI checks that previously ran only in CI's Config compiler job. A wiring test fails when a scripts/check-native-*-plan-cli.ts is missing from it or from CI. Refs HACK-1290 Refs HACK-1294 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
The check and typecheck package scripts now run check:bun-version first, so the reviewed task catalog's pins for them were stale (stale_recipe). Pins are the canonical-JSON digests the catalog computes; test:models is unchanged. Refs HACK-1290, HACK-1157 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 22, 2026
roodboi
marked this pull request as ready for review
October 22, 2026 08:10
roodboi
added a commit
that referenced
this pull request
Oct 22, 2026
…ved (#389) ## Root cause On Linux CI (`linux-process-lifetime`, PR #381), the test "a starter lost after readiness leaves a claimed helper that another client reuses and releases" failed at the last assertion. `releaseSharedHostHelper({ holder: "branch-b" })` returned `{ state: "stopped" }`, and then the fixture's `isAlive(owned)` returned `true`. The product and the fixture used different liveness rules: - The product (`readCensus`) already counts a zombie as exited. Release returns `stopped` only after a census shows every tree member gone or a zombie. - The fixture's `isAlive` matched any `ps` row with the same PID and birth time. That includes a zombie. In this test the starter is killed with SIGKILL, so the helper is orphaned. On Linux, init or a subreaper reaps it, and that can lag behind the stop. During that window the helper is a zombie. Release correctly reported `stopped`, but the zombie-blind fixture still saw the helper as alive. On macOS, launchd reaps orphans almost at once, so the window is rarely seen. The same mismatch applies to the time-zone test, whose first starter also exits and orphans the helper. The only other way to get that result is `lstart` drift that flips back within milliseconds. A zombie is the simple explanation that fits. The new assertions would still fail, and show the problem, if a non-zombie row were ever left after `stopped`. ## Contract (decided and documented) `stopped` means a census, taken after the signals, observed every member of the captured tree exited within one bounded, named wait: 1. TERM, then poll for `graceMs` (`SHARED_HOST_RELEASE_GRACE_MS` = 2000 ms by default; a failed start uses 1000 ms). 2. KILL to the members still running, then poll for up to `SHARED_HOST_KILL_CONFIRM_MS` (1000 ms). 3. Otherwise the result is `stop-unconfirmed` (the existing closed state for an uncertain stop), and the claim is kept. Exit is observed when the PID is gone or is a zombie. A PID that is still running never counts as exited, even with another birth time. Linux `lstart` can drift, so that row may be the same helper. Before this change such a row read as "exited", so a live process could be reported `stopped`. A census that fails during the wait now keeps polling until the deadline instead of escalating to KILL early. ## Fix - `src/lib/shared-host-lifecycle.ts` - Exported one liveness predicate, `isSharedHostProcessRunning`: exact PID and birth, not a zombie. The product uses it internally. - Added `exitObserved` for the stop wait. - Named the waits, and folded the two poll loops into one bounded `awaitExit`. - `tests/shared-host-lifecycle.test.ts` - `isAlive` now uses the product predicate. - Added `isPresent` and `waitReaped` for the one check that really is about reaping. `waitGone` now means "exited". - `docs/reference/shared-host-lifecycle.md`: new "What `stopped` means" section, plus new fixture rows. ## Tests (each new assertion proven by mutation) | New test / assertion | Mutation | Result | | --- | --- | --- | | Helper ignores TERM for 600 ms (`slow-term` fixture), 10 s grace. Release returns `stopped` only after the exit, the helper is not running at return, the TERM-path marker exists, and elapsed time is at least about 600 ms. | M1: return `stopped` right after TERM | fails (plus the census test below) | | Same | M4: TERM grace forced to 0 (straight to KILL) | fails (no marker) | | Simulated census: helper runs for 3 polls after TERM, then shows as a zombie. Expect `stopped` on poll 4, TERM only. | M1, M4 | fail | | Simulated census: PID keeps running with a drifted birth. Expect `stop-unconfirmed`, TERM then KILL, claim kept with no holders. | M2: exit observed by birth match (the old rule) | fails | | Existing unreaped-zombie test now also asserts the fixture `isAlive` is false while the row is still a zombie. This is the deterministic local repro of the HACK-1336 failure mode. | M3: zombie-blind fixture `isAlive` (the old helper) | fails | | Same | M5: product predicate counts zombies as running | fails | M6 (dropping the `!row.zombie` clause inside `exitObserved`) is an equivalent mutant, because `readCensus` already removes zombie rows. The clause is kept as defense in depth. ## Evidence - Bun 1.4.2 (`mise x bun@1.4.2`), `bun install --frozen-lockfile`; `bun.lock` unchanged. - Local (macOS, loaded host, load average about 28 on 18 cores, plus 4 extra CPU spinners): `bun test tests/shared-host-lifecycle.test.ts` passed 5 out of 5 runs, 36 tests each, 0 failures. All mutations above were run locally. - Root `bun run check` (includes `privacy:check: ok`) and `bun run --cwd packages/cli typecheck`: exit 0. The Biome warnings shown are pre-existing in other files. - gitleaks 8.30.1 over `origin/next..HEAD` (patch export): no leaks. - Full suite on the M3 at the exact pushed commit `89b6221d` (detached worktree, Bun 1.4.2): 7678 pass, 126 skip, 5 fail, 548 files, 3421 s. `tests/shared-host-lifecycle.test.ts` passed there. None of the 5 failures is in a file this PR changes, and the changed module is imported only by the shared-host tests and fixtures: - `repository catalog > the checked-in .hack/tasks.json recipes are all reviewed`: fails in detached worktrees (HACK-1358). It passes in this checkout (18/0) and in the hosted `test` job. - `new no-host death uses real read-only quiescence facts…` and `new no-host recovery retains original bindings on last-await` (`tests/native-authored-project-recovery.test.ts`): these also fail when rerun serially with `--timeout 60000` on this Mac (16/2). The same two failures appear in full-suite logs for other branches, including a baseline run, so they predate this PR. Reported separately. - `original capture uses its live session…` (`native-authored-process-incarnation`): rerun serially on this Mac, 10/0. - `legacy linked admission shares all six managed layers…` (`native-compose-adoption-env- inputs`, 15 s timeout): rerun serially with `--timeout 60000` on this Mac, 35/0. - Hosted checks on this PR at `89b6221d`: all pass, including `linux-process-lifetime` (the step with `tests/shared-host-lifecycle.test.ts`: 67 pass, 0 fail) and `test`. ## Gaps - The Linux zombie window cannot be forced on macOS, where launchd reaps orphans at once. The Linux receipt is the hosted `linux-process-lifetime` job on this PR. One green hosted run does not prove the intermittent failure is gone. The deterministic proof is the zombie test plus mutation M3. - `lstart` drift outside a stop (a release that reads the claim as stale) is still the documented gap under "Not covered here". This PR only makes drift during a stop fail closed. - The serial reruns of the M3 failures ran on this Mac, not on the M3. A queued M3 rerun waited almost three hours behind other units for the shared M3 lock, so I stopped it without touching the lock. - No CLI surface, agent guidance, or `package.json` scripts changed, so there is nothing to regenerate or re-pin. Refs HACK-1336 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
package.jsonpinspackageManager: bun@1.4.2, andmise.toml,.tool-versions, every workflow and the runtime images use 1.4.2. Nothing checked the Bun that actually ran local gates, though. A Bun ahead of mise's shims onPATH(here mise'slatest1.4.3) skips the checkout'smise.toml. A Bun 1.4.2 macOS SIGSTOP difference made a test pass locally on 1.4.3 and then fail in CI.scripts/check-config-compiler-cli.ts,scripts/check-native-{env,host,routing,endpoint,process,acquisition}-plan-cli.ts,scripts/check-native-config-import-cli.ts) ran only inside CI's macOS "Config compiler" job. No local command ran them, so a behavior change passed every local gate and failed CI.Fix
scripts/check-bun-version.ts(bun run check:bun-version). It comparesBun.versionwith the exactpackageManagerpin. On a mismatch it exits 1 with the fix:mise x bun@<pin> -- bun run <script>. SettingHACK_ALLOW_BUN_VERSION_MISMATCH=1(exactly1) continues with a warning for a deliberate experiment.test,checkandtypecheckand inpackages/clitest,checkandtypecheck.turbo.jsonpasses the override through viaglobalPassThroughEnv, so package tasks honor it. CI already installs 1.4.2, and the.hacktoolchain image installs frommise.toml, so both pass unchanged.bun run check:native-plan-cli(scripts/check-native-plan-clis.ts). It runs the eight compiled-bundle checks one at a time, in CI order, and stops at the first failure. It refuses to start withoutdist/hackanddist/hack-config-compiler, and tells you how to build them.AGENTS.mdand.ai/skills/hack-repo-verify/SKILL.mddocument the guard, the override and the new command. These are contributor-only files. Consumer guidance (src/agents/instruction-source.ts) did not change, so no plugin regeneration or render fingerprint is needed.Tests
tests/bun-version-guard.test.tscovers four things. First, the match, mismatch and override verdicts (only"1"overrides) and the rejection of inexact pins. Second, the real script exits 1 on a mismatch and 0 with the override, run in a temp copy with an isolatedHOME. Third,mise.toml,.tool-versionsand every literal workflowbun-versionagree withpackageManager. Fourth, every entry point starts with the guard, and turbo passes the override through.tests/native-plan-cli-wiring.test.tsfails when anyscripts/check-native-*-plan-cli.ts, the config-import check or the config-compiler check is missing from the local command or from a CI step. It also checks that the runner refuses to start without the built bundle.1.test.typecheck.Evidence (macOS arm64, local)
bunis 1.4.3:bun run check:bun-versionexits 1 with the message.bun run teststops before turbo.HACK_ALLOW_BUN_VERSION_MISMATCH=1 bun run typecheck --forcewarns at the root and inside@hack/cli:typecheck, which proves the pass-through, and both tasks succeed.mise x bun@1.4.2 -- bun run check:bun-versionexits 0.bun run build,bun run build:config-compiler(rustc 1.97.1), thenmise x bun@1.4.2 -- bun run check:native-plan-cliall pass. All eight checks pass, including acquisition (198 checks) and config import (49 checks).bun run --cwd packages/cli typecheck: 0 errors.bun run --cwd packages/cli check: 0 errors (existing warnings only).bun run check: passes.bunx ultracite checkon the changed files: clean.bun run test(full suite, Bun 1.4.2, host load average 50 to 130 from parallel agents, 2h07m): the guard passed and turbo ran. 6785 tests passed and 1197 failed across 120 files. All 5 new tests passed in the suite.--timeout 60000. 99 files passed. 21 files still fail, allnative-compose-*/native-authored-*source-CLI fixtures. Most fail with the fixture's inner 15 sSIGKILL(exit 137) orprivate-uncertainrefusals.origin/next'spackage.json,packages/cli/package.jsonandturbo.jsonrestored on the same host gave identical results.native-compose-warm-up-command: same exit 137.native-compose-down-commandplusnative-compose-storage-fence-cost: 1 pass / 17 fail in both runs.bun testdoes not run any code this PR changes, so I count these failures as host and environment failures, not as caused by this change. I did not get a fully green local suite on this host.origin/next..HEAD: no leaks.Remaining gaps
bun test <paths>andbun index.tsdo not run the guard. Only thebun runentry points do; this is documented.check:native-plan-clicovers the CLI checks only. CI's "Config compiler" job also lints those scripts, runs their transport tests, runsbuild:releaseand validates the schema corpus in Python. The docs say to lint changed scripts explicitly.d9d6376d) is based onec223cdb, the commit I ran the full suite against.nexthas since gained four unrelated commits (feat(agents): read-only reviewed task catalog #353, fix(env): keep stored ciphertext when a secret value is unchanged #340, feat(runtime-core): admit stopped clean development owners within reserve #344, fix(native): keep storage carrier commands alive through host stalls #356). None of them touches these files, and the head merges cleanly (git merge-tree). The focused tests also pass after rebasing onto2f434217.tests/setup-migrate-command.test.tsresetsroot = ""inafterEach. A CLI call that outlives its timed-out test then passesroot: ""toprivateCliEnv. That resolves to the checkout, soisolatedDockerClientEnvwritesdocker-client/config.jsoninto the repository root. I saw this during the loaded full run.privateCliEnvshould refuse an empty or relativeroot.Refs HACK-1290
Refs HACK-1294
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.