Skip to content

fix: fail local gates on a Bun pin mismatch and run native plan CLI checks locally - #381

Merged
roodboi merged 3 commits into
nextfrom
wf/g17-local-gates-b19
Oct 22, 2026
Merged

roodboi merged 3 commits into
nextfrom
wf/g17-local-gates-b19

Conversation

@roodboi

@roodboi roodboi commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Root cause

  • HACK-1290: package.json pins packageManager: bun@1.4.2, and mise.toml, .tool-versions, every workflow and the runtime images use 1.4.2. Nothing checked the Bun that actually ran local gates, though. A Bun ahead of mise's shims on PATH (here mise's latest 1.4.3) skips the checkout's mise.toml. A Bun 1.4.2 macOS SIGSTOP difference made a test pass locally on 1.4.3 and then fail in CI.
  • HACK-1294: the compiled-bundle CLI checks (scripts/check-config-compiler-cli.ts, scripts/check-native-{env,host,routing,endpoint,process,acquisition}-plan-cli.ts, scripts/check-native-config-import-cli.ts) ran only inside CI's macOS "Config compiler" job. No local command ran them, so a behavior change passed every local gate and failed CI.

Fix

  • New scripts/check-bun-version.ts (bun run check:bun-version). It compares Bun.version with the exact packageManager pin. On a mismatch it exits 1 with the fix: mise x bun@<pin> -- bun run <script>. Setting HACK_ALLOW_BUN_VERSION_MISMATCH=1 (exactly 1) continues with a warning for a deliberate experiment.
  • The guard runs first in root test, check and typecheck and in packages/cli test, check and typecheck. turbo.json passes the override through via globalPassThroughEnv, so package tasks honor it. CI already installs 1.4.2, and the .hack toolchain image installs from mise.toml, so both pass unchanged.
  • New bun run check:native-plan-cli (scripts/check-native-plan-clis.ts). It runs the eight compiled-bundle checks one at a time, in CI order, and stops at the first failure. It refuses to start without dist/hack and dist/hack-config-compiler, and tells you how to build them.
  • AGENTS.md and .ai/skills/hack-repo-verify/SKILL.md document the guard, the override and the new command. These are contributor-only files. Consumer guidance (src/agents/instruction-source.ts) did not change, so no plugin regeneration or render fingerprint is needed.

Tests

  • tests/bun-version-guard.test.ts covers four things. First, the match, mismatch and override verdicts (only "1" overrides) and the rejection of inexact pins. Second, the real script exits 1 on a mismatch and 0 with the override, run in a temp copy with an isolated HOME. Third, mise.toml, .tool-versions and every literal workflow bun-version agree with packageManager. Fourth, every entry point starts with the guard, and turbo passes the override through.
  • tests/native-plan-cli-wiring.test.ts fails when any scripts/check-native-*-plan-cli.ts, the config-import check or the config-compiler check is missing from the local command or from a CI step. It also checks that the runner refuses to start without the built bundle.
  • Mutation-proven: each of these 8 mutations made at least one test fail, and the restored code passes 8/8.
    • Always match.
    • Treat any truthy override as 1.
    • Drop the exit code on a mismatch.
    • Remove one check from the list.
    • Remove the guard from root test.
    • Remove the guard from CLI typecheck.
    • Remove the turbo pass-through.
    • Ignore missing prerequisites.

Evidence (macOS arm64, local)

  • Plain bun is 1.4.3:
    • bun run check:bun-version exits 1 with the message.
    • bun run test stops before turbo.
    • HACK_ALLOW_BUN_VERSION_MISMATCH=1 bun run typecheck --force warns at the root and inside @hack/cli:typecheck, which proves the pass-through, and both tasks succeed.
  • mise x bun@1.4.2 -- bun run check:bun-version exits 0.
  • bun run build, bun run build:config-compiler (rustc 1.97.1), then mise x bun@1.4.2 -- bun run check:native-plan-cli all pass. All eight checks pass, including acquisition (198 checks) and config import (49 checks).
  • Bun 1.4.2 gates:
    • bun run --cwd packages/cli typecheck: 0 errors.
    • bun run --cwd packages/cli check: 0 errors (existing warnings only).
    • bun run check: passes.
    • bunx ultracite check on the changed files: clean.
    • bun run test (full suite, Bun 1.4.2, host load average 50 to 130 from parallel agents, 2h07m): the guard passed and turbo ran. 6785 tests passed and 1197 failed across 120 files. All 5 new tests passed in the suite.
    • I reran each of the 120 failing files serially with --timeout 60000. 99 files passed. 21 files still fail, all native-compose-* / native-authored-* source-CLI fixtures. Most fail with the fixture's inner 15 s SIGKILL (exit 137) or private-uncertain refusals.
    • Baseline A/B with origin/next's package.json, packages/cli/package.json and turbo.json restored on the same host gave identical results. native-compose-warm-up-command: same exit 137. native-compose-down-command plus native-compose-storage-fence-cost: 1 pass / 17 fail in both runs.
    • Direct bun test does not run any code this PR changes, so I count these failures as host and environment failures, not as caused by this change. I did not get a fully green local suite on this host.
  • gitleaks 8.30.1 over origin/next..HEAD: no leaks.

Remaining gaps

Refs HACK-1290
Refs HACK-1294

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

…hecks locally

Root test, check and typecheck, and the CLI package's test, check and
typecheck, now run `bun run check:bun-version` first. It fails when the
running Bun differs from the package.json packageManager pin that CI
installs, unless HACK_ALLOW_BUN_VERSION_MISMATCH=1 is set deliberately.
Turbo passes that variable through so package tasks honor it.

`bun run check:native-plan-cli` runs the compiled-bundle CLI checks that
previously ran only in CI's Config compiler job. A wiring test fails when
a scripts/check-native-*-plan-cli.ts is missing from it or from CI.

Refs HACK-1290
Refs HACK-1294

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

HACK-1290

HACK-1294

@blacksmith-sh

This comment has been minimized.

roodboi and others added 2 commits October 21, 2026 20:17
The check and typecheck package scripts now run check:bun-version first, so the
reviewed task catalog's pins for them were stale (stale_recipe). Pins are the
canonical-JSON digests the catalog computes; test:models is unchanged.

Refs HACK-1290, HACK-1157

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@roodboi
roodboi marked this pull request as ready for review October 22, 2026 08:10
@roodboi
roodboi merged commit 7459df6 into next Oct 22, 2026
16 checks passed
@roodboi
roodboi deleted the wf/g17-local-gates-b19 branch October 22, 2026 08:10
roodboi added a commit that referenced this pull request Oct 22, 2026
…ved (#389)

## Root cause

On Linux CI (`linux-process-lifetime`, PR #381), the test "a starter lost after readiness leaves
a claimed helper that another client reuses and releases" failed at the last assertion.
`releaseSharedHostHelper({ holder: "branch-b" })` returned `{ state: "stopped" }`, and then the
fixture's `isAlive(owned)` returned `true`.

The product and the fixture used different liveness rules:

- The product (`readCensus`) already counts a zombie as exited. Release returns `stopped` only
  after a census shows every tree member gone or a zombie.
- The fixture's `isAlive` matched any `ps` row with the same PID and birth time. That includes a
  zombie.

In this test the starter is killed with SIGKILL, so the helper is orphaned. On Linux, init or a
subreaper reaps it, and that can lag behind the stop. During that window the helper is a zombie.
Release correctly reported `stopped`, but the zombie-blind fixture still saw the helper as
alive. On macOS, launchd reaps orphans almost at once, so the window is rarely seen. The same
mismatch applies to the time-zone test, whose first starter also exits and orphans the helper.

The only other way to get that result is `lstart` drift that flips back within milliseconds. A
zombie is the simple explanation that fits. The new assertions would still fail, and show the
problem, if a non-zombie row were ever left after `stopped`.

## Contract (decided and documented)

`stopped` means a census, taken after the signals, observed every member of the captured tree
exited within one bounded, named wait:

1. TERM, then poll for `graceMs` (`SHARED_HOST_RELEASE_GRACE_MS` = 2000 ms by default; a failed
start uses 1000 ms). 2. KILL to the members still running, then poll for up to
`SHARED_HOST_KILL_CONFIRM_MS` (1000 ms). 3. Otherwise the result is `stop-unconfirmed` (the
existing closed state for an uncertain stop), and the claim is kept.

Exit is observed when the PID is gone or is a zombie. A PID that is still running never counts
as exited, even with another birth time. Linux `lstart` can drift, so that row may be the same
helper. Before this change such a row read as "exited", so a live process could be reported
`stopped`. A census that fails during the wait now keeps polling until the deadline instead of
escalating to KILL early.

## Fix

- `src/lib/shared-host-lifecycle.ts`
  - Exported one liveness predicate, `isSharedHostProcessRunning`: exact PID and birth, not a
  zombie. The product uses it internally.
  - Added `exitObserved` for the stop wait.
  - Named the waits, and folded the two poll loops into one bounded `awaitExit`.
- `tests/shared-host-lifecycle.test.ts`
  - `isAlive` now uses the product predicate.
  - Added `isPresent` and `waitReaped` for the one check that really is about reaping.
  `waitGone` now means "exited".
- `docs/reference/shared-host-lifecycle.md`: new "What `stopped` means" section, plus new
  fixture rows.

## Tests (each new assertion proven by mutation)

| New test / assertion | Mutation | Result |
| --- | --- | --- |
| Helper ignores TERM for 600 ms (`slow-term` fixture), 10 s grace. Release returns `stopped` only after the exit, the helper is not running at return, the TERM-path marker exists, and elapsed time is at least about 600 ms. | M1: return `stopped` right after TERM | fails (plus the census test below) |
| Same | M4: TERM grace forced to 0 (straight to KILL) | fails (no marker) |
| Simulated census: helper runs for 3 polls after TERM, then shows as a zombie. Expect `stopped` on poll 4, TERM only. | M1, M4 | fail |
| Simulated census: PID keeps running with a drifted birth. Expect `stop-unconfirmed`, TERM then KILL, claim kept with no holders. | M2: exit observed by birth match (the old rule) | fails |
| Existing unreaped-zombie test now also asserts the fixture `isAlive` is false while the row is still a zombie. This is the deterministic local repro of the HACK-1336 failure mode. | M3: zombie-blind fixture `isAlive` (the old helper) | fails |
| Same | M5: product predicate counts zombies as running | fails |

M6 (dropping the `!row.zombie` clause inside `exitObserved`) is an equivalent mutant, because
`readCensus` already removes zombie rows. The clause is kept as defense in depth.

## Evidence

- Bun 1.4.2 (`mise x bun@1.4.2`), `bun install --frozen-lockfile`; `bun.lock` unchanged.
- Local (macOS, loaded host, load average about 28 on 18 cores, plus 4 extra CPU spinners): `bun
  test tests/shared-host-lifecycle.test.ts` passed 5 out of 5 runs, 36 tests each, 0 failures.
  All mutations above were run locally.
- Root `bun run check` (includes `privacy:check: ok`) and `bun run --cwd packages/cli
  typecheck`: exit 0. The Biome warnings shown are pre-existing in other files.
- gitleaks 8.30.1 over `origin/next..HEAD` (patch export): no leaks.
- Full suite on the M3 at the exact pushed commit `89b6221d` (detached worktree, Bun 1.4.2):
  7678 pass, 126 skip, 5 fail, 548 files, 3421 s. `tests/shared-host-lifecycle.test.ts` passed
  there. None of the 5 failures is in a file this PR changes, and the changed module is imported
  only by the shared-host tests and fixtures:
  - `repository catalog > the checked-in .hack/tasks.json recipes are all reviewed`: fails in
  detached worktrees (HACK-1358). It passes in this checkout (18/0) and in the hosted `test`
  job.
  - `new no-host death uses real read-only quiescence facts…` and `new no-host recovery retains
  original bindings on last-await` (`tests/native-authored-project-recovery.test.ts`): these
  also fail when rerun serially with `--timeout 60000` on this Mac (16/2). The same two failures
  appear in full-suite logs for other branches, including a baseline run, so they predate this
  PR. Reported separately.
  - `original capture uses its live session…` (`native-authored-process-incarnation`): rerun
  serially on this Mac, 10/0.
  - `legacy linked admission shares all six managed layers…` (`native-compose-adoption-env-
  inputs`, 15 s timeout): rerun serially with `--timeout 60000` on this Mac, 35/0.
- Hosted checks on this PR at `89b6221d`: all pass, including `linux-process-lifetime` (the step
  with `tests/shared-host-lifecycle.test.ts`: 67 pass, 0 fail) and `test`.

## Gaps

- The Linux zombie window cannot be forced on macOS, where launchd reaps orphans at once. The
  Linux receipt is the hosted `linux-process-lifetime` job on this PR. One green hosted run does
  not prove the intermittent failure is gone. The deterministic proof is the zombie test plus
  mutation M3.
- `lstart` drift outside a stop (a release that reads the claim as stale) is still the
  documented gap under "Not covered here". This PR only makes drift during a stop fail closed.
- The serial reruns of the M3 failures ran on this Mac, not on the M3. A queued M3 rerun waited
  almost three hours behind other units for the shared M3 lock, so I stopped it without touching
  the lock.
- No CLI surface, agent guidance, or `package.json` scripts changed, so there is nothing to
  regenerate or re-pin.

Refs HACK-1336

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant