[roodboi]

Root cause

The hosted test job ran every bun run test file (568 root tests/ files plus packages/db) in one Turbo task on blacksmith-6vcpu-macos-15. That step alone took about 39 min (next run 39873410893: Tests (Turbo) 23:42:18 to 00:21:20). Slower runners reached the 60 min job limit and were cancelled. Turbo buffers task output until the end, so a cancelled run did not show which file was running.

Fix

• scripts/test-shard-plan.ts finds the same file set that bun run test runs: root tests/ (with bunfig.toml root = "./tests" and its preload) and packages/db (from its own directory). It splits the files into shards, longest first. Durations come from scripts/test-shard-durations.json. That table was first built from the per-test times in the hosted log above. A file with no recorded duration is planned at 5 s, and each file adds 0.3 s for process start. The split is deterministic.
• scripts/test-shard.ts run --shard=i/4:
  • runs check:bun-version and test:preflight first, as bun run test does;
  • runs each file with bun test ./<file> in its own process, from the file's suite directory, so the bunfig preload and the db path aliases apply;
  • prints start and PASS/FAIL lines with the file's duration and the shard's elapsed time, and a heartbeat every 2 min while a file runs;
  • stops a file after 12 min, and stops the shard at 30 min or on SIGINT/SIGTERM (which is what a GitHub cancellation or timeout sends). It then prints the running file, the files not started and the 10 slowest files;
  • fails when a file fails, times out, or Bun does not report exactly 1 file run;
  • writes per-file durations, uploaded as the test-shard-durations-<i> artifact. update-durations merges those artifacts back into the table.
• Coverage check, run by every shard before it starts and by verify --count=4: the files found on disk must equal the files git ls-files lists. The union of all shards must equal that set, with no file in two shards. The set of workspaces with a test script, and the bunfig test root, must match the known suites.
• .github/workflows/ci.yml:
  • adds the test-shard matrix (test shard i/4, fail-fast: false, timeout-minutes: 40). It keeps the existing runner, Bun 1.4.2, frozen install, Rust 1.97.1 and the compiler build on every shard. Typecheck, privacy, lint, the CLI build and the release smoke run on shard 1 only.
  • test is now a fan-in job (needs: test-shard, if: always()). It passes only when every shard succeeded, so the required check name test does not change. No admin change to the ruleset is needed.
• scripts/prerelease-plan.ts CI_JOBS now also requires each test shard i/4 job, the same way it requires the docker-e2e shards.

Tests

tests/test-shard.test.ts (37 tests) covers:

• shard parsing and test-file name matching;
• partition determinism, exact cover, balance, the fallback duration, and zero-second files being spread across shards;
• the coverage check, with drop, duplicate and extra-file cases;
• fixture git checkouts for drift, ignored files and a missing suite root;
• the current checkout's 4-shard plan against git ls-files;
• the workspace and bunfig checks;
• the runner: per-file lines, file timeout, deadline, interrupt, the stop report, and the file-count rule;
• real bun test child processes: exit status, suite cwd, stopping a hung file, and not waiting for a leftover process that holds stderr open;
• the workflow wiring and the prerelease gate.

Mutation proofs (each mutation was applied, the focused file was run, then the file was restored). Each one made at least one test fail:

Mutation	Failing tests
partition drops the last file	7
no per-file start cost added	2
coverage ignores missing files	2
coverage ignores duplicate files	1
no disk-vs-git drift check	1
no workspace check	1
any file count counts as a pass	1
no file timeout	2
stop report omits the running file	1
test fan-in without always()	1
prerelease gate drops one shard	1
wait for stderr end of file	1
no SIGTERM on abort	1
wrong suite cwd	1

CLI mutation: when the partition drops a file, bun scripts/test-shard.ts verify --count=4 exits 2 with not run by any shard (1): tests/zig-aarch64-musl-linker.test.ts. Restored, it exits 0 with all files in 4 shards (571 files when first written, 573 at 503dc18d, 576 at 57736870; planned about 10m25s to 10m32s per shard).

Evidence

• Bun 1.4.2 (mise x bun@1.4.2), Rust 1.97.1.
• Local (loaded M5 Mac, private HOME/HACK_HOME/DOCKER_CONFIG, HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK=1):
  • bun run check (includes privacy:check) passed;
  • bun run --cwd packages/cli typecheck passed;
  • actionlint 1.7.12 found nothing on the 4 workflows;
  • focused tests passed: test-shard, prerelease-plan, e2e-shard, native-plan-cli-wiring, ci-bun-crash-census and agent-instruction-source (135 tests);
  • gitleaks 8.30.1 over origin/next..HEAD found no leaks.
• Full suite: hosted CI is the full-suite evidence for this unit, so no M3 run is needed. Each hosted run executes every test file across the 4 shards; the fan-in test check passes only when all 4 shards pass.
• Hosted shard runs: see "Hosted runs" below (3 runs, 2 of them green).

Hosted runs

Times are the Tests (shard) step; the whole job adds about 1 to 1.5 min of setup. Every shard job has timeout-minutes: 40.

Run	Head	Shard 1/4	Shard 2/4	Shard 3/4	Shard 4/4	Result
39888691204	a33511f4	11m48s (failed, test bug)	11m39s	10m32s	12m57s	red: one test, see below
39977193863	503dc18d	11m49s	11m30s	11m44s	12m16s	green, all 20 checks
40057462690	57736870	10m12s	10m39s	9m49s	11m29s	green, all 20 checks

Whole shard jobs took 10m34s to 13m54s across the 3 runs, well under the 40 min timeout. For comparison, next's serial test job (39916341281, 7d9c89f8) took 40m17s, with Tests (Turbo) at 39m21s, and run 39783802503 was cancelled at 60m25s.

Same file set as the serial job: the shard logs of run 39977193863 list 573 unique PASS files. next's serial log (39916341281) lists 572 files; the extra one is the new tests/test-shard.test.ts. Unique skipped tests are the same 157 names on both. Passed tests are 8057 on next and 8094 on the PR; the 37 extra are the new test-shard tests. Running each file in its own process did not skip anything new.

Why run 39888691204 was red

Shard 1 failed one test in tests/project-exec-command.test.ts: "exec warns value-free on stderr only after the service env changed since start". The clack CoreDNS/Caddy warnings went to stdout instead of stderr.

This was a test that depended on file order, not a sharding bug and not a known flake. In the old single bun test process, an earlier file had already set HACK_LOGGER=console. Run on its own, on a host without gum, the logger falls back to clack, which writes to stdout. Repro on origin/next, running only that file with gum removed from PATH: 4 pass, 1 fail, the same test.

Fix (503dc18d, test: pin the console logger in the exec env-change test): the test file sets HACK_LOGGER=console in beforeAll and restores the previous value in afterAll. With the fix the file gives 5 pass on its own; with next's version of the file put back it gives 4 pass, 1 fail again. Run 39977193863 at 503dc18d is green.

Round 2 (verifier follow-up)

• The verifier found that this body still said PENDING for the hosted and full-suite evidence, quoted 571 files instead of the plan's count, and did not explain the red run. This body now lists the hosted runs, the red-run cause and fix, and drops the M3 item.
• File counts: the plan had 571 files when the CLI mutation above was first written, 573 at 503dc18d, and 576 after merging next at 5208ee54 (which adds test: fail and contain process-state leaks between test files #405, the global-state-guard preload, and fix(agents): name the permission fix and keep a value-free cause on input access refusals #452).
• New head 57736870: merge of next 5208ee54 into 503dc18d. The merge was clean; no PR file changed.
• Local gates at 57736870 (Bun 1.4.2, private HOME/HACK_HOME/DOCKER_CONFIG, HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK=1, a bogus DOCKER_HOST):
  • gitleaks 8.30.1 dir over the git log -p origin/next..HEAD export: no leaks;
  • bun run check (includes privacy:check): 2 of 2 tasks passed;
  • bun run --cwd packages/cli typecheck: exit 0;
  • focused tests test-shard, prerelease-plan and project-exec-command: 99 pass, 0 fail;
  • bun scripts/test-shard.ts verify --count=4: 576 files in 4 shards, each file exactly once, planned about 10m31s per shard. 7 files from next have no recorded duration yet and use the 5 s fallback.
• Hosted CI at 57736870: run 40057462690: all 20 checks green, including test shard 1/4 to 4/4 and the test fan-in. The shard logs list 576 unique PASS files and no FAIL line, which matches the local plan. Shard jobs took 10m34s to 12m11s.
• Required check: still test (ruleset on next). The fan-in job keeps that name, so no admin change is needed.

Round 3 (re-check)

• Remote head is still 57736870, and next is still 5208ee54, which is an ancestor of the head. No merge or new commit was needed, so nothing was pushed this round.
• Re-checked this body against GitHub: gh pr checks 442 shows all 20 checks green at 57736870. The Tests (shard) step times of run 40057462690 match the table (10m12s, 10m39s, 9m49s, 11m29s). Its shard logs list 576 unique PASS files and no FAIL line.
• bun scripts/test-shard.ts verify --count=4 at 57736870: 576 files in 4 shards, each exactly once, planned 10m31s to 10m32s per shard.
• The Tests section above now gives the file count at each head instead of only the first count (571).

Gaps

• Each file now runs in its own process instead of sharing one bun test process. Files no longer share module state or global hooks across files. That is stricter isolation, so a test that depended on another file running first in the same process would now fail. The red run 39888691204 was such a case: a test passed in the shared process only because an earlier file had set HACK_LOGGER=console. It is fixed in 503dc18d (see "Why run 39888691204 was red"). The green hosted runs are the evidence for the current tree.
• scripts/ci-bun-crash-census.ts matches one exact job name, and its default is test. For runs after this change the Bun logs are in the test shard i/4 jobs, so the census needs --job "test shard i/4" once per shard. This is documented in the verify skill. The script itself is not changed here.
• After a stop, a child that ignores SIGTERM is SIGKILLed after 10 s. Its own descendants are not process-group killed, because changing process groups could change the lifecycle tests.

Refs HACK-1381

🤖 Generated with Claude Code

[linear-code]

HACK-1381