Repository navigation
test: keep the write guard, pid handshakes and store cases reliable under load (HACK-1439) - #496
Merged
Merged
Conversation
…nder load (HACK-1439) - checkout write guard: bound each git call (CHECKOUT_GIT_BUDGET_MS, killed and refused closed at the bound) and register the end-of-run afterAll with CHECKOUT_GUARD_HOOK_BUDGET_MS instead of Bun's 5 s default, which killed git mid-status and failed runs whose cases all passed. Guarded fixture keep-alive cases get a named budget too. - native-runtime-client: one bounded wait-for-complete-content helper backs waitForPid and waitForFileText; the cancellation case waits for a complete pid before it aborts (its fixture now publishes a torn handshake every run), and no case waits for existence and then reads. - private-env-file and native-compose-storage-witness-xattr-owner: a file-scoped named default budget replaces Bun's 5 s default and the fixed 30_000 literals; captured carriers outlive their case budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…load (HACK-1439) Four store cases (warm no-op up/restart, warm enrollment, adjacent verification) passed the 180 s budget in every run of a wave at a sustained load average of 110-190 on an 18-CPU host, while their assertions held; the slowest took 63 s at a load of about 90. Raise STORE_CASE_BUDGET_MS to 600 s: it is a hang guard for fixed store work, not a speed assertion. The fake carrier lifetime stays twice the budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keeps HACK-1440's store call counts in the xattr budget comment alongside the HACK-1439 load measurements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move waitForFileContent to tests/helpers/wait-for-file-content.ts and route the create-then-write readers found by verification through it: - native-compose-adoption-job-worktrees: the descendant pid was read after an existence wait, so an empty file parsed as 0. - native-compose-route-claims: the rollback barrier was read after the writer was killed, so a kill inside the empty-file window lost the reference for good. The reference is now taken before the kill, and the fixture holds that window open on every run. No assertion changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current state (head 4fced95)
Reworked after verification of 384bf68 failed. This head merges
origin/nextat ae032e7 and moveswaitForFileContenttotests/helpers/wait-for-file-content.ts. The two reads named in the verdict now use it: the descendant pid innative-compose-adoption-job-worktrees.test.ts, and the rollback barrier innative-compose-route-claims.test.ts. The barrier's reference is now taken before the writer is killed, and its fixture holds the empty-file window open on every run. No assertion changed.Verified locally on Bun 1.4.2 with a private HOME and HACK_HOME, at load averages of about 30-75 from other agents:
Unexpected end of JSON input); unmutated, that case passes 5 of 5. The adoption site, with a temporary torn fixture and a wider deadline, fails 4 of 5 on the> 0assertion and once on the deadline below; with the helper it passes 4 of 5, with one failure ondeadline expired.native-runtime-client,checkout-write-guard,private-env-file.native-compose-adoption-job-worktreespassed 2 of 3 andnative-compose-route-claimspassed 2 of 3. The failing cases are timing-bound under load; see the open problem below.bun run --cwd packages/cli typecheck, Biome on the changed files, and gitleaks over the new commits all pass.Open problem, not fixed here: "inherited-pipe descendants withhold cancellation settlement…" still fails under host load, with or without this change (paired runs at load 45-60: 25 of 30 before, 27 of 30 after; at load 10-16: 2 of 40 before, 3 of 40 after, on Bun 1.4.4). Its 250 ms capture deadline covers starting both processes, so a slow start gets the process group killed before the file is written or while it is still empty. Other cases in these two files also failed under load at their 1 s capture deadlines or Bun's 5 s default: "an exited leader…", "expired readiness…", and route-claims cases such as "explicit verified stop…". They were left alone rather than loosened.
Not re-run at this head:
native-compose-storage-witness-xattr-owner.test.ts, the full suite, and any load matrix. The tables below describe 384bf68 and earlier. Hosted CI at this head was not yet reported when this was written.Refs HACK-1439. Based on
nextafter #480 (HACK-1422), which touched three of these files, landed.nextwas merged in again at 148ef8f, which brought in #494 (HACK-1440) and #497.Test-only. No assertion was removed or weakened; the only changes are bounds on waits, named Bun timeouts, and fixture timing that makes a race happen on every run.
Root causes
installCheckoutWriteGuardregistered its end-of-runafterAllwithout a timeout, so it ran under Bun's 5 s default hook timeout. Under host load the synchronousgit status --untracked-files=allpassed 5 s. Bun then killed git mid-read ("killed 1 dangling process"), git returned non-zero, and the guard refused with "git status failed" on a run whose cases all passed.native-runtime-client.test.ts, the "cancellation starts at most one native request and reaps only its owned child" case waited only for the pid file to exist, then aborted.writeFileSynccreates the file before it writes the digits, so the abort could SIGKILL the child in between. The file then stayed""andNumber("")gave0.private-env-file.test.tsandnative-compose-storage-witness-xattr-owner.test.tsdo a fixed amount of store file work (env layer lock, staged write and rename; durable generation publish/enroll/verify/retire) and wait on no timer. Under load they passed Bun's 5 s default or the fixed30_000literals while their assertions still held.Fix
tests/helpers/checkout-write-guard.tsgitcall is bounded byCHECKOUT_GIT_BUDGET_MS(60 s,spawnSynctimeout+ SIGKILL).git status did not finish within N ms. A real failure still givesgit status failed. The guard never treats an unreadable checkout as clean.afterAllhook is registered withCHECKOUT_GUARD_HOOK_BUDGET_MS(status bound + 30 s). A slow status therefore ends as the guard's named refusal, not as a Bun kill.checkout-write-guard.test.ts: the guarded fixtures' keep-alive cases getKEEP_ALIVE_CASE_TIMEOUT_MS. The latespawnSyncblocks their event loop while a coldbunchild starts, and under load that passed 5 s. This was found while running the file under load.tests/native-runtime-client.test.tswaitForFileContenthelper now backswaitForPidandwaitForFileText(fix(env): retry a private env read that races an atomic replace instead of refusing it as a symlink #480 added the latter for thefinishedfile).waitForFileis removed. Each of its three call sites now waits for complete content: the cancellation case's pid before it aborts, the abort-mode keeper pid, and the late-writerfinishedtext.native-runtime-metadata,-not-running-cli,-preflight): none waits for a file.private-env-file.test.tsandnative-compose-storage-witness-xattr-owner.test.tssetDefaultTimeout:ENV_STORE_CASE_BUDGET_MS(60 s) and fix(env): retry a private env read that races an atomic replace instead of refusing it as a symlink #480'sSTORE_CASE_BUDGET_MS. Every case and hook gets a named budget.STORE_CASE_BUDGET_MSis raised from 180 s to 600 s. In one wave at a sustained load average of 110-190, every run failed the same four store cases (warm no-op up/restart, warm enrollment, adjacent verification) at 180 s while their assertions held. The slowest case had taken 63 s at a load of about 90. Those measurements predate perf(native-compose): cut redundant store rechecks per proof step #494, which cut the store's call counts. The budget is a hang guard, so it is sized with room to spare.30_000literals in the xattr file are removed so the named default applies.FAKE_CARRIER_LIFETIME_MS. A slow case can then only time out cleanly; it cannot fail because the carrier deadline expired. The one case that mocksDate.nowkeeps its own clock deadline.setDefaultTimeoutapplies to this file's cases,test.eachrows and hooks, and does not leak into a later file in the same run.Tests and mutation proofs
exec sleep 30, budget 200 ms.spawnSynctimeout, the guard returns[]after 30 s (no refusal): the test fails.timedOutbranch, the message is "git status failed": the test fails.killed 1 dangling process+checkout write guard: git status failed.waitForPid(pidFile)with an existence wait fails 3 of 3 runs withExpected: <pid> Received: 0, the reported failure. The fixed version passes 3 of 3.Evidence (local, macOS, 18 CPUs; ambient load from other agents, plus a
yesload generator stopped in a trap)Each run is a full
bun test <file>with a private HOME, HACK_HOME and DOCKER_CONFIG, and the checkout write guard preloaded. The load average is the 1-minute value, sampled as each run ended.At the exact PR head 384bf68:
yesloadprivate-env-file.test.tscheckout-write-guard.test.tsnative-runtime-client.test.tsnative-compose-storage-witness-xattr-owner.test.tsuptimeread 110-290 while the fifth wave ranEarlier rounds of the same change:
Gates:
bun run check(includes privacy:check) passes.bun run typecheckandbun run --cwd packages/cli typecheckreport 0 errors.ultracite checkon the five changed files is clean. gitleaks 8.30.1 overorigin/next..HEADfinds no leaks. The CLI surface and agent guidance did not change, so no generated docs needed regenerating.Full suite: hosted CI run 40143709633 passed at the exact PR head 384bf68, including
testshards 1-4. All 20 checks are green. The earlier head 7dc0359 was also fully green.Gaps
🤖 Generated with Claude Code