Repository navigation
test: make process-timing tests tolerate host load - #358
Merged
Merged
Conversation
Replace wall-clock guesses in three tests that failed intermittently on loaded hosts (HACK-1280): - The hanging-probe test now proves the kill only after the fake Docker publishes its pid through an atomic marker handshake. A probe timeout that fires before the handshake is inconclusive and escalates through named rungs (500 ms, 2 s, 8 s); every rung still requires the timeout refusal, and the reap check polls to a named deadline instead of a fixed 50 ms sleep. - The native compose ownership file sets a named 60 s default test budget: its scenarios launch many cold fake-Docker processes, and an expired test kept writing into the next test's shared root. - The initial forwarding trace test uses named bounds for each forwarded command (30 s) and for the test (150 s) instead of 3 s and 5 s. No product timeout or assertion changes. Refs HACK-1280 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs HACK-1280
What changed
Three tests failed now and then on loaded hosts and passed on rerun. Each one depended on a wall-clock guess. This PR changes only test files. No product timeout and no assertion is weakened.
a real hanging process times out and its inherited-pipe descendant is killed(tests/native-compose-ownership.test.ts)started/descendantpid file. The test then failed withENOENT. A fixedBun.sleep(50)also guessed how long reaping takes.HANG_PROBE_TIMEOUTS_MS = [500, 2000, 8000]. Every attempt must still refuse withE_NATIVE_COMPOSE_PROBE_TIMEOUT. The fixed sleep is replaced by polling up toHANG_REAP_DEADLINE_MS. If no attempt reaches the handshake, the test fails with an explicit error.custom bridge driver, internal flag, aliases, generation and unexpected endpoint drift refuse(same file)root(fixture.json,commands). This caused cascading failures in tests that were otherwise healthy.setDefaultTimeout(FAKE_DOCKER_TEST_TIMEOUT_MS)(60 s). It applies to this file only; I checked that Bun 1.4.2 scopes it to the file. Each probe keeps its own default or test-chosen timeout.initial forwarding preserves bytes/exits but records no unknown output, args or stderr(tests/native-process-policy-initial-trace.test.ts)execbound inside a 5 s test.FORWARDED_DOCKER_EXEC_TIMEOUT_MS(30 s) andINITIAL_FORWARDING_TEST_TIMEOUT_MS(150 s). These only stop a wedged child from hanging the run. Exit codes, bytes and trace order are still checked exactly.Most of the second file's diff is the formatter re-indenting the test after the timeout argument was added. Review with
git diff -w.Evidence
Host: macOS, 18 cores, pinned Bun 1.4.2. Synthetic load: 36 busy-loop processes (2× cores). Each round runs N concurrent copies of the focused test (
bun test <file> -t <name>) to mimic the parallel suite. Other agents were also using the host, so the 1-minute load average ranged from 20 to 110.origin/nextcopy)ENOENT … /startedat ~525 mstoEqualexit check at ~3.0 s (exec bound)Whole
tests/native-compose-ownership.test.ts, 4 concurrent copies under the same load:empty fresh inventory…,owned resources yield…,fully completed real queries…andabort before spawn…)Mutation checks (each test still fails when the product behaviour breaks)
killProbeno longer SIGKILLs the owned process groupexitedWithin→ false: the descendant survived)timedOutE_NATIVE_COMPOSE_PROBE, not…_TIMEOUT)internalunexpected probe success)trace unavailable)exitCode17 → 0)Required checks
bun run --cwd packages/cli typecheck: passbun run --cwd packages/cli check: 0 errors (34 existing warnings, none in the changed files);biome checkon both changed files is cleanbun test tests/cli-spawn-env-isolation.test.ts: 7/7 passdockeron a temporary PATH.Remaining gaps
tests/native-process-policy-initial-trace.test.tsstill fail under the same stress, with 8 concurrent copies:trace replay exposes startup vs strict ownership…andtrace replay re-observes a restarting endpoint placeholder member…. Cause:tests/e2e/native-process-policy-initial-replay.tsgives its replay subprocess a 5000 msexecbound (timeoutMs: opts.timeoutMs ?? 5000), and it reportsreplay unavailable. This is outside HACK-1280 scope and is left for a follow-up.initial forwarding preserves a multicall Docker entry…andmalformed UTF-8 keeps…in the same file still use 3000 ms bounds. They did not fail under this stress but have the same shape.f8cbd52c:test(Typecheck, Quality, Tests), all four docker-e2e shards, Candidate core (ubuntu), Config compiler (ubuntu x64/arm), Runtime state models, linux-process-lifetime, runtime-images and secret-scan all pass. The three macOS jobs (Candidate core, Config compiler latest/15-intel) were still queued for a runner when this was written.🤖 Generated with Claude Code