Skip to content

test: make process-timing tests tolerate host load - #358

Merged
roodboi merged 1 commit into
nextfrom
wf/g12-load-tolerant-tests-b17
Oct 20, 2026
Merged

roodboi merged 1 commit into
nextfrom
wf/g12-load-tolerant-tests-b17

Conversation

@roodboi

@roodboi roodboi commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Refs HACK-1280

What changed

Three tests failed now and then on loaded hosts and passed on rerun. Each one depended on a wall-clock guess. This PR changes only test files. No product timeout and no assertion is weakened.

  1. a real hanging process times out and its inherited-pipe descendant is killed (tests/native-compose-ownership.test.ts)
    • Cause: the probe timeout was 500 ms. A cold Bun start of the fake Docker can take longer than that on a loaded host, so the probe killed the process before it wrote its started/descendant pid file. The test then failed with ENOENT. A fixed Bun.sleep(50) also guessed how long reaping takes.
    • Fix: the fake Docker now writes its pid marker atomically (write to a temp file, then rename). The test checks that the process is dead only after this handshake. If the probe times out before the handshake, that attempt counts as inconclusive, never as a pass, and the test tries again with the next value in HANG_PROBE_TIMEOUTS_MS = [500, 2000, 8000]. Every attempt must still refuse with E_NATIVE_COMPOSE_PROBE_TIMEOUT. The fixed sleep is replaced by polling up to HANG_REAP_DEADLINE_MS. If no attempt reaches the handshake, the test fails with an explicit error.
  2. custom bridge driver, internal flag, aliases, generation and unexpected endpoint drift refuse (same file)
    • Cause: the test runs 9 scenarios in sequence, and each one starts several cold fake-Docker processes. Bun's default 5 s test budget ends up measuring how fast the host starts processes, not the product. A worse problem: when a test in this file runs past its budget, it keeps running and writes into the next test's shared root (fixture.json, commands). This caused cascading failures in tests that were otherwise healthy.
    • Fix: the file sets a named default, setDefaultTimeout(FAKE_DOCKER_TEST_TIMEOUT_MS) (60 s). It applies to this file only; I checked that Bun 1.4.2 scopes it to the file. Each probe keeps its own default or test-chosen timeout.
  3. initial forwarding preserves bytes/exits but records no unknown output, args or stderr (tests/native-process-policy-initial-trace.test.ts)
    • Cause: each forwarded command cold-starts two Bun processes, the TS trace forwarder and the fake Docker, under a 3 s exec bound inside a 5 s test.
    • Fix: named bounds FORWARDED_DOCKER_EXEC_TIMEOUT_MS (30 s) and INITIAL_FORWARDING_TEST_TIMEOUT_MS (150 s). These only stop a wedged child from hanging the run. Exit codes, bytes and trace order are still checked exactly.

Most of the second file's diff is the formatter re-indenting the test after the timeout argument was added. Review with git diff -w.

Evidence

Host: macOS, 18 cores, pinned Bun 1.4.2. Synthetic load: 36 busy-loop processes (2× cores). Each round runs N concurrent copies of the focused test (bun test <file> -t <name>) to mimic the parallel suite. Other agents were also using the host, so the 1-minute load average ranged from 20 to 110.

Test Before (origin/next copy) After: 20 consecutive rounds
hanging process / descendant (4 copies per round) 0/5 rounds; 9/20 copies passed; every failure was ENOENT … /started at ~525 ms 20/20 rounds, 80/80 copies (load average 81–88; another 80/80 on an earlier revision of this PR, where some rounds used the 8 s rung)
custom bridge refusals (4 copies per round) 0/5 rounds; 0/20 copies passed; all timed out at 5 s 20/20 rounds, 80/80 copies (load average 70–109)
initial forwarding (12 copies per round) 0/3 rounds; 9/36 copies passed; all failed the toEqual exit check at ~3.0 s (exec bound) 20/20 rounds, 240/240 copies (load average 57–87)

Whole tests/native-compose-ownership.test.ts, 4 concurrent copies under the same load:

  • before: 5–9 failures per copy (seven 5 s timeouts, plus cascade failures in empty fresh inventory…, owned resources yield…, fully completed real queries… and abort before spawn…)
  • after: 64/64 tests passed in every copy

Mutation checks (each test still fails when the product behaviour breaks)

Mutation Test Result
killProbe no longer SIGKILLs the owned process group hanging process fails (exitedWithin → false: the descendant survived)
probe timer stops the child but does not set timedOut hanging process fails (code is E_NATIVE_COMPOSE_PROBE, not …_TIMEOUT)
bridge-policy check ignores internal custom bridge refusals fails (unexpected probe success)
trace forwarder records unknown argv as a query initial forwarding fails (trace unavailable)
trace forwarder returns 0 instead of the child exit code initial forwarding fails (exitCode 17 → 0)

Required checks

  • bun run --cwd packages/cli typecheck: pass
  • bun run --cwd packages/cli check: 0 errors (34 existing warnings, none in the changed files); biome check on both changed files is clean
  • bun test tests/cli-spawn-env-isolation.test.ts: 7/7 pass
  • No Docker daemon, real HOME or keychain is used. All Docker interaction goes through the existing fake docker on a temporary PATH.

Remaining gaps

  • Two other tests in tests/native-process-policy-initial-trace.test.ts still fail under the same stress, with 8 concurrent copies: trace replay exposes startup vs strict ownership… and trace replay re-observes a restarting endpoint placeholder member…. Cause: tests/e2e/native-process-policy-initial-replay.ts gives its replay subprocess a 5000 ms exec bound (timeoutMs: opts.timeoutMs ?? 5000), and it reports replay unavailable. This is outside HACK-1280 scope and is left for a follow-up.
  • initial forwarding preserves a multicall Docker entry… and malformed UTF-8 keeps… in the same file still use 3000 ms bounds. They did not fail under this stress but have the same shape.
  • Hosted CI on f8cbd52c: test (Typecheck, Quality, Tests), all four docker-e2e shards, Candidate core (ubuntu), Config compiler (ubuntu x64/arm), Runtime state models, linux-process-lifetime, runtime-images and secret-scan all pass. The three macOS jobs (Candidate core, Config compiler latest/15-intel) were still queued for a runner when this was written.

🤖 Generated with Claude Code

Replace wall-clock guesses in three tests that failed intermittently on
loaded hosts (HACK-1280):

- The hanging-probe test now proves the kill only after the fake Docker
  publishes its pid through an atomic marker handshake. A probe timeout
  that fires before the handshake is inconclusive and escalates through
  named rungs (500 ms, 2 s, 8 s); every rung still requires the timeout
  refusal, and the reap check polls to a named deadline instead of a
  fixed 50 ms sleep.
- The native compose ownership file sets a named 60 s default test
  budget: its scenarios launch many cold fake-Docker processes, and an
  expired test kept writing into the next test's shared root.
- The initial forwarding trace test uses named bounds for each forwarded
  command (30 s) and for the test (150 s) instead of 3 s and 5 s.

No product timeout or assertion changes.

Refs HACK-1280

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Oct 19, 2026

Copy link
Copy Markdown

HACK-1280

@roodboi
roodboi marked this pull request as ready for review October 20, 2026 13:56
@roodboi
roodboi merged commit 0c12701 into next Oct 20, 2026
16 checks passed
@roodboi
roodboi deleted the wf/g12-load-tolerant-tests-b17 branch October 20, 2026 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant