Skip to content

test: fix all 182 failures unmasked by #2172 + closeTab last-tab race#2230

Open
time-attack wants to merge 7 commits into
garrytan:mainfrom
time-attack:pr-2172-test-env
Open

test: fix all 182 failures unmasked by #2172 + closeTab last-tab race#2230
time-attack wants to merge 7 commits into
garrytan:mainfrom
time-attack:pr-2172-test-env

Conversation

@time-attack

Copy link
Copy Markdown

What this PR does

This is the follow-up to #2172. That PR stopped test teardowns from silently killing the bun test process partway through the run (exit 0, no summary, ~34 of 434 files executed). With the suite finally able to run to completion, it exposed 182 pre-existing test failures that the early exit had been masking — this PR fixes all of them, plus one real product bug the investigation uncovered.

This branch contains #2172 (merged locally at its head, e7c37d76) because the suite cannot run far enough to verify these fixes without it. Merge #2172 first; this PR's diff then reduces to its own five commits.

Before / after (full free suite, identical command)

main + #2172, before this PR with this PR
Result 6,109 pass / 184 fail 6,239 pass / 0 real fails / 168 skip
Files 434, full summary 433 (two stale files deleted), full summary

The two residual "fails" in the after-run summary are module-load artifacts (see "Not fixed here" below) that appear identically in the before-run — the before-run's summary said 184 while printing 182 real failure lines; same two phantoms.

The five commits

1. fix(browse) — closeTab race loses last-tab auto-create (product code, the only source change)

BrowserManager.closeTab checked tabId === this.activeTabId after await page.close(). But the page 'close' event handler (wired in wirePageEvents) can run during that await and reassign activeTabId first. When it won the race, the "closed the last tab → auto-create a blank one" branch never executed, leaving the manager with zero tabs and every subsequent command failing with No active page. Event-dispatch order varies across Playwright/Chromium versions and machines, which is why this raced only on some setups. Fix: capture wasActive before closing, and only reassign activeTabId if it no longer points at a live tab (respecting a valid choice the event handler already made).

2. test(sidebar) — 92 failures: tests for an architecture that no longer exists

The v1.14 sidebar rip (#1216) replaced the chat-queue path (sidebar-agent.ts, /sidebar-command, /sidebar-chat, /sidebar-agent/*) with the interactive claude PTY terminal, and deleted the main chat-queue test files — but missed these. They have been failing invisibly ever since.

  • Deleted sidebar-integration.test.ts — every test drives removed endpoints; its two "passing" tests only asserted 401s on paths that no longer exist. PTY-path integration coverage lives in terminal-agent-integration.test.ts.
  • Deleted security-sidepanel-dom.test.ts — drives the #security-banner chat-pane DOM that was ripped with the chat queue (sidepanel.js documents the SEC shield as "not driven yet"). Server-side security-stack coverage (content-security, security, adversarial suites) is untouched.
  • sidebar-ux.test.ts (1670 → ~960 lines) — removed blocks pinning the dead architecture (the rip itself is already pinned as absence tests in sidebar-tabs.test.ts); re-pinned invariants that survived in new form: shutdown now asserts identity-based killAgentByRecord (not pkill), cleanup prompt asserts PTY injection via gstackInjectToTerminal, CSSOM guard asserts the typed DOMException catch.
  • sidebar-tabs.test.ts — two static-grep anchors updated to current source.

3. test(browse) — 63 failures: bun-version-dependent hook semantics + hermeticity

The dominant root cause (55 failures in commands.test.ts): bun 1.2.x runs describe-scoped beforeAll hooks eagerly at file load, in file order; newer bun runs them lazily. Three beforeAll(goto …) navigation hooks clobbered each other on 1.2.x, tests ran against the wrong page, 5s bun timeouts left stale 30s Playwright waits whose delayed rejections got mis-attributed to later tests, and a mid-cascade restoreState failure killed the shared session for the rest of the file. Converted the three hooks to beforeEach (identical semantics on every bun version; the local fixture server makes it ~ms per test). This is why the file passed 243/243 on machines with newer bun but failed 55 deterministically on bun 1.2.19.

Also in this commit:

  • content-security.test.ts: the content-filter describe called clearContentFilters() and never restored the module-load-registered urlBlocklistFilter. The module instance is shared across every test file in the process, so security-integration.test.ts failed downstream. Added an afterAll restore; the downstream assertion (whose job is to catch exactly this unregistration) is untouched.
  • watchdog.test.ts: hardcoded ports 34901–34903 → BROWSE_PORT=0. The fixed ports collided (EADDRINUSE) with sibling worktree runs of the same file; the tests never dial the port.
  • gstack-update-check.test.ts: pin GSTACK_HOME/GSTACK_STATE_ROOT to the test's state dir so env leaked by other files can't redirect the script's config resolution.
  • terminal-agent.test.ts / terminal-agent-integration.test.ts: static-grep anchors re-pinned after the maybeSpawnPty / args-aware canDispatchOverTunnel refactors (the new anchors pin the equivalent, stronger invariants); the PTY round-trip test gets a feature-detected skipIf(!BUN_HAS_PTY) (bun 1.2.x Bun.spawn has no terminal option — probed, not version-sniffed); the Sec-WebSocket-Protocol success-path handshake is rewritten as a raw TCP handshake via node:net because fetch() with Upgrade headers never resolves for 101 responses on bun 1.2.x (same assertions, truer to the test's stated intent).
  • dual-listener.test.ts: pinned literal updated to the current args-aware tunnel-dispatch source.

4. test(hermetic) — 20 failures: an env-restore leak and machine-state assumptions

The best bug of the batch: gstack-memory-helpers.test.ts saved GSTACK_HOME/GBRAIN_HOME/HOME/PATH in beforeEach but restored in afterAll — so after test #1, the "saved" value was the previous test's temp dir, and the restore permanently leaked a deleted gstack-test-engine-* temp path into process.env for every file that ran after it. That single leak caused 11 failures in two unrelated files (gstack-config.test.ts, make-pdf format-gate.test.ts). Proven both directions: the exact failures reproduce by running the leaky file first pre-fix, and everything is green post-fix including a cold-daemon run. Fixed by capturing once at collection time (same pattern as brain-cache-roundtrip.test.ts), plus defense-in-depth env scrubbing in gstack-config.test.ts's runner.

Also: gbrain-detect-install.test.ts appends a bun-only shim dir to its deliberate SAFE_PATH (the script's #!/usr/bin/env -S bun run shebang needs bun, but ~/.bun/bin also carries a real gbrain on dev machines); gen-skill-docs / openclaw-native-skills strict-YAML tests gate on Bun.YAML availability (lands in bun 1.3; CI pins latest so coverage is preserved where enforced); codex-resume-flag-semantics upgrades its availability probe from which codex to codex --version exit-0 (a half-installed npm wrapper resolves but cannot execute).

5. test(design) — 7 failures: interface drift and a millisecond truncation

  • feedback-roundtrip.test.ts (0/6 passing, fails identically on unmodified main): the test passed the BrowserManager where handleWriteCommand/handleReadCommand now take a TabSession. The manager delegates enough of the session surface to limp through setup but lacks clearLoadedHtml, which goto calls — hence TypeError on every test. Now calls via bm.getActiveSession() exactly like the canonical caller in commands.test.ts. Product code was correct and is untouched.
  • variants-retry-after.test.ts: the test built its Retry-After HTTP-date from Date.now() + 3000, and toUTCString() truncates milliseconds — silently shaving up to 999ms off the encoded wait, failing whenever the wall clock was >~500ms into a second. A coin-flip test. The base timestamp is now aligned to a whole second; assertions unchanged.

Ground rules the fixes followed

  • No assertion was weakened to match broken output. Every fix is either a source fix, a corrected test recipe, a hermeticity repair, or a feature-detected skip for a capability genuinely absent on the machine (CI, pinned to bun-version: latest, still runs everything).
  • Test deletions are limited to files whose entire subject was deliberately removed from the product, with replacement coverage identified above.
  • No VERSION / CHANGELOG / package-version changes — versioning is deliberately left to a later /ship pass.

Verification

  • Full free suite (identical command, same machine): 184 fail → 0 real fails, full summary printed, exit code now honest.
  • Per-file standalone: every touched file green standalone (commands.test.ts 243/243 in 28.9s; whole browse-core cluster in one process: 449 pass / 1 skip / 0 fail).
  • The final full-suite run passed even while two runaway sibling checkouts were hammering the same browser test file at 100% CPU — the reworked hooks are robust under contention.
  • Paid evals (test:evals) were not run locally; the only product-code change is the closeTab fix, which CI's diff-based selection will cover.

Known issues deliberately not fixed here (pre-existing, identical before/after)

  • 83 module-load "errors" in test/skill-e2e-* / skill-llm-eval files on bun 1.2.x (the test(name, {timeout}, fn) options-object signature needs newer bun). These files are --ignored by the free-suite script and run in CI with latest bun; they still load when the free suite is invoked with positional dir filters.
  • 2 phantom summary "fails" from module-scope throws: test/gbrain-refresh-install-render.test.ts (source-anchor miss) and ios-qa/daemon/test/daemon-integration.test.ts (afterEach undefined at load on bun 1.2.x).
  • TUNNEL_PATHS in browse/src/server.ts still allowlists the removed /sidebar-chat route (requests 404; editing the tunnel security surface was out of scope), and package.json engines still claims bun >=1.0.0 while parts of the suite genuinely want ≥1.3.

🤖 Generated with Claude Code

sneakygriff and others added 7 commits July 4, 2026 08:16
Several browse and design test files armed `setTimeout(() => process.exit(0),
500)` inside afterAll as a workaround for a browser close() that can hang. That
assumes each test file runs in its own process. It doesn't: `bun test` runs
EVERY file in one shared process. The 500ms timer therefore fired partway
through a LATER test file and terminated the entire run with exit code 0 and no
summary. Because the exit code was 0, the suite could not gate — only a fraction
of the files ran, and every downstream failure was silently masked.

The fix removes the delayed process.exit from teardown and instead time-boxes
the resource cleanup: browser teardowns race `close()` against a short timeout
and abandon it if it hangs (the child is reaped at real process exit), and the
design daemon /shutdown test stubs `process.exit` around the shutdown timers so
they fire harmlessly. A new static-guard test, test/no-suicide-exit.test.ts,
scans every *.test.ts and fails if any file schedules a delayed process.exit,
so the pattern cannot silently return. test/user-slug-fallback.test.ts is made
deterministic via HOME isolation.

Verified: test/no-suicide-exit.test.ts + test/user-slug-fallback.test.ts pass
(15/15); browse/test/commands.test.ts passes standalone (243/243) and now runs
to a full summary; design/test/daemon.test.ts passes (34/34) with no early exit.

Honest note: with the suite now able to run to completion, it surfaces
pre-existing failures that the early exit had been hiding — e.g.
design/test/feedback-roundtrip.test.ts fails identically on unmodified
origin/main (a `session.clearLoadedHtml` stub gap), independent of this change.
These are not introduced here; this change only makes them visible so they can
be gated on and fixed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ins the race

The page 'close' event handler can run while page.close() is awaited and
reassign activeTabId before closeTab's post-close active check, so the
last-tab auto-create branch never ran and the manager was left with zero
tabs. Capture wasActive before close(); only reassign activeTabId when it
no longer points at a live tab. Event-dispatch order varies across
Playwright/Chromium versions, which is why this raced on some machines only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v1.14 sidebar rip (PR garrytan#1216) replaced the chat-queue path with the
claude PTY terminal but missed these test files; the teardown process.exit
bug then hid their failures until the suite could run to completion again.

- sidebar-integration.test.ts, security-sidepanel-dom.test.ts: deleted —
  every test drives removed endpoints (/sidebar-command, /sidebar-chat,
  /sidebar-agent/*) or the ripped #security-banner chat UI. PTY-path
  coverage lives in terminal-agent-integration.test.ts; the rip itself is
  pinned as absence tests in sidebar-tabs.test.ts.
- sidebar-ux.test.ts: removed stale blocks pinning the dead architecture;
  re-pinned invariants that survived in new form (identity-based
  killAgentByRecord, PTY cleanup-prompt injection, typed DOMException
  catch, lazy terminal spawn).
- sidebar-tabs.test.ts: two assertions updated to the current source shape.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ate restore

- commands.test.ts: describe-scoped beforeAll(goto) hooks become
  beforeEach — bun 1.2.x runs describe-scoped beforeAll eagerly at file
  start in file order, so the three navigations clobbered each other and
  cascaded into 55 wrong-page / dead-session failures; beforeEach is
  correctly scoped on all bun versions.
- content-security.test.ts: restore the module-load urlBlocklistFilter
  after the content-filter describe — the module instance is shared
  across files, and leaving the registry empty broke
  security-integration.test.ts downstream.
- watchdog.test.ts: hardcoded ports 34901-34903 → BROWSE_PORT=0 (random),
  ending EADDRINUSE collisions with sibling worktree runs.
- gstack-update-check.test.ts: pin GSTACK_HOME/GSTACK_STATE_ROOT to the
  test's state dir so leaked env from other files can't redirect config.
- terminal-agent(.integration).test.ts: re-pin static-grep anchors to the
  current maybeSpawnPty/canDispatchOverTunnel source; feature-detected
  skip for Bun.spawn PTY support (absent on bun 1.2.x); success-path
  WebSocket handshake via raw TCP (fetch() never resolves 101 upgrades
  on bun 1.2.x).
- dual-listener.test.ts: update pinned literal to the args-aware
  canDispatchOverTunnel(body?.command, body?.args) form.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ent capabilities

- gstack-memory-helpers.test.ts: env vars were saved in beforeEach but
  restored in afterAll, so the restore leaked the previous test's dead
  temp dir into GSTACK_HOME/GBRAIN_HOME/HOME/PATH for every file that ran
  later (broke gstack-config and make-pdf format-gate in full-suite runs).
  Capture once at collection time instead.
- gstack-config.test.ts: scrub inherited GSTACK_STATE_ROOT/GSTACK_HOME in
  run() unless a test passes them explicitly (defense in depth).
- gbrain-detect-install.test.ts: append a bun-only shim dir to SAFE_PATH —
  the script's '#!/usr/bin/env -S bun run' shebang needs bun on PATH, but
  ~/.bun/bin also carries a real gbrain on dev machines.
- gen-skill-docs / openclaw-native-skills: skipIf(!Bun.YAML) — parser
  lands in bun 1.3; CI pins bun latest so strict-YAML coverage remains.
- codex-resume-flag-semantics: availability probe upgraded from
  'which codex' to 'codex --version' exit 0 (half-installed wrapper
  resolved but could not execute).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…runcation

- feedback-roundtrip.test.ts: handleWrite/ReadCommand's third param is a
  TabSession, not the BrowserManager — the manager delegates enough of the
  session surface to limp through setup but lacks clearLoadedHtml, so
  goto threw on every test. Call via bm.getActiveSession() like
  commands.test.ts does.
- variants-retry-after.test.ts: the Retry-After HTTP-date was built from
  Date.now()+3000, and toUTCString() truncates milliseconds — shaving up
  to 999ms off the encoded wait and failing whenever the clock was late
  in a second. Align the base to a whole second; assertions unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@trunk-io

trunk-io Bot commented Jul 10, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants