perf: three-tier performance testing (PR-gated counts, nightly timings, local-only CPU/GPU)

[simion]

Implements README roadmap item 12 (performance benchmarks in CI), scoped by what a CI runner can actually measure honestly.

Research doc: docs/research/perf-ci.md.

The finding that shaped this

The roadmap item names four targets: idle CPU, cold start, main-thread jank, RSS growth. Three durations and a percentage. None gate cleanly on macos-14 (3 M1 cores, virtualised, no Metal Performance Shaders in the guest) when termic's real deltas look like performance.md's "hidden 0.23% CPU vs visible 0.33%".

I got Orca wrong twice before getting it right, both times by inferring instead of reading. Corrected picture, verified across all 28 of their workflows:

Where	What
Every PR (Linux)	counts and static facts: selector fan-out, max-lines ratchet, asset budget
Nightly (Linux, xvfb)	app-level latency budgets, enforced (the gate script chains the budget check and propagates its exit code)
Never in CI	bench:idle-cpu, bench:startup, bench:main-thread-jank — invoked by zero workflows

So the axis is what the metric is, not gated vs ungated. Their one runtime PR gate, check:zustand-selector-fanout, guards exactly termic's bear trap 5. The metrics they never run in CI are exactly the four our roadmap names.

Three tiers

Tier 0 — PR-gated (src/store/selectorFanout.test.ts). 500 mounted useTaskTabs subscribers, 1000 setSidebarWidth writes (a sidebar drag), assert zero snapshot invalidations. Models useSyncExternalStore exactly. Counts are machine-independent, so this holds on a 3-core VM.

Verified it can fail, rather than trusting it green:

Injected regression	Result
?? [] instead of EMPTY_TABS	identity test fails
selector derives a fresh array	500,000 invalidations vs 0

Selector bodies are exported from app.ts so the test measures the real selectors, not a copy that drifts. No behaviour change: the hooks already passed a fresh arrow per render, and useSyncExternalStore compares snapshots, never selector identity.

Tier 2 — nightly, ungated (perf/, .github/workflows/perf.yml). Cron 03:30 UTC plus workflow_dispatch, macos-14, never on PRs. Startup timing and RSS growth. Writes a $GITHUB_STEP_SUMMARY table so the numbers are readable without downloading anything, plus a 90-day JSON artifact so a metric can earn a threshold from data later.

Tier 3 — local only (bench/). Idle CPU, GPU, compositor. Promoted out of gitignored scratchpad, where it was one rm -rf from gone. It encodes seven measurement traps, each of which produces a plausible wrong number rather than an error, and it already refuted a claim that shipped into the 0.26.0 changelog.

make perf       # both sections, reported separately
make perf-ci    # nightly suite only

Idle CPU is deliberately excluded from the nightly, matching Orca's own practice.

Running it locally found five real bugs

The suite is only worth anything if it was actually executed, so I ran make perf on the Mac Studio three times. Everything below was found by running it, not by review:

1. Startup spec failed outright. WKWebView freezes rAF for an occluded window (documented in docs/automation.md, which I'd read and then ignored), and a WebdriverIO-driven window is always occluded. Fixed with a timeout fallback, and then properly by switching the primary metric to the Paint Timing API, which is engine-recorded and rAF-independent.
2. RSS summed 10 WebKit helpers across three unrelated apps, adding ~570 MiB of someone else's memory. Helpers are parented to launchd so ppid cannot attribute them; now takes the lowest pid above the app's, per helper type. 10 helpers → 3.
3. memory.growth came out at -355 MiB. A fixed 5s baseline caught the startup peak and then RSS decayed, so it measured settling rather than leaking. This is GH macOS: expose the terminalGpuEnabled toggle on Mac — WebGL renderer costs ~33pp of WindowServer CPU and ~44pp of GPU on macOS 26 #140 trap 6 ("poll until quiet instead") which I had documented in bench/README.md and then walked straight into. Now polls until RSS is stable, and labels the row when it never settles.
4. A Section 1 failure aborted make perf before Section 2 ran, hiding half the report over an unrelated failure. Status is now captured and re-raised at the end.
5. perf/ was in no tsconfig, so none of it was typechecked. Added to e2e/tsconfig.json.

One thing I want to flag as not a bug, because I initially wrote a code comment claiming it was: the ~1.6 GiB app RSS is real and reproducible for the debug build, not a misattribution. The anchored pgrep is defensive hardening; measured on a real run the unanchored pattern matched exactly one process too. Comment corrected.

Answered: WebGL on the runner

webglRenderer: Apple GPU / hardwareWebgl: looks hardware-backed, locally. That was the open question gating any future frame-timing work. The nightly records it every run, so we get the CI answer on the first scheduled execution and notice if it ever changes.

Deliberately not done

Tier 1 e2e invariants (hidden-pane canvas count, PTY event coalescing, sidebar render count) are still proposals in the research doc. No thresholds anywhere: a metric earns one after its spread is known, and there is no spread data yet.

Docs: perf/README.md, bench/README.md, docs/performance.md (new Measuring section), CLAUDE.md. Also fixes CLAUDE.md's stale "laptop-only, no CI" claim about the e2e suite, which has run on macos-14 for a while.

No CHANGELOG entry, per CLAUDE.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_01F2JzhUsFa9YLjBBvUJQGYj