[denolfe]

Extends the PR bench job to gate first-frame mount time and per-step scroll latency, not just --render startup. Both harnesses now emit machine-readable JSON, and a new comparison mode diffs the PR against a baseline built from main.

The recent progressive-mount work moved the first-frame and scroll numbers, but CI still measured only --render startup. A regression in mount or scroll latency could merge unnoticed. This wires the existing bench/first-frame.tsx and bench/scroll.tsx harnesses into the same PR gate.

Key Changes

• Harness JSON output
  • first-frame.tsx and scroll.tsx gain --json, printing one metrics payload per doc.
  • first-frame.tsx --runs N relaunches itself N times and reports the p50 (median sample). Process startup is part of first-frame and can only be sampled across fresh processes.
• Metric comparison
  • bench-compare.ts gains compareMetrics and renderMetricTable. A --metrics CLI mode diffs two harness payloads and gates on the existing warn/fail ratios.
  • --append stacks the render, first-frame, and scroll tables into one sticky comment.
• CI wiring
  • main.yml generates large.md, then runs both harnesses on the PR and baseline builds. All three tables post to the PR.

Design Decisions

• Structured JSON diff over hyperfine for scroll. Scroll cost is per-step latency. Hyperfine wall time would be dominated by mount, so the harness emits its own p50 and the comparison diffs those numbers.
• Baseline runs the PR harness. The main worktree predates the --json flag, so CI copies the PR harness scripts into it. Both builds then run identical measurement code against different src.
• Soft gate. The new compare steps use continue-on-error, matching the job-level setting. A regression annotates the sticky comment but never blocks merge.

Overall Flow

Previously the bench job ran a single hyperfine --render comparison. The flow now adds harness runs on both builds and a JSON diff:

flowchart TD
  A[Generate large.md] --> B[Copy PR harness into baseline worktree]
  B --> C[Run first-frame + scroll on PR build]
  B --> D[Run first-frame + scroll on baseline build]
  C --> E[Diff JSON payloads, apply warn/fail ratios]
  D --> E
  E --> F[Post render, first-frame, scroll tables]

Loading

References / Links

• Security hardening for GitHub Actions: confirms fork-PR token and secret isolation for the added untrusted-code execution.

[github-actions]

Startup benchmark (--render test/exhaustive.md, linux-x64)

build	mean	ratio	verdict
baseline (main@3b6d9f2)	380.7ms ± 7.1ms	—
PR	386.6ms ± 10.8ms	1.02×	✅ ok

Thresholds: warn ≥ 1.1×, fail ≥ 1.25×. Baseline built from main.

First-frame (large.md, runs=10)

metric	baseline (main@3b6d9f2)	PR	ratio	verdict
first-frame	290.6ms	299.4ms	1.03×	✅ ok

Thresholds: warn ≥ 1.1×, fail ≥ 1.25×. Baseline built from main.

Scroll (large.md)

metric	baseline (main@3b6d9f2)	PR	ratio	verdict
full_mount	12256.0ms	13035.0ms	1.06×	✅ ok
step_p50	28.3ms	30.5ms	1.08×	✅ ok
step_search_p50	30.0ms	29.6ms	0.99×	✅ ok
stepMatch_p50	22.8ms	23.6ms	1.03×	✅ ok
commit	152.6ms	155.6ms	1.02×	✅ ok

Thresholds: warn ≥ 1.1×, fail ≥ 1.25×. Baseline built from main.