ci(verify): perf budget for export, seek and playback

[richiemcilroy]

Adds a Perf budget check: a pull request that makes export, seek or playback measurably slower than its base now fails.

How it measures

flowchart LR
    A[PR head] --> C[perf_build.py: engine suite binaries]
    B[merge base] --> D[perf_build.py: same, same machine]
    C --> E[perf_ab.py: interleaved pairs]
    D --> E
    E --> F{median per-pair CPU ratio more than 25% above base, slower in every pair?}
    F -->|yes| G[fail]
    F -->|no| H[pass]

Loading

• Both builds run on one Blacksmith macOS 26 machine (blacksmith-6vcpu-macos-26) and share one target cache (target/perf-verify/, release without LTO).
• Each workload gets one untimed warm-up per side, then pairs that alternate which side runs first.
• The metric is the test process's CPU time (user + system, from wait4 rusage). These VMs have no hardware instruction counters, and CPU time ignores scheduling noise better than wall time. Wall time is reported and warns at 8%.
• If a case passes on one side and fails on the other, that workload is not timed. Engine verify reports the case failure.

Workload	Cases	Verdict
export-basic	export/basic/*	fails past +25%, warns past +8%
export-jitter-camera-40s	export/jitter-camera-40s/*	fails past +25%, warns past +8%
seek-storm	seek/*/storm30/*	fails past +25%, warns past +8%
playback-basic	playback/basic/*	reported only

When it runs

• Ready pull requests that change an engine crate or a high-risk region, the same plan.py risk scope as the Engine verify FULL tier. A change to the perf budget's own files also runs it.
• Draft pull requests and everything else get a green Perf budget verdict from a 2-vCPU Ubuntu job, with no macOS job.
• workflow_dispatch accepts any base ref.

Calibration

Head and base with identical engine code (this pull request changes only CI files), 3 pairs each:

Workload	Apple silicon	Blacksmith, run 38995088761	Blacksmith, run 38998485953	Blacksmith, run 39000364369
export-basic	-0.0%	+0.8%	-1.1%	-2.1%
export-jitter-camera-40s	+0.5%	-4.4%	-1.2%	+10.8%
seek-storm	-0.5%	+0.9%	-5.0%	outcomes differed
playback-basic	+0.6%	+15.5%	+8.7%	-0.6%

• A Blacksmith VM's speed drifts within one job: the same seek-storm code ran 10.1 to 13.2 s per run, and one job put identical export code +10.8% slower. The verdict uses the median of the per-pair CPU ratios, since the two runs of a pair share the machine's state. It fails only past +25%, well above any noise measured, so the check stops large slowdowns (a decode per frame, a quadratic loop) without failing on the machine. Changes between +8% and +25% are flagged as warnings in the summary. Real-time playback's CPU time varied by up to 15% for the same code, so it is only reported.
• A run where a test binary exits on a signal or reports no cases fails as "not measured". Before that fix, the first run passed with every binary aborting at launch (FFmpeg frameworks not found).
• Perf budget becomes a required check once a few runs have stayed green.

Confidence Score: 5/5

The PR appears safe to merge, with no new blocking issues found.

Summary

Adds a performance check that builds and times head and base on one macOS runner.

• Selected pull requests now get a performance check against their base.
• Export, seek, and playback runs now get paired performance measurements.
• The comparison now builds separate test binaries for each revision.

_{Reviews (4) · Last reviewed commit: "ci(verify): perf budget compares runs wh..." · Reviewed by Greptile}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
cap-status	Ready	Preview	Oct 17, 2026 5:28am UTC

[capy-ai]

Capy found 3 potential issues (3 medium).

View 1 other finding in Capy.

[capy-ai]

Capy found no issues.

View 1 other finding in Capy.

[capy-ai]

Capy found no issues.

[richiemcilroy]

Addressed in 1a18450:

1. Valid: the base worktree now gets the head's generated .cargo/config.toml (FFMPEG_DIR), so both sides link the same FFmpeg.
2. Already fixed in b1928a7: perf-build.py links target/Frameworks (where scripts/setup.js puts it) and fails on macOS if there is none.
3. Valid. b1928a7 already fails the check when any run exits on a signal or reports no cases. 1a18450 also compares the case outcomes of every run, warm-ups included, and skips the timing when any of them differ.
4. Valid: the workflow header is down to why head and base run on one machine. The comments left in the scripts record the measured noise floor and platform facts.
5. Valid: renamed to perf-ab.py and perf-build.py.

[richiemcilroy]

@greptileai review

[capy-ai]

Capy found no issues.

[capy-ai]

Capy found no issues.

[superagent-security]

Superagent found 1 security concern(s).

[capy-ai]

Capy found no issues.

[capy-ai]

Capy found no issues.

View 1 other finding in Capy.

[richiemcilroy]

@greptileai review

[capy-ai]

Capy found no issues.

[richiemcilroy]

@greptileai review