ci(verify): nightly diff against the cap-v0.6.0 baseline

[richiemcilroy]

Adds a Baseline diff workflow. Every nightly Engine verify and real-capture run on main is now compared with the cap-v0.6.0 production baseline, case by case. The check fails when main is worse than 0.6.0 anywhere.

How it works

flowchart LR
    N[nightly run on main: Engine verify or real capture] -->|workflow_run| D[Baseline diff]
    B[latest run of the same workflow on scratch/prod-baseline] --> D
    D --> R{WORSE rows?}
    R -->|yes| F[fail, table in the summary]
    R -->|no| P[pass]
    W[weekly cron] --> RB[re-run the baseline workflows]

Loading

• scripts/verify/diff.py gains --fail-on-worse. It exits 1 when a lane has WORSE case rows, or junit rows that went from pass to fail or error. A unit test that main no longer has (renamed or removed with its code) is listed but not counted.
• scripts/verify/baseline-na.tsv lists cases that cannot exist on 0.6.0 (the captions suite).
• The job runs on a 2-vCPU Ubuntu runner and downloads only the JSON and junit reports; recordings are deleted.
• A weekly cron re-runs both baseline workflows on scratch/prod-baseline, so the baseline artifacts never expire.
• workflow_dispatch compares any main run, optionally against a chosen baseline run.

First result

Main at faf032527 (Engine verify run 38950431408) against baseline run 38900902858:

Lane	WORSE	BETTER	EQUAL
linux	3	133	90
macos	0	52	151
macos-vt	0	13	98
windows	0	49	116

The three Linux rows are FFmpeg same-clip cross-fades with stale frames (editing/export/basic/seed-5, editing/export/multi-segment/seed-7, editing/export/same-clip-transition/forward/ffmpeg). #2661 fixes all three.

Confidence Score: 5/5

The latest changes appear safe to merge.

Summary

Adds nightly comparisons against the cap-v0.6.0 baseline, with weekly baseline refreshes and report-only recording artifacts.

• Nightly verification on main is compared with the cap-v0.6.0 baseline.

_{Reviews (9) · Last reviewed commit: "ci(verify): baseline lane check lists di..." · Reviewed by Greptile}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
cap-status	Ready	Preview	Oct 17, 2026 11:52pm UTC

[capy-ai]

Capy found 2 potential issues (2 medium).

View 1 other finding in Capy.

[richiemcilroy]

Addressed in 6d549ae:

1. Valid: the weekly refresh now dispatches real capture as FULL, repeat=3, runners=both, the same shape as main's nightly run, so recording measures have enough samples on both sides.
2. Valid. The baseline run is the latest completed one that was not cancelled; baseline runs fail by design, since 0.6.0 has known bugs. --fail-on-worse now also fails when main has a lane with no baseline records, which catches a baseline that stopped before a runner reported. Cases that only main has stay listed and are not counted: new cases cannot exist on 0.6.0.
3. Not changing: verify-scratch holds recordings only for failing attempts unless keep_artifacts is set, and the frozen baseline branch's workflow cannot gain a report-only artifact without re-running it. The job deletes the recordings right after download.
4. Valid: the header now only says why the weekly refresh exists.

[richiemcilroy]

@greptileai review

[superagent-security]

Superagent found 1 security concern(s).

[capy-ai]

Capy found no issues.

[richiemcilroy]

Addressed in 64f98b3: --fail-on-worse now also fails when more than a tenth of a lane's main cases have no baseline record, or when the lane has none. A baseline that stopped partway, or a baseline suite that has fallen behind main's, fails the check with a message to re-run or refresh it. New cases on main below that share are listed and not counted. On the recordings: as above, verify-scratch only holds failing attempts, and the job deletes them right after download.

[richiemcilroy]

@greptileai review

[capy-ai]

Capy found no issues.

[richiemcilroy]

Issue 1, in 0d7f07b: the baseline is now the latest completed run in which neither the run nor any job was cancelled or timed out, so every runner reported. Each lane still has to match at least nine tenths of main's cases. Failed baseline runs stay eligible, because 0.6.0 has known failures.

[richiemcilroy]

@greptileai review

[capy-ai]

Capy found no issues.

[richiemcilroy]

Both addressed in c2d41ba:

1. The baseline is picked only from runs that concluded success or failure (0.6.0 has known failures), and only when no job in the run was cancelled or timed out. --fail-on-worse then fails when a lane main has is missing from the baseline, or when more than a tenth of a lane's main cases have no baseline record.
2. Real capture now also uploads its reports alone as verify-recording-report-<runner>. The diff downloads those, falls back to the full artifact only for runs from before this change (deleting the recordings right away), and diff.py maps both names to the same lane. floor.py looks up verify-recording-<runner> by exact name, so the gate is unaffected.

[richiemcilroy]

@greptileai review

[capy-ai]

Capy found 1 potential issue (1 medium).

[richiemcilroy]

In 78d5720:

1. Valid. The job lists the run's artifacts first and downloads with one pattern: Engine verify reports, the new real-capture report artifacts, or, for runs from before them, the full real-capture artifacts. A run with none of these fails, and so does any download error; nothing is ignored.
2. --status completed is no longer in the workflow. Since 0d7f07b the baseline is the latest run that concluded success or failure, with no cancelled or timed-out job. --fail-on-worse then fails when one of main's lanes is missing from the baseline, or when more than a tenth of a lane's main cases have no baseline record.

[richiemcilroy]

@greptileai review

[capy-ai]

Capy found no issues.

[richiemcilroy]

In da45510 the workflow also checks this before diffing: every lane main reported (artifact names with tier and shard dropped, as diff.py names lanes) must have baseline reports, or the job fails and names the missing lanes. The line Greptile cites (--status completed) has not been in the workflow since 0d7f07b.

[richiemcilroy]

@greptileai review

[capy-ai]

Capy found no issues.

[richiemcilroy]

@greptileai review

[capy-ai]

Capy found no issues.

[richiemcilroy]

Dry run of this workflow's diff job, its run steps executed as written against real runs, in 6d4c54c:

• Engine verify nightly 38999256504 vs baseline 38900902858: lanes covered. The diff fails with 3 Linux WORSE rows (editing/export/basic/seed-5, multi-segment/seed-7, same-clip-transition/forward/ffmpeg). That nightly started 10 minutes before fix(rendering): stale frames in same-clip cross-fades and seeks with the FFmpeg decoder #2661, which fixes those three, merged.
• Real capture nightly 39000171559 vs baseline 38950453529: fails at "Baseline covers main's runners" with the baseline run has no reports for: verify-recording-macos-15 verify-recording-macos-26, because that baseline ran on Blacksmith only.

The dry run also showed that the lane step's find -printf failure inside <( ) did not fail the step. It now lists directories with a shell glob, captures both lane lists before comparing them, and fails when main has no report directories. I dispatched a full real-capture baseline (tier=full repeat=3 runners=both) so the first nightly diff after merge has every lane.

@greptileai review

[capy-ai]

Capy found no issues.