Skip to content

feat: remote verification orchestration and classified acceptance evidence - #292

Merged
turinglambdaai merged 3 commits into
mainfrom
feature/verify-remote-orchestration
Oct 19, 2026
Merged

turinglambdaai merged 3 commits into
mainfrom
feature/verify-remote-orchestration

Conversation

@turinglambdaai

Copy link
Copy Markdown
Owner

Closes #287 (the two items the matrix slice left open).

1. raco rivet verify-remote — orchestrate the remote legs from one editing host

raco rivet verify-remote --repo OWNER/REPO --ref <branch-or-sha> [--workflow verify.yml] [--platform linux]...

The command dispatches each platform's acceptance leg on GitHub Actions, waits for the dispatched runs, downloads the archived acceptance --json reports, and folds them into the shared matrix aggregation — so an agent editing on one OS receives the other platforms' evidence instead of assuming correctness from local results alone.

Design points:

  • It drives the gh CLI and nothing else. Defaults (verify.yml, acceptance-report-<platform>) encode the convention the product CI guide documents; --workflow/--artifact-prefix/--platform/--timeout-seconds/--output-directory adapt it.
  • Dispatch runs are identified against a pre-dispatch baseline (same workflow, same commit, unseen ids), so a concurrent session's runs can never be mistaken for this verification.
  • Runs are collected keying off artifact names, not job names — a product's matrix workflow may name its jobs however it likes, and per-platform collection still works.
  • A red run still hands over its archived evidence: the watch exit code only sets the conclusion, never skips the download, because the failing platform's report is exactly what the matrix exists to show.
  • Exit code is green only when the matrix passes, every requested platform delivered a report, and no collection failed. --json adds per-run conclusions and report paths to the matrix document.
  • Testable without a network: the entire gh boundary is injectable, and the tests script dispatch failures, delayed run lists, red legs, missing artifacts, timeouts, and argument validation against the real collection path.

The product CI guide (docs/ci.md, docs/ci.zh-CN.md) now archives every leg's report as acceptance-report-<platform> with if: always() — the exact convention the command consumes.

2. Classified evidence as first-class report fields

Manifest evidence entries accept a classified form alongside legacy strings:

(evidence . (#hasheq((path . ".rivet/acceptance/windows/ax.json")
                     (kind . accessibility-snapshot))
             #hasheq((path . ".rivet/acceptance/windows/screen.png")
                     (kind . screenshot))
             ".rivet/acceptance/windows/trace.json"))   ; generic

Kinds: accessibility-snapshot, screenshot, interaction-trace, log, generic (bare paths stay legal as generic). Accessibility snapshots are documented as the primary cross-platform record; screenshots are supplemental because native presentations are expected to differ.

The --json report now pins every declared entry's kind, presence, byte size, and SHA-256 — an archived report audits the artifacts it vouches for even when the files live in another platform's artifact store. missing-evidence stays for backward-compatible tooling.

Validation

  • raco test rivet/tests/: 10110 tests pass locally (verify-remote +16 acceptance cases + all prior suites).
  • raco rivet verify-remote --help dispatches through the lazy command loader; usage lists the command.

Finishes the two items #287 left open after the matrix landed.

raco rivet verify-remote dispatches the per-platform acceptance legs of one
commit on GitHub Actions, waits for the dispatched runs, downloads the
archived acceptance --json reports, and folds them into the shared
verification matrix. It drives the gh CLI and nothing else: workflow names
and artifact names stay the product's own conventions, with verify.yml and
acceptance-report-<platform> as the defaults the product CI guide now
documents. Runs are identified against a pre-dispatch baseline so a
concurrent session's runs cannot be mistaken for this verification, and a
red run still hands over its archived evidence because the watch exit code
only sets the conclusion, never skips the download. The command exits green
only when the matrix passes and every requested platform delivered a report.

Acceptance evidence entries are now classified — accessibility-snapshot,
screenshot, interaction-trace, log, or generic — with plain string paths
staying legal as generic. The report pins each declared artifact's kind,
presence, byte size, and SHA-256, so an archived report audits the
artifacts it vouches for even when the files are fetched from another
platform's artifact store.

Closes #287.
raco test runs with the test file's directory as the working directory, so
the default output directory sprouted rivet/tests/.rivet during CI and the
source-package smoke rightly rejected it. Every scenario now pins an
isolated output directory.
The output directory was created before dispatch, so a failed dispatch or
a discovery timeout left an empty .rivet/verify-remote behind in the
caller's working directory — which the source-package smoke rightly
rejects. Create it after the dispatched runs are confirmed.
@turinglambdaai
turinglambdaai force-pushed the feature/verify-remote-orchestration branch from 7f1b9db to 545c224 Compare October 19, 2026 17:56
@turinglambdaai
turinglambdaai merged commit ff3eb0e into main Oct 19, 2026
23 checks passed
@turinglambdaai
turinglambdaai deleted the feature/verify-remote-orchestration branch October 19, 2026 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

desktop: make feature parity and cross-platform agent verification first-class gates

1 participant