From 211a873683e3b90115572ce5e2f381f71c027f8d Mon Sep 17 00:00:00 2001 From: jrtxio Date: Tue, 20 Oct 2026 01:02:26 +0800 Subject: [PATCH 1/3] feat: orchestrate remote verification and classify acceptance evidence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Finishes the two items #287 left open after the matrix landed. raco rivet verify-remote dispatches the per-platform acceptance legs of one commit on GitHub Actions, waits for the dispatched runs, downloads the archived acceptance --json reports, and folds them into the shared verification matrix. It drives the gh CLI and nothing else: workflow names and artifact names stay the product's own conventions, with verify.yml and acceptance-report- as the defaults the product CI guide now documents. Runs are identified against a pre-dispatch baseline so a concurrent session's runs cannot be mistaken for this verification, and a red run still hands over its archived evidence because the watch exit code only sets the conclusion, never skips the download. The command exits green only when the matrix passes and every requested platform delivered a report. Acceptance evidence entries are now classified — accessibility-snapshot, screenshot, interaction-trace, log, or generic — with plain string paths staying legal as generic. The report pins each declared artifact's kind, presence, byte size, and SHA-256, so an archived report audits the artifacts it vouches for even when the files are fetched from another platform's artifact store. Closes #287. --- AGENTS.md | 4 +- CHANGELOG.md | 13 ++ docs/ci.md | 31 ++- docs/ci.zh-CN.md | 31 ++- docs/product-acceptance.md | 47 +++- rivet-cli/acceptance.rkt | 108 +++++++-- rivet-cli/inspect.rkt | 3 + rivet-cli/main.rkt | 9 + rivet-cli/verify-remote.rkt | 412 ++++++++++++++++++++++++++++++++++ rivet/tests/acceptance.rkt | 111 ++++++++- rivet/tests/verify-remote.rkt | 258 +++++++++++++++++++++ 11 files changed, 1004 insertions(+), 23 deletions(-) create mode 100644 rivet-cli/verify-remote.rkt create mode 100644 rivet/tests/verify-remote.rkt diff --git a/AGENTS.md b/AGENTS.md index b859563b..695e9239 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -80,7 +80,9 @@ required acceptance suites pass on Windows, macOS, and Linux. The local operating system determines only which checks can run locally; it does not reduce the required verification scope. Archive every platform leg's `acceptance --json` report and prove the union with -`raco rivet matrix --json` (issue #287). Classify desktop +`raco rivet matrix --json`; `raco rivet verify-remote` +dispatches those legs and pulls their archived reports into the matrix from +this host when the platforms are not local (issue #287). Classify desktop capabilities as guaranteed, conditional, or platform-specific as documented in `docs/product-acceptance.md`. diff --git a/CHANGELOG.md b/CHANGELOG.md index afa9f485..f3136b28 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,19 @@ ## Unreleased +- Make verification evidence and orchestration first-class for distributed + agents (issue #287). `raco rivet verify-remote` dispatches the + per-platform acceptance legs of one commit on GitHub Actions, waits for the + runs, downloads the archived `acceptance --json` reports, and folds them + into the verification matrix, so an agent edits on one OS and is judged by + the others. Acceptance evidence entries are now classified — + `accessibility-snapshot`, `screenshot`, `interaction-trace`, `log`, or + `generic` — and the report pins each declared artifact's kind, presence, + byte size, and SHA-256, while plain string evidence paths stay legal as + `generic`. The product CI guide archives every leg's report as + `acceptance-report-` with `if: always()`, which is exactly the + convention the new command consumes. + - Add the crash-safe process-external Linux AppImage handoff (issue #290). `AppImageHandoff::LaunchReplacement` in the linked `rivet::platform` target writes a generated first-party shell script — the counterpart of the diff --git a/docs/ci.md b/docs/ci.md index a81ec5af..63389f13 100644 --- a/docs/ci.md +++ b/docs/ci.md @@ -60,7 +60,13 @@ jobs: - run: raco pkg install --auto --no-docs 'https://github.com/turinglambdaai/rivet.git#v0.12.0' - run: raco rivet doctor - run: raco rivet schema check rivet-schema.json - - run: raco rivet acceptance --json + - run: raco rivet acceptance --json > acceptance-report.json + shell: bash + - uses: actions/upload-artifact@v7 + if: always() + with: + name: acceptance-report-windows + path: acceptance-report.json - run: raco rivet build macos-host: @@ -80,7 +86,13 @@ jobs: - run: raco pkg install --auto --no-docs 'https://github.com/turinglambdaai/rivet.git#v0.12.0' - run: raco rivet doctor - run: raco rivet schema check rivet-schema.json - - run: raco rivet acceptance --json + - run: raco rivet acceptance --json > acceptance-report.json + shell: bash + - uses: actions/upload-artifact@v7 + if: always() + with: + name: acceptance-report-macos + path: acceptance-report.json - run: raco rivet build linux-host: @@ -97,10 +109,23 @@ jobs: run: raco pkg install --auto --no-docs 'https://github.com/turinglambdaai/rivet.git#v0.12.0' - run: raco rivet doctor - run: raco rivet schema check rivet-schema.json - - run: raco rivet acceptance --json + - run: raco rivet acceptance --json > acceptance-report.json + shell: bash + - uses: actions/upload-artifact@v7 + if: always() + with: + name: acceptance-report-linux + path: acceptance-report.json - run: raco rivet build ``` +Each job writes its `acceptance --json` report to a file and archives it as +`acceptance-report-` with `if: always()`, so a red leg still hands +over its evidence. Those names are exactly what +[`raco rivet verify-remote`](product-acceptance.md#orchestrate-remote-verification-from-one-host) +expects: from any editing host it dispatches the legs, waits for the runs, +downloads the archived reports, and folds them into the verification matrix. + The Linux order is load-bearing: install the Rivet package **after** `setup-embed-racket`. The action's minimal Racket build intentionally ships without packages; installing Rivet afterward resolves `cext-lib`, `crypto-lib`, diff --git a/docs/ci.zh-CN.md b/docs/ci.zh-CN.md index 1db6f68d..182d26b3 100644 --- a/docs/ci.zh-CN.md +++ b/docs/ci.zh-CN.md @@ -55,7 +55,13 @@ jobs: - run: raco pkg install --auto --no-docs 'https://github.com/turinglambdaai/rivet.git#v0.12.0' - run: raco rivet doctor - run: raco rivet schema check rivet-schema.json - - run: raco rivet acceptance --json + - run: raco rivet acceptance --json > acceptance-report.json + shell: bash + - uses: actions/upload-artifact@v7 + if: always() + with: + name: acceptance-report-windows + path: acceptance-report.json - run: raco rivet build macos-host: @@ -75,7 +81,13 @@ jobs: - run: raco pkg install --auto --no-docs 'https://github.com/turinglambdaai/rivet.git#v0.12.0' - run: raco rivet doctor - run: raco rivet schema check rivet-schema.json - - run: raco rivet acceptance --json + - run: raco rivet acceptance --json > acceptance-report.json + shell: bash + - uses: actions/upload-artifact@v7 + if: always() + with: + name: acceptance-report-macos + path: acceptance-report.json - run: raco rivet build linux-host: @@ -92,10 +104,23 @@ jobs: run: raco pkg install --auto --no-docs 'https://github.com/turinglambdaai/rivet.git#v0.12.0' - run: raco rivet doctor - run: raco rivet schema check rivet-schema.json - - run: raco rivet acceptance --json + - run: raco rivet acceptance --json > acceptance-report.json + shell: bash + - uses: actions/upload-artifact@v7 + if: always() + with: + name: acceptance-report-linux + path: acceptance-report.json - run: raco rivet build ``` +每个 job 都把 `acceptance --json` 报告写入文件，并以 +`acceptance-report-` 为名、`if: always()` 归档——失败的腿也要交出 +证据。这些名字正是 +[`raco rivet verify-remote`](product-acceptance.md#orchestrate-remote-verification-from-one-host) +的默认约定：在任一编辑主机上触发各平台腿、等待运行、下载归档报告并折叠成 +验证矩阵。 + Linux 中的顺序是必要条件：必须在 `setup-embed-racket` **之后**安装 Rivet。 该 action 准备的是刻意不附带任何包的 minimal Racket；此时再安装 Rivet， `cext-lib`、`crypto-lib`、`keel` 等完整依赖才会进入嵌入式 runtime 实际读取的 diff --git a/docs/product-acceptance.md b/docs/product-acceptance.md index b5a1ed30..d281368d 100644 --- a/docs/product-acceptance.md +++ b/docs/product-acceptance.md @@ -33,8 +33,12 @@ Commit the manifest at the project root: (platforms . (windows)) (command . ("pwsh" "-File" "scripts/e2e/windows.ps1")) (timeout-seconds . 900) - (evidence . (".rivet/acceptance/windows/trace.json" - ".rivet/acceptance/windows/accessibility.json")))))) + (evidence + . (#hasheq((path . ".rivet/acceptance/windows/accessibility.json") + (kind . accessibility-snapshot)) + #hasheq((path . ".rivet/acceptance/windows/screen.png") + (kind . screenshot)) + ".rivet/acceptance/windows/trace.json")))))) ``` The supported suite kinds are: @@ -56,6 +60,16 @@ suite starts and then requires the suite to recreate it, so a stale artifact from an earlier green run cannot satisfy the current run. `native-ui`, `package`, and `release` suites must declare at least one evidence file. +Evidence entries are classified so remote consumers know what a green suite +actually produced. A bare path string means the `generic` kind; a hash declares +`path` and one `kind` of `accessibility-snapshot`, `screenshot`, +`interaction-trace`, `log`, or `generic`. Accessibility snapshots are the +primary cross-platform record — semantic UI state that is comparable across +platforms — while screenshots are supplemental because native presentations +are expected to differ. The `--json` report pins every declared entry's kind, +presence, byte size, and SHA-256, so an archived report audits the artifacts +it vouches for. + `requirements` is the reviewable policy. For every kind/platform pair there must be at least one matching suite or the manifest is rejected before a child process starts. Product owners should protect changes to requirements, CI, @@ -148,6 +162,35 @@ only when every required kind/platform cell is proven. Feed the reports for the commit under verification; the matrix aggregates what it is given and does not heuristically prefer fresher runs. +## Orchestrate remote verification from one host + +`raco rivet verify-remote` closes the loop for an agent editing away from the +platform legs: it dispatches each platform's acceptance leg on GitHub Actions, +waits for the runs, downloads the archived `acceptance --json` reports, and +folds them into the matrix: + +```bash +raco rivet verify-remote --repo OWNER/REPO --ref +raco rivet verify-remote --repo OWNER/REPO --ref --platform linux +raco rivet verify-remote --repo OWNER/REPO --ref --json +``` + +The command drives the `gh` CLI and nothing else. The defaults encode one +convention — workflow `verify.yml`, artifacts named +`acceptance-report-` — and `--workflow`/`--artifact-prefix` adapt it +to a product's own naming. The workflow contract is: a `workflow_dispatch` +trigger with a `platform` input (`linux`, `windows`, or `macos`), one +`raco rivet acceptance --json` per leg, and the report uploaded with +`if: always()` so a red leg still hands over its evidence. Re-running only the +affected platform is just `--platform `; the final verdict still needs +every required platform's report, because missing platforms are never silently +green. + +The command exits green only when the matrix passes and every requested +platform delivered a report. Run collections land under +`.rivet/verify-remote/` (adjust with `--output-directory`), and `--json` adds +the per-run conclusions and report paths to the matrix document. + ## What counts as complete RackUnit is the normal owner of shared Racket behavior. A product that claims diff --git a/rivet-cli/acceptance.rkt b/rivet-cli/acceptance.rkt index b65bb2ed..fd4d1a62 100644 --- a/rivet-cli/acceptance.rkt +++ b/rivet-cli/acceptance.rkt @@ -13,19 +13,29 @@ racket/port racket/string racket/system + "../rivet/distribution/crypto.rkt" "project.rkt") (provide default-acceptance-manifest-name acceptance-kinds + evidence-kinds supported-platforms (struct-out acceptance-suite) load-acceptance-manifest acceptance-report + evidence-entries run-acceptance-command) (define default-acceptance-manifest-name "rivet-acceptance.rktd") (define acceptance-kinds '(unit contract bridge native-build native-ui package release)) +;; Normalized evidence classes (issue #287): accessibility snapshots are the +;; primary cross-platform acceptance evidence, screenshots are supplemental +;; because native UIs are expected to look different, and traces plus logs +;; carry the interaction and failure narrative. Plain string paths in a +;; manifest stay legal and mean 'generic. +(define evidence-kinds + '(accessibility-snapshot screenshot interaction-trace log generic)) (define supported-platforms '(windows macos linux)) (define maximum-captured-output-bytes (* 64 1024)) @@ -95,10 +105,64 @@ (string-ci=? (path->string (first parts)) ".rivet") (string-ci=? (path->string (second parts)) "acceptance"))))) -(define (valid-evidence? value) - (and (list? value) - (andmap acceptance-evidence-path-string? value) - (= (length value) (length (remove-duplicates value string=?))))) +;; Manifest evidence entries are either a bare path string (legacy shape, +;; normalized to the 'generic kind) or a hash with exactly `path` and `kind`. +;; Internally each entry is a (path . kind) pair. +(define (parse-evidence-entry path suite-id raw) + (cond + [(string? raw) + (unless (acceptance-evidence-path-string? raw) + (manifest-error path + "invalid suite evidence path" + "suite" suite-id + "expected" + "a unique file path below .rivet/acceptance/" + "value" raw)) + (cons raw 'generic)] + [(hash? raw) + (define path-value + (hash-ref raw 'path + (lambda () + (manifest-error path + "invalid suite evidence entry" + "suite" suite-id + "expected" "a hash with path and kind" + "value" raw)))) + (unless (acceptance-evidence-path-string? path-value) + (manifest-error path + "invalid suite evidence path" + "suite" suite-id + "expected" + "a unique file path below .rivet/acceptance/" + "value" path-value)) + (define kind-value + (hash-ref raw 'kind + (lambda () + (manifest-error path + "invalid suite evidence entry" + "suite" suite-id + "expected" "a hash with path and kind" + "value" raw)))) + (unless (memq kind-value evidence-kinds) + (manifest-error path + "invalid suite evidence kind" + "suite" suite-id + "expected" evidence-kinds + "value" kind-value)) + (unless (= (length (hash-keys raw)) 2) + (manifest-error path + "invalid suite evidence entry" + "suite" suite-id + "expected" "a hash with exactly path and kind" + "value" raw)) + (cons path-value kind-value)] + [else + (manifest-error path + "invalid suite evidence entry" + "suite" suite-id + "expected" + "a path string or a hash with path and kind" + "value" raw)])) (define (parse-suite path raw) (unless (hash? raw) @@ -134,14 +198,17 @@ "suite" id "expected" "integer from 1 through 7200" "value" timeout-seconds)) - (define evidence (hash-ref raw 'evidence '())) - (unless (valid-evidence? evidence) + (define evidence-raw (hash-ref raw 'evidence '())) + (define evidence + (map (lambda (entry) (parse-evidence-entry path id entry)) evidence-raw)) + (unless (= (length evidence) + (length (remove-duplicates evidence equal?))) (manifest-error path "invalid suite evidence" "suite" id "expected" "unique file paths below .rivet/acceptance/" - "value" evidence)) + "value" evidence-raw)) (when (and (memq kind '(native-ui package release)) (null? evidence)) (manifest-error path @@ -280,7 +347,8 @@ (define (prepare-evidence! project suite) ;; Evidence belongs under Rivet's generated root, so removing stale files is ;; both safe and necessary: an earlier green run must not satisfy this run. - (for ([relative (in-list (acceptance-suite-evidence suite))]) + (for ([entry (in-list (acceptance-suite-evidence suite))]) + (define relative (car entry)) (define path (project-path project relative)) (for/fold ([parent (rivet-project-root project)]) ([part (in-list @@ -302,9 +370,23 @@ (delete-file path)))) (define (evidence-missing project suite) - (for/list ([relative (in-list (acceptance-suite-evidence suite))] - #:unless (file-exists? (project-path project relative))) - relative)) + (for/list ([entry (in-list (acceptance-suite-evidence suite))] + #:unless (file-exists? (project-path project (car entry)))) + (car entry))) + +;; First-class evidence records (issue #287): the report pins each declared +;; artifact's kind, presence, size, and digest so remote consumers can audit +;; what a green suite actually produced without trusting prose. +(define (evidence-entries project suite) + (for/list ([entry (in-list (acceptance-suite-evidence suite))]) + (define relative (car entry)) + (define path (project-path project relative)) + (define present (file-exists? path)) + (hash 'path relative + 'kind (symbol->string (cdr entry)) + 'present present + 'bytes (if present (file-size path) 'null) + 'sha256 (if present (string-downcase (sha256-file/hex path)) 'null)))) (define (execute-suite project suite) (define command (acceptance-suite-command suite)) @@ -334,7 +416,7 @@ (hash 'id (symbol->string (acceptance-suite-id suite)) 'kind (symbol->string (acceptance-suite-kind suite)) 'command (acceptance-suite-command suite) - 'evidence (acceptance-suite-evidence suite) + 'evidence (evidence-entries project suite) 'status "fail" 'duration-ms (inexact->exact @@ -371,7 +453,7 @@ (hash 'id (symbol->string (acceptance-suite-id suite)) 'kind (symbol->string (acceptance-suite-kind suite)) 'command (acceptance-suite-command suite) - 'evidence (acceptance-suite-evidence suite) + 'evidence (evidence-entries project suite) 'status (if passed? "pass" "fail") 'duration-ms (inexact->exact diff --git a/rivet-cli/inspect.rkt b/rivet-cli/inspect.rkt index 3f5b982d..0198c869 100644 --- a/rivet-cli/inspect.rkt +++ b/rivet-cli/inspect.rkt @@ -294,6 +294,9 @@ (list (action "inspect" "raco rivet inspect --json" #f) (action "diagnose" "raco rivet doctor --json" #f) (action "acceptance" "raco rivet acceptance --json" #f) + (action "verify-remote" + "raco rivet verify-remote --repo OWNER/REPO --ref HEAD" + #f) (action "inspect-schema" "raco rivet schema --json" #f) (action "save-schema-baseline" "raco rivet schema --output rivet-schema.json" diff --git a/rivet-cli/main.rkt b/rivet-cli/main.rkt index 0b38dfd1..a32f2b42 100644 --- a/rivet-cli/main.rkt +++ b/rivet-cli/main.rkt @@ -43,6 +43,9 @@ " raco rivet matrix ... aggregate per-platform acceptance reports into the\n" " cross-platform verification matrix\n" " raco rivet matrix ... --json emit the machine-readable matrix\n" + " raco rivet verify-remote --repo OWNER/REPO --ref REF [--workflow verify.yml]\n" + " trigger the per-platform acceptance legs and\n" + " fold their archived reports into the matrix\n" " raco rivet schema --json emit the current versioned API schema\n" " raco rivet schema --output write a schema compatibility baseline\n" " raco rivet schema check reject breaking changes from a baseline\n" @@ -120,6 +123,12 @@ ;; is judging (issue #287). (define run-matrix-command (command "matrix" 'run-matrix-command)) (exit (run-matrix-command matrix-args))] + [(cons "verify-remote" verify-remote-args) + ;; Remote orchestration triggers the platform legs and pulls their + ;; archived reports into the matrix from one editing host (issue #287). + (define run-verify-remote-command + (command "verify-remote" 'run-verify-remote-command)) + (exit (run-verify-remote-command verify-remote-args))] [(list "schema" "--json") (define schema-snapshot (command "codegen" 'schema-snapshot)) (write-json (schema-snapshot (current-project!))) diff --git a/rivet-cli/verify-remote.rkt b/rivet-cli/verify-remote.rkt new file mode 100644 index 00000000..013f3ced --- /dev/null +++ b/rivet-cli/verify-remote.rkt @@ -0,0 +1,412 @@ +#lang racket/base + +;; Remote verification orchestration (issue #287). The local operating system +;; is an editing environment, not the boundary of verification: this command +;; triggers the per-platform acceptance legs of one commit on GitHub Actions, +;; waits for the dispatched runs, downloads the archived `acceptance --json` +;; reports, and folds them into the shared verification matrix. It drives the +;; `gh` CLI and nothing else, so workflow names and artifact names stay the +;; product's own conventions. + +(require json + racket/file + racket/list + racket/match + racket/path + racket/port + racket/string + "acceptance.rkt" + "matrix.rkt") + +(provide run-verify-remote-command) + +(define default-workflow "verify.yml") +(define default-artifact-prefix "acceptance-report-") +(define default-timeout-seconds 1800) +(define default-output-directory ".rivet/verify-remote") +(define run-list-poll-seconds 5) + +(define usage-text + (string-append + "raco rivet verify-remote --repo OWNER/REPO --ref REF" + " [--workflow verify.yml] [--platform linux]..." + " [--artifact-prefix acceptance-report-]" + " [--timeout-seconds 1800] [--output-directory DIR] [--json]")) + +(define (verify-remote-error message . fields) + (apply raise-arguments-error 'rivet-verify-remote message fields)) + +(define (non-empty-text? value) + (and (string? value) (positive? (string-length value)))) + +(define (current-inexact-seconds) + (/ (current-inexact-milliseconds) 1000.0)) + +;; One collected workflow run: how it concluded, which acceptance artifacts +;; it archived, and which loaded platform reports it produced. +(struct run-outcome (run-id conclusion reports detail) #:transparent) + +;; One gh invocation: (values exit-code stdout stderr). The command never +;; decides trust from exit codes it did not see, so every caller funnels +;; through this injectable boundary. Tests parameterize `current-run-gh` +;; with a scripted replacement. +(define (default-run-gh argv) + (define gh (find-executable-path "gh")) + (unless gh + (verify-remote-error + "the gh CLI was not found on PATH" + "fix" "install the GitHub CLI and authenticate with `gh auth login`")) + (define-values (child stdout stdin stderr) + (apply subprocess #f #f #f gh argv)) + (close-output-port stdin) + (define stdout-thread (thread (lambda () (port->string stdout)))) + (define stderr-thread (thread (lambda () (port->string stderr)))) + (subprocess-wait child) + (values (subprocess-status child) + (thread-wait stdout-thread) + (thread-wait stderr-thread))) + +(define current-run-gh (make-parameter default-run-gh)) + +(define (trimmed text) + (string-trim (or text ""))) + +(define (run-gh-must who argv) + ;; who is the surrounding step; the failure detail carries gh's stderr. + (define-values (code stdout stderr) ((current-run-gh) argv)) + (unless (zero? code) + (verify-remote-error (format "~a failed" who) + "command" (string-join (cons "gh" argv) " ") + "detail" (trimmed stderr))) + stdout) + +(define (gh-json who stdout) + (with-handlers + ([exn:fail? + (lambda (error) + (verify-remote-error "gh emitted unparseable JSON" + "command" who + "message" (exn-message error)))]) + (read-json (open-input-string stdout)))) + +(define (resolve-commit-sha repo ref) + (define sha + (trimmed + (run-gh-must "commit resolution" + (list "api" (format "repos/~a/commits/~a" repo ref) + "--jq" ".sha")))) + (unless (regexp-match? #px"^[0-9a-f]{40}$" sha) + (verify-remote-error "commit resolution returned no commit sha" + "repo" repo + "ref" ref + "detail" sha)) + sha) + +;; Dispatch runs are identified against this baseline so a concurrent +;; session's older runs can never be mistaken for this verification. +(define (listed-dispatch-runs repo workflow) + (define runs + (gh-json "gh run list" + (run-gh-must + "run listing" + (list "run" "list" + "--repo" repo + "--workflow" workflow + "--event" "workflow_dispatch" + "--json" "databaseId,headSha" + "--limit" "50")))) + (unless (list? runs) + (verify-remote-error "gh run list returned no run array" + "repo" repo + "workflow" workflow)) + (for/list ([run (in-list runs)]) + (unless (and (hash? run) + (exact-integer? (hash-ref run 'databaseId #f)) + (string? (hash-ref run 'headSha #f))) + (verify-remote-error "gh run list returned a malformed run" + "repo" repo + "value" run)) + run)) + +(define (dispatch-platform repo workflow sha platform) + (run-gh-must "workflow dispatch" + (list "workflow" "run" workflow + "--repo" repo + "--ref" sha + "-f" (format "platform=~a" platform)))) + +(define (wait-for-dispatched-runs repo workflow sha baseline expected + timeout-seconds sleep now) + (define started (now)) + (let poll () + (define runs (listed-dispatch-runs repo workflow)) + (define discovered + (remove-duplicates + (for/list ([run (in-list runs)] + #:when (and (string=? (hash-ref run 'headSha) sha) + (not (member (hash-ref run 'databaseId) + baseline =)))) + (hash-ref run 'databaseId)) + =)) + (cond + [(>= (length discovered) expected) discovered] + [(> (- (now) started) timeout-seconds) + (verify-remote-error + "dispatched verification runs never appeared" + "workflow" workflow + "commit" sha + "expected-runs" expected + "discovered-runs" (length discovered) + "timeout-seconds" timeout-seconds)] + [else (sleep run-list-poll-seconds) (poll)]))) + +;; A red run must still hand over its archived reports: the watch exit code +;; only sets the conclusion, never skips the download, because a failing +;; platform's evidence is exactly what the matrix exists to show. +(define (collect-run run-id repo artifact-prefix platforms output-directory) + (define-values (watch-code _watch-stdout _watch-stderr) + ((current-run-gh) + (list "run" "watch" (number->string run-id) + "--repo" repo + "--exit-status" + "--interval" "10"))) + (define conclusion (if (zero? watch-code) "success" "failure")) + (define run-directory + (build-path output-directory (format "run-~a" run-id))) + ;; Stale artifacts from an earlier collection of the same run must not + ;; satisfy this one, mirroring the evidence hygiene of acceptance suites. + (when (directory-exists? run-directory) + (delete-directory/files run-directory)) + (define-values (_dl-code _dl-stdout _dl-stderr) + ((current-run-gh) + (list "run" "download" (number->string run-id) + "--repo" repo + "--dir" (path->string run-directory)))) + (define artifact-directories + (if (directory-exists? run-directory) + (for/list ([entry (in-list (directory-list run-directory #:build? #t))] + #:when (directory-exists? entry)) + entry) + '())) + (define reports + (append* + (for*/list ([artifact-directory (in-list artifact-directories)] + [artifact (in-value + (path->string + (file-name-from-path artifact-directory)))] + [platform (in-list platforms)] + #:when (string=? + artifact + (string-append artifact-prefix + (symbol->string platform)))) + (define json-paths + (sort (find-files (lambda (path) + (equal? (path-get-extension path) + #".json")) + artifact-directory) + stringstring)) + (unless (pair? json-paths) + (verify-remote-error + "the archived acceptance artifact contains no JSON report" + "artifact" artifact)) + (for/list ([path (in-list json-paths)]) + (hash 'platform (symbol->string platform) + 'path (path->string path) + 'report (load-acceptance-report path)))))) + (run-outcome run-id conclusion reports + (if (zero? watch-code) #f (format "run concluded ~a" conclusion)))) + +(define (parse-arguments args) + (let loop ([remaining args] + [repo #f] + [ref #f] + [workflow default-workflow] + [platforms '()] + [artifact-prefix default-artifact-prefix] + [timeout-seconds default-timeout-seconds] + [output-directory default-output-directory] + [json? #f]) + (match remaining + ['() + (unless repo + (verify-remote-error "missing required --repo option" + "usage" usage-text)) + (unless ref + (verify-remote-error "missing required --ref option" + "usage" usage-text)) + (define requested + (remove-duplicates (reverse (map string->symbol platforms)) eq?)) + (for ([platform (in-list requested)]) + (unless (memq platform supported-platforms) + (verify-remote-error "unknown --platform" + "expected" + (map symbol->string supported-platforms) + "value" platform))) + (unless (and (exact-integer? timeout-seconds) + (positive? timeout-seconds)) + (verify-remote-error "invalid --timeout-seconds" + "expected" "positive integer" + "value" timeout-seconds)) + (values repo + ref + workflow + (if (null? requested) supported-platforms requested) + artifact-prefix + timeout-seconds + output-directory + json?)] + [(list* "--repo" value tail) + (when repo (verify-remote-error "duplicate --repo option")) + (unless (regexp-match? #px"^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$" value) + (verify-remote-error "invalid --repo" + "expected" "OWNER/REPO" + "value" value)) + (loop tail value ref workflow platforms artifact-prefix + timeout-seconds output-directory json?)] + [(list* "--ref" value tail) + (when ref (verify-remote-error "duplicate --ref option")) + (unless (non-empty-text? value) + (verify-remote-error "invalid --ref" "value" value)) + (loop tail repo value workflow platforms artifact-prefix + timeout-seconds output-directory json?)] + [(list* "--workflow" value tail) + (unless (non-empty-text? value) + (verify-remote-error "invalid --workflow" "value" value)) + (loop tail repo ref value platforms artifact-prefix + timeout-seconds output-directory json?)] + [(list* "--platform" value tail) + (loop tail repo ref workflow (cons value platforms) artifact-prefix + timeout-seconds output-directory json?)] + [(list* "--artifact-prefix" value tail) + (unless (non-empty-text? value) + (verify-remote-error "invalid --artifact-prefix" "value" value)) + (loop tail repo ref workflow platforms value + timeout-seconds output-directory json?)] + [(list* "--timeout-seconds" value tail) + (define parsed + (and (string? value) + (regexp-match? #px"^[0-9]+$" value) + (string->number value))) + (loop tail repo ref workflow platforms artifact-prefix + parsed output-directory json?)] + [(list* "--output-directory" value tail) + (unless (non-empty-text? value) + (verify-remote-error "invalid --output-directory" "value" value)) + (loop tail repo ref workflow platforms artifact-prefix + timeout-seconds value json?)] + [(cons "--json" tail) + (when json? (verify-remote-error "duplicate --json option")) + (loop tail repo ref workflow platforms artifact-prefix + timeout-seconds output-directory #t)] + [_ + (verify-remote-error "unexpected argument" + "usage" usage-text + "arguments" args)]))) + +(define (display-outcomes repo ref sha workflow outcomes) + (printf "Rivet remote verification: ~a @ ~a (~a)\n" + repo (substring sha 0 (min 12 (string-length sha))) workflow) + (for ([outcome (in-list outcomes)]) + (define platforms + (remove-duplicates + (map (lambda (report) (hash-ref report 'platform)) + (run-outcome-reports outcome)))) + (printf " run ~a: ~a\n" + (run-outcome-run-id outcome) + (if (null? platforms) + (or (run-outcome-detail outcome) "no acceptance artifacts") + (format "~a — ~a, ~a report~a" + (string-join platforms ", ") + (or (run-outcome-detail outcome) "success") + (length (run-outcome-reports outcome)) + (if (= (length (run-outcome-reports outcome)) 1) + "" + "s")))))) + +(define (run-verify-remote-command args + #:run-gh [run-gh default-run-gh] + #:sleep [wake-after + (lambda (seconds) + (sleep seconds))] + #:now [now current-inexact-seconds]) + (define-values (repo ref workflow platforms artifact-prefix + timeout-seconds output-directory json?) + (parse-arguments args)) + ;; Bind the injectable boundary once so every helper sees the same scripted + ;; gh during tests. + (parameterize ([current-run-gh run-gh]) + (define sha (resolve-commit-sha repo ref)) + (make-directory* output-directory) + (define baseline + (map (lambda (run) (hash-ref run 'databaseId)) + (listed-dispatch-runs repo workflow))) + ;; Dispatch one leg per platform, then discover the runs those + ;; dispatches produced. Collection keys off artifact names, so a + ;; product's matrix workflow may name its jobs however it likes. + (for ([platform (in-list platforms)]) + (dispatch-platform repo workflow sha platform)) + (define run-ids + (wait-for-dispatched-runs repo workflow sha baseline (length platforms) + timeout-seconds wake-after now)) + (define outcomes + (for/list ([run-id (in-list (sort run-ids <))]) + (with-handlers + ([exn:fail? + (lambda (error) + (run-outcome run-id "collection-failed" '() + (exn-message error)))]) + (collect-run run-id repo artifact-prefix platforms + output-directory)))) + (define loaded-reports + (append* (map (lambda (outcome) (run-outcome-reports outcome)) + outcomes))) + (define covered + (remove-duplicates + (map (lambda (report) (hash-ref report 'platform)) + loaded-reports))) + (define uncovered + (for/list ([platform (in-list platforms)] + #:unless (member (symbol->string platform) covered)) + (symbol->string platform))) + (define matrix + (and (pair? loaded-reports) + (matrix-report (map (lambda (report) + (hash-ref report 'report)) + loaded-reports)))) + (cond + [json? + (write-json + (hash 'format-version 1 + 'repo repo + 'ref ref + 'sha sha + 'workflow workflow + 'runs (for/list ([outcome (in-list outcomes)]) + (hash 'run-id (run-outcome-run-id outcome) + 'conclusion (run-outcome-conclusion outcome) + 'detail (or (run-outcome-detail outcome) 'null) + 'reports (for/list ([report + (in-list + (run-outcome-reports + outcome))]) + (hash 'platform + (hash-ref report 'platform) + 'path (hash-ref report 'path))))) + 'matrix (or matrix 'null))) + (newline)] + [else + (display-outcomes repo ref sha workflow outcomes) + (unless (null? uncovered) + (printf " missing platform legs: ~a\n" (string-join uncovered ", "))) + (when matrix + (display-matrix matrix))]) + (if (and matrix + (string=? (hash-ref matrix 'status) "pass") + (null? uncovered) + (not (findf (lambda (outcome) + (string=? (run-outcome-conclusion outcome) + "collection-failed")) + outcomes))) + 0 + 1))) diff --git a/rivet/tests/acceptance.rkt b/rivet/tests/acceptance.rkt index 36fae089..db78297b 100644 --- a/rivet/tests/acceptance.rkt +++ b/rivet/tests/acceptance.rkt @@ -178,7 +178,116 @@ RKT (check-equal? (hash-ref (first (hash-ref stale-report 'suites)) 'missing-evidence) '(".rivet/acceptance/stale.txt")) - (check-false (file-exists? stale-evidence))) + (check-false (file-exists? stale-evidence)) + + ;; Evidence is classified (issue #287): accessibility snapshots are the + ;; primary cross-platform record, screenshots are supplemental, and plain + ;; string paths stay legal as the generic kind. + (write-manifest + root + #<string + (hasheq 'format-version 1 + 'project "demo" + 'platform platform + 'manifest "rivet-acceptance.rktd" + 'configured-suites 1 + 'configured-requirements + (list (hasheq 'kind "native-ui" + 'platforms requirement-platforms)) + 'selected-suite 'null + 'status status + 'passed (if (string=? status "pass") 1 0) + 'failed (if (string=? status "pass") 0 1) + 'suites + (list (hasheq 'id (string-append platform "-ui") + 'kind "native-ui" + 'status status))))) + +;; Builds the injectable gh boundary: records every invocation, replays the +;; run list progression, and materializes archived artifacts as real files so +;; the collection path runs for real. +(define (make-fake-gh! + #:runs-before [runs-before "[]"] + #:runs-after [runs-after "[]"] + #:runs-after-on-call [runs-after-on-call 2] + #:watch-codes [watch-codes (hash)] + #:artifacts-by-run [artifacts-by-run (hash)] + #:dispatch-code [dispatch-code 0] + #:log [log #f]) + (define run-list-calls 0) + (define (flag-value argv flag) + (let loop ([rest argv]) + (match rest + [(list (== flag) next _ ...) next] + [(cons _ tail) (loop tail)] + [_ #f]))) + (lambda (argv) + (when log (log argv)) + (match argv + [(list "api" _ "--jq" ".sha") + (values 0 sha-under-test "")] + [(list "run" "list" _ ...) + (set! run-list-calls (+ run-list-calls 1)) + (values 0 + (if (>= run-list-calls runs-after-on-call) + runs-after + runs-before) + "")] + [(list "workflow" "run" _ ...) + (if (zero? dispatch-code) + (values 0 "" "") + (values dispatch-code "" "no workflow_dispatch trigger"))] + [(list "run" "watch" raw-id _ ...) + (define run-id (string->number raw-id)) + (values (hash-ref watch-codes run-id 0) "" "")] + [(list "run" "download" raw-id _ ...) + (define run-id (string->number raw-id)) + (define directory + (and run-id (flag-value argv "--dir"))) + (for ([spec (in-list (hash-ref artifacts-by-run run-id '()))] + #:when directory) + (define artifact-directory + (build-path (string->path directory) + (string-append "acceptance-report-" + (car spec)))) + (make-directory* artifact-directory) + (call-with-output-file + (build-path artifact-directory "report.json") + (lambda (output) (display (cdr spec) output)) + #:exists 'truncate/replace)) + (values 0 "" "")] + [_ + (values 1 "" (format "unexpected gh invocation: ~a" argv))]))) + +(define (all-green-artifacts) + (hash 101 (list (cons "linux" (report-text "linux" "pass"))) + 102 (list (cons "macos" (report-text "macos" "pass"))) + 103 (list (cons "windows" (report-text "windows" "pass"))))) + +(define (green-runs-json) + (jsexpr->string + (list (hasheq 'databaseId 101 'headSha sha-under-test) + (hasheq 'databaseId 102 'headSha sha-under-test) + (hasheq 'databaseId 103 'headSha sha-under-test)))) + +(define (capture-output thunk) + (define out (open-output-string)) + (define exit-code + (parameterize ([current-output-port out]) (thunk))) + (values exit-code (get-output-string out))) + +(define (sleep-counter) + (define count 0) + (values (lambda (_) (set! count (+ count 1))) + (lambda () count))) + +;; Three green legs fold into a passing matrix and exit 0. +(let-values ([(wake count) (sleep-counter)]) + (define exit-code + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main") + #:run-gh (make-fake-gh! + #:runs-after (green-runs-json) + #:artifacts-by-run (all-green-artifacts)) + #:sleep wake + #:now (lambda () 0))) + (check-equal? exit-code 0) + (check-equal? (count) 0)) + +;; The same command in --json mode reports the runs and the matrix verdict. +(let () + (define-values (exit-code output) + (capture-output + (lambda () + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main" "--json") + #:run-gh (make-fake-gh! + #:runs-after (green-runs-json) + #:artifacts-by-run (all-green-artifacts)) + #:sleep (lambda (_) (void)) + #:now (lambda () 0))))) + (check-equal? exit-code 0) + (define parsed (read-json (open-input-string output))) + (check-equal? (hash-ref parsed 'repo) "acme/demo") + (check-equal? (hash-ref parsed 'sha) sha-under-test) + (check-equal? (length (hash-ref parsed 'runs)) 3) + (check-equal? (hash-ref (hash-ref parsed 'matrix) 'status) "pass") + (check-equal? + (length (hash-ref (hash-ref parsed 'matrix) 'cells)) 3)) + +;; A red run still hands over its archived evidence; the failing platform's +;; report fails its matrix cell and the command exits 1. +(let () + (define artifacts + (hash 101 (list (cons "linux" (report-text "linux" "fail"))) + 102 (list (cons "macos" (report-text "macos" "pass"))) + 103 (list (cons "windows" (report-text "windows" "pass"))))) + (define-values (exit-code output) + (capture-output + (lambda () + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main") + #:run-gh (make-fake-gh! + #:runs-after (green-runs-json) + #:watch-codes (hash 101 1) + #:artifacts-by-run artifacts) + #:sleep (lambda (_) (void)) + #:now (lambda () 0))))) + (check-equal? exit-code 1) + (check-true (string-contains? output "FAIL"))) + +;; A leg that archives nothing leaves its platform uncovered; the matrix can +;; stay green for what it saw but the command still fails the verification. +(let () + (define work (make-temporary-file "rivet-verify-~a" 'directory)) + (define-values (exit-code output) + (capture-output + (lambda () + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main" "--platform" "linux" + "--output-directory" (path->string work)) + #:run-gh (make-fake-gh! + #:runs-after (jsexpr->string + (list (hasheq 'databaseId 101 + 'headSha sha-under-test))) + #:artifacts-by-run (hash)) + #:sleep (lambda (_) (void)) + #:now (lambda () 0))))) + (check-equal? exit-code 1) + (check-true (string-contains? output "missing platform legs: linux")) + (check-true (string-contains? output "run 101: no acceptance artifacts"))) + +;; The dispatcher waits across poll cycles before the runs appear. The +;; report only requires the linux leg, so a single green leg passes. +(let-values ([(wake count) (sleep-counter)]) + (define work (make-temporary-file "rivet-verify-~a" 'directory)) + (define exit-code + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main" "--platform" "linux" + "--output-directory" (path->string work)) + #:run-gh (make-fake-gh! + #:runs-after (jsexpr->string + (list (hasheq 'databaseId 101 + 'headSha sha-under-test))) + #:runs-after-on-call 4 + #:artifacts-by-run + (hash 101 (list (cons "linux" + (report-text "linux" "pass" + (list "linux")))))) + #:sleep wake + #:now (lambda () 0))) + (check-equal? exit-code 0) + (check-true (>= (count) 1))) + +;; Runs that never appear fail loudly instead of blocking forever. The +;; injected clock advances past the timeout on its second reading. +(let ([tick 0]) + (check-exn + #rx"dispatched verification runs never appeared" + (lambda () + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main") + #:run-gh (make-fake-gh!) + #:sleep (lambda (_) (void)) + #:now (lambda () + (set! tick (+ tick 100000)) + tick))))) + +;; A workflow without a dispatch trigger fails the dispatch step. +(check-exn + #rx"workflow dispatch failed" + (lambda () + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main") + #:run-gh (make-fake-gh! #:dispatch-code 1) + #:sleep (lambda (_) (void)) + #:now (lambda () 0)))) + +;; Argument validation stays fail-closed. +(check-exn #rx"missing required --repo" + (lambda () + (run-verify-remote-command + (list "--ref" "main") + #:run-gh (make-fake-gh!) + #:now (lambda () 0)))) +(check-exn #rx"invalid --repo" + (lambda () + (run-verify-remote-command + (list "--repo" "acme-demo" "--ref" "main") + #:run-gh (make-fake-gh!) + #:now (lambda () 0)))) +(check-exn #rx"unknown --platform" + (lambda () + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main" + "--platform" "freebsd") + #:run-gh (make-fake-gh!) + #:now (lambda () 0)))) From d136b377c45d941737437128fd4d7ff656e95a00 Mon Sep 17 00:00:00 2001 From: jrtxio Date: Tue, 20 Oct 2026 01:31:49 +0800 Subject: [PATCH 2/3] fix: keep test verification state out of the source tree raco test runs with the test file's directory as the working directory, so the default output directory sprouted rivet/tests/.rivet during CI and the source-package smoke rightly rejected it. Every scenario now pins an isolated output directory. --- rivet/tests/verify-remote.rkt | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/rivet/tests/verify-remote.rkt b/rivet/tests/verify-remote.rkt index bf643456..43dda29a 100644 --- a/rivet/tests/verify-remote.rkt +++ b/rivet/tests/verify-remote.rkt @@ -116,9 +116,11 @@ ;; Three green legs fold into a passing matrix and exit 0. (let-values ([(wake count) (sleep-counter)]) + (define work (make-temporary-file "rivet-verify-~a" 'directory)) (define exit-code (run-verify-remote-command - (list "--repo" "acme/demo" "--ref" "main") + (list "--repo" "acme/demo" "--ref" "main" + "--output-directory" (path->string work)) #:run-gh (make-fake-gh! #:runs-after (green-runs-json) #:artifacts-by-run (all-green-artifacts)) @@ -129,11 +131,13 @@ ;; The same command in --json mode reports the runs and the matrix verdict. (let () + (define work (make-temporary-file "rivet-verify-~a" 'directory)) (define-values (exit-code output) (capture-output (lambda () (run-verify-remote-command - (list "--repo" "acme/demo" "--ref" "main" "--json") + (list "--repo" "acme/demo" "--ref" "main" "--json" + "--output-directory" (path->string work)) #:run-gh (make-fake-gh! #:runs-after (green-runs-json) #:artifacts-by-run (all-green-artifacts)) @@ -151,6 +155,7 @@ ;; A red run still hands over its archived evidence; the failing platform's ;; report fails its matrix cell and the command exits 1. (let () + (define work (make-temporary-file "rivet-verify-~a" 'directory)) (define artifacts (hash 101 (list (cons "linux" (report-text "linux" "fail"))) 102 (list (cons "macos" (report-text "macos" "pass"))) @@ -159,7 +164,8 @@ (capture-output (lambda () (run-verify-remote-command - (list "--repo" "acme/demo" "--ref" "main") + (list "--repo" "acme/demo" "--ref" "main" + "--output-directory" (path->string work)) #:run-gh (make-fake-gh! #:runs-after (green-runs-json) #:watch-codes (hash 101 1) From 545c224323c53af0be396bcde2f40176600736fc Mon Sep 17 00:00:00 2001 From: jrtxio Date: Tue, 20 Oct 2026 01:54:59 +0800 Subject: [PATCH 3/3] fix: write verification output only once runs exist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The output directory was created before dispatch, so a failed dispatch or a discovery timeout left an empty .rivet/verify-remote behind in the caller's working directory — which the source-package smoke rightly rejects. Create it after the dispatched runs are confirmed. --- rivet-cli/verify-remote.rkt | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/rivet-cli/verify-remote.rkt b/rivet-cli/verify-remote.rkt index 013f3ced..f259d43f 100644 --- a/rivet-cli/verify-remote.rkt +++ b/rivet-cli/verify-remote.rkt @@ -337,7 +337,6 @@ ;; gh during tests. (parameterize ([current-run-gh run-gh]) (define sha (resolve-commit-sha repo ref)) - (make-directory* output-directory) (define baseline (map (lambda (run) (hash-ref run 'databaseId)) (listed-dispatch-runs repo workflow))) @@ -349,6 +348,9 @@ (define run-ids (wait-for-dispatched-runs repo workflow sha baseline (length platforms) timeout-seconds wake-after now)) + ;; Nothing is written until the runs exist: a failed dispatch or a + ;; timeout leaves no empty output directory behind. + (make-directory* output-directory) (define outcomes (for/list ([run-id (in-list (sort run-ids <))]) (with-handlers