diff --git a/AGENTS.md b/AGENTS.md index b859563b..695e9239 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -80,7 +80,9 @@ required acceptance suites pass on Windows, macOS, and Linux. The local operating system determines only which checks can run locally; it does not reduce the required verification scope. Archive every platform leg's `acceptance --json` report and prove the union with -`raco rivet matrix --json` (issue #287). Classify desktop +`raco rivet matrix --json`; `raco rivet verify-remote` +dispatches those legs and pulls their archived reports into the matrix from +this host when the platforms are not local (issue #287). Classify desktop capabilities as guaranteed, conditional, or platform-specific as documented in `docs/product-acceptance.md`. diff --git a/CHANGELOG.md b/CHANGELOG.md index afa9f485..f3136b28 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,19 @@ ## Unreleased +- Make verification evidence and orchestration first-class for distributed + agents (issue #287). `raco rivet verify-remote` dispatches the + per-platform acceptance legs of one commit on GitHub Actions, waits for the + runs, downloads the archived `acceptance --json` reports, and folds them + into the verification matrix, so an agent edits on one OS and is judged by + the others. Acceptance evidence entries are now classified — + `accessibility-snapshot`, `screenshot`, `interaction-trace`, `log`, or + `generic` — and the report pins each declared artifact's kind, presence, + byte size, and SHA-256, while plain string evidence paths stay legal as + `generic`. The product CI guide archives every leg's report as + `acceptance-report-` with `if: always()`, which is exactly the + convention the new command consumes. + - Add the crash-safe process-external Linux AppImage handoff (issue #290). `AppImageHandoff::LaunchReplacement` in the linked `rivet::platform` target writes a generated first-party shell script — the counterpart of the diff --git a/docs/ci.md b/docs/ci.md index a81ec5af..63389f13 100644 --- a/docs/ci.md +++ b/docs/ci.md @@ -60,7 +60,13 @@ jobs: - run: raco pkg install --auto --no-docs 'https://github.com/turinglambdaai/rivet.git#v0.12.0' - run: raco rivet doctor - run: raco rivet schema check rivet-schema.json - - run: raco rivet acceptance --json + - run: raco rivet acceptance --json > acceptance-report.json + shell: bash + - uses: actions/upload-artifact@v7 + if: always() + with: + name: acceptance-report-windows + path: acceptance-report.json - run: raco rivet build macos-host: @@ -80,7 +86,13 @@ jobs: - run: raco pkg install --auto --no-docs 'https://github.com/turinglambdaai/rivet.git#v0.12.0' - run: raco rivet doctor - run: raco rivet schema check rivet-schema.json - - run: raco rivet acceptance --json + - run: raco rivet acceptance --json > acceptance-report.json + shell: bash + - uses: actions/upload-artifact@v7 + if: always() + with: + name: acceptance-report-macos + path: acceptance-report.json - run: raco rivet build linux-host: @@ -97,10 +109,23 @@ jobs: run: raco pkg install --auto --no-docs 'https://github.com/turinglambdaai/rivet.git#v0.12.0' - run: raco rivet doctor - run: raco rivet schema check rivet-schema.json - - run: raco rivet acceptance --json + - run: raco rivet acceptance --json > acceptance-report.json + shell: bash + - uses: actions/upload-artifact@v7 + if: always() + with: + name: acceptance-report-linux + path: acceptance-report.json - run: raco rivet build ``` +Each job writes its `acceptance --json` report to a file and archives it as +`acceptance-report-` with `if: always()`, so a red leg still hands +over its evidence. Those names are exactly what +[`raco rivet verify-remote`](product-acceptance.md#orchestrate-remote-verification-from-one-host) +expects: from any editing host it dispatches the legs, waits for the runs, +downloads the archived reports, and folds them into the verification matrix. + The Linux order is load-bearing: install the Rivet package **after** `setup-embed-racket`. The action's minimal Racket build intentionally ships without packages; installing Rivet afterward resolves `cext-lib`, `crypto-lib`, diff --git a/docs/ci.zh-CN.md b/docs/ci.zh-CN.md index 1db6f68d..182d26b3 100644 --- a/docs/ci.zh-CN.md +++ b/docs/ci.zh-CN.md @@ -55,7 +55,13 @@ jobs: - run: raco pkg install --auto --no-docs 'https://github.com/turinglambdaai/rivet.git#v0.12.0' - run: raco rivet doctor - run: raco rivet schema check rivet-schema.json - - run: raco rivet acceptance --json + - run: raco rivet acceptance --json > acceptance-report.json + shell: bash + - uses: actions/upload-artifact@v7 + if: always() + with: + name: acceptance-report-windows + path: acceptance-report.json - run: raco rivet build macos-host: @@ -75,7 +81,13 @@ jobs: - run: raco pkg install --auto --no-docs 'https://github.com/turinglambdaai/rivet.git#v0.12.0' - run: raco rivet doctor - run: raco rivet schema check rivet-schema.json - - run: raco rivet acceptance --json + - run: raco rivet acceptance --json > acceptance-report.json + shell: bash + - uses: actions/upload-artifact@v7 + if: always() + with: + name: acceptance-report-macos + path: acceptance-report.json - run: raco rivet build linux-host: @@ -92,10 +104,23 @@ jobs: run: raco pkg install --auto --no-docs 'https://github.com/turinglambdaai/rivet.git#v0.12.0' - run: raco rivet doctor - run: raco rivet schema check rivet-schema.json - - run: raco rivet acceptance --json + - run: raco rivet acceptance --json > acceptance-report.json + shell: bash + - uses: actions/upload-artifact@v7 + if: always() + with: + name: acceptance-report-linux + path: acceptance-report.json - run: raco rivet build ``` +每个 job 都把 `acceptance --json` 报告写入文件,并以 +`acceptance-report-` 为名、`if: always()` 归档——失败的腿也要交出 +证据。这些名字正是 +[`raco rivet verify-remote`](product-acceptance.md#orchestrate-remote-verification-from-one-host) +的默认约定:在任一编辑主机上触发各平台腿、等待运行、下载归档报告并折叠成 +验证矩阵。 + Linux 中的顺序是必要条件:必须在 `setup-embed-racket` **之后**安装 Rivet。 该 action 准备的是刻意不附带任何包的 minimal Racket;此时再安装 Rivet, `cext-lib`、`crypto-lib`、`keel` 等完整依赖才会进入嵌入式 runtime 实际读取的 diff --git a/docs/product-acceptance.md b/docs/product-acceptance.md index b5a1ed30..d281368d 100644 --- a/docs/product-acceptance.md +++ b/docs/product-acceptance.md @@ -33,8 +33,12 @@ Commit the manifest at the project root: (platforms . (windows)) (command . ("pwsh" "-File" "scripts/e2e/windows.ps1")) (timeout-seconds . 900) - (evidence . (".rivet/acceptance/windows/trace.json" - ".rivet/acceptance/windows/accessibility.json")))))) + (evidence + . (#hasheq((path . ".rivet/acceptance/windows/accessibility.json") + (kind . accessibility-snapshot)) + #hasheq((path . ".rivet/acceptance/windows/screen.png") + (kind . screenshot)) + ".rivet/acceptance/windows/trace.json")))))) ``` The supported suite kinds are: @@ -56,6 +60,16 @@ suite starts and then requires the suite to recreate it, so a stale artifact from an earlier green run cannot satisfy the current run. `native-ui`, `package`, and `release` suites must declare at least one evidence file. +Evidence entries are classified so remote consumers know what a green suite +actually produced. A bare path string means the `generic` kind; a hash declares +`path` and one `kind` of `accessibility-snapshot`, `screenshot`, +`interaction-trace`, `log`, or `generic`. Accessibility snapshots are the +primary cross-platform record — semantic UI state that is comparable across +platforms — while screenshots are supplemental because native presentations +are expected to differ. The `--json` report pins every declared entry's kind, +presence, byte size, and SHA-256, so an archived report audits the artifacts +it vouches for. + `requirements` is the reviewable policy. For every kind/platform pair there must be at least one matching suite or the manifest is rejected before a child process starts. Product owners should protect changes to requirements, CI, @@ -148,6 +162,35 @@ only when every required kind/platform cell is proven. Feed the reports for the commit under verification; the matrix aggregates what it is given and does not heuristically prefer fresher runs. +## Orchestrate remote verification from one host + +`raco rivet verify-remote` closes the loop for an agent editing away from the +platform legs: it dispatches each platform's acceptance leg on GitHub Actions, +waits for the runs, downloads the archived `acceptance --json` reports, and +folds them into the matrix: + +```bash +raco rivet verify-remote --repo OWNER/REPO --ref +raco rivet verify-remote --repo OWNER/REPO --ref --platform linux +raco rivet verify-remote --repo OWNER/REPO --ref --json +``` + +The command drives the `gh` CLI and nothing else. The defaults encode one +convention — workflow `verify.yml`, artifacts named +`acceptance-report-` — and `--workflow`/`--artifact-prefix` adapt it +to a product's own naming. The workflow contract is: a `workflow_dispatch` +trigger with a `platform` input (`linux`, `windows`, or `macos`), one +`raco rivet acceptance --json` per leg, and the report uploaded with +`if: always()` so a red leg still hands over its evidence. Re-running only the +affected platform is just `--platform `; the final verdict still needs +every required platform's report, because missing platforms are never silently +green. + +The command exits green only when the matrix passes and every requested +platform delivered a report. Run collections land under +`.rivet/verify-remote/` (adjust with `--output-directory`), and `--json` adds +the per-run conclusions and report paths to the matrix document. + ## What counts as complete RackUnit is the normal owner of shared Racket behavior. A product that claims diff --git a/rivet-cli/acceptance.rkt b/rivet-cli/acceptance.rkt index b65bb2ed..fd4d1a62 100644 --- a/rivet-cli/acceptance.rkt +++ b/rivet-cli/acceptance.rkt @@ -13,19 +13,29 @@ racket/port racket/string racket/system + "../rivet/distribution/crypto.rkt" "project.rkt") (provide default-acceptance-manifest-name acceptance-kinds + evidence-kinds supported-platforms (struct-out acceptance-suite) load-acceptance-manifest acceptance-report + evidence-entries run-acceptance-command) (define default-acceptance-manifest-name "rivet-acceptance.rktd") (define acceptance-kinds '(unit contract bridge native-build native-ui package release)) +;; Normalized evidence classes (issue #287): accessibility snapshots are the +;; primary cross-platform acceptance evidence, screenshots are supplemental +;; because native UIs are expected to look different, and traces plus logs +;; carry the interaction and failure narrative. Plain string paths in a +;; manifest stay legal and mean 'generic. +(define evidence-kinds + '(accessibility-snapshot screenshot interaction-trace log generic)) (define supported-platforms '(windows macos linux)) (define maximum-captured-output-bytes (* 64 1024)) @@ -95,10 +105,64 @@ (string-ci=? (path->string (first parts)) ".rivet") (string-ci=? (path->string (second parts)) "acceptance"))))) -(define (valid-evidence? value) - (and (list? value) - (andmap acceptance-evidence-path-string? value) - (= (length value) (length (remove-duplicates value string=?))))) +;; Manifest evidence entries are either a bare path string (legacy shape, +;; normalized to the 'generic kind) or a hash with exactly `path` and `kind`. +;; Internally each entry is a (path . kind) pair. +(define (parse-evidence-entry path suite-id raw) + (cond + [(string? raw) + (unless (acceptance-evidence-path-string? raw) + (manifest-error path + "invalid suite evidence path" + "suite" suite-id + "expected" + "a unique file path below .rivet/acceptance/" + "value" raw)) + (cons raw 'generic)] + [(hash? raw) + (define path-value + (hash-ref raw 'path + (lambda () + (manifest-error path + "invalid suite evidence entry" + "suite" suite-id + "expected" "a hash with path and kind" + "value" raw)))) + (unless (acceptance-evidence-path-string? path-value) + (manifest-error path + "invalid suite evidence path" + "suite" suite-id + "expected" + "a unique file path below .rivet/acceptance/" + "value" path-value)) + (define kind-value + (hash-ref raw 'kind + (lambda () + (manifest-error path + "invalid suite evidence entry" + "suite" suite-id + "expected" "a hash with path and kind" + "value" raw)))) + (unless (memq kind-value evidence-kinds) + (manifest-error path + "invalid suite evidence kind" + "suite" suite-id + "expected" evidence-kinds + "value" kind-value)) + (unless (= (length (hash-keys raw)) 2) + (manifest-error path + "invalid suite evidence entry" + "suite" suite-id + "expected" "a hash with exactly path and kind" + "value" raw)) + (cons path-value kind-value)] + [else + (manifest-error path + "invalid suite evidence entry" + "suite" suite-id + "expected" + "a path string or a hash with path and kind" + "value" raw)])) (define (parse-suite path raw) (unless (hash? raw) @@ -134,14 +198,17 @@ "suite" id "expected" "integer from 1 through 7200" "value" timeout-seconds)) - (define evidence (hash-ref raw 'evidence '())) - (unless (valid-evidence? evidence) + (define evidence-raw (hash-ref raw 'evidence '())) + (define evidence + (map (lambda (entry) (parse-evidence-entry path id entry)) evidence-raw)) + (unless (= (length evidence) + (length (remove-duplicates evidence equal?))) (manifest-error path "invalid suite evidence" "suite" id "expected" "unique file paths below .rivet/acceptance/" - "value" evidence)) + "value" evidence-raw)) (when (and (memq kind '(native-ui package release)) (null? evidence)) (manifest-error path @@ -280,7 +347,8 @@ (define (prepare-evidence! project suite) ;; Evidence belongs under Rivet's generated root, so removing stale files is ;; both safe and necessary: an earlier green run must not satisfy this run. - (for ([relative (in-list (acceptance-suite-evidence suite))]) + (for ([entry (in-list (acceptance-suite-evidence suite))]) + (define relative (car entry)) (define path (project-path project relative)) (for/fold ([parent (rivet-project-root project)]) ([part (in-list @@ -302,9 +370,23 @@ (delete-file path)))) (define (evidence-missing project suite) - (for/list ([relative (in-list (acceptance-suite-evidence suite))] - #:unless (file-exists? (project-path project relative))) - relative)) + (for/list ([entry (in-list (acceptance-suite-evidence suite))] + #:unless (file-exists? (project-path project (car entry)))) + (car entry))) + +;; First-class evidence records (issue #287): the report pins each declared +;; artifact's kind, presence, size, and digest so remote consumers can audit +;; what a green suite actually produced without trusting prose. +(define (evidence-entries project suite) + (for/list ([entry (in-list (acceptance-suite-evidence suite))]) + (define relative (car entry)) + (define path (project-path project relative)) + (define present (file-exists? path)) + (hash 'path relative + 'kind (symbol->string (cdr entry)) + 'present present + 'bytes (if present (file-size path) 'null) + 'sha256 (if present (string-downcase (sha256-file/hex path)) 'null)))) (define (execute-suite project suite) (define command (acceptance-suite-command suite)) @@ -334,7 +416,7 @@ (hash 'id (symbol->string (acceptance-suite-id suite)) 'kind (symbol->string (acceptance-suite-kind suite)) 'command (acceptance-suite-command suite) - 'evidence (acceptance-suite-evidence suite) + 'evidence (evidence-entries project suite) 'status "fail" 'duration-ms (inexact->exact @@ -371,7 +453,7 @@ (hash 'id (symbol->string (acceptance-suite-id suite)) 'kind (symbol->string (acceptance-suite-kind suite)) 'command (acceptance-suite-command suite) - 'evidence (acceptance-suite-evidence suite) + 'evidence (evidence-entries project suite) 'status (if passed? "pass" "fail") 'duration-ms (inexact->exact diff --git a/rivet-cli/inspect.rkt b/rivet-cli/inspect.rkt index 3f5b982d..0198c869 100644 --- a/rivet-cli/inspect.rkt +++ b/rivet-cli/inspect.rkt @@ -294,6 +294,9 @@ (list (action "inspect" "raco rivet inspect --json" #f) (action "diagnose" "raco rivet doctor --json" #f) (action "acceptance" "raco rivet acceptance --json" #f) + (action "verify-remote" + "raco rivet verify-remote --repo OWNER/REPO --ref HEAD" + #f) (action "inspect-schema" "raco rivet schema --json" #f) (action "save-schema-baseline" "raco rivet schema --output rivet-schema.json" diff --git a/rivet-cli/main.rkt b/rivet-cli/main.rkt index 0b38dfd1..a32f2b42 100644 --- a/rivet-cli/main.rkt +++ b/rivet-cli/main.rkt @@ -43,6 +43,9 @@ " raco rivet matrix ... aggregate per-platform acceptance reports into the\n" " cross-platform verification matrix\n" " raco rivet matrix ... --json emit the machine-readable matrix\n" + " raco rivet verify-remote --repo OWNER/REPO --ref REF [--workflow verify.yml]\n" + " trigger the per-platform acceptance legs and\n" + " fold their archived reports into the matrix\n" " raco rivet schema --json emit the current versioned API schema\n" " raco rivet schema --output write a schema compatibility baseline\n" " raco rivet schema check reject breaking changes from a baseline\n" @@ -120,6 +123,12 @@ ;; is judging (issue #287). (define run-matrix-command (command "matrix" 'run-matrix-command)) (exit (run-matrix-command matrix-args))] + [(cons "verify-remote" verify-remote-args) + ;; Remote orchestration triggers the platform legs and pulls their + ;; archived reports into the matrix from one editing host (issue #287). + (define run-verify-remote-command + (command "verify-remote" 'run-verify-remote-command)) + (exit (run-verify-remote-command verify-remote-args))] [(list "schema" "--json") (define schema-snapshot (command "codegen" 'schema-snapshot)) (write-json (schema-snapshot (current-project!))) diff --git a/rivet-cli/verify-remote.rkt b/rivet-cli/verify-remote.rkt new file mode 100644 index 00000000..f259d43f --- /dev/null +++ b/rivet-cli/verify-remote.rkt @@ -0,0 +1,414 @@ +#lang racket/base + +;; Remote verification orchestration (issue #287). The local operating system +;; is an editing environment, not the boundary of verification: this command +;; triggers the per-platform acceptance legs of one commit on GitHub Actions, +;; waits for the dispatched runs, downloads the archived `acceptance --json` +;; reports, and folds them into the shared verification matrix. It drives the +;; `gh` CLI and nothing else, so workflow names and artifact names stay the +;; product's own conventions. + +(require json + racket/file + racket/list + racket/match + racket/path + racket/port + racket/string + "acceptance.rkt" + "matrix.rkt") + +(provide run-verify-remote-command) + +(define default-workflow "verify.yml") +(define default-artifact-prefix "acceptance-report-") +(define default-timeout-seconds 1800) +(define default-output-directory ".rivet/verify-remote") +(define run-list-poll-seconds 5) + +(define usage-text + (string-append + "raco rivet verify-remote --repo OWNER/REPO --ref REF" + " [--workflow verify.yml] [--platform linux]..." + " [--artifact-prefix acceptance-report-]" + " [--timeout-seconds 1800] [--output-directory DIR] [--json]")) + +(define (verify-remote-error message . fields) + (apply raise-arguments-error 'rivet-verify-remote message fields)) + +(define (non-empty-text? value) + (and (string? value) (positive? (string-length value)))) + +(define (current-inexact-seconds) + (/ (current-inexact-milliseconds) 1000.0)) + +;; One collected workflow run: how it concluded, which acceptance artifacts +;; it archived, and which loaded platform reports it produced. +(struct run-outcome (run-id conclusion reports detail) #:transparent) + +;; One gh invocation: (values exit-code stdout stderr). The command never +;; decides trust from exit codes it did not see, so every caller funnels +;; through this injectable boundary. Tests parameterize `current-run-gh` +;; with a scripted replacement. +(define (default-run-gh argv) + (define gh (find-executable-path "gh")) + (unless gh + (verify-remote-error + "the gh CLI was not found on PATH" + "fix" "install the GitHub CLI and authenticate with `gh auth login`")) + (define-values (child stdout stdin stderr) + (apply subprocess #f #f #f gh argv)) + (close-output-port stdin) + (define stdout-thread (thread (lambda () (port->string stdout)))) + (define stderr-thread (thread (lambda () (port->string stderr)))) + (subprocess-wait child) + (values (subprocess-status child) + (thread-wait stdout-thread) + (thread-wait stderr-thread))) + +(define current-run-gh (make-parameter default-run-gh)) + +(define (trimmed text) + (string-trim (or text ""))) + +(define (run-gh-must who argv) + ;; who is the surrounding step; the failure detail carries gh's stderr. + (define-values (code stdout stderr) ((current-run-gh) argv)) + (unless (zero? code) + (verify-remote-error (format "~a failed" who) + "command" (string-join (cons "gh" argv) " ") + "detail" (trimmed stderr))) + stdout) + +(define (gh-json who stdout) + (with-handlers + ([exn:fail? + (lambda (error) + (verify-remote-error "gh emitted unparseable JSON" + "command" who + "message" (exn-message error)))]) + (read-json (open-input-string stdout)))) + +(define (resolve-commit-sha repo ref) + (define sha + (trimmed + (run-gh-must "commit resolution" + (list "api" (format "repos/~a/commits/~a" repo ref) + "--jq" ".sha")))) + (unless (regexp-match? #px"^[0-9a-f]{40}$" sha) + (verify-remote-error "commit resolution returned no commit sha" + "repo" repo + "ref" ref + "detail" sha)) + sha) + +;; Dispatch runs are identified against this baseline so a concurrent +;; session's older runs can never be mistaken for this verification. +(define (listed-dispatch-runs repo workflow) + (define runs + (gh-json "gh run list" + (run-gh-must + "run listing" + (list "run" "list" + "--repo" repo + "--workflow" workflow + "--event" "workflow_dispatch" + "--json" "databaseId,headSha" + "--limit" "50")))) + (unless (list? runs) + (verify-remote-error "gh run list returned no run array" + "repo" repo + "workflow" workflow)) + (for/list ([run (in-list runs)]) + (unless (and (hash? run) + (exact-integer? (hash-ref run 'databaseId #f)) + (string? (hash-ref run 'headSha #f))) + (verify-remote-error "gh run list returned a malformed run" + "repo" repo + "value" run)) + run)) + +(define (dispatch-platform repo workflow sha platform) + (run-gh-must "workflow dispatch" + (list "workflow" "run" workflow + "--repo" repo + "--ref" sha + "-f" (format "platform=~a" platform)))) + +(define (wait-for-dispatched-runs repo workflow sha baseline expected + timeout-seconds sleep now) + (define started (now)) + (let poll () + (define runs (listed-dispatch-runs repo workflow)) + (define discovered + (remove-duplicates + (for/list ([run (in-list runs)] + #:when (and (string=? (hash-ref run 'headSha) sha) + (not (member (hash-ref run 'databaseId) + baseline =)))) + (hash-ref run 'databaseId)) + =)) + (cond + [(>= (length discovered) expected) discovered] + [(> (- (now) started) timeout-seconds) + (verify-remote-error + "dispatched verification runs never appeared" + "workflow" workflow + "commit" sha + "expected-runs" expected + "discovered-runs" (length discovered) + "timeout-seconds" timeout-seconds)] + [else (sleep run-list-poll-seconds) (poll)]))) + +;; A red run must still hand over its archived reports: the watch exit code +;; only sets the conclusion, never skips the download, because a failing +;; platform's evidence is exactly what the matrix exists to show. +(define (collect-run run-id repo artifact-prefix platforms output-directory) + (define-values (watch-code _watch-stdout _watch-stderr) + ((current-run-gh) + (list "run" "watch" (number->string run-id) + "--repo" repo + "--exit-status" + "--interval" "10"))) + (define conclusion (if (zero? watch-code) "success" "failure")) + (define run-directory + (build-path output-directory (format "run-~a" run-id))) + ;; Stale artifacts from an earlier collection of the same run must not + ;; satisfy this one, mirroring the evidence hygiene of acceptance suites. + (when (directory-exists? run-directory) + (delete-directory/files run-directory)) + (define-values (_dl-code _dl-stdout _dl-stderr) + ((current-run-gh) + (list "run" "download" (number->string run-id) + "--repo" repo + "--dir" (path->string run-directory)))) + (define artifact-directories + (if (directory-exists? run-directory) + (for/list ([entry (in-list (directory-list run-directory #:build? #t))] + #:when (directory-exists? entry)) + entry) + '())) + (define reports + (append* + (for*/list ([artifact-directory (in-list artifact-directories)] + [artifact (in-value + (path->string + (file-name-from-path artifact-directory)))] + [platform (in-list platforms)] + #:when (string=? + artifact + (string-append artifact-prefix + (symbol->string platform)))) + (define json-paths + (sort (find-files (lambda (path) + (equal? (path-get-extension path) + #".json")) + artifact-directory) + stringstring)) + (unless (pair? json-paths) + (verify-remote-error + "the archived acceptance artifact contains no JSON report" + "artifact" artifact)) + (for/list ([path (in-list json-paths)]) + (hash 'platform (symbol->string platform) + 'path (path->string path) + 'report (load-acceptance-report path)))))) + (run-outcome run-id conclusion reports + (if (zero? watch-code) #f (format "run concluded ~a" conclusion)))) + +(define (parse-arguments args) + (let loop ([remaining args] + [repo #f] + [ref #f] + [workflow default-workflow] + [platforms '()] + [artifact-prefix default-artifact-prefix] + [timeout-seconds default-timeout-seconds] + [output-directory default-output-directory] + [json? #f]) + (match remaining + ['() + (unless repo + (verify-remote-error "missing required --repo option" + "usage" usage-text)) + (unless ref + (verify-remote-error "missing required --ref option" + "usage" usage-text)) + (define requested + (remove-duplicates (reverse (map string->symbol platforms)) eq?)) + (for ([platform (in-list requested)]) + (unless (memq platform supported-platforms) + (verify-remote-error "unknown --platform" + "expected" + (map symbol->string supported-platforms) + "value" platform))) + (unless (and (exact-integer? timeout-seconds) + (positive? timeout-seconds)) + (verify-remote-error "invalid --timeout-seconds" + "expected" "positive integer" + "value" timeout-seconds)) + (values repo + ref + workflow + (if (null? requested) supported-platforms requested) + artifact-prefix + timeout-seconds + output-directory + json?)] + [(list* "--repo" value tail) + (when repo (verify-remote-error "duplicate --repo option")) + (unless (regexp-match? #px"^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$" value) + (verify-remote-error "invalid --repo" + "expected" "OWNER/REPO" + "value" value)) + (loop tail value ref workflow platforms artifact-prefix + timeout-seconds output-directory json?)] + [(list* "--ref" value tail) + (when ref (verify-remote-error "duplicate --ref option")) + (unless (non-empty-text? value) + (verify-remote-error "invalid --ref" "value" value)) + (loop tail repo value workflow platforms artifact-prefix + timeout-seconds output-directory json?)] + [(list* "--workflow" value tail) + (unless (non-empty-text? value) + (verify-remote-error "invalid --workflow" "value" value)) + (loop tail repo ref value platforms artifact-prefix + timeout-seconds output-directory json?)] + [(list* "--platform" value tail) + (loop tail repo ref workflow (cons value platforms) artifact-prefix + timeout-seconds output-directory json?)] + [(list* "--artifact-prefix" value tail) + (unless (non-empty-text? value) + (verify-remote-error "invalid --artifact-prefix" "value" value)) + (loop tail repo ref workflow platforms value + timeout-seconds output-directory json?)] + [(list* "--timeout-seconds" value tail) + (define parsed + (and (string? value) + (regexp-match? #px"^[0-9]+$" value) + (string->number value))) + (loop tail repo ref workflow platforms artifact-prefix + parsed output-directory json?)] + [(list* "--output-directory" value tail) + (unless (non-empty-text? value) + (verify-remote-error "invalid --output-directory" "value" value)) + (loop tail repo ref workflow platforms artifact-prefix + timeout-seconds value json?)] + [(cons "--json" tail) + (when json? (verify-remote-error "duplicate --json option")) + (loop tail repo ref workflow platforms artifact-prefix + timeout-seconds output-directory #t)] + [_ + (verify-remote-error "unexpected argument" + "usage" usage-text + "arguments" args)]))) + +(define (display-outcomes repo ref sha workflow outcomes) + (printf "Rivet remote verification: ~a @ ~a (~a)\n" + repo (substring sha 0 (min 12 (string-length sha))) workflow) + (for ([outcome (in-list outcomes)]) + (define platforms + (remove-duplicates + (map (lambda (report) (hash-ref report 'platform)) + (run-outcome-reports outcome)))) + (printf " run ~a: ~a\n" + (run-outcome-run-id outcome) + (if (null? platforms) + (or (run-outcome-detail outcome) "no acceptance artifacts") + (format "~a — ~a, ~a report~a" + (string-join platforms ", ") + (or (run-outcome-detail outcome) "success") + (length (run-outcome-reports outcome)) + (if (= (length (run-outcome-reports outcome)) 1) + "" + "s")))))) + +(define (run-verify-remote-command args + #:run-gh [run-gh default-run-gh] + #:sleep [wake-after + (lambda (seconds) + (sleep seconds))] + #:now [now current-inexact-seconds]) + (define-values (repo ref workflow platforms artifact-prefix + timeout-seconds output-directory json?) + (parse-arguments args)) + ;; Bind the injectable boundary once so every helper sees the same scripted + ;; gh during tests. + (parameterize ([current-run-gh run-gh]) + (define sha (resolve-commit-sha repo ref)) + (define baseline + (map (lambda (run) (hash-ref run 'databaseId)) + (listed-dispatch-runs repo workflow))) + ;; Dispatch one leg per platform, then discover the runs those + ;; dispatches produced. Collection keys off artifact names, so a + ;; product's matrix workflow may name its jobs however it likes. + (for ([platform (in-list platforms)]) + (dispatch-platform repo workflow sha platform)) + (define run-ids + (wait-for-dispatched-runs repo workflow sha baseline (length platforms) + timeout-seconds wake-after now)) + ;; Nothing is written until the runs exist: a failed dispatch or a + ;; timeout leaves no empty output directory behind. + (make-directory* output-directory) + (define outcomes + (for/list ([run-id (in-list (sort run-ids <))]) + (with-handlers + ([exn:fail? + (lambda (error) + (run-outcome run-id "collection-failed" '() + (exn-message error)))]) + (collect-run run-id repo artifact-prefix platforms + output-directory)))) + (define loaded-reports + (append* (map (lambda (outcome) (run-outcome-reports outcome)) + outcomes))) + (define covered + (remove-duplicates + (map (lambda (report) (hash-ref report 'platform)) + loaded-reports))) + (define uncovered + (for/list ([platform (in-list platforms)] + #:unless (member (symbol->string platform) covered)) + (symbol->string platform))) + (define matrix + (and (pair? loaded-reports) + (matrix-report (map (lambda (report) + (hash-ref report 'report)) + loaded-reports)))) + (cond + [json? + (write-json + (hash 'format-version 1 + 'repo repo + 'ref ref + 'sha sha + 'workflow workflow + 'runs (for/list ([outcome (in-list outcomes)]) + (hash 'run-id (run-outcome-run-id outcome) + 'conclusion (run-outcome-conclusion outcome) + 'detail (or (run-outcome-detail outcome) 'null) + 'reports (for/list ([report + (in-list + (run-outcome-reports + outcome))]) + (hash 'platform + (hash-ref report 'platform) + 'path (hash-ref report 'path))))) + 'matrix (or matrix 'null))) + (newline)] + [else + (display-outcomes repo ref sha workflow outcomes) + (unless (null? uncovered) + (printf " missing platform legs: ~a\n" (string-join uncovered ", "))) + (when matrix + (display-matrix matrix))]) + (if (and matrix + (string=? (hash-ref matrix 'status) "pass") + (null? uncovered) + (not (findf (lambda (outcome) + (string=? (run-outcome-conclusion outcome) + "collection-failed")) + outcomes))) + 0 + 1))) diff --git a/rivet/tests/acceptance.rkt b/rivet/tests/acceptance.rkt index 36fae089..db78297b 100644 --- a/rivet/tests/acceptance.rkt +++ b/rivet/tests/acceptance.rkt @@ -178,7 +178,116 @@ RKT (check-equal? (hash-ref (first (hash-ref stale-report 'suites)) 'missing-evidence) '(".rivet/acceptance/stale.txt")) - (check-false (file-exists? stale-evidence))) + (check-false (file-exists? stale-evidence)) + + ;; Evidence is classified (issue #287): accessibility snapshots are the + ;; primary cross-platform record, screenshots are supplemental, and plain + ;; string paths stay legal as the generic kind. + (write-manifest + root + #<string + (hasheq 'format-version 1 + 'project "demo" + 'platform platform + 'manifest "rivet-acceptance.rktd" + 'configured-suites 1 + 'configured-requirements + (list (hasheq 'kind "native-ui" + 'platforms requirement-platforms)) + 'selected-suite 'null + 'status status + 'passed (if (string=? status "pass") 1 0) + 'failed (if (string=? status "pass") 0 1) + 'suites + (list (hasheq 'id (string-append platform "-ui") + 'kind "native-ui" + 'status status))))) + +;; Builds the injectable gh boundary: records every invocation, replays the +;; run list progression, and materializes archived artifacts as real files so +;; the collection path runs for real. +(define (make-fake-gh! + #:runs-before [runs-before "[]"] + #:runs-after [runs-after "[]"] + #:runs-after-on-call [runs-after-on-call 2] + #:watch-codes [watch-codes (hash)] + #:artifacts-by-run [artifacts-by-run (hash)] + #:dispatch-code [dispatch-code 0] + #:log [log #f]) + (define run-list-calls 0) + (define (flag-value argv flag) + (let loop ([rest argv]) + (match rest + [(list (== flag) next _ ...) next] + [(cons _ tail) (loop tail)] + [_ #f]))) + (lambda (argv) + (when log (log argv)) + (match argv + [(list "api" _ "--jq" ".sha") + (values 0 sha-under-test "")] + [(list "run" "list" _ ...) + (set! run-list-calls (+ run-list-calls 1)) + (values 0 + (if (>= run-list-calls runs-after-on-call) + runs-after + runs-before) + "")] + [(list "workflow" "run" _ ...) + (if (zero? dispatch-code) + (values 0 "" "") + (values dispatch-code "" "no workflow_dispatch trigger"))] + [(list "run" "watch" raw-id _ ...) + (define run-id (string->number raw-id)) + (values (hash-ref watch-codes run-id 0) "" "")] + [(list "run" "download" raw-id _ ...) + (define run-id (string->number raw-id)) + (define directory + (and run-id (flag-value argv "--dir"))) + (for ([spec (in-list (hash-ref artifacts-by-run run-id '()))] + #:when directory) + (define artifact-directory + (build-path (string->path directory) + (string-append "acceptance-report-" + (car spec)))) + (make-directory* artifact-directory) + (call-with-output-file + (build-path artifact-directory "report.json") + (lambda (output) (display (cdr spec) output)) + #:exists 'truncate/replace)) + (values 0 "" "")] + [_ + (values 1 "" (format "unexpected gh invocation: ~a" argv))]))) + +(define (all-green-artifacts) + (hash 101 (list (cons "linux" (report-text "linux" "pass"))) + 102 (list (cons "macos" (report-text "macos" "pass"))) + 103 (list (cons "windows" (report-text "windows" "pass"))))) + +(define (green-runs-json) + (jsexpr->string + (list (hasheq 'databaseId 101 'headSha sha-under-test) + (hasheq 'databaseId 102 'headSha sha-under-test) + (hasheq 'databaseId 103 'headSha sha-under-test)))) + +(define (capture-output thunk) + (define out (open-output-string)) + (define exit-code + (parameterize ([current-output-port out]) (thunk))) + (values exit-code (get-output-string out))) + +(define (sleep-counter) + (define count 0) + (values (lambda (_) (set! count (+ count 1))) + (lambda () count))) + +;; Three green legs fold into a passing matrix and exit 0. +(let-values ([(wake count) (sleep-counter)]) + (define work (make-temporary-file "rivet-verify-~a" 'directory)) + (define exit-code + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main" + "--output-directory" (path->string work)) + #:run-gh (make-fake-gh! + #:runs-after (green-runs-json) + #:artifacts-by-run (all-green-artifacts)) + #:sleep wake + #:now (lambda () 0))) + (check-equal? exit-code 0) + (check-equal? (count) 0)) + +;; The same command in --json mode reports the runs and the matrix verdict. +(let () + (define work (make-temporary-file "rivet-verify-~a" 'directory)) + (define-values (exit-code output) + (capture-output + (lambda () + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main" "--json" + "--output-directory" (path->string work)) + #:run-gh (make-fake-gh! + #:runs-after (green-runs-json) + #:artifacts-by-run (all-green-artifacts)) + #:sleep (lambda (_) (void)) + #:now (lambda () 0))))) + (check-equal? exit-code 0) + (define parsed (read-json (open-input-string output))) + (check-equal? (hash-ref parsed 'repo) "acme/demo") + (check-equal? (hash-ref parsed 'sha) sha-under-test) + (check-equal? (length (hash-ref parsed 'runs)) 3) + (check-equal? (hash-ref (hash-ref parsed 'matrix) 'status) "pass") + (check-equal? + (length (hash-ref (hash-ref parsed 'matrix) 'cells)) 3)) + +;; A red run still hands over its archived evidence; the failing platform's +;; report fails its matrix cell and the command exits 1. +(let () + (define work (make-temporary-file "rivet-verify-~a" 'directory)) + (define artifacts + (hash 101 (list (cons "linux" (report-text "linux" "fail"))) + 102 (list (cons "macos" (report-text "macos" "pass"))) + 103 (list (cons "windows" (report-text "windows" "pass"))))) + (define-values (exit-code output) + (capture-output + (lambda () + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main" + "--output-directory" (path->string work)) + #:run-gh (make-fake-gh! + #:runs-after (green-runs-json) + #:watch-codes (hash 101 1) + #:artifacts-by-run artifacts) + #:sleep (lambda (_) (void)) + #:now (lambda () 0))))) + (check-equal? exit-code 1) + (check-true (string-contains? output "FAIL"))) + +;; A leg that archives nothing leaves its platform uncovered; the matrix can +;; stay green for what it saw but the command still fails the verification. +(let () + (define work (make-temporary-file "rivet-verify-~a" 'directory)) + (define-values (exit-code output) + (capture-output + (lambda () + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main" "--platform" "linux" + "--output-directory" (path->string work)) + #:run-gh (make-fake-gh! + #:runs-after (jsexpr->string + (list (hasheq 'databaseId 101 + 'headSha sha-under-test))) + #:artifacts-by-run (hash)) + #:sleep (lambda (_) (void)) + #:now (lambda () 0))))) + (check-equal? exit-code 1) + (check-true (string-contains? output "missing platform legs: linux")) + (check-true (string-contains? output "run 101: no acceptance artifacts"))) + +;; The dispatcher waits across poll cycles before the runs appear. The +;; report only requires the linux leg, so a single green leg passes. +(let-values ([(wake count) (sleep-counter)]) + (define work (make-temporary-file "rivet-verify-~a" 'directory)) + (define exit-code + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main" "--platform" "linux" + "--output-directory" (path->string work)) + #:run-gh (make-fake-gh! + #:runs-after (jsexpr->string + (list (hasheq 'databaseId 101 + 'headSha sha-under-test))) + #:runs-after-on-call 4 + #:artifacts-by-run + (hash 101 (list (cons "linux" + (report-text "linux" "pass" + (list "linux")))))) + #:sleep wake + #:now (lambda () 0))) + (check-equal? exit-code 0) + (check-true (>= (count) 1))) + +;; Runs that never appear fail loudly instead of blocking forever. The +;; injected clock advances past the timeout on its second reading. +(let ([tick 0]) + (check-exn + #rx"dispatched verification runs never appeared" + (lambda () + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main") + #:run-gh (make-fake-gh!) + #:sleep (lambda (_) (void)) + #:now (lambda () + (set! tick (+ tick 100000)) + tick))))) + +;; A workflow without a dispatch trigger fails the dispatch step. +(check-exn + #rx"workflow dispatch failed" + (lambda () + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main") + #:run-gh (make-fake-gh! #:dispatch-code 1) + #:sleep (lambda (_) (void)) + #:now (lambda () 0)))) + +;; Argument validation stays fail-closed. +(check-exn #rx"missing required --repo" + (lambda () + (run-verify-remote-command + (list "--ref" "main") + #:run-gh (make-fake-gh!) + #:now (lambda () 0)))) +(check-exn #rx"invalid --repo" + (lambda () + (run-verify-remote-command + (list "--repo" "acme-demo" "--ref" "main") + #:run-gh (make-fake-gh!) + #:now (lambda () 0)))) +(check-exn #rx"unknown --platform" + (lambda () + (run-verify-remote-command + (list "--repo" "acme/demo" "--ref" "main" + "--platform" "freebsd") + #:run-gh (make-fake-gh!) + #:now (lambda () 0))))