diff --git a/AGENTS.md b/AGENTS.md index b0c1aec..b859563 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,6 +75,15 @@ development substitute, or refresh expected evidence merely to make a change green. A skipped, unrun, timed-out, or wrong-platform required suite means the work is not complete. +A change touching a guaranteed desktop capability is incomplete until the +required acceptance suites pass on Windows, macOS, and Linux. The local +operating system determines only which checks can run locally; it does not +reduce the required verification scope. Archive every platform leg's +`acceptance --json` report and prove the union with +`raco rivet matrix --json` (issue #287). Classify desktop +capabilities as guaranteed, conditional, or platform-specific as documented +in `docs/product-acceptance.md`. + For this repository, inspect the nearest implementation and its tests before editing. The shared wire contract is documented in `docs/protocol.md` and exercised by `rivet/tests/protocol-golden.txt` across Racket, C++, Swift, and Kotlin. diff --git a/CHANGELOG.md b/CHANGELOG.md index efbe64f..58ae4f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,17 @@ ## Unreleased +- Make cross-platform verification first-class. `raco rivet matrix + ... [--json]` folds the per-platform `acceptance --json` + reports for one commit into a single machine-readable verdict: a required + kind/platform cell is proven only by a passing suite on that platform, + missing platforms are never silently green, and the exit code is green only + when every required cell is proven — so an agent editing on one OS can + still be judged by the other platforms' evidence. `docs/product-acceptance.md` + now defines the guaranteed / conditional / platform-specific capability + classification, and the maintainer contract states that the local + operating system does not reduce the required verification scope. + - Ship the Windows runtime source list as a machine-consumable property sheet. `platform/windows/rivet-runtime-sources.props` is now the single author of the host library's `` set — `rivet-host.vcxproj` diff --git a/docs/agent-native.md b/docs/agent-native.md index 569fe08..9624520 100644 --- a/docs/agent-native.md +++ b/docs/agent-native.md @@ -210,6 +210,14 @@ output bounds, checks exit status and declared artifacts, and returns a machine-readable report suitable for a required CI check. See [product acceptance evidence](product-acceptance.md). +The local operating system bounds only which suites can run locally, not the +required verification scope. Archive every platform leg's +`acceptance --json` report as a CI artifact and judge the union with +`raco rivet matrix --json`: a change touching a guaranteed +desktop capability is incomplete until the required suites are proven on +every declared platform. Fix, then re-run only the affected platform's leg +and re-aggregate. + `build` validates the Racket schema and regenerates typed native clients. `package` creates and immediately verifies a development distributable. `verify` can audit the artifact again without rebuilding it. Production mode diff --git a/docs/product-acceptance.md b/docs/product-acceptance.md index 9d88686..b5a1ed3 100644 --- a/docs/product-acceptance.md +++ b/docs/product-acceptance.md @@ -89,6 +89,65 @@ commit with the acceptance report. A dirty, diverged, or unreachable checkout is a reason to stop and report the state, not a reason to stash, reset, or claim evidence against an unknown base. +## Classify desktop capabilities + +Not every desktop capability carries the same parity obligation. Declare each +one as one of three categories and let the category drive the requirements +list: + +- **Guaranteed** — the same semantics are required on every declared desktop + platform; a missing platform is a release failure. These become + `requirements` entries with all desktop platforms and suites of the matching + kinds. +- **Conditional capability** — the same API and semantics everywhere, but + runtime availability may differ: tray, global shortcuts, secure storage, + notifications. The adapter must report supported/unavailable/permission + state explicitly instead of failing silently, and suites assert the + reported state, not unconditional success. +- **Platform-specific** — intentionally native-only behavior with no forced + cross-platform abstraction. It is excluded from the requirements list; + platform-specific does not mean unverified, it means verified by that + platform's own suites. + +The goal is semantic parity, not identical visual behavior. Native SwiftUI, +WinUI, and GTK presentations are expected to differ; the acceptance evidence +(accessibility state, RPC effects, declared artifacts) must not. + +A Guaranteed feature is not complete until every declared platform has run +its suites — which is what the matrix below proves. + +## Aggregate the cross-platform matrix + +The local operating system is an editing environment, not the boundary of +verification. Each platform leg archives its acceptance report as a CI +artifact; the matrix folds the reports for one commit into a single verdict: + +```bash +raco rivet matrix windows-report.json macos-report.json linux-report.json +raco rivet matrix windows-report.json --json +``` + +```text +Rivet verification matrix: Taskly - 3 reports + + requirement windows macos linux + contract PASS PASS PASS + native-ui PASS PASS FAIL + +Cross-platform status: FAIL +``` + +The rules are deterministic: a cell is proven only by a passing suite of that +kind in a report from that platform; ran-but-failed evidence fails the cell; +no evidence for a platform means missing, never silently green. All reports +must describe the same project, so one product's green leg cannot vouch for +another. A diagnostic `--suite` run is accepted and flagged as partial in the +matrix, but merge gates should feed complete reports. The exit code is green +only when every required kind/platform cell is proven. + +Feed the reports for the commit under verification; the matrix aggregates +what it is given and does not heuristically prefer fresher runs. + ## What counts as complete RackUnit is the normal owner of shared Racket behavior. A product that claims diff --git a/rivet-cli/acceptance.rkt b/rivet-cli/acceptance.rkt index 7728094..b65bb2e 100644 --- a/rivet-cli/acceptance.rkt +++ b/rivet-cli/acceptance.rkt @@ -17,6 +17,7 @@ (provide default-acceptance-manifest-name acceptance-kinds + supported-platforms (struct-out acceptance-suite) load-acceptance-manifest acceptance-report diff --git a/rivet-cli/main.rkt b/rivet-cli/main.rkt index bf486a4..0b38dfd 100644 --- a/rivet-cli/main.rkt +++ b/rivet-cli/main.rkt @@ -40,6 +40,9 @@ " raco rivet inspect --json emit the agent-readable project contract\n" " raco rivet acceptance run the current platform's declared acceptance suites\n" " raco rivet acceptance --json emit bounded machine-readable acceptance evidence\n" + " raco rivet matrix ... aggregate per-platform acceptance reports into the\n" + " cross-platform verification matrix\n" + " raco rivet matrix ... --json emit the machine-readable matrix\n" " raco rivet schema --json emit the current versioned API schema\n" " raco rivet schema --output write a schema compatibility baseline\n" " raco rivet schema check reject breaking changes from a baseline\n" @@ -111,6 +114,12 @@ (define run-acceptance-command (command "acceptance" 'run-acceptance-command)) (exit (run-acceptance-command (current-project!) acceptance-args))] + [(cons "matrix" matrix-args) + ;; Matrix aggregation consumes report artifacts, not a project checkout, + ;; so it runs anywhere: on an editing host far from the platform legs it + ;; is judging (issue #287). + (define run-matrix-command (command "matrix" 'run-matrix-command)) + (exit (run-matrix-command matrix-args))] [(list "schema" "--json") (define schema-snapshot (command "codegen" 'schema-snapshot)) (write-json (schema-snapshot (current-project!))) diff --git a/rivet-cli/matrix.rkt b/rivet-cli/matrix.rkt new file mode 100644 index 0000000..d1eebd7 --- /dev/null +++ b/rivet-cli/matrix.rkt @@ -0,0 +1,300 @@ +#lang racket/base + +;; Cross-platform verification matrix (issue #287). Every platform leg already +;; emits a machine-readable `raco rivet acceptance --json` report; the matrix +;; folds those reports into one verdict so an agent editing on any OS can see +;; which required kind/platform cells are proven, failing, or missing. The +;; local operating system is an editing environment, not the boundary of +;; verification. + +(require json + racket/file + racket/list + racket/match + racket/string + "acceptance.rkt") + +(provide load-acceptance-report + matrix-report + display-matrix + run-matrix-command) + +;; Report files are JSON, so every enum arrives as a string; normalize at the +;; load boundary so aggregation, ordering, and cell lookup work on symbols. +(define (kind-symbol path value) + (or (for/or ([kind (in-list acceptance-kinds)]) + (and (string=? value (symbol->string kind)) kind)) + (report-argument-error "acceptance report has an invalid member" + "path" path + "member" 'kind + "expected" (format "one of ~a" acceptance-kinds) + "value" value))) + +(define (platform-symbol path value) + (or (for/or ([platform (in-list supported-platforms)]) + (and (string=? value (symbol->string platform)) platform)) + (report-argument-error "acceptance report has an invalid member" + "path" path + "member" 'platform + "expected" "windows, macos, or linux" + "value" value))) + +(define (platform-list path value) + (unless (and (list? value) + (pair? value) + (andmap string? value)) + (report-argument-error "acceptance report has an invalid member" + "path" path + "member" 'platforms + "expected" "non-empty list of platform names" + "value" value)) + (map (lambda (item) (platform-symbol path item)) value)) + +(define (report-argument-error message . fields) + (apply raise-arguments-error 'rivet-matrix message fields)) + +(define (json-member report path key predicate expected) + (define item + (hash-ref report key + (lambda () + (report-argument-error + "acceptance report is missing a required member" + "path" path + "member" key)))) + (unless (predicate item) + (report-argument-error + "acceptance report has an invalid member" + "path" path + "member" key + "expected" expected + "value" item)) + item) + +(define (report-requirement-pairs path requirement) + (unless (hash? requirement) + (report-argument-error "acceptance report requirement must be an object" + "path" path + "value" requirement)) + (define kind + (kind-symbol path + (json-member requirement path 'kind string? "kind name string"))) + (define platforms + (platform-list + path + (json-member requirement path 'platforms list? + "non-empty list of platform names"))) + (for/list ([platform (in-list platforms)]) (cons kind platform))) + +;; A report is trusted only after its shape validates; an aggregation gate +;; that silently tolerates malformed evidence would reinterpret completion. +(define (load-acceptance-report path) + (unless (file-exists? path) + (report-argument-error "acceptance report file is missing" "path" path)) + (define raw + (with-handlers + ([exn:fail? + (lambda (error) + (report-argument-error + "acceptance report is not valid JSON" + "path" path + "message" (exn-message error)))]) + (call-with-input-file path read-json))) + (unless (hash? raw) + (report-argument-error "acceptance report must be a JSON object" + "path" path)) + (json-member raw path 'format-version + (lambda (value) (eqv? value 1)) + "1") + (define project + (json-member raw path 'project + (lambda (value) (and (string? value) (positive? (string-length value)))) + "non-empty string")) + (define platform + (platform-symbol + path + (json-member raw path 'platform string? "platform name string"))) + (define selected-suite + (json-member raw path 'selected-suite + (lambda (value) (or (string? value) (eq? value 'null))) + "suite id string or null")) + (define raw-requirements + (json-member raw path 'configured-requirements list? "list of objects")) + (define raw-suites + (json-member raw path 'suites + (lambda (value) (and (list? value) (pair? value))) + "non-empty list of objects")) + (define suites + (for/list ([suite (in-list raw-suites)]) + (unless (hash? suite) + (report-argument-error "acceptance report suite must be an object" + "path" path)) + (hash 'id (json-member suite path 'id + (lambda (value) + (and (string? value) (positive? (string-length value)))) + "non-empty string") + 'kind (kind-symbol path + (json-member suite path 'kind + string? "kind name string")) + 'status (json-member suite path 'status + (lambda (value) (or (string=? value "pass") + (string=? value "fail"))) + "\"pass\" or \"fail\"")))) + (hash 'path (if (path? path) (path->string path) path) + 'project project + 'platform platform + 'partial (not (eq? selected-suite 'null)) + 'requirements + (remove-duplicates + (append* + (for/list ([requirement (in-list raw-requirements)]) + (report-requirement-pairs path requirement)))) + 'suites suites)) + +;; Cell rule: a required kind/platform pair is proven when at least one +;; supplied report for that platform shows a passing suite of that kind; any +;; ran-but-not-passed evidence fails the cell; nothing ran means missing. +;; Feeding the current commit's reports is the caller's curation duty, so the +;; rule stays deterministic instead of heuristically picking fresher runs. +(define (cell-status kind platform reports) + (define relevant + (append* + (for/list ([report (in-list reports)] + #:when (eq? platform (hash-ref report 'platform))) + (for/list ([suite (in-list (hash-ref report 'suites))] + #:when (eq? kind (hash-ref suite 'kind))) + (hash-ref suite 'status))))) + (cond [(member "pass" relevant) "pass"] + [(pair? relevant) "fail"] + [else "missing"])) + +;; Reports must describe one product: a matrix spanning projects would let a +;; green leg of one product vouch for another. +(define (matrix-report reports) + (unless (pair? reports) + (report-argument-error "matrix aggregation needs at least one report")) + (define projects + (remove-duplicates (map (lambda (report) (hash-ref report 'project)) reports))) + (unless (= 1 (length projects)) + (report-argument-error "matrix reports must all describe the same project" + "projects" projects)) + (define kind-order + (lambda (left right) + (define left-kind (index-of acceptance-kinds (car left))) + (define right-kind (index-of acceptance-kinds (car right))) + (or (< left-kind right-kind) + (and (= left-kind right-kind) + (< (index-of supported-platforms (cdr left)) + (index-of supported-platforms (cdr right))))))) + (define requirements + (sort (remove-duplicates + (append* (map (lambda (report) (hash-ref report 'requirements)) reports))) + kind-order)) + (define cells + (for/list ([requirement (in-list requirements)]) + (hash 'kind (symbol->string (car requirement)) + 'platform (symbol->string (cdr requirement)) + 'status (cell-status (car requirement) (cdr requirement) reports)))) + (hash 'format-version 1 + 'project (first projects) + 'reports (for/list ([report (in-list reports)]) + (hash 'path (hash-ref report 'path) + 'platform (symbol->string (hash-ref report 'platform)) + 'partial (hash-ref report 'partial) + 'status (if (for/and ([suite (in-list (hash-ref report 'suites))]) + (string=? (hash-ref suite 'status) "pass")) + "pass" + "fail"))) + 'cells cells + 'status (if (andmap (lambda (cell) (string=? (hash-ref cell 'status) "pass")) + cells) + "pass" + "fail"))) + +(define (string-pad-right text width) + (if (>= (string-length text) width) + text + (string-append text (make-string (- width (string-length text)) #\space)))) + +(define (string-pad-left text width) + (if (>= (string-length text) width) + text + (string-append (make-string (- width (string-length text)) #\space) text))) + +(define (cell-text status) + (case status [("pass") "PASS"] [("fail") "FAIL"] [else "-"])) + +(define (display-matrix matrix) + (define kinds + (remove-duplicates (map (lambda (cell) (hash-ref cell 'kind)) + (hash-ref matrix 'cells)))) + (define platforms (map symbol->string supported-platforms)) + (define cell-map + (for/hash ([cell (in-list (hash-ref matrix 'cells))]) + (values (cons (hash-ref cell 'kind) (hash-ref cell 'platform)) + (hash-ref cell 'status)))) + (define kind-width + (for/fold ([width (string-length "requirement")]) + ([kind (in-list kinds)]) + (max width (string-length kind)))) + (define report-count (length (hash-ref matrix 'reports))) + (printf "Rivet verification matrix: ~a - ~a report~a\n" + (hash-ref matrix 'project) report-count + (if (= report-count 1) "" "s")) + (newline) + (printf " ~a" (string-pad-right "requirement" kind-width)) + (for ([platform (in-list platforms)]) + (printf " ~a" (string-pad-left platform 8))) + (newline) + (for ([kind (in-list kinds)]) + (printf " ~a" (string-pad-right kind kind-width)) + (for ([platform (in-list platforms)]) + (printf " ~a" + (string-pad-left + (cell-text (hash-ref cell-map (cons kind platform) "missing")) 8))) + (newline)) + (newline) + (for ([report (in-list (hash-ref matrix 'reports))]) + (printf " ~a ~a~a ~a\n" + (string-pad-left (hash-ref report 'platform) 8) + (string-pad-left (hash-ref report 'status) 4) + (if (hash-ref report 'partial) + " (partial: --suite run)" + (make-string 24 #\space)) + (hash-ref report 'path))) + (newline) + (printf "Cross-platform status: ~a\n" (string-upcase (hash-ref matrix 'status)))) + +(define (parse-arguments args) + (let loop ([remaining args] + [json? #f] + [reports '()]) + (match remaining + ['() + (unless (pair? reports) + (raise-arguments-error + 'rivet-matrix + "expected at least one acceptance report path" + "usage" "raco rivet matrix ... [--json]")) + (values json? (reverse reports))] + [(cons "--json" tail) + (when json? + (raise-arguments-error 'rivet-matrix "duplicate --json option")) + (loop tail #t reports)] + [(cons path tail) + (loop tail json? (cons path reports))] + [_ + (raise-arguments-error + 'rivet-matrix + "expected ... [--json]" + "arguments" args)]))) + +(define (run-matrix-command args) + (define-values (json? paths) (parse-arguments args)) + (define reports (map load-acceptance-report paths)) + (define matrix (matrix-report reports)) + (if json? + (begin + (write-json matrix) + (newline)) + (display-matrix matrix)) + (if (string=? (hash-ref matrix 'status) "pass") 0 1)) diff --git a/rivet/tests/acceptance-matrix.rkt b/rivet/tests/acceptance-matrix.rkt new file mode 100644 index 0000000..99d6a89 --- /dev/null +++ b/rivet/tests/acceptance-matrix.rkt @@ -0,0 +1,180 @@ +#lang racket/base + +;; The cross-platform matrix is the aggregation gate over per-platform +;; acceptance reports: it may prove a cell only from a passing suite on that +;; platform's report, must fail closed on malformed evidence, and must never +;; let one product's green leg vouch for another. + +(require json + racket/file + racket/list + racket/port + racket/runtime-path + racket/string + rackunit + "../../rivet-cli/matrix.rkt") + +(define temp-root (make-temporary-file "rivet-matrix-test-~a" 'directory)) + +(define (write-report! name content) + (define path (build-path temp-root name)) + (call-with-output-file path + #:exists 'truncate/replace + (lambda (output) (display content output))) + path) + +(define (suite id kind status) + (hash 'id id 'kind kind 'status status)) + +(define (report #:project [project "Taskly"] + #:platform platform + #:selected-suite [selected-suite 'null] + #:requirements requirements + #:suites suites) + ;; A real report carries the whole manifest policy in + ;; configured-requirements — every required kind with every platform it + ;; must run on — not just the declaring leg's row. + (hash 'format-version 1 + 'project project + 'platform platform + 'manifest "rivet-acceptance.rktd" + 'configured-suites (length suites) + 'configured-requirements + (for/list ([requirement (in-list requirements)]) + (hash 'kind (car requirement) + 'platforms (cdr requirement))) + 'selected-suite selected-suite + 'status (if (andmap (lambda (suite) + (string=? (hash-ref suite 'status) "pass")) + suites) + "pass" + "fail") + 'passed (count (lambda (suite) (string=? (hash-ref suite 'status) "pass")) + suites) + 'failed (- (length suites) + (count (lambda (suite) + (string=? (hash-ref suite 'status) "pass")) + suites)) + 'suites suites)) + +;; JSON round trips symbols through strings; report files store strings. +(define (jsonify value) + (define (stringify item) + (cond [(symbol? item) (symbol->string item)] + [(hash? item) + (for/hash ([(key item) (in-hash item)]) + (values key (stringify item)))] + [(list? item) (map stringify item)] + [else item])) + (with-output-to-string (lambda () (write-json (stringify value))))) + +(define (file-report path-name report) + (define path (write-report! path-name (jsonify report))) + (load-acceptance-report path)) + +(dynamic-wind + void + (lambda () + ;; Full green evidence on every required platform proves the matrix. + (define windows + (file-report + "windows.json" + (report #:platform "windows" + #:requirements '((contract windows linux) (native-ui windows linux)) + #:suites (list (suite "core" "contract" "pass") + (suite "windows-ui" "native-ui" "pass"))))) + (define linux + (file-report + "linux.json" + (report #:platform "linux" + #:requirements '((contract windows linux) (native-ui windows linux)) + #:suites (list (suite "core" "contract" "pass") + (suite "linux-ui" "native-ui" "fail"))))) + (define matrix (matrix-report (list windows linux))) + (check-equal? (hash-ref matrix 'project) "Taskly") + (check-equal? (hash-ref matrix 'status) "fail") + (check-equal? + (hash-ref matrix 'cells) + (list (hash 'kind "contract" 'platform "windows" 'status "pass") + (hash 'kind "contract" 'platform "linux" 'status "pass") + (hash 'kind "native-ui" 'platform "windows" 'status "pass") + (hash 'kind "native-ui" 'platform "linux" 'status "fail"))) + (check-equal? (length (hash-ref matrix 'reports)) 2) + + ;; A platform with no report is missing, never silently green. + (define windows-only + (matrix-report (list windows))) + (check-equal? (hash-ref windows-only 'status) "fail") + (check-true + (pair? (member (hash 'kind "contract" 'platform "linux" 'status "missing") + (hash-ref windows-only 'cells))) + "an unreported platform must be a missing cell") + + ;; A partial (--suite) run still contributes its own kind's evidence but is + ;; flagged, so a gate reading the artifact sees the reduced scope. + (define partial + (file-report + "partial.json" + (report #:platform "macos" + #:selected-suite "core" + #:requirements '((contract macos)) + #:suites (list (suite "core" "contract" "pass"))))) + (check-true (hash-ref partial 'partial)) + (define partial-matrix (matrix-report (list partial))) + (check-equal? (hash-ref partial-matrix 'cells) + (list (hash 'kind "contract" 'platform "macos" 'status "pass"))) + (check-true + (hash-ref (first (hash-ref partial-matrix 'reports)) 'partial)) + + ;; Malformed evidence fails closed instead of aggregating silently. + (for ([text (in-list + '("not json" + "[]" + "{\"format-version\": 2, \"project\": \"Taskly\"}" + "{\"format-version\": 1, \"project\": \"Taskly\"}"))]) + (define path (write-report! "broken.json" text)) + (check-exn #rx"acceptance report" + (lambda () (load-acceptance-report path)) + (format "malformed report must be rejected: ~a" text))) + (check-exn #rx"acceptance report file is missing" + (lambda () (load-acceptance-report + (build-path temp-root "absent.json")))) + + ;; One product per matrix: a green leg of another product cannot vouch. + (define other + (file-report + "other.json" + (report #:project "Other" + #:platform "macos" + #:requirements '((contract macos)) + #:suites (list (suite "core" "contract" "pass"))))) + (check-exn #rx"same project" + (lambda () (matrix-report (list windows other)))) + + ;; The command exit code follows the matrix verdict and --json emits it. + (define exit-code + (parameterize ([current-output-port (open-output-nowhere)]) + (run-matrix-command + (list (path->string (build-path temp-root "windows.json")) + (path->string (build-path temp-root "linux.json")))))) + (check-equal? exit-code 1) + (define json-output (open-output-string)) + (define json-exit-code + (parameterize ([current-output-port json-output]) + (run-matrix-command + (list "--json" + (path->string (build-path temp-root "partial.json")))))) + (check-equal? json-exit-code 0) + (define parsed (read-json (open-input-string (get-output-string json-output)))) + (check-equal? (hash-ref parsed 'status) "pass") + (check-equal? (hash-ref parsed 'project) "Taskly") + + ;; Argument errors: no reports, unknown options. + (check-exn #rx"at least one acceptance report" + (lambda () (run-matrix-command '()))) + (check-exn #rx"duplicate --json" + (lambda () (run-matrix-command + (list "--json" "--json" "report.json"))))) + (lambda () + (when (directory-exists? temp-root) + (delete-directory/files temp-root))))