Skip to content

desktop: make feature parity and cross-platform agent verification first-class gates #287

Closed

Description

@turinglambdaai

Problem

Rivet's macOS implementation tends to become the de facto reference implementation, while Windows and Linux lag behind in feature completeness and refinement.

This is partly inherent to Rivet's architecture: backend/schema/RVT1 are shared, but product behavior above that boundary can still drift across SwiftUI, WinUI, and GTK. It is also amplified by agent development: an agent working on macOS can validate macOS locally, but cannot directly observe the real WinUI or Linux/GTK result.

The result is two related failure modes:

  1. Feature parity drift — a feature is considered "done" after the macOS implementation works, while Windows/Linux remain incomplete or behaviorally different.
  2. Local-OS verification bias — an agent treats the current OS as the effective verification scope, instead of validating all required desktop targets.

Proposal

Make desktop parity and distributed platform verification first-class Rivet contracts.

1. Classify desktop capabilities

Define three categories:

  • Guaranteed — same semantics are required on Windows/macOS/Linux. Missing support is a release/acceptance failure.
  • Conditional capability — same API/semantics, but runtime availability may differ (for example tray, global shortcuts, secure storage, notifications). Adapters must report supported/unavailable/permission/provider state explicitly.
  • Platform-specific — intentionally native-only capabilities with no forced cross-platform abstraction.

The goal is semantic parity, not identical visual behavior.

2. Make feature parity executable

Extend the acceptance model so a cross-platform feature can declare its required platforms and suites.

Conceptually:

(feature 'export-document
  #:platforms '(windows macos linux)
  #:required? #t)

A Guaranteed desktop feature should not be considered complete until the required acceptance scenarios pass on all declared platforms.

Suggested definition of done:

  • shared semantic contract
  • backend/schema changes
  • macOS native implementation
  • Windows native implementation
  • Linux native implementation
  • macOS acceptance
  • Windows acceptance
  • Linux acceptance

This should build on the existing rivet-acceptance.rktd, native UI E2E, capability discovery, and schema gates rather than introducing another parallel test system.

3. Add cross-platform verification orchestration for agents

The local OS should be treated as an editing environment, not the boundary of verification.

A coding agent should be able to:

  1. run fast local checks;
  2. trigger Windows/macOS/Linux verification jobs;
  3. receive structured machine-readable results;
  4. inspect accessibility snapshots, screenshots, logs, and failing assertions;
  5. fix the implementation and re-run only the affected platform/suite.

Each platform job should emit a normalized result artifact, for example:

{
  "platform": "windows-x64",
  "result": "failed",
  "feature": "export-document",
  "stage": "native-ui",
  "expected": "Save dialog opened",
  "actual": "No dialog appeared",
  "accessibility_snapshot": "uia.json",
  "screenshot": "failure.png"
}

Accessibility/semantic evidence should be primary; screenshots are supplemental because native SwiftUI, WinUI, and GTK UIs are expected to look different.

4. Add a verification matrix view

Consider a command such as:

raco rivet matrix
raco rivet matrix --json

It could aggregate local and remote acceptance evidence:

Desktop verification matrix

                     local   remote
Racket contract       PASS      -
macOS native UI       PASS      -
Windows build           -      PASS
Windows native UI       -      FAIL
Linux build             -      PASS
Linux native UI         -      PASS

Cross-platform status: FAIL

The exact command name is not important; the important part is one machine-readable source of truth for platform completeness.

5. Use tiered verification

Avoid making every edit wait for the full release pipeline.

  • Tier 0 / local: unit, schema, codegen
  • Tier 1 / remote quick: per-platform compile + targeted tests
  • Tier 2 / PR: package + native UI acceptance
  • Tier 3 / release: installer, signing, real-hardware/system-integration checks

This keeps agent feedback fast while still preventing parity drift.

Agent contract

Add an explicit rule to the maintainer/agent contract along these lines:

A change touching a Guaranteed desktop capability is incomplete until the required acceptance suites pass on Windows, macOS, and Linux. The local operating system determines only which checks can run locally; it does not reduce the required verification scope.

Non-goals

  • Do not introduce a cross-platform UI DSL.
  • Do not require pixel-identical UIs across platforms.
  • Do not treat Linux desktop-service availability differences as bugs when they are explicitly modeled as capabilities.
  • Do not require full release/signing validation for every edit.

Expected outcome

Rivet should move from:

shared backend + three native implementations that can drift

toward:

shared behavior contract + native platform adapters whose parity is continuously proven

This would also make Rivet substantially more agent-native: an agent could develop from one OS while still receiving actionable evidence from the other platforms instead of assuming correctness from local results alone.

Activity

  1. turinglambdaai commented on Oct 19, 2026

    @turinglambdaai
    OwnerAuthor

    First slice landed in #289 (merged as ddaf976), built on the existing acceptance model rather than a parallel system:

    Now executable

    • raco rivet matrix <report.json>... [--json] aggregates the per-platform acceptance --json reports for one commit into the verification matrix from this proposal. Cell rules are deterministic: proven only by a passing suite of that kind on that platform, ran-but-failed fails, an unreported platform is missing, never silently green; all reports must describe one project; exit code is green only when every required cell is proven. A diagnostic --suite run is accepted but flagged partial.
    • The agent contract rule is in AGENTS.md: the local operating system determines only which checks can run locally; it does not reduce the required verification scope.
    • docs/product-acceptance.md adds the guaranteed / conditional / platform-specific capability classification — conditional adapters must report availability state explicitly instead of failing silently — and the tiering guidance stays grounded in the existing suite kinds (unit/contract/bridge ≈ Tier 0, native-build ≈ Tier 1, native-ui/package ≈ Tier 2, release ≈ Tier 3), so no new test taxonomy was introduced.

    Still open on this issue

    • Orchestrating the remote legs from one editing host (trigger Windows/macOS/Linux jobs and pull their artifacts automatically — today the agent fetches the archived reports and runs the matrix itself).
    • Richer normalized per-suite artifacts in the JSON (accessibility snapshots and screenshots as first-class fields; accessibility evidence already flows through native-ui suite evidence).

    Closing the remaining items keeps main green per slice; the matrix command is designed to be the aggregation point they land behind.

  2. turinglambdaai commented on Oct 19, 2026

    @turinglambdaai
    OwnerAuthor

    Both remaining items landed in #292 (rebased onto the AppImage handoff merge), closing the slice list:

    Remote orchestration from one editing host

    • raco rivet verify-remote dispatches each platform's acceptance leg of one commit on GitHub Actions, waits for the dispatched runs, downloads the archived acceptance --json reports, and folds them into raco rivet matrix — the aggregation point this issue designated. It drives the gh CLI and nothing else: workflow verify.yml and artifacts named acceptance-report-<platform> are the documented defaults, and --workflow/--artifact-prefix/--platform adapt to a product's own naming. Runs are identified against a pre-dispatch baseline so concurrent sessions cannot be confused, runs are collected by artifact name rather than job names, and a red run still hands over its archived evidence because the watch exit code only sets the conclusion, never skips the download. Re-running one affected platform is --platform <name>; the exit code stays green only when the matrix passes and every requested platform delivered a report. docs/ci.md / docs/ci.zh-CN.md now archive every leg's report with if: always() under exactly those names.

    Richer normalized per-suite artifacts

    • Manifest evidence entries accept a classified form (path + kind: accessibility-snapshot, screenshot, interaction-trace, log, or generic; plain strings stay legal as generic). The --json report now pins each declared entry's kind, presence, byte size, and SHA-256, so an archived report audits the artifacts it vouches for even when the files live in another platform's artifact store. Accessibility snapshots are documented as the primary cross-platform record, screenshots as supplemental. missing-evidence remains for existing tooling.

    With the matrix slice from #289, all five proposal sections now have first-class support: capability classification, executable parity (matrix), orchestration (verify-remote), the matrix view, and tiering guidance grounded in the existing suite kinds. The agent contract in AGENTS.md names both commands. Local validation: raco test rivet/tests/ green (10110 cases on Windows; the Linux roundtrip job drives the handoff tests end to end).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions