Skip to content

test: add a stable provider compatibility matrix and contract fixtures #47

Description

@EmersonBraun

Parent: #37
Depends on: #41, #42, #43, #44, #45

Objective

Prevent provider-specific regressions by making compatibility claims executable and offline by default.

Scope

  • Define the stable-provider matrix for grok-cli and opencode-cli, including transport, supported trust modes, required lenses, output envelope, and failure semantics.
  • Add deterministic fixtures for ACP/headless startup, strict JSON output, malformed output, timeout, auth failure, permission failure, unavailable binary, and redaction.
  • Run the matrix against the same engine contracts used by the CLI, not duplicated test-only logic.
  • Keep live provider smoke tests opt-in and clearly separated from pull-request checks.
  • Fail the matrix when a declared stable provider lacks a fixture or contract assertion.
  • Document how a provider moves from experimental to stable and how unknown versions are handled.

Acceptance criteria

  • Offline CI proves both stable CLI providers satisfy the common contract.
  • Each provider/transport combination declares whether it is supported; unsupported combinations are tested as explicit failures.
  • No fixture contains credentials or requires a live network.
  • A provider output change cannot silently bypass schema validation.
  • The compatibility matrix is reviewable as a small generated or declarative artifact.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions