ci: verify and bind matrix evidence to release and scorecard claims #2601

Goal

Make local install-matrix results admissible as release and scorecard evidence only when they are schema-valid, current, and bound to the exact Base revision and test environment.

Background

The local four-account matrix (#2559–#2563) produces JSON attached to PRs, but the current plan has no verifier or release-gate consumer. A manually selected or stale result must not support a January compatibility claim or replace the immutable macOS/Ubuntu BOM evidence.

Tracking: #2555. Depends on #2559, #2561, and #2562.

Scope

• Define a versioned matrix-result schema covering Base commit, scenario version, platform, macOS version, architecture, route, shell, account/cell, preconditions, step results, and completion time.
• Add a read-only verifier that rejects stale commits, missing required cells, schema violations, edited/ambiguous results, and mismatched scenario versions.
• Add a release/scorecard adapter that records matrix evidence separately from required hosted BOM evidence.
• Preserve redacted logs and finding IDs without copying secrets or personal data into release artifacts.
• Add seeded invalid fixtures for stale, partial, wrong-route, wrong-shell, and tampered results.

Acceptance Criteria

• Results from test: add the local install-matrix harness (scenario runner, runas contract, JSON results) #2559–test: matrix scenarios - agent shell modes, idempotency, and uninstall residue #2563 validate against a committed schema and bind to a full Base SHA.
• The verifier fails closed for stale, partial, tampered, or wrong-cell evidence.
• A release checklist and docs+test: January readiness scorecard - every public claim mapped to evidence #2595 can consume a verified result by stable evidence ID.
• Matrix evidence cannot substitute for the required immutable BOM combinations.
• A known-bad fixture proves the verifier goes red.

Validation

BASE_CLI_SOURCE_DIR=../base-cli/lib/python \
PYTHONPATH=../base-cli/lib/python:lib/python:cli/python \
python -m pytest tests/contracts <new matrix evidence tests>
python tools/release_train_check.py --format json
git diff --check

Non-Goals

• Do not turn the four local accounts into hosted CI.
• Do not sign or publish private machine details.
• Do not weaken the immutable release-BOM gate.

Project Fields

• Status: Ready
• Priority: P1
• Area: CI
• Initiative: Contract Hardening
• Size: M
• Milestone: v1.12.0
• Category: ci

Agent Assignment

Assignee: codeforester. Implementation-ready after the matrix result shape is confirmed; use an issue-backed worktree.