Skip to content

ci: add reusable remote consumer and backward compatibility verification - #1125

Draft
omermorad wants to merge 3 commits into
fix/v4-release-regressionsfrom
ci/previous-release-verification
Draft

omermorad wants to merge 3 commits into
fix/v4-release-regressionsfrom
ci/previous-release-verification

Conversation

@omermorad

@omermorad omermorad commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Run isolated consumer tests against selectable published npm packages or candidate tarballs on GitHub-hosted runners. Test source and package target are independent; each comparison first verifies the published baseline, then runs the identical consumer against the candidate with the same external dependency versions.

Two comparisons use the same reusable workflow:

  • Previous release consumers: frozen tests from the published release across ten consumers and Node 22, 24, and 26.
  • Current regression consumers: newly authored examples from the candidate commit, checked against both releases. These are explicitly separate from the historical suite. The TestBed subclass consumer currently demonstrates the unresolved source-compatibility break.

Separate entry workflows handle manual execution and baseline policy. Three composites handle tool setup, input preparation, and isolated execution. Node modules handle Git snapshots, package selection, integrity checks, installed-package verification, and results. Release simulation includes the comparison gate; nothing is published.

Every job summary shows separate baseline and target outcomes. Failed consumer output appears in the job log; artifacts retain test snapshots, package tarballs, result records, installed dependency inventories, logs, and lockfiles.

Validation:

  • The first hosted historical comparison passed all 30 consumer/runtime jobs, each running baseline and candidate.
  • The dedicated hosted regression consumer passed against published 4.1.2 and failed against candidate 4.2.0 with the two real TS2417 subclass errors: https://github.com/suites-dev/unit/actions/runs/42768289923/job/129670111652
  • Local standalone published-version execution and an intentionally broken-candidate control behaved as expected.
  • Seven harness tests pass, including baseline-failure blocking, candidate-failure classification, phase reporting, snapshot integrity, and installed-package provenance checks. Actionlint and formatting pass.

Stacked on #1123. Publication remains paused; the subclass API decision and other reported release findings remain unresolved. A green historical suite does not prove contracts absent from that suite. Manual dispatch becomes available after the workflow reaches the default branch.

Runbook: docs/e2e-verification.md.

@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.64%. Comparing base (355e8a0) to head (b143091).

Additional details and impacted files
@@                     Coverage Diff                     @@
##           fix/v4-release-regressions    #1125   +/-   ##
===========================================================
  Coverage                       93.64%   93.64%           
===========================================================
  Files                              29       29           
  Lines                             708      708           
  Branches                          124      124           
===========================================================
  Hits                              663      663           
  Misses                             27       27           
  Partials                           18       18           
Flag Coverage Δ
core.unit 97.00% <ø> (ø)
di.injectionjs 100.00% <ø> (ø)
di.inversify 87.73% <ø> (ø)
di.nestjs 87.25% <ø> (ø)
doubles.jest 100.00% <ø> (ø)
doubles.vitest 97.82% <ø> (ø)
unit 84.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant