Skip to content

ci(astro-kit): kit visual compares with --ci and reports on every pull request (PR B of 2) - #586

Merged
ABB65 merged 6 commits into
mainfrom
feat/kit-visual-ci-enforce
Oct 16, 2026
Merged

ABB65 merged 6 commits into
mainfrom
feat/kit-visual-ci-enforce

Conversation

@ABB65

@ABB65 ABB65 commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

The kit visual CI, PR B of 2: CI now compares and fails, and kit-visual can be a required check. Reader: t2. GENERIC.

#585 (PR A) put the kit visual in its own job inside Playwright's image and committed the first linux baselines (831 PNGs, compare run green with 0 written). This PR makes the job enforce them.

The change (.github/workflows/kit-visual.yml)

  • --ci on every run except a hand-run with update. A drawing that moves (more than MAX_DIFF_RATIO, 0.2%), a size that changes, or a fixture with no linux baseline fails the job.
  • It reports on every pull request, so it can be required (ORK's call, after t2's note that a path-filtered workflow never reports):
    • There is no workflow-level paths: filter.
    • The changes job (plain ubuntu, about 10 s) checks out with fetch-depth: 0 and greps git diff --name-only <base> HEAD for the watched paths: packages/astro-kit/, templates/astro-starter/, packages/types/, packages/query/, scripts/local-sdk.mjs, pnpm-lock.yaml and this workflow. A hand-run (no pull request) always draws.
    • kit-visual has needs: changes and if: needs.changes.outputs.kit == 'true'. A job skipped by its if counts as passing for a required check, so a pull request outside those paths gets a passing kit-visual without pulling the image.
  • Uploads only what is useful. On a failure, only the failing shots (*.actual.png, *.diff.png) go to the artifact astro-kit-visual-linux-failed. With update, every baseline goes to astro-kit-visual-linux. A passing compare uploads nothing; before, it was 50 MB per run.
  • tests/kit-visual-ci.test.ts now also pins no paths: filter, the if gate, every watched path in the grep, and --ci on the run line (4/4).

Branch protection

The check to require is kit-visual (the job id, in workflow "Kit visual"). changes reports too, but it does not need to be required. Changing the protection setting is ORK's, after this merges.

Proof

  • This PR's own run touches the workflow, so kit-visual draws and compares with --ci. It passed in 4m49s, with nothing written: run 38901105701.
  • Scratch draft PR test: kit-visual gate probe (do not merge) #587 changed one fixture (closed afterwards, branch deleted). kit-visual must fail and upload the failed shots.
    • A heading text change failed, as it should: run 38901900689. stats/three-plain @390 changed size (390×428 vs 392), and @768 and @1280 had 3183 px differ. The artifact astro-kit-visual-linux-failed (138 KB) held only those shots.
    • The first version changed only one digit (480 → 481), and it passed: run 38901149275. A digit is under the 0.2% tolerance at these sizes. Tightening the linux tolerance is a separate decision, not part of this PR.
  • Scratch draft PR test: kit-visual gate probe (do not merge) #588 touched only the root README (closed afterwards, branch deleted). kit-visual must be skipped and pass. changes passed in 8 s, and kit-visual was skipping (0 s), which counts as passing: run 38901174244.

Changesets

None: CI only.

ABB65 added 5 commits October 16, 2026 15:10
…e, with --ci for a missing baseline

The verify job drew the kit on ubuntu-latest against visual/linux, which was never committed: every baseline was
written and the step could not fail on a pixel. The kit visual now runs in its own workflow inside
mcr.microsoft.com/playwright:v1.63.0-noble (the lockfile's version, pinned by a test), only when the kit, the starter,
the SDK, the lockfile or the workflow change, reading the repository only. visual.mjs --ci fails on a missing
baseline (keeping the shot) instead of writing it; a hand-run with `update` uploads every linux baseline to commit.
…kit-visual run in Playwright's image

831 PNGs from run 38898912113 (mcr.microsoft.com/playwright:v1.63.0-noble), every fixture of the 35 components at
390/768/1280; a sample checked by eye (each component at 390 and 1280, the ja/ar/upper wordmarks, the serif heading
hooks). From here the kit-visual run compares against them.
…l request (PR B of 2)

The kit-visual job now runs visual.mjs --ci, so a drawing that moves, a size that changes or a fixture with no linux
baseline fails. It is a required check: the workflow runs on every pull request, a light `changes` job reads the
files the pull request changes, and kit-visual is skipped (which passes) unless the kit, the starter, the SDK, the
lockfile or the workflow changed, so other pull requests do not pull the image. Only failing shots are uploaded,
or every baseline on a hand-run with `update`.
…force

# Conflicts:
#	.github/workflows/kit-visual.yml
#	packages/astro-kit/tests/kit-visual-ci.test.ts
@netlify

netlify Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for contentrain-ai ready!

Name Link
🔨 Latest commit 4201dac
🔍 Latest deploy log https://app.netlify.com/projects/contentrain-ai/deploys/6ad21d992d6e68b889b1cb7f
😎 Deploy Preview https://deploy-preview-586--contentrain-ai.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@ABB65

ABB65 commented Oct 16, 2026

Copy link
Copy Markdown
Member Author

Read @67919bf0 (t2): one-flag HOLD, otherwise OK.

Must-fix: add --no-renames to git diff --name-only. Rename detection is on by default, so a file moved out of a watched path (say packages/astro-kit/src/x.ts → packages/other/x.ts) lists only its new path. kit is then false and a kit change goes undrawn. With --no-renames both sides of the move are listed. A test line: expect(workflow).toContain('git diff --no-renames --name-only').

The 0.2% tolerance question is fine to leave as a separate decision.

@ABB65

ABB65 commented Oct 16, 2026

Copy link
Copy Markdown
Member Author

OK @4201dac5 (t2): --no-renames is in, and the test pins it. Merge once kit-visual and verify on this head are green.

@ABB65
ABB65 merged commit d9a6a8c into main Oct 16, 2026
8 checks passed
@ABB65
ABB65 deleted the feat/kit-visual-ci-enforce branch October 16, 2026 13:02
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 16, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant