Skip to content

ci(e2e): run the smoke suite in the pinned Playwright image - #1355

Merged
mforce merged 4 commits into
mainfrom
ci/e2e-playwright-image
Oct 23, 2026
Merged

mforce merged 4 commits into
mainfrom
ci/e2e-playwright-image

Conversation

@mforce

@mforce mforce commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

The E2E smoke job no longer installs Chromium with apt. The suite and the canary now run inside Microsoft's Playwright image, pinned by digest, and the job itself stays on the runner.

Why

npx playwright install --with-deps chromium fetched fonts and system libraries from azure.archive.ubuntu.com on every shard. On 2026-10-23 that mirror was slow for some runner regions:

A rerun still depends on how fast the mirror is at that moment. This change removes the download, so the job no longer depends on the mirror.

What changed

  • e2e-smoke.yml pins PLAYWRIGHT_IMAGE to mcr.microsoft.com/playwright:v1.62.1-resolute@sha256:aebd85bc…. resolute is Ubuntu 26.04, the same OS as runs-on: ubuntu-26.04. The image ships Chromium 151.0.7922.34 and its libraries in /ms-playwright.
  • The "Install Chromium" step is gone. A new early step pulls the image and checks its Playwright version (see below).
  • The smoke step and the dispatch-only canary step run through a new script, tools/simulation/ui/in-playwright-image.sh, which calls docker run --rm --init --ipc=host --network host --user <runner uid:gid>. It mounts the whole repo at its own path and passes through every environment variable the suite reads.
  • npm ci still runs on the host under setup-node 26. The suite runs on the image's Node 24.18.1.
  • The workflow header, tools/simulation/ui/README.md and the header comment in src/browser.ts now describe the image instead of playwright install.

Design choices

  • Only the Playwright steps run in the image. reset.sh drives the host's Docker compose, and the suite expects the stack at 127.0.0.1:8081 (src/env.ts), so running the whole job in a container: would mean reworking the stack boot. --network host reaches the stack as before.
  • The whole repo is mounted, not just tools/simulation/ui. src/i18n.ts imports the SPA's catalogs from web/src, and cast.ts reads ../.sim-cast.json.
  • --user is the runner's uid and gid. Screenshots, videos and reports that the container writes stay removable by the runner user, so the "Remove credential-bearing report artifacts" gate and the artifact upload work unchanged. The harness files are mode 0600 and stay readable inside. Nothing new is printed, uploaded or baked into an image.
  • One wrapper script instead of two inline docker run commands, so the smoke and canary steps cannot drift apart.
  • npm ci stays on the host. setup-node has to stay anyway, because verify-harness.sh calls node. On Linux, no package in the lock has an install script or a native binary (only fsevents, which is macOS-only), so the host-installed node_modules runs on the image's Node.
  • CLUCKWORK_E2E_BUNDLED_BROWSER=1 is unchanged. Inside the image it resolves to Playwright's own build under /ms-playwright.

The version check

Dependabot does not watch tools/simulation/ui, so nothing bumps the image when someone bumps @playwright/test. The new step "Pull the Playwright image and check it matches the lock" runs right after checkout, before the app image build. It compares the image's own driverVersion (from /ms-playwright/.docker-info) with the lock's @playwright/test version and fails with an error that names the tag to pin.

Two local review rounds each found a gap in the first versions of this check:

  • Round 1: the first check compared the lock with the image tag's text. With the lock and the tag both changed to 1.63.0 and the old digest kept, it passed, even though the image still held 1.62.1. The check now reads the version from inside the pulled image.
  • Round 2: with the lock entry missing and the image metadata set to {}, both values read as null, compared equal, and passed. Both reads now fail unless they return a non-empty string.

Mutation results for the final check, run on the step script taken from the workflow at a6a5569:

Case Exit
Current state 0
Lock and tag at 1.63.0, old digest 1
Lock alone at 1.63.0 1
Lock entry missing, image metadata {} 5
Image metadata {} alone 5
Lock entry missing alone 5
Both files empty 4

What I checked, and what nobody has run yet

I checked these on my machine with the script's exact flags. Nobody started a sim stack.

  • npx playwright --version in the image prints 1.62.1, the lock's version. Headless Chromium launches and takes a screenshot.
  • playwright test --list --shard=N/3 gives the same tests in the image as on the host: 48, 47 and 43, with no differences.
  • Files the container writes belong to the host user, who can delete them, and a mode-0600 .sim-cast.json is readable inside.
  • actionlint and shellcheck report nothing, and the two ImagePin_IsOneIdenticalString guards pass.

The reviews, run in throwaway copies of the repo, repeated the shard lists and the Chromium launch. They also made a test fail on purpose inside the container and ran the unchanged cleanup step on its output as the host user. It removed the report HTML, traces and error-context files and kept the PNG and WebM files.

This PR's own E2E run is the first full run of the suite in the image. All three shards passed, with the same test counts as the runs before the change (see below).

Not run yet:

  • The image's disk use on a hosted runner. It is 3.6 GB unpacked, and no run so far has measured free disk space around the pull.
  • The dispatch-only canary against a seeded stack.

Before and after

Before, from the E2E runs on #1346 and #1350 on 2026-10-23:

Run Region Install Chromium Job
#1346 shard 1/3 centralus 16 s 5 min 13 s
#1346 shard 3/3 centralus 17 s 4 min 39 s
#1346 shard 2/3 (rerun) westus3 cancelled after 8 min 8 s 10 min 18 s; cancelled by the 10-minute cap
#1350 shard 2/3 (rerun) centralus 19 s 4 min 51 s
#1350 shard 3/3 (rerun) westus2 20 s 5 min 7 s

After, from this PR's E2E run (run 40006224315, head 82c680f):

Shard Region Pull and check Job Tests
1/3 eastus 27 s 5 min 37 s 48 passed
2/3 westus2 20 s 5 min 10 s 46 passed, 1 skipped
3/3 westus3 30 s 5 min 22 s 43 passed

The skipped test is the slow session-refresh.spec.ts expiry spec, which a PR run skips by design. The green shard 2/3 run on #1350 skipped the same one. Each shard logged Playwright 1.62.1 in both the image and the lock and chromium: Playwright's downloaded build. The three logs have no apt Get: or Fetched lines and no playwright install call, and the cleanup step printed strip verified on each.

On runners where the mirror was fast, the 20 to 30 s pull takes up to 14 s longer than the 16 to 20 s install did. Whole jobs took 5 min 10 s to 5 min 37 s, against 4 min 39 s to 5 min 13 s for the healthy runs before. The westus3 shard, the region of the cancelled rerun, finished in 5 min 22 s. That was a different hour, so it does not prove the mirror was still slow then. It does show the job no longer touches the mirror.

The Install Chromium step apt-fetched fonts and libraries on every shard, and a slow Ubuntu mirror held it past the 10-minute PR cap. The suite and the canary now run through tools/simulation/ui/in-playwright-image.sh in mcr.microsoft.com/playwright v1.62.1-resolute, pinned by digest, with --network host and the runner's uid. An early step fails when the image tag and the lock's @playwright/test version differ.
The digest decides what runs, so a bump that changed the lock and the tag but kept the old digest passed the tag check. The check now pulls the image first and compares /ms-playwright/.docker-info driverVersion with the lock.
jq -r printed null for a missing lock entry or image driverVersion, so both missing compared equal and passed. Both reads now fail with jq -e unless they return a non-empty string, and name what is missing.
Only one job was cancelled inside the install step; the two westus jobs finished it and ran out of time in the suite.
@mforce
mforce merged commit 172f19a into main Oct 23, 2026
22 checks passed
@mforce
mforce deleted the ci/e2e-playwright-image branch October 23, 2026 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant