Repository navigation
ci(e2e): run the smoke suite in the pinned Playwright image - #1355
Merged
Merged
Conversation
The Install Chromium step apt-fetched fonts and libraries on every shard, and a slow Ubuntu mirror held it past the 10-minute PR cap. The suite and the canary now run through tools/simulation/ui/in-playwright-image.sh in mcr.microsoft.com/playwright v1.62.1-resolute, pinned by digest, with --network host and the runner's uid. An early step fails when the image tag and the lock's @playwright/test version differ.
The digest decides what runs, so a bump that changed the lock and the tag but kept the old digest passed the tag check. The check now pulls the image first and compares /ms-playwright/.docker-info driverVersion with the lock.
jq -r printed null for a missing lock entry or image driverVersion, so both missing compared equal and passed. Both reads now fail with jq -e unless they return a non-empty string, and name what is missing.
Only one job was cancelled inside the install step; the two westus jobs finished it and ran out of time in the suite.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The E2E smoke job no longer installs Chromium with apt. The suite and the canary now run inside Microsoft's Playwright image, pinned by digest, and the job itself stays on the runner.
Why
npx playwright install --with-deps chromiumfetched fonts and system libraries from azure.archive.ubuntu.com on every shard. On 2026-10-23 that mirror was slow for some runner regions:Fetched 24.9 MB in 6min 5s (68.3 kB/s). Its rerun ran in westus3 and was still in the install step when the 10-minute PR cap (e2e-smoke.yml:119) cancelled it, 8 min 8 s after the step started.24.9 MB in 6min 5s, then hit the cap.A rerun still depends on how fast the mirror is at that moment. This change removes the download, so the job no longer depends on the mirror.
What changed
e2e-smoke.ymlpinsPLAYWRIGHT_IMAGEtomcr.microsoft.com/playwright:v1.62.1-resolute@sha256:aebd85bc….resoluteis Ubuntu 26.04, the same OS asruns-on: ubuntu-26.04. The image ships Chromium 151.0.7922.34 and its libraries in/ms-playwright.tools/simulation/ui/in-playwright-image.sh, which callsdocker run --rm --init --ipc=host --network host --user <runner uid:gid>. It mounts the whole repo at its own path and passes through every environment variable the suite reads.npm cistill runs on the host under setup-node 26. The suite runs on the image's Node 24.18.1.tools/simulation/ui/README.mdand the header comment insrc/browser.tsnow describe the image instead ofplaywright install.Design choices
reset.shdrives the host's Docker compose, and the suite expects the stack at127.0.0.1:8081(src/env.ts), so running the whole job in acontainer:would mean reworking the stack boot.--network hostreaches the stack as before.tools/simulation/ui.src/i18n.tsimports the SPA's catalogs fromweb/src, andcast.tsreads../.sim-cast.json.--useris the runner's uid and gid. Screenshots, videos and reports that the container writes stay removable by the runner user, so the "Remove credential-bearing report artifacts" gate and the artifact upload work unchanged. The harness files are mode 0600 and stay readable inside. Nothing new is printed, uploaded or baked into an image.docker runcommands, so the smoke and canary steps cannot drift apart.npm cistays on the host. setup-node has to stay anyway, becauseverify-harness.shcallsnode. On Linux, no package in the lock has an install script or a native binary (onlyfsevents, which is macOS-only), so the host-installednode_modulesruns on the image's Node.CLUCKWORK_E2E_BUNDLED_BROWSER=1is unchanged. Inside the image it resolves to Playwright's own build under/ms-playwright.The version check
Dependabot does not watch
tools/simulation/ui, so nothing bumps the image when someone bumps@playwright/test. The new step "Pull the Playwright image and check it matches the lock" runs right after checkout, before the app image build. It compares the image's owndriverVersion(from/ms-playwright/.docker-info) with the lock's@playwright/testversion and fails with an error that names the tag to pin.Two local review rounds each found a gap in the first versions of this check:
{}, both values read asnull, compared equal, and passed. Both reads now fail unless they return a non-empty string.Mutation results for the final check, run on the step script taken from the workflow at a6a5569:
{}{}aloneWhat I checked, and what nobody has run yet
I checked these on my machine with the script's exact flags. Nobody started a sim stack.
npx playwright --versionin the image prints 1.62.1, the lock's version. Headless Chromium launches and takes a screenshot.playwright test --list --shard=N/3gives the same tests in the image as on the host: 48, 47 and 43, with no differences..sim-cast.jsonis readable inside.ImagePin_IsOneIdenticalStringguards pass.The reviews, run in throwaway copies of the repo, repeated the shard lists and the Chromium launch. They also made a test fail on purpose inside the container and ran the unchanged cleanup step on its output as the host user. It removed the report HTML, traces and error-context files and kept the PNG and WebM files.
This PR's own E2E run is the first full run of the suite in the image. All three shards passed, with the same test counts as the runs before the change (see below).
Not run yet:
Before and after
Before, from the E2E runs on #1346 and #1350 on 2026-10-23:
After, from this PR's E2E run (run 40006224315, head 82c680f):
The skipped test is the slow
session-refresh.spec.tsexpiry spec, which a PR run skips by design. The green shard 2/3 run on #1350 skipped the same one. Each shard loggedPlaywright 1.62.1 in both the image and the lockandchromium: Playwright's downloaded build. The three logs have no aptGet:orFetchedlines and noplaywright installcall, and the cleanup step printedstrip verifiedon each.On runners where the mirror was fast, the 20 to 30 s pull takes up to 14 s longer than the 16 to 20 s install did. Whole jobs took 5 min 10 s to 5 min 37 s, against 4 min 39 s to 5 min 13 s for the healthy runs before. The westus3 shard, the region of the cancelled rerun, finished in 5 min 22 s. That was a different hour, so it does not prove the mirror was still slow then. It does show the job no longer touches the mirror.