Skip to content

fix(ci): run browser tests in pinned Playwright containers - #2584

Merged
jbeda merged 1 commit into
mainfrom
fix/playwright-ci-provisioning
Oct 27, 2026
Merged

jbeda merged 1 commit into
mainfrom
fix/playwright-ci-provisioning

Conversation

@jbeda

@jbeda jbeda commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Run SDK and Studio browser jobs in the official Playwright 1.64.1 Noble image, pinned by digest, with a non-root user, process reaping, shared memory, and explicit Bash.
  • Remove CI browser downloads and the SDK browser cache; preserve all browser projects, single-worker suites, relevance gates, prebuilt same-checkout daemon, and existing timeout budgets. Local browser-install tasks are unchanged.
  • Upload retained browser failure diagnostics with unique per-project artifact names.
  • Group Playwright npm/image updates in Renovate and validate image/package version agreement in the always-run changes job and local Actions lint.

Why

In the latest 100 CI runs examined (October 24–27), 15 of 79 browser-enabled runs had a confirmed 20-minute timeout. All confirmed timeout jobs stopped in SDK Playwright dependency installation before tests started. Sample logs show very slow Ubuntu apt mirror downloads, even with a browser-cache hit; actual SDK browser test steps typically take 6–7 seconds.

This follows the upstream-supported GitHub Actions container pattern and Docker guidance. Upstream discourages browser caching because Linux OS dependencies still require installation. Studio previously downloaded Chromium without --with-deps; it adopts the same container pattern for consistency, not because it has the identical apt failure.

Verification

  • Passed: new offline contract tests, negative/mutation cases, existing path-classifier/workflow tests, and diff whitespace checks.
  • Passed: all five projects in the exact pinned image, non-root, CI-like HOME, pinned Task 3.54.0 and real pinned pnpm/action-setup. No browser or apt installation. SDK: 6 tests; Studio: desktop 70 passed/34 existing skips, mobile 97 passed/7 existing skips.
  • Passed: isolated forced browser failure retained a valid trace ZIP and screenshot.
  • Passed: focused independent DevOps review, no blockers/important findings.
  • Passed locally: complete offline task test, task test:race, task lint, task lint:actions, and go run ./cmd/mecademo (tool call, permission ask/approval, result). Aggregate tests used concurrency 2 and race builds used GOFLAGS=-p=2. Actions lint passed with checksummed disposable cosign tooling after an initial cold-build timeout and missing-tool diagnosis; no unrelated code changes.
  • Initial hosted Actions run: passed, including all five browser jobs and actual non-root action/tool-cache setup and cleanup. SDK: WebKit 55s, Chromium 67s, Firefox 70s. Studio: mobile 3m35s, desktop 4m26s. All reported applicable checks passed before marking ready; draft-inapplicable/live checks skipped. Marking ready may trigger fresh checks on this same commit.

Approved plan and research archive.

API compatibility

CI/configuration only. No wire, Go source, or stored-data contract changes; no API major or feature change and no operator upgrade steps.

Independent of #2583; that branch is unchanged.

Avoid runtime apt mirror dependency installation in SDK browser jobs. Keep image and npm versions aligned, retain failure diagnostics, and use the same provisioning for Studio.

Co-authored-by: Mecatl <noreply@mecatl.dev>
@jbeda

jbeda commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

Approved implementation plan: Playwright CI provisioning

  • Slug: playwright-ci-provisioning
  • Source: .scratch/playwright-ci-research.md (local provenance)
  • Branch: fix/playwright-ci-provisioning
  • Baseline: 55b980d34b4518221c9385a2ab32e2bb5c3f4f8a
  • Updated: 2026-10-27
  • The following describes intended scope and verification, not completed gate results. Actual results belong in the PR body.

Research and decision

Primary sources:

Playwright documents two first-class GitHub Actions patterns: host jobs that install JavaScript dependencies and run playwright install --with-deps, or official Playwright container jobs with browsers and OS libraries already installed. The existing host pattern is supported, but apt mirror downloads caused the observed SDK timeouts. Upstream explicitly discourages browser caching because restoring binaries costs roughly as much as downloading them and Linux system dependencies still need installation.

Use the official mcr.microsoft.com/playwright:v1.64.1-noble job container, pinned to a verified digest. It includes browsers and OS dependencies, not the project's Playwright npm package; retain frozen pnpm installation and explicit Node/pnpm/Task setup.

Scope and contracts

  • Convert all three SDK browser projects and both Studio browser projects to the same official image.
  • Preserve existing relevance gates, read-only permissions, secret-free PR execution, prebuilt same-checkout mecated artifact, single-worker execution, browser coverage, and suite/global timeouts. Keep the 20-minute job ceiling.
  • Remove browser caching and browser/OS installation from containerized CI. Preserve local developer browser-install tasks.
  • Use non-root execution where compatible and recommended --init --ipc=host; do not add privileged mode, SYS_ADMIN, or a Docker socket mount. Explicitly select Bash for run steps.
  • Upload retained failure traces/screenshots as matrix-specific artifacts; do not add retries to mask provisioning failures.
  • Keep Playwright image and npm package versions aligned. Both SDK and Studio currently pin @playwright/test to 1.64.1. Coordinate Renovate npm/image updates and add an offline version-skew guard.
  • No public wire, Go source, or stored-data contract change; no major or feature bump.
  • Keep this independent of feat(mcp): support static bearer credentials in auth.yaml #2583.

Correction to initial research: Studio previously installed Chromium without --with-deps; it adopts the same container pattern for consistency, not because it exhibited the identical runtime apt failure.

Tasks

  1. Update SDK/Studio workflow provisioning and failure-artifact delivery without changing coverage, gates, local install tasks, or timeout budgets.
  2. Coordinate Playwright package/image updates in Renovate. Add isolated offline configuration regression checks, including mismatched package/image versions and missing pins/provisioning regressions. Run them in the always-run changes job and local Actions lint.
  3. Validate and review the integrated change, then commit, push, and create this PR. No merge or deployment is authorized.

Planned verification

  • Verify the official image digest and platform manifests.
  • Validate workflow syntax and existing classifier/workflow contracts.
  • Exercise all three SDK projects and both Studio projects in the exact pinned image with non-root user, CI-like HOME, explicit Node/pnpm/Task tooling, and prebuilt same-checkout daemon. Verify package caches, artifact permissions, browser paths, and daemon binary compatibility without an in-container Go toolchain.
  • Force an isolated browser-test failure and verify retained diagnostics exist.
  • Exercise image/package mismatch rejection, coordinated dependency updates, and unchanged local install paths.
  • Run repository-required offline test/race, lint, and demo gates; obtain focused independent CI/DevOps review.
  • Validate hosted Actions separately: local Docker does not establish GitHub runner permissions, action/tool-cache behavior, or measured CI provisioning improvement.

No timeout increases, test retries, coverage reductions, custom browser-server arrangement, or custom image build are in scope. No unresolved product decision remains; hosted runtime behavior requires empirical validation.

@jbeda
jbeda marked this pull request as ready for review October 27, 2026 20:37
@jbeda
jbeda requested review from rdimitrov and samuv and removed request for rdimitrov October 27, 2026 21:19
@jbeda
jbeda merged commit 38c5023 into main Oct 27, 2026
76 checks passed
@jbeda
jbeda deleted the fix/playwright-ci-provisioning branch October 27, 2026 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant