chore: add Playwright configuration for E2E tests by 3w36zj6 · Pull Request #41 · typst-community/typst-docs-web

3w36zj6 · 2025-11-30T10:17:34Z

close #40

This pull request introduces end-to-end testing using Playwright to implement Visual Regression Testing (VRT). This will explicitly manage UI layout specifications and prevent unintended changes.

Changes

Add dependencies
Add Playwright configuration
Add CI
- What metadata and assets should be included in the test environment?
- Should we inspect all pages, or perform selective testing by page kind?

netlify · 2025-11-30T10:17:39Z

✅ Deploy Preview for typst-docs-web ready!

Name	Link
🔨 Latest commit	`6715ddb`
🔍 Latest deploy log	https://app.netlify.com/projects/typst-docs-web/deploys/692ee496be8f000008108f99
😎 Deploy Preview	https://deploy-preview-41--typst-docs-web.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

3w36zj6 · 2025-11-30T10:19:30Z

We need to exclude Playwright test files in our Vitest configuration.

YDX-2147483647

I suggest documenting the workflow in README.md or tests/e2e/README.md. It did take me some time to figure it out.

Execute playwright install chromium.
Prepare a suitable docs.json and metadata.json.
Start dev and keep it running.
Run test:e2e.

suitable: await page.goto("/docs/") requires the basePath to be /docs/.

3w36zj6 · 2025-11-30T14:47:16Z

Since this PR also sets up the CI workflow, I think a simple addition to the documentation should be sufficient for now.

As for setting up the assets required for E2E tests, I suggest extracting the hardcoded parts from the current CI into a separate bash script. This way, we can reuse the same logic both in development and CI environments, which should make things much more convenient. What do you think?

…ng page state

3w36zj6 · 2025-12-02T13:15:00Z

Since Playwright's VRT snapshots are unstable in the GitHub Actions environment, we've set up automatic commits of the results generated in CI. This makes it convenient for pull request reviewers to visually check for UI changes. Developers can also use VRT locally as needed.

What do you think about this approach?

Copilot

Pull request overview

This pull request introduces end-to-end testing infrastructure using Playwright for Visual Regression Testing (VRT) to manage UI layout specifications and prevent unintended changes. The implementation includes test configuration, a shell script for fetching documentation assets, and CI workflow updates.

Key changes:

Adds Playwright E2E testing framework with visual snapshot testing for documentation pages
Introduces scripts/fetch-docs-assets.sh to standardize asset fetching across CI and local development
Configures automated E2E testing in GitHub Actions CI workflow

Reviewed changes

Copilot reviewed 8 out of 204 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
`playwright.config.ts`	Configures Playwright test runner with browser settings, timeouts, and snapshot comparison thresholds
`vitest.config.ts`	Excludes E2E tests from vitest runner to prevent conflicts
`tests/e2e/index.test.ts`	Implements VRT for documentation pages by iterating through docs.json and capturing full-page screenshots
`tests/e2e/index.test.ts-snapshots/*.png`	Binary snapshot files for baseline comparison (not reviewed in detail)
`scripts/fetch-docs-assets.sh`	Shell script to download docs.json, assets, and favicon from GitHub releases
`package.json`	Adds Playwright dependencies and test scripts
`bun.lock`	Lock file updates for new dependencies
`mise.toml`	Adds shellcheck and shfmt tools for shell script quality
`README.md`	Documents E2E testing setup and usage instructions
`.gitignore`	Ignores Playwright-generated files and reports
`.github/workflows/ci.yaml`	Refactors asset fetching to use new script and adds E2E test job with snapshot auto-commit

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-02T13:20:22Z

tests/e2e/index.test.ts

+
+const docsJson = readFileSync("./public/docs.json", "utf-8");
+const docs = JSON.parse(docsJson) as unknown as Page[];
+const [flattenedPages, _] = flattenDocs(docs);


The unused variable _ on line 8 should be removed. When destructuring an array and intentionally ignoring a value, you can simply omit the variable or use array destructuring without assigning the unwanted value.

const [flattenedPages] = flattenDocs(docs);

Suggested change

const [flattenedPages, _] = flattenDocs(docs);

const [flattenedPages] = flattenDocs(docs);

Copilot · 2025-12-02T13:20:22Z

tests/e2e/index.test.ts

+	"/docs/reference/symbols/emoji/",
+]);
+
+flattenedPages.forEach((page, _) => {


The forEach loop has an unused second parameter _ (index) that should be removed for cleaner code:

flattenedPages.forEach((page) => {

Suggested change

flattenedPages.forEach((page, _) => {

flattenedPages.forEach((page) => {

playwright.config.ts

Copilot · 2025-12-02T13:20:23Z

.github/workflows/ci.yaml

+      - name: Install Playwright browsers
+        run: bun playwright install --with-deps
+      - name: Run Playwright E2E tests
+        run: bun run test:e2e:update


The CI workflow runs with --update-snapshots flag on line 117, which will automatically update snapshots even if they differ. This defeats the purpose of visual regression testing in CI.

For CI, you should:

Run tests normally (without --update-snapshots) to detect visual changes

Only update snapshots manually when changes are intentional

The current setup will never fail on visual regressions since it always updates snapshots to match

Consider changing line 117 to:

run: bun run test:e2e

And only use test:e2e:update locally or in a separate workflow for intentional updates.

Suggested change

run: bun run test:e2e:update

run: bun run test:e2e

Copilot · 2025-12-02T13:20:23Z

scripts/fetch-docs-assets.sh

+	ESCAPED_BASE=$(printf '%s' "$BASE" | sed 's|/|\\/|g')
+	sed -i "s/\/DOCS-BASE\//${ESCAPED_BASE}/g" "${DEST}/docs.json"


Security concern: The script escapes the BASE variable for sed but doesn't validate or sanitize it. A malicious BASE value could potentially:

Break the sed command with special characters

Execute unintended commands if BASE contains shell metacharacters

Consider adding validation:

# Validate BASE format (should start and end with /) if [[ ! "$BASE" =~ ^/.*/$ ]]; then echo "Error: BASE must start and end with '/' (e.g., '/docs/')" >&2 exit 2 fi

Copilot · 2025-12-02T13:20:24Z

README.md

+# Install Playwright browsers and dependencies
+bun playwright install --with-deps
+


The documentation mentions using bun playwright install --with-deps but doesn't explain what --with-deps does or that it requires sudo privileges on Linux. Consider adding a note:

# Install Playwright browsers and dependencies (may require sudo on Linux) bun playwright install --with-deps

Or provide alternative instructions for users who can't use sudo:

# Install browsers only (no system dependencies) bun playwright install

Suggested change

# Install Playwright browsers and dependencies

bun playwright install --with-deps

# Install Playwright browsers and dependencies (may require sudo on Linux)

bun playwright install --with-deps

# If you cannot use sudo or do not want to install system dependencies,

# you can install only the Playwright browsers:

bun playwright install

Copilot · 2025-12-02T13:20:25Z

playwright.config.ts

+	expect: {
+		timeout: 100_000,
+		/* Configure snapshot comparison */
+		toHaveScreenshot: { maxDiffPixelRatio: 0.05 },


[nitpick] The maxDiffPixelRatio of 0.05 (5% difference allowed) is quite permissive for visual regression testing. This means up to 5% of pixels can differ before a test fails, which could allow significant visual regressions to pass undetected. Consider:

Starting with a stricter threshold (e.g., 0.01 or 1%)

If certain pages have legitimate minor variations, configure per-test thresholds

A 5% threshold may be too lenient to catch meaningful UI changes

Suggested change

toHaveScreenshot: { maxDiffPixelRatio: 0.05 },

toHaveScreenshot: { maxDiffPixelRatio: 0.01 },

Copilot · 2025-12-02T13:20:25Z

.github/workflows/ci.yaml

+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+
+          git add tests/e2e/
+          if ! git diff --staged --quiet; then
+            git commit -m "test: update VRT snapshots" || echo "No changes to commit"
+            git push origin HEAD:${{ github.event.pull_request.head.ref }}
+          else
+            echo "No snapshot changes detected"


The snapshot commit logic (lines 118-129) will push directly to the PR branch, which can create issues:

If the PR is from a fork, this will fail due to permission issues

Auto-committing snapshots on every CI run can create commit spam

The condition on line 126 references github.event.pull_request.head.ref which may not exist for workflow_dispatch or push events

Consider either:

Remove auto-commit logic and require developers to update snapshots locally

Only commit snapshots when explicitly triggered (e.g., via a label or comment)

Add proper guards to check if this is a PR from the same repository

Suggested change

git config user.name "github-actions[bot]"

git config user.email "41898282+github-actions[bot]@users.noreply.github.com"

git add tests/e2e/

if ! git diff --staged --quiet; then

git commit -m "test: update VRT snapshots" || echo "No changes to commit"

git push origin HEAD:${{ github.event.pull_request.head.ref }}

else

echo "No snapshot changes detected"

# Only attempt to commit and push if this is a PR from the same repository

if [ "${{ github.event_name }}" = "pull_request" ] && [ "${{ github.event.pull_request.head.repo.full_name }}" = "${{ github.repository }}" ]; then

git config user.name "github-actions[bot]"

git config user.email "41898282+github-actions[bot]@users.noreply.github.com"

git add tests/e2e/

if ! git diff --staged --quiet; then

git commit -m "test: update VRT snapshots" || echo "No changes to commit"

git push origin HEAD:${{ github.event.pull_request.head.ref }}

else

echo "No snapshot changes detected"

fi

else

echo "Skipping snapshot commit: not a PR from the same repository."

Copilot · 2025-12-02T13:20:25Z

scripts/fetch-docs-assets.sh

+else
+	# Use portable sed: escape slashes
+	ESCAPED_BASE=$(printf '%s' "$BASE" | sed 's|/|\\/|g')
+	sed -i "s/\/DOCS-BASE\//${ESCAPED_BASE}/g" "${DEST}/docs.json"


The shell script lacks proper error handling for the sed command on line 91. If sed -i is not available (e.g., on macOS which requires sed -i ''), the script will fail. The portable approach for in-place editing varies between GNU sed and BSD sed.

Consider using a more portable approach:

ESCAPED_BASE=$(printf '%s' "$BASE" | sed 's|/|\\/|g') if sed --version >/dev/null 2>&1; then # GNU sed sed -i "s/\/DOCS-BASE\//${ESCAPED_BASE}/g" "${DEST}/docs.json" else # BSD sed (macOS) sed -i '' "s/\/DOCS-BASE\//${ESCAPED_BASE}/g" "${DEST}/docs.json" fi

Or create a temporary file for true portability:

sed "s/\/DOCS-BASE\//${ESCAPED_BASE}/g" "${DEST}/docs.json" > "${DEST}/docs.json.tmp" mv "${DEST}/docs.json.tmp" "${DEST}/docs.json"

Suggested change

sed -i "s/\/DOCS-BASE\//${ESCAPED_BASE}/g" "${DEST}/docs.json"

sed "s/\/DOCS-BASE\//${ESCAPED_BASE}/g" "${DEST}/docs.json" > "${DEST}/docs.json.tmp" && mv "${DEST}/docs.json.tmp" "${DEST}/docs.json"

Copilot · 2025-12-02T13:20:26Z

scripts/fetch-docs-assets.sh

+cat >"${DEST}/metadata.json" <<EOF
+{
+  "\$schema": "../metadata.schema.json",
+  "language": "en-US",
+  "version": "${VERSION}",
+  "typstOfficialUrl": "https://typst.app",
+  "typstOfficialDocsUrl": "https://typst.app/docs/",
+  "githubOrganizationUrl": "https://github.com/${ORG}",
+  "socialLinks": [
+    { "url": "https://github.com/${ORG}/typst-docs-web" },
+    {
+      "title": "Discord (Typst)",
+      "url": "https://discord.gg/2uDybryKPe"
+    }
+  ],
+  "originUrl": "https://example.com/",
+  "basePath": "${BASE}",
+  "displayTranslationStatus": false
+}
+EOF


The JSON in metadata.json is not properly escaped for shell heredoc. If VERSION, ORG, or BASE contain special characters (like quotes or backslashes), the generated JSON will be invalid.

Consider using jq to generate the JSON safely:

jq -n \ --arg version "$VERSION" \ --arg org "$ORG" \ --arg base "$BASE" \ '{ "$schema": "../metadata.schema.json", "language": "en-US", "version": $version, "typstOfficialUrl": "https://typst.app", "typstOfficialDocsUrl": "https://typst.app/docs/", "githubOrganizationUrl": ("https://github.com/" + $org), "socialLinks": [ { "url": ("https://github.com/" + $org + "/typst-docs-web") }, { "title": "Discord (Typst)", "url": "https://discord.gg/2uDybryKPe" } ], "originUrl": "https://example.com/", "basePath": $base, "displayTranslationStatus": false }' > "${DEST}/metadata.json"

Alternatively, escape the variables properly if staying with heredoc.

Suggested change

cat >"${DEST}/metadata.json" <<EOF

{

"\$schema": "../metadata.schema.json",

"language": "en-US",

"version": "${VERSION}",

"typstOfficialUrl": "https://typst.app",

"typstOfficialDocsUrl": "https://typst.app/docs/",

"githubOrganizationUrl": "https://github.com/${ORG}",

"socialLinks": [

{ "url": "https://github.com/${ORG}/typst-docs-web" },

{

"title": "Discord (Typst)",

"url": "https://discord.gg/2uDybryKPe"

}

],

"originUrl": "https://example.com/",

"basePath": "${BASE}",

"displayTranslationStatus": false

}

EOF

jq -n \

--arg version "$VERSION" \

--arg org "$ORG" \

--arg base "$BASE" \

'{

"$schema": "../metadata.schema.json",

"language": "en-US",

"version": $version,

"typstOfficialUrl": "https://typst.app",

"typstOfficialDocsUrl": "https://typst.app/docs/",

"githubOrganizationUrl": ("https://github.com/" + $org),

"socialLinks": [

{ "url": ("https://github.com/" + $org + "/typst-docs-web") },

{

"title": "Discord (Typst)",

"url": "https://discord.gg/2uDybryKPe"

}

],

"originUrl": "https://example.com/",

"basePath": $base,

"displayTranslationStatus": false

}' > "${DEST}/metadata.json"

YDX-2147483647 · 2025-12-03T03:30:03Z

To be honest, I'm not familiar with Playwright VRT. My closest previous experience was in a LaTeX template project.

Workflow for the LaTeX project

Before releasing a new version, we download the source code of the previous version from GitHub Releases, compile the two versions into PDFs locally, and then compare them. (It's solidified as a script.)
We don't run this workflow for most PRs, because we don't have time and energy to do so.
We use two PDF comparison tools. One is very strict, used to determine which templates have changed (the project has multiple templates). The other one is relatively loose and comes with a GUI, allowing for manual judgment of regression issues.
We do not keep PDFs compiled from the old source codes. This is because we add new sample contents from time to time. As a result, the old source code has to be compiled with the new content before comparison, eliminating the use of saving PDF compiled from old source code + old content.

I think the current approach is kind of problematic.

The screenshots are specific to the platform. On my Windows machine, playwright complaints:

A snapshot doesn't exist at tests\e2e\index.test.ts-snapshots\*-chromium-win32.png

Despite this, most (115 of 194, ~59%) tests can pass after renaming *-linux.png to *-win32.png.

Main causes of failed tests:
- The screenshot was taken too early, before the outline updates its state. (image 1)
  (I feel like that everything becomes slower on Windows.)
- The width of <main> is slightly smaller, then the line breaking algorithm (image 1 & 2) and CSS breakpoints (example blocks in image 3) exaggerate the difference.
All pages in v0.14.0 en-US docs are committed into the repo.
- That's 80 MiB. These screenshots will bloat the repo size, making it difficult for new comers to clone.
  
  How about publishing those screenshots to GitHub Actions/Releases? We can download them when needed.
- That introduces ~200 repetitive PNGs. There might be too much noise when reviewing pull requests.
  For example, changing a gap-1 in <FunctionParameters> to gap-2 will fail 14 tests.
  
  Perhaps it would be better to extract several independent and typical fragments from docs.json? This also helps to adapt to the future situation when docs.json is replaced by typst.
- Only en-US is covered. Features like translation status are not tested.

Besides, there are commented codes in playwright.config.ts. Is that intentional?

3w36zj6 · 2025-12-15T11:50:47Z

Testing every single page would probably be overkill. I think it's sufficient to pick a few representative pages from each category and check those.

I'll review Copilot's suggestions later.

chore: add Playwright configuration for E2E tests

9ded103

3w36zj6 requested a review from YDX-2147483647 as a code owner November 30, 2025 10:17

YDX-2147483647 reviewed Nov 30, 2025

View reviewed changes

3w36zj6 and others added 12 commits December 1, 2025 17:14

refactor: extract docs fixtures download to script

7fefd7d

chore: add Vitest configuration to exclude E2E test cases

97e9fe2

test: add all page snapshots

e269d9c

ci: add Playwright E2E tests

a462fd8

chore: configure timeouts for Playwright tests

d80b6a3

docs: add E2E testing instructions to README

8751aad

chore: configure screenshot tolerance

23ebee1

test: add snapshots

0af7f30

test: stabilize Playwright tests by increasing timeouts and normalizi…

ca179a7

…ng page state

ci: automate snapshot updates

f735fe7

Merge f735fe7 into dc5ef5c

ef3c9a0

test: update VRT snapshots

6715ddb

3w36zj6 requested a review from Copilot December 2, 2025 13:11

Copilot started reviewing on behalf of 3w36zj6 December 2, 2025 13:11 View session

Copilot finished reviewing on behalf of 3w36zj6 December 2, 2025 13:15

Copilot AI reviewed Dec 2, 2025

View reviewed changes

YDX-2147483647 mentioned this pull request Dec 5, 2025

fix: import translations lazily #43

Merged

	const [flattenedPages, _] = flattenDocs(docs);
	const [flattenedPages] = flattenDocs(docs);

	flattenedPages.forEach((page, _) => {
	flattenedPages.forEach((page) => {

		ESCAPED_BASE=$(printf '%s' "$BASE" \| sed 's\|/\|\\/\|g')
		sed -i "s/\/DOCS-BASE\//${ESCAPED_BASE}/g" "${DEST}/docs.json"

		# Install Playwright browsers and dependencies
		bun playwright install --with-deps

	toHaveScreenshot: { maxDiffPixelRatio: 0.05 },
	toHaveScreenshot: { maxDiffPixelRatio: 0.01 },

	sed -i "s/\/DOCS-BASE\//${ESCAPED_BASE}/g" "${DEST}/docs.json"
	sed "s/\/DOCS-BASE\//${ESCAPED_BASE}/g" "${DEST}/docs.json" > "${DEST}/docs.json.tmp" && mv "${DEST}/docs.json.tmp" "${DEST}/docs.json"

Conversation

3w36zj6 commented Nov 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

netlify bot commented Nov 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for typst-docs-web ready!

Uh oh!

3w36zj6 commented Nov 30, 2025

Uh oh!

YDX-2147483647 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

3w36zj6 commented Nov 30, 2025

Uh oh!

3w36zj6 commented Dec 2, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

YDX-2147483647 commented Dec 3, 2025

Uh oh!

3w36zj6 commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

3w36zj6 commented Nov 30, 2025 •

edited

Loading

netlify bot commented Nov 30, 2025 •

edited

Loading

YDX-2147483647 left a comment •

edited

Loading