Mimic WPT runner

This repository contains a CDP testharness runner and analysis tooling, not a fork of web-platform-tests. The WPT corpus and browser binaries live separately. The default is a fixed 15-URL sample. A large selection requires explicit --full; setup never starts a multi-hour browser run automatically.

Recorded baseline

The completed sample ran on 2026-10-03 in Ubuntu 24.04.3 / WSL2, x86_64, with fresh headless contexts and concurrency 4. All 97 subtest IDs were observed on all three backends. It is a small, control-gated sample, not a full-WPT compatibility score. Encoding contributes 49 of the 97 assertions. Percentages sum counts, not category averages.

Backend	Files PASS / FAIL	Subtests PASS / FAIL	Chrome-relative common PASS	Chrome PASS ID coverage
Chrome 152.0.7977.82	15 / 0	97 / 0	control	100%
Lightpanda 1.0.0	9 / 6	77 / 20	77/97 = 79.38%	100%
Mimic measured source	7 / 8	27 / 70	27/97 = 27.84%	100%

TIMEOUT, ERROR, CRASH, NOTRUN and SKIP were zero in this completed sample. Paired IDs: 27 common PASS, 20 common FAIL, 50 Lightpanda-PASS/Mimic-FAIL and zero Mimic-PASS/Lightpanda-FAIL. See sample data and baseline provenance.

Actual WPT category/path	Chrome PASS IDs	Lightpanda PASS	Mimic PASS
dom	13	13	13
dom/events (inside dom)	1	1	1
uievents	1	1	1
fetch	2	2	2
xhr	6	6	6
websockets	12	2	2
workers	1	0	0
webstorage	4	4	2
html	2	0	0
html/canvas (inside html)	1	0	0
css	4	0	0
webgl	3	0	0
encoding	49	49	1

Nested Events and Canvas rows are already included in DOM and HTML. Do not add them again. webstorage does not mean all storage APIs; IndexedDB and most other WPT categories were not measured by this tiny sample.

Selection used one SHA-ranked ordinary-timeout URL per listed path/category among prior Chrome PASS controls; compared runtime outcomes did not influence selection. The seed and IDs are in selection-receipt.json. WS and WSS echo variants were added as protocol controls. This selection is intentionally biased towards working Chrome controls and cannot support a representative product-wide percentage.

Pinned sources and binaries

• WPT fork: 159253365efa0bea03b8f633a1aeceb9a45c43dc.
• Official demo methodology: 59c2528e390984e622e7c24069842806ea94c0cb.
• Lightpanda browser 1.0.0: source 588f6223b9cae8a2406aeef035ed9363a3e404fd, official amd64 Debian artifact, binary SHA256 aa5a4b8ed53d1e38b3c73f5b2647d0a84a82e6744557f45f9a9c85858aa031c3.
• Chrome for Testing: Linux 152.0.7977.82, revision 1669021, binary SHA256 115ee0fc6d26768ec757db2b0c6f54554dece46b59e96c7eca653c944db7f3de.
• Measured Mimic runtime: source 7f55808f3c5d35fc48c8a62ae6ac0017ca653b9e, stock isolated Linux build, vcs.modified=false, Go 1.26.4; binary SHA256 f3dfcf275709551564c7043c5ac81d0aa8c68fa9f54ab7d5cc096dde6f63f0da.

The runtime tag wpt-baseline-2026-10-03 points to the measured runtime snapshot before WPT-oriented fixes. A README-only later commit is not the measured binary. The same tag in this repository identifies the corresponding portable tooling; these are two distinct repositories and commit IDs.

All binaries executed on Linux. Their JavaScript identities differ: Mimic exposes a frozen Windows Chrome 152 profile; Lightpanda's CDP metadata reports a macOS Chrome 124 profile. Lightpanda's binary version is independently checked using lightpanda version; its CDP Chrome product string is not its release version.

The preserved historical full manifest is version 9, SHA256 71d83ad56757e268a267bd44279cd5f5a11fb66d3f2f3ace55ff357b9a562fc5. Only inventory/digest metadata is committed. A newly generated sample-only manifest has a different digest by design. The original corrected measured adapter is retained in provenance/measured-adapter.py for inspection, not as the portable entry point. Packaging added explicit paths, guards and cancellation; no new live browser validation was performed for this publication.

Setup on Linux or WSL

Use Python 3.10+, Git, OpenSSL and separately installed exact Linux browser binaries. Do not use a personal browser profile, headful browser, system Chrome substitution or an unpinned Lightpanda nightly.

python3 -m venv .venv
. .venv/bin/activate
python3 -m pip install -r requirements.txt

# Keep the corpus outside this repository. This is a large shallow corpus clone,
# not a browser test run.
git init ../wpt-corpus
git -C ../wpt-corpus remote add origin https://github.com/lightpanda-io/wpt.git
git -C ../wpt-corpus fetch --depth=1 origin 159253365efa0bea03b8f633a1aeceb9a45c43dc
git -C ../wpt-corpus checkout --detach FETCH_HEAD

# Uses WPT's official SourceFile/Manifest classifier on only the sample sources.
python3 tools/prepare.py --wpt-root ../wpt-corpus --output artifacts/setup
python3 tools/reporter.py install --wpt-root ../wpt-corpus --state artifacts/reporter

prepare.py imports the pinned WPT tool dependencies through tools.localpaths. The separate upstream wpt serve command may bootstrap its own virtualenv; follow the pinned WPT prerequisites if required. Neither preparation command launches a browser. The reporter installer refuses already modified upstream files and restores only when its installed hash still matches, preserving outside edits.

DNS and process-only TLS

Canonical WPT origins need loopback mappings. If existing names resolve to loopback, use that setup. Otherwise review the generated 62-name block. Applying it is a separate, explicit privileged action; no script invokes sudo on your behalf. Windows hosts is never touched.

python3 tools/hosts.py prepare --wpt-root ../wpt-corpus --block artifacts/hosts-block.txt
cat artifacts/hosts-block.txt
# Only after approving this exact temporary block:
sudo python3 tools/hosts.py apply --block artifacts/hosts-block.txt --state artifacts/hosts-change.json

The guard refuses existing WPT names, non-loopback mappings and overwritten receipts. It saves original bytes and the exact added block. Removal later deletes only that block and preserves concurrent edits; if the block changed, it refuses a blind restore. Backups may contain machine-local settings and stay ignored in artifacts/.

python3 tools/tls_bundle.py --system-ca /etc/ssl/certs/ca-certificates.crt \
  --wpt-root ../wpt-corpus --output artifacts/wpt-ca-bundle.pem
python3 ../wpt-corpus/wpt serve --config samples/2026-10-03/server-config.json

The bundle combines existing public CAs and the public WPT CA. It is never installed in system trust. The explicit runner flag --trust-wpt-ca enables only research-process trust: Chrome's exception is pinned to the WPT server SPKI; Lightpanda uses --ca-cert; Mimic receives SSL_CERT_FILE. No global TLS verification bypass is used. The runner verifies the pinned WPT source/key and requires canonical origins to resolve only to loopback.

Run the default sample

In another shell, after readiness checks on the manually started server:

python3 runner.py --wpt-root ../wpt-corpus \
  --manifest artifacts/setup/MANIFEST.json --output artifacts/run \
  --chrome-binary /path/to/chrome-linux64/chrome \
  --lightpanda-binary /path/to/lightpanda \
  --mimic-binary /path/to/measured/mimic \
  --mimic-source 7f55808f3c5d35fc48c8a62ae6ac0017ca653b9e \
  --trust-wpt-ca --ca-bundle artifacts/wpt-ca-bundle.pem --concurrency 4
python3 report.py artifacts/run
python3 chrome_relative.py artifacts/run
python3 subpaths.py artifacts/run
python3 missing.py artifacts/run

Supply a different explicit --mimic-source when measuring a future candidate. Binary hashes and actual launch argv are recorded; there is no quiet binary or reference version substitution. Concurrency is bounded to 1–4. Every output directory must be new. Browser execution is Linux-only; offline analysis also works on Windows. The runner closes only processes it launched in its own process groups and fresh profiles. It does not use a system-wide browser kill command.

Afterwards stop only your separately launched WPT server (Ctrl-C), and restore only changes made by these helpers:

sudo python3 tools/hosts.py remove --block artifacts/hosts-block.txt --state artifacts/hosts-change.json
python3 tools/reporter.py restore --wpt-root ../wpt-corpus --state artifacts/reporter

Explicit full testharness scope

This can take many hours. First verify a small common protocol smoke and budget resources. These commands are intentionally separate from the default sample:

python3 tools/prepare.py --wpt-root ../wpt-corpus --output artifacts/full-setup --full
python3 runner.py --full --selection artifacts/full-setup/selection.json \
  --manifest artifacts/full-setup/MANIFEST.json --wpt-root ../wpt-corpus \
  --output artifacts/full-run --chrome-binary /path/to/chrome \
  --lightpanda-binary /path/to/lightpanda --mimic-binary /path/to/mimic \
  --mimic-source YOUR_EXACT_SOURCE_SHA --trust-wpt-ca \
  --ca-bundle artifacts/wpt-ca-bundle.pem --concurrency 2

The pinned historical manifest contains 38,674 testharness URL variants / 31,566 sources in 256 actual top-level categories. The historical plan marked 4,027 file SKIPs: 3,845 source-detected driver dependencies, 78 jsshell variants, 102 WebTransport fixtures and 2 PAC tests. This is a selection plan, not evidence that all those rows executed. Driver detection is static; transitive dependencies can still surface as harness failures/timeouts and must be investigated explicitly.

Only testharness callbacks are executed here. Manual, reftest, printreftest, visual, wdspec, aamtest, crashtest and test262 manifest types are inventoried, not implemented by this adapter. Pixel/reftest coverage cannot be inferred from JavaScript Canvas/WebGL assertions. WebTransport/HTTP3 and PAC fixtures are not configured. Device/input/permission automation and GPU presentation need separate validation. Stock Lightpanda lacks the CI WPT-extension build used by its official dashboard; the fork's testdriver depends on Lightpanda-specific helpers.

Results and denominators

Assertions come from unchanged WPT tests and testharness.js; Chrome is an environment control, not a vote that overrides WPT assertions. Standard numeric statuses and completion status are preserved. Ordinary controller deadlines are 30 seconds, long deadlines 60 seconds, including CDP work. All commands share the remaining whole-file deadline rather than a separate 3-second query limit. Observed partial callbacks survive timeout. The unmodified testharness has its own ordinary 10-second / long 60-second timers. DNS/TLS network errors remain provisional signals because tests can deliberately trigger failed connections.

Files and subtests are counted separately: PASS, FAIL, TIMEOUT, ERROR, CRASH, NOTRUN and SKIP. Unobserved IDs are not invented assertions or silently PASSed. An ID is an exact normalized WPT URL variant plus its unmodified subtest name. Duplicate names are ambiguous; paired output marks AMBIGUOUS, and metrics exclude ambiguity only for the affected backend/identity.

• Chrome-relative pass rate on common subtests: runtime PASS intersect Chrome PASS, divided by Chrome PASS IDs identically observed in that runtime.
• Coverage: identifiable common Chrome PASS IDs / all identifiable Chrome PASS IDs. Missing and ambiguous runtime IDs are separately counted.
• Literal rate: the same PASS intersection / all identifiable Chrome PASS IDs.
• Three-backend-common rate: both runtimes use exactly the same Chrome PASS denominator identifiable on all three backends.

A high conditional rate with low coverage is not broad compatibility. Runtime PASS on Chrome FAIL remains in raw/paired WPT outcomes but does not enter the Chrome-PASS-relative metric. Totals use counts, never an unweighted mean of category percentages. CSV/JSON outputs contain denominators and category counts.

Other attempts: kept separate

• Completed final sample: the 15-URL canonical run above, corrected collector, all three backends, 97 common IDs. Sanitized raw and summaries are committed.
• Flawed diagnostic set: 748 localhost URLs on all three backends. It had unresolved multi-origin/TLS fixtures and a 3-second CDP-query bias. Lightpanda's conditional rate was 64.67% at 98.56% coverage; current Mimic's was 32.56% at 30.47% coverage. These figures are diagnostics, not a comparable full score.
• Interrupted full series: user cancelled after 3,087 Chrome files (1,944 PASS, 560 FAIL, 47 TIMEOUT, 27 ERROR, 509 SKIP); other backends had not run. It is censored and cannot be compared with old localhost candidates. An earlier collection attempt was also invalidated because optional post-deadline metadata discarded partial callbacks; the collector was fixed and a new series started.

Large original logs, full raw streams, binaries, CA keys and machine-specific receipts remain local and are not published here. The published sample removes transport/navigation IDs and retains assertion evidence. Original and published raw hashes are recorded separately. No GitHub Release or compatibility badge is associated with this baseline.

Offline checks and licensing

python3 -m unittest discover -s tests -v

Checks recompute 77/97 and 27/97 from published raw; test missing/ambiguous ID denominators, preservation of partial callbacks, shared CDP deadlines, and safe hosts restoration against temporary files. They do not launch browsers or change system hosts/trust.

See THIRD_PARTY_NOTICES.md. The original tooling license has not yet been selected; no permissive license is implied. Upstream notices are preserved without relicensing upstream material.