Skip to content

test(example): agent-device visual regression PoC for Surface - #5115

Draft
JKobrynski wants to merge 18 commits into
mainfrom
poc/agent-device-visual
Draft

test(example): agent-device visual regression PoC for Surface#5115
JKobrynski wants to merge 18 commits into
mainfrom
poc/agent-device-visual

Conversation

@JKobrynski

@JKobrynski JKobrynski commented Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Motivation

Our component tests assert style props and tree snapshots; they cannot tell whether a component looks right. This is a proof of concept for screenshot-based visual regression tests using agent-device against the existing example-app screens, scoped to Surface since most other components are mid-rework.

The question: can agent-device screenshot Surface deterministically enough to diff against a baseline, and does that diff catch a realistic regression?

What we found

  1. Capture is pixel-deterministic on iOS and Android: 0 changed pixels across warm captures and a relaunch, at every threshold tried (0.1 and 0.02 on iOS, down to 0.01 on Android).
  2. agent-device's default diff threshold (0.1) hides a real Surface regression. A one-level shadow change (elevation 1 rendered as 2) diffs to 0 on both platforms at the default; at 0.02 it is caught cleanly with the same pixel count on three captures each (10,179 px iOS, 9,336 px Android). This is documented behaviour, but any suite built on this must set the threshold explicitly.
  3. Dev-client chrome gets into screenshots. The Expo dev launcher, the iOS first-run onboarding sheet and the floating Tools button each produced false diffs with the app unchanged (2,822 px Android, 3,916 px iOS). A release build removes the class.
  4. Web: open and full-page screenshots work; per-section --crop-on is refused, so a web leg would be full-viewport diffs. Not pursued for a Surface-only PoC.
  5. One agent-device gap: implicit sessions were keyed by cwd only, so two platforms from one checkout needed a hand-named --session. We filed and then withdrew an issue about it; the maintainers fixed it independently in fix(daemon): key the implicit session by workspace and platform agent-device#2587, not yet in a release as of 0.21.3.

Three other things looked like tool issues and were checked against the docs, source and tracker before filing anything: bare find pressing its match (documented; find ... list is the read-only form, and callstack/agent-device#1625 already tracks the ergonomics), snapshot --scope returning success with empty nodes when nothing matches (a docs-clarity remark at most), and screenshot --crop-on being refused on web (an explicit UNSUPPORTED_OPERATION, consistent with "preliminary web support"). None was filed; whether any of them deserves an upstream report is the agent-device maintainers' call and the README records the exact responses. The README also lists what a runner has to handle around the dev client. The tool was driven through its CLI (agent-device 0.21.0); its Node client was not used because it requires adding agent-device as a dependency, which this PoC avoided. If the approach is adopted, the runner should switch to the client.

Full numbers and reproduction steps are in example/visual/README.md; every number resolves to a row in example/visual/evidence/results.csv; the raw per-command JSON behind those rows is on branch poc/agent-device-visual-runner, not in this PR.

Related issue

None. Requested by the maintainers in the team channel as a follow-up to the Surface rework; the goal was a proof of concept before deciding on a visual test suite.

Screenshots / Videos

Realistic break at 0.02, iOS and Android (red = changed pixels, the ring on the Elevation 1 card):

The dev-client's floating Tools button inside the Android crop, app unchanged:

Web, full-page capture (per-section crop refused):

Changes

  • example/src/Examples/SurfaceExample.tsx: testID on the two List.Sections so screenshot --crop-on can target them. Only app change; no library code touched.
  • example/visual/README.md: results, caveats, one reproduction sequence for both platforms, and what a real runner has to handle.
  • example/visual/env.json: the device profiles the baselines are valid for (simulator name and runtime, AVD name, API level, density). No UDID or serial.
  • example/visual/__baselines__/{ios,android}/: four PNGs.
  • example/visual/evidence/: results.csv (74 diff rows; the last column is the raw JSON path on the runner branch), the diff images, three small tree and response excerpts.
  • example/visual/.gitignore: ignores artifacts/, where the runner writes captures, diff images and summary.json.
  • example/visual/run.mjs: one script that runs the loop on either platform: resolve the device from the profile (simulator by name plus runtime through simctl, emulator by AVD name through agent-device devices), boot it if it is down, relaunch, wait for ready, dismiss dev-client chrome, navigate to Surface, capture, diff. Exit 1 on a regression, 2 on a setup error (no matching device, wrong system image, refused open), 3 on a capture-size mismatch; --update re-baselines. run.test.mjs covers its pure helpers with node:test, no device needed.
  • example/package.json: visual script running run.mjs, test:visual running run.test.mjs.

Scope

A PoC. The raw per-command JSON and the full accessibility-tree dumps behind the evidence stay on branch poc/agent-device-visual-runner; this PR carries the summaries they were computed from.

Test plan

With the app installed on an iPhone 17 Pro simulator (iOS 26.5) or an AVD named Pixel_10_Pro (API 37, 480 dpi) and Metro running: yarn example visual --platform ios / --platform android. The runner finds and boots the device itself; expect PASS on both stories (verified 2026-09-16 on both platforms, including cold boots). iOS works on any Mac with that simulator; Android needs the AVD named exactly that, since an AVD name is the only handle agent-device exposes for a stopped emulator. Or follow "Reproducing by hand" in the README. Expect differentPixels: 0 against the committed baselines; apply the one-line Surface break described there, relaunch the app so it fetches the bundle, capture, diff: about 10,179 px (iOS) or 9,336 px (Android) at 0.02 and match: true at 0.1.

Out of scope

A web loop, CI, other components, dark theme, cross-platform comparison.

@JKobrynski
test(example): add agent-device visual regression PoC for Surface
32391c4 
Proof of concept for screenshot-based visual regression testing using
agent-device against the existing example screens, scoped to Surface.

- example/visual/run.mjs: one command per platform (iOS simulator,
  Android emulator) that opens the example app, navigates to the Surface
  screen, crops the "Elevated surface" / "Flat surface" sections, and
  diffs them against committed baselines. Exits 1 on regression;
  --update re-baselines.
- example/visual/README.md: results. Noise floor is 0 px on both
  platforms across warm captures and a relaunch. agent-device's default
  diff threshold (0.1) misses a realistic one-level shadow regression on
  both platforms; 0.02 catches it with noise still at 0.
- example/visual/env.json: pinned device/OS/density/versions the
  baselines are valid for.
- SurfaceExample: testID + accessible on the two List.Sections so
  `screenshot --crop-on` can target them.

Twelve agent-device dogfooding issues were recorded locally for filing.
@JKobrynski
test(example): address review on the visual regression PoC
e4f0311 
- Drop `accessible` from the two List.Sections. With it, iOS collapsed each
  section into a single accessibility element (XCUITest tree evidence in
  evidence/a11y/). `testID` alone resolves `screenshot --crop-on` on both
  platforms with identical rects, so the example app's accessibility tree
  is unchanged by the hook.
- Rename the test ids to surface-example-{elevated,flat} so they cannot be
  confused with the library defaults removed in #5088 / #5099. Baselines
  renamed to __baselines__/<platform>/<story>.png on both platforms.
- Commit the evidence: raw diff JSON behind every table, break diff images,
  issues.md, the a11y trees, and the final script runs.
- run.mjs: env.json mismatch and crop-size mismatch are hard failures
  (exit 2 / 3) unless --force; --update can create a missing baseline;
  always relaunch so the bundle is fresh (Fast Refresh was seen to stop
  reaching Android); handle the Expo dev menu and dev launcher; because
  the example app persists navigation state, go Back to the list root and
  press the Surface row by position, not the header by label; --force-full
  on every snapshot; summary.json written on every exit; --out.
- Re-measure the realistic break three times per platform at pre-declared
  thresholds: bit-identical (iOS 10,179 px, Android 9,336 px at 0.02;
  0 at 0.1) and identical to the original single captures.
- README: status, evidence links, a11y section, re-measurement section,
  issues 13-18, updated Running-it.
@JKobrynski
test(example): address round-2 review on the visual regression PoC
63e1699 
- Entry point: replace `import.meta.main` (Node >= 24.2 only) with a
  realpath comparison of argv[1] and the module URL. On Node 20/22 the
  script previously printed nothing and exited 0. Verified on 20.20.2.
- `--update` skips the capture-size check and prints the captured
  dimensions, so a new story can get a baseline; regression test added.
- `--force` covers the device check only; a size mismatch in diff mode
  has no override.
- Remove the dev-launcher geometric fallback, the unreachable size-check
  branch, the duplicate iOS version comparison and unused exports.
- List-root detection requires the "Examples" title; argument errors
  write a summary.json too.
- Strip npm_config_* from the spawned environment so the nested
  `npx agent-device` works when the script itself runs under `npx -p`.
- run.test.mjs (node:test, 6 cases) and evidence/summarize.mjs, the
  committed generator behind evidence/summary.md.
- Evidence: drop the 12 wait.json files and two full-tree snapshots;
  add the round-2 runs on Node 24 and Node 20.
- README: Android section no longer credits `accessible`; Node
  requirement, test and generator documented; env.json drops the
  parent-commit pin.
@JKobrynski
test(example): address round-3 review on the visual regression PoC
e9245dc 
- Evidence citations: every path in evidence/issues.md now resolves to a
  committed file. Issue 15 re-captured (evidence/runner/ios-*), issues 4
  and 14 rewritten onto committed evidence; the two lost logs are stated
  as not kept.
- Crop size is checked against the baseline PNG's own header, per story,
  so `--update` on a new story is followed by a passing run; env.json
  drops the per-platform crop pins and totalPixels.
- Drop the arg-error summary machinery; an argument error prints and
  exits 2.
- Device observers are injected; the --force test no longer shells out.
  `yarn example test:visual` runs the seven node:test cases; README says
  it is manual, not Jest/hook/CI. README Node version is 20+ throughout.
- Guardrails triggered for real: exit 2 on a UDID that is not attached,
  exit 3 on an 804x428 capture against the 3x baseline
  (evidence/runs/ios/round3-exit*).
- New: the Expo dev-client's floating "Tools" button inside the crop
  produced a deterministic false FAIL on Android with src clean
  (issue 19). The runner now detects it, disables it via the dev menu,
  or stops with exit 2; evidence in evidence/runner/android-* and
  evidence/runs/android/round3-*.
@JKobrynski
test(example): cut the visual regression PoC back to PoC scope
e59fd2e 
Three review rounds turned the reproduction script into a 1,200-line
runner and the evidence folder into 29,000 lines of accessibility-tree
dumps. Neither is what a proof of concept needs. This commit keeps:

- the results (README), the pinned environment, the four baselines and
  the two test ids;
- the diff JSON every table is computed from, the diff images, and the
  19 agent-device issues with their evidence;
- two small excerpts replacing the full accessibility-tree snapshots,
  each saying how to regenerate the full tree;
- hand-run CLI commands to reproduce, instead of a script.

Removed from this PR: run.mjs, run.test.mjs, evidence/summarize.mjs and
summary.md, evidence/runs, evidence/runner, the full evidence/a11y
trees, and the example/package.json test script. All of it is preserved
unchanged on branch poc/agent-device-visual-runner, and the README
lists what a real runner would have to handle, learned from it.
@JKobrynski
test(example): collapse PoC evidence into one results table
7917bdf 
96 per-threshold diff JSON files, each ~20 lines, become one
evidence/results.csv with 72 rows (platform, capture, threshold,
pixels, regions, match, source file). Keep one diff image per platform
for the realistic and gross breaks plus the Tools-button false FAIL.
Raw JSON and the other images stay on poc/agent-device-visual-runner.
This was referenced Sep 14, 2026
Closed
Closed
Closed
Closed
Closed
@JKobrynski
test(example): validate the PoC against the ask and close the gaps
122a82c 
- Web: actually tried. open and full-page screenshot work; --crop-on is
  refused (CROP_TARGET_NOT_ACCEPTED / PENDING_PIXEL_IDENTITY_EVIDENCE).
  Transcript in evidence/web-excerpt.json, issue 20.
- One reproduction sequence for both platforms with a table of the
  platform-specific values, instead of two diverging command blocks;
  the Android dev-launcher step is now in it. Points at the runner
  branch as the single-script form.
- Five issues filed on callstack/agent-device (#2579-#2583) and linked
  from issues.md.
- No em dashes; three self-certifying phrases removed.
@JKobrynski
test(example): drop references to upstream issues that were withdrawn
cf143f9
@JKobrynski
test(example): drop the Node-client entry from the agent-device findings
731699e
@JKobrynski
test(example): re-verify the agent-device findings
21dc2ab 
All 19 re-checked against 0.21.3 docs and source, the emulator, and the upstream tracker. None was an unreported agent-device bug: one fixed upstream (#2580), two documented, the rest our build, our runner, or not reproducible. File restructured accordingly; original numbers kept in brackets.
@JKobrynski
test(example): iOS re-verification of the findings; README matches
28b4779 
Baselines hold bit-for-bit on Xcode 27.0 once dev-client chrome is off; the realistic break still measures 10,179 px at 0.02 and 0 at 0.1. Corrections from the iOS pass: the iOS dev menu is a one-time onboarding scrim, not a per-relaunch sheet; the floating Tools button also crosses the iOS crop (3,916 px); scroll bottom succeeds on a clean screen; the stale-ref hint quote was wrong; a no-match --scope returns empty nodes with success. README no longer calls any finding an agent-device issue.
@JKobrynski
test(example): extend the dev-client caveat to the ios chrome
a7dbc70
@JKobrynski
test(example): drop the agent-device findings file
a81fde3 
Re-verification showed none of the recorded observations was an agent-device bug, so the file documented nothing. The practical lessons that survive are in the README's runner section.
@JKobrynski
test(example): back the dev-client numbers, fix the region count
bd9de7b 
Two results.csv rows for the Tools-button false FAILs (Android 2,822 px, iOS 3,916 px); the iOS realistic break has 3 regions, not one; state that the Node client was not used and why.
@JKobrynski
test(example): bring the runner back into the PoC
f138b25 
run.mjs (one script for both platforms), run.test.mjs (seven node:test cases) and evidence/summarize.mjs return from the runner branch unchanged; example/package.json gains test:visual. The raw JSON and tree dumps stay on that branch. Verified today: both platforms PASS through the script against the committed baselines.
@github-actions

github-actions Bot commented Sep 16, 2026

Copy link
Copy Markdown

Found potential problems with the pull request:

  • The description is too long. Please keep it under 1000 characters.

@JKobrynski
test(example): drop the unrunnable summarizer, one node-client story
4abbd01
@JKobrynski
test(example): resolve the device from its profile, drop --force
393780c 
env.json pins the simulator name plus runtime and the AVD name, API level and density; no UDID or serial. run.mjs resolves the instance through simctl and agent-device devices, boots it with agent-device boot, binds fixed sessions paper-visual-ios / paper-visual-android on open, retries open once when the session is bound to a device that is gone, and sets adb reverse after an Android boot. --force is removed; a wrong Android image is exit 2 with the re-baseline steps. Three new node:test cases with captured device shapes. yarn example visual is the entry point.
@JKobrynski
test(example): address review, settle after adb keys, reject empty story
5c8ad73 
Runner: poll the tree after adb keyevents instead of snapshotting at once; reject an empty --story list, which produced a green run with no captures; exit 2 with a summary-less message when --out cannot be created; fail on a diff response without a boolean match; verify the PNG signature; keep the Android block out of the iOS summary; clearer scroll-budget and unparseable-output messages; remove em dashes. Tests: parseArgs and isAvailable cases, 11 total. README: relaunch not metro reload, gross-break 0.1 images, flags, summary.json caveat, npx as a CI blocker in follow-ups, results.csv provenance. results.csv: one wrong threshold, one unquoted field. env.json: drop dead diffThresholds and a note masquerading as a baseline.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@JKobrynski