Skip to content

P2: add deterministic native-material visual regression coverage #928

Description

@RonenMars

Goal

Turn the current ad-hoc native Liquid Glass screenshots into a deterministic visual-regression test harness for Settings across every retained palette on iOS and Android.

This implements the native-material/theme slice of #610 rather than replacing its broader Maestro coverage work.

Why

The current palette capture flow is useful for manual review, but it has no comparator or CI gate.
It can also accidentally record global error UI over Settings when an unreachable saved server is present.
For example, the translucent card seen in the Android Catppuccin and Latte captures was the app-wide BannerHost error overlay (History failed to load), not Android material or a theme surface.

A visual baseline must represent a known-good, no-server state so design/contrast review is not obscured by unrelated network failures.

Scope

  • Keep the retained palette matrix as the source of truth: 9 dark palettes and 7 light palettes.
  • Capture the Settings viewport for every palette on both platforms.
  • iOS references must run on an iOS 26+ simulator where expo-glass-effect is available, so they exercise native Liquid Glass.
  • Android references must exercise the supported expo-blur palette-aware fallback; they are not expected to reproduce Apple's native material.
  • Test both a representative dark palette and a representative light palette visually in the review output, while retaining all 16 baseline captures for diffing.
  • Keep screenshots as committed, reviewable repository assets; disposable Maestro artifacts remain ignored.

Deterministic capture state

Provide a fixture/setup path that establishes all of the following before a screenshot is accepted:

  • onboarding complete;
  • no paired or saved server;
  • no query, server, history, slow-query, or global BannerHost alert visible;
  • Settings route open;
  • stable window size/device profile recorded with the baseline metadata;
  • fonts and animations settled before capture.

Do not use an Expo development-client clearState: true flow as the only setup mechanism: it clears the remembered Metro endpoint and can return the app to Expo Dev Launcher instead of Threadbase.

The capture runner should fail fast if a known overlay/modal/banner test ID or its visible copy is present.

Implementation outline

  1. Add a platform-aware visual-capture command around the existing Maestro Settings palette flow.
  2. Add a clean-state setup helper or release-style fixture route so the command does not depend on stale emulator/server state.
  3. Store baselines under a platform/device/runtime hierarchy, for example: e2e/visual/native-liquid-glass/settings-themes/<platform>/<device>-<os>/.
  4. Add a small manifest next to the images with platform, device profile, OS/runtime version, app build/commit, color scheme, palette IDs, viewport dimensions, and capture command.
  5. Compare fresh captures against committed baselines in CI using a documented image-diff tool and explicit tolerances/masks for unavoidable system-chrome variation.
  6. Upload actual, expected, and diff images as CI artifacts on failure.
  7. Add an intentional-baseline-update workflow that requires review and updates the manifest alongside the PNGs.
  8. Make the runner report whether it used iOS native Liquid Glass or the Android/unsupported-platform blur fallback.

Acceptance criteria

  • A single documented command captures all 16 Settings palettes on iOS and Android from a clean, no-server state.
  • The command refuses to accept captures with a global error banner/modal, dev launcher, developer menu, or React Native error overlay visible.
  • CI fails when a meaningful visual difference exceeds the documented tolerance.
  • CI failure artifacts clearly show expected, actual, and diff images with palette and platform names.
  • Baseline metadata identifies iOS native Glass versus Android blur fallback.
  • The suite is stable across two consecutive runs on each supported platform.
  • Existing functional Maestro coverage remains runnable independently of visual comparison.

Relevant code and assets

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Soon, but does not gate a releasee2eMaestro / end-to-end test suiteenhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions