Add an End-to-End test suite for the Flutter SDK - #31
Open
brionmario wants to merge 3 commits into
Open
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
11 tasks
brionmario
force-pushed
the
mobile-e2e-tests
branch
from
September 1, 2026 12:42
5c8b3f2 to
343527a
Compare
Drive the Quickstart through real authentication against a real ThunderID server using Maestro: sign in and sign out, then register a new account and sign in as it. The suite lives in tests/e2e and one script owns the whole run, so CI executes exactly what a contributor runs locally. run-e2e.sh drives an iOS Simulator and needs a Mac; run-e2e.ps1 drives an Android emulator and is the path for contributors on Windows. This is the only layer that exercises the Dart and native halves together, since the unit tests mock the method channel. Expose flow field and action identifiers to the platform accessibility tree. A widget Key never leaves the Flutter tree, so nothing driving the app from outside could see these fields. They now also carry Semantics.identifier, which maps to resource-id on Android and accessibilityIdentifier on iOS. The identifier resolves the server's field identifier rather than its ref, which is what keeps one set of selectors working across all three SDKs; the ref remains the key used for form state and submission. Semantics.identifier requires Flutter 3.19, so the declared minimum moves up from 3.16. flow_form.dart previously had no Semantics at all, which also left it short of the accessibility rule in the repository's own guidelines. Add ThunderIDConfig.allowInsecureConnections and forward it across the method channel. Without it there was no way for a Flutter app to reach a development server over the self-signed certificate ThunderID generates for localhost, which blocked local development on Android as much as it blocked these tests. The native Android SDK honours it for loopback hosts only, and the sample enables it for debug builds only. Refs thunder-id/thunderid#5181 Signed-off-by: Brion <info@brionmario.com>
brionmario
force-pushed
the
mobile-e2e-tests
branch
from
September 1, 2026 15:07
343527a to
71b01d1
Compare
Every flow starts here, and a bare assertVisible gets Maestro's short default lookup timeout rather than the generous budget the rest of the suite uses. On a loaded CI emulator the landing screen does not always render inside it, which failed the Android sign-up flow 19 seconds in, before the flow had done anything. Wait for it explicitly instead. Signed-off-by: Brion <info@brionmario.com>
CI installed whatever Maestro was newest at the time, so the test runner changed under the suite between runs with nothing in the repository to show it: CI has been on 2.10.0 since it shipped while a contributor following the README gets whatever is current, which makes a CI-only failure impossible to reproduce faithfully. Pin it, and bump deliberately after checking the flows. The JUnit report gives a failed run a machine-readable result instead of a console log to scrape, and CI now collects it with the other debug artifacts. Signed-off-by: Brion <info@brionmario.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
The Flutter SDK had unit tests, but they mock the method channel, so they cover the Dart half in isolation and never exercise the bridge to the native SDKs. That leaves the most valuable failures uncovered: a flow step the SDK renders but cannot submit, a field the server renames, or an argument dropped on its way across the channel.
This adds a Maestro suite that drives the Quickstart sample against a real ThunderID server. It signs a user in, signs them out, registers a new account, and signs in as that new account. It is the only layer that tests the Dart and native halves together.
Two SDK changes were needed to make it possible, and both fix problems that affect users beyond the tests.
Approach
Maestro rather than
integration_test. The sign-in and sign-up forms are not static widgets: the server returns a flow definition and the SDK renders it. All three mobile SDKs tag the resulting fields the same way, so one set of selectors and one set of flows works across iOS, Android and Flutter.One script owns the run.
tests/e2e/run-e2e.shstarts a server, provisions the test application and user, builds and installs the sample, and runs the flows. CI calls the same script through a composite action.run-e2e.shdrives an iOS Simulator and needs a Mac;run-e2e.ps1drives an Android emulator and is the path for contributors on Windows.Exposing identifiers to the platform accessibility tree
flow_form.darttagged its fields and actions with a widgetKey. AKeyis internal to the Flutter tree and never reaches the platform accessibility tree, so nothing driving the app from outside could see them, and the tags looked correct in source while being unreachable in practice.They now also carry
Semantics.identifier, which maps toresource-idon Android andaccessibilityIdentifieron iOS. TheKeyis kept so widget tests can keep finding these fields by key.The identifier resolves the server's field
identifierrather than itsref. That is deliberate: the iOS and Android SDKs both tag from the identifier, so preferring it here is what keeps one set of selectors working across all three. Therefremains the key used for form state and submission, so nothing about submission changes.Semantics.identifierrequires Flutter 3.19, so the declared minimum moves from 3.16 to 3.19 and the Dart constraint from 3.2 to 3.3. This is the one change reviewers may want to weigh: it affects consumers pinned to 3.16 through 3.18.Worth noting separately that
flow_form.dartpreviously contained noSemanticsat all, which also left it short of the accessibility rule in this repository's own guidelines.Reaching a development server
ThunderIDConfighad no way to accept the self-signed certificate ThunderID generates forlocalhost. The native Android SDK hasallowInsecureConnections, but the plugin never forwarded it, so a Flutter app simply could not talk to a local server on Android. That blocked local development, not only these tests: anyone following the Flutter quickstart againsthttps://localhost:8090on Android hit the same wall.ThunderIDConfig.allowInsecureConnectionsis now forwarded across the method channel. It is narrow by construction:localhost,127.0.0.1,::1,10.0.2.2) and raises a configuration error otherwise. That change is in the companion Android PR.kDebugMode, which is compiled out of release builds.NSAppTransportSecurityexemption at the app level.Related Issues
Related PRs
Checklist
tests/e2e/README.md, plus the SDK Development section of the contributor docstests/e2e/flows/signin.yaml,tests/e2e/flows/signup.yamlbreaking changelabel added.The minimum Flutter version moving from 3.16 to 3.19 is the only compatibility change. I have left the breaking changes section and label unticked because it does not alter any API, but if the project treats a minimum version bump as breaking, say so and I will fill that section in.
Security checks
The credentials in this PR (
e2e_mobile_user/TestPassword@123) belong to a throwaway account the suite creates on a local server. They are not secrets and grant nothing anywhere else.Verification
flutter analyzereports no issues andflutter testpasses, 66 tests. The suite was run end to end against a freshly downloaded ThunderID v1.0.1 on an iOS Simulator, 2 of 2 flows passing.