ci: tiered integration testing — per-PR mock tier + nightly device tier [QA proposal]

.github/workflows/ci-integration-device.yml

@@ -0,0 +1,90 @@
+# Nightly device-bound integration flows on the lab router (tier: device).
+# Classification and rationale: integration_test/FLOW-TIERS.md
+# RUNNER: waits on a self-hosted runner (label: lab-lan) on the QA lab LAN with
+#   the lab router reachable at 192.168.1.1. A Jetson box is being provisioned
+#   for this. Router admin password comes from the ROUTER_ADMIN_PASSWORD secret.
+#   OPEN (review W-5): integration_web.sh does not read that env var yet; the
+#   password must be threaded into the harness config before first live run.
+# NOTE: this tier uses the repo's own harness (integration_web.sh +
+#   shell_scripts/*) which handles factory reset, PnP skip, and device-mode waits.
+name: Integration (device tier, nightly)
+
+on:
+  schedule:
+    - cron: '30 18 * * *'   # 02:30 Taipei / 18:30 UTC, outside CBT lab hours
+  workflow_dispatch:
+    inputs:
+      tags:
+        description: 'test_meta tags to run (default = device-bound set)'
+        required: false
+        default: 'pnp,wifi,internet_settings,speed_test'
+
+permissions:
+  contents: read
+
+concurrency:
+  group: integration-device
+  cancel-in-progress: false   # never interrupt a run that may have factory-reset the router
+
+jobs:
+  device-flows:
+    runs-on: [self-hosted, lab-lan]
+    timeout-minutes: 240
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup Flutter
+        uses: subosito/flutter-action@v2
+        with:
+          channel: stable
+          cache: true
+
+      - name: Setup SSH for Private Deps
+        uses: webfactory/ssh-agent@v0.9.0
+        with:
+          ssh-private-key: ${{ secrets.UI_KIT_DEPLOY_KEY }}
+
+      - name: Configure Git SSH
+        run: git config --global url."git@github.com:".insteadOf "https://github.com/"
+
+      - name: Setup Environment
+        run: |
+          mkdir -p assets/agents
+          if [ -f assets/agents/env.template ]; then
+            cp assets/agents/env.template assets/agents/.env
+          else
+            touch assets/agents/.env
+          fi
+
+      - name: Install Dependencies
+        run: flutter pub get
+
+      # -k is deliberate: the lab router serves a self-signed cert on an isolated LAN.
+      - name: Preflight - router reachable
+        run: |
+          curl -sk --max-time 10 https://192.168.1.1/ > /dev/null
+          bash integration_test/shell_scripts/get_device_info.sh
+
+      # Device-bound flows per integration_test/FLOW-TIERS.md.
+      # integration_web.sh resolves tags via test_meta/*.json and runs each flow's
+      # declared setUp (factory_reset, skip_setup, pre_paired) before flutter drive.
+      # inputs.tags goes through an env var, never inline interpolation (review W-4).
+      - name: Run device-bound flows
+        env:
+          ROUTER_ADMIN_PASSWORD: ${{ secrets.ROUTER_ADMIN_PASSWORD }}
+          TAGS_INPUT: ${{ inputs.tags || 'pnp,wifi,internet_settings,speed_test' }}
+        run: |
+          bash integration_web.sh -t "$TAGS_INPUT"
+
+      - name: Generate report
+        if: always()
+        run: bash integration_test/shell_scripts/generate_integration_report.sh || true
+
+      - name: Upload report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: device-tier-report-${{ github.run_number }}
+          path: build/integration_test/
+          retention-days: 30

.github/workflows/ci-integration-mock.yml

@@ -0,0 +1,107 @@
+# Per-PR mockable integration flows (tier: mock).
+# Classification and rationale: integration_test/FLOW-TIERS.md
+# Mock JNAP layer flag confirmed in review round 1: the app reads the 'force'
+#   dart-define (lib/constants/build_config.dart), same as integration_web.sh.
+name: Integration (mock tier, per-PR)
+
+on:
+  pull_request:
+    branches: [main, 'dev-1.*']   # integration_test/ exists on the 1.x line only
+    paths:
+      - 'lib/**'
+      - 'integration_test/**'
+      - 'pubspec.yaml'
+
+permissions:
+  contents: read
+
+concurrency:
+  group: integration-mock-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  mock-flows:
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    strategy:
+      fail-fast: false
+      matrix:
+        # 13 mockable flows per integration_test/FLOW-TIERS.md
+        flow:
+          - administration_test
+          - advanced_routing_test
+          - apps_and_gaming_test
+          - dashboard_home_test
+          - dhcp_reservations_test
+          - dmz_test
+          - firewall_test
+          - instant_admin_test
+          - instant_privacy_test
+          - instant_safety_test
+          - local_network_settings_test
+          - menu_test
+          - recovery_and_login_test
+      max-parallel: 4
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup Flutter
+        uses: subosito/flutter-action@v2
+        with:
+          channel: stable
+          cache: true
+
+      - name: Setup SSH for Private Deps
+        uses: webfactory/ssh-agent@v0.9.0
+        with:
+          ssh-private-key: ${{ secrets.UI_KIT_DEPLOY_KEY }}
+
+      - name: Configure Git SSH
+        run: git config --global url."git@github.com:".insteadOf "https://github.com/"
+
+      # env.template is not tracked in git; fall back to an empty .env (review W-1).
+      - name: Setup Environment
+        run: |
+          mkdir -p assets/agents
+          if [ -f assets/agents/env.template ]; then
+            cp assets/agents/env.template assets/agents/.env
+          else
+            touch assets/agents/.env
+          fi
+
+      - name: Install Dependencies
+        run: flutter pub get
+
+      - name: Setup Chrome + chromedriver
+        uses: browser-actions/setup-chrome@v1
+        with:
+          install-chromedriver: true
+
+      - name: Start chromedriver
+        run: |
+          chromedriver --port=4444 &
+          for i in $(seq 1 30); do
+            curl -sf http://127.0.0.1:4444/status > /dev/null && break
+            sleep 1
+          done
+
+      # NOTE: this bypasses integration_web.sh, so test_meta setUp/tearDown hooks do
+      # NOT run. Mock-tier flows must be self-contained (see FLOW-TIERS.md caveat).
+      - name: Run ${{ matrix.flow }} (mocked JNAP)
+        run: |
+          flutter drive \
+            --driver=test_driver/integration_test.dart \
+            --target=integration_test/${{ matrix.flow }}.dart \
+            --dart-define=force=local \
+            -d web-server --headless \
+            --browser-name=chrome --driver-port=4444
+
+      - name: Upload failure artifacts
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: mock-flow-failure-${{ matrix.flow }}
+          path: |
+            build/integration_test/
+          retention-days: 7

integration_test/FLOW-TIERS.md

@@ -0,0 +1,85 @@
+# Integration Flow Tiers: Mockable vs Device-Bound
+
+> Proposal from QA (Deven). Splits the 21 integration flows into a per-PR mock tier
+> and a nightly device tier so the suite runs automatically instead of manually.
+> Companion workflows: `.github/workflows/ci-integration-mock.yml` and
+> `ci-integration-device.yml`.
+
+## Classification principle
+
+Every flow can technically run with mocked JNAP. The split is about where the
+regression value lives:
+
+- **MOCKABLE**: the flow asserts UI behavior and the UI-to-JNAP contract (forms,
+  validation, CRUD echo, navigation). A mocked JNAP layer preserves nearly all of its
+  regression value. Run per-PR, headless, no router.
+- **DEVICE-BOUND**: the flow's value depends on real device behavior (radio apply,
+  WAN state machines, device-mode transitions, real measurements). Run nightly on the
+  lab router via `integration_web.sh`.
+
+**Mock-tier caveat (review W-6):** the mock workflow calls `flutter drive` directly,
+so the `setUp`/`tearDown` hooks declared in `test_meta/*.json` do NOT run in the mock
+tier. Mock-tier flows must therefore be self-contained against the mocked JNAP state.
+Whether all 13 actually are is a first-run validation item.
+
+**Tag caveat (review W-7):** `test_meta` tags cross tiers (e.g. `menu` is on the
+mockable `menu_test` AND device-bound wifi/speed flows). Do not use bare tags to
+select a tier with `integration_web.sh -t`; the `tier` field (open item 3) is the
+real fix.
+
+## The split (13 mockable / 8 device-bound)
+
+### Mockable, per-PR tier
+
+| Flow | Why mockable |
+|---|---|
+| `administration_test` | Toggle persistence is a JNAP Set/Get echo, no HW behavior asserted |
+| `advanced_routing_test` | Route CRUD plus validation error rendering |
+| `apps_and_gaming_test` | DDNS / port-forwarding / triggering form CRUD |
+| `dashboard_home_test` | Outer-layer cards and navigation. OPEN (review W-8): test_meta says the flow includes Speed Test interaction; whether that triggers a real measurement (which a mock run cannot answer) or only renders the card needs confirming before this row is trusted |
+| `dhcp_reservations_test` | Add/edit reservation UI plus Get-back verification |
+| `dmz_test` | Enable switch plus IP/MAC form management |
+| `firewall_test` | Switch states plus IPv6 port-service CRUD |
+| `instant_admin_test` | Password/timezone forms; manual-FW-update entry stubbed |
+| `instant_privacy_test` | UI contract is mockable. Live runs also risk kicking the test client off WiFi (MAC filter), an extra reason to keep it in the mock tier |
+| `instant_safety_test` | Enable flow, JNAP contract only |
+| `local_network_settings_test` | Hostname/LAN IP/DHCP-pool forms; mock avoids the real re-IP dance. Open: confirm the test does not assert post-re-IP reachability (mock cannot answer that) |
+| `menu_test` | Pure navigation, every card and button on the dashboard menu |
+| `recovery_and_login_test` | Login and forgot-password flow against mocked auth responses |
+
+### Device-bound, nightly tier
+
+| Flow | Why device-bound |
+|---|---|
+| `factory_reset_setup_test` | Full PnP on a genuinely reset device; device-mode state machine |
+| `prepaired_reset_setup_test` | Pre-paired PnP path; mesh pairing state is real-device-only |
+| `incredible_wifi_test` | Band settings must apply to real radios |
+| `incredible_wifi_advanced_and_mac_filtering_test` | Advanced radio switches, MLO alert, MAC filtering against real clients |
+| `internet_settings_set_1_test` | IPv4 DHCP/Static/PPPoE, real WAN state transitions |
+| `internet_settings_set_2_test` | PPTP/L2TP/Bridge mode, WAN mode switching |
+| `internet_settings_set_3_test` | IPv6 DHCP/Static/PPPoE, dual-stack behavior |
+| `speed_test_test` | Real measurement; meta itself says only run on devices supporting speed test |
+
+## Open items for review
+
+1. ~~Mock flag~~ **RESOLVED (review round 1, C-1):** the app reads the `force`
+   dart-define (`lib/constants/build_config.dart`); the workflow now passes
+   `--dart-define=force=local`, matching `integration_web.sh:219`.
+2. **Two rows worth a second opinion**: `dashboard_home_test` (Speed Test interaction
+   depth, see W-8 note in its row) and `local_network_settings_test` (post-re-IP
+   reachability assertion).
+3. **`tier` field proposal**: add `"tier": "mock" | "device"` to `test_meta/*.json` so
+   selection is data-driven instead of hardcoded lists, and cross-tier tags (W-7) stop
+   being a foot-gun. Happy to do this as a follow-up once the classification is agreed.
+4. **Model genericity**: flows must stay model-generic across M60/M61/M62. Known model
+   difference: band count (6 GHz coverage in `incredible_wifi_test` is conditional on
+   tri-band hardware). Proposal is to data-drive model differences, never branch test
+   logic on model.
+5. **2.x line**: `integration_test/` exists on `main` and the 1.x line only. Should the
+   suite (or the mock tier at least) be ported to the `usp`/2.x line?
+6. **Admin password threading (review W-5)**: the device workflow exports
+   `ROUTER_ADMIN_PASSWORD`, but `integration_web.sh` never reads it; the harness gets
+   the password from `is_default_admin_password_or_get_password()` (default or config
+   JSON). Before the first live nightly run, thread the secret into the harness config
+   (or agree the lab device stays on the label-default password, which factory resets
+   restore anyway).