ci: tiered integration testing — per-PR mock tier + nightly device tier [QA proposal]#1149
ci: tiered integration testing — per-PR mock tier + nightly device tier [QA proposal]#1149DevenDucommun wants to merge 2 commits into
Conversation
Splits the 21 integration flows into a mockable per-PR tier (13 flows, headless, no router) and a device-bound nightly tier (8 flows, lab router via integration_web.sh). Classification and rationale in integration_test/FLOW-TIERS.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
AustinChangLinksys
left a comment
AustinChangLinksys
left a comment
There was a problem hiding this comment.
🤖 Automated Review — Round 1 · 61bef92..ec12768 (full)
Verdict: 💬 COMMENT — 1 Critical finding (--dart-define key mismatch renders mock tier permanently inoperative); manual decision required before merge.
| Conf. | Where | Issue (one-liner) | |
|---|---|---|---|
| 🔴 | 🟢 High | ci-integration-mock.yml:71 |
[both reviewers] --dart-define=FORCE_COMMAND_TYPE=local does not match app's force key — mock JNAP layer never activates |
| 🟢 High | ci-integration-mock.yml:64 |
assets/agents/env.template missing from git — Setup Environment step fails, all 13 mock flows break immediately |
|
| 🟢 High | ci-integration-device.yml (setup steps) |
device workflow missing SSH key setup (UI_KIT_DEPLOY_KEY) — flutter pub get will fail if private deps present |
|
| 🟢 High | ci-integration-device.yml (setup steps) |
device workflow missing .env init step present in mock workflow and generate-screenshots.yml |
|
| 🟢 High | ci-integration-device.yml:55 |
inputs.tags directly interpolated into shell run: — injection surface on self-hosted runner |
|
| 🟢 High | ci-integration-device.yml:50-51 |
ROUTER_ADMIN_PASSWORD set as env but never consumed by integration_web.sh — device flows silently fail with wrong password |
|
| 🟢 High | ci-integration-mock.yml:65-77 |
mock workflow calls flutter drive directly, silently skips test_meta setUp/tearDown hooks declared in all 13 flow JSONs |
|
| 🟢 High | integration_test/FLOW-TIERS.md |
menu tag spans both tiers — integration_web.sh -t menu will mix mock and device-bound flows |
|
| 🟢 High | integration_test/FLOW-TIERS.md:28 |
dashboard_home_test mockable rationale ("speed-test card stubbed") contradicts test_meta description which includes Speed Test interaction |
|
| 💡 | 🟢 High | both new workflows | Missing permissions: blocks — workflows run with default (potentially over-broad) GITHUB_TOKEN |
| 💡 | 🟢 High | ci-integration-mock.yml:63-64 |
chromedriver backgrounded with & without readiness check — flutter drive may connect before chromedriver is ready |
| 💡 | 🟢 High | ci-integration-device.yml:41 |
curl -sk skips TLS validation; acceptable on isolated lab LAN but should be documented explicitly |
| 💡 | 🟢 High | ci-integration-device.yml:18,51 |
Hardcoded tag list will drift from tier classification once flows are added; propose adopting tier field (Open item #3) for selection |
| 💡 | 🟡 Med | integration_test/FLOW-TIERS.md:35 |
local_network_settings_test rationale should clarify validation boundary (does mock verify post-re-IP reachability?) |
| 💡 | 🟢 High | integration_test/shell_scripts/jnap.sh:1 |
Pre-existing shebang typo #!/uer/bin/bash — opportunity to fix while touching this area |
Confidence: 🟢High = code-verified · 🟡Med = located + reasoned, not fully confirmed · ⚪Low = speculative, please double-check.
Items marked [both reviewers] were independently flagged by two agents → higher confidence.
🔴 Critical — Complete Evidence Chain
C-1 · --dart-define Key Mismatch — Mock JNAP Layer Never Activates
Severity: Critical | Confidence: High | Both reviewers independently confirmed
Location chain:
.github/workflows/ci-integration-mock.yml:71-- wrong key injectedlib/constants/build_config.dart:44-45-- actual key the app readsintegration_web.sh:219-- correct existing usage
(1) Code in the new workflow (ci-integration-mock.yml:71):
--dart-define=FORCE_COMMAND_TYPE=local \(2) Code the app actually reads (build_config.dart:44-45):
static ForceCommand forceCommandType = ForceCommand.reslove(
const String.fromEnvironment('force', defaultValue: 'none'));The app reads key force, not FORCE_COMMAND_TYPE.
(3) Correct usage in the existing harness (integration_web.sh:219):
--dart-define=force="${force}" \ # force defaults to "local" (line 29)Why it breaks: String.fromEnvironment('force', ...) receives no matching define -> returns 'none' -> ForceCommand.reslove('none') -> ForceCommand.none -> BuildConfig.isLocal() returns false -> JNAP calls are not intercepted -> all 13 mock-tier flows attempt to connect to a real JNAP endpoint (https://localhost/JNAP/) that does not exist on ubuntu-latest -> all flows fail with network errors rather than valid mock results.
Trigger condition: Every PR that touches lib/**, integration_test/**, or pubspec.yaml triggers ci-integration-mock.yml. All 13 matrix jobs will fail from day one.
Fix:
# ci-integration-mock.yml line 71
--dart-define=force=local \Also update FLOW-TIERS.md Open item #1 with the confirmed correct flag name.
⚠️ Warnings — Evidence and Fixes
W-1 · assets/agents/env.template Missing from Git — Mock CI Breaks Immediately
Confidence: High | [single - Reviewer A]
Location: ci-integration-mock.yml:64
- name: Setup Environment
run: cp assets/agents/env.template assets/agents/.envgit ls-files assets/agents/ returns empty. GitHub Actions run: steps default to set -eo pipefail; cp failure (exit 1) terminates the step -> all 13 matrix jobs fail at Setup Environment before any test runs.
Fix: Either commit a sanitized assets/agents/env.template to the repo, or add a guard:
run: |
mkdir -p assets/agents
[ -f assets/agents/env.template ] && cp assets/agents/env.template assets/agents/.env || touch assets/agents/.envW-2 · Device Workflow Missing SSH Key Setup — flutter pub get Fails
Confidence: High | [single - Reviewer B]
Location: ci-integration-device.yml (between checkout and flutter pub get)
ci-integration-mock.yml:49-56 and generate-screenshots.yml:137-143 both include the webfactory/ssh-agent@v0.9.0 step with UI_KIT_DEPLOY_KEY. ci-integration-device.yml does not. If pubspec.yaml references private git dependencies (e.g. ui_kit_library), flutter pub get on the device runner will fail with SSH auth errors.
Fix: Add the same two SSH setup steps before Install Dependencies in ci-integration-device.yml.
W-3 · Device Workflow Missing .env Init Step
Confidence: High | [single - Reviewer B]
Location: ci-integration-device.yml (setup steps)
Both ci-integration-mock.yml:62 and generate-screenshots.yml:146 include the cp assets/agents/env.template assets/agents/.env step. ci-integration-device.yml has no equivalent. integration_web.sh calls init_config / set_global_config which may depend on .env being present.
Fix: Add the same Setup Environment step in ci-integration-device.yml.
W-4 · inputs.tags Directly Interpolated into Shell — Injection Surface
Confidence: High | [single - Reviewer A]
Location: ci-integration-device.yml:55
run: |
bash integration_web.sh -t "${{ inputs.tags || 'pnp,wifi,internet_settings,speed_test' }}"${{ inputs.tags }} is a free-text workflow_dispatch input interpolated directly into a run: shell block. GitHub's own hardening guide explicitly prohibits this pattern. On a self-hosted runner with lab LAN access (192.168.1.1 reachable), the attack surface is meaningful for anyone with repo write access.
Fix:
env:
TAGS_INPUT: ${{ inputs.tags || 'pnp,wifi,internet_settings,speed_test' }}
run: |
bash integration_web.sh -t "$TAGS_INPUT"W-5 · ROUTER_ADMIN_PASSWORD Never Consumed — Device Flows Silently Fail
Confidence: High | [single - Reviewer A]
Location: ci-integration-device.yml:50-51
env:
ROUTER_ADMIN_PASSWORD: ${{ secrets.ROUTER_ADMIN_PASSWORD }}Searched all .sh and .dart files — zero references to $ROUTER_ADMIN_PASSWORD. The harness reads the admin password via is_default_admin_password_or_get_password() in utils.sh:190-195, which returns "admin" or a config JSON value — never the environment secret. If the lab router password is not the default "admin", all JNAP calls return 401 and flows fail silently.
Fix: Thread the secret into the harness via a --admin-password "$ROUTER_ADMIN_PASSWORD" argument to integration_web.sh, or inject it into the config JSON before running.
W-6 · Mock Workflow Directly Calls flutter drive — test_meta setUp Hooks Silently Skipped
Confidence: High | [single - Reviewer B]
Location: ci-integration-mock.yml:65-77
All 13 mockable flows' test_meta/*.json files declare setUp hooks (e.g. "setUp": ["factory_reset", "skip_setup"]). integration_web.sh:193-213 executes these hooks before each case. The mock workflow calls flutter drive directly, bypassing the harness entirely. If any test's Dart code assumes initial state prepared by setUp, it will fail or produce invalid results without explanation.
Fix: Document in FLOW-TIERS.md that mock-tier flows must be self-contained (no setUp pre-state assumed), or add a no-op mock setUp path in integration_web.sh.
W-7 · menu Tag Spans Both Tiers — integration_web.sh -t menu Mixes Flows
Confidence: High | [single - Reviewer B]
Location: integration_test/FLOW-TIERS.md (classification table)
menu_test.json has "tags": ["menu"] (mockable); incredible_wifi_test.json, speed_test_test.json, and incredible_wifi_advanced_and_mac_filtering_test.json also have the menu tag but are device-bound. integration_web.sh:57-67 does pure string tag matching with no tier filter.
Fix: Implement Open item #3 ("tier" field in test_meta JSONs + -r flag in harness), or add a warning in FLOW-TIERS.md about cross-tier tag overlap.
W-8 · dashboard_home_test Mockable Rationale Contradicts test_meta
Confidence: High | [single - Reviewer B]
Location: integration_test/FLOW-TIERS.md:28
FLOW-TIERS.md: "speed-test card stubbed" vs test_meta description: "Test... including Devices, Privacy, Speed Test, and Wi-Fi". If dashboard_home_test triggers a real speed-test measurement request, mock-tier runner will have no device to answer it.
Fix: Resolve Open item #2 — confirm whether the test merely renders the Speed Test card UI or triggers a measurement, and update the rationale.
✅ What looks good
concurrency.cancel-in-progress: falseon device workflow correctly prevents mid-run factory-reset interruption.concurrencygroup includesgithub.refon mock workflow — per-branch isolation, stale runs cancelled on new push.strategy.fail-fast: falseon mock matrix — a single flow failure does not abort the other 12.timeout-minutes: 240on device workflow — realistic for PnP + factory-reset cycle.upload-artifactwithif: always()on both workflows — failure artifacts preserved for investigation.matrix.flowlist is static YAML — no runtime user input interpolated into flow names, no injection risk there.max-parallel: 4cap — friendly to shared runner pool.retention-daystiered correctly: 7d for mock (frequent), 30d for device nightly (less frequent, higher debug value).- FLOW-TIERS.md Open items are honest and specific — author proactively identified unresolved questions.
Cross-reviewed by two independent agents (security+correctness / architecture+maintainability). Automated — please sanity-check before merge.
AustinChangLinksys
added
the
need-decide
- mock: --dart-define=force=local (C-1, matches build_config.dart 'force' key and integration_web.sh:219), env.template guard (W-1), chromedriver readiness wait, permissions block - device: SSH-agent + env setup steps (W-2/W-3), tags via env var not inline interpolation (W-4), permissions block, TLS + password notes - FLOW-TIERS: open item 1 resolved, mock-tier self-containment caveat (W-6), cross-tier tag warning (W-7), dashboard_home rationale honesty (W-8), W-5 password threading recorded as open item Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for the fast round 1, this was exactly the review the draft needed. Pushed Fixed
Documented as open rather than fixed
Still a draft until W-5/W-6 have answers and the lab runner exists (Jetson provisioning is planned for next week). |
AustinChangLinksys
left a comment
AustinChangLinksys
left a comment
There was a problem hiding this comment.
🤖 Automated Review — Round 2 · 61bef92..4e13f92 (full)
Verdict: ✅ APPROVE — Prior Critical (C-1, --dart-define key mismatch) is correctly fixed; no new Critical findings. 4 Warnings remain (chromedriver silent-failure gap, duplicated bootstrap, unpinned action, deferred password threading).
| Conf. | Where | Issue (one-liner) | |
|---|---|---|---|
| 🟢High | ci-integration-mock.yml:60-65 |
Chromedriver readiness loop exits code 0 even if chromedriver never starts — flutter drive gets cryptic connect error |
|
| 🟢High | Both workflows, bootstrap steps | 5-step setup sequence duplicated verbatim — DRY violation, version-drift risk across workflows | |
| 🟢High | ci-integration-mock.yml (Setup Chrome step) |
browser-actions/setup-chrome@v1 pinned at major tag only — no SHA pin on ubuntu-latest supply-chain boundary |
|
| 🟢High | ci-integration-device.yml:63 |
W-5 (acknowledged deferred): ROUTER_ADMIN_PASSWORD exported but never read by harness — silent wrong-password on non-default lab router |
|
| 🟡Med | ci-integration-mock.yml:72 |
Static matrix.flow path pattern safe today; latent traversal risk if matrix ever made dynamic — add safety comment |
|
| 💡 | 🟢High | Both workflows, long-running steps | [both reviewers] No per-step timeout-minutes — concurrency.cancel-in-progress: false + 4h job ceiling = runner monopolized on hang |
| 💡 | 🟡Med | ci-integration-device.yml, device-flows job |
No environment: protection gate — ROUTER_ADMIN_PASSWORD + UI_KIT_DEPLOY_KEY exposed without approval gate |
| 💡 | 🟡Med | ci-integration-mock.yml, Upload step |
Artifact content from build/integration_test/ undocumented — confirm mock runs write no sensitive data |
| 💡 | 🟢High | ci-integration-mock.yml:40 |
max-parallel: 4 has no rationale comment (runner quota cap? cost? measured optimum?) |
| 💡 | 🟡Med | integration_test/FLOW-TIERS.md, Open items |
Open items 2–6 have no owner/milestone — consider converting to GitHub Issues to avoid tracking drift |
| 💡 | ⚪Low | ci-integration-mock.yml:10-14 |
pubspec.lock absent from paths: trigger — lockfile-only dep upgrade won't trigger mock CI |
Confidence: 🟢High = code-verified · 🟡Med = located + reasoned, not fully confirmed · ⚪Low = speculative, please double-check.
Items marked [both reviewers] were independently flagged by two agents → higher confidence.
⚠️ Warnings — Evidence and Fixes
W-new-1 · Chromedriver Readiness Loop Exits Code 0 on Failure
Confidence: High | [single — Reviewer B]
Location: ci-integration-mock.yml:60-65 (Start chromedriver step)
chromedriver --port=4444 &
for i in $(seq 1 30); do
curl -sf http://127.0.0.1:4444/status > /dev/null && break
sleep 1
doneThe for loop exits with the exit code of the last command executed. If break was never hit (chromedriver never became ready), the last command is sleep 1, which exits 0. The step succeeds silently and the subsequent flutter drive step fails with a cryptic "cannot connect to driver" error rather than an actionable "chromedriver not ready" diagnostic.
Note: Reviewer A accepted the loop addition as sufficient fix from Round 1. Reviewer B's analysis of bash exit-code semantics is technically correct — the fix is incomplete.
Fix:
chromedriver --port=4444 &
READY=0
for i in $(seq 1 30); do
curl -sf http://127.0.0.1:4444/status > /dev/null && READY=1 && break
sleep 1
done
if [ "$READY" -ne 1 ]; then
echo "::error::chromedriver did not become ready on port 4444 within 30 seconds"
exit 1
fiW-new-2 · 5-Step Bootstrap Sequence Duplicated Verbatim Across Both Workflows
Confidence: High | [single — Reviewer B]
Location: Both ci-integration-mock.yml and ci-integration-device.yml, steps 1–5
The following steps are copy-pasted identically:
actions/checkout@v4Setup Flutter(subosito/flutter-action@v2, channel: stable, cache: true)Setup SSH for Private Deps(webfactory/ssh-agent@v0.9.0)Configure Git SSH(git config --global url rewrite)Setup Environment(mkdir + cp/touch .env)
This is ~25 lines of duplicated YAML. When the Flutter channel bumps, the SSH agent action updates, or the .env path changes, both files must be updated in sync. FLOW-TIERS.md open item 5 (2.x porting) would duplicate this block a third time.
Fix: Extract into .github/actions/setup-flutter-env/action.yml composite action, or a reusable workflow. Both integration workflows then call uses: ./.github/actions/setup-flutter-env.
W-new-3 · browser-actions/setup-chrome@v1 Pinned Only at Major Version
Confidence: High | [single — Reviewer B]
Location: ci-integration-mock.yml (Setup Chrome + chromedriver step)
- name: Setup Chrome + chromedriver
uses: browser-actions/setup-chrome@v1
with:
install-chromedriver: true@v1 is the least-specific pin in both workflows (others use @v4, @v2, @v0.9.0). This action installs binaries onto ubuntu-latest; a supply-chain compromise or breaking change pushes transparently to all mock-tier PR runs. GitHub's own hardening guide and OSSF Scorecard flag this pattern.
Fix: Pin to the current commit SHA with a version comment:
uses: browser-actions/setup-chrome@<40-char-SHA> # v1.x.xW-5 (Deferred/Acknowledged) · ROUTER_ADMIN_PASSWORD Exported but Harness Never Reads It
Confidence: High | [single — Reviewer A] — acknowledged in FLOW-TIERS.md open item 6
Location: ci-integration-device.yml:63 (env block in "Run device-bound flows" step) + FLOW-TIERS.md open item 6
The secret is correctly exported as an env var but integration_web.sh reads the admin password via is_default_admin_password_or_get_password() (config JSON or label-default), never $ROUTER_ADMIN_PASSWORD. FLOW-TIERS.md open item 6 explicitly tracks this: "Before the first live nightly run, thread the secret into the harness config."
Acceptable for now — factory resets restore the default password. Risk: if the lab router is ever hardened with a non-default password, nightly runs fail with silent 401s.
Fix (before first live run): Pass --admin-password "$ROUTER_ADMIN_PASSWORD" to integration_web.sh, or add a pre-flight credential-validation step.
W-new-4 · Static matrix.flow Path Pattern — Latent Traversal Risk
Confidence: Medium | [single — Reviewer A]
Location: ci-integration-mock.yml:72 (flutter drive --target line)
--target=integration_test/${{ matrix.flow }}.dart \Matrix is currently a static hardcoded YAML allowlist — current risk is low. If the matrix is ever made dynamic (e.g., via fromJson reading FLOW-TIERS.md), a crafted entry like ../../../etc/passwd would produce --target=integration_test/../../../etc/passwd.dart, enabling path traversal.
Fix (minimal): Add a safety comment:
# SECURITY: matrix.flow is a static allowlist. Do NOT make this dynamic
# (e.g. via fromJson) without adding path-sanitization or allowlist validation.✅ What looks good
- C-1 RESOLVED:
--dart-define=force=localnow matches the keylib/constants/build_config.dartreads — mock JNAP layer activates correctly. - W-1 RESOLVED: Both workflows guard
.envinit withif [ -f env.template ]; then cp … else touch …; fi— graceful fallback when template not in git. - W-2 RESOLVED: Device workflow now includes
webfactory/ssh-agent@v0.9.0+Configure Git SSH— private deps will resolve on lab runner. - W-3 RESOLVED: Device workflow has a Setup Environment step matching the mock workflow.
- W-4 RESOLVED:
inputs.tagsthreaded viaTAGS_INPUTenv var — no inline${{ }}interpolation inside shellrun:block. - Permissions RESOLVED: Both workflows declare
permissions: contents: read— GITHUB_TOKEN scoped to minimum. - chromedriver readiness PARTIALLY ADDRESSED: Polling loop added (see W-new-1 for the remaining gap).
- TLS skip documented:
curl -skcomment explains self-signed cert on isolated lab LAN. - W-6/W-7/W-8: All three acknowledged with clear caveats in both FLOW-TIERS.md and workflow inline comments — honest about architectural limitations.
concurrency.cancel-in-progress: falseon device workflow correctly prevents mid-factory-reset interruption.strategy.fail-fast: falseon mock matrix — one flow failure does not abort the other 12.- Static matrix list has no user-input interpolation for flow names.
- Artifact
retention-daystiered correctly: 7d mock (frequent), 30d device nightly (higher debug value). - FLOW-TIERS.md open items are specific, honest, and forward-looking.
Cross-reviewed by two independent agents (security+correctness / architecture+maintainability). Automated — please sanity-check before merge.
|
Thanks for picking this up, Deven. Worth some context up front: we actually ran this suite against real hardware in an automated setup for the better part of a month a while back. We stopped not because the flows were wrong, but because of stability — the device-tier reality of flaky runs and unreliable device initialization/reset made the results untrustworthy enough that the signal wasn't worth it. So the value of this PR isn't "finally automating it" — it's the tiering idea: pulling the mockable UI/contract flows out from under that device-stability problem so they can run reliably without a router. That framing is exactly right, and That said, I'd like us to treat this as a design proposal to align on, not a change to merge yet — and before we do, I want to pin down a few things about the suite itself, because they change what these workflows should look like. (The bot's APPROVE is on YAML quality — key, injection, permissions — it can't speak to whether the suite actually runs green, which is the real gate.) 1. What These flows are grey-box, in-process E2E: they're compiled into the app binary and import app internals directly (
2. What we're testing (goal) There are two orthogonal goals hiding in one PR:
Right now the device tier is bound to a nightly cron on HEAD, which reads like an extension of goal 1. If the intent is goal 2 (validating releases), it should trigger on a release tag / dispatch with a ref, and we should surface the firmware × model matrix (this also connects to open item 4 on model genericity). Worth deciding which goal it serves. Trigger cost — should the full suite gate every PR? The mock tier currently fires on every PR touching 3. Constraints / what actually blocks each tier
Suggestion: split this into two PRs. The mock tier can land on its own once we've done the first-run validation and get real per-PR value. The device tier should wait for the runner, the On the open questions:
Really appreciate the work — this moved the suite forward. Let's lock the positioning and split the tiers, and the mock half can start earning its keep quickly. |
|
Austin, this is a great review and it sent me back to the source with fresh eyes. A couple of things changed once I actually reproduced the state, and some of them are corrections to my own PR, so let me lay it out straight. The honest headline: the mock tier does not work yet, and not for the reasons my FLOW-TIERS notes implied. I had been treating Second, the current runs are red before any of that even matters. The last run at On the missing And the runner plan needs to reopen. Flutter publishes no official Linux ARM64 SDK, so the Jetson can't just pull the pinned toolchain. Options are a community ARM64 build, an x64 runner, or building the engine. Not settled. Here is where I land, and it lines up with your split:
On 2.x, I agree the in-process suite is 1.x-source-bound and should not be mechanically ported. I am reading your steer as a direction and not a ruling, so I am treating 2.x as still open rather than settled. The one thing I want to keep is the fast contract-level regression the mock tier was supposed to give. If we are going Playwright plus screenshot on the Bottom line: the classification stands as a proposal, but I oversold the mock tier as runnable. Contract double and clean bootstrap first, then we validate the 13 for real. Thanks for catching this before it went anywhere. |
|
Deven — this is exactly the kind of dig-into-the-source reply that makes the difference, thank you. A few things land on my side: On the On Two docs attached for context — this is the history the PR is picking up:
Agreed on the plan, as you laid it out:
On 2.x — yes, treat it as open, not settled. And I think your framing is the right one: if Thanks again — better to find all this now than after it shipped. linksys_now_using_flutter_integration_testing.md |
|
Austin, the two docs did not come through. Nothing is attached on the comment for me, and I cannot find either file in the repo, so I think the upload did not take. Could you re-attach |
|
Hi @DevenDucommun |
2 participants
Summary
QA proposal to get the existing
integration_test/suite (21 flows) running automatically instead of manually. It splits the flows into two tiers:lib/,integration_test/, orpubspec.yaml(1.x line branches only).integration_web.shharness.Full classification with per-flow reasoning:
integration_test/FLOW-TIERS.md.What this PR contains
.github/workflows/ci-integration-mock.yml(per-PR, ubuntu, headless Chrome).github/workflows/ci-integration-device.yml(nightly cron at 02:30 Taipei, self-hostedlab-lanrunner)integration_test/FLOW-TIERS.md(the classification + open items)Status / what we need from you
This is a draft for async review and discussion. Nothing runs until it lands and prerequisites are met:
--dart-define=FORCE_COMMAND_TYPE=localactivates the mocked JNAP layer the demo builds use. Please correct if wrong. This is the main thing blocking the mock tier.dashboard_home_test,local_network_settings_test).lab-lan; the device workflow stays dormant until it's registered and theROUTER_ADMIN_PASSWORDsecret is set. Expected when Deven is back in the lab.tierfield in test_meta (proposal, follow-up PR):"tier": "mock" | "device"so selection is data-driven instead of the hardcoded matrix.integration_test/only exists onmain/1.x. Worth porting the mock tier tousp?Context: QA validated the enforcement side of this suite live on the lab M62DU-EU (FW 1.0.12) this week; a Playwright browser-layer smoke suite (login + CPE-SEC-15 sweep) also went green against the same unit and will follow once the runner exists.
🤖 Generated with Claude Code