Flaky device test: DebuggingTest.ApplicationRunsWithoutDebugger ('Activity should have started' + flaky build failures, CoreCLR, ~12 hits/5 days)

[simonrozsival]

Summary

DebuggingTest.ApplicationRunsWithoutDebugger is a significant repeating flake on the public dotnet-android pipeline: 12 failed occurrences across 7 builds in 5 days (2026‑06‑25 → 06‑29), every one in an (Auto‑Retry) run (i.e. it failed in the original run and again on retry). It is not config‑specific and not machine‑specific.

Example (from the report that prompted this): build 1486352, ApplicationRunsWithoutDebugger(False,False,True,CoreCLR):

Activity should have started.
  Expected: True
  But was:  False
  at ...DebuggingTest.ApplicationRunsWithoutDebugger(...) DebuggingTest.cs:line 94

Occurrences (dnceng-public def 333, 2026‑06‑25 → 06‑29)

Build	PR	Date	Config (isRelease,extractNativeLibs,useEmbeddedDex,runtime)	Mode	Build result
1481768	#11702	06‑25	(False,False,False,CoreCLR)	activity-timeout	failed
1481768	#11702	06‑25	(False,True,True,CoreCLR)	activity-timeout	failed
1483246	#11759	06‑26	(False,False,False,CoreCLR)	activity-timeout	failed
1483246	#11759	06‑26	(True,True,False,CoreCLR)	activity-timeout	failed
1483642	#11762	06‑26	(False,False,False,CoreCLR)	activity-timeout	failed
1483642	#11762	06‑26	(True,True,False,CoreCLR)	activity-timeout	failed
1484806	#11780	06‑28	(False,True,False,CoreCLR)	build-failure	succeeded
1484820	#11781	06‑28	(True,True,True,CoreCLR)	build-failure	succeeded
1484820	#11781	06‑28	(False,False,True,CoreCLR)	build-failure	succeeded
1485678	#11772	06‑29	(True,False,False,CoreCLR)	activity-timeout	failed
1485683	#11617	06‑29	(True,True,False,CoreCLR)	build-failure	failed
1485683	#11617	06‑29	(False,False,False,CoreCLR)	build-failure	failed

(3 of the 7 builds still ended succeeded — the test recovered on a later retry slice — confirming flakiness rather than a hard regression.)

Patterns

• Two distinct failure modes (do not conflate):
  • A — activity start timeout (the reported symptom): Activity should have started. Expected: True But was: False at DebuggingTest.cs:94 — WaitForActivityToStart(..., 30) times out; the app built+installed but the activity didn't appear within 30 s.
  • B — flaky build failure: Xamarin.ProjectTools.FailedBuildException : Build failure: UnnamedProject.csproj at Builder.cs:417 — the MSBuild step itself failed before deploy/run. This mode warrants a separate look at the captured build.log.
• Runtime is not discriminating. Every hit is CoreCLR only because the test does Assert.Ignore for NativeAOT (known crash, see the TODO in ApplicationRunsWithoutDebugger). So CoreCLR is the only runnable parametrization, not a signal.
• Config is not discriminating. Failures span isRelease both true/false, extractNativeLibs both, useEmbeddedDex both. Most frequent are (False,False,False) ×4 and (True,True,False) ×3, but no single parametrization dominates.
• Machine is not discriminating. 14 occurrences spread over 10 distinct emulator machines (sjc20-*, sat12-*, iad01-*); the only repeats are two configs failing in the same run on the same machine. ⇒ systemic timing/environment flake, not a bad node.
• The test calls SwitchUser() (secondary-user path), so emulator user-switch + activity-launch timing is in play, but the symptom is a launch/build timeout, not XA0137.

Related lower-frequency flakes (same suite, same "didn't start/click in time" family)

These were called out alongside but currently appear once each in the window — tracked here for context, not yet independently significant:

• CheckXamarinFormsAppDeploysAndAButtonWorks(CoreCLR) — 1× (build 1481768): Button Should have been Clicked. Expected: True But was: False.
• AppWithStyleableUsageRuns(False,False,True,CoreCLR) — 1× (build 1485365): Activity should have started.

Suggested actions

1. Triage ApplicationRunsWithoutDebugger as the primary flake. Split the two modes:
   • Mode A: capture logcat.log + a screenshot on timeout and/or raise the 30 s WaitForActivityToStart budget; investigate emulator activity-start latency after SwitchUser().
   • Mode B: pull the referenced build.log to identify the flaky MSBuild failure — likely a separate root cause.
2. Keep the two sibling tests on watch; promote to their own issue if frequency rises.

Related

• Flaky test analysis: Xamarin.Android-PR pipeline, April 27 – May 11 2026 (follow-up to #11203) #11320 / Investigate ApplicationRunsWithDebuggerAndBreaks flakiness: Fast Deployment + guest user (XA0137) + test-code bugs (follow-up to #11320) #11322 — flaky ApplicationRunsWith**Debugger**AndBreaks analysis (different test, but same suite/infra)
• Flaky test analysis: Xamarin.Android-PR pipeline, April 10–24 2026 #11203 — earlier flaky-test analysis (closed)