test(e2e): stop dash-08/dash-09 racing the validity window

[rominf]

Problem

E2E tests is a required check and has been failing on dash-08 / dash-09 since #329 landed. The timeline points at scenario count rather than at any one change:

• feat(therock): support ROCm 10 "next" install layout, additive (EAI-8431) #329 merged at 2026-09-11T13:28Z.
• Every E2E tests run that completed before that (85–88 scenarios) passed.
• Every run after it (92 scenarios) has failed on these two, and only these two.
• Reruns do not help — feat(therock): support ROCm 10 "next" install layout, additive (EAI-8431) #329 itself failed them 3/3.

The production code is fine. These scenarios assert correct behaviour and then lose a race against it.

Root cause

The held-value window is clamp(3 × instance_tick, 6 s, 30 s), which at the default 2 s cadence clamps to the 6 s floor. Expiry is measured from the last successful observation (GenerationObservationTracker::snapshot).

The scenario flips the mock to failure, polls until one 503 has been served, then reads the screen. By then the last good observation is already up to one cadence old, so only ~4 s of the window is left.

The contention that eats those 4 s is not ambient CPU load — it is the suite itself. The no-GPU lane runs max_concurrent_scenarios(64) (tests/e2e-cucumber/tests/e2e.rs), so up to 64 scenarios, each with a PTY, a daemon and a mock server, run at once on a small runner. The daemon's ticker is MissedTickBehavior::Skip, so under starvation it drops scrapes and the last good observation ages out. More scenarios in the binary means more pressure, which is why the failure appeared exactly when the count grew.

Fix

Pin the scenario's scrape cadence to 10 s through dashboard.daemon.instance_tick_secs — the ordinary config field, written into the scenario's already-isolated config dir. That saturates the 30 s ceiling and leaves roughly 20 s of margin instead of 4 s. No production change and no test-only seam.

Two smaller changes:

• dash-09 no longer asserts the held boundary. It duplicated dash-08 verbatim and was the only reason dash-09 inherited the same race. The scenario still tells the whole story, because it asserts throughput is visible before the failure.
• Expiry polls instead of sleeping a fixed span. That direction cannot race — extra delay only makes the value more expired — so polling is both faster in the common case and tolerant of a slow runner.

Both scenarios' comments still claimed the fix was unimplemented and the assertions were expected to fail (runner.rs has held gen_trackers across failures since EAI-7960 landed). Corrected.

Verification

The failure reproduces locally by recreating the real condition — the full suite at concurrency 64, pinned to 2 cores under load. Two scenarios in isolation never reproduce it, however much CPU load is applied; the pressure has to come from the other scenarios.

run	result	unexpected failures
before	90 scenarios, 86 passed, 4 failed	2 — dash-08, dash-09, same message as CI
after	90 scenarios, 88 passed, 2 failed	0 (the 2 are pre-existing xfails)
after, repeated	90 scenarios, 88 passed, 2 failed	0

Also confirmed the config genuinely reaches the daemon rather than the test passing for the old reasons: dash-08 takes 30 s with it and 15 s without, which is the cadence change showing up in the scrape schedule.

cargo clippy --locked --workspace --all-targets -D warnings, cargo clippy --locked -p e2e-cucumber --test e2e -D warnings and cargo fmt --check are clean. rocm-dash-core's 39 injected-time observation tests are untouched — the boundary arithmetic was always covered deterministically there; what these scenarios add is the wiring (tracker → snapshot → rendered screen), which has no coverage at the runner level.

Tradeoffs

• Wall time. dash-08 grows to roughly 25 s and dash-09 to roughly 45 s. Accepted: it is mostly sleeping rather than CPU, against a permanently red required check.
• Still time-based. The margin is 5× larger, not infinite. True determinism would need a clock seam in the daemon (runner.rs calls Utc::now() directly), which is the test-backdoor pattern Remove the e2e-test-hooks feature #349 exists to delete.
• 30 s is the ceiling. MAX_VALIDITY caps what config can buy. If it ever flakes again, the fallback is to drop these two from E2E and add runner-level wiring coverage instead.

Closes #379

[rominf]

Superseded by #380, which landed the same fix about half an hour before I opened this — my mistake for not re-checking #379 immediately before opening rather than only at the start of the work.

#380's logical-clock seam is the stronger fix for the hold boundary: it cannot be crossed by host scheduling at all, whereas this PR only widened the margin from ~4 s to ~20 s through dashboard.daemon.instance_tick_secs and stayed time-based.

Nothing here is worth salvaging into a follow-up. The one durable artifact — how to actually reproduce the failure, which turns on the suite's own max_concurrent_scenarios(64) rather than ambient CPU load — is now recorded on #379. I've also noted on #349 that #380 adds a new e2e-test-hooks consumer, so deleting that feature now has to answer for dash-08/dash-09's determinism too.