e2e: dash-08/dash-09 validity-window scenarios are flaky under runner load (EAI-7960 regression tests race real wall-clock)

[juhovainio]

Summary

`dash.feature`'s EAI-7960 scenarios assert against a real wall-clock validity window instead of a mocked/injected clock, and flake under GitHub-hosted-runner CPU contention when the shared `e2e` test binary runs enough scenarios.

• `@id:dash-gen-tps-held-after-scrape-failure` (dash-08 — "Gen throughput stays visible for the validity window after a scrape failure")
• `@id:dash-gen-tps-expiry-boundary` (dash-09 — "Gen throughput expires after the validity window following sustained failure")

Both assert runner.rs's gen_tps hold/expiry behavior against clamp(3 × instance_tick, 6s, 30s) — a real-time window — rather than a mocked clock.

Evidence (from PR #329)

The GitHub-hosted `E2E tests` job failed 3/3 reruns on PR #329's branch, each time on the same assertion:

```
Scenario: dash-08 - Gen throughput stays visible for the validity window after a scrape failure
✘ Then generation throughput remains visible within the validity window
Step panicked. Captured output: EAI-7960 REGRESSION: gen throughput ("tok/s") was cleared
immediately after the first failed scrape instead of being held for the validity window
(clamp(3 × instance_tick, 6 s, 30 s)).
```

One rerun also tripped the sibling `dash-09` expiry-boundary scenario on the identical kind of assertion.

Both scenarios pass 100% of the time locally, alone or together:

```
cargo xtask e2e -- -n 'dash-08|dash-09'

2 scenarios (2 passed), 18 steps (18 passed)

```

`main`'s `E2E tests` job has not failed on this recently (0 unexpected failures across its last several runs), but it currently runs 82 scenarios versus 87 on PR #329's branch — the extra scenario load from that PR's new hermetic `therock-next-*` scenarios plausibly tips this already-marginal real-time assertion into failing under runner contention, since all scenarios share one test binary/process.

Suggested fix

Make `dash-08`/`dash-09` deterministic by driving the validity window off a mocked/injected clock (matching the pattern other timing-sensitive dash scenarios in this suite already use), instead of racing real wall-clock time against CI runner scheduling jitter.

Impact

Intermittent, load-dependent CI failures on the `E2E tests` (GitHub-hosted) required check, unrelated to the actual change under review, that can block otherwise-ready PRs and require repeated reruns.

[rominf]

Fixed by #380 — adding a reproduction recipe here since the issue is the canonical place to look if this class recurs.

The contention source is the suite's own concurrency, not ambient CPU load. The no-GPU lane runs max_concurrent_scenarios(64) (tests/e2e-cucumber/tests/e2e.rs), so up to 64 scenarios — each with a PTY, a daemon and a mock server — run at once. The daemon's ticker is MissedTickBehavior::Skip, so under starvation it drops scrapes and the last successful observation ages past the validity window while the assertion waits.

That distinction matters for reproducing it:

# Reproduces: full suite, small core count, under load
taskset -c 0,1 cargo xtask e2e
# => 2 unexpected failures, dash-08 + dash-09, same message as CI

# Does NOT reproduce, at any CPU load:
cargo xtask e2e -- -n 'dash-08|dash-09'

I ran the narrow selection at load average 30 and pinned to two cores and it passed every time; only the full suite reproduces, because the pressure has to come from the other scenarios. That also explains the scenario-count correlation in the original report — 82 and 87 passed, 92 failed.

For the record, the pre-#380 code fails this way reliably (90 scenarios, 86 passed, 4 failed, 2 unexpected), which is what makes it a usable regression check rather than a coin flip.