installer-tests: the prereqs install step is unbounded, so a mirror stall reads as a 20-minute cancel

[LukasWodka]

Summary

A distro-mirror stall in the Prereqs — <distro> matrix job of .github/workflows/installer-tests.yaml runs until the job's timeout-minutes: 20 fires and then reads as cancelled, with no log line saying why. It happened twice on 2026-09-04 on two package managers. The step that runs the prereqs is:

- name: Install prerequisites in ${{ matrix.distro }}
  run: |
    docker run --rm -v "$PWD:/src:ro" -w /src "$DISTRO" \
      bash scripts/tests/distro-prereqs.sh

What IS already bounded, so this is not "add a timeout" as first written. scripts/tests/distro-prereqs.sh bounds its own bootstrap through _pm_run (three attempts of timeout ${TB_PM_TIMEOUT:-60} <pm> …, then a ::error:: naming mirror connectivity — added for exactly this class after backend#2859), and its header says the real installer functions it then invokes use common.sh's bounded probes. The sibling Pull <distro> (bounded, retried) step and Install bats (bounded, retried) bound their commands the same way. So somewhere in this leg a command reached the 20-minute cap past those bounds, and the ticket's job is to find which. Two candidates, neither verified yet because the cancelled job's step log is not retrievable after the cap fires:

1. _pm_run falls through to an unbounded call when the container has no timeout binary — measured false (see comment below): rockylinux:9 ships /usr/bin/timeout via coreutils-single, so the bounded helpers did bound what they wrapped. The fall-through itself remains a fail-open worth closing, but it is not this leg's mechanism.
2. A later installer function runs a package-manager or network command that is not routed through a bounded probe (docker-ce repo setup, conntrack, helm download). If so, the fix is at that call site, not in the workflow.

Measured

leg	evidence
Prereqs — rockylinux:9 (run 33889848786, attempt 1, PR #983)	step "Install prerequisites in rockylinux:9" ran 15:40:32 → 16:00:41, the 20-minute cap to the second; the other 24 legs of the run were green. Conclusion cancelled. Nothing in the PR's diff reaches the prereqs path.
Installer tests / PATH persist (the leg behind #980)	apt-get update could not reach the Ubuntu mirror; gh run rerun --failed passed with no code change. #980 blamed a merge for it and had to be corrected.

Both cost 20 minutes of wall clock and a reviewer holding an approval on a red check that was never about the diff.

Proposed fix

1. First, make the cap legible: a timeout on the docker run itself, sized just under timeout-minutes (say 18 min), so the step FAILS with the script's own last stdout/stderr in the log instead of being cancelled by the job cap with nothing to read. That alone turns the next occurrence into evidence about which command stalled.
2. Close the fail-open regardless of which candidate measures true: _pm_run (distro-prereqs.sh) and _bounded (common.sh) both fall through to a bare "$@" when neither timeout nor gtimeout exists, so an absent guard reads as a satisfied one — a distro without coreutils timeout passes the "bounded call" check by never being bounded. The probe should refuse (or install coreutils first) when command -v timeout is empty, the same way the repo's other guards fail closed.
3. Then, if the evidence names it, route the unbounded installer call site through the bounded probe.

A mirror stall should end as a failed step with a reason in its log, never a cancelled job whose only evidence is that its start and end timestamps match timeout-minutes.

[LukasWodka]

Measured: candidate (1) is false. docker run --rm rockylinux:9 bash -c 'command -v timeout; rpm -q coreutils coreutils-single' →

/usr/bin/timeout
package coreutils is not installed
coreutils-single-8.32-34.el9.aarch64

The image ships timeout through coreutils-single (measured on the arm64 manifest; the package set is the same across the multi-arch image), so _pm_run and common.sh's _bounded did not fall through to an unbounded call on this leg. That leaves candidate (2): a command in the prereqs path that is not routed through a bounded probe at all — install_docker_engine's repo setup / curl | bash, install_kubectl/install_k3d/install_helm downloads, or a dnf call made directly rather than through _pm_run. The cancelled job's step log is not retrievable after the cap fires, so which one it was cannot be read off run 33889848786; the first fix step in the body (a timeout on the docker run itself, just under the job cap) is what makes the next occurrence name it.

The fail-open in _pm_run/_bounded (unbounded when timeout is absent) is still worth closing as a hardening — an image without coreutils would pass the bounded-call check by never being bounded — but it is not what happened here.

[LukasWodka]

Frequency, measured — and the bounding this ticket asked for is working

Adding the number this ticket does not have. No scope change: the diagnosis and the proposed fix below are unchanged, this is just how often it fires and where.

The bounded-failure half shipped, and it shows

A PATH persist — ubuntu:22.04 failure today reads:

##[warning] package-manager step stalled or failed (attempt 1/3): apt-get update -qq
##[warning] package-manager step stalled or failed (attempt 2/3): apt-get update -qq
##[warning] package-manager step stalled or failed (attempt 3/3): apt-get update -qq
##[error]   package-manager step failed after 3 bounded attempts: apt-get update -qq
            — distro-mirror connectivity inside the CI container, not this PR. Re-run this job.
##[error]   PATH-persist check in ubuntu:22.04 exceeded the 12m step bound (timeout exit 137)
            - the last lines above name the command that stalled

That is exactly what this ticket asked for: a named cause instead of a silent 20-minute cancelled. The _pm_run bounding plus the 12m step bound are doing their job. What is left is the flake itself.

How often, over the last 39 decided runs (2026-09-10 → 2026-09-11)

success	20
failure	10
cancelled (superseded)	9
failure share of decided (failure + success)	33 %

It is not attributable to any PR — it fails on the protected branches too

develop                            2 failures
main                               1
release-train/to-main              1
ci/2460-pin-staleness-guard        2
ci/mirror-publish-guard            2
feat/2461-control-plane-trim       1
fix/3556-fresh-install-digest-gate 1

Seven branches, three of them develop / main / a promotion. So "re-run, it is not your PR" is literally true — and that is the problem, because it is true a third of the time.

Why the number matters more than the individual failures

Installer tests is not a required check on develop (the required list is Unit tests, Lint, quality/{gitleaks,house-rules,shellcheck,action-pins}, version-bump-gate/version-check, Source-of-truth drift, chart content ⇒ Chart.yaml version bump, Helm unit tests). So it blocks nothing — which is precisely what makes a 33 % failure rate corrosive rather than merely annoying: a check that is wrong a third of the time and cannot block stops being read, including on the days it is right. That is the "never land a red gate / it trains people to skip the tier" failure mode, arrived at from the other direction.

Suggested next step, unchanged in spirit

Take the mirror off the critical path rather than retrying harder — cache or pre-bake the distro images so apt-get update is not what stands between a PR and a verdict. Retrying more, or auto-re-running on the now machine-classifiable error, hides the flake at the cost of making the signal meaningless.

No action requested on this comment; recording the measurement so the next person to hit it does not have to re-derive it. Third time it has been diagnosed from scratch.