From e9a7d68a5e36f0711ca58dfc081796fa10563fcf Mon Sep 17 00:00:00 2001 From: Lukas Wuttke Date: Fri, 11 Sep 2026 11:44:24 +0200 Subject: [PATCH 1/3] fix(installer-tests): bound apt's own sockets so a stalled mirror fails in seconds, not 12 minutes The `Prereqs` / `PATH persist` matrix jobs fail ~33 % of the time. The error they print blames "distro-mirror connectivity ... Re-run this job", and that advice does not work: two consecutive attempts on 2026-09-11 failed identically. WHAT ACTUALLY HAPPENS. `_pm_run` bounds each package-manager attempt from OUTSIDE with `timeout 60`. apt itself has no socket bound, so on a blackholed route -- packets dropped rather than refused -- it waits, emitting NOTHING: no `Err:`, no `W:`, no `E:`. All three attempts are 60 s of silence, apt learns nothing between them, and the arithmetic runs the job out: _bootstrap_host installs sudo and curl INDEPENDENTLY -> 2 x _pm_run(apt-get update) -> 2 x (3 attempts x 60 s + 15 s backoff) ~ 6.5 min -> the remaining work then hits the job's 12m bound -> exit 137 A refused connection errors instantly; only a silent stall produces that signature, and apt's default socket timeout outlasts the external kill every time. So the bound was in the wrong place, not missing. THE FIX. Give apt the bounds it can act on: -o Acquire::http::Timeout=10 -o Acquire::https::Timeout=10 -o Acquire::Retries=3 -o Acquire::ForceIPv4=true The first three are what tracebloc-engine's test workflow already applies to its own `apt-get update`, which does not exhibit this failure. A stall now costs 10 s and apt retries INSIDE one attempt instead of burning a whole _pm_run cycle; worst case the bootstrap fails honestly in well under a minute instead of consuming the job. ForceIPv4 is the mitigation for the LIKELY cause, and is labelled as such in the code: a container with no working IPv6 egress resolves an AAAA, connects and waits -- the exact silent-hang signature. A stalled run emits no apt output, so nothing in the logs proves it. It is here because it is cheap and harmless on an IPv4-only path, not because it was measured. apt-only, deliberately: dnf/yum/zypper/pacman take none of these flags and would fail on an unknown option -- turning a mirror stall on one distro into a hard argument error on four. Also corrected the error message. "Re-run this job" was wrong twice over: re-running inherits the same unbounded apt, and the wording framed our own missing flags as somebody else's outage. It now says re-running often does not help, and tells the reader that missing `Acquire::*` bounds on the attempts above IS the bug. Verified by running the real script in the real container the CI job uses: `docker run --rm -v "$PWD:/src:ro" -w /src ubuntu:24.04 bash scripts/tests/distro-prereqs.sh` -> exit 0, bootstrap completed, zero "stalled" warnings. Options confirmed accepted by a real apt (exit 0, not 100). shellcheck -S warning -x clean; manifest unchanged (this file is not on the bootstrap's fetch surface). Co-Authored-By: Claude Opus 5 --- scripts/tests/distro-prereqs.sh | 34 +++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/scripts/tests/distro-prereqs.sh b/scripts/tests/distro-prereqs.sh index 3ba94d75..577f57a6 100755 --- a/scripts/tests/distro-prereqs.sh +++ b/scripts/tests/distro-prereqs.sh @@ -55,11 +55,41 @@ _pm_run() { # bounded + retried package-manager invocation; $@ = the PM argv # `cancelled` (the failure class of #2859). [ "$i" -lt 3 ] && sleep $((i * 5)) done - echo "::error::package-manager step failed after 3 bounded attempts: $* — distro-mirror connectivity inside the CI container, not this PR. Re-run this job." >&2 + echo "::error::package-manager step failed after 3 bounded attempts: $* — the package manager could not reach its mirrors from inside the CI container. This is NOT this PR's diff. Re-running often does NOT help: two consecutive attempts failed identically on 2026-09-11. If apt, the attempts above should carry Acquire::* bounds; if they did not, that is the bug." >&2 return 1 } +# APT NEEDS ITS OWN SOCKET BOUND, not just the external one _pm_run applies. +# +# `timeout 60` around apt kills a stalled fetch, but apt never learns anything: it +# emits NO output, retries nothing, and the next attempt stalls identically. The +# observed failure is three attempts of pure silence -- no `Err:`, no `W:`, no +# `E:` -- then the job's 12m bound killing the container with exit 137. A refused +# connection errors instantly; only a BLACKHOLED route (packets dropped, not +# rejected) hangs like that, and apt's default socket timeout is long enough to +# outlast the external kill every time. +# +# So bound apt where apt can act on it. `Acquire::http::Timeout=10` turns a +# 60-second silent kill into a 10-second error apt can retry, and +# `Acquire::Retries=3` lets it ride out a transient stall inside ONE attempt +# instead of burning a whole _pm_run cycle. Same options tracebloc-engine's +# test workflow already uses on its own `apt-get update`, which does not exhibit +# this failure. +# +# ForceIPv4 is the MITIGATION FOR THE LIKELY CAUSE, not a proven one: a container +# with no working IPv6 egress resolves an AAAA, connects, and waits -- the exact +# silent-hang signature. There is no apt output to prove that from a stalled run, +# so this is here because it is cheap and cannot hurt an IPv4-only path, not +# because the logs named it. +# +# apt-only, deliberately: dnf/yum/zypper/pacman below take none of these flags +# and would fail on an unknown option, converting a mirror stall into a hard +# argument error on four distros to fix it on one. +_APT_BOUND='-o Acquire::http::Timeout=10 -o Acquire::https::Timeout=10 -o Acquire::Retries=3 -o Acquire::ForceIPv4=true' + _pm_install_one() { # install a single package with whatever PM exists - if command -v apt-get >/dev/null 2>&1; then _pm_run apt-get update -qq && _pm_run apt-get install -y -qq "$1" + if command -v apt-get >/dev/null 2>&1; then + # shellcheck disable=SC2086 # _APT_BOUND is a deliberate word-split argv + _pm_run apt-get update -qq $_APT_BOUND && _pm_run apt-get install -y -qq $_APT_BOUND "$1" elif command -v dnf >/dev/null 2>&1; then _pm_run dnf install -y -q "$1" elif command -v yum >/dev/null 2>&1; then _pm_run yum install -y -q "$1" elif command -v zypper >/dev/null 2>&1; then _pm_run zypper --non-interactive install "$1" From fc042658104591e5c66a39b4edcedc319973ed46 Mon Sep 17 00:00:00 2001 From: Lukas Wuttke Date: Fri, 11 Sep 2026 12:04:32 +0200 Subject: [PATCH 2/3] fix(installer-tests): one bounded package-manager runner, sourced by both harnesses (backend#2460 follow-up) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bugbot, and it is the right call: the previous commit fixed ONE of two copies. `scripts/tests/path-persist.sh` carried its own `_pm_run` plus the same bare `apt-get update -qq` and the same "Re-run this job" message. So `Prereqs` was fixed and `PATH persist` — two of the three jobs that were actually failing — was not. The two copies were identical in logic and had already drifted in prose; this change proved they drift in behaviour too, inside the very PR meant to stop the hang. So the runner moves to `scripts/tests/_pm.sh` and both harnesses source it. One copy cannot half-ship. WHAT IS SHARED: `_pm_run`, its error message, and `_APT_BOUND` — the parts that were byte-identical apart from a comment. WHAT IS NOT: each harness's own `_pm_install_one`. Those genuinely differ — path-persist supports apk, distro-prereqs does not — and folding them together would invent a package-manager matrix neither one tests. Sourcing is safe in this context and deliberately chosen over duplication: both harnesses run with the repo mounted at /src and `-w /src`, so `${BASH_SOURCE[0]%/*}/_pm.sh` is a local file read needing no network and no tools beyond the shell already running. Verified in BOTH container families the jobs use — ubuntu:24.04 and alpine:3.20 (path-persist's alpine leg installs bash first, then execs it) — `_pm_run` defined and `_APT_BOUND` set in each. Verified end to end: `docker run --rm -v "$PWD:/src:ro" -w /src ubuntu:24.04 bash scripts/tests/distro-prereqs.sh` -> exit 0, ZERO "stalled or failed" warnings. shellcheck -S warning -x clean on all three files; bash -n clean. Exactly one `_pm_run()` and one `_APT_BOUND=` in the tree, both in _pm.sh. Manifest unchanged — none of these are on the bootstrap's fetch surface. Co-Authored-By: Claude Opus 5 --- scripts/tests/_pm.sh | 65 +++++++++++++++++++++++++++++++++ scripts/tests/distro-prereqs.sh | 60 ++---------------------------- scripts/tests/path-persist.sh | 40 +++----------------- 3 files changed, 75 insertions(+), 90 deletions(-) create mode 100644 scripts/tests/_pm.sh diff --git a/scripts/tests/_pm.sh b/scripts/tests/_pm.sh new file mode 100644 index 00000000..fadf93c5 --- /dev/null +++ b/scripts/tests/_pm.sh @@ -0,0 +1,65 @@ +# shellcheck shell=bash +# ============================================================================= +# _pm.sh — the bounded package-manager runner, shared by the container harnesses +# ----------------------------------------------------------------------------- +# Sourced by scripts/tests/distro-prereqs.sh and scripts/tests/path-persist.sh. +# Both run INSIDE a plain distro container with the repo mounted at /src, so a +# sibling source is just a local file read — it needs no network and no tools +# beyond the shell that is already running. +# +# WHY THIS FILE EXISTS. The two harnesses each carried their own copy of +# `_pm_run`, identical in logic and already drifted in prose. When the apt +# socket bounds below were added to fix a 12-minute silent hang, they landed on +# ONE copy — so `Prereqs` was fixed and `PATH persist`, which fails the same +# way, was not. Bugbot caught that inside the very PR that introduced it +# (client#1051). One copy, sourced twice, is the only version of this that +# cannot half-ship. +# +# What is NOT here: each harness's own `_pm_install_one`. Those genuinely +# differ — path-persist supports apk, distro-prereqs does not — and folding +# them together would invent a package-manager matrix neither one tests. +# ============================================================================= + +# APT NEEDS ITS OWN SOCKET BOUND, not just the external one _pm_run applies. +# +# `timeout 60` around apt kills a stalled fetch, but apt never learns anything: +# it emits NO output, retries nothing, and the next attempt stalls identically. +# The observed failure is three attempts of pure silence — no `Err:`, no `W:`, +# no `E:` — then the job's outer bound killing the container with exit 137. A +# refused connection errors instantly; only a BLACKHOLED route (packets dropped, +# not rejected) hangs like that, and apt's default socket timeout outlasts the +# external kill every time. So the bound was in the wrong place, not missing. +# +# Timeout + Retries are what tracebloc-engine's test workflow already applies to +# its own `apt-get update`, which does not exhibit this failure. +# +# ForceIPv4 is the MITIGATION FOR THE LIKELY CAUSE, not a proven one: a container +# with no working IPv6 egress resolves an AAAA, connects, and waits — the exact +# silent-hang signature. A stalled run emits no apt output, so nothing in the +# logs proves it. It is here because it is cheap and cannot hurt an IPv4-only +# path, not because it was measured. +# +# apt-only, deliberately: dnf/yum/zypper/apk/pacman take none of these flags and +# would fail on an unknown option — turning a mirror stall on one distro into a +# hard argument error on five. +_APT_BOUND='-o Acquire::http::Timeout=10 -o Acquire::https::Timeout=10 -o Acquire::Retries=3 -o Acquire::ForceIPv4=true' + +# Bounded + retried package-manager invocation; "$@" = the PM argv. +# +# `command -v` (not has()) because this runs BEFORE common.sh is sourced; +# notices go to stderr. +_pm_run() { + local i + for i in 1 2 3; do + if command -v timeout >/dev/null 2>&1; then timeout "${TB_PM_TIMEOUT:-60}" "$@" && return 0 + elif command -v gtimeout >/dev/null 2>&1; then gtimeout "${TB_PM_TIMEOUT:-60}" "$@" && return 0 + else "$@" && return 0; fi + echo "::warning::package-manager step stalled or failed (attempt $i/3): $*" >&2 + # No backoff after the LAST attempt, so a dead mirror fails RED here well + # under the job's timeout-minutes rather than running the clock out into a + # silent `cancelled` (the failure class of backend#2859). + [ "$i" -lt 3 ] && sleep $((i * 5)) + done + echo "::error::package-manager step failed after 3 bounded attempts: $* — the package manager could not reach its mirrors from inside the CI container. This is NOT this PR's diff. Re-running often does NOT help: two consecutive attempts failed identically on 2026-09-11. If apt, the attempts above should carry Acquire::* bounds; if they did not, that is the bug." >&2 + return 1 +} diff --git a/scripts/tests/distro-prereqs.sh b/scripts/tests/distro-prereqs.sh index 577f57a6..52fc4e59 100755 --- a/scripts/tests/distro-prereqs.sh +++ b/scripts/tests/distro-prereqs.sh @@ -29,62 +29,10 @@ set -uo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" LIB="$HERE/../lib" - -# ── Make the container resemble a real host ────────────────────────────────── -# Anyone running the real installer reached it via `curl | bash`, so curl always -# exists and the box has sudo. Minimal base images ship neither — install them -# up front (we are root here) so the rest of the run mirrors a real machine. -# These bootstrap installs run in an EPHEMERAL CI container against distro mirrors -# that occasionally STALL rather than fail. An unbounded package-manager call then -# hangs until the job's `timeout-minutes`, which GitHub reports as `cancelled` — NOT -# a red check — so the failure is silent (this is the exact class that bit the -# sibling path-persist job's opensuse leg on 2026-08-31; backend#2859). Bound each -# attempt and retry: a transient stall recovers, a dead mirror fails FAST with an -# honest error. `command -v` (not has()) because this runs BEFORE common.sh is -# sourced; notices go to stderr. NB: this bounds the harness's own bootstrap only — -# the real installer functions invoked below use common.sh's bounded probes. -_pm_run() { # bounded + retried package-manager invocation; $@ = the PM argv - local i - for i in 1 2 3; do - if command -v timeout >/dev/null 2>&1; then timeout "${TB_PM_TIMEOUT:-60}" "$@" && return 0 - elif command -v gtimeout >/dev/null 2>&1; then gtimeout "${TB_PM_TIMEOUT:-60}" "$@" && return 0 - else "$@" && return 0; fi - echo "::warning::package-manager step stalled or failed (attempt $i/3): $*" >&2 - # No backoff after the LAST attempt, so a dead mirror fails RED here well under - # the job's timeout-minutes rather than running the clock out into a silent - # `cancelled` (the failure class of #2859). - [ "$i" -lt 3 ] && sleep $((i * 5)) - done - echo "::error::package-manager step failed after 3 bounded attempts: $* — the package manager could not reach its mirrors from inside the CI container. This is NOT this PR's diff. Re-running often does NOT help: two consecutive attempts failed identically on 2026-09-11. If apt, the attempts above should carry Acquire::* bounds; if they did not, that is the bug." >&2 - return 1 -} -# APT NEEDS ITS OWN SOCKET BOUND, not just the external one _pm_run applies. -# -# `timeout 60` around apt kills a stalled fetch, but apt never learns anything: it -# emits NO output, retries nothing, and the next attempt stalls identically. The -# observed failure is three attempts of pure silence -- no `Err:`, no `W:`, no -# `E:` -- then the job's 12m bound killing the container with exit 137. A refused -# connection errors instantly; only a BLACKHOLED route (packets dropped, not -# rejected) hangs like that, and apt's default socket timeout is long enough to -# outlast the external kill every time. -# -# So bound apt where apt can act on it. `Acquire::http::Timeout=10` turns a -# 60-second silent kill into a 10-second error apt can retry, and -# `Acquire::Retries=3` lets it ride out a transient stall inside ONE attempt -# instead of burning a whole _pm_run cycle. Same options tracebloc-engine's -# test workflow already uses on its own `apt-get update`, which does not exhibit -# this failure. -# -# ForceIPv4 is the MITIGATION FOR THE LIKELY CAUSE, not a proven one: a container -# with no working IPv6 egress resolves an AAAA, connects, and waits -- the exact -# silent-hang signature. There is no apt output to prove that from a stalled run, -# so this is here because it is cheap and cannot hurt an IPv4-only path, not -# because the logs named it. -# -# apt-only, deliberately: dnf/yum/zypper/pacman below take none of these flags -# and would fail on an unknown option, converting a mirror stall into a hard -# argument error on four distros to fix it on one. -_APT_BOUND='-o Acquire::http::Timeout=10 -o Acquire::https::Timeout=10 -o Acquire::Retries=3 -o Acquire::ForceIPv4=true' +# The bounded package-manager runner and apt's socket bounds live in ONE place; +# both container harnesses source it. See scripts/tests/_pm.sh for why. +# shellcheck source=scripts/tests/_pm.sh +. "${BASH_SOURCE[0]%/*}/_pm.sh" _pm_install_one() { # install a single package with whatever PM exists if command -v apt-get >/dev/null 2>&1; then diff --git a/scripts/tests/path-persist.sh b/scripts/tests/path-persist.sh index 7735a5a6..1f1f9c2f 100755 --- a/scripts/tests/path-persist.sh +++ b/scripts/tests/path-persist.sh @@ -78,42 +78,14 @@ umask 022 DEFAULT_CLI_REF="https://github.com/tracebloc/cli/releases/latest/download/install.sh" CLI_REF="${TRACEBLOC_CLI_REF:-$DEFAULT_CLI_REF}" CLI_VERSION="${TRACEBLOC_CLI_VERSION:-}" - -# ── Make the container resemble a real host ────────────────────────────────── -# A customer reached install.sh via `curl | sh`, so curl always exists. Minimal -# base images may ship neither curl nor a shell beyond /bin/sh — install what we -# need so the run mirrors a real machine. We are root in the container. -# Package installs run in an EPHEMERAL CI container against distro mirrors that -# occasionally STALL rather than fail. An unbounded `zypper`/`apt`/`dnf` then hangs -# until the job's `timeout-minutes`, and GitHub reports that as `cancelled` — NOT a -# red check — so a scheduled run can rot unnoticed for a week (backend#2859: -# opensuse/leap:15.6 hung the full 20 min on 2026-08-31 while every other leg, and -# this same script run by hand, went green in ~80s). Same infra-stall class the -# workflow already bounds one layer out for `docker pull` (#525/#592) — the -# container's own package installs were the layer left unbounded. Bound each attempt -# and retry: a transient stall recovers (→ green); a genuinely dead mirror fails -# FAST with an honest error (→ red), never a silent 20-minute cancel. `command -v` -# (not has()) so this is safe even before common.sh is sourced; notices go to stderr -# so a caller's `>/dev/null` can't swallow them. -_pm_run() { # bounded + retried package-manager invocation; $@ = the PM argv - local i - for i in 1 2 3; do - if command -v timeout >/dev/null 2>&1; then timeout "${TB_PM_TIMEOUT:-60}" "$@" && return 0 - elif command -v gtimeout >/dev/null 2>&1; then gtimeout "${TB_PM_TIMEOUT:-60}" "$@" && return 0 - else "$@" && return 0; fi - echo "::warning::package-manager step stalled or failed (attempt $i/3): $*" >&2 - # No backoff after the LAST attempt. Sized so the pathological all-installs- - # stall case (a few best-effort installs × 3 bounded attempts) stays well under - # the job's 20-min timeout-minutes, so a dead mirror fails RED here — it does - # not run the clock out into a silent `cancelled` (the very failure of #2859). - [ "$i" -lt 3 ] && sleep $((i * 5)) - done - echo "::error::package-manager step failed after 3 bounded attempts: $* — distro-mirror connectivity inside the CI container, not this PR. Re-run this job." >&2 - return 1 -} +# The bounded package-manager runner and apt's socket bounds live in ONE place; +# both container harnesses source it. See scripts/tests/_pm.sh for why. +# shellcheck source=scripts/tests/_pm.sh +. "${BASH_SOURCE[0]%/*}/_pm.sh" _pm_install() { # install one or more packages with whatever PM exists; best-effort - if command -v apt-get >/dev/null 2>&1; then _pm_run apt-get update -qq && _pm_run apt-get install -y -qq "$@" + # shellcheck disable=SC2086 # _APT_BOUND is a deliberate word-split argv + if command -v apt-get >/dev/null 2>&1; then _pm_run apt-get update -qq $_APT_BOUND && _pm_run apt-get install -y -qq $_APT_BOUND "$@" elif command -v dnf >/dev/null 2>&1; then _pm_run dnf install -y -q "$@" elif command -v yum >/dev/null 2>&1; then _pm_run yum install -y -q "$@" elif command -v zypper >/dev/null 2>&1; then _pm_run zypper --non-interactive --quiet install "$@" From c2e884d05328e0cf7c1e08cf97ddf78d75ba634b Mon Sep 17 00:00:00 2001 From: Lukas Wuttke Date: Fri, 11 Sep 2026 14:15:04 +0200 Subject: [PATCH 3/3] fix(installer-tests): drop ForceIPv4, and record why the runner's https fix cannot be copied here MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit tracebloc-engine#1029 landed the same class of fix on the RUNNER and measured the actual cause, which corrects two things here. WHAT IT MEASURED. The mirrorlist lists the archive over http first and https second; from 06:33Z on 2026-09-11 http stopped answering while https answered at once, so apt walked 52 index URLs at ~23 s each before falling back — 20 minutes per step. It fixed that by rewriting the mirrorlist to https. It also measured that Timeout/Retries ALONE still left a 29-minute stall, i.e. "bounded but not to anything useful". 1. ForceIPv4 is REMOVED. It was here on my hypothesis that a blackholed AAAA caused the stall. The measured cause is the SCHEME, not the address family. The flag was harmless, but its stated reason was wrong and a flag shipped on a contradicted hypothesis gets copied forward as fact. 2. The https rewrite is NOT adopted, and the comment says why with the measurement. These harnesses run inside a BARE container, and ubuntu:24.04 ships no ca-certificates, so rewriting its sources to https makes every index fetch fail verification: W: Failed to fetch https://…/InRelease Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. apt-get update -> exit 0 (warnings only — "succeeds" fetching nothing) apt-get install -> exit 100, package not installed An update that exits 0 having fetched nothing is WORSE than the hang: the failure moves to whatever needed the package. https needs ca-certificates, installing which needs apt, which is the circle. Measured in the image the job actually uses, not reasoned about. The bounds stay, and are adequate at THIS scale in a way they were not on the runner: #1029's 29 minutes was ~52 index URLs; a bare container lists four suites, so a total stall costs minutes and ends in an honest error rather than a 12-minute silent kill. Also kept from #1029: `Acquire::https::Timeout` alongside the http one, because `Acquire::http::Timeout` does not govern https connections. A LIMIT OF MY VERIFICATION, said plainly: the container run I used to check this (exit 0, zero stalls) was from a network where http://archive answers. It cannot reproduce the CI failure, so it proves the change is not broken — not that it cures the stall. The claim is bounded-and-honest failure, and #1029's numbers are the evidence for the bound's size. Co-Authored-By: Claude Opus 5 --- scripts/tests/_pm.sh | 45 +++++++++++++++++++++++++++++++++----------- 1 file changed, 34 insertions(+), 11 deletions(-) diff --git a/scripts/tests/_pm.sh b/scripts/tests/_pm.sh index fadf93c5..eea111dd 100644 --- a/scripts/tests/_pm.sh +++ b/scripts/tests/_pm.sh @@ -26,23 +26,46 @@ # it emits NO output, retries nothing, and the next attempt stalls identically. # The observed failure is three attempts of pure silence — no `Err:`, no `W:`, # no `E:` — then the job's outer bound killing the container with exit 137. A -# refused connection errors instantly; only a BLACKHOLED route (packets dropped, -# not rejected) hangs like that, and apt's default socket timeout outlasts the -# external kill every time. So the bound was in the wrong place, not missing. +# refused connection errors instantly; only a stalled one hangs like that, and +# apt's default socket timeout outlasts the external kill every time. So the +# bound was in the wrong place, not missing. # -# Timeout + Retries are what tracebloc-engine's test workflow already applies to -# its own `apt-get update`, which does not exhibit this failure. +# WHY NOT THE FIX tracebloc-engine#1029 USED. That change found the specific +# cause on the RUNNER — the mirrorlist lists the archive over http first and +# https second, http stopped answering on 2026-09-11, and apt walked 52 index +# URLs before falling back — and fixed it by rewriting the mirrorlist to https. +# THAT MUST NOT BE COPIED HERE. These harnesses run inside a BARE distro +# container, and ubuntu:24.04 ships no ca-certificates: rewriting its sources to +# https makes every index fetch fail certificate verification. Measured, in the +# image the job actually uses: # -# ForceIPv4 is the MITIGATION FOR THE LIKELY CAUSE, not a proven one: a container -# with no working IPv6 egress resolves an AAAA, connects, and waits — the exact -# silent-hang signature. A stalled run emits no apt output, so nothing in the -# logs proves it. It is here because it is cheap and cannot hurt an IPv4-only -# path, not because it was measured. +# W: Failed to fetch https://…/InRelease Certificate verification failed: +# The certificate is NOT trusted. The certificate issuer is unknown. +# apt-get update -> exit 0 (warnings only — it "succeeds" fetching nothing) +# apt-get install -> exit 100, the package is not installed +# +# An update that exits 0 having fetched nothing is worse than the hang, because +# the failure moves to whatever needed the package. So https is not available to +# us until something installs ca-certificates, which needs apt, which is the +# circle. The bounds below are what IS available. +# +# They are adequate here in a way they were not on the runner. #1029 measured +# Timeout/Retries alone still stalling 29 minutes, but that was ~52 index URLs; +# a bare container lists four suites, so a total stall costs minutes and ends in +# an honest error rather than a 12-minute silent kill. Bounded to something +# useful, at this scale. # # apt-only, deliberately: dnf/yum/zypper/apk/pacman take none of these flags and # would fail on an unknown option — turning a mirror stall on one distro into a # hard argument error on five. -_APT_BOUND='-o Acquire::http::Timeout=10 -o Acquire::https::Timeout=10 -o Acquire::Retries=3 -o Acquire::ForceIPv4=true' +# +# NOTE `Acquire::http::Timeout` does not govern https connections (#1029), hence +# both. No ForceIPv4: an earlier version of this carried it on the theory that a +# blackholed AAAA caused the stall. #1029 then measured the real cause on the +# runner to be the SCHEME, not the address family. The flag was harmless but its +# stated reason was wrong, and a flag shipped on a contradicted hypothesis is +# the kind of thing that gets copied forward as fact. +_APT_BOUND='-o Acquire::http::Timeout=10 -o Acquire::https::Timeout=10 -o Acquire::Retries=3' # Bounded + retried package-manager invocation; "$@" = the PM argv. #