Run the infrastructure bench suites weekly, and fix what they found

[danielrmerskine]

What

Of the 38 bash integration suites, CI executes exactly one. This adds Bench: Extended (bench-extended.yml), a weekly job on the self-hosted bench that runs five of the seven test/integration/infrastructure/ suites -- the box-level ones needing no instruments, so the bench stays dark for the whole run.

Saturday 14:17 UTC, clear of the nightly's worst case. Leaf workflow on the shared hardware-ci concurrency group. No pull_request trigger, ever: public repo, self-hosted runner. Failure updates the shared bench-alert issue; there is deliberately no recovery job, because closing that alert is the nightly's signal and a green weekly must never mask a nightly that is still failing.

The gate

Each suite declares a baseline -- how many of its checks are known to fail today -- and tools/bench_suite_gate.sh ratchets two-sided:

failures vs baseline	result
more	red: new breakage beyond the known gap
equal	green, with a warning naming the count
fewer	green, with a notice telling you to lower it

That is what lets this land green without softening a single assertion. Every non-zero baseline names the issue that owns it. A suite that produces no summary at all -- a crash, a step timeout -- is red regardless of its baseline; that case is not a pass.

Suite	Baseline	What it is
deployment.sh	0 / 44	clean
devenv.sh	0 / 13	clean
nets.sh	0 / 25	clean
box_config.sh	6 / 56	sysctl/cargo apply and clear, plus two others -- #275
generic.sh	5 / 95	hello under sustained repetition -- #273

Three of the five expect zero failures. The two that do not each name the issue that owns them, and the baseline comes down as those are fixed.

Reading the gate is the point: a continue-on-error step shows a green tick whether its suite passed, failed, or timed out. The step ticks cannot be trusted and the annotations can.

Two suites are deliberately not wired

boxes_config.sh -- its discovery section inventories every box reachable from the runner's network and prints the results. A network inventory does not belong in a CI log. It stays runnable by hand; wiring it needs a CI-safe output mode (counts, not names) first.

python.sh -- it does not test the CLI we ship. 53 of its checks call lager python -c "<code>" for inline execution and there is no -c option; the runner reports Error: No such option '-c' and the suite fails 43 of 84. Three more pass --image, also gone. It needs the rewrite devenv.sh already had, against the current interface, and that is its own reviewable change. Baselining 43 would record "calls an option that does not exist" as the expected state and leave the ratchet watching for a 44th.

What the runs found

Eight supervised bench rounds. Test defects, fixed here:

• lager boxes add now requires --user; 33 call sites predated that.
• deployment.sh read the removed lager duts registry in three places and called lager duts --help. Its set -e meant the first failure killed the run before its own summary printed, hiding every later result.
• devenv.sh tested the removed named-environment CLI; rewritten against the current per-project model. 13/13.
• generic.sh asserted that boxes add --ip rejects 999.999.999.999 and not.an.ip.address. It does not, by design -- --ip takes an IP or a DNS hostname, and both are valid hostname syntax. Those checks now cover what the validator genuinely refuses.
• Assertions pinned to strings the CLI no longer prints: nets --help, boxes edit, defaults add.
• deployment.sh inspected containers named controller and python. There is one container, named lager. Both checks failed on every box, reporting a stale expectation as a box fault.

Product behaviour, filed separately: lager hello failing under sustained repetition (#273) and box config sysctl and cargo entries not applying or clearing (#275). python.sh additionally surfaced --timeout never firing and a detached launch that does not return (#274).

The box lock

The CLI releases its auto-lock three ways -- try/finally, atexit, and a SIGTERM/SIGINT/SIGHUP handler -- but a step timeout ends in SIGKILL, which cannot be caught. The client-side release never runs, and the box-side TTL (1800s by default) becomes the only backstop. An expired lock is reaped automatically and needs no --force; an unexpired one blocks the next run outright.

That collides for back-to-back runs -- a re-run after a timeout, or dispatches during a debugging session, land well inside 30 minutes. It happened twice during this validation. It is not a threat to the schedule: the nightly and this weekly are hours apart, so any leaked lock has long expired by then.

This workflow force-releases before its connectivity check, matching the call update-regression.yml already makes for its own leak path. Forcing is safe here because the hardware-ci concurrency group serialises bench workflows against this box, so a lock still held when the job starts belongs to a run that is no longer running.

Also

static-checks now shellchecks and syntax-checks tools/ as well as test/. Those scripts are CI's own machinery -- one files public issues, one gates these suites -- and were never linted. They pass with no exclusions at all.

Verification

• Final pre-merge run green, with every baseline at its measured value.
• nets.sh and half of generic.sh's failures turned out to be caused by
  python.sh running ahead of them in the same job; with it unwired, nets.sh
  passes all 25 and generic.sh drops from 10 failures to 5. Failures that
  belong to no suite are exactly what a baseline must not record.
• shellcheck -S warning and bash -n clean across test/ and tools/.
• The temporary branch-push trigger and its matching notify-failure guard are removed in the same commit; the final on: block is schedule + workflow_dispatch only.