ci: retire the queued-job and actions-budget monitors (Phase 7 step 6) - #357
Draft
kyle-sexton wants to merge 2 commits into
Draft
ci: retire the queued-job and actions-budget monitors (Phase 7 step 6)#357kyle-sexton wants to merge 2 commits into
kyle-sexton wants to merge 2 commits into
Conversation
Deletes queued-job-monitor.yml and actions-budget-monitor.yml along with their scripts (queue-monitor.cjs, queue-monitor.test.cjs, budget-monitor.cjs, budget-monitor.test.cjs) and their documentation pages (docs/queue-monitor.md, docs/actions-budget-monitor.md). Both workflows are superseded by pool-alert.yml in github-iac, per the ci-perf Phase 7 routing retirement plan (melodic-software/github-iac#378, sub-issue #425), step 6. incident-issue.cjs is also deleted. A grep sweep of the tree after the other deletions showed its only requirers were queue-monitor.cjs, queue-monitor.test.cjs and budget-monitor.cjs, all removed in this change, so no surviving workflow or script depends on it. README.md's two links to the deleted docs are removed, and docs/releases.md's description of the release-tag verification job is updated to say "workflow-script tests" instead of naming the now-deleted queue-monitor tests specifically. release/dependencies.json's actions/create-github-app-token entry is removed because it was minted only by queued-job-monitor.yml to read the observer App installation, and Test-ReleasePins.ps1 fails closed on any manifest entry no workflow uses. docs/topics/repo-simplify-sweep/RUN-STATE.md and docs/releases.md's historical note about the v0.1.20 tag reservation still name these files; both are historical records of past runs and are left untouched. No ci-workflows pin changed. ci-runner stays pinned at v0.22.2 (5776760254f8b63cba44e896f51604cb755350d9) by decision; this change touches none of its governed references. This PR opens as a draft. Its merge gates on pool-alert.yml showing seven green scheduled runs over seven consecutive days (currently 4 of 7). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Companion to 83d76f6, staged separately because git rm only staged the deletions themselves. - README.md: remove the two "Further documentation" links that pointed at the deleted docs/queue-monitor.md and docs/actions-budget-monitor.md. - docs/releases.md: the release-tag verification job description named "queue-monitor tests" specifically; renamed to "workflow-script tests" since queue-monitor.test.cjs no longer exists. - release/dependencies.json: removed the actions/create-github-app-token entry. Test-ReleasePins.ps1 fails closed on any githubActions manifest entry not used by a workflow, and queued-job-monitor.yml was the only caller of that action. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
kyle-sexton
added a commit
that referenced
this pull request
Sep 8, 2026
…sture (Phase 9b) (#359) No related issue: melodic-software/github-iac#378 tracks the ci-perf program (Phase 9b). Draft until Phase 7 steps 5 and 7 land. ## Summary This repository's README documented a routing contract that no longer exists. Its "Routing and fallback contract" section described three selector policies (`hosted-only`, `prefer-self-hosted`, `self-hosted-only`), the liveness rule that chose between them, and a recovery procedure that flips the affected repository's effective policy to `hosted-only` through the audited `github-iac` routing-control workflow and then re-runs all jobs to force a fresh selector verdict. ci-perf Phase 7 deleted every part of that. The `select-runner` reusable workflow is gone (ci-workflows#569, merged as `541ee4e90d12d77a90a3ddd72a3af9bc78634ea7`, released as v0.23.0), its grammar is out of the `runner-policy` component (melodic-software/standards#556, merged as `771a796628f325c3c418c7b397d09fb7211e2972`), and the routing-control workflow that wrote `CI_RUNNER_POLICY` is deleted (github-iac#453, merged as `4c5937e6b6067552d11b87abef04620191dda503`). **Nothing about this controller changed.** It still supplies runners inside its governed name and label namespaces, and a busy runner still queues rather than diverting work. What changed sits entirely upstream of it: an eligible private job names `melodic-ubuntu-24.04-x64` as a literal in its own `runs-on`, and nothing decides placement at run time. The decision record is github-iac ADR 0014, which supersedes ADR 0004. ## Fix `README.md`: - **Intro paragraph.** "The centrally governed routing policy decides whether local capacity is preferred, required, or bypassed" and "serve the organization's governed `self-hosted-only` routing default" both describe a policy that is gone. - **Runtime architecture mermaid.** The `Reusable workflow selector` node with its "managed route" and "hosted route" edges becomes a workflow job naming one label or the other. - **"Routing and fallback contract" becomes "Routing contract"**, rewritten. The three policies, the liveness rule, the fail-open/fail-closed selector behaviour and the two-minute `ubuntu-slim` selector control job are recorded as retired with their citing SHAs. Two consequences a reader actually needs are stated plainly: a rerun no longer recomputes anything about placement, so **Re-run all jobs** and a failed-job rerun are equivalent as far as routing is concerned; and there is no consumer-side recovery from an unavailable fleet at all, so affected jobs queue until a host returns. The unrelated stale-re-run hazard is pointed at github-iac `POSTURE.md` rather than restated. - **Rollback order.** Its first step was "set routing `hosted-only`". There is no routing variable, so the rollback now says what a fleet it cannot restore actually means: the affected jobs queue until it is back. It also states what the escape would really cost, because the escape is not one edit: widening the reason set is a governed standards change **and** each affected repository still needs its own per-job `exceptions` entry plus a `runs-on` change before anything routes differently. A reader planning a rollback window needs that stated, not silently dropped. **There is no consumer-side recovery from a fleet outage, and the doc now says so.** An earlier draft offered a `runner-policy.json` exception as the substitute for the deleted `hosted-only` flip. That path does not exist: `reason` must be a member of the closed `hostedExceptionReasons` set (`dependabot`, `docker-socket`, `job-container`, `privileged-control-plane`, `publication`, `service-container`, `windows`), enforced at `runner-policy.mjs:348`, and none of them expresses "the fleet is down". Both the routing section and the rollback order now say the affected jobs queue until a host returns, and that widening that set is a governed standards change, which on its own routes nothing. Two further precision points carried over from the sibling PRs' review, so this document does not repeat their errors: - **`CI_RUNNER_POLICY` is described as present and unread, not as deleted.** The organization variable is live at `self-hosted-only`; only its removal from the github-iac Pulumi program is decided, pending that repository's Phase 7 step 5 apply. Nothing reads it, because the selector that consumed it is deleted. - **`hosted-exception-required` is named as a finding, not a config key.** A reader following the other phrasing would write a config the schema rejects (the repository-policy schema is `additionalProperties: false` and permits no such top-level key). The real shape is an entry under `exceptions`, keyed `<workflow path>#<jobId>`, carrying a `reason` from the closed enum plus a `justification`. - **The ADR 0014 reference names github-iac#466 rather than a `main` URL**, which would 404 until that PR merges. "Supersedes ADR 0004" likewise becomes true only when #466 lands; both PRs are gated on the same Phase 7 steps. `docs/roadmap.md`: - The isolated-Linux-VM backend section required a future backend to preserve "the audited hosted-only cutoff with a full workflow rerun that recomputes selector eligibility". There is no eligibility to recompute and no cutoff to preserve. - The cost-reporting section required separating "selector spend from workload spend". The selector's `ubuntu-slim` control job is retired, so there is no selector spend. ## Verification - `npx markdownlint-cli2 README.md docs/roadmap.md`: 0 issues. - No em-dashes in added lines (`git diff -U0 | grep "^+" | grep -c "—"` prints `0`). - Every SHA cited is a full 40 characters and was resolved against the local clones before being written. - In the two **changed** files, every surviving `selector` / `CI_RUNNER_POLICY` hit is deliberate retirement prose, not an instruction. This is scoped to the changed files on purpose: `docs/queue-monitor.md` lines 104 to 108 still instruct an operator to flip `CI_RUNNER_POLICY` to `hosted-only` and expect the selector to re-run, and `.github/scripts/queue-monitor.cjs:23` ships the same text in incident output. Both are deleted by ci-runner#357 and are deliberately left to it rather than conflicted with here. **Merge-order note:** if this PR lands first, the README contradicts that shipped incident text until #357 follows. - Claims about what was deleted were checked against `origin/main` of the ci-workflows clone: `select-runner.yml`, `selector-conformance.yml` and `queue-monitor-liveness.yml` are absent from `.github/workflows/`. **Expected interaction with ci-runner#357.** That draft also edits `README.md`, but only to remove two entries from the "Further documentation" list (lines 435 onward) as it deletes the two monitors. My edits are in the intro, the mermaid block, the routing section and the rollback paragraph, so the two diffs touch disjoint hunks and merge cleanly in either order. #357 is gated on a seven-day green `pool-alert.yml` window; this PR is gated on Phase 7 steps 5 and 7. Whichever lands second may need a trivial rebase. ## Related - melodic-software/github-iac#378, the ci-perf program; github-iac#466, the Phase 9b PR carrying ADR 0014 and the completed `POSTURE.md`. - ci-workflows#569, merged as `541ee4e90d12d77a90a3ddd72a3af9bc78634ea7`: deletes `select-runner.yml`. - melodic-software/standards#556, merged as `771a796628f325c3c418c7b397d09fb7211e2972`: retires the selector grammar. - github-iac#453, merged as `4c5937e6b6067552d11b87abef04620191dda503`: deletes the routing-control workflow this README's recovery procedure pointed at. - ci-runner#357, the queue-monitor and budget-monitor retirement, gated on the pool-alert window. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No related issue: melodic-software/github-iac#378 tracks the ci-perf program (Phase 7 step 6, sub-issue melodic-software/github-iac#425). Draft until pool-alert.yml shows seven green scheduled runs over seven days.
Summary
Retires ci-runner's two selector-era monitors,
queued-job-monitor.ymlandactions-budget-monitor.yml, and everything that existed only to serve them.Both are superseded by
pool-alert.ymlinmelodic-software/github-iac, perthe ci-perf Phase 7 routing retirement plan, step 6.
Fix
Deleted:
.github/workflows/queued-job-monitor.yml.github/workflows/actions-budget-monitor.yml.github/scripts/queue-monitor.cjsand.github/scripts/queue-monitor.test.cjs.github/scripts/budget-monitor.cjsand.github/scripts/budget-monitor.test.cjs.github/scripts/incident-issue.cjsdocs/queue-monitor.mddocs/actions-budget-monitor.md(entirely about the deletedactions-budget-monitor.yml; not on the plan's literal deletion list, butwholly orphaned by this change, so it is deleted alongside its sibling
rather than left describing a workflow that no longer exists)
incident-issue.cjsdecision: the plan flags it UNVERIFIED whether it has aconsumer other than
queued-job-monitor.yml. After staging the other eightdeletions,
git grep -n incident-issueon the tree showed its only requirerswere
queue-monitor.cjs,queue-monitor.test.cjs, andbudget-monitor.cjs,all removed in this change. No surviving workflow or script requires it, so
it is deleted with the rest.
Fixed references:
README.md: removed the two "Further documentation" links to the deleteddocs.
docs/releases.md: the release-tag verification job description named"queue-monitor tests" specifically; reworded to "workflow-script tests"
since
queue-monitor.test.cjsno longer exists. Left untouched: thehistorical note that tag
v0.1.20is reserved by a past zizmor failure onactions-budget-monitor.yml, a record of history rather than a livereference.
release/dependencies.json: removed theactions/create-github-app-tokenentry.
queued-job-monitor.ymlwas its only caller (minting a short-livedinstallation token to read the observer App), and
scripts/Test-ReleasePins.ps1fails closed on anygithubActionsmanifestentry that no workflow uses. It does not list
queue-monitor.cjsorbudget-monitor.cjsby name, so no other edit was needed there..github/scripts/workflow-pin-metadata.test.cjs: no edit needed. Itdiscovers ci-workflows callers by scanning
.github/workflows/*.ymldynamically rather than enumerating files by name, and neither deleted
workflow carried a
uses: melodic-software/ci-workflows/...reference, soits governed-reference inventory is unaffected by construction.
Left untouched and reported rather than edited (historical records, not live
references):
docs/topics/repo-simplify-sweep/RUN-STATE.mdstill namesqueue-monitor.cjs,queue-monitor.test.cjs,budget-monitor.cjs,budget-monitor.test.cjs, andincident-issue.cjsas part of a pasttidying-sweep run record, and
docs/worker-image.mdnamesqueue-monitor-liveness.yml, an unrelated ci-workflows reusable workflow ina different repository that this change does not touch.
No ci-workflows pin moved. ci-runner stays pinned at v0.22.2
(
5776760254f8b63cba44e896f51604cb755350d9) throughout;git diff origin/maincarries no
ci-workflows/.../@<sha>line change.Verification
node --test .github/scripts/*.test.cjs: 21 pass, 0 fail.pwsh -File scripts/Test-ReleasePins.ps1: "Release and workflow pins areinternally consistent." (exit 0).
actionlint -shellcheck= -pyflakes=: no findings (exit 0).the planned files deleted plus the two additions above and the
incident-issue.cjsgrep proof; noci-workflowsSHA changed anywhere;the tests and release-pins script both exit 0;
pool-alert.yml(whichlives in
github-iac, not this repository) is untouched; and the grepsweep is clean outside historical documents.
Merge gate
This PR stays draft until
pool-alert.ymlinmelodic-software/github-iacshows seven green scheduled runs over seven consecutive days. Reading taken
2026-09-07 with the plan's exact form:
Result:
{"conclusions":["success"],"days":4,"runs":4}. Four of sevenrequired consecutive green scheduled days, all green so far. Expected to
clear around 2026-09-10.
Related
melodic-software/github-iac#378 (ci-perf program), melodic-software/github-iac#425
(Phase 7 sub-issue),
docs/topics/ci-perf/waves/routing-retirement/PLAN.mdstep 6 in
melodic-software/github-iac.