docs: retire the routing and fallback contract for the fleet-first posture (Phase 9b) - #359
Merged
Conversation
…sture (Phase 9b) The README's "Routing and fallback contract" section documented three selector policies (hosted-only, prefer-self-hosted, self-hosted-only), a liveness rule, and a recovery procedure that flips CI_RUNNER_POLICY and re-runs all jobs to get a fresh selector verdict. None of that works: ci-perf Phase 7 deleted the select-runner reusable workflow and standards#556 removed its grammar, so an eligible private job now names melodic-ubuntu-24.04-x64 as a literal and nothing decides placement at run time. What the controller does is unchanged; what changed sits upstream of it. The section is rewritten to say so, and the two consequences a reader needs are stated plainly: a rerun no longer recomputes anything about placement, and recovery from unavailable local capacity is a reviewed hosted-exception-required key in the consumer repository rather than a variable flip. The CI_RUNNER_POLICY organization variable is described as still present and unread rather than as deleted. It is live at self-hosted-only; only its Pulumi declaration is decided for deletion, pending the owner's Phase 7 step 5 apply. Also corrected: the intro paragraph and the mermaid node, which both named the selector; the rollback order, whose first step was setting routing to hosted-only; and two lines in docs/roadmap.md that assumed a selector to recompute eligibility and selector spend to separate from workload spend. Refs: melodic-software/github-iac#378 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t does not exist Two corrections from review, one of which was written but never committed onto the pushed branch. hosted-exception-required is a finding rule name, not a config key. The repository-policy schema is additionalProperties false and permits no such top-level key, so a reader following the old wording writes a config the analyzer rejects. The real shape is an entry under exceptions, keyed <workflow path>#<jobId>, with a reason and a justification. More seriously, the rewritten recovery path did not exist. reason must be a member of the closed hostedExceptionReasons set (dependabot, docker-socket, job-container, privileged-control-plane, publication, service-container, windows), enforced at runner-policy.mjs:348, and none of those expresses "the fleet is down". A consumer cannot declare a fleet-outage exception at all, so offering one as the rollback's substitute for the deleted hosted-only flip sent an operator down a path that fails closed. Both the routing section and the rollback order now say what is true: affected jobs queue until a host returns, and widening that set is a governed standards change, not a consumer edit. The ADR 0014 link is replaced by a citation of github-iac#466, which adds the record; it is not on that repository's main yet, so the URL would have 404'd. Refs: melodic-software/github-iac#378 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Re-verification found the previous fix introduced its own error. The rollback said the only shortcut is a governed standards change "not an edit in the affected repository". Both are required: runner-policy.mjs:3132-3147 fails an eligible private job unless it declares a per-job exceptions entry in its OWN .github/runner-policy.json, and the job's runs-on must change too. Widening the reason set routes nothing on its own. The sentence also contradicted this README's own routing section five paragraphs earlier. "The fleet's availability is now the whole of private CI's availability" was an absolute the rest of the document avoids: jobs already holding a hosted exception run on GitHub-hosted capacity and are unaffected by a fleet outage. "Serve every eligible private job" left "eligible" undefined while the mermaid block three lines below shows excepted jobs going hosted. It now says what it means: every private job that names the fleet label. Also cites github-iac OrgCiRouting.cs for the CI_RUNNER_POLICY claim, which was true but uncheckable as written. Refs: melodic-software/github-iac#378 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
kyle-sexton
marked this pull request as ready for review
September 8, 2026 08:52
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 268972c99f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…eleted routing procedure Codex P1, and it is right. This branch's README says there is no routing cutoff, while queued-job-monitor.yml stays scheduled every 15 minutes and queue-monitor.cjs publishes incident instructions telling a responder to follow the audited CI routing-control procedure, set CI_RUNNER_POLICY=hosted-only, and re-run so the selector picks hosted capacity. All three of those are deleted. Merging the doc alone would leave an alert that misdirects the person handling a fleet outage, so the runbook is corrected in the same change rather than left to the separate monitor retirement. The recovery summary now states what is true: no rerun changes placement, queued work waits for capacity to return, recovery is bringing a host back, and moving a job to hosted capacity is not an incident-time action because it needs a per-job exceptions entry plus a runs-on change and the reason set has no fleet-outage member. docs/queue-monitor.md mirrors that text verbatim and is updated with it. The test that pinned the old wording is rewritten to pin the new contract, plus two guards that fail if the runbook ever again links a responder to the dead procedure or tells one to set the deleted variable. 37 of 37 pass. ci-runner#357 still deletes this monitor once its pool-alert window closes; this change only makes the interim safe. Refs: melodic-software/github-iac#378 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No related issue: melodic-software/github-iac#378 tracks the ci-perf program (Phase 9b). Draft until Phase 7 steps 5 and 7 land.
Summary
This repository's README documented a routing contract that no longer exists. Its "Routing and fallback contract" section described three selector policies (
hosted-only,prefer-self-hosted,self-hosted-only), the liveness rule that chose between them, and a recovery procedure that flips the affected repository's effective policy tohosted-onlythrough the auditedgithub-iacrouting-control workflow and then re-runs all jobs to force a fresh selector verdict.ci-perf Phase 7 deleted every part of that. The
select-runnerreusable workflow is gone (ci-workflows#569, merged as541ee4e90d12d77a90a3ddd72a3af9bc78634ea7, released as v0.23.0), its grammar is out of therunner-policycomponent (melodic-software/standards#556, merged as771a796628f325c3c418c7b397d09fb7211e2972), and the routing-control workflow that wroteCI_RUNNER_POLICYis deleted (github-iac#453, merged as4c5937e6b6067552d11b87abef04620191dda503).Nothing about this controller changed. It still supplies runners inside its governed name and label namespaces, and a busy runner still queues rather than diverting work. What changed sits entirely upstream of it: an eligible private job names
melodic-ubuntu-24.04-x64as a literal in its ownruns-on, and nothing decides placement at run time. The decision record is github-iac ADR 0014, which supersedes ADR 0004.Fix
README.md:self-hosted-onlyrouting default" both describe a policy that is gone.Reusable workflow selectornode with its "managed route" and "hosted route" edges becomes a workflow job naming one label or the other.ubuntu-slimselector control job are recorded as retired with their citing SHAs. Two consequences a reader actually needs are stated plainly: a rerun no longer recomputes anything about placement, so Re-run all jobs and a failed-job rerun are equivalent as far as routing is concerned; and there is no consumer-side recovery from an unavailable fleet at all, so affected jobs queue until a host returns. The unrelated stale-re-run hazard is pointed at github-iacPOSTURE.mdrather than restated.hosted-only". There is no routing variable, so the rollback now says what a fleet it cannot restore actually means: the affected jobs queue until it is back. It also states what the escape would really cost, because the escape is not one edit: widening the reason set is a governed standards change and each affected repository still needs its own per-jobexceptionsentry plus aruns-onchange before anything routes differently. A reader planning a rollback window needs that stated, not silently dropped.There is no consumer-side recovery from a fleet outage, and the doc now says so. An earlier draft offered a
runner-policy.jsonexception as the substitute for the deletedhosted-onlyflip. That path does not exist:reasonmust be a member of the closedhostedExceptionReasonsset (dependabot,docker-socket,job-container,privileged-control-plane,publication,service-container,windows), enforced atrunner-policy.mjs:348, and none of them expresses "the fleet is down". Both the routing section and the rollback order now say the affected jobs queue until a host returns, and that widening that set is a governed standards change, which on its own routes nothing.Two further precision points carried over from the sibling PRs' review, so this document does not repeat their errors:
CI_RUNNER_POLICYis described as present and unread, not as deleted. The organization variable is live atself-hosted-only; only its removal from the github-iac Pulumi program is decided, pending that repository's Phase 7 step 5 apply. Nothing reads it, because the selector that consumed it is deleted.hosted-exception-requiredis named as a finding, not a config key. A reader following the other phrasing would write a config the schema rejects (the repository-policy schema isadditionalProperties: falseand permits no such top-level key). The real shape is an entry underexceptions, keyed<workflow path>#<jobId>, carrying areasonfrom the closed enum plus ajustification.mainURL, which would 404 until that PR merges. "Supersedes ADR 0004" likewise becomes true only when #466 lands; both PRs are gated on the same Phase 7 steps.docs/roadmap.md:ubuntu-slimcontrol job is retired, so there is no selector spend.Verification
npx markdownlint-cli2 README.md docs/roadmap.md: 0 issues.git diff -U0 | grep "^+" | grep -c "—"prints0).selector/CI_RUNNER_POLICYhit is deliberate retirement prose, not an instruction. This is scoped to the changed files on purpose:docs/queue-monitor.mdlines 104 to 108 still instruct an operator to flipCI_RUNNER_POLICYtohosted-onlyand expect the selector to re-run, and.github/scripts/queue-monitor.cjs:23ships the same text in incident output. Both are deleted by ci-runner#357 and are deliberately left to it rather than conflicted with here. Merge-order note: if this PR lands first, the README contradicts that shipped incident text until ci: retire the queued-job and actions-budget monitors (Phase 7 step 6) #357 follows.origin/mainof the ci-workflows clone:select-runner.yml,selector-conformance.ymlandqueue-monitor-liveness.ymlare absent from.github/workflows/.Expected interaction with ci-runner#357. That draft also edits
README.md, but only to remove two entries from the "Further documentation" list (lines 435 onward) as it deletes the two monitors. My edits are in the intro, the mermaid block, the routing section and the rollback paragraph, so the two diffs touch disjoint hunks and merge cleanly in either order. #357 is gated on a seven-day greenpool-alert.ymlwindow; this PR is gated on Phase 7 steps 5 and 7. Whichever lands second may need a trivial rebase.Related
POSTURE.md.541ee4e90d12d77a90a3ddd72a3af9bc78634ea7: deletesselect-runner.yml.771a796628f325c3c418c7b397d09fb7211e2972: retires the selector grammar.4c5937e6b6067552d11b87abef04620191dda503: deletes the routing-control workflow this README's recovery procedure pointed at.