Skip to content

docs: retire the routing and fallback contract for the fleet-first posture (Phase 9b) - #359

Merged
kyle-sexton merged 4 commits into
mainfrom
docs/ci-perf-phase-9b
Sep 8, 2026
Merged

docs: retire the routing and fallback contract for the fleet-first posture (Phase 9b)#359
kyle-sexton merged 4 commits into
mainfrom
docs/ci-perf-phase-9b

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

No related issue: melodic-software/github-iac#378 tracks the ci-perf program (Phase 9b). Draft until Phase 7 steps 5 and 7 land.

Summary

This repository's README documented a routing contract that no longer exists. Its "Routing and fallback contract" section described three selector policies (hosted-only, prefer-self-hosted, self-hosted-only), the liveness rule that chose between them, and a recovery procedure that flips the affected repository's effective policy to hosted-only through the audited github-iac routing-control workflow and then re-runs all jobs to force a fresh selector verdict.

ci-perf Phase 7 deleted every part of that. The select-runner reusable workflow is gone (ci-workflows#569, merged as 541ee4e90d12d77a90a3ddd72a3af9bc78634ea7, released as v0.23.0), its grammar is out of the runner-policy component (melodic-software/standards#556, merged as 771a796628f325c3c418c7b397d09fb7211e2972), and the routing-control workflow that wrote CI_RUNNER_POLICY is deleted (github-iac#453, merged as 4c5937e6b6067552d11b87abef04620191dda503).

Nothing about this controller changed. It still supplies runners inside its governed name and label namespaces, and a busy runner still queues rather than diverting work. What changed sits entirely upstream of it: an eligible private job names melodic-ubuntu-24.04-x64 as a literal in its own runs-on, and nothing decides placement at run time. The decision record is github-iac ADR 0014, which supersedes ADR 0004.

Fix

README.md:

  • Intro paragraph. "The centrally governed routing policy decides whether local capacity is preferred, required, or bypassed" and "serve the organization's governed self-hosted-only routing default" both describe a policy that is gone.
  • Runtime architecture mermaid. The Reusable workflow selector node with its "managed route" and "hosted route" edges becomes a workflow job naming one label or the other.
  • "Routing and fallback contract" becomes "Routing contract", rewritten. The three policies, the liveness rule, the fail-open/fail-closed selector behaviour and the two-minute ubuntu-slim selector control job are recorded as retired with their citing SHAs. Two consequences a reader actually needs are stated plainly: a rerun no longer recomputes anything about placement, so Re-run all jobs and a failed-job rerun are equivalent as far as routing is concerned; and there is no consumer-side recovery from an unavailable fleet at all, so affected jobs queue until a host returns. The unrelated stale-re-run hazard is pointed at github-iac POSTURE.md rather than restated.
  • Rollback order. Its first step was "set routing hosted-only". There is no routing variable, so the rollback now says what a fleet it cannot restore actually means: the affected jobs queue until it is back. It also states what the escape would really cost, because the escape is not one edit: widening the reason set is a governed standards change and each affected repository still needs its own per-job exceptions entry plus a runs-on change before anything routes differently. A reader planning a rollback window needs that stated, not silently dropped.

There is no consumer-side recovery from a fleet outage, and the doc now says so. An earlier draft offered a runner-policy.json exception as the substitute for the deleted hosted-only flip. That path does not exist: reason must be a member of the closed hostedExceptionReasons set (dependabot, docker-socket, job-container, privileged-control-plane, publication, service-container, windows), enforced at runner-policy.mjs:348, and none of them expresses "the fleet is down". Both the routing section and the rollback order now say the affected jobs queue until a host returns, and that widening that set is a governed standards change, which on its own routes nothing.

Two further precision points carried over from the sibling PRs' review, so this document does not repeat their errors:

  • CI_RUNNER_POLICY is described as present and unread, not as deleted. The organization variable is live at self-hosted-only; only its removal from the github-iac Pulumi program is decided, pending that repository's Phase 7 step 5 apply. Nothing reads it, because the selector that consumed it is deleted.
  • hosted-exception-required is named as a finding, not a config key. A reader following the other phrasing would write a config the schema rejects (the repository-policy schema is additionalProperties: false and permits no such top-level key). The real shape is an entry under exceptions, keyed <workflow path>#<jobId>, carrying a reason from the closed enum plus a justification.
  • The ADR 0014 reference names github-iac#466 rather than a main URL, which would 404 until that PR merges. "Supersedes ADR 0004" likewise becomes true only when #466 lands; both PRs are gated on the same Phase 7 steps.

docs/roadmap.md:

  • The isolated-Linux-VM backend section required a future backend to preserve "the audited hosted-only cutoff with a full workflow rerun that recomputes selector eligibility". There is no eligibility to recompute and no cutoff to preserve.
  • The cost-reporting section required separating "selector spend from workload spend". The selector's ubuntu-slim control job is retired, so there is no selector spend.

Verification

  • npx markdownlint-cli2 README.md docs/roadmap.md: 0 issues.
  • No em-dashes in added lines (git diff -U0 | grep "^+" | grep -c "—" prints 0).
  • Every SHA cited is a full 40 characters and was resolved against the local clones before being written.
  • In the two changed files, every surviving selector / CI_RUNNER_POLICY hit is deliberate retirement prose, not an instruction. This is scoped to the changed files on purpose: docs/queue-monitor.md lines 104 to 108 still instruct an operator to flip CI_RUNNER_POLICY to hosted-only and expect the selector to re-run, and .github/scripts/queue-monitor.cjs:23 ships the same text in incident output. Both are deleted by ci-runner#357 and are deliberately left to it rather than conflicted with here. Merge-order note: if this PR lands first, the README contradicts that shipped incident text until ci: retire the queued-job and actions-budget monitors (Phase 7 step 6) #357 follows.
  • Claims about what was deleted were checked against origin/main of the ci-workflows clone: select-runner.yml, selector-conformance.yml and queue-monitor-liveness.yml are absent from .github/workflows/.

Expected interaction with ci-runner#357. That draft also edits README.md, but only to remove two entries from the "Further documentation" list (lines 435 onward) as it deletes the two monitors. My edits are in the intro, the mermaid block, the routing section and the rollback paragraph, so the two diffs touch disjoint hunks and merge cleanly in either order. #357 is gated on a seven-day green pool-alert.yml window; this PR is gated on Phase 7 steps 5 and 7. Whichever lands second may need a trivial rebase.

Related

  • melodic-software/github-iac#378, the ci-perf program; github-iac#466, the Phase 9b PR carrying ADR 0014 and the completed POSTURE.md.
  • ci-workflows#569, merged as 541ee4e90d12d77a90a3ddd72a3af9bc78634ea7: deletes select-runner.yml.
  • feat(runner-policy)!: retire the selector grammar (Phase 7 step 2) standards#556, merged as 771a796628f325c3c418c7b397d09fb7211e2972: retires the selector grammar.
  • github-iac#453, merged as 4c5937e6b6067552d11b87abef04620191dda503: deletes the routing-control workflow this README's recovery procedure pointed at.
  • ci-runner#357, the queue-monitor and budget-monitor retirement, gated on the pool-alert window.

kyle-sexton and others added 3 commits September 8, 2026 00:58
…sture (Phase 9b)

The README's "Routing and fallback contract" section documented three selector
policies (hosted-only, prefer-self-hosted, self-hosted-only), a liveness rule,
and a recovery procedure that flips CI_RUNNER_POLICY and re-runs all jobs to
get a fresh selector verdict. None of that works: ci-perf Phase 7 deleted the
select-runner reusable workflow and standards#556 removed its grammar, so an
eligible private job now names melodic-ubuntu-24.04-x64 as a literal and
nothing decides placement at run time.

What the controller does is unchanged; what changed sits upstream of it. The
section is rewritten to say so, and the two consequences a reader needs are
stated plainly: a rerun no longer recomputes anything about placement, and
recovery from unavailable local capacity is a reviewed hosted-exception-required
key in the consumer repository rather than a variable flip.

The CI_RUNNER_POLICY organization variable is described as still present and
unread rather than as deleted. It is live at self-hosted-only; only its Pulumi
declaration is decided for deletion, pending the owner's Phase 7 step 5 apply.

Also corrected: the intro paragraph and the mermaid node, which both named the
selector; the rollback order, whose first step was setting routing to
hosted-only; and two lines in docs/roadmap.md that assumed a selector to
recompute eligibility and selector spend to separate from workload spend.

Refs: melodic-software/github-iac#378

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t does not exist

Two corrections from review, one of which was written but never committed onto
the pushed branch.

hosted-exception-required is a finding rule name, not a config key. The
repository-policy schema is additionalProperties false and permits no such
top-level key, so a reader following the old wording writes a config the
analyzer rejects. The real shape is an entry under exceptions, keyed
<workflow path>#<jobId>, with a reason and a justification.

More seriously, the rewritten recovery path did not exist. reason must be a
member of the closed hostedExceptionReasons set (dependabot, docker-socket,
job-container, privileged-control-plane, publication, service-container,
windows), enforced at runner-policy.mjs:348, and none of those expresses "the
fleet is down". A consumer cannot declare a fleet-outage exception at all, so
offering one as the rollback's substitute for the deleted hosted-only flip sent
an operator down a path that fails closed. Both the routing section and the
rollback order now say what is true: affected jobs queue until a host returns,
and widening that set is a governed standards change, not a consumer edit.

The ADR 0014 link is replaced by a citation of github-iac#466, which adds the
record; it is not on that repository's main yet, so the URL would have 404'd.

Refs: melodic-software/github-iac#378

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Re-verification found the previous fix introduced its own error.

The rollback said the only shortcut is a governed standards change "not an edit
in the affected repository". Both are required: runner-policy.mjs:3132-3147
fails an eligible private job unless it declares a per-job exceptions entry in
its OWN .github/runner-policy.json, and the job's runs-on must change too.
Widening the reason set routes nothing on its own. The sentence also
contradicted this README's own routing section five paragraphs earlier.

"The fleet's availability is now the whole of private CI's availability" was an
absolute the rest of the document avoids: jobs already holding a hosted
exception run on GitHub-hosted capacity and are unaffected by a fleet outage.

"Serve every eligible private job" left "eligible" undefined while the mermaid
block three lines below shows excepted jobs going hosted. It now says what it
means: every private job that names the fleet label.

Also cites github-iac OrgCiRouting.cs for the CI_RUNNER_POLICY claim, which was
true but uncheckable as written.

Refs: melodic-software/github-iac#378

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review September 8, 2026 08:52
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 8, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-08T08:55:59.974253Z 268972c Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 268972c99f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread README.md
…eleted routing procedure

Codex P1, and it is right. This branch's README says there is no routing cutoff,
while queued-job-monitor.yml stays scheduled every 15 minutes and
queue-monitor.cjs publishes incident instructions telling a responder to follow
the audited CI routing-control procedure, set CI_RUNNER_POLICY=hosted-only, and
re-run so the selector picks hosted capacity. All three of those are deleted.
Merging the doc alone would leave an alert that misdirects the person handling a
fleet outage, so the runbook is corrected in the same change rather than left to
the separate monitor retirement.

The recovery summary now states what is true: no rerun changes placement, queued
work waits for capacity to return, recovery is bringing a host back, and moving a
job to hosted capacity is not an incident-time action because it needs a per-job
exceptions entry plus a runs-on change and the reason set has no fleet-outage
member. docs/queue-monitor.md mirrors that text verbatim and is updated with it.

The test that pinned the old wording is rewritten to pin the new contract, plus
two guards that fail if the runbook ever again links a responder to the dead
procedure or tells one to set the deleted variable. 37 of 37 pass.

ci-runner#357 still deletes this monitor once its pool-alert window closes; this
change only makes the interim safe.

Refs: melodic-software/github-iac#378

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton merged commit 3948a53 into main Sep 8, 2026
26 checks passed
@kyle-sexton
kyle-sexton deleted the docs/ci-perf-phase-9b branch September 8, 2026 09:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant