Post-mortem: CI outage across private repos, 2026-10-19 #40

Impact: from about 01:18Z to 03:10Z on 2026-10-19, CI stopped for several private repos and one job waited 40+ minutes. Merges were held for the whole window, and no bad code shipped.

Timeline (UTC)

• 01:18 Jobs on ubuntu-latest began failing in about 2 s with "recent account payments have failed": the GitHub-hosted minutes for private repos were used up.
• 02:16 A repo moved to [self-hosted, railway-private-pg], but its jobs stayed queued, because RUNNER_SCOPES is a per-repo allowlist and the repo was not on it. The scope was added at about 02:30.
• 02:26 A job in another repo waited 22+ minutes. Its run was pending, not queued, and QueuedJobs listed only queued and in_progress runs.
• 02:40–02:55 Jobs on the pool hit two further problems:
  • vitest workers ran under bun because the image has no Node, and crashed. The fix is actions/setup-node in that workflow.
  • shivammathur/setup-php needs sudo, which the runner user lacks, so it hung. Tracked in shivammathur/setup-php hangs on the runner image #38 and docs: explain why setup-php hangs on the pool and add a PHP derived image #39.
• 03:09 fix(ghapp): serve queued jobs of pending runs #37 (serve the queued jobs of pending runs) was deployed to both controllers, and the stuck job was picked up straight away.

Causes: three separate gaps that surfaced one at a time.

1. Private repos were left on GitHub-hosted runners.
2. The scope allowlist didn't cover them.
3. The controller ignored pending runs.

What stops a recurrence

• fix(ghapp): serve queued jobs of pending runs #37: the controller now reads pending runs.
• Every active private repo is now in scope or has a migration open.
• README: a table of which setup-* actions work on the pool (docs: explain why setup-php hangs on the pool and add a PHP derived image #39), with notes on the no-Node and no-sudo images.
• Deploying a controller is a manual step after merge: set the new image SHA and redeploy. Documented for the steward.
• The team now has an incident procedure (one owner and one issue per incident), so related symptoms get connected faster.