From d8057a490ee6f4ba7a64c5e9863fe575da06ad40 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 27 Oct 2026 18:49:12 -0700 Subject: [PATCH 1/2] chore(quest): claim quest/m0/ci/select-audit.md Quest-Claim: ea6124f3a33b2530 From 3532ac3fee67db3172be7f4a823db6a3a3670218 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 27 Oct 2026 19:11:42 -0700 Subject: [PATCH 2/2] fix(ci): select a merge group's changes from main Replay every Nightly full-gate failure since the gate entered Nightly against the selected queue gate. The only missed edge: Check diffed each merge group from the entry ahead of it, so under HEADGREEN a passing head merged failed entries whose changes it never selected. Diff from origin/main instead. Co-Authored-By: Claude Opus 5.5 --- .github/ci-selection-audit.md | 97 +++++++++++++++++++++++++++++------ .github/workflows/check.yml | 11 ++-- quest/m0/ci/README.md | 1 - quest/m0/ci/select-audit.md | 29 ----------- sh/dispatch-query.test.py | 5 +- sh/gh/queue.test.ts | 4 ++ 6 files changed, 95 insertions(+), 52 deletions(-) delete mode 100644 quest/m0/ci/select-audit.md diff --git a/.github/ci-selection-audit.md b/.github/ci-selection-audit.md index 5613ab77ff6..226bb3af047 100644 --- a/.github/ci-selection-audit.md +++ b/.github/ci-selection-audit.md @@ -2,8 +2,9 @@ The shared-input rules rarely force every module. Keep the conservative Rust lockfile rule rather than add a second dependency resolver. Fix the missing -Cargo-configuration edge now. Replaying historical full-run failures for missed -edges remains open. +Cargo-configuration edge now. Replaying the full-gate failures found no missed +path edge; it found one missed queue edge, now fixed: a merge group diffed from +the entry ahead of it rather than from main. ## Observation window and method @@ -107,17 +108,83 @@ settle the rule decisions above without a new measurement window. ## Missed edges -Two recent full-run failures illustrate why replay needs to distinguish selector -bugs from other failures. The [October 25 full gate](https://github.com/moq-dev/moq/actions/runs/40192013303/job/121164806731) -failed because its shallow checkout had no `HEAD~1` for a tooling test. That is a -checkout-history problem, not a missed package edge. The same Nightly's -[feature lane](https://github.com/moq-dev/moq/actions/runs/40192013303/job/121164807008) -failed on no-default-feature rustdoc links in moq-net. The current selected -net-feature boundary already invokes that documentation build; workspace feature -unification is separately being addressed in -[moq-dev/moq#6830](https://github.com/moq-dev/moq/pull/6830) and the async-free +### Full-gate replay + +Nightly began running the queue's gate over every package (`just gate`) on +October 20, in [moq-dev/moq#6038](https://github.com/moq-dev/moq/pull/6038); before that no full run existed to +compare against. Every scheduled Nightly gate from October 20 through October 27 +is below. Other Nightly lanes, such as the feature matrix, run work the queue never +selects, so they cannot expose a selection miss. + +| Nightly | Gate | Cause | Selection miss | +| --- | --- | --- | --- | +| [October 20](https://github.com/moq-dev/moq/actions/runs/39421743227) | passed | | | +| [October 21](https://github.com/moq-dev/moq/actions/runs/39600361153/job/119223010596) | failed | Dangling quest link | Yes, the queue base below | +| [October 22](https://github.com/moq-dev/moq/actions/runs/39779729461/job/119827590096) | failed | Nine `@moq/watch` CMAF decoder tests | No | +| [October 23](https://github.com/moq-dev/moq/actions/runs/39951870847) | passed | | | +| [October 24](https://github.com/moq-dev/moq/actions/runs/40090959108/job/120863338990) | failed | Shallow checkout has no `HEAD~1` | No | +| [October 25](https://github.com/moq-dev/moq/actions/runs/40192013303/job/121164806731) | failed | Same | No | +| [October 26](https://github.com/moq-dev/moq/actions/runs/40332938121) | passed | | | +| [October 27](https://github.com/moq-dev/moq/actions/runs/40506938673/job/122198526371) | timed out | No output for its last 36 minutes | No | + +- October 21: [moq-dev/moq#5849](https://github.com/moq-dev/moq/pull/5849) deleted a quest that + [moq-dev/moq#6122](https://github.com/moq-dev/moq/pull/6122) had just linked. Its own queue entry + [failed on that link](https://github.com/moq-dev/moq/actions/runs/39572680575), yet it merged when the next entry, + [moq-dev/moq#6012](https://github.com/moq-dev/moq/pull/6012), [passed](https://github.com/moq-dev/moq/actions/runs/39578702167). That entry diffed from + the failed entry's head, touched no quest, and so never ran the quest check. +- October 22: the queue entry for [moq-dev/moq#6219](https://github.com/moq-dev/moq/pull/6219) + [ran these tests](https://github.com/moq-dev/moq/actions/runs/39747822278), 387 passing. At the Nightly head, the whole + `@moq/watch` suite passes locally while `src/video/decoder.test.ts` alone fails + the same nine tests: they depend on test-file order, which differs between + runners. That file passes alone on current main. +- October 24 and 25: the queue fetches full history; Nightly checks out shallowly. + [moq-dev/moq#6708](https://github.com/moq-dev/moq/pull/6708) made the test build its own history. +- October 27: outside the window. The job hit its 60-minute timeout 36 minutes + after its last output, a hang rather than a narrower selection. + +The feature lane's no-default-feature rustdoc failure in +[that October 25 Nightly](https://github.com/moq-dev/moq/actions/runs/40192013303/job/121164807008) is a workspace feature +unification problem, addressed in [moq-dev/moq#6830](https://github.com/moq-dev/moq/pull/6830) and the async-free contract in [moq-dev/moq#6851](https://github.com/moq-dev/moq/pull/6851). -These observations do not substitute for replaying all month's full-run failures -against the contemporaneous selected gate. That replay remains in -[its quest](/quest/m0/ci/select-audit.md). No claim of complete missed-edge -coverage is made here. + +### Queue entries merged without passing + +The October 21 miss is one instance of a wider hole. From October 20 23:58 to +October 22 23:33 UTC, the main ruleset's merge queue used the `HEADGREEN` +grouping strategy, which merges every entry under a passing head, while Check +diffed each entry from `merge_group.base_sha`, the entry ahead of it. A head +therefore vouched for entries whose changes it never selected. Thirty-seven +attempts landed on main without their head ever passing, all within +that interval and none under `ALLGREEN`: + +| Failure carried into main | Attempts | +| --- | ---: | +| Dangling quest links, four different deletions | 13 | +| `@moq/net` request and draining-session tests | 6 | +| Cancelled | 5 | +| JS doc comments missing on exported error members | 3 | +| `moq-cli` redundant-importer bufferbloat test | 3 | +| `moq-video` hev1 CMAF round trip | 2 | +| `moq-srt` late-payload host instant | 2 | +| `moq-net` test build, a call with the wrong argument count | 1 | +| JS type check, a removed `Manual.advanceAsync` | 1 | +| `moq-relay` GOAWAY deadline drain | 1 | + +The static failures (links, doc comments, type and compile errors) are +deterministic breakage that reached main. The test failures were not reproduced +here, so some may be flakes. Either way, a narrower selection is what let them +through: the passing head never ran them. + +Check now diffs a merge group from `origin/main`, the main commit its head is +built on, so a head's run selects every entry it can merge, whichever grouping +strategy the ruleset uses. `sh/gh/queue.test.ts` pins the workflow base, and +`sh/dispatch-query.test.py` covers an unmerged entry ahead. The change costs +nothing at today's settings: of 487 attempts from October 22 23:33 UTC to the +window end, with one entry built at a time, none started before the entry ahead +had merged. With five entries built at once, October 10 through 20, 806 of 1,047 +attempts started on an unmerged entry and would now also select its changes. + +Five pull requests merged outside the queue after October 10, four of them on +October 23 between 21:33 and 22:56 UTC while the ruleset had no merge queue rule. +They skipped Check entirely, which no selection rule can address, and none caused +a full-gate failure above. diff --git a/.github/workflows/check.yml b/.github/workflows/check.yml index 4ebaf334566..be43e0f9bf4 100644 --- a/.github/workflows/check.yml +++ b/.github/workflows/check.yml @@ -41,13 +41,18 @@ jobs: environment: ${{ (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/heads/gh-readonly-queue/main/')) && 'cache-write' || '' }} # Backstop a hang instead of holding the runner for the 6-hour default. timeout-minutes: 60 + env: + # Diff a merge group from main, not from the entry ahead of it: when the + # queue requires only a group's head to pass, that run vouches for every + # entry it merges. Building one entry at a time, the base is main anyway. + CI_BASE: ${{ github.base_ref == 'release' && '--all' || 'origin/main' }} steps: - name: Checkout uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - # Full history so `just ci` can diff against the PR or merge group base. + # Full history so `just ci` can diff against main. fetch-depth: 0 - uses: ./.github/actions/just @@ -57,8 +62,6 @@ jobs: - name: Select work id: scope run: just ci-select "$CI_BASE" >> "$GITHUB_OUTPUT" - env: - CI_BASE: ${{ github.base_ref == 'release' && '--all' || github.event.merge_group.base_sha }} - uses: ./.github/actions/nix if: steps.scope.outputs.needed == 'true' && runner.environment != 'self-hosted' @@ -86,8 +89,6 @@ jobs: export NEXTEST_TEST_THREADS=24 fi nix develop --command just ci "$CI_BASE" - env: - CI_BASE: ${{ github.base_ref == 'release' && '--all' || github.event.merge_group.base_sha }} # Cargo's per-crate timings for the Rust test builds, so a slow crate # shows without rebuilding. The job summary has the per-module times. diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 2c991353a34..8ecfe3664f7 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -132,7 +132,6 @@ The historical samples locate candidates; they are not controlled benchmarks. - [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - measure the hosted Nightly jobs' hits, restore time, and expiry once the bump lands - [Verify the Nix cache](/quest/m0/ci/nix-cache-verify.md) - confirm CI and releases use `nix.moq.dev` and retire Cachix - [Unify dependency features](/quest/m0/ci/hakari.md) - a cargo-hakari workspace-hack so every job compiles each dependency alike and mbx hits across jobs -- [Replay full-run failures against the selected gate](/quest/m0/ci/select-audit.md) - every dependency edge `just ci` misses is found and fixed with a test - [First shared FFI release](/quest/m0/ci/nightly-ffi-reuse.md) - condition: the backport merges and the next `moq-ffi-v*` tag runs `Release FFI`; verify the wrappers - [Fewer BTreeMap instantiations](/quest/m0/ci/btree-keys.md) - moq-net shares BTreeMap key and value types, kept only if its CPU time drops - [Halve the slowest tests' CPU](/quest/m0/ci/slow-tests.md) - the slowest 1% of tests take under half their 289 CPU-seconds, with coverage kept diff --git a/quest/m0/ci/select-audit.md b/quest/m0/ci/select-audit.md deleted file mode 100644 index fc4b365aac0..00000000000 --- a/quest/m0/ci/select-audit.md +++ /dev/null @@ -1,29 +0,0 @@ -# [S] Replay full-run failures against CI's selected gate - -## Goal - -Find the dependency edges `just ci`'s selection misses. Replay each change -whose full run (Nightly or `just gate`) failed where its selected queue run -passed, and fix every missed edge with a test in `sh/changed.test.py`, -`sh/dispatch-query.test.py`, `sh/rs/select.test.py`, or their neighbors. - -## Plan - -- [The CI selection report](/.github/ci-selection-audit.md) covers widening - frequency for September 26 through October 25 and its per-attempt evidence. - Extend it with the replay, rather than starting a second report. -- Replay against the selection rules each change's queue run used, read from - that head's `sh/dispatch.sh` and `sh/rs/select.sh`. -- Separate actual missed edges from failures a narrower selection cannot cause, - such as checkout-history problems and feature unification already owned by - [moq-dev/moq#6830](https://github.com/moq-dev/moq/pull/6830) and - [moq-dev/moq#6851](https://github.com/moq-dev/moq/pull/6851). -- Decided with the maintainer on 2026-10-26: the report's incidence counts and - archived elapsed durations stand in for runner CPU time, so no CPU - measurement window is needed. Cargo.lock stays a whole-workspace trigger; a - lockfile-diff selector would narrow at most 75 attempts a month. -- kixelated/rust2's - [MIR test selection](https://github.com/kixelated/rust2/blob/main/quest/m3/test-select.md) - measures a finer selector against this one. - -Public API: none. diff --git a/sh/dispatch-query.test.py b/sh/dispatch-query.test.py index 0dab2a9a3b6..01eb407089f 100644 --- a/sh/dispatch-query.test.py +++ b/sh/dispatch-query.test.py @@ -124,12 +124,13 @@ def test_working_tree_changes_and_deletions_count(self): def test_release_pr_base_and_merge_group_base(self): self.env["GITHUB_BASE_REF"] = "release" self.expect(True, base="", rust=False) + # The entry queued ahead changes Rust and has not merged; this one changes a quest. self.change("rs/moq-net/src/lib.rs") self.git("add", ".") self.git("commit", "-qm", "queued source") - group_base = self.git("rev-parse", "HEAD") self.change("quest/m1/example.md") - self.expect(True, base=group_base, rust=False) + # Check's merge group base: a passing head can merge the entry ahead, so it selects that too. + self.expect(True, base="origin/main", rust=True) self.expect(True, base="origin/release") def test_invalid_base_fails_instead_of_reporting_no_work(self): diff --git a/sh/gh/queue.test.ts b/sh/gh/queue.test.ts index d6aab23b059..bea261ca0e5 100644 --- a/sh/gh/queue.test.ts +++ b/sh/gh/queue.test.ts @@ -35,6 +35,10 @@ for (const job of [check.jobs.check]) { assert.equal(condition(job.if, "merge_group", "", true, "destroyed"), false); } assert.deepEqual(check.on.merge_group.types, ["checks_requested", "destroyed"]); +// A merge group selects from main: a passing head can merge the entries ahead of it. +assert.equal(condition(check.jobs.check.env.CI_BASE, "merge_group", ""), "origin/main"); +assert.equal(condition(check.jobs.check.env.CI_BASE, "pull_request", "release"), "--all"); +for (const step of check.jobs.check.steps) assert.equal(step.env?.CI_BASE, undefined, `${step.name} overrides CI_BASE`); // biome-ignore lint/suspicious/noTemplateCurlyInString: GitHub expression is literal YAML. assert.equal(check.concurrency.group, "check-${{ github.ref }}"); assert.equal(check.concurrency["cancel-in-progress"], true);