Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 82 additions & 15 deletions .github/ci-selection-audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@

The shared-input rules rarely force every module. Keep the conservative Rust
lockfile rule rather than add a second dependency resolver. Fix the missing
Cargo-configuration edge now. Replaying historical full-run failures for missed
edges remains open.
Cargo-configuration edge now. Replaying the full-gate failures found no missed
path edge; it found one missed queue edge, now fixed: a merge group diffed from
the entry ahead of it rather than from main.

## Observation window and method

Expand Down Expand Up @@ -107,17 +108,83 @@ settle the rule decisions above without a new measurement window.

## Missed edges

Two recent full-run failures illustrate why replay needs to distinguish selector
bugs from other failures. The [October 25 full gate](https://github.com/moq-dev/moq/actions/runs/40192013303/job/121164806731)
failed because its shallow checkout had no `HEAD~1` for a tooling test. That is a
checkout-history problem, not a missed package edge. The same Nightly's
[feature lane](https://github.com/moq-dev/moq/actions/runs/40192013303/job/121164807008)
failed on no-default-feature rustdoc links in moq-net. The current selected
net-feature boundary already invokes that documentation build; workspace feature
unification is separately being addressed in
[moq-dev/moq#6830](https://github.com/moq-dev/moq/pull/6830) and the async-free
### Full-gate replay

Nightly began running the queue's gate over every package (`just gate`) on
October 20, in [moq-dev/moq#6038](https://github.com/moq-dev/moq/pull/6038); before that no full run existed to
compare against. Every scheduled Nightly gate from October 20 through October 27
is below. Other Nightly lanes, such as the feature matrix, run work the queue never
selects, so they cannot expose a selection miss.

| Nightly | Gate | Cause | Selection miss |
| --- | --- | --- | --- |
| [October 20](https://github.com/moq-dev/moq/actions/runs/39421743227) | passed | | |
| [October 21](https://github.com/moq-dev/moq/actions/runs/39600361153/job/119223010596) | failed | Dangling quest link | Yes, the queue base below |
| [October 22](https://github.com/moq-dev/moq/actions/runs/39779729461/job/119827590096) | failed | Nine `@moq/watch` CMAF decoder tests | No |
| [October 23](https://github.com/moq-dev/moq/actions/runs/39951870847) | passed | | |
| [October 24](https://github.com/moq-dev/moq/actions/runs/40090959108/job/120863338990) | failed | Shallow checkout has no `HEAD~1` | No |
| [October 25](https://github.com/moq-dev/moq/actions/runs/40192013303/job/121164806731) | failed | Same | No |
| [October 26](https://github.com/moq-dev/moq/actions/runs/40332938121) | passed | | |
| [October 27](https://github.com/moq-dev/moq/actions/runs/40506938673/job/122198526371) | timed out | No output for its last 36 minutes | No |

- October 21: [moq-dev/moq#5849](https://github.com/moq-dev/moq/pull/5849) deleted a quest that
[moq-dev/moq#6122](https://github.com/moq-dev/moq/pull/6122) had just linked. Its own queue entry
[failed on that link](https://github.com/moq-dev/moq/actions/runs/39572680575), yet it merged when the next entry,
[moq-dev/moq#6012](https://github.com/moq-dev/moq/pull/6012), [passed](https://github.com/moq-dev/moq/actions/runs/39578702167). That entry diffed from
the failed entry's head, touched no quest, and so never ran the quest check.
- October 22: the queue entry for [moq-dev/moq#6219](https://github.com/moq-dev/moq/pull/6219)
[ran these tests](https://github.com/moq-dev/moq/actions/runs/39747822278), 387 passing. At the Nightly head, the whole
`@moq/watch` suite passes locally while `src/video/decoder.test.ts` alone fails
the same nine tests: they depend on test-file order, which differs between
runners. That file passes alone on current main.
- October 24 and 25: the queue fetches full history; Nightly checks out shallowly.
[moq-dev/moq#6708](https://github.com/moq-dev/moq/pull/6708) made the test build its own history.
- October 27: outside the window. The job hit its 60-minute timeout 36 minutes
after its last output, a hang rather than a narrower selection.

The feature lane's no-default-feature rustdoc failure in
[that October 25 Nightly](https://github.com/moq-dev/moq/actions/runs/40192013303/job/121164807008) is a workspace feature
unification problem, addressed in [moq-dev/moq#6830](https://github.com/moq-dev/moq/pull/6830) and the async-free
contract in [moq-dev/moq#6851](https://github.com/moq-dev/moq/pull/6851).
These observations do not substitute for replaying all month's full-run failures
against the contemporaneous selected gate. That replay remains in
[its quest](/quest/m0/ci/select-audit.md). No claim of complete missed-edge
coverage is made here.

### Queue entries merged without passing

The October 21 miss is one instance of a wider hole. From October 20 23:58 to
October 22 23:33 UTC, the main ruleset's merge queue used the `HEADGREEN`
grouping strategy, which merges every entry under a passing head, while Check
diffed each entry from `merge_group.base_sha`, the entry ahead of it. A head
therefore vouched for entries whose changes it never selected. Thirty-seven
attempts landed on main without their head ever passing, all within
that interval and none under `ALLGREEN`:

| Failure carried into main | Attempts |
| --- | ---: |
| Dangling quest links, four different deletions | 13 |
| `@moq/net` request and draining-session tests | 6 |
| Cancelled | 5 |
| JS doc comments missing on exported error members | 3 |
| `moq-cli` redundant-importer bufferbloat test | 3 |
| `moq-video` hev1 CMAF round trip | 2 |
| `moq-srt` late-payload host instant | 2 |
| `moq-net` test build, a call with the wrong argument count | 1 |
| JS type check, a removed `Manual.advanceAsync` | 1 |
| `moq-relay` GOAWAY deadline drain | 1 |

The static failures (links, doc comments, type and compile errors) are
deterministic breakage that reached main. The test failures were not reproduced
here, so some may be flakes. Either way, a narrower selection is what let them
through: the passing head never ran them.

Check now diffs a merge group from `origin/main`, the main commit its head is
built on, so a head's run selects every entry it can merge, whichever grouping
strategy the ruleset uses. `sh/gh/queue.test.ts` pins the workflow base, and
`sh/dispatch-query.test.py` covers an unmerged entry ahead. The change costs
nothing at today's settings: of 487 attempts from October 22 23:33 UTC to the
window end, with one entry built at a time, none started before the entry ahead
had merged. With five entries built at once, October 10 through 20, 806 of 1,047
attempts started on an unmerged entry and would now also select its changes.

Five pull requests merged outside the queue after October 10, four of them on
October 23 between 21:33 and 22:56 UTC while the ruleset had no merge queue rule.
They skipped Check entirely, which no selection rule can address, and none caused
a full-gate failure above.
11 changes: 6 additions & 5 deletions .github/workflows/check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,18 @@ jobs:
environment: ${{ (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/heads/gh-readonly-queue/main/')) && 'cache-write' || '' }}
# Backstop a hang instead of holding the runner for the 6-hour default.
timeout-minutes: 60
env:
# Diff a merge group from main, not from the entry ahead of it: when the
# queue requires only a group's head to pass, that run vouches for every
# entry it merges. Building one entry at a time, the base is main anyway.
CI_BASE: ${{ github.base_ref == 'release' && '--all' || 'origin/main' }}

steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# Full history so `just ci` can diff against the PR or merge group base.
# Full history so `just ci` can diff against main.
fetch-depth: 0

- uses: ./.github/actions/just
Expand All @@ -57,8 +62,6 @@ jobs:
- name: Select work
id: scope
run: just ci-select "$CI_BASE" >> "$GITHUB_OUTPUT"
env:
CI_BASE: ${{ github.base_ref == 'release' && '--all' || github.event.merge_group.base_sha }}

- uses: ./.github/actions/nix
if: steps.scope.outputs.needed == 'true' && runner.environment != 'self-hosted'
Expand Down Expand Up @@ -86,8 +89,6 @@ jobs:
export NEXTEST_TEST_THREADS=24
fi
nix develop --command just ci "$CI_BASE"
env:
CI_BASE: ${{ github.base_ref == 'release' && '--all' || github.event.merge_group.base_sha }}

# Cargo's per-crate timings for the Rust test builds, so a slow crate
# shows without rebuilding. The job summary has the per-module times.
Expand Down
1 change: 0 additions & 1 deletion quest/m0/ci/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,6 @@ The historical samples locate candidates; they are not controlled benchmarks.
- [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - measure the hosted Nightly jobs' hits, restore time, and expiry once the bump lands
- [Verify the Nix cache](/quest/m0/ci/nix-cache-verify.md) - confirm CI and releases use `nix.moq.dev` and retire Cachix
- [Unify dependency features](/quest/m0/ci/hakari.md) - a cargo-hakari workspace-hack so every job compiles each dependency alike and mbx hits across jobs
- [Replay full-run failures against the selected gate](/quest/m0/ci/select-audit.md) - every dependency edge `just ci` misses is found and fixed with a test
- [First shared FFI release](/quest/m0/ci/nightly-ffi-reuse.md) - condition: the backport merges and the next `moq-ffi-v*` tag runs `Release FFI`; verify the wrappers
- [Fewer BTreeMap instantiations](/quest/m0/ci/btree-keys.md) - moq-net shares BTreeMap key and value types, kept only if its CPU time drops
- [Halve the slowest tests' CPU](/quest/m0/ci/slow-tests.md) - the slowest 1% of tests take under half their 289 CPU-seconds, with coverage kept
29 changes: 0 additions & 29 deletions quest/m0/ci/select-audit.md

This file was deleted.

5 changes: 3 additions & 2 deletions sh/dispatch-query.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -124,12 +124,13 @@ def test_working_tree_changes_and_deletions_count(self):
def test_release_pr_base_and_merge_group_base(self):
self.env["GITHUB_BASE_REF"] = "release"
self.expect(True, base="", rust=False)
# The entry queued ahead changes Rust and has not merged; this one changes a quest.
self.change("rs/moq-net/src/lib.rs")
self.git("add", ".")
self.git("commit", "-qm", "queued source")
group_base = self.git("rev-parse", "HEAD")
self.change("quest/m1/example.md")
self.expect(True, base=group_base, rust=False)
# Check's merge group base: a passing head can merge the entry ahead, so it selects that too.
self.expect(True, base="origin/main", rust=True)
self.expect(True, base="origin/release")

def test_invalid_base_fails_instead_of_reporting_no_work(self):
Expand Down
4 changes: 4 additions & 0 deletions sh/gh/queue.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,10 @@ for (const job of [check.jobs.check]) {
assert.equal(condition(job.if, "merge_group", "", true, "destroyed"), false);
}
assert.deepEqual(check.on.merge_group.types, ["checks_requested", "destroyed"]);
// A merge group selects from main: a passing head can merge the entries ahead of it.
assert.equal(condition(check.jobs.check.env.CI_BASE, "merge_group", ""), "origin/main");
assert.equal(condition(check.jobs.check.env.CI_BASE, "pull_request", "release"), "--all");
for (const step of check.jobs.check.steps) assert.equal(step.env?.CI_BASE, undefined, `${step.name} overrides CI_BASE`);
// biome-ignore lint/suspicious/noTemplateCurlyInString: GitHub expression is literal YAML.
assert.equal(check.concurrency.group, "check-${{ github.ref }}");
assert.equal(check.concurrency["cancel-in-progress"], true);
Expand Down
Loading