Repository navigation
ci(nightly): delete unreported relay measurements - #7603
Conversation
Quest-Claim: 097afce7f7c9a6e5
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
kixelated
changed the title
kixelated
left a comment
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit: e079631
No actionable bugs found. Removing the three unreported measurement jobs is a proportionate simplification: on-demand diagnostics and deterministic profile-tool tests remain, while paired benchmark reporting, release artifacts, and the other Nightly coverage are unchanged. The remaining matrix needs neither the runner override nor perf sysctl setup.
Verification: static review of the workflow, related recipes and documentation. GitHub's November 1 job records confirm the 180-job status breakdown and the removed jobs' approximately 55.5 runner-minutes; this is not a measured wall-time improvement. I did not run tests/builds or dispatch workflows, and did not independently reproduce the full 13-night or billing audit.
(Written by OpenAI)
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
- Delete three Nightly relay measurement jobs —
bench-audio,bench-shard, andbench-profilematrix entries removed from.github/workflows/nightly.yml; their recipes stay as documented local diagnostics. - Remove now-unused job plumbing — hardcodes
runs-on: ubuntu-24.04-arm(the onlyrunneroverride was the deleted Relay profiles job) and drops theperfmatrix key plus theAllow perf kernel samplingsysctl step. - Update
bench/README.md— states the audio grouping bench, shard sweep, and profile capture are on-demand, not nightly series. - Finish the one-off m1 quest — deletes
quest/m1/prune-ci.md, removes its link fromquest/m1/README.md, and repointsquest/m1/prune-tests.mdto the recurring CI audit. - Refresh
quest/m2/audit-ci.md— rewrites baselines and per-workflow/per-job tables (56 workflows, 180 Nightly jobs) and trims the relay-benches bullet fromquest/m2/bench-sources.md.
I verified the cleanup independently: no matrix.runner/matrix.perf references remain, no test-inventory target is orphaned (bench/run.sh has no test-runner pattern and stays reached via just gh criterion, while bench/relay.test.py and bench/profile.test.py stay reached via just gh check/gh test), no dangling prune-ci references remain, and every quest link in the changed markdown resolves. The workflow-policy checks pass on the change (bun test sh/gh/queue.test.ts, bash sh/gh/alert.sh check-coverage, bun test sh/gh/setup.test.ts). The audit counts are internally consistent (56 workflow rows, 180 job rows, 28 skipped leaving 152 executed, 19 failures plus one cancelled), and the local perf_event_paranoid requirement the removed step satisfied is still documented in bench/README.md:232.
deepseek-v4.1-flash (free via Pullfrog for OSS) ｜ 𝕏
|
No actionable comments were generated in the recent review.
 |
| Check name | Status | Explanation |
|---|---|---|
| Title check |  |
The title clearly and concisely describes the main change: removing Nightly relay measurements that do not report results. |
| Linked Issues check | |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | |
Check skipped because no linked issues were found for this pull request. |
| Description check | |
The description is closely related to the changeset. It explains that three Nightly relay measurement jobs (audio grouping relay benchmark, relay shard sweep, relay profiles) are deleted from nightly.… |
Full details: Description check
-
Explanation
The description is closely related to the changeset. It explains that three Nightly relay measurement jobs (audio grouping relay benchmark, relay shard sweep, relay profiles) are deleted from nightly.yml, with runner-minute costs and the reasons for removal. It also covers the refresh of quest/m2/audit-ci.md, the removal of the one-off m1 quest and its references, the removal of the unused matrix runner override and perf sysctl step, and the validation performed. The author supplies concrete, specific details that match the raw summary of file changes.
Finishing Touches
-
Simplify code
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
|
codex / Codex land fixer note:
(written by codex / Codex) |
|
codex / Codex land fixer: Conflicts with the base: quest/m2/audit-ci.md (written by codex / Codex) |
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
- Merged
mainand repaired the stale join reference — the deleted/quest/m2/ietf-join-after-finish.mdlink now points at#7306(merged Rust publisher fix) and/quest/m2/ietf-js-joining-fetch.md(JS half), in both the verdict bullet and the earlier-failure note. - Added the Queue workflow —
queue.ymlrow appears in the workflow table and the merge-queue baseline now says#7612replacedquest land servewith GitHub's single-PR merge queue on the self-hosted runner (main PRs skip Check and Queue; merge group runsjust ci), matchingCONTRIBUTING.mdandcheck.yml. - Preserved open context — capture/play build decision, the Windows
reconnect::a_protocol_violation_ends_reconnectrecheck, and operational lessons (just gh pages-clean) stay in the recurring audit. - Clarified totals and ownership — the 747.2-minute total is stated as summed before per-row rounding, and the applied deletion links to this PR.
Verified the touched claims: 57 workflow files match 57 audit rows (56 at the sample + Queue afterward), #7306 and #7612 are merged, the reconnect test still exists, just gh pages-clean is in nightly.yml, and every local link in audit-ci.md resolves with no dangling ietf-join-after-finish references.
deepseek-v4.1-flash (free via Pullfrog for OSS) ｜ 𝕏
Main finished quest/m1/prune-ci.md (#7603) and repointed prune-tests.md's CI link; this branch finishes prune-tests.md, so both quests stay deleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep main's `benchmark_datagram_delivery` run in the rust-test group as its own nextest run, and drop the relay load benches from the quest now that #7603 removed them from Nightly. Main's additions to the rust-test group must print measured CSV too: - `benchmark_datagram_delivery` printed its counts as dimensions, which would churn every series name; they become `*_packets` columns, and `received` (offered minus lost) goes. - moq-congestion's new GCC `benchmark_per_packet` printed key=value, and its name matched BBR's in the same package, so the report would refuse both. It now prints CSV, and rust-test series keep the test's module path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem
Three Nightly relay measurement jobs consume runner time without publishing their measurements or enforcing a performance comparison. The recurring CI audit still describes the retired hosted merge queue and October 22 workflow inventory.
Approach
Delete the three jobs from
nightly.ymlin one workflow commit. The following are actual elapsed runner-minutes, including setup and teardown, from all 13 scheduled Nightlies on October 20 through November 1:All 39 job executions succeeded. Total observed cost: 747.2 runner-minutes; latest night: 55.5 minutes. These are removed job costs, not a measured Nightly wall-time improvement. Keep the local diagnostic commands and deterministic profile-tool tests; update their documentation. Remove the now-unused matrix runner override and perf sysctl step.
Refresh
quest/m2/audit-ci.mdin place: all 56 workflows in the November 1 sample plus the subsequently added Queue workflow, and all 180 jobs in that scheduled Nightly, monthly billing, current runner-minute samples, failure classifications, visibility gaps and ownership. The November 1 Nightly executed 152 jobs: 132 succeeded, 19 failed, one was cancelled; 28 were skipped. Keep its unique fuzzing, model, feature, platform, ABI and packaging signal. Successful runs are coverage evidence, not proof of a caught regression.Complete and delete the one-off m1 quest and repair its references. The recurring audit retains its release prerequisite and schedule; no successor is created. Merge current main, preserve the Rust publisher fix and JS joining FETCH ownership references, and distinguish the historical gate sample from the current merge queue. Retain the earlier capture/play decision, Windows reconnect recheck and operational lessons for the recurring audit.
Recorded choices and overlap
Apply the maintainer's November 1 default-to-deleting decision. Do not remove release targets or published artifacts. Keep CodSpeed's distinct instruction-count signal under the recorded October 28 decision, rather than treating it as duplicate timing. Newly added weekly mutation testing has no scheduled cost sample yet and tests a distinct security/grant/resume/expiry boundary.
Concrete ownership checked before edits:
Impact
Alternatives
Publishing more nightly series would retain recurring work without an identified consumer. Moving these already-nightly jobs off the merge path saves nothing. Retain the on-demand diagnostics for investigations and the existing visible/paired benchmark lanes.
Validation
7682d12c1a2592190d521d551b43e486d1d84b00,quest gate nix develop --ignore-environment --keep HOME --keep USER --keep PATH --command just ci origin/main: passed, 37 seconds. All selected modules passed, including benchmark checks, workflow/security lint, alert coverage, test inventory, GitHub tooling tests, quest, Markdown, shell, TOML, Nix, justfile and large-file checks./dev/kvm. All 30 dispatcher fixture tests and their selection benchmark passed through the standard gate. The earlier missing-KVM handback does not reproduce here. The host-dependent fixture fix remains with moq-dev/moq#7598; no duplicate fix or CI bypass was added.quest check --base origin/main: passed, 604 documents.git diff --check origin/main...HEAD: passed. All 57 current workflow files have audit rows; November 1's 180-job cost/status inventory remains historical evidence. Re-fetched all 39 removed-job executions across the 13 nights: 44,829 seconds, or 747.15 runner-minutes before rounding.just gate, Nightly dispatch, release build or interop run was needed or claimed. No public API or wire changes.Review
Claude Code,
claude-opus-5-5, medium effort, localquest review, round 2 reviewed head7682d12c1a2592190d521d551b43e486d1d84b00againstorigin/main(7168b8f9720df554bf53f39cd5729afefd4bcaa8): no findings.The preceding round's findings are resolved:
The original implementation review also completed two clean/fixed rounds on
e07963121b47854f66e5d5979c8ee58e98b06e26; the review above covers this repaired head.Follow-ups
No successor or follow-up quest created or planned. Existing owners retain the failures/reporting work above. Release target/artifact removal remains a maintainer decision. No new decisions. The parent retains queue/merge ownership; this implementation does not merge or monitor.
(Written by GPT-6.1 Sol)