diff --git a/quest/m1/README.md b/quest/m1/README.md index b3b330fc544..304b093e503 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -141,7 +141,6 @@ lite-07 stream type here. - [Dead code and API surface audit](/quest/m1/audit-dead-code.md) - repeat the removal audit before the next major release - [Dependency and alternatives audit](/quest/m1/audit-deps.md) - dependencies are current, deduplicated, needed, and the best maintained option -- [Test cost audit](/quest/m1/audit-test-cost.md) - the slowest, flakiest, and least valuable tests are fixed, deleted, or planned - [Interop coverage](/quest/m1/interop-coverage/README.md) - the riskiest moq-transport features moq-net exposes get cross-implementation tests over the even drafts 14 to 22 - [Runner nightly](/quest/m1/interop-runner-nightly.md) - a nightly job runs the upstream interop runner against our relay and clients, so newly registered implementations with a published image are tested without a change here - [CI workflow audit](/quest/m1/audit-ci.md) - every workflow and job gives a signal someone sees, at a cost that signal justifies, or is fixed, moved, or deleted diff --git a/quest/m1/audit-ci.md b/quest/m1/audit-ci.md index c789f38f494..a1dd4b8bf44 100644 --- a/quest/m1/audit-ci.md +++ b/quest/m1/audit-ci.md @@ -5,7 +5,7 @@ Every CI workflow and job earns its place: its results reach someone (a gate, a Discord alert, or a published report) at a cost proportional to that signal. Each one that doesn't is fixed, moved, or deleted. Individual tests stay with -the [test cost audit](/quest/m1/audit-test-cost.md), and runner and cache +the [test cost audit](/quest/m2/audit-test-cost.md), and runner and cache infrastructure with [CI cost and latency](/quest/m0/ci/README.md). ## Plan diff --git a/quest/m1/audit-test-cost.md b/quest/m1/audit-test-cost.md deleted file mode 100644 index 4594e22a945..00000000000 --- a/quest/m1/audit-test-cost.md +++ /dev/null @@ -1,51 +0,0 @@ -# [M] Test cost audit - -## Goal - -The slowest, flakiest, and least valuable tests are sped up, fixed, deleted, -or planned, so the suite costs what it earns. Workflows and jobs belong to the -[CI workflow audit](/quest/m1/audit-ci.md). - -## Plan - -Recurs once per major release (decided 2026-10-19). The PR that finishes -this run also adds the next one with the same file name at the end of the next -milestone's `Required` (m0 to m1, m1 to m2). The successor keeps the Goal -and this paragraph; the rest of its Plan is rewritten with what this run -learned: baselines, results, and what to check first. A successor placed in m2 -also lists the [release cut](/quest/m2/release/cut.md) under `Required`, so it -waits for the release in flight to ship. Only `/quest-delete` ends the -recurrence. Fixes land in the run's PR, one commit each, rather than as follow-ups. -Only a change that needs a maintainer decision (public API, scope) or is -independently landable and too large to review alongside the rest becomes a -follow-up through `/quest-plan`. - -Use nextest's normal CI log to rank the slowest tests (`slow-timeout` is 30s). -Flakes are tracked by hand in [test flakes](/quest/m1/test-flakes-2/README.md), -and `retries = 0` is deliberate. A test that sleeps on the wall clock should -mock time (AGENTS.md). Delete tests whose coverage another test already -provides. Failed Nightly, Interop, and Platform jobs, and what each job costs, moved to -the [CI workflow audit](/quest/m1/audit-ci.md) (2026-10-20). - -What the m0 run (2026-10-19) fixed: the ACME rate-limit test's released port, -the stats session graph's recursive Send proof (send-depth and fuzz), -bench/ffi-gen's stale wasm-bindgen lock, moq-uring's fixed 1 ms calibration -slack, the Nightly JS benchmark stall, the CI selection test's feature count, -and the stats producer benchmark lint. It planned -[redundant failover drift](/quest/m0/redundant-failover-drift.md) -and [macOS QUIC connects](https://github.com/moq-dev/moq/pull/6110). - -Baselines from queue run -[39371990135](https://github.com/moq-dev/moq/actions/runs/39371990135): -Test took 17 min and Check 15 min; nextest ran 7,657 tests in 220 s. The -slowest were the three new redundant-importer exports (37, 29 and 15 s, -waiting out a killed importer's 10 s QUIC idle timeout), the CLI -restart/stitch case (18.6 s), same-instance linger (17.5 s), TS source-limit -timing (16.5 s), the impaired flapping-peer drill (16.4 s), and the one-day -jitter drift simulation (14.7 s). Apart from the redundant exports, the -leaders match the earlier sample and cover different behaviors. - -Still to measure: a JS net benchmark-only edit selects WASM and the net -dependents, and a stats benchmark-only edit selects relay and CLI tests, docs, -and CLI's optional play build. Compare that cost with the regression signal -before changing selection. diff --git a/quest/m2/README.md b/quest/m2/README.md index 9e68f74f899..4e5ef467c06 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -193,3 +193,4 @@ ranked just below deterministic TS legs, which it requires. - [Drop dead items](/quest/m2/drop-dead-items.md) - the audit's uncalled public items and test helpers are removed - [Test coverage audit](/quest/m2/audit-test-coverage.md) - repeat the coverage audit against the m1 baselines after the next release ships - [Documentation audit](/quest/m2/audit-docs.md) - repeat the docs audit after the release ships, against the m1 baselines +- [Test cost audit](/quest/m2/audit-test-cost.md) - the slowest, flakiest, and least valuable tests are fixed, deleted, or planned once the release ships diff --git a/quest/m2/audit-test-cost.md b/quest/m2/audit-test-cost.md new file mode 100644 index 00000000000..7c04af09962 --- /dev/null +++ b/quest/m2/audit-test-cost.md @@ -0,0 +1,78 @@ +# [M] Test cost audit + +## Goal + +The slowest, flakiest, and least valuable tests are sped up, fixed, deleted, +or planned, so the suite costs what it earns. Workflows and jobs belong to the +[CI workflow audit](/quest/m1/audit-ci.md). + +## Plan + +Recurs once per major release (decided 2026-10-19). The PR that finishes +this run also adds the next one with the same file name at the end of the next +milestone's `Required` (m0 to m1, m1 to m2). The successor keeps the Goal +and this paragraph; the rest of its Plan is rewritten with what this run +learned: baselines, results, and what to check first. A successor placed in m2 +also lists the [release cut](/quest/m2/release/cut.md) under `Required`, so it +waits for the release in flight to ship. Only `/quest-delete` ends the +recurrence. Fixes land in the run's PR, one commit each, rather than as follow-ups. +Only a change that needs a maintainer decision (public API, scope) or is +independently landable and too large to review alongside the rest becomes a +follow-up through `/quest-plan`. + +Rank tests from nextest's lines in a full-selection queue log (`slow-timeout` +is 60 s under the `ci` profile). Flakes are tracked by hand in their own +quests, and `retries = 0` is deliberate. A test that sleeps on the wall clock +should mock time (AGENTS.md). Delete tests whose coverage another test already +provides. Repeat each local measurement: parallel agents load the host, so a +single run can be off by 2x. Pin to the queue VM's 8 CPUs with `taskset -c +0-7` when timing the whole suite. A test whose time differs between runs by +about 3 s is waiting out a QUIC draining period, not computing. + +What the m1 run (2026-10-21) fixed: the 900 s TS export past-limit failure the +jitter buffer's tests already cover (16.6 s), the close-code tests dialing +`localhost` against an IPv4-only server (3 s to 0.5 s each on the NixOS +runner, which lists `::1` first), and the split-accept driver tests' 333 ms +initial RTT (3 s to 0.2 s each). A benchmark-only edit now selects only its +own Rust crate or JS workspace; a moq-stats bench edit had run the relay and +CLI suites. To catch the `localhost` race on a host without it, run the +suite under an `/etc/hosts` that lists `::1` first (`unshare -rm`, then a +bind mount over `/etc/hosts`); only the close-code tests regressed. +[moq-dev/moq#6192](https://github.com/moq-dev/moq/pull/6192) landed alongside: +moq-auth's re-checks on the paused clock, moq-c's encoder probe, moq-gst's and +moq-relay's idle and retry waits, and moq-cli's export process tests moved to +nightly. + +Baselines from queue run +[39546261474](https://github.com/moq-dev/moq/actions/runs/39546261474), a full +selection on the 8-core runner VM: nextest ran 9,312 tests in 127 s. Their +summed time was 995 s, so the run is packed by slot time; the 48 tests over +5 s were 480 s of it. By crate: moq-cli 225 s, moq-relay 202 s, moq-mux 130 s, +moq-net 73 s, moq-tokio 59 s, moq-gst 56 s, moq-uring 41 s. + +Check first: + +- Test threads. Pinned to 8 CPUs on a loaded host, the whole suite took + 105 to 113 s at 8 threads (about 185% CPU), 60 s at 16 (315%), 47 s at 24 + (420%), and 41 s at 32 (500%): the suite needs about 200 CPU-seconds, and + most slots wait on sockets, subprocesses, and wall-clock timeouts. + Oversubscribing the queue job is a maintainer decision, since it loads + every timing-sensitive test and a flake ejects a queue entry. Only the + known redundant-importer flake failed, once, at 16 threads. +- moq-gst: about ten tests per run each pay 3 s, at random. `stop` closes the + client, which waits out the draining period (three PTOs at the 333 ms + initial RTT) of a dial to the tests' unreachable relay that never got an + answer. A user stopping a pipeline whose relay is down waits the same. Ask + whether moq-quic may skip draining for a connection that never heard from + its peer. +- moq-uring: three endpoint tests wait out moq-quic's fixed 10 s handshake + idle timeout (30 s together). Neither moq-uring's `Transport` nor + `--quic-idle-timeout` sets it; ask whether the configured idle timeout + should bound the handshake too (public API). +- CPU-bound simulations in a debug build: five TS export timing cases (3 to + 11 s each) and the jitter buffer's day of drift (6 to 13 s), whose `Steer` + rebuilds the hull and median over its whole window on every step. + +## Required + +- [Cut and publish the prepared breaking release](/quest/m2/release/cut.md) - the release in flight ships before this run starts diff --git a/rs/moq-mux/src/container/ts/export_timing_test.rs b/rs/moq-mux/src/container/ts/export_timing_test.rs index 39447181a5b..8de164526b6 100644 --- a/rs/moq-mux/src/container/ts/export_timing_test.rs +++ b/rs/moq-mux/src/container/ts/export_timing_test.rs @@ -739,15 +739,9 @@ async fn a_source_past_the_limit_is_counted() { assert!(stats.release.out_of_tolerance > 0, "not counted"); } -/// The same source fails the export once it has used half the delay, about ten minutes in, -/// rather than drop nearly every frame from then on. -#[tokio::test(start_paused = true)] -#[should_panic(expected = "past the 30 ppm the output's may follow")] -async fn a_source_past_the_limit_fails_once_half_the_delay_is_gone() { - on_a_scaled_clock(1.0004, 900_000_000 / VIDEO_US).await; -} - -/// A source 22% slow, as an importer starved of CPU, fails the export within seconds. +/// A source 22% slow, as an importer starved of CPU, fails the export within seconds. Past the +/// limit, a source fails once it has used half the delay however far off it runs; the jitter +/// buffer's own tests time that at 40 and 400 ppm. #[tokio::test(start_paused = true)] #[should_panic(expected = "the source's clock runs -220")] async fn a_source_far_off_real_time_fails_within_seconds() { diff --git a/rs/moq-tokio/src/quic/driver/tests.rs b/rs/moq-tokio/src/quic/driver/tests.rs index 9a649a92383..8fe55e02445 100755 --- a/rs/moq-tokio/src/quic/driver/tests.rs +++ b/rs/moq-tokio/src/quic/driver/tests.rs @@ -476,6 +476,14 @@ fn blocking_server_pair( let mut server_config = crate::quic::driver::ServerConfig::with_crypto(Arc::new( QuicServerConfig::try_from(server_crypto).unwrap(), )); + // Loopback answers in microseconds. The default 333 ms initial RTT would hold every + // connection closed before its handshake in draining for three seconds. + let transport = Arc::new({ + let mut transport = TransportConfig::default(); + transport.initial_rtt(Duration::from_millis(10)); + transport + }); + server_config.transport_config(transport.clone()); let blocker = Arc::new(HandshakeBlocker::default()); server_config.crypto = Arc::new(BlockingServerConfig { inner: server_config.crypto.clone(), @@ -505,9 +513,10 @@ fn blocking_server_pair( .with_root_certificates(roots) .with_no_client_auth(); client_crypto.alpn_protocols = vec![client_alpn.to_vec()]; - client.set_default_client_config(ClientConfig::new(Arc::new( - QuicClientConfig::try_from(client_crypto).unwrap(), - ))); + let mut client_config = + ClientConfig::new(Arc::new(QuicClientConfig::try_from(client_crypto).unwrap())); + client_config.transport_config(transport); + client.set_default_client_config(client_config); (client, server, blocker) } diff --git a/rs/moq-tokio/tests/integration/close_code.rs b/rs/moq-tokio/tests/integration/close_code.rs index 1a45c9cf6d4..dcb62c99c92 100644 --- a/rs/moq-tokio/tests/integration/close_code.rs +++ b/rs/moq-tokio/tests/integration/close_code.rs @@ -93,9 +93,11 @@ async fn client_error(url: &str, websocket: bool) -> moq_tokio::Error { async fn check(case: Case) { let server = serve(case).await; + // The server binds IPv4 only, so `localhost` would race a dead `::1` on hosts that list it + // first, paying the Happy Eyeballs stagger on every dial. let urls = [ - (format!("https://localhost:{}/", server.quic), false), - (format!("moqt://localhost:{}/", server.quic), false), + (format!("https://127.0.0.1:{}/", server.quic), false), + (format!("moqt://127.0.0.1:{}/", server.quic), false), (format!("ws://127.0.0.1:{}/", server.websocket), true), ]; diff --git a/sh/js/workspace.py b/sh/js/workspace.py index 60ccb077a42..19cf4f73f18 100644 --- a/sh/js/workspace.py +++ b/sh/js/workspace.py @@ -65,6 +65,8 @@ def selection(root, paths): "inputs": inputs(root, directory)} selected = set() + # A workspace's `bench/` is no export, so no dependent can import it: only its owner checks it. + benched = set() fallback = paths is None for path in paths or []: owners = [(name, p["directory"]) for name, p in packages.items() @@ -72,7 +74,8 @@ def selection(root, paths): if not owners: fallback = True break - selected.add(max(owners, key=lambda owner: len(owner[1]))[0]) + name, directory = max(owners, key=lambda owner: len(owner[1])) + (benched if path.startswith(directory + "/bench/") else selected).add(name) # Manifests miss a package reading another tree's files, so add those readers too. selected |= {name for name, package in packages.items() if any(reads(path, i) for i in package["inputs"])} @@ -88,6 +91,7 @@ def selection(root, paths): if expanded == selected: break selected = expanded + selected |= benched # A selected consumer can need an unchanged dependency's generated bindings. dependencies = set(selected) diff --git a/sh/js/workspace.test.py b/sh/js/workspace.test.py index 2ecd0587a32..d90ede5bff5 100644 --- a/sh/js/workspace.test.py +++ b/sh/js/workspace.test.py @@ -89,6 +89,11 @@ def test_transitive_dependents_from_manifests(self): self.assertFalse(plan["all"]) self.assertFalse(plan["wasm"]) + def test_a_bench_selects_only_its_workspace(self): + self.assertEqual(list(self.plan(["js/core/bench/frames.ts"])["packages"]), ["@example/core"]) + self.assertEqual(set(self.plan(["js/core/bench/frames.ts", "js/core/src/index.ts"])["packages"]), + {"@example/core", "@example/view", "docs"}) + def test_leaf_does_not_select_its_dependencies(self): self.assertEqual(list(self.plan(["doc/index.md"])["packages"]), ["docs"]) diff --git a/sh/rs/select.sh b/sh/rs/select.sh index ed31575a88d..c84ae0b11fa 100755 --- a/sh/rs/select.sh +++ b/sh/rs/select.sh @@ -48,9 +48,13 @@ select_packages() { return fi - local seeds edges selected + local seeds benches edges selected + # A crate's `benches/` builds only its own bench targets, which no other crate can + # depend on and the test build never compiles, so it selects that crate alone + # (for its lint) rather than every dependent's tests. + benches=$(sed -n 's|^rs/\([^/]*\)/benches/.*|\1|p' "$list") seeds=$({ - sed -n 's|^rs/\([^/]*\)/.*|\1|p' "$list" + sed -n '\|^rs/[^/]*/benches/|d; s|^rs/\([^/]*\)/.*|\1|p' "$list" jq -r --rawfile list "$list" '($list | split("\n")) as $paths | .packages[] | (.manifest_path | split("/")) as $p | select($p[-4] == "demo") | select(any($paths[]; startswith("demo/\($p[-3])/\($p[-2])/"))) | .name' <<<"$metadata" @@ -74,7 +78,8 @@ select_packages() { } while (grew) for (name in want) if (want[name]) print name } - ' <<<"$edges" | sort -u) + ' <<<"$edges") + selected=$(printf '%s\n%s\n' "$selected" "$benches" | sed '/^$/d' | sort -u) [[ -n "$selected" ]] || return 0 diff --git a/sh/rs/select.test.py b/sh/rs/select.test.py index 1cf3b15358c..7d7f9bee0cf 100644 --- a/sh/rs/select.test.py +++ b/sh/rs/select.test.py @@ -54,6 +54,16 @@ def test_demo_crate_follows_its_dependencies(self): self.assertTrue(any("/demo/carla/bridge#moq-carla-bridge" in call for call in calls), calls) self.assertEqual(self.selected("demo/carla/web/src/main.ts"), []) + def test_a_bench_selects_only_its_crate(self): + bench = "rs/moq-stats/benches/producer_tick.rs" + for action in ("lint", "test"): + with self.subTest(action=action): + calls = self.selected(bench, action=action) + self.assertTrue(any("#moq-stats@" in call or "/rs/moq-stats#" in call for call in calls), calls) + self.assertFalse(any("moq-relay" in call or "moq-cli" in call for call in calls), calls) + calls = self.selected(bench, "rs/moq-stats/src/lib.rs", action="test") + self.assertTrue(any("/rs/moq-relay#" in call for call in calls), calls) + class RecipeTest(unittest.TestCase):