Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,6 @@ lite-07 stream type here.

- [Dead code and API surface audit](/quest/m1/audit-dead-code.md) - repeat the removal audit before the next major release
- [Dependency and alternatives audit](/quest/m1/audit-deps.md) - dependencies are current, deduplicated, needed, and the best maintained option
- [Test cost audit](/quest/m1/audit-test-cost.md) - the slowest, flakiest, and least valuable tests are fixed, deleted, or planned
- [Interop coverage](/quest/m1/interop-coverage/README.md) - the riskiest moq-transport features moq-net exposes get cross-implementation tests over the even drafts 14 to 22
- [Runner nightly](/quest/m1/interop-runner-nightly.md) - a nightly job runs the upstream interop runner against our relay and clients, so newly registered implementations with a published image are tested without a change here
- [CI workflow audit](/quest/m1/audit-ci.md) - every workflow and job gives a signal someone sees, at a cost that signal justifies, or is fixed, moved, or deleted
Expand Down
2 changes: 1 addition & 1 deletion quest/m1/audit-ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
Every CI workflow and job earns its place: its results reach someone (a gate,
a Discord alert, or a published report) at a cost proportional to that signal.
Each one that doesn't is fixed, moved, or deleted. Individual tests stay with
the [test cost audit](/quest/m1/audit-test-cost.md), and runner and cache
the [test cost audit](/quest/m2/audit-test-cost.md), and runner and cache
infrastructure with [CI cost and latency](/quest/m0/ci/README.md).

## Plan
Expand Down
51 changes: 0 additions & 51 deletions quest/m1/audit-test-cost.md

This file was deleted.

1 change: 1 addition & 0 deletions quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,3 +193,4 @@ ranked just below deterministic TS legs, which it requires.
- [Drop dead items](/quest/m2/drop-dead-items.md) - the audit's uncalled public items and test helpers are removed
- [Test coverage audit](/quest/m2/audit-test-coverage.md) - repeat the coverage audit against the m1 baselines after the next release ships
- [Documentation audit](/quest/m2/audit-docs.md) - repeat the docs audit after the release ships, against the m1 baselines
- [Test cost audit](/quest/m2/audit-test-cost.md) - the slowest, flakiest, and least valuable tests are fixed, deleted, or planned once the release ships
78 changes: 78 additions & 0 deletions quest/m2/audit-test-cost.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# [M] Test cost audit

## Goal

The slowest, flakiest, and least valuable tests are sped up, fixed, deleted,
or planned, so the suite costs what it earns. Workflows and jobs belong to the
[CI workflow audit](/quest/m1/audit-ci.md).

## Plan

Recurs once per major release (decided 2026-10-19). The PR that finishes
this run also adds the next one with the same file name at the end of the next
milestone's `Required` (m0 to m1, m1 to m2). The successor keeps the Goal
and this paragraph; the rest of its Plan is rewritten with what this run
learned: baselines, results, and what to check first. A successor placed in m2
also lists the [release cut](/quest/m2/release/cut.md) under `Required`, so it
waits for the release in flight to ship. Only `/quest-delete` ends the
recurrence. Fixes land in the run's PR, one commit each, rather than as follow-ups.
Only a change that needs a maintainer decision (public API, scope) or is
independently landable and too large to review alongside the rest becomes a
follow-up through `/quest-plan`.

Rank tests from nextest's lines in a full-selection queue log (`slow-timeout`
is 60 s under the `ci` profile). Flakes are tracked by hand in their own
quests, and `retries = 0` is deliberate. A test that sleeps on the wall clock
should mock time (AGENTS.md). Delete tests whose coverage another test already
provides. Repeat each local measurement: parallel agents load the host, so a
single run can be off by 2x. Pin to the queue VM's 8 CPUs with `taskset -c
0-7` when timing the whole suite. A test whose time differs between runs by
about 3 s is waiting out a QUIC draining period, not computing.

What the m1 run (2026-10-21) fixed: the 900 s TS export past-limit failure the
jitter buffer's tests already cover (16.6 s), the close-code tests dialing
`localhost` against an IPv4-only server (3 s to 0.5 s each on the NixOS
runner, which lists `::1` first), and the split-accept driver tests' 333 ms
initial RTT (3 s to 0.2 s each). A benchmark-only edit now selects only its
own Rust crate or JS workspace; a moq-stats bench edit had run the relay and
CLI suites. To catch the `localhost` race on a host without it, run the
suite under an `/etc/hosts` that lists `::1` first (`unshare -rm`, then a
bind mount over `/etc/hosts`); only the close-code tests regressed.
[moq-dev/moq#6192](https://github.com/moq-dev/moq/pull/6192) landed alongside:
moq-auth's re-checks on the paused clock, moq-c's encoder probe, moq-gst's and
moq-relay's idle and retry waits, and moq-cli's export process tests moved to
nightly.

Baselines from queue run
[39546261474](https://github.com/moq-dev/moq/actions/runs/39546261474), a full
selection on the 8-core runner VM: nextest ran 9,312 tests in 127 s. Their
summed time was 995 s, so the run is packed by slot time; the 48 tests over
5 s were 480 s of it. By crate: moq-cli 225 s, moq-relay 202 s, moq-mux 130 s,
moq-net 73 s, moq-tokio 59 s, moq-gst 56 s, moq-uring 41 s.

Check first:

- Test threads. Pinned to 8 CPUs on a loaded host, the whole suite took
105 to 113 s at 8 threads (about 185% CPU), 60 s at 16 (315%), 47 s at 24
(420%), and 41 s at 32 (500%): the suite needs about 200 CPU-seconds, and
most slots wait on sockets, subprocesses, and wall-clock timeouts.
Oversubscribing the queue job is a maintainer decision, since it loads
every timing-sensitive test and a flake ejects a queue entry. Only the
known redundant-importer flake failed, once, at 16 threads.
- moq-gst: about ten tests per run each pay 3 s, at random. `stop` closes the
client, which waits out the draining period (three PTOs at the 333 ms
initial RTT) of a dial to the tests' unreachable relay that never got an
answer. A user stopping a pipeline whose relay is down waits the same. Ask
whether moq-quic may skip draining for a connection that never heard from
its peer.
- moq-uring: three endpoint tests wait out moq-quic's fixed 10 s handshake
idle timeout (30 s together). Neither moq-uring's `Transport` nor
`--quic-idle-timeout` sets it; ask whether the configured idle timeout
should bound the handshake too (public API).
- CPU-bound simulations in a debug build: five TS export timing cases (3 to
11 s each) and the jitter buffer's day of drift (6 to 13 s), whose `Steer`
rebuilds the hull and median over its whole window on every step.

## Required

- [Cut and publish the prepared breaking release](/quest/m2/release/cut.md) - the release in flight ships before this run starts
12 changes: 3 additions & 9 deletions rs/moq-mux/src/container/ts/export_timing_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -739,15 +739,9 @@ async fn a_source_past_the_limit_is_counted() {
assert!(stats.release.out_of_tolerance > 0, "not counted");
}

/// The same source fails the export once it has used half the delay, about ten minutes in,
/// rather than drop nearly every frame from then on.
#[tokio::test(start_paused = true)]
#[should_panic(expected = "past the 30 ppm the output's may follow")]
async fn a_source_past_the_limit_fails_once_half_the_delay_is_gone() {
on_a_scaled_clock(1.0004, 900_000_000 / VIDEO_US).await;
}

/// A source 22% slow, as an importer starved of CPU, fails the export within seconds.
/// A source 22% slow, as an importer starved of CPU, fails the export within seconds. Past the
/// limit, a source fails once it has used half the delay however far off it runs; the jitter
/// buffer's own tests time that at 40 and 400 ppm.
#[tokio::test(start_paused = true)]
#[should_panic(expected = "the source's clock runs -220")]
async fn a_source_far_off_real_time_fails_within_seconds() {
Expand Down
15 changes: 12 additions & 3 deletions rs/moq-tokio/src/quic/driver/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -476,6 +476,14 @@ fn blocking_server_pair(
let mut server_config = crate::quic::driver::ServerConfig::with_crypto(Arc::new(
QuicServerConfig::try_from(server_crypto).unwrap(),
));
// Loopback answers in microseconds. The default 333 ms initial RTT would hold every
// connection closed before its handshake in draining for three seconds.
let transport = Arc::new({
let mut transport = TransportConfig::default();
transport.initial_rtt(Duration::from_millis(10));
transport
});
server_config.transport_config(transport.clone());
let blocker = Arc::new(HandshakeBlocker::default());
server_config.crypto = Arc::new(BlockingServerConfig {
inner: server_config.crypto.clone(),
Expand Down Expand Up @@ -505,9 +513,10 @@ fn blocking_server_pair(
.with_root_certificates(roots)
.with_no_client_auth();
client_crypto.alpn_protocols = vec![client_alpn.to_vec()];
client.set_default_client_config(ClientConfig::new(Arc::new(
QuicClientConfig::try_from(client_crypto).unwrap(),
)));
let mut client_config =
ClientConfig::new(Arc::new(QuicClientConfig::try_from(client_crypto).unwrap()));
client_config.transport_config(transport);
client.set_default_client_config(client_config);

(client, server, blocker)
}
Expand Down
6 changes: 4 additions & 2 deletions rs/moq-tokio/tests/integration/close_code.rs
Original file line number Diff line number Diff line change
Expand Up @@ -93,9 +93,11 @@ async fn client_error(url: &str, websocket: bool) -> moq_tokio::Error {

async fn check(case: Case) {
let server = serve(case).await;
// The server binds IPv4 only, so `localhost` would race a dead `::1` on hosts that list it
// first, paying the Happy Eyeballs stagger on every dial.
let urls = [
(format!("https://localhost:{}/", server.quic), false),
(format!("moqt://localhost:{}/", server.quic), false),
(format!("https://127.0.0.1:{}/", server.quic), false),
(format!("moqt://127.0.0.1:{}/", server.quic), false),
(format!("ws://127.0.0.1:{}/", server.websocket), true),
];

Expand Down
6 changes: 5 additions & 1 deletion sh/js/workspace.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,14 +65,17 @@ def selection(root, paths):
"inputs": inputs(root, directory)}

selected = set()
# A workspace's `bench/` is no export, so no dependent can import it: only its owner checks it.
benched = set()
fallback = paths is None
for path in paths or []:
owners = [(name, p["directory"]) for name, p in packages.items()
if path == p["directory"] or path.startswith(p["directory"] + "/")]
if not owners:
fallback = True
break
selected.add(max(owners, key=lambda owner: len(owner[1]))[0])
name, directory = max(owners, key=lambda owner: len(owner[1]))
(benched if path.startswith(directory + "/bench/") else selected).add(name)
# Manifests miss a package reading another tree's files, so add those readers too.
selected |= {name for name, package in packages.items()
if any(reads(path, i) for i in package["inputs"])}
Expand All @@ -88,6 +91,7 @@ def selection(root, paths):
if expanded == selected:
break
selected = expanded
selected |= benched

# A selected consumer can need an unchanged dependency's generated bindings.
dependencies = set(selected)
Expand Down
5 changes: 5 additions & 0 deletions sh/js/workspace.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,11 @@ def test_transitive_dependents_from_manifests(self):
self.assertFalse(plan["all"])
self.assertFalse(plan["wasm"])

def test_a_bench_selects_only_its_workspace(self):
self.assertEqual(list(self.plan(["js/core/bench/frames.ts"])["packages"]), ["@example/core"])
self.assertEqual(set(self.plan(["js/core/bench/frames.ts", "js/core/src/index.ts"])["packages"]),
{"@example/core", "@example/view", "docs"})

def test_leaf_does_not_select_its_dependencies(self):
self.assertEqual(list(self.plan(["doc/index.md"])["packages"]), ["docs"])

Expand Down
11 changes: 8 additions & 3 deletions sh/rs/select.sh
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,13 @@ select_packages() {
return
fi

local seeds edges selected
local seeds benches edges selected
# A crate's `benches/` builds only its own bench targets, which no other crate can
# depend on and the test build never compiles, so it selects that crate alone
# (for its lint) rather than every dependent's tests.
benches=$(sed -n 's|^rs/\([^/]*\)/benches/.*|\1|p' "$list")
seeds=$({
sed -n 's|^rs/\([^/]*\)/.*|\1|p' "$list"
sed -n '\|^rs/[^/]*/benches/|d; s|^rs/\([^/]*\)/.*|\1|p' "$list"
jq -r --rawfile list "$list" '($list | split("\n")) as $paths | .packages[]
| (.manifest_path | split("/")) as $p | select($p[-4] == "demo")
| select(any($paths[]; startswith("demo/\($p[-3])/\($p[-2])/"))) | .name' <<<"$metadata"
Expand All @@ -74,7 +78,8 @@ select_packages() {
} while (grew)
for (name in want) if (want[name]) print name
}
' <<<"$edges" | sort -u)
' <<<"$edges")
selected=$(printf '%s\n%s\n' "$selected" "$benches" | sed '/^$/d' | sort -u)

[[ -n "$selected" ]] || return 0

Expand Down
10 changes: 10 additions & 0 deletions sh/rs/select.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,16 @@ def test_demo_crate_follows_its_dependencies(self):
self.assertTrue(any("/demo/carla/bridge#moq-carla-bridge" in call for call in calls), calls)
self.assertEqual(self.selected("demo/carla/web/src/main.ts"), [])

def test_a_bench_selects_only_its_crate(self):
bench = "rs/moq-stats/benches/producer_tick.rs"
for action in ("lint", "test"):
with self.subTest(action=action):
calls = self.selected(bench, action=action)
self.assertTrue(any("#moq-stats@" in call or "/rs/moq-stats#" in call for call in calls), calls)
self.assertFalse(any("moq-relay" in call or "moq-cli" in call for call in calls), calls)
calls = self.selected(bench, "rs/moq-stats/src/lib.rs", action="test")
self.assertTrue(any("/rs/moq-relay#" in call for call in calls), calls)



class RecipeTest(unittest.TestCase):
Expand Down
Loading