From 69c946eab88453015fdfc9a2aad21fc2a7c69521 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Thu, 22 Oct 2026 17:58:47 -0700 Subject: [PATCH 01/16] chore(quest): claim quest/m1/stats-group-tracks.md Quest-Claim: 97337cfb7225eebf From ff8adb317baca76a5e744a690e392db987489e22 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Thu, 22 Oct 2026 18:26:53 -0700 Subject: [PATCH 02/16] feat(stats): publish groups as tracks of one node broadcast A new `Layout::Track` producer layout (relay `[stats] layout = "track"`) keeps the single `/node/` broadcast and carries each depth group as a compressed `/group.json` track, listed through TRACKS while the group has entries plus the linger. Group churn no longer costs an announce on every relay, a reader holds one subscription per group, and an oversize group ends only its own track. Each instance of a group track mints an epoch carried in its frames, so a group returning after its linger restarts its counters explicitly. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01EeyiEy5FWz3z44DYZVo7vU --- Cargo.lock | 1 + doc/bin/relay/config.md | 5 +- doc/concept/stats.md | 66 ++- quest/m1/README.md | 1 - quest/m1/stats-group-tracks.md | 51 -- rs/moq-relay/src/config.rs | 11 +- rs/moq-relay/src/settings.rs | 3 + rs/moq-relay/src/stats.rs | 50 +- rs/moq-stats/Cargo.toml | 1 + rs/moq-stats/benches/producer_tick.rs | 71 ++- rs/moq-stats/src/consume.rs | 35 +- rs/moq-stats/src/lib.rs | 68 ++- rs/moq-stats/src/produce.rs | 756 ++++++++++++++++++++++---- rs/moq-stats/src/produce_bench.rs | 15 +- 14 files changed, 959 insertions(+), 175 deletions(-) delete mode 100644 quest/m1/stats-group-tracks.md diff --git a/Cargo.lock b/Cargo.lock index b8cb020c2fd..42f53221597 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5283,6 +5283,7 @@ dependencies = [ "moq-net", "serde", "serde_json", + "serde_with", "thiserror 2.0.21", "tokio", "tracing", diff --git a/doc/bin/relay/config.md b/doc/bin/relay/config.md index b76ef9e601c..a9c3078fc49 100644 --- a/doc/bin/relay/config.md +++ b/doc/bin/relay/config.md @@ -337,6 +337,7 @@ prefix = ".stats" # Broadcasts appear under /node//node/ depth 0: one broadcast per node //node/ depth N: one broadcast per group per node +/node/ depth N, track layout: one broadcast per node, + with a /group.json track per group ``` - `` tells relays sharing a cluster apart. It may span several segments @@ -162,7 +164,8 @@ name with its label and a slash: tier `region/sjc` publishes `region/sjc/publisher.json`. Appending `.z` selects the [compressed](#compressed) encoding of the same track: `publisher.json.z`. -`totals.json` and the default tier's six per-tier tracks always exist. A named +`totals.json` and, outside the [track layout](#group-tracks), the default +tier's six per-tier tracks always exist. A named tier's are created on its first recorded traffic, but a subscriber may ask for them earlier: any name of the shape `[/]{publisher,subscriber,sessions}.json[.z]` is accepted and held open @@ -176,16 +179,39 @@ leaves. A prefix track works the same way for any broadcast prefix in the group: ask for `/traffic.json` (`traffic.json` for the empty prefix). A label or prefix outside the group, or any other name, is refused. +### Group tracks + +The track layout (`layout = "track"` on the relay) publishes every group of a +node on its one broadcast. Each group is a `/group.json` track +(`group.json` for the empty group) carrying all of the group's tiers, both +directions, and its sessions, so a reader holds one subscription per group. +The broadcast also carries `totals.json` for the whole node and serves presence +and prefix tracks for any label or prefix, but has no per-tier tracks. + +A reader discovers the live groups from the broadcast's track listing (TRACKS +in moq-lite 07, `SUBSCRIBE_TRACKS` in moq-transport draft 18 and later): a +group's track is listed from its first entry until it has lingered empty, then +unlisted and ended. The node's announcement never changes, so group churn +costs no announce traffic across a cluster. Over a hop that cannot carry a +listing (moq-lite 06 and older, or moq-transport before draft 18) the listing +is empty or refused; a reader that knows a group's name can still subscribe to +it while it is listed, and is refused otherwise. + +A group that returns after its linger is a new instance of its track: its +frames carry a new `epoch` and count from zero, and its group numbers continue +past the old instance's. A frame too large to cache ends only that group's +track. + ## Frames -Every frame is a JSON object mapping a key to an entry. A track with no -entries holds `{}`. +Every frame is a JSON object mapping a key to an entry, except a +[group](#group) track's. A track with no entries holds `{}`. The producer drains its counters every interval (one second by default) and writes only when a track's frame changed, so silence means nothing moved, not that the producer is gone. The track ends when the producer does, or at -depth 1 or more when its group's broadcast is unannounced; the group may -return later as a new broadcast. A frame too large to cache (32 MB) ends its +depth 1 or more when its group's broadcast is unannounced or its group track +unlisted; the group may return later as a new broadcast or track instance. A frame too large to cache (32 MB) ends its track with a `FrameTooLarge` error. The `.z` flavor meets the same limit by its uncompressed size, since a reader inflates the whole group. @@ -210,6 +236,29 @@ A lingering group's frames repeat its final totals. A reader that misses every frame from the group's last change until it unannounces loses that epoch's tail, so a meter that only reads totals under-counts by it. +### Group + +```json +{ + "epoch": "0192f4e6-7b8a-7c3d-9e1f-2a3b4c5d6e7f", + "tiers": { + "": { + "publisher": { ... }, "subscriber": { ... }, "sessions": { ... }, + "paths": { "acme/live": { "publisher": { ... }, "subscriber": { ... } } }, + "labels": { "acme": { ... } } + } + } +} +``` + +`epoch` is a UUIDv7 minted for each instance of the group's track. `tiers` +maps each tier the group recorded on since then: its top-level `publisher`, +`subscriber`, and `sessions` are the group's [totals](#totals), so a totals +reader parses them unchanged; `paths` holds each broadcast path's +[Traffic](#traffic) and `labels` each stats label's [Sessions](#sessions), with +the same lifetimes as the [per-tier tracks](#per-tier-tracks). A reader keeps +each epoch's counters apart, as it does for a broadcast's epochs. + ### Per-tier tracks A traffic entry appears from its first nonzero counter until the relay drops @@ -397,7 +446,12 @@ defaults a missing field to zero, so it can read an older relay. ## Encodings Both encodings carry identical frames; pick by bandwidth. Only the track name -says which one a track uses: the payload has no marker. +says which one a track uses: the payload has no marker. A group track has one +name and no `.z` sibling: it sets the track's +[compressed](/concept/moq-lite#subscriptions) flag, so a session that can +signal compression carries it DEFLATEd and every other reads an inflated copy. +Its frames read like the compressed encoding's after inflating: a snapshot +opening each group, then merge patches. ### Plain diff --git a/quest/m1/README.md b/quest/m1/README.md index 65910341b09..24cda476695 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -78,7 +78,6 @@ tracks, so per-group stats stop costing cluster announces. - [Announce cursors stay linear](/quest/m1/announce-cursor-cpu.md) - a withdrawal costs each announce cursor O(K log K), and a no-op route change allocates nothing - [Path length setting](/quest/m1/max-hops-setting.md) - operators cap hop paths near their mesh diameter instead of 32 - [Draining relays don't pull peers back](/quest/m1/drain-peer-redial.md) - cluster peers fail over instead of redialing a draining relay, and peer redials back off with jitter -- [Stats groups as tracks](/quest/m1/stats-group-tracks.md) - one stable node broadcast per stats producer, with each group a produced track listed by TRACKS - [Routing guide](/quest/m1/routing-guide.md) - concepts and practical outbound federation setup - [Controlled time](/quest/m1/time/README.md) - every crate and package reads time through moq-clock or @moq/clock, with regression tests for the changed contracts - [Own the QUIC stack](/quest/m1/quic/README.md) - quinn hard-forked in-tree as `moq-quic`, carrying BBR, reliable reset, hierarchical scheduling, peer limits, and endpoint sharding diff --git a/quest/m1/stats-group-tracks.md b/quest/m1/stats-group-tracks.md deleted file mode 100644 index 2358b97e055..00000000000 --- a/quest/m1/stats-group-tracks.md +++ /dev/null @@ -1,51 +0,0 @@ -# [M] Stats groups as tracks of one node broadcast - -## Goal - -A stats producer can publish one `/node/` broadcast per node -and carry each group as its own produced track, listed and withdrawn through -TRACKS, instead of announcing one broadcast per group per node. A reader -discovers a node's active groups with one Tracks stream and subscribes only -to the groups it wants. The node's announcement stays stable for the -producer's life, so group churn costs no announce traffic across a cluster. -The existing depth layouts keep their meaning. - -## Plan - -Decided 2026-10-22 (maintainer, from a downstream clustering audit): depth N -announces one broadcast per group per node, so every group that goes idle -past its linger and returns costs a broadcast end and start on every relay -in a cluster, and a reader following every group holds one subscription per -tier and role per group. TRACKS (lite-07-wip) already lists a broadcast's -produced tracks with live adds and removes, so groups become tracks. - -- A new producer layout, selected by config beside `depth`. The broadcast - path is depth 0's `/node/`. Each group is one produced track - carrying every tier, role, and the sessions counts for that group, so a - reader holds one subscription per group. Use the - [compressed track](/quest/m1/compress/README.md) flag rather than a `.z` - sibling if it has landed; otherwise keep today's encodings and say so. -- A group's track is listed when it first has traffic and removed after the - linger, as its broadcast is today. A name listed again means the same - content (TRACKS contract), so a reader must be able to tell continuation - from a counter reset without double counting cumulative values. Settle - and document which: counters kept across removal for the producer's life, - or a reset the frame makes explicit. -- Frame size limits apply per group track, so one oversize group no longer - ends every group's stats on that node. -- Discovery is TRACKS only. Over a hop that cannot carry TRACKS the listing - is empty; the docs say so rather than adding a second index. -- `totals.json` and the existing depth layouts are unchanged. Removing depth - N is a later decision, not this quest's. - -Tests: groups appear and disappear in a listing through a relay; a group -removed and re-listed reads continuous or explicitly reset counters, never -double; an oversize group leaves other groups readable; a reader of one -group never receives another's entries. - -Docs: `doc/concept/stats.md` gains the layout. Public API: a producer config -option. Wire: none beyond TRACKS. - -## Related - -- [Finalize moq-lite-07](/quest/m1/lite07-finalize.md) - TRACKS ships in the finalized wire diff --git a/rs/moq-relay/src/config.rs b/rs/moq-relay/src/config.rs index 5e6a1370668..b9f044224c0 100644 --- a/rs/moq-relay/src/config.rs +++ b/rs/moq-relay/src/config.rs @@ -745,7 +745,12 @@ root = ["ca.pem"] /// Bare defaults loaded from TOML survive when the CLI does not mention them. #[test] fn cli_does_not_clobber_toml_stats_enabled() { - let _env = EnvGuard::clear(&["MOQ_STATS_ENABLED", "MOQ_STATS_DEPTH", "MOQ_STATS_LINGER"]); + let _env = EnvGuard::clear(&[ + "MOQ_STATS_ENABLED", + "MOQ_STATS_DEPTH", + "MOQ_STATS_LAYOUT", + "MOQ_STATS_LINGER", + ]); let toml = r#" [stats] @@ -753,6 +758,7 @@ enabled = true interval = 5 node = "localhost" depth = 2 +layout = "track" linger = "2m" "#; let dir = std::env::temp_dir().join("moq-relay-config-test"); @@ -770,15 +776,18 @@ linger = "2m" assert_eq!(config.stats.interval, 5); assert_eq!(config.stats.node.as_deref(), Some("localhost")); assert_eq!(config.stats.depth, 2); + assert_eq!(config.stats.layout, crate::stats::Layout::Track); assert_eq!(config.stats.linger(), Some(std::time::Duration::from_secs(120))); let args = vec![ std::ffi::OsString::from("moq-relay"), std::ffi::OsString::from(&path), std::ffi::OsString::from("--stats-linger=30s"), + std::ffi::OsString::from("--stats-layout=broadcast"), ]; let config = Config::parse_and_merge(args).expect("config load"); assert_eq!(config.stats.linger(), Some(std::time::Duration::from_secs(30))); + assert_eq!(config.stats.layout, crate::stats::Layout::Broadcast); } /// Bare runtime defaults loaded from TOML survive when the CLI omits them. diff --git a/rs/moq-relay/src/settings.rs b/rs/moq-relay/src/settings.rs index 25ad57508b0..e39e5a6f0a9 100644 --- a/rs/moq-relay/src/settings.rs +++ b/rs/moq-relay/src/settings.rs @@ -216,6 +216,9 @@ struct Stats { #[usage(env = "MOQ_STATS_DEPTH", cli("--stats-depth"))] depth: Option, + #[usage(env = "MOQ_STATS_LAYOUT", cli("--stats-layout"))] + layout: Option, + #[usage(env = "MOQ_STATS_LINGER", cli("--stats-linger"))] linger: Option, } diff --git a/rs/moq-relay/src/stats.rs b/rs/moq-relay/src/stats.rs index 21f5f2fdaf8..760de4a54ae 100644 --- a/rs/moq-relay/src/stats.rs +++ b/rs/moq-relay/src/stats.rs @@ -19,7 +19,8 @@ use serde::{Deserialize, Serialize}; /// carries plain `.json` tracks of cumulative counters (per-tier totals, /// per-path traffic, per-root sessions, and per-root presence on request) plus compressed /// `.json.z` siblings; see `moq_stats` for the wire format and per-field -/// semantics. +/// semantics. [`Self::layout`] `track` keeps the one broadcast and publishes +/// each [`Self::depth`] group as a track on it instead. #[derive(usage_rs::Args, Clone, Debug, Deserialize, Serialize)] #[usage(unknown_flags = "error", args_override_self = false)] #[serde(default, deny_unknown_fields)] @@ -79,12 +80,29 @@ pub struct Config { )] pub depth: usize, + /// Where each `depth` group is published: `broadcast` (the default) + /// announces one broadcast per group per node; `track` keeps the single + /// `/node/` broadcast and lists each group as a + /// `/group.json` track on it, so group churn costs no announce + /// traffic. + /// + /// See [`moq_stats::produce::Layout`]. + #[usage( + long = "stats-layout", + env = "MOQ_STATS_LAYOUT", + value_enum, + default = "broadcast", + setting = "stats.layout" + )] + pub layout: Layout, + /// How long a group's stats broadcast stays announced after the group's /// last session and traffic leave, e.g. "5m" or "30s". Defaults to 5 /// minutes. A group that returns within it keeps its broadcast, so viewer /// churn doesn't unannounce and re-announce it across the mesh; one that /// returns after it announces under a new epoch. Only applies at `depth` 1 - /// or more. See [`moq_stats::produce::Config::linger`]. + /// or more; with `layout = "track"`, it is how long a group's track stays + /// listed. See [`moq_stats::produce::Config::linger`]. #[usage(skip)] #[serde(with = "crate::duration::serde_option")] pub linger: Option, @@ -102,6 +120,7 @@ impl Default for Config { interval: 1, node: None, depth: 0, + layout: Layout::Broadcast, linger: None, linger_arg: None, } @@ -129,17 +148,40 @@ impl Config { let interval = Duration::from_secs(self.interval.max(1)); let node = self.node.clone().map(PathOwned::from); let depth = self.depth; + let layout = self.layout; let linger = self.linger(); - tracing::info!(prefix, interval_secs = interval.as_secs(), node = ?node, depth, ?linger, "stats publishing enabled"); + tracing::info!(prefix, interval_secs = interval.as_secs(), node = ?node, depth, ?layout, ?linger, "stats publishing enabled"); let mut config = moq_stats::produce::Config::new() .with_origin(origin) .with_prefix(prefix) .with_interval(interval) .with_node(node) - .with_depth(depth); + .with_depth(depth) + .with_layout(layout.into()); if let Some(linger) = linger { config = config.with_linger(linger); } moq_stats::Producer::new(config) } } + +/// Where each stats group is published; see [`moq_stats::produce::Layout`]. +#[derive(usage_rs::ValueEnum, Clone, Copy, Debug, Default, PartialEq, Eq, Deserialize, Serialize)] +#[usage(ignore_case)] +#[serde(rename_all = "kebab-case")] +pub enum Layout { + /// One broadcast per group per node. + #[default] + Broadcast, + /// One broadcast per node, with each group a track on it. + Track, +} + +impl From for moq_stats::produce::Layout { + fn from(layout: Layout) -> Self { + match layout { + Layout::Broadcast => Self::Broadcast, + Layout::Track => Self::Track, + } + } +} diff --git a/rs/moq-stats/Cargo.toml b/rs/moq-stats/Cargo.toml index 45eca996a13..41edd88a4b2 100644 --- a/rs/moq-stats/Cargo.toml +++ b/rs/moq-stats/Cargo.toml @@ -24,6 +24,7 @@ moq-json = { workspace = true } moq-net = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } +serde_with = { workspace = true } thiserror = { workspace = true } tracing = { workspace = true } web-async = { workspace = true } diff --git a/rs/moq-stats/benches/producer_tick.rs b/rs/moq-stats/benches/producer_tick.rs index 0d18b6bc7d3..c6a28d554b6 100644 --- a/rs/moq-stats/benches/producer_tick.rs +++ b/rs/moq-stats/benches/producer_tick.rs @@ -17,6 +17,10 @@ //! tracks: each row walks its ancestors once a prefix is held, so the cost //! grows with rows times depth, plus each held track's encode. //! +//! `stats/groups` sweeps the track layout over held paths and the groups they +//! spread across, each group one track: the rows cost the same however they +//! split, so a slope in groups is per-track overhead. +//! //! Run `cargo bench -p moq-stats --features bench --bench producer_tick`. // Custom timings measure the real time the work took, not the ambient clock. @@ -94,14 +98,21 @@ struct Relay { } impl Relay { - /// `depth` is each path's segment count, at least 2. - fn build(paths: usize, tiers: usize, changed_percent: usize, depth: usize) -> Self { + /// `depth` is each path's segment count, at least 2. With `groups`, the + /// paths spread over that many first segments and the driver publishes each + /// as a group track. + fn build(paths: usize, tiers: usize, changed_percent: usize, depth: usize, groups: Option) -> Self { let registry = stats::Registry::new( stats::Config::new().with_exclude(moq_net::Pattern::subtree(".stats").expect("literal prefix")), ); let (origin, origin_driver) = origin::Producer::new(origin::Config::default()); let nested = "/s".repeat(depth - 2); - let path_names: Vec = (0..paths).map(|index| format!("room/{index:06}{nested}")).collect(); + let path_names: Vec = (0..paths) + .map(|index| match groups { + Some(groups) => format!("g{:04}/{index:06}{nested}", index % groups), + None => format!("room/{index:06}{nested}"), + }) + .collect(); let publishers: Vec<_> = path_names .iter() .map(|path| { @@ -128,7 +139,11 @@ impl Relay { registry.report(&mut report); assert_eq!(report.traffic.len(), paths * tiers, "one held entry per path and tier"); - let driver = Driver::new(registry.clone(), origin.clone()).expect("stats broadcast"); + let driver = match groups { + Some(_) => Driver::tracks(registry.clone(), origin.clone()), + None => Driver::new(registry.clone(), origin.clone()), + } + .expect("stats broadcast"); Self { registry, tiers: tier_labels, @@ -232,7 +247,7 @@ fn bench(c: &mut Criterion) { let mut relay = None; group.bench_function(id, |b| { let relay = relay.get_or_insert_with(|| { - let mut relay = Relay::build(paths, tiers, changed, 2); + let mut relay = Relay::build(paths, tiers, changed, 2, None); let row = sample(&mut relay); let micros = row.elapsed.as_secs_f64() * 1e6; println!( @@ -381,7 +396,7 @@ fn prefix(c: &mut Criterion) { let mut relay = None; group.bench_function(id, |b| { let relay = relay.get_or_insert_with(|| { - let mut relay = Relay::build(paths, 1, 10, depth); + let mut relay = Relay::build(paths, 1, 10, depth, None); black_box(relay.driver.tick()); relay.hold(held); let row = sample(&mut relay); @@ -409,5 +424,47 @@ fn prefix(c: &mut Criterion) { group.finish(); } -criterion_group!(benches, bench, presence, prefix); +const GROUP_PATHS: [usize; 2] = [1_000, 10_000]; + +/// Groups the paths spread across, each a track. +const GROUPS: [usize; 3] = [1, 100, 1_000]; + +/// What the track layout costs a tick, with a tenth of the paths moving. +fn groups(c: &mut Criterion) { + println!("paths groups allocs/tick us/tick"); + + let mut group = c.benchmark_group("stats/groups"); + group.sample_size(10); + group.warm_up_time(Duration::from_millis(200)); + group.measurement_time(Duration::from_secs(1)); + for &paths in &GROUP_PATHS { + for &groups in GROUPS.iter().filter(|&&groups| groups <= paths) { + let id = BenchmarkId::from_parameter(format!("{paths}p_{groups}g")); + group.throughput(Throughput::Elements(paths as u64)); + let mut relay = None; + group.bench_function(id, |b| { + let relay = relay.get_or_insert_with(|| { + let mut relay = Relay::build(paths, 1, 10, 2, Some(groups)); + let row = sample(&mut relay); + let micros = row.elapsed.as_secs_f64() * 1e6; + println!("{paths:>5} {groups:>6} {allocs:>11} {micros:>8.1}", allocs = row.allocs); + relay + }); + b.iter_custom(|iters| { + let mut total = Duration::ZERO; + for _ in 0..iters { + relay.touch(); + let start = Instant::now(); + black_box(relay.driver.tick()); + total += start.elapsed(); + } + total + }); + }); + } + } + group.finish(); +} + +criterion_group!(benches, bench, presence, prefix, groups); criterion_main!(benches); diff --git a/rs/moq-stats/src/consume.rs b/rs/moq-stats/src/consume.rs index fa515e62dbd..fcd3e2351b3 100644 --- a/rs/moq-stats/src/consume.rs +++ b/rs/moq-stats/src/consume.rs @@ -4,8 +4,8 @@ use moq_net::stats::{Role, Tier}; use moq_net::{AsPath, broadcast}; use crate::{ - PrefixFrame, PresenceFrame, Result, SessionsFrame, TotalsFrame, TrafficFrame, prefix_track, presence_track, - sessions_track, totals_track, traffic_track, + GroupFrame, PrefixFrame, PresenceFrame, Result, SessionsFrame, TotalsFrame, TrafficFrame, group_track, + prefix_track, presence_track, sessions_track, totals_track, traffic_track, }; /// Configuration for a [`Consumer`]. Construct with [`Config::new`] @@ -39,7 +39,10 @@ impl Config { /// [`Self::traffic`], [`Self::sessions`], [`Self::presence`], and /// [`Self::prefix`]. A named tier's tracks are held open with an empty frame /// until the tier records, and a label's presence track and a prefix's track -/// are produced only while subscribed. +/// are produced only while subscribed. In the +/// [track layout](crate::produce::Layout::Track), discover live groups from +/// the broadcast's [track listing](moq_net::broadcast::Consumer::tracks) with +/// [`crate::parse_group_track`] and read each with [`Self::group`]. pub struct Consumer { broadcast: broadcast::Consumer, config: Config, @@ -98,6 +101,17 @@ impl Consumer { }) } + /// Subscribe to group `group`'s track, awaiting the subscription handshake. + /// The track compresses itself, so [`Config::compression`] does not apply. + /// Fails with [`NotFound`](moq_net::Error::NotFound) unless the group is + /// listed. + pub async fn group(&self, group: impl AsPath) -> Result { + let track = self.broadcast.track(&group_track(group))?.subscribe(None).await?; + Ok(Group { + inner: moq_json::snapshot::Consumer::new(track, Default::default()), + }) + } + async fn subscribe(&self, name: &str) -> Result> { let track = self.broadcast.track(name)?.subscribe(None).await?; let mut config = moq_json::snapshot::consumer::Config::default(); @@ -174,6 +188,21 @@ impl Prefix { } } +/// A typed reader over one group's track; see [`Traffic`]. +pub struct Group { + inner: moq_json::snapshot::Consumer, +} + +impl Group { + /// The next frame, or `None` once the track ends (the producer went away + /// or the group lingered out of the listing). Fails with + /// [`FrameTooLarge`](moq_net::Error::FrameTooLarge) once the frame outgrows + /// the cache. + pub async fn next(&mut self) -> Result> { + Ok(self.inner.latest().await?.map(|value| value.value)) + } +} + #[cfg(test)] mod tests { /// A millisecond track: the frames written here carry timestamps. diff --git a/rs/moq-stats/src/lib.rs b/rs/moq-stats/src/lib.rs index d4a358fca7a..9de1b5cd8d3 100644 --- a/rs/moq-stats/src/lib.rs +++ b/rs/moq-stats/src/lib.rs @@ -31,8 +31,14 @@ //! broadcast prefix's `/traffic.json` track of [`Prefix`] counters. Every //! track has a `.z` sibling encoded with [`moq_json::snapshot`]; compute names //! with [`totals_track`], [`traffic_track`], [`sessions_track`], -//! [`presence_track`], and [`prefix_track`]. The full -//! contract (paths, tracks, both encodings, and counter semantics) is at +//! [`presence_track`], and [`prefix_track`]. +//! +//! The [track layout](produce::Layout::Track) instead keeps the one +//! `/node/` broadcast and carries each group as a compressed +//! `/group.json` track of [`GroupFrame`]s, listed through the +//! broadcast's track listing while the group is live; compute and parse names +//! with [`group_track`] and [`parse_group_track`]. The full contract (paths, +//! tracks, both encodings, and counter semantics) is at //! . pub mod aggregate; @@ -48,7 +54,7 @@ use std::collections::BTreeMap; /// can depend on this crate alone. pub use moq_net::stats::{Failure, Failures, Handle, Presence, Refusal, Refusals, Registry, Role, Tier, Traffic}; -use moq_net::{AsPath, Path, PathOwned}; +use moq_net::{AsPath, Epoch, Path, PathOwned}; /// One frame off a traffic track: cumulative counters keyed by broadcast path. pub type TrafficFrame = BTreeMap; @@ -129,6 +135,43 @@ pub struct Prefix { pub children: BTreeMap, } +/// One frame off a group track: one instance of the track, and its counters. +#[serde_with::serde_as] +#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[non_exhaustive] +pub struct GroupFrame { + /// Minted each time the group's track is created. Every counter in a frame + /// with a new epoch starts from zero, so a reader keeps each epoch's counters + /// apart rather than diffing across them. + #[serde_as(as = "serde_with::DisplayFromStr")] + pub epoch: Epoch, + /// Each tier's [`Group`] counters, keyed by tier label (`""` for the default + /// tier). + #[serde(default)] + pub tiers: BTreeMap, +} + +/// One tier's counters on a group track: the group's totals for the track's +/// epoch, then its live per-path traffic and per-label sessions. The totals sit +/// at the top level, so a [`Totals`] reader parses them unchanged. +#[derive(Debug, Default, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[serde(default)] +#[non_exhaustive] +pub struct Group { + /// Egress: everything the group's paths sent this epoch. + pub publisher: Traffic, + /// Ingress: everything the group's paths received this epoch. + pub subscriber: Traffic, + /// Sessions connected under the group's stats labels this epoch. + pub sessions: Presence, + /// Each live broadcast path's [`Usage`], as the per-tier traffic tracks + /// carry it. + pub paths: BTreeMap, + /// Each live stats label's [`Presence`], as the per-tier sessions track + /// carries it. + pub labels: BTreeMap, +} + /// Suffix appended to a plain track name for its compressed sibling. pub const COMPRESSED_SUFFIX: &str = ".z"; @@ -141,6 +184,9 @@ const PRESENCE: &str = "presence.json"; /// The last segment of a prefix track's plain name. const PREFIX: &str = "traffic.json"; +/// The last segment of a group track's name. +const GROUP: &str = "group.json"; + /// The traffic track name for a tier and role: `.json` at the prefix root /// on the default tier (`publisher.json` / `subscriber.json`), `/.json` /// on a named one, plus [`COMPRESSED_SUFFIX`] when `compressed`. @@ -187,6 +233,22 @@ pub fn prefix_track(path: impl AsPath, compressed: bool) -> String { keyed_track(path.as_path(), PREFIX, compressed) } +/// The group track name for a group: `/group.json`, or `group.json` for +/// the empty group. One name, compressed by the track itself, with no `.z` +/// sibling. Published only in the [track layout](produce::Layout::Track). +pub fn group_track(group: impl AsPath) -> String { + keyed_track(group.as_path(), GROUP, false) +} + +/// The group a group track name carries; `None` for any other name. The +/// inverse of [`group_track`], for reading a stats broadcast's track listing. +pub fn parse_group_track(name: &str) -> Option { + match parse_keyed_track(name, GROUP)? { + (group, false) => Some(group), + (_, true) => None, + } +} + /// `/`, or `` for the empty key, plus the compressed suffix. fn keyed_track(key: Path, kind: &str, compressed: bool) -> String { let mut name = match key.is_empty() { diff --git a/rs/moq-stats/src/produce.rs b/rs/moq-stats/src/produce.rs index 4f8d113c3c2..dee454ecf5b 100644 --- a/rs/moq-stats/src/produce.rs +++ b/rs/moq-stats/src/produce.rs @@ -55,14 +55,19 @@ pub struct Config { /// plus the [`Self::linger`], and a group returning after that announces /// under a new epoch counted from zero. At depth `0`, the single broadcast /// and its epoch last for the producer's life. + /// + /// [`Layout::Track`] keeps the single broadcast and makes each group a track. pub depth: usize, - /// How long a group broadcast stays announced after its group's last entry - /// leaves. Default 5 minutes. + /// Whether a group is a broadcast or a track. Default [`Layout::Broadcast`]. + pub layout: Layout, + /// How long a group broadcast (or, in [`Layout::Track`], a group track) + /// stays published after its group's last entry leaves. Default 5 minutes. /// /// A group that returns within the linger keeps its broadcast, so a viewer /// leaving and another arriving causes no unannounce and re-announce across /// the mesh. While it lingers empty, every track reads `{}`. Zero - /// unannounces on the first drain the group is empty. Unused at depth `0`. + /// unannounces on the first drain the group is empty. Unused at depth `0` + /// in [`Layout::Broadcast`]. /// /// A return after the linger is a new epoch, which an aggregating reader /// keeps as its own node for its grace, so a short linger on a flapping @@ -70,6 +75,24 @@ pub struct Config { pub linger: Duration, } +/// Where a [`Producer`] publishes each group of [`Config::depth`] leading +/// path segments. +#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)] +pub enum Layout { + /// One broadcast per group per node at `//node/`, each + /// with the totals and per-tier tracks for its group. + #[default] + Broadcast, + /// One `/node/` broadcast for the producer's life. Each group + /// is a compressed [`group_track`](crate::group_track) on it, listed while + /// the group has entries plus the [`Config::linger`], beside the node's + /// `totals.json` and its presence and prefix tracks. There are no per-tier + /// tracks. A group that returns after its linger is a new track instance + /// with a new [`GroupFrame::epoch`](crate::GroupFrame::epoch), counted from + /// zero. + Track, +} + impl Config { /// A config with default settings: no origin (no-op), `.stats` prefix, 1s /// interval, and no node suffix. Call [`Self::with_origin`] to actually @@ -81,6 +104,7 @@ impl Config { node: None, interval: Duration::from_secs(1), depth: 0, + layout: Layout::Broadcast, linger: DEFAULT_LINGER, } } @@ -116,8 +140,15 @@ impl Config { self } - /// Set how long an empty group broadcast stays announced (default 5 - /// minutes). See [`Self::linger`]. + /// Set whether a group is a broadcast or a track (default + /// [`Layout::Broadcast`]). See [`Self::layout`]. + pub fn with_layout(mut self, layout: Layout) -> Self { + self.layout = layout; + self + } + + /// Set how long an empty group stays published (default 5 minutes). See + /// [`Self::linger`]. pub fn with_linger(mut self, linger: Duration) -> Self { self.linger = linger; self @@ -187,6 +218,7 @@ impl Producer { node, interval, depth, + layout, linger, } = config; // An empty path after normalization is indistinguishable from "no node @@ -212,6 +244,7 @@ impl Producer { prefix, node, depth, + layout, linger, interval, }; @@ -238,6 +271,7 @@ struct Task { prefix: PathOwned, node: Option, depth: usize, + layout: Layout, linger: Duration, interval: Duration, } @@ -269,6 +303,24 @@ impl Task { fn node(&self) -> Option<&str> { self.node.as_ref().map(moq_net::Path::as_str) } + + /// How many leading segments pick a row's broadcast: the track layout has + /// only the node's. + fn broadcast_depth(&self) -> usize { + match self.layout { + Layout::Broadcast => self.depth, + Layout::Track => 0, + } + } + + /// How many leading segments pick a row's group track, `None` in the + /// broadcast layout, which has none. + fn track_depth(&self) -> Option { + match self.layout { + Layout::Broadcast => None, + Layout::Track => Some(self.depth), + } + } } /// The publish task's state, kept across drains so a steady-state drain @@ -293,7 +345,7 @@ impl Drain { /// refused it. fn new(task: Task) -> Option { let mut groups = HashMap::new(); - if task.depth == 0 { + if task.broadcast_depth() == 0 { let group = GroupPublisher::create(&task, &Path::empty())?; groups.insert(String::new(), group); } @@ -318,14 +370,15 @@ impl Drain { group.session_rows.clear(); } + let depth = self.task.broadcast_depth(); for (i, entry) in self.report.traffic.iter().enumerate() { - let key = group_key(entry.path.as_str(), self.task.depth); + let key = group_key(entry.path.as_str(), depth); if let Some(group) = Self::group(&self.task, &mut self.groups, &mut self.refused, key) { group.traffic_rows.push(i); } } for (i, entry) in self.report.sessions.iter().enumerate() { - let key = group_key(entry.label.as_str(), self.task.depth); + let key = group_key(entry.label.as_str(), depth); if let Some(group) = Self::group(&self.task, &mut self.groups, &mut self.refused, key) { group.session_rows.push(i); } @@ -366,7 +419,8 @@ impl Drain { fn publish(&mut self, now: Instant) { // At depth 0 the single broadcast stays for the producer's life; a // group broadcast lives while its group has entries, plus the linger. - let Task { depth, linger, .. } = self.task; + let depth = self.task.broadcast_depth(); + let linger = self.task.linger; for (_, group) in self .groups .extract_if(|_, group| depth > 0 && group.lingered(now, linger)) @@ -377,7 +431,7 @@ impl Drain { } for group in self.groups.values_mut() { - group.flush(self.tick); + group.flush(self.tick, now, linger); // Serve consumer requests for tracks no drain has created yet: a // tier's tracks appear lazily on its first traffic, and a label's @@ -454,9 +508,9 @@ impl Serialize for Frame { /// /// The shared allocator is why this wraps the encoder rather than a /// `moq_json::snapshot::Producer`, which numbers its own groups. -struct Snapshot { +struct Snapshot { track: track::Producer, - encoder: moq_json::snapshot::Encoder>, + encoder: moq_json::snapshot::Encoder, group: Option, sequence: Arc, /// The sequence of the newest group written, which a refresh has to pass. @@ -464,7 +518,7 @@ struct Snapshot { deltas: bool, } -impl Snapshot { +impl Snapshot { fn new(track: track::Producer, config: moq_json::snapshot::Config, sequence: Arc) -> Self { Self { track, @@ -487,7 +541,7 @@ impl Snapshot { /// [`moq_net::Error::FrameTooLarge`]: skipping it would leave readers on the /// previous snapshot with no sign it stopped tracking. A compressed frame /// meets the same limit by its uncompressed size, which the encoder refuses. - fn update(&mut self, value: &Frame) -> moq_json::Result> { + fn update(&mut self, value: &T) -> moq_json::Result> { if self.track.is_closed() { return Ok(None); } @@ -546,8 +600,8 @@ impl Snapshot { /// writing each frame as its own single-frame group; the compressed side uses /// merge-patch deltas inside a shared DEFLATE window. struct TrackPair { - plain: Snapshot, - compressed: Snapshot, + plain: Snapshot>, + compressed: Snapshot>, /// This drain's entries, published and cleared by [`Self::publish`]. frame: Frame, /// Last plain payload written, kept so a later unchanged tick can still @@ -906,6 +960,14 @@ struct GroupPublisher { session_rows: Vec, /// When the group's last entry left; `None` while it has entries. empty_since: Option, + /// Whether the broadcast carries per-tier tracks; the track layout's group + /// tracks carry their entries instead. + tiers: bool, + /// How many leading segments pick a row's group track; `None` outside the + /// track layout. + track_depth: Option, + /// Each live group's track, keyed by group. + group_tracks: HashMap, } /// The plain track names one tier's entries land on. @@ -1108,6 +1170,156 @@ impl PrefixTrack { } } +/// One group's track in the [`Layout::Track`] layout: listed from the group's +/// first entry until it lingers empty, then finished. A return after that is a +/// new instance under a new epoch, counted from zero, so it holds no state for +/// groups that left. +struct GroupTrack { + /// The track name, for logs. + name: String, + snapshot: Snapshot, + /// This drain's frame, refilled in place every drain. + body: GroupBody, + /// Whether this drain carried a row for the group. + live: bool, + /// When the group's last entry left; `None` while it has entries. + empty_since: Option, +} + +/// A group track's frame, byte-identical to a [`GroupFrame`](crate::GroupFrame) +/// once sorted. +#[serde_with::serde_as] +#[derive(Serialize)] +struct GroupBody { + #[serde_as(as = "serde_with::DisplayFromStr")] + epoch: Epoch, + tiers: Frame, +} + +/// One tier's entry on a group track; serializes as a [`Group`](crate::Group). +#[derive(Default, Serialize)] +struct GroupTier { + /// The closing readouts of entries that ended this epoch. + #[serde(skip)] + folded: Totals, + /// `folded` plus this drain's live readouts. + #[serde(flatten)] + totals: Totals, + paths: Frame, + labels: Frame, +} + +/// Group tracks are timed like every stats track, and keep merge-patch deltas +/// inside the track's own DEFLATE window, so a joiner asks for a fresh group +/// once the deltas outweigh eight snapshots. +fn group_info() -> track::Info { + info().with_compress(true).with_refresh(8) +} + +impl GroupTrack { + fn create(broadcast: &broadcast::Producer, group: &str, sequence: Arc) -> Result { + let name = crate::group_track(group); + let track = broadcast.create_track(name.as_str(), group_info())?; + Ok(Self { + name, + snapshot: Snapshot::new(track, moq_json::snapshot::Config::default(), sequence), + body: GroupBody { + // A new instance of the name: readers keep its counters apart from + // an earlier instance's. + epoch: Epoch::mint(), + tiers: Frame::default(), + }, + live: false, + empty_since: None, + }) + } + + /// Start a drain: every tier holds its folded totals and no live entries. + fn reset(&mut self) { + self.live = false; + for (_, tier) in &mut self.body.tiers.entries { + tier.totals = tier.folded; + tier.paths.entries.clear(); + tier.labels.entries.clear(); + } + } + + /// `tier`'s entry, inserted on first sight, marking the group live. + fn tier(&mut self, tier: &Tier) -> &mut GroupTier { + self.live = true; + let entries = &mut self.body.tiers.entries; + let index = match entries.iter().position(|(label, _)| label == tier.label()) { + Some(index) => index, + None => { + entries.push((tier.label().clone(), GroupTier::default())); + entries.len() - 1 + } + }; + &mut entries[index].1 + } + + /// Add one traffic row, listing its path when either side has moved. + fn add_traffic(&mut self, entry: &TrafficEntry, listed: bool) { + let tier = self.tier(&entry.tier); + let add = |totals: &mut Totals| { + totals.publisher.add(entry.publisher); + totals.subscriber.add(entry.subscriber); + }; + add(&mut tier.totals); + if entry.closed { + add(&mut tier.folded); + } + if listed { + tier.paths.entries.push(( + entry.path.clone(), + Usage { + publisher: entry.publisher, + subscriber: entry.subscriber, + }, + )); + } + } + + /// Add one sessions row; every row is listed, as on the sessions track. + fn add_sessions(&mut self, label: &PathOwned, tier: &Tier, presence: Presence, closed: bool) { + let tier = self.tier(tier); + tier.totals.sessions.add(presence); + if closed { + tier.folded.sessions.add(presence); + } + tier.labels.entries.push((label.clone(), presence)); + } + + /// Whether this drain left the group empty for at least `linger`. + fn lingered(&mut self, now: Instant, linger: Duration) -> bool { + if self.live { + self.empty_since = None; + return false; + } + let since = *self.empty_since.get_or_insert(now); + now.saturating_duration_since(since) >= linger + } + + /// Write this drain's frame; moq-json skips an unchanged one. A frame too + /// large to cache ends this group's track alone. + fn publish(&mut self) { + let tiers = &mut self.body.tiers.entries; + tiers.sort_unstable_by(|a, b| a.0.cmp(&b.0)); + for (_, tier) in tiers.iter_mut() { + tier.paths.entries.sort_unstable_by(|a, b| a.0.cmp(&b.0)); + tier.labels.entries.sort_unstable_by(|a, b| a.0.cmp(&b.0)); + } + if let Err(err) = self.snapshot.update(&self.body) { + tracing::warn!(?err, name = self.name, "stats: failed to write group frame"); + } + } + + /// Finish the track, a deliberate end that also unlists it. + fn finish(mut self) { + let _ = self.snapshot.finish(); + } +} + /// Each prefix of `path` from the root down to `path` itself, paired with the /// next segment below it (`None` at `path`). fn ancestors(path: &str) -> impl Iterator)> { @@ -1143,10 +1355,11 @@ impl GroupPublisher { let sequence = Arc::new(AtomicU64::new(0)); let mut traffic = TrackFamily::new(sequence.clone()); let mut sessions = TrackFamily::new(sequence.clone()); + let tiers = task.layout == Layout::Broadcast; // The totals and the default tier's tracks always exist, even while idle. let tier = Tier::default(); - for role in [Role::Publisher, Role::Subscriber] { + for role in [Role::Publisher, Role::Subscriber].into_iter().filter(|_| tiers) { let name = traffic_track(&tier, role, false); match TrackPair::create(&broadcast, &name, sequence.clone()) { Ok(pair) => { @@ -1159,13 +1372,15 @@ impl GroupPublisher { } } let name = sessions_track(&tier, false); - match TrackPair::create(&broadcast, &name, sequence.clone()) { - Ok(pair) => { - sessions.tracks.insert(name, pair); - } - Err(err) => { - tracing::warn!(?err, name, "stats: failed to create track"); - return None; + if tiers { + match TrackPair::create(&broadcast, &name, sequence.clone()) { + Ok(pair) => { + sessions.tracks.insert(name, pair); + } + Err(err) => { + tracing::warn!(?err, name, "stats: failed to create track"); + return None; + } } } let name = totals_track(false); @@ -1197,6 +1412,9 @@ impl GroupPublisher { traffic_rows: Vec::new(), session_rows: Vec::new(), empty_since: None, + tiers, + track_depth: task.track_depth(), + group_tracks: HashMap::new(), }) } @@ -1226,6 +1444,9 @@ impl GroupPublisher { prefixes, traffic_rows, session_rows, + tiers, + track_depth, + group_tracks, .. } = self; @@ -1266,23 +1487,42 @@ impl GroupPublisher { } } + for track in group_tracks.values_mut() { + track.reset(); + } + for &i in traffic_rows.iter() { let entry = &report.traffic[i]; - let names = names - .entry(entry.tier.clone()) - .or_insert_with(|| TierNames::new(&entry.tier)); let slots = local .entry(entry.path.clone()) .or_default() .entry(entry.tier.clone()) .or_default(); slots.seen = tick; + let names = match tiers { + true => Some( + &*names + .entry(entry.tier.clone()) + .or_insert_with(|| TierNames::new(&entry.tier)), + ), + false => None, + }; process_slot(entry.publisher, &mut slots.publisher, |snap| { - traffic.push(broadcast, requested, &names.publisher, entry.path.clone(), snap); + if let Some(names) = names { + traffic.push(broadcast, requested, &names.publisher, entry.path.clone(), snap); + } }); process_slot(entry.subscriber, &mut slots.subscriber, |snap| { - traffic.push(broadcast, requested, &names.subscriber, entry.path.clone(), snap); + if let Some(names) = names { + traffic.push(broadcast, requested, &names.subscriber, entry.path.clone(), snap); + } }); + if let Some(depth) = *track_depth { + let key = group_key(entry.path.as_str(), depth); + if let Some(track) = Self::group_track(broadcast, group_tracks, &traffic.sequence, key) { + track.add_traffic(entry, slots.publisher.moved || slots.subscriber.moved); + } + } } // The registry reports a label only while a session or request holds its @@ -1290,25 +1530,61 @@ impl GroupPublisher { // held by an idle request must not drop out and return looking reset. for &i in session_rows.iter() { let entry = &report.sessions[i]; - let names = names - .entry(entry.tier.clone()) - .or_insert_with(|| TierNames::new(&entry.tier)); - sessions.push( - broadcast, - requested, - &names.sessions, - entry.label.clone(), - entry.presence, - ); + if *tiers { + let names = names + .entry(entry.tier.clone()) + .or_insert_with(|| TierNames::new(&entry.tier)); + sessions.push( + broadcast, + requested, + &names.sessions, + entry.label.clone(), + entry.presence, + ); + } + if let Some(depth) = *track_depth { + let key = group_key(entry.label.as_str(), depth); + if let Some(track) = Self::group_track(broadcast, group_tracks, &traffic.sequence, key) { + track.add_sessions(&entry.label, &entry.tier, entry.presence, entry.closed); + } + } + } + } + + /// Get or create the track for group `key`, `None` if the broadcast refused it. + fn group_track<'a>( + broadcast: &broadcast::Producer, + tracks: &'a mut HashMap, + sequence: &Arc, + key: &str, + ) -> Option<&'a mut GroupTrack> { + if !tracks.contains_key(key) { + match GroupTrack::create(broadcast, key, sequence.clone()) { + Ok(track) => { + tracks.insert(key.to_string(), track); + } + Err(err) => { + tracing::warn!(?err, group = key, "stats: failed to create group track"); + return None; + } + } } + tracks.get_mut(key) } /// Publish the pending frames, then drop change-detection state for /// entries this drain did not carry (the registry pruned them). - fn flush(&mut self, tick: u64) { + fn flush(&mut self, tick: u64, now: Instant, linger: Duration) { self.traffic.flush(); self.sessions.flush(); + for (_, track) in self.group_tracks.extract_if(|_, track| track.lingered(now, linger)) { + track.finish(); + } + for track in self.group_tracks.values_mut() { + track.publish(); + } + self.totals.sums.fill(&mut self.totals.pair.frame); self.totals.pair.publish(crate::TOTALS); for track in self.presence.values_mut() { @@ -1390,7 +1666,9 @@ impl GroupPublisher { } continue; } - let Some(shape) = requested_track_shape(request.name()) else { + // The track layout has no per-tier tracks, and serves a group's track + // only while it is listed. + let Some(shape) = requested_track_shape(request.name()).filter(|_| self.tiers) else { request.reject(moq_net::Error::NotFound); continue; }; @@ -1491,6 +1769,9 @@ impl GroupPublisher { for track in self.prefixes.values_mut() { track.pair.finish(); } + for (_, track) in self.group_tracks.drain() { + track.finish(); + } self.broadcast.close(); } } @@ -1614,7 +1895,7 @@ mod tests { fn a_joiner_over_the_ratio_gets_a_snapshot_next_tick() { let broadcast = moq_net::broadcast::Info::new().produce(); let track = broadcast.create_track("t.json.z", compressed_info()).unwrap(); - let mut snapshot = Snapshot::::new( + let mut snapshot = Snapshot::>::new( track.clone(), moq_json::snapshot::Config::default(), Arc::new(AtomicU64::new(0)), @@ -1686,56 +1967,67 @@ mod tests { } } + /// A relay origin, and the cache pool its copies are billed to. + fn relay() -> (origin::Producer, moq_net::cache::Pool) { + let config = origin::Config::default(); + let pool = config.pool.clone(); + let (origin, driver) = origin::Producer::new(config); + tokio::spawn(moq_net::time::run(driver)); + (origin, pool) + } + + /// Drive `drain` until `future` resolves. + async fn pumping(drain: &mut Drain, future: impl std::future::Future) -> T { + pumping_at(drain, Instant::now(), future).await + } + + /// Drive `drain` at `now` until `future` resolves. + async fn pumping_at(drain: &mut Drain, now: Instant, future: impl std::future::Future) -> T { + let mut future = std::pin::pin!(future); + for _ in 0..1000 { + drain.collect(); + drain.publish(now); + if let Some(result) = futures::FutureExt::now_or_never(future.as_mut()) { + return result; + } + tokio::task::yield_now().await; + } + panic!("protocol request did not settle while pumping the stats producer"); + } + + /// Serve `upstream` to `downstream` over an in-memory `version` session. + async fn link( + upstream: &origin::Producer, + downstream: &origin::Producer, + version: moq_net::Version, + ) -> [moq_net::Session; 2] { + let mut config = moq_net::transport::memory::Config::default(); + config.protocol = Some(version.to_string()); + let (client_transport, server_transport) = moq_net::transport::memory::pair(config); + let client = moq_net::Client::new() + .with_versions(version.into()) + .with_subscriber(downstream.clone().peer()); + let server = moq_net::Server::new() + .with_versions(version.into()) + .with_publisher(upstream.consume().with_hidden(true)); + let client = async { + let (session, driver) = client.connect(moq_clock::now(), client_transport).await.unwrap(); + tokio::spawn(moq_net::time::run(driver)); + session + }; + let server = async { + let (session, driver) = server.accept(moq_clock::now(), server_transport).await.unwrap(); + tokio::spawn(moq_net::time::run(driver)); + session + }; + let (a, b) = tokio::join!(client, server); + [a, b] + } + /// Real protocol relays keep their idle track cache while upstream presence /// is reclaimed. A re-hold must refresh that cache before exposing any frame. #[tokio::test(start_paused = true)] async fn reheld_wire_presence_refreshes_two_relay_caches() { - fn relay() -> (origin::Producer, moq_net::cache::Pool) { - let config = origin::Config::default(); - let pool = config.pool.clone(); - let (origin, driver) = origin::Producer::new(config); - tokio::spawn(moq_net::time::run(driver)); - (origin, pool) - } - async fn pumping(drain: &mut Drain, future: impl std::future::Future) -> T { - let mut future = std::pin::pin!(future); - for _ in 0..1000 { - drain.collect(); - drain.publish(Instant::now()); - if let Some(result) = futures::FutureExt::now_or_never(future.as_mut()) { - return result; - } - tokio::task::yield_now().await; - } - panic!("protocol request did not settle while pumping the stats producer"); - } - async fn link( - upstream: &origin::Producer, - downstream: &origin::Producer, - version: moq_net::Version, - ) -> [moq_net::Session; 2] { - let mut config = moq_net::transport::memory::Config::default(); - config.protocol = Some(version.to_string()); - let (client_transport, server_transport) = moq_net::transport::memory::pair(config); - let client = moq_net::Client::new() - .with_versions(version.into()) - .with_subscriber(downstream.clone().peer()); - let server = moq_net::Server::new() - .with_versions(version.into()) - .with_publisher(upstream.consume().with_hidden(true)); - let client = async { - let (session, driver) = client.connect(moq_clock::now(), client_transport).await.unwrap(); - tokio::spawn(moq_net::time::run(driver)); - session - }; - let server = async { - let (session, driver) = server.accept(moq_clock::now(), server_transport).await.unwrap(); - tokio::spawn(moq_net::time::run(driver)); - session - }; - let (a, b) = tokio::join!(client, server); - [a, b] - } for version in ["moq-lite-06", "moq-lite-07-wip"].map(|value| value.parse().unwrap()) { for compressed in [false, true] { let registry = Registry::new(moq_net::stats::Config::new()); @@ -1746,6 +2038,7 @@ mod tests { prefix: ".stats".into(), node: None, depth: 0, + layout: Layout::Broadcast, linger: DEFAULT_LINGER, interval: Duration::from_secs(1), }) @@ -2741,6 +3034,7 @@ mod tests { prefix: PathOwned::from(".stats"), node: None, depth: 0, + layout: Layout::Broadcast, linger: DEFAULT_LINGER, interval: Duration::from_secs(1), }) @@ -3011,6 +3305,7 @@ mod tests { prefix: PathOwned::from(".stats"), node: None, depth: 0, + layout: Layout::Broadcast, linger: DEFAULT_LINGER, interval: Duration::from_secs(1), }) @@ -3415,7 +3710,16 @@ mod tests { // into the pending frames allocates nothing, however many broadcasts // and tiers there are, or whether a prefix is held. Encoding the frames // (moq-json) is out of scope. - for (depth, hold) in [(0, false), (1, false), (0, true), (1, true)] { + let cases = [(0, false), (1, false), (0, true), (1, true)]; + for (layout, (depth, hold)) in [Layout::Broadcast, Layout::Track] + .into_iter() + .flat_map(|layout| cases.map(|case| (layout, case))) + { + // The broadcast that holds `room0/cam`'s prefix track. + let key = match layout { + Layout::Broadcast => group_key("room0/cam", depth), + Layout::Track => "", + }; for (broadcasts, tiers) in [(1, 1), (16, 1), (1, 4), (16, 4)] { let registry = Registry::new(moq_net::stats::Config::new()); let mut feeds = Vec::new(); @@ -3435,6 +3739,7 @@ mod tests { prefix: PathOwned::from(".stats"), node: None, depth, + layout, linger: DEFAULT_LINGER, interval: Duration::from_secs(1), }) @@ -3448,7 +3753,7 @@ mod tests { // Hold a prefix with a child per tier, as a reader would. let _held = hold.then(|| { let tick = drain.tick; - let group = drain.groups.get_mut(group_key("room0/cam", depth)).expect("group"); + let group = drain.groups.get_mut(key).expect("group"); group.adopt_prefix(&drain.report, PathOwned::from("room0"), PendingPair::default(), tick); group.prefixes["room0"].pair.plain.track.subscribe(None) }); @@ -3468,16 +3773,275 @@ mod tests { assert!(pending > 0, "the drain carried entries"); assert_eq!( allocs, 0, - "depth {depth}, {broadcasts} broadcasts x {tiers} tiers, held prefix {hold}" + "{layout:?} depth {depth}, {broadcasts} broadcasts x {tiers} tiers, held prefix {hold}" ); if hold { - assert_eq!( - drain.groups[group_key("room0/cam", depth)].prefixes.len(), - 1, - "still held" - ); + assert_eq!(drain.groups[key].prefixes.len(), 1, "still held"); } } } } + + /// A producer publishing each first path segment as a track of `.stats/node/sjc`. + fn track_producer(linger: Duration) -> (Producer, origin::Producer) { + let origin = produce_origin(); + let producer = Producer::new( + Config::new() + .with_origin(origin.clone()) + .with_node(PathOwned::from("sjc")) + .with_depth(1) + .with_layout(Layout::Track) + .with_linger(linger), + ); + (producer, origin) + } + + /// Read `reader` until its track ends, returning how it ended and the last + /// frame before that. + async fn drain_group(reader: &mut crate::consume::Group) -> (crate::Result<()>, Option) { + let mut last = None; + loop { + match reader.next().await { + Ok(Some(frame)) => last = Some(frame), + Ok(None) => return (Ok(()), last), + Err(err) => return (Err(err), last), + } + } + } + + /// The next listing change to a group track, as `(group, listed)`, skipping + /// every other track. + async fn next_listed(drain: &mut Drain, now: Instant, tracks: &mut broadcast::Tracks) -> (String, bool) { + loop { + let event = pumping_at(drain, now, tracks.next()) + .await + .expect("listing") + .expect("live"); + let (name, listed) = match event { + broadcast::TracksEvent::Added { name, .. } => (name, true), + broadcast::TracksEvent::Removed { name } => (name, false), + broadcast::TracksEvent::Synced => continue, + }; + if let Some(group) = crate::parse_group_track(&name) { + return (group.as_str().to_string(), listed); + } + } + } + + /// Groups are listed through a relay as they gain entries, stay listed for + /// the linger once empty, and are unlisted after it, all on one broadcast + /// whose announcement never changes. + #[tokio::test(start_paused = true)] + async fn group_tracks_list_and_unlist_through_a_relay() { + let registry = Registry::new(moq_net::stats::Config::new()); + let origin = produce_origin(); + let mut drain = Drain::new(Task { + registry: registry.clone(), + origin: origin.clone(), + prefix: ".stats".into(), + node: None, + depth: 1, + layout: Layout::Track, + linger: Duration::from_secs(10), + interval: Duration::from_secs(1), + }) + .unwrap(); + let (edge, _cache) = relay(); + let _link = link(&origin, &edge, "moq-lite-07-wip".parse().unwrap()).await; + let start = Instant::now(); + + let acme = registry.tier(Tier::default()).session("acme/live"); + drain.collect(); + drain.publish(start); + let consumer = edge.consume().with_hidden(true); + let mut announced = consumer.announced(); + let (route, active) = next_update(&mut announced).await.unwrap(); + assert!(active); + assert_eq!(route.prefix.as_str(), ".stats/node"); + let broadcast = consumer.request_broadcast(".stats/node", None).await.unwrap(); + let mut tracks = broadcast.tracks().unwrap(); + assert_eq!(next_listed(&mut drain, start, &mut tracks).await, ("acme".into(), true)); + + let beta = registry.tier(Tier::default()).session("beta"); + assert_eq!(next_listed(&mut drain, start, &mut tracks).await, ("beta".into(), true)); + + // Empty, acme lingers: still listed, its totals keeping the ended session. + drop(acme); + let lingering = start + Duration::from_secs(9); + let stats = crate::Consumer::new(broadcast.clone(), Default::default()); + let mut acme = pumping_at(&mut drain, lingering, stats.group("acme")).await.unwrap(); + let mut frame = pumping_at(&mut drain, lingering, acme.next()).await.unwrap().unwrap(); + while !frame.tiers[""].labels.is_empty() { + frame = pumping_at(&mut drain, lingering, acme.next()).await.unwrap().unwrap(); + } + assert_eq!(frame.tiers[""].sessions.sessions_ended, 1); + + // Past the linger, acme alone is unlisted and its subscription ends. + let lingered = start + Duration::from_secs(20); + assert_eq!( + next_listed(&mut drain, lingered, &mut tracks).await, + ("acme".into(), false) + ); + assert!(pumping_at(&mut drain, lingered, acme.next()).await.unwrap().is_none()); + let mut beta_reader = pumping_at(&mut drain, lingered, stats.group("beta")).await.unwrap(); + let frame = pumping_at(&mut drain, lingered, beta_reader.next()) + .await + .unwrap() + .unwrap(); + assert_eq!(frame.tiers[""].labels.keys().collect::>(), ["beta"]); + + // A return is listed again on the same broadcast, with no announce. + let _acme = registry.tier(Tier::default()).session("acme/live"); + assert_eq!( + next_listed(&mut drain, lingered, &mut tracks).await, + ("acme".into(), true) + ); + drop(beta); + assert!( + futures::FutureExt::now_or_never(next_update(&mut announced)).is_none(), + "group churn costs no announce" + ); + } + + /// A group that returns after its linger is a new track instance: a new + /// epoch counted from zero, so a reader summing each epoch never counts the + /// old one twice, and group numbers that continue past the old instance's. + #[tokio::test(start_paused = true)] + async fn a_relisted_group_starts_a_new_epoch_from_zero() { + let (producer, origin) = track_producer(Duration::ZERO); + let registry = producer.registry(); + let (_, broadcast) = announced(&origin).await; + let stats = crate::Consumer::new(broadcast.clone(), Default::default()); + let sequence = async |broadcast: &moq_net::broadcast::Consumer| { + let mut track = subscribe(broadcast, "acme/group.json").await; + track.next_group().await.unwrap().unwrap().sequence + }; + + let session = registry.tier(Tier::default()).session("acme/live"); + drive_tick().await; + let mut reader = stats.group("acme").await.expect("listed"); + let first = reader.next().await.unwrap().unwrap(); + assert_eq!(first.tiers[""].sessions.sessions_started, 1); + let old_sequence = sequence(&broadcast).await; + + drop(session); + for _ in 0..3 { + drive_tick().await; + } + let (end, last) = drain_group(&mut reader).await; + end.expect("the group's track finishes cleanly"); + let last = last.unwrap_or(first.clone()); + assert_eq!(last.epoch, first.epoch); + assert_eq!(last.tiers[""].sessions.sessions_ended, 1); + + let _session = registry.tier(Tier::default()).session("acme/live"); + drive_tick().await; + let second = stats + .group("acme") + .await + .expect("listed again") + .next() + .await + .unwrap() + .unwrap(); + assert!(second.epoch > first.epoch, "a returning group mints a new epoch"); + assert_eq!( + second.tiers[""].sessions.sessions_started, 1, + "a new epoch counts from zero" + ); + assert!( + sequence(&broadcast).await > old_sequence, + "group numbers continue past the old instance" + ); + assert_eq!( + default_totals(&broadcast).await.sessions.sessions_started, + 2, + "the node totals carry both epochs" + ); + } + + /// One group's frame outgrowing the cache ends that group's track alone. + /// + /// Read off the broadcast itself: through an origin, the subscriber's resume + /// re-requests the ended track and reads the refusal instead of the cause. + #[tokio::test(start_paused = true)] + async fn an_oversize_group_leaves_other_groups_readable() { + let registry = Registry::new(moq_net::stats::Config::new()); + let mut drain = Drain::new(Task { + registry: registry.clone(), + origin: produce_origin(), + prefix: ".stats".into(), + node: None, + depth: 1, + layout: Layout::Track, + linger: Duration::ZERO, + interval: Duration::from_secs(1), + }) + .unwrap(); + let stats = crate::Consumer::new(drain.groups[""].broadcast.consume(), Default::default()); + + let _acme = registry.tier(Tier::default()).session("acme"); + let _big = registry.tier(Tier::default()).session("big"); + let mut acme = pumping(&mut drain, stats.group("acme")).await.unwrap(); + let mut big = pumping(&mut drain, stats.group("big")).await.unwrap(); + assert!(pumping(&mut drain, acme.next()).await.unwrap().is_some()); + assert!(pumping(&mut drain, big.next()).await.unwrap().is_some()); + + let huge = format!("big/{}", "a".repeat(moq_net::group::MAX_CACHE_BYTES as usize)); + let _huge = registry.tier(Tier::default()).session(huge.as_str()); + let (end, _) = pumping(&mut drain, drain_group(&mut big)).await; + assert!( + matches!( + end, + Err(crate::Error::Json(moq_json::Error::Net(moq_net::Error::FrameTooLarge))) + ), + "{end:?}" + ); + + let _more = registry.tier(Tier::default()).session("acme/more"); + let mut frame = pumping(&mut drain, acme.next()).await.unwrap().unwrap(); + while frame.tiers[""].labels.len() < 2 { + frame = pumping(&mut drain, acme.next()).await.unwrap().unwrap(); + } + assert_eq!(frame.tiers[""].labels.keys().collect::>(), ["acme", "acme/more"]); + } + + /// A group's track carries only its own paths and labels, and the track + /// layout has no per-tier tracks. + #[tokio::test(start_paused = true)] + async fn a_group_track_carries_only_its_own_entries() { + let (producer, origin) = track_producer(DEFAULT_LINGER); + let registry = producer.registry(); + let (_, broadcast) = announced(&origin).await; + let stats = crate::Consumer::new(broadcast.clone(), Default::default()); + + // Each feed holds a session labelled `feed`, its own group. + let _acme = feed(registry, Tier::default(), "acme/live", true, 1, 100).await; + let _beta = feed(registry, Tier::default(), "beta/live", true, 1, 50).await; + drive_tick().await; + + let frame = stats.group("acme").await.unwrap().next().await.unwrap().unwrap(); + let acme = &frame.tiers[""]; + assert_eq!(acme.paths.keys().collect::>(), ["acme/live"]); + assert_eq!(acme.paths["acme/live"].publisher.bytes, 100); + assert_eq!(acme.publisher.bytes, 100, "the totals are the group's alone"); + assert!(acme.labels.is_empty()); + + let frame = stats.group("beta").await.unwrap().next().await.unwrap().unwrap(); + assert_eq!(frame.tiers[""].paths.keys().collect::>(), ["beta/live"]); + assert_eq!(frame.tiers[""].publisher.bytes, 50); + + let frame = stats.group("feed").await.unwrap().next().await.unwrap().unwrap(); + assert!(frame.tiers[""].paths.is_empty()); + assert_eq!(frame.tiers[""].labels["feed"].sessions_started, 2); + + assert_eq!( + default_totals(&broadcast).await.publisher.bytes, + 150, + "the node totals sum every group" + ); + let refused = broadcast.track("publisher.json").unwrap().subscribe(None); + let (refused, ()) = tokio::join!(refused, drive_tick()); + assert!(refused.is_err(), "no per-tier tracks in the track layout"); + } } diff --git a/rs/moq-stats/src/produce_bench.rs b/rs/moq-stats/src/produce_bench.rs index 84418f2e631..abf33eaa8dc 100644 --- a/rs/moq-stats/src/produce_bench.rs +++ b/rs/moq-stats/src/produce_bench.rs @@ -5,7 +5,7 @@ use std::time::Duration; use moq_net::{PathOwned, origin, stats::Registry, track}; -use super::{Drain, PendingPair, Task}; +use super::{Drain, Layout, PendingPair, Task}; /// Plain and compressed publisher-frame sizes from one tick, in bytes. /// @@ -33,12 +33,23 @@ impl Driver { /// [`crate::Producer::new`] does at depth 0. `None` if the origin refuses /// the broadcast. pub fn new(registry: Registry, origin: origin::Producer) -> Option { + Self::build(registry, origin, 0, Layout::Broadcast) + } + + /// [`Self::new`] in the track layout at depth 1: each first path segment + /// is a group track of `.stats/node`. + pub fn tracks(registry: Registry, origin: origin::Producer) -> Option { + Self::build(registry, origin, 1, Layout::Track) + } + + fn build(registry: Registry, origin: origin::Producer, depth: usize, layout: Layout) -> Option { let task = Task { registry, origin, prefix: PathOwned::from(".stats"), node: None, - depth: 0, + depth, + layout, linger: Duration::from_secs(300), interval: Duration::from_secs(1), }; From 3f144371ef7affb1db1c2de025e03e9b1ad2f1ac Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Thu, 22 Oct 2026 18:38:35 -0700 Subject: [PATCH 03/16] refactor(stats): tidy the track layout's default-tier setup and pin its frame shape Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01EeyiEy5FWz3z44DYZVo7vU --- rs/moq-stats/src/lib.rs | 2 +- rs/moq-stats/src/produce.rs | 63 ++++++++++++++++++++++++++++--------- 2 files changed, 50 insertions(+), 15 deletions(-) diff --git a/rs/moq-stats/src/lib.rs b/rs/moq-stats/src/lib.rs index 9de1b5cd8d3..be74da7adf9 100644 --- a/rs/moq-stats/src/lib.rs +++ b/rs/moq-stats/src/lib.rs @@ -21,7 +21,7 @@ //! (default prefix `.stats`), or one per group of leading broadcast-path //! segments at `//node/`; parse announce paths back with //! [`parse_node_path`]. Each announcement carries a fresh -//! [`Epoch`](moq_net::Epoch) on its route, so a restarted node or a group +//! [`Epoch`] on its route, so a restarted node or a group //! returning from idle is a new broadcast at the same path; a reader tells //! them apart by the announced route's epoch. Every broadcast carries a //! `totals.json` track of each [`Tier`]'s [`Totals`] for the epoch, each tier diff --git a/rs/moq-stats/src/produce.rs b/rs/moq-stats/src/produce.rs index dee454ecf5b..c71b757eaf6 100644 --- a/rs/moq-stats/src/produce.rs +++ b/rs/moq-stats/src/produce.rs @@ -1357,22 +1357,23 @@ impl GroupPublisher { let mut sessions = TrackFamily::new(sequence.clone()); let tiers = task.layout == Layout::Broadcast; - // The totals and the default tier's tracks always exist, even while idle. - let tier = Tier::default(); - for role in [Role::Publisher, Role::Subscriber].into_iter().filter(|_| tiers) { - let name = traffic_track(&tier, role, false); - match TrackPair::create(&broadcast, &name, sequence.clone()) { - Ok(pair) => { - traffic.tracks.insert(name, pair); - } - Err(err) => { - tracing::warn!(?err, name, "stats: failed to create track"); - return None; + // The totals and, outside the track layout, the default tier's tracks + // always exist, even while idle. + if tiers { + let tier = Tier::default(); + for role in [Role::Publisher, Role::Subscriber] { + let name = traffic_track(&tier, role, false); + match TrackPair::create(&broadcast, &name, sequence.clone()) { + Ok(pair) => { + traffic.tracks.insert(name, pair); + } + Err(err) => { + tracing::warn!(?err, name, "stats: failed to create track"); + return None; + } } } - } - let name = sessions_track(&tier, false); - if tiers { + let name = sessions_track(&tier, false); match TrackPair::create(&broadcast, &name, sequence.clone()) { Ok(pair) => { sessions.tracks.insert(name, pair); @@ -3668,6 +3669,40 @@ mod tests { assert_eq!(serde_json::to_vec(&Frame::::default()).unwrap(), b"{}"); } + #[test] + fn group_body_serializes_like_a_group_frame() { + let mut usage = Usage::default(); + usage.subscriber.bytes = 5; + let mut presence = Presence::default(); + presence.sessions_started = 2; + let mut tier = GroupTier::default(); + tier.folded.publisher.bytes = 99; + tier.totals.publisher.bytes = 7; + tier.totals.sessions.sessions_started = 2; + tier.paths.entries.push((PathOwned::from("acme/live"), usage)); + tier.labels.entries.push((PathOwned::from("acme"), presence)); + let body = GroupBody { + epoch: Epoch::mint(), + tiers: Frame { + entries: vec![(PathOwned::from(""), tier)], + }, + }; + + let mut group = crate::Group::default(); + group.publisher.bytes = 7; + group.sessions.sessions_started = 2; + group.paths.insert("acme/live".into(), usage); + group.labels.insert("acme".into(), presence); + let frame: crate::GroupFrame = serde_json::from_slice(&serde_json::to_vec(&body).unwrap()).unwrap(); + assert_eq!(frame.epoch, body.epoch); + assert_eq!(frame.tiers, BTreeMap::from([(String::new(), group)])); + assert_eq!( + serde_json::to_vec(&frame).unwrap(), + serde_json::to_vec(&body).unwrap(), + "the producer writes what the public type reads, folded totals left out" + ); + } + /// Counts this thread's allocations, so the test below measures only its /// own drain while other tests run in parallel. mod counting { From e510cbe60427465911aa7ffdea530b97c362585b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 08:30:03 -0700 Subject: [PATCH 04/16] fix(net): forget a track name once its track is gone A broadcast kept a sequence namespace for every track name it ever wrote, so a long-lived broadcast churning names (stats group tracks at depth 2+, which a subscriber can mint) grew without bound. The broadcast now keeps nothing per name beyond its live tracks: a replacement starts over at 0, and the publisher that recreates a name picks group sequences past its predecessor's, as a name always means the same content. Mirrored in js/net, which also drops the removed-producer handoff that only existed to guard the shared namespace. Co-Authored-By: Claude Opus 5.5 --- doc/lib/js/net.md | 2 +- doc/lib/rs/moq-net.md | 2 +- js/net/MIGRATE.md | 7 + js/net/src/broadcast.test.ts | 186 ++---------------- js/net/src/broadcast.ts | 7 +- js/net/src/internal.ts | 56 ------ js/net/src/track.test.ts | 14 -- js/net/src/track.ts | 124 ++---------- rs/moq-net/MIGRATE.md | 6 + rs/moq-net/src/model/broadcast.rs | 158 ++++----------- rs/moq-net/src/model/track.rs | 169 ++-------------- rs/moq-net/src/model/track/demand.rs | 40 ++-- .../tests/integration/empty_replacement.rs | 124 ------------ rs/moq-net/tests/integration/main.rs | 1 - rs/moq-net/tests/integration/route_change.rs | 7 +- 15 files changed, 128 insertions(+), 775 deletions(-) delete mode 100644 rs/moq-net/tests/integration/empty_replacement.rs diff --git a/doc/lib/js/net.md b/doc/lib/js/net.md index f0c964bc2d7..c27983dd6c4 100644 --- a/doc/lib/js/net.md +++ b/doc/lib/js/net.md @@ -43,7 +43,7 @@ broadcast.announce(); - **GOAWAY moves the session.** The connection dials the replacement at once while the old session keeps serving its groups, up to `goaway.handover` (default 10s, or the relay's deadline when that is sooner). New requests keep opening on the old session until the replacement's route outranks it, which is a restart without an epoch: an unpinned announced request follows the replacement, while a blind or epoch-pinned request stays on its accepted instance. With an epoch, which travels only when `moq-lite-07-wip` is offered, the replacement serves the same broadcast, so open subscriptions move across without missing a group. An empty GOAWAY redials the same URL. A redirect stays on the same host unless `goaway.redirect` is `"follow"`. - **One send estimate per connection.** `Bandwidth.Allocator` divides it by track priority, max-min fair within a tier. An idle track claims nothing. Publishers reserve against it so their targets sum to the estimate. - **List a broadcast's tracks.** `for await (const event of broadcast.tracks())` reports each track the broadcast lists (`added`, with its `Track.Info`), one `synced`, then `added` and `removed` as tracks come and go, until the broadcast ends; leaving the loop stops listing. Over a session it needs `moq-lite-07-wip`, syncs at once with what this side holds, and the peer's tracks follow as `added`. A track served only on request is never listed. See [discovery](/concept/moq-lite#discovery). -- **One track per name.** Concurrent subscriptions share one request and producer. `createTrack` on a name with a request still queued answers it. The application owns sequence numbers: a track that replaces an ended one, created or requested, throws on an append that would reuse a sequence the name already wrote, so it writes its first group with an explicit sequence. Only a new broadcast starts over. +- **One track per name.** Concurrent subscriptions share one request and producer. `createTrack` on a name with a request still queued answers it. The application owns sequence numbers: the broadcast keeps nothing of an ended track, so a track that replaces it, created or requested, starts at 0. A name always means the same content, so the replacement continues past its predecessor's groups by writing its first group with an explicit sequence. - **Timing is per track, with no default.** A track that declares a `timescale` carries a `timestamp` on every frame; one without is [untimed](/concept/moq-lite#subscriptions) and its frames carry none. Omitting it means untimed, as in Rust. Nothing stamps a frame for you: the publisher passes the timestamp, and the `Timed` type (`{ value, at }`) carries one into the `@moq/json` and `@moq/flate` producers. - **Subscriber staleness is `maxDelay`.** It counts from a group's successor, by media time and by wall clock, whichever reaches it first. Passing the old `maxAge` key throws a `TypeError` naming `maxDelay`. Publisher retention is the separate `Track.Info.maxAge`. - **Hidden paths** stay out of discovery unless the announce request opts in. See [hidden broadcasts](/concept/moq-lite#hidden-broadcasts). diff --git a/doc/lib/rs/moq-net.md b/doc/lib/rs/moq-net.md index d9295f9ad4a..8a8abe1723b 100644 --- a/doc/lib/rs/moq-net.md +++ b/doc/lib/rs/moq-net.md @@ -20,7 +20,7 @@ What you use it for, beyond what the concept page already describes: - **Rank several CDNs.** Dial each with its own subscribe handle, `origin.clone().with_preference(origin::Preference::new(rank))` (higher wins; 0 is neutral and a negative rank demotes), all on one origin. Requests use the preferred link that has a route and fail over to the next; see [multiple CDNs](/concept/moq-lite#multiple-cdns). - **Follow restarts.** `consumer.announced_broadcast(path, None)` hands out each publisher instance serving a path from `available()`: the first once announced, then each one that takes over or returns after an end, like JS `request(path, { announced: true })`. A broadcast already resolved stays on the old instance until dropped or its route goes. `consumer.follow(path)` yields the underlying start, update, restart, and end of the route a request resolves. - **Discovery follows what you watch.** On moq-lite, a client session asks the relay for announcements only under the prefixes its subscribe origin's readers watch, with `announced()`, `routed_broadcast`, or `announced_broadcast`, stops asking about a prefix 2s after nothing at or beneath it is watched, and asks again on reconnect. A broadcast resolved through a discovered route keeps its path watched until dropped. `request_broadcast` watches nothing, so a path nobody watches is unroutable. A server session, or a client fed into a `peer()` or `upstream()` origin handle, asks for everything it may, so a relay discovers everything. An HTTP gateway (the moq-hls `Server`, the moq-rtc WHEP router) watches its whole scope while it lives, so the paths its callers name never open a request each. moq-transport sessions, and moq-lite-03/04 sessions (whose announcements mark no initial set), still ask for everything. -- **One track per name.** Concurrent subscriptions share one request and producer. The application owns sequence numbers: a producer that replaces an ended one gets `Error::Duplicate` from an append that would reuse a sequence the name already wrote, so it starts with `create_group` at an explicit sequence. Only a new broadcast starts over. Readers of an aborted track resume onto the replacement only when the broadcast announces an epoch; otherwise their subscription ends with the error. +- **One track per name.** Concurrent subscriptions share one request and producer. The application owns sequence numbers: the broadcast keeps nothing of an ended track, so a producer that replaces it starts at 0. A name always means the same content, so the replacement continues past its predecessor's groups with `create_group` at an explicit sequence. Readers of an aborted track resume onto the replacement only when the broadcast announces an epoch; otherwise their subscription ends with the error. - **End a broadcast with `finish()`.** When the input ends, finish the tracks, then call `broadcast.finish()`. The returned handle keeps the broadcast announced and serving its finished tracks for `linger()`, their longest max-age, so a subscriber still catching up can drain them. Await `wait()` on a runtime, or drop the handle once your own clock passes `linger()`. Dropping it early is the interrupt, and `close()` on another handle ends it at once. - **Publish only while someone is watching.** `demand()` on a track, group, or broadcast says whether a subscriber is attached, which is how capture and transcode skip work nobody asked for. Holding a broadcast or track consumer is not demand: a track consumer requests nothing until it queries, subscribes, or fetches, and those count until dropped. A shared fetch stays up until its last reader leaves. - **Use the model without a runtime.** Disable default features on `moq-net` to use its `poll_*` APIs and origin driver with caller-supplied time. The default `async` feature adds async helpers, `time::run`, and both wire sessions. diff --git a/js/net/MIGRATE.md b/js/net/MIGRATE.md index bc64e99cadd..b18995be0c9 100644 --- a/js/net/MIGRATE.md +++ b/js/net/MIGRATE.md @@ -4,6 +4,13 @@ How to update your code for each breaking change, newest first. ## Unreleased +### A replacement track starts over + +A broadcast no longer remembers the sequences an ended track of a name wrote, so its state stays bounded by its live tracks. +A track that replaces an ended one, created or requested, appends from 0 instead of throwing `duplicate sequence`, `close()` ends at its own writes, and `finishAt()` and its first frames are no longer checked against the predecessor. +A producer removed with `removeTrack` stays writable after a replacement claims the name; its writes reach only its own subscribers. +A publisher that recreates a name continues past the groups it already wrote with `writeGroup` at an explicit sequence, such as a timestamp. + ### `Track.Producer.largest()` is a getter `Track.Producer.largest()` returns the track's largest location synchronously, `undefined` before any, instead of a promise for the next newer one. diff --git a/js/net/src/broadcast.test.ts b/js/net/src/broadcast.test.ts index 6251563cf24..8155e1bdeff 100644 --- a/js/net/src/broadcast.test.ts +++ b/js/net/src/broadcast.test.ts @@ -1,9 +1,8 @@ import { expect, setSystemTime, test } from "bun:test"; import { Manual, Milli, Nano } from "@moq/clock"; import { Consumer as BroadcastConsumer, Producer as BroadcastProducer } from "./broadcast.ts"; -import { GroupTooLarge, NotFound, ProtocolViolation, TimestampMismatch } from "./error.ts"; +import { GroupTooLarge, NotFound } from "./error.ts"; import { Producer as GroupProducer, MAX_GROUP_FRAMES } from "./group.ts"; -import { TrackSequences } from "./internal.ts"; import { textFrame } from "./mock.ts"; import { Producer as OriginProducer } from "./origin.ts"; import * as Path from "./path.ts"; @@ -100,81 +99,9 @@ test("a disposed staged track refuses a later commit", async () => { await expect(info).rejects.toThrow("staged track dropped"); }); -// A replacement continues the name's timeline, so its groups can't start before the highest one an -// earlier producer checked, while a new broadcast starts over. -test("a replacement producer refuses a rewind past its predecessor", () => { - const broadcast = new BroadcastProducer(); - const first = broadcast.createTrack("media", { timescale: Timescale.MILLI }); - first.appendGroup().writeFrame(textFrame("a", Timestamp.fromMillis(1000))); - first.close(); - - const second = broadcast.createTrack("media", { timescale: Timescale.MILLI }); - const group = new GroupProducer(1); - second.writeGroup(group); - expect(() => group.writeFrame(textFrame("b", Timestamp.fromMillis(500)))).toThrow(TimestampMismatch); - group.writeFrame(textFrame("b", Timestamp.fromMillis(1000))); - - const next = new BroadcastProducer(); - const track = next.createTrack("media", { timescale: Timescale.MILLI }); - track.appendGroup().writeFrame(textFrame("c", Timestamp.fromMillis(500))); - - second.close(); - broadcast.close(); - next.close(); -}); - -// A replacement that closes before writing anything still ends past its predecessor's groups: -// an end below them would contradict the ones subscribers already received. -test("an empty replacement ends past its predecessor's groups", async () => { - const broadcast = new BroadcastProducer(); - const first = broadcast.createTrack("media", { timescale: Timescale.MILLI }); - first.appendGroup().close(); - first.close(); - - const second = broadcast.createTrack("media", { timescale: Timescale.MILLI }); - const subscriber = second.subscribe(); - expect(() => second.finishAt({ group: 0, frame: 0 })).toThrow(ProtocolViolation); - expect(() => second.finishAt({ group: 0, frame: 1 })).toThrow(ProtocolViolation); - second.close(); - expect(await subscriber.finished()).toEqual({ group: 1, frame: 0 }); - broadcast.close(); -}); - -test("a removed producer writes nothing once its replacement claims the name", async () => { - const broadcast = new BroadcastProducer(); - const first = broadcast.createTrack("media", { timescale: Timescale.MILLI }); - first.appendGroup().close(); - broadcast.removeTrack("media"); - - const second = broadcast.createTrack("media", { timescale: Timescale.MILLI }); - const subscriber = second.subscribe(); - second.close(); - expect(() => first.appendGroup()).toThrow("replaced"); - expect(() => first.writeGroup(new GroupProducer(1))).toThrow("replaced"); - expect(() => first.appendDatagram(Timestamp.fromMillis(0), new Uint8Array())).toThrow("replaced"); - expect(await subscriber.finished()).toEqual({ group: 1, frame: 0 }); - first.close(); - broadcast.close(); -}); - -test("a removed producer's abort ignores its replacement's groups", async () => { - const broadcast = new BroadcastProducer(); - const first = broadcast.createTrack("media", { timescale: Timescale.MILLI }); - first.finishAt({ group: 2, frame: 0 }); - broadcast.removeTrack("media"); - - const second = broadcast.createTrack("media", { timescale: Timescale.MILLI }); - for (let i = 0; i < 3; i++) second.appendGroup().close(); - const abort = new Error("cut off"); - first.close(abort); - expect(first.closed.peek()).toBe(abort); - second.close(); - broadcast.close(); -}); - -// A name keeps one sequence namespace across producer replacements. The application owns where a -// replacement starts: an append that would reuse an earlier producer's sequence throws. -test("a replacement producer refuses to append a sequence the name already wrote", async () => { +// A broadcast keeps nothing of a name once its track is gone: a replacement numbers and times its +// groups from scratch, and the application owns keeping them past its predecessor's. +test("a replacement producer starts over", async () => { const broadcast = new BroadcastProducer(); // Pull before subscribing: a broadcast nobody serves on demand answers NotFound instead. @@ -182,39 +109,29 @@ test("a replacement producer refuses to append a sequence the name already wrote const firstSubscriber = broadcast.track("media").subscribe(); const firstRequest = await firstPull; if (!firstRequest) throw new Error("expected first request"); - const firstProducer = firstRequest.accept({ timescale: Timescale.MILLI }); - expect(firstProducer.appendGroup().sequence).toBe(0); - expect(firstProducer.appendDatagram(Timestamp.fromMillis(0), new Uint8Array())).toBe(1); - firstProducer.writeGroup(new GroupProducer(8)); - firstProducer.insertDatagram(12, Timestamp.fromMillis(0), new Uint8Array()); + const first = firstRequest.accept({ timescale: Timescale.MILLI }); + const group = new GroupProducer(8); + first.writeGroup(group); + group.writeFrame(textFrame("a", Timestamp.fromMillis(1000))); + group.close(); firstSubscriber.close(); - firstProducer.close(); + first.close(); const secondSubscriber = broadcast.track("media").subscribe(); const secondRequest = await wireOf(broadcast).requested(); if (!secondRequest) throw new Error("expected second request"); - const secondProducer = secondRequest.accept({ timescale: Timescale.MILLI }); - expect(() => secondProducer.appendGroup()).toThrow("duplicate sequence"); - expect(() => secondProducer.appendDatagram(Timestamp.fromMillis(0), new Uint8Array())).toThrow( - "duplicate sequence", - ); - - // An explicit first write picks the start, and appends continue from there. - secondProducer.writeGroup(new GroupProducer(13)); - expect(secondProducer.appendGroup().sequence).toBe(14); - - const nextGeneration = new BroadcastProducer(); - const nextPull = wireOf(nextGeneration).requested(); - const nextSubscriber = nextGeneration.track("media").subscribe(); - const nextRequest = await nextPull; - if (!nextRequest) throw new Error("expected next-generation request"); - expect(nextRequest.accept({ timescale: Timescale.MILLI }).appendGroup().sequence).toBe(0); - + const second = secondRequest.accept({ timescale: Timescale.MILLI }); + const appended = second.appendGroup(); + expect(appended.sequence).toBe(0); + appended.writeFrame(textFrame("b", Timestamp.fromMillis(500))); secondSubscriber.close(); - secondProducer.close(); - nextSubscriber.close(); + second.close(); + + broadcast.removeTrack("media"); + const third = broadcast.createTrack("media", { timescale: Timescale.MILLI }); + third.finishAt({ group: 0, frame: 0 }); + third.close(); broadcast.close(); - nextGeneration.close(); }); test("createTrack answers a queued request", async () => { @@ -272,69 +189,6 @@ test("createTrack does not take over a request a handler already pulled", async broadcast.close(); }); -test("created and requested tracks share a name's sequences", async () => { - const broadcast = new BroadcastProducer(); - const first = broadcast.createTrack("media", { timescale: Timescale.MILLI }); - expect(first.appendGroup().sequence).toBe(0); - first.writeGroup(new GroupProducer(8)); - first.insertDatagram(12, Timestamp.fromMillis(0), new Uint8Array()); - first.close(); - - // A replacement refuses to append past nothing, since that would reuse every explicit write. - const second = broadcast.createTrack("media", { timescale: Timescale.MILLI }); - expect(() => second.appendGroup()).toThrow("duplicate sequence"); - second.writeGroup(new GroupProducer(13)); - second.close(); - - // An on-demand producer checks the same namespace, and so does a later created track. - broadcast.removeTrack("media"); - const pulled = wireOf(broadcast).requested(); - const subscriber = broadcast.track("media").subscribe(); - const request = await pulled; - if (!request) throw new Error("expected request"); - const requested = request.accept({ timescale: Timescale.MILLI }); - expect(() => requested.appendDatagram(Timestamp.fromMillis(0), new Uint8Array())).toThrow("duplicate sequence"); - requested.insertDatagram(14, Timestamp.fromMillis(0), new Uint8Array()); - subscriber.close(); - requested.close(); - - const later = broadcast.createTrack("media", { timescale: Timescale.MILLI }); - expect(() => later.appendGroup()).toThrow("duplicate sequence"); - later.close(); - - // Only a new broadcast starts over. - const next = new BroadcastProducer(); - expect(next.createTrack("media", { timescale: Timescale.MILLI }).appendGroup().sequence).toBe(0); - broadcast.close(); - next.close(); -}); - -test("names that never wrote leave nothing behind", () => { - const sequences = new TrackSequences(); - sequences.get("written").next = 1; - const held = sequences.get("held"); - held.holders = 1; - - // A peer asking for arbitrary names cannot grow the map: unused names are swept as it grows. - for (let i = 0; i < 1000; i++) sequences.get(`unused-${i}`); - expect(sequences.size).toBeLessThan(64); - - // Written and held names survive every sweep. - expect(sequences.get("written").next).toBe(1); - expect(sequences.get("held")).toBe(held); -}); - -test("a written name refuses to reuse its sequences after many other names", () => { - const broadcast = new BroadcastProducer(); - const written = broadcast.createTrack("written"); - written.appendGroup().close(); - written.close(); - for (let i = 0; i < 1000; i++) broadcast.createTrack(`unused-${i}`).close(); - - expect(() => broadcast.createTrack("written").appendGroup()).toThrow("duplicate sequence"); - broadcast.close(); -}); - test("publishing-side subscriptions coalesce onto one request", async () => { const broadcast = new BroadcastProducer(); const pulled = wireOf(broadcast).requested(); diff --git a/js/net/src/broadcast.ts b/js/net/src/broadcast.ts index 77a17e4cc2c..53ae20b56be 100644 --- a/js/net/src/broadcast.ts +++ b/js/net/src/broadcast.ts @@ -9,7 +9,7 @@ import type * as Epoch from "./epoch.ts"; import { NotFound } from "./error.ts"; import type { Consumer as GroupConsumer } from "./group.ts"; import { Route } from "./hop.ts"; -import { hooks, TrackSequences } from "./internal.ts"; +import { hooks } from "./internal.ts"; import { Lister, Listing, listTracks, type Tracks } from "./listing.ts"; import * as Path from "./path.ts"; import * as track from "./track.ts"; @@ -63,8 +63,6 @@ class BroadcastState { // publisher that has started pulling them. The counterpart of a live Rust `broadcast::Dynamic`; // without one, a track nobody publishes is `NotFound` rather than a request nobody will answer. served = false; - // Each name's sequence namespace, shared by every producer of the name, created or requested. - sequences = new TrackSequences(); // Live consumer handles sharing this state (see {@link Consumer.clone}). The broadcast // closes once the last one closes, so a shared consumer can be handed to several callers. consumers = 0; @@ -203,7 +201,7 @@ function logical(state: BroadcastState, name: string): { producer: track.Produce }); state.requested.mutate((requested) => { - requested.push(hooks.makeRequest({ name, producer, sequences: state.sequences, pending: state.pending })); + requested.push(hooks.makeRequest({ name, producer, pending: state.pending })); }); return { producer, requested: true }; @@ -441,7 +439,6 @@ export class Producer { } const producer = new track.Producer(name).accept(info); - hooks.bindSequence(producer, this.#state.sequences.get(name)); adopt(this.#state, producer, track.infoDefaults(info)); return producer; } diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index 5c79dbb77de..2c20a9ec4b0 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -204,55 +204,6 @@ export interface ReadGroupFrame { complete(): void; } -/** One past the highest sequence written under a broadcast track's name, by any of its producers. */ -export interface TrackSequence { - next: number; - /** Open producers writing through this namespace. */ - holders: number; - /** The open producer the name's newest claim went to; a later claim supersedes it. */ - owner?: Producer; - /** - * The highest-sequence checked group start any producer of the name wrote, which bounds a - * replacement's groups the way the eviction watermark bounds a track's own. - */ - bound?: { sequence: number; start: Timestamp }; -} - -/** - * Each name's sequence namespace for one broadcast generation, outliving the producers that - * serve it so a replacement that would append a sequence already written is refused. Written - * namespaces are kept until the broadcast ends, since forgetting one would let a later producer - * reuse its sequences. Namespaces nothing holds or wrote are swept as the map grows, like the - * Rust map at capacity, so requests for names never served stay bounded. - */ -export class TrackSequences { - #map = new Map(); - // The size that triggers the next sweep, doubling past what survives so sweeps stay amortized. - #sweepAt = 16; - - /** The namespace for `name`, created if the name is new. */ - get(name: string): TrackSequence { - const existing = this.#map.get(name); - if (existing) return existing; - - if (this.#map.size >= this.#sweepAt) { - for (const [key, sequence] of this.#map) { - if (sequence.next === 0 && sequence.holders === 0) this.#map.delete(key); - } - this.#sweepAt = Math.max(16, this.#map.size * 2); - } - - const sequence = { next: 0, holders: 0 }; - this.#map.set(name, sequence); - return sequence; - } - - /** How many names hold a namespace. */ - get size(): number { - return this.#map.size; - } -} - /** What a resumed reader's {@link ResumeSource} sees of the subscriber it serves. */ export interface ResumeShell { /** The reader's own preferences, mirrored into every copy it subscribes. */ @@ -378,8 +329,6 @@ export interface TrackRequestOptions { name: string; /** The producer that will serve the request. */ producer: Producer; - /** Sequence namespaces shared by the broadcast generation. */ - sequences: TrackSequences; /** Requests not yet accepted or rejected by the publisher. */ pending: Set; } @@ -433,8 +382,6 @@ export const hooks: { startTrack: (producer: Producer, served: { start?: number; requested?: number }) => void; /** Mint a track {@link Request}; assigned by `track.ts`. */ makeRequest: (options: TrackRequestOptions) => Request; - /** Have a producer write through a broadcast's sequence namespace for its name. */ - bindSequence: (producer: Producer, sequence: TrackSequence) => void; /** The newest frame a producer's cache holds, read without subscribing; assigned by `track.ts`. */ cachedLargest: (producer: Producer) => Location | undefined; /** @@ -611,9 +558,6 @@ export const hooks: { makeRequest: () => { throw new Error("track.ts not loaded"); }, - bindSequence: () => { - throw new Error("track.ts not loaded"); - }, cachedLargest: () => { throw new Error("track.ts not loaded"); }, diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index 2265f894540..91e4d4be4e1 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -1,6 +1,5 @@ import { expect, setSystemTime, spyOn, test } from "bun:test"; import { Manual, Milli, Nano } from "@moq/clock"; -import { Producer as BroadcastProducer } from "./broadcast.ts"; import { ProtocolViolation, StreamCode, StreamError, TimestampMismatch, TooFarBehind } from "./error.ts"; import { Producer as GroupProducer, MAX_GROUP_FRAMES } from "./group.ts"; import { Version as IetfVersion } from "./ietf/version.ts"; @@ -186,19 +185,6 @@ test("a fixed start is no join", () => { subscriber.close(); }); -test("a replacement wants the group past its predecessor", () => { - const broadcast = new BroadcastProducer(); - const first = broadcast.createTrack("test"); - first.writeGroup(new GroupProducer(7)); - first.close(); - - const second = broadcast.createTrack("test"); - const subscriber = second.subscribe(); - expect(second.demand().wanted.peek()).toBe(8); - subscriber.close(); - broadcast.close(); -}); - test("refresh must be a u32 ratio", () => { expect(() => infoDefaults({ refresh: -1 })).toThrow(RangeError); expect(() => infoDefaults({ refresh: 0.5 })).toThrow(RangeError); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 757f6b3cb27..d5068d19347 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -37,8 +37,6 @@ import { removeRun, runGaps, type TrackRequestOptions, - type TrackSequence, - type TrackSequences, } from "./internal.ts"; import { grace, Ledger } from "./ledger.ts"; import { Timescale, type Timestamp } from "./time.ts"; @@ -317,13 +315,11 @@ export class Request { readonly name: string; #producer: Producer; - #sequences: TrackSequences; #pending: Set; private constructor(options: TrackRequestOptions) { this.name = options.name; this.#producer = options.producer; - this.#sequences = options.sequences; this.#pending = options.pending; this.#pending.add(this); } @@ -363,7 +359,6 @@ export class Request { */ stage(info: Partial = {}): Staged { this.#pending.delete(this); - bindProducerSequence(this.#producer, this.#sequences.get(this.name)); return new Staged(this.#producer, info); } @@ -611,7 +606,6 @@ type CachedGroup = { // A group's checked start, kept by sequence. type Start = { sequence: number; start: Timestamp }; -let bindProducerSequence: (producer: Producer, sequence: TrackSequence) => void; // A fresh read handle on a group the producer still caches; assigned in its static block. let cachedMirror: (producer: Producer, sequence: number) => GroupConsumer | undefined; // Whether the producer still caches a group that has not failed; assigned in its static block. @@ -713,17 +707,9 @@ export class Producer { // The producer's own state is the source of truth (info/closed); subscribers // read mirrored sinks, never this state directly. #state = new TrackState(); - // The name's sequence namespace, which records every sequence any producer of the name - // wrote. Until a broadcast binds it, a namespace of its own. An open producer holds it. - #sequence: TrackSequence = { next: 0, holders: 1 }; // One past the highest group or datagram this producer received, like the Rust - // `max_sequence`, and where the next append lands. The namespace above can run ahead of - // it: earlier producers of the same broadcast track advanced it too. + // `max_sequence`, and where the next append lands. #received = 0; - // Where the namespace stood when a replacement claimed the name. From then on it is this - // producer's inherited edge, so the replacement's groups never count as its progress, and - // it writes nothing more, so it can't land a group past the replacement's end. - #handoff?: number; // Recently written source groups, retained for replay to late subscribers and // pruned once idle for longer than the cache window. Each entry tracks the mirror @@ -758,8 +744,8 @@ export class Producer { // The lowest group they want while no refresh is pending, never lowered. #floor = 0; // A join over the refresh ratio asked for a group past every sequence written, and none - // has landed yet. Read against the namespace rather than stored as a number, so a - // datagram taking the next sequence can never leave the ask behind. + // has landed yet. Read against `#received` rather than stored as a number, so a datagram + // taking the next sequence can never leave the ask behind. #refresh = false; #demand: Demand; // The largest group's content may still grow: it has not closed cleanly, and it is not a @@ -796,31 +782,6 @@ export class Producer { producer.#owe(); for (const state of [producer.#state, ...producer.#sinks]) state.startChanged.update((rev) => rev + 1); }; - // Anything written before binding still counts, so the namespace never goes backwards. - bindProducerSequence = (producer, sequence) => { - const own = producer.#sequence; - if (own === sequence) return; - if (producer.#state.closed.peek() === undefined) { - const owner = sequence.owner; - if (owner && owner.#state.closed.peek() === undefined) owner.#handoff = sequence.next; - sequence.owner = producer; - } - sequence.next = Math.max(sequence.next, producer.#received); - // What earlier producers of the name wrote is gone from this track's cache, so their - // highest checked start is its eviction watermark. - const previous = sequence.bound; - if (previous && (!producer.#evicted || producer.#evicted.sequence < previous.sequence)) { - producer.#evicted = previous; - } - if (own.bound && (!previous || previous.sequence < own.bound.sequence)) sequence.bound = own.bound; - producer.#sequence = sequence; - // A replacement's subscribers want the group past what earlier producers wrote. - if (sequence.next > producer.#received) producer.#want(sequence.next); - if (producer.#state.closed.peek() !== undefined) return; - own.holders--; - sequence.holders++; - }; - hooks.bindSequence = bindProducerSequence; hooks.cachedLargest = (producer) => producer.#cachedLargest(); hooks.linger = (producer) => { const maxAge = producer.#state.info.peek()?.maxAge; @@ -1018,17 +979,10 @@ export class Producer { this.#reportWanted(); } - // Raise the wanted floor, reporting it while anyone subscribes. - #want(group: number): void { - if (group <= this.#floor) return; - this.#floor = group; - this.#reportWanted(); - } - - // Report the wanted group while anyone subscribes: the namespace's next free sequence - // while a refresh is pending. + // Report the wanted group while anyone subscribes: the next free sequence while a refresh + // is pending. #reportWanted(): void { - const wanted = this.#refresh ? Math.max(this.#sequence.next, this.#floor) : this.#floor; + const wanted = this.#refresh ? Math.max(this.#received, this.#floor) : this.#floor; this.#wanted.set(this.#sinks.size > 0 ? wanted : undefined); } @@ -1190,8 +1144,6 @@ export class Producer { } entry.bound = start; const sequence = entry.group.sequence; - const shared = this.#sequence.bound; - if (!shared || shared.sequence < sequence) this.#sequence.bound = { sequence, start }; const last = this.#bounds.at(-1); if (!last || last.sequence < sequence) this.#bounds.push({ sequence, start }); else this.#bounds.splice(timelineIndex(this.#bounds, sequence), 0, { sequence, start }); @@ -1363,14 +1315,13 @@ export class Producer { this.#prune(); } - // Record a written sequence, here and in the name's namespace. It raises the largest too, - // at the head of an empty group, so a later group below it never changes it. A group - // (`answers`) at or past the namespace's next free sequence answers a pending refresh; - // anything else written there moves the sequence it asks for. + // Record a written sequence. It raises the largest too, at the head of an empty group, so + // a later group below it never changes it. A group (`answers`) at or past the next free + // sequence answers a pending refresh; anything else written there moves the sequence it + // asks for. #advance(sequence: number, answers: boolean): void { - const next = this.#sequence.next; - this.#received = Math.max(this.#received, sequence + 1); - this.#sequence.next = Math.max(next, sequence + 1); + const next = this.#received; + this.#received = Math.max(next, sequence + 1); this.#advanceLargest({ group: sequence, frame: 0 }); this.#endLargest(); if (!this.#refresh || sequence < next) return; @@ -1381,18 +1332,10 @@ export class Producer { this.#reportWanted(); } - // The sequence the next append takes: one past this producer's own writes. The - // application owns where a replacement of the name starts, so an append that would - // reuse a sequence an earlier producer wrote throws instead of renumbering. + // The sequence the next append takes: one past this producer's own writes. #nextSequence(): number { const next = this.#received; this.#writable(next); - const written = this.#sequence.next; - if (next < written) { - throw new Error( - `duplicate sequence: ${this.name} already wrote up to ${written - 1}, so start a replacement with an explicit sequence`, - ); - } return next; } @@ -1414,7 +1357,6 @@ export class Producer { // Refuse a write once the track is closed, or at or past its declared end. #writable(sequence: number): void { if (this.#state.closed.peek() !== undefined) throw new Error("track is closed"); - if (this.#handoff !== undefined) throw new Error(`track ${this.name} was replaced`); const end = this.#state.end.peek(); if (end !== undefined && sequence >= finalSequence(end)) { throw new Error(`sequence ${sequence} is at or past the track's end ${end.group}/${end.frame}`); @@ -1566,20 +1508,9 @@ export class Producer { * at or past it is refused. Unlike {@link close} it is not terminal: call `close()` once * the remaining groups are written. Readers end once the live edge reaches the end's * group, independently of closure. Throws if the track is closed, already has an end, or - * `end` falls before or inside a group already produced, including one an earlier producer - * of the name wrote, which a replacement continues. + * `end` would orphan a group or frame already produced. */ finishAt(end: Location): void { - // A replacement continues its name's timeline, so it can't end before or inside a group - // an earlier producer wrote; only its own last group can be cut mid-way. - const ownLast = end.frame !== 0 && this.#received === end.group + 1; - if (end.group < this.#inherited() && !ownLast) { - throw new ProtocolViolation(`track end ${end.group}/${end.frame} is inside the name's earlier groups`); - } - this.#finishAt(end); - } - - #finishAt(end: Location): void { if (this.#state.closed.peek() !== undefined) throw new Error("track is closed"); const declared = this.#state.end.peek(); if (declared !== undefined) throw new Error(`track already ends at ${declared.group}/${declared.frame}`); @@ -1587,12 +1518,11 @@ export class Producer { } // The wire's end: repeating it is a no-op, and a mid-group end may refine the group - // boundary it falls just before. Mirrors the Rust `finish_at` on a `received()` copy, - // which answers to its publisher's end rather than the name's local writes. + // boundary it falls just before. Mirrors the Rust `finish_at` on a `received()` copy. #endPending(end: Location): void { const declared = this.#state.end.peek(); if (declared === undefined) { - this.#finishAt(end); + this.finishAt(end); } else if (declared.group !== end.group || declared.frame !== end.frame) { if (declared.frame !== 0 || end.frame === 0 || finalSequence(end) !== declared.group) { throw new ProtocolViolation(`conflicting track end: ${end.group}/${end.frame}`); @@ -1651,18 +1581,6 @@ export class Producer { return !this.#arrivals.covers(Math.min(floor, final), final); } - // One past the highest sequence produced: where the live edge stands against the end. A - // replacement starts at its name's, since what earlier producers wrote never arrives here. - // Mirrors the Rust `edge`. - #liveEdge(): number { - return Math.max(this.#received, this.#inherited()); - } - - // One past what the name's producers wrote, up to the replacement that superseded this one. - #inherited(): number { - return this.#handoff ?? this.#sequence.next; - } - #declareEnd(end: Location): void { const { group, frame } = end; if (!Number.isSafeInteger(group) || group < 0 || !Number.isSafeInteger(frame) || frame < 0) { @@ -1694,7 +1612,7 @@ export class Producer { #endLargest(): void { if (this.#state.largestEnded.peek() || this.#largestOpen) return; const end = this.#state.end.peek(); - if (end === undefined || this.#liveEdge() < finalSequence(end)) return; + if (end === undefined || this.#received < finalSequence(end)) return; for (const state of [this.#state, ...this.#sinks]) state.largestEnded.set(true); } @@ -1719,17 +1637,13 @@ export class Producer { // A gap past the grace was lost before the abort rather than cut off by it. if (abort && this.#tailPending) this.#arrivals.expire(now(), this.#grace()); if (abort && this.#settled()) abort = undefined; - // A replacement continues its name's timeline: ending below an earlier producer's - // groups would contradict the ones subscribers already received. if (declare && abort === undefined && this.#state.end.peek() === undefined) { - this.#declareEnd({ group: this.#liveEdge(), frame: 0 }); + this.#declareEnd({ group: this.#received, frame: 0 }); } // Nobody will finish these, so a subscriber that has not taken one yet never sees it. // Not evicted: a reader already holding one keeps its frames and sees the abort. const open = abort ? this.#cache.filter((entry) => entry.group.closed.peek() === undefined) : []; closeTrackState(this.#state, abort); - this.#sequence.holders--; - if (this.#sequence.owner === this) this.#sequence.owner = undefined; for (const { group } of this.#cache) group.close(abort); for (const entry of open) { this.#unlink(entry); @@ -1752,7 +1666,7 @@ export class Producer { const end = this.#state.end.peek(); if (end === undefined) return false; const final = finalSequence(end); - if (this.#liveEdge() < final || this.#owes(final)) return false; + if (this.#received < final || this.#owes(final)) return false; return this.#cache.every(({ group }) => group.sequence >= final || group.closed.peek() === null); } diff --git a/rs/moq-net/MIGRATE.md b/rs/moq-net/MIGRATE.md index 5b362199db7..d6c1c88acb8 100644 --- a/rs/moq-net/MIGRATE.md +++ b/rs/moq-net/MIGRATE.md @@ -4,6 +4,12 @@ How to update your code for each breaking change, newest first. ## Unreleased +### A replacement track starts over (#6428) + +A broadcast no longer remembers the sequences an ended track of a name wrote, so its state stays bounded by its live tracks. +A track that replaces an ended one, created or requested, appends from 0 instead of returning `Error::Duplicate`, `finish()` ends at its own writes, and `finish_at()` and its first frames are no longer checked against the predecessor. +A publisher that recreates a name continues past the groups it already wrote with `create_group` at an explicit sequence, such as a timestamp. + ### Sessions box their transport (#6176) `Driver` and `server::Handshake` lose their transport type parameter: write `moq_net::Driver` and `moq_net::server::Handshake`. diff --git a/rs/moq-net/src/model/broadcast.rs b/rs/moq-net/src/model/broadcast.rs index f4b0af63950..cd93344e032 100644 --- a/rs/moq-net/src/model/broadcast.rs +++ b/rs/moq-net/src/model/broadcast.rs @@ -10,7 +10,6 @@ //! [Info] is the broadcast's static metadata, fixed for its lifetime. use crate::{cache, stats, track}; use std::{ - collections::HashMap, sync::Arc, task::{Poll, ready}, time::Duration, @@ -116,12 +115,6 @@ struct BroadcastState { // coalescing onto it there). requests: Requests, track::Request>, - // Each name's sequence namespace, outliving the producers that serve it: a name means - // the same content for the broadcast's whole life, so a replacement producer that would - // append a sequence an earlier one wrote is refused. Entries nothing holds or wrote are - // swept as the map grows, so requests for names never served stay bounded. - sequences: HashMap, track::Sequence>, - // The tracks the broadcast lists, and the readers listing them; see [`Tracks`]. listing: listing::ListingState, @@ -148,14 +141,6 @@ impl BroadcastState { } } - /// The sequence namespace for `name`, shared with every earlier track of the name. - fn sequence(&mut self, name: &Arc) -> track::Sequence { - if self.sequences.len() == self.sequences.capacity() { - self.sequences.retain(|_, sequence| !sequence.is_unused()); - } - self.sequences.entry(name.clone()).or_default().clone() - } - /// Resolve every name the broadcast never filled, so subscribers waiting on a /// [`track::Info`] that can no longer arrive fail with `err` instead of parking. /// @@ -381,10 +366,7 @@ impl Producer { return Ok(track.listed(unlist)); } - let sequence = state.sequence(&name); - let track = track::Producer::new(self.info.clone(), name.clone(), info.clone()) - .with_stats(self.stats.clone()) - .with_sequence(sequence); + let track = track::Producer::new(self.info.clone(), name.clone(), info.clone()).with_stats(self.stats.clone()); state.insert_track(track.weak())?; let unlist = Unlist::list(&self.state, &mut state.listing, name, &info); Ok(track.listed(unlist)) @@ -404,9 +386,7 @@ impl Producer { pub fn reserve_track(&self, name: impl Into>) -> Result { let name = name.into(); let mut state = self.state.lock(); - let request = track::Request::new(self.info.clone(), name.clone()) - .with_stats(self.stats.clone()) - .with_sequence(state.sequence(&name)); + let request = track::Request::new(self.info.clone(), name.clone()).with_stats(self.stats.clone()); state.insert_track(request.weak())?; Ok(request.listed(ListOnAccept { state: self.state.clone(), @@ -1119,7 +1099,7 @@ impl Consumer { // requests map, and the FIFO order. The request inherits the broadcast's // cache pool through its `Arc`, same as a producer-created track. let name: Arc = name.into(); - let request = track::Request::new(self.info.clone(), name.clone()).with_sequence(state.sequence(&name)); + let request = track::Request::new(self.info.clone(), name.clone()); let resolved = request.join(subscription); // With no handler alive to serve it, the request is dropped: `NotFound` beats @@ -1806,126 +1786,58 @@ mod test { track2.assert_group(); } - /// A name keeps one sequence namespace for the broadcast's life. The application owns - /// where a replacement starts: an append that would reuse an earlier producer's sequence, - /// explicit writes included, is refused, while a new broadcast starts over. + /// A broadcast keeps nothing of a name once its track is gone: a replacement numbers and + /// times its groups from scratch, and the application owns keeping them past its + /// predecessor's. #[test] - fn replacement_refuses_reused_sequences() { + fn replacement_starts_over() { + let at = |millis| crate::Timestamp::from_millis(millis).unwrap(); let producer = Info::new().produce(); let mut dynamic = producer.dynamic(); let consumer = producer.consume(); let track = consumer.track("media").unwrap().subscribe(None); let mut first = dynamic.assert_request().accept(crate::track::Info::timed()); - assert_eq!(first.append_group().unwrap().sequence, 0); - assert_eq!(first.append_datagram(crate::Timestamp::ZERO, b"x").unwrap(), 1); - let _open = first.create_group(crate::group::Info { sequence: 8 }).unwrap(); - first.insert_datagram(12, crate::Timestamp::ZERO, b"x").unwrap(); + let mut group = first.create_group(crate::group::Info { sequence: 8 }).unwrap(); + group.write_frame(at(1000), b"a".as_slice()).unwrap(); + group.finish().unwrap(); first.finish().unwrap(); drop((first, track)); let track = consumer.track("media").unwrap().subscribe(None); - let mut second = dynamic.assert_request().accept(crate::track::Info::timed()); - assert!(matches!(second.append_group(), Err(Error::Duplicate))); - assert!(matches!( - second.append_datagram(crate::Timestamp::ZERO, b"x"), - Err(Error::Duplicate) - )); - - // An explicit first write picks the start, and appends continue from there. - let _open = second.create_group(crate::group::Info { sequence: 13 }).unwrap(); - assert_eq!(second.append_datagram(crate::Timestamp::ZERO, b"x").unwrap(), 14); - second.finish().unwrap(); - drop((second, track)); + let second = dynamic.assert_request().accept(crate::track::Info::timed()); + let mut group = second.append_group().unwrap(); + assert_eq!(group.sequence, 0); + group.write_frame(at(500), b"b".as_slice()).unwrap(); + drop((group, second, track)); - // A track the producer creates itself checks the same namespace. - let third = producer.create_track("media", crate::track::Info::timed()).unwrap(); - assert!(matches!(third.append_group(), Err(Error::Duplicate))); - assert_eq!( - third - .create_group(crate::group::Info { sequence: 15 }) - .unwrap() - .sequence, - 15 - ); - - let next = Info::new().produce(); - assert_eq!( - next.create_track("media", crate::track::Info::timed()) - .unwrap() - .append_group() - .unwrap() - .sequence, - 0 - ); - } - - /// A replacement can't declare an end before or inside a group an earlier producer of the - /// name wrote: a subscriber that received that group would read it as a contradiction. - #[test] - fn replacement_refuses_an_end_below_its_predecessor() { - let producer = Info::new().produce(); - let first = producer.create_track("media", crate::track::Info::timed()).unwrap(); - first.append_group().unwrap().finish().unwrap(); - first.finish().unwrap(); - drop(first); - - let mut second = producer.create_track("media", crate::track::Info::timed()).unwrap(); - for end in [ - crate::track::Position::group(0), - crate::track::Position { group: 0, frame: 1 }, - ] { - assert!( - matches!(second.finish_at(end), Err(Error::ProtocolViolation)), - "{end:?}" - ); - } - second.finish_at(crate::track::Position::group(1)).unwrap(); - } - - /// A replacement continues the name's timeline, so its groups can't start before the - /// highest one an earlier producer checked, while a new broadcast starts over. - #[test] - fn replacement_refuses_a_rewind_past_its_predecessor() { - let at = |millis| crate::Timestamp::from_millis(millis).unwrap(); - let producer = Info::new().produce(); - - let first = producer.create_track("media", crate::track::Info::timed()).unwrap(); - let mut group = first.append_group().unwrap(); - group.write_frame(at(1000), b"a".as_slice()).unwrap(); - group.finish().unwrap(); - drop((group, first)); - - let second = producer.create_track("media", crate::track::Info::timed()).unwrap(); - let mut group = second.create_group(crate::group::Info { sequence: 1 }).unwrap(); - assert!(matches!( - group.write_frame(at(500), b"b".as_slice()), - Err(Error::TimestampMismatch) - )); - group.write_frame(at(1000), b"b".as_slice()).unwrap(); - - let next = Info::new().produce(); - let track = next.create_track("media", crate::track::Info::timed()).unwrap(); - track - .append_group() - .unwrap() - .write_frame(at(500), b"c".as_slice()) - .unwrap(); + let mut third = producer.create_track("media", crate::track::Info::timed()).unwrap(); + third.finish_at(crate::track::Position::group(0)).unwrap(); } - /// Names requested but never served leave nothing behind, so a peer asking for - /// arbitrary names cannot grow the broadcast. + /// Names that are gone leave nothing behind, served or not, so churning distinct names + /// keeps the broadcast bounded by its live tracks. #[test] - fn unserved_names_are_forgotten() { + fn gone_names_are_forgotten() { let producer = Info::new().produce(); let mut dynamic = producer.dynamic(); let consumer = producer.consume(); for i in 0..1000 { - let track = consumer.track(&i.to_string()).unwrap().subscribe(None); - drop(dynamic.assert_request()); - drop(track); + let track = consumer.track(&format!("requested/{i}")).unwrap().subscribe(None); + let served = dynamic.assert_request().accept(crate::track::Info::timed()); + served.append_group().unwrap().finish().unwrap(); + drop((served, track)); + + let track = consumer.track(&format!("refused/{i}")).unwrap().subscribe(None); + drop((dynamic.assert_request(), track)); + + let created = producer + .create_track(format!("created/{i}"), crate::track::Info::timed()) + .unwrap(); + created.append_group().unwrap().finish().unwrap(); } - assert!(producer.state.lock().sequences.len() < 16); + let state = producer.state.lock(); + assert!(state.tracks.len() < 16, "{} names retained", state.tracks.len()); } #[moq_clock::test] diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index d45ab47a735..ff2ba17e17b 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -34,7 +34,7 @@ use std::{ ops::{self, Bound, RangeBounds}, sync::Arc, sync::OnceLock, - sync::atomic::{AtomicBool, AtomicU64, Ordering}, + sync::atomic::{AtomicBool, Ordering}, sync::{Mutex, PoisonError}, task::{Poll, ready}, time::Duration, @@ -244,49 +244,6 @@ impl Info { } } -/// The sequence namespace a track name keeps within its broadcast, shared by every -/// producer that serves the name, so a replacement continues the same timeline: it can't -/// append a sequence an earlier one wrote, nor start a group before an earlier one's. -#[derive(Clone, Default)] -pub(super) struct Sequence(Arc); - -#[derive(Default)] -struct SequenceState { - // One past the highest sequence written. - next: AtomicU64, - // The highest-sequence checked start written, which bounds a replacement's groups the - // way the eviction watermark bounds a track's own. - bound: Mutex>, -} - -impl Sequence { - /// One past the highest sequence any producer of the name wrote. - fn next(&self) -> u64 { - self.0.next.load(Ordering::Relaxed) - } - - fn advance(&self, sequence: u64) { - self.0.next.fetch_max(sequence.saturating_add(1), Ordering::Relaxed); - } - - fn bound(&self) -> Option<(u64, Timestamp)> { - *self.0.bound.lock().unwrap_or_else(PoisonError::into_inner) - } - - /// Keep group `sequence`'s checked `start` if it is the highest-sequence one yet. - fn share_bound(&self, sequence: u64, start: Timestamp) { - let mut bound = self.0.bound.lock().unwrap_or_else(PoisonError::into_inner); - if bound.is_none_or(|(highest, _)| sequence > highest) { - *bound = Some((sequence, start)); - } - } - - /// Whether nothing holds it and nothing was written, so forgetting it changes nothing. - pub(super) fn is_unused(&self) -> bool { - Arc::strong_count(&self.0) == 1 && self.next() == 0 - } -} - /// Whether a track's cache reflects its live feed; see [`TrackState::set_idle`]. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] enum Feed { @@ -379,10 +336,6 @@ pub(crate) struct TrackState { first_produced: Option, newest_produced: Option, - // The name's sequence namespace within the broadcast, which can run past this track's - // `max_sequence` when an earlier producer of the name wrote further. - sequence: Sequence, - // The checked starts cached groups hold, by sequence (see [`group::Origin::Bound`]), so a // first frame finds its neighbors without walking groups not yet stamped. Filled through // the read guard a first frame holds, so it needs its own lock, never contended. @@ -893,8 +846,8 @@ impl TrackState { self.cache.wakes().started(sequence); } - /// Record group `sequence`'s just-checked `start` as a bound for its neighbors and the - /// name's later producers. A group stamped after it left the cache bounds nothing. + /// Record group `sequence`'s just-checked `start` as a bound for its neighbors. A group + /// stamped after it left the cache bounds nothing. fn share_bound(&self, sequence: u64, start: Timestamp) { if self.lookup.get(&sequence).and_then(Slot::bound) != Some(start) { return; @@ -903,7 +856,6 @@ impl TrackState { .lock() .unwrap_or_else(PoisonError::into_inner) .insert(sequence, start); - self.sequence.share_bound(sequence, start); } /// Take group `sequence` out of the cache, raising the eviction watermark to its start. @@ -1736,33 +1688,23 @@ impl TrackState { state } - /// Record a written sequence, here and in the name's shared namespace. It raises the - /// largest too, so a later group below it never changes it, even one below a reached end. + /// Record a written sequence. It raises the largest too, so a later group below it never + /// changes it, even one below a reached end. /// /// `answers` when it is a group subscribers receive, which answers a pending refresh. fn advance(&mut self, sequence: u64, answers: bool) { self.max_sequence = Some(self.max_sequence.map_or(sequence, |max| max.max(sequence))); - let next = self.sequence.next(); - self.sequence.advance(sequence); self.cache.advance_largest(Position::group(sequence)); - let mut wakes = self.subscriptions.lock().wrote(sequence, next, answers); + let mut wakes = self.subscriptions.lock().wrote(sequence, answers); wakes.wake(); } /// The sequence the next append takes: one past this track's own writes. - /// - /// The application owns where a replacement of the name starts, so an append that - /// would reuse a sequence an earlier producer wrote is [`Error::Duplicate`] instead of - /// renumbering. fn next_sequence(&self) -> Result { - let next = match self.max_sequence { + Ok(match self.max_sequence { Some(max) => max.checked_add(1).ok_or(coding::BoundsExceeded)?, None => 0, - }; - if next < self.sequence.next() { - return Err(Error::Duplicate); - } - Ok(next) + }) } /// Reject a sequence that is still cached; a dead (aborted or evicted) @@ -2190,11 +2132,8 @@ impl TrackState { } /// One past the highest sequence produced: where the live edge stands against the end. - /// A replacement starts at its name's, since what earlier producers wrote never arrives here. fn edge(&self) -> u64 { - self.max_sequence - .map_or(0, |max| max.saturating_add(1)) - .max(self.sequence.next()) + self.max_sequence.map_or(0, |max| max.saturating_add(1)) } /// Whether a group below `fin` the wire subscription still owes has yet to arrive. @@ -2662,13 +2601,6 @@ impl Producer { self } - /// Continue the name's sequence namespace within its broadcast. Set by the broadcast - /// before the track is handed out. - pub(super) fn with_sequence(self, sequence: Sequence) -> Self { - set_sequence(&self.state, sequence); - self - } - /// Keep the broadcast's listing of this track until it leaves the table. pub(super) fn listed(self, unlist: broadcast::Unlist) -> Self { self.alive.list(unlist); @@ -2715,8 +2647,9 @@ impl Producer { /// Create a new group with the next sequence number: one past this producer's own writes. /// - /// [`Error::Duplicate`] if an earlier producer of the name in this broadcast already wrote - /// that sequence. A replacement picks where it starts with [`Self::create_group`]. + /// The broadcast remembers nothing of an earlier producer of the name, so a replacement + /// starts at 0. Since a name always means the same content, a replacement continues past + /// its predecessor's groups with [`Self::create_group`]. pub fn append_group(&self) -> Result { let mut state = self.modify()?; let sequence = state.next_sequence()?; @@ -2746,8 +2679,7 @@ impl Producer { /// before 05, or stream-only transports like WebSocket) never deliver them. Keep payloads well under the 1200-byte minimum path MTU. /// A datagram is never cached or served by a fetch, so use a group for anything a late /// joiner needs. An origin publisher uses this; a relay preserving upstream numbering - /// uses [`Self::insert_datagram`]. Like [`Self::append_group`], it is [`Error::Duplicate`] - /// if an earlier producer of the name already wrote that sequence. + /// uses [`Self::insert_datagram`]. /// /// Pass `None` on an untimed track; a datagram that doesn't match its track is /// [`Error::TimestampMismatch`]. @@ -2916,8 +2848,7 @@ impl Producer { /// Mark the track as finished after the last appended group. /// - /// Ends the track at the head of the group after the largest one written to its name, - /// including by an earlier producer this one replaces. + /// Ends the track at the head of the group after the current max_sequence. /// No new groups at or above it can be appended. /// NOTE: Old groups with lower sequence numbers can still arrive. pub fn finish(&self) -> Result<()> { @@ -2926,9 +2857,6 @@ impl Producer { Some(max) => max.checked_add(1).ok_or(coding::BoundsExceeded)?, None => 0, }; - // A replacement continues its name's timeline: ending below an earlier producer's - // groups would contradict the ones subscribers already received. - let final_sequence = final_sequence.max(state.sequence.next()); state.set_end(Position::group(final_sequence)) } @@ -2949,21 +2877,13 @@ impl Producer { /// /// Repeating the declared end is a no-op, and a mid-group end may refine the group /// boundary it falls just before (a peer may name only the boundary first). Any other - /// second end is [`Error::ProtocolViolation`], and so is one before or inside a group an - /// earlier producer of the name wrote, which a replacement continues. + /// second end is [`Error::ProtocolViolation`]. /// /// On a [received](Self::received) copy, groups below the end may still be in flight: /// readers wait for a missing one until a grace passes, the subscription's max delay /// or one second without one, before ending at the boundary. pub fn finish_at(&mut self, end: Position) -> Result<()> { let mut state = self.modify()?; - // A replacement continues its name's timeline, so it can't end before or inside a group - // an earlier producer wrote; only its own last group can be cut mid-way. A received - // copy answers to its publisher's end, which the session judges. - let own_last = end.frame != 0 && state.max_sequence == Some(end.group); - if !state.received && end.group < state.sequence.next() && !own_last { - return Err(Error::ProtocolViolation); - } match state.end { Some(declared) if declared == end => return Ok(()), Some(declared) if declared.frame == 0 && end.frame != 0 && end.final_sequence() == Some(declared.group) => { @@ -3518,26 +3438,6 @@ impl Drop for Dynamic { } } -fn set_sequence(state: &kio::Producer, sequence: Sequence) { - let Ok(mut state) = state.write() else { - return; - }; - debug_assert!( - state.max_sequence.is_none(), - "a track joins its namespace before writing" - ); - // What earlier producers of the name wrote is gone from this track's cache, so their - // highest checked start is its eviction watermark. - state.evicted = sequence.bound(); - // A replacement's consumers want the group past what earlier producers wrote. - let next = sequence.next(); - state.sequence = sequence; - let subscriptions = state.subscriptions.clone(); - drop(state); - let mut wakes = subscriptions.lock().want(next); - wakes.wake(); -} - /// Ends the track when the last [`Producer`] or [`Dynamic`] drops. /// /// A refcount rather than a "am I the last one?" check inside `Drop`: that answer is a @@ -3888,8 +3788,7 @@ impl Demand { if !self.is_used() { return None; } - let (subscriptions, namespace) = self.subscriptions(); - let wanted = subscriptions.lock().wanted(&namespace); + let wanted = self.subscriptions().lock().wanted(); Some(wanted) } @@ -3904,10 +3803,9 @@ impl Demand { kio::wait(|waiter| self.poll_wanted_changed(waiter)).await } - /// The demand aggregate, and the name's sequence namespace a pending refresh reads. - fn subscriptions(&self) -> (kio::Lock, Sequence) { - let state = self.state.read(); - (state.subscriptions.clone(), state.sequence.clone()) + /// The demand aggregate. + fn subscriptions(&self) -> kio::Lock { + self.state.read().subscriptions.clone() } /// Poll for [`wanted`](Self::wanted) to differ from what this handle last returned. @@ -3917,8 +3815,7 @@ impl Demand { DemandState::Active => true, DemandState::Idle => false, }; - let (subscriptions, namespace) = self.subscriptions(); - let wanted = subscriptions.lock().poll_wanted(&namespace, waiter); + let wanted = self.subscriptions().lock().poll_wanted(waiter); let next = used.then_some(wanted); if self.prev == Some(next) { return Poll::Pending; @@ -7396,13 +7293,6 @@ impl Request { self } - /// Continue the name's sequence namespace within its broadcast. Set by the broadcast - /// before the request is visible to anyone. - pub(super) fn with_sequence(self, sequence: Sequence) -> Self { - set_sequence(&self.state, sequence); - self - } - /// The requested track name. pub fn name(&self) -> &str { &self.name @@ -15873,25 +15763,6 @@ mod test { assert_eq!(producer.demand().wanted(), Some(0)); } - /// A replacement's consumers want the group past what earlier producers of the name - /// wrote, where its first group has to start. - #[test] - fn a_replacement_wants_past_its_predecessor() { - let broadcast = broadcast::Info::default().produce(); - let first = broadcast.create_track("test", Info::default()).unwrap(); - let _open = first.create_group(group::Info { sequence: 7 }).unwrap(); - first.finish().unwrap(); - drop(first); - - let second = broadcast.create_track("test", Info::default()).unwrap(); - let mut demand = second.demand(); - assert_eq!(wanted_changed(&mut demand), Some(None)); - let _subscriber = second.subscribe(None); - assert_eq!(wanted_changed(&mut demand), Some(Some(8))); - let _open = second.create_group(group::Info { sequence: 8 }).unwrap(); - assert_eq!(wanted_changed(&mut demand), None); - } - /// The demand reports the track's closure as its abort reason. #[test] fn wanted_changed_errors_once_closed() { diff --git a/rs/moq-net/src/model/track/demand.rs b/rs/moq-net/src/model/track/demand.rs index 80618c813b2..38e7c101eaa 100644 --- a/rs/moq-net/src/model/track/demand.rs +++ b/rs/moq-net/src/model/track/demand.rs @@ -9,7 +9,7 @@ use std::{ time::Duration, }; -use super::{Order, Sequence, Subscription, TrackState, clamp_combined}; +use super::{Order, Subscription, TrackState, clamp_combined}; mod ranges; use ranges::Ranges; @@ -25,9 +25,11 @@ pub(super) struct Subscriptions { /// [`super::Demand::wanted`]. wanted: u64, /// A join over the track's refresh ratio asked for a group past every sequence written, - /// and no such group has landed yet. Read against the namespace rather than stored as a - /// number, so a datagram taking the next sequence can never leave the ask behind. + /// and no such group has landed yet. Read against `next` rather than stored as a number, + /// so a datagram taking the next sequence can never leave the ask behind. refresh: bool, + /// One past the highest sequence written, which a pending refresh asks for. + next: u64, waiters: kio::WaiterList, } @@ -83,30 +85,18 @@ impl Subscriptions { } /// The lowest group consumers want, whether or not anyone consumes now: the next free - /// sequence of `namespace` while a refresh is pending. - /// - /// Read under this lock, so a write that moves the namespace after it wakes the waiters - /// armed here (see [`Self::wrote`]). - pub(super) fn wanted(&self, namespace: &Sequence) -> u64 { + /// sequence while a refresh is pending. + pub(super) fn wanted(&self) -> u64 { match self.refresh { - true => namespace.next().max(self.wanted), + true => self.next.max(self.wanted), false => self.wanted, } } /// [`Self::wanted`], arming `waiter` for its next change. - pub(super) fn poll_wanted(&mut self, namespace: &Sequence, waiter: &kio::Waiter) -> u64 { + pub(super) fn poll_wanted(&mut self, waiter: &kio::Waiter) -> u64 { waiter.register(&mut self.waiters); - self.wanted(namespace) - } - - /// Raise the floor of [`Self::wanted`] to `group`, taking the wakes only when it moved. - pub(super) fn want(&mut self, group: u64) -> kio::WaiterList { - if group <= self.wanted { - return Default::default(); - } - self.wanted = group; - self.waiters.take() + self.wanted() } /// A join over the refresh ratio: ask for a group past every sequence written. Joins @@ -118,10 +108,12 @@ impl Subscriptions { self.waiters.take() } - /// Sequence `sequence` was written at or past `next`, the namespace's next free sequence - /// before it. A group subscribers receive answers a pending refresh; anything else, such - /// as a datagram, moves the sequence it asks for. - pub(super) fn wrote(&mut self, sequence: u64, next: u64, answers: bool) -> kio::WaiterList { + /// Sequence `sequence` was written. One at or past the next free sequence moves it: a + /// group subscribers receive answers a pending refresh, and anything else, such as a + /// datagram, moves the sequence it asks for. + pub(super) fn wrote(&mut self, sequence: u64, answers: bool) -> kio::WaiterList { + let next = self.next; + self.next = next.max(sequence.saturating_add(1)); if !self.refresh || sequence < next { return Default::default(); } diff --git a/rs/moq-net/tests/integration/empty_replacement.rs b/rs/moq-net/tests/integration/empty_replacement.rs deleted file mode 100644 index 8c34a501ea8..00000000000 --- a/rs/moq-net/tests/integration/empty_replacement.rs +++ /dev/null @@ -1,124 +0,0 @@ -#![cfg(feature = "async")] - -//! A dynamic handler serves a track, the relay's copy goes idle, and the next request gets a -//! replacement producer that finishes before writing anything. The relay still caches the -//! first producer's group, so the replacement's end must not fall below it: an end there -//! contradicts a received group, which the relay treats as a protocol violation and closes -//! the whole session. - -use crate::support; - -use std::time::Duration; - -use moq_net::{Hop, Timestamp, Version, broadcast, track}; -use support::harness::{MockConnectOptions, connect_mock}; - -const TIMEOUT: Duration = Duration::from_secs(5); - -fn produce_origin(hop: u64) -> moq_net::origin::Producer { - let (producer, driver) = moq_net::origin::Producer::new(moq_net::origin::Config::new(Hop::new(hop).unwrap())); - support::harness::spawn(driver); - producer -} - -/// Subscribe to `name` through `remote`, answering the request with a new producer. -async fn serve( - remote: &broadcast::Consumer, - dynamic: &mut broadcast::Dynamic, - name: &str, -) -> (track::Producer, track::Subscriber) { - let answer = async { - let request = moq_clock::timeout(TIMEOUT, dynamic.requested_track()) - .await - .expect("the request never reached the publisher") - .unwrap(); - request.accept(track::Info::default().with_timescale(moq_net::Timescale::MILLI)) - }; - let subscribing = moq_clock::timeout(TIMEOUT, remote.track(name).unwrap().subscribe(None)); - let (producer, subscriber) = futures::future::join(answer, subscribing).await; - let subscriber = subscriber.expect("subscribe hung").expect("subscribe failed"); - (producer, subscriber) -} - -async fn empty_replacement_ends_after_the_cached_group(version: &str) { - let version: Version = version.parse().unwrap(); - let publisher = produce_origin(1); - let broadcast = publisher.create_broadcast("room").unwrap(); - let mut dynamic = broadcast.dynamic(); - broadcast.announce(Default::default()).unwrap(); - - let relay = produce_origin(2); - let mut options = MockConnectOptions::new(version); - options.server_publish = Some(publisher.consume()); - options.client_subscribe = Some(relay.clone()); - let pair = connect_mock(options).await; - - let consumer = relay.consume(); - moq_clock::timeout(TIMEOUT, consumer.routed("room")) - .await - .unwrap() - .unwrap(); - let remote = moq_clock::timeout(TIMEOUT, consumer.request_broadcast("room", None)) - .await - .unwrap() - .unwrap(); - - // The first producer writes group 0, which the relay caches. - let (first, mut subscriber) = serve(&remote, &mut dynamic, "feed").await; - let mut group = first.append_group().unwrap(); - group.write_frame(Timestamp::ZERO, &b"first"[..]).unwrap(); - group.finish().unwrap(); - let group = moq_clock::timeout(TIMEOUT, subscriber.recv_group()) - .await - .unwrap() - .unwrap() - .unwrap(); - assert_eq!(group.sequence, 0, "{version:?}"); - drop((group, subscriber)); - - // The relay's copy goes idle and cancels upstream; the handler finishes and lets go. - moq_clock::timeout(TIMEOUT, support::unwanted(first.demand())) - .await - .unwrap() - .unwrap(); - first.finish().unwrap(); - drop(first); - - // The idle copy, still holding group 0, subscribes again, and the replacement finishes - // before writing anything. - let (second, mut subscriber) = serve(&remote, &mut dynamic, "feed").await; - second.finish().unwrap(); - assert_eq!(second.end(), Some(track::Position::group(1)), "{version:?}"); - - // Its own readers end at once: the earlier producer's groups never arrive here. - let mut local = second.subscribe(None); - let ended = moq_clock::timeout(TIMEOUT, local.recv_group()) - .await - .expect("a local reader of the replacement never ended"); - assert!(matches!(ended, Ok(None)), "{version:?}"); - - // The relay's copy ends after the group it cached, and its session stays up. - loop { - match moq_clock::timeout(TIMEOUT, subscriber.recv_group()) - .await - .expect("the replacement never ended") - { - Ok(Some(_)) => {} - Ok(None) => break, - Err(err) => panic!("{version:?}: the replacement's end failed the subscription: {err}"), - } - } - if let Ok(err) = moq_clock::timeout(TIMEOUT, pair.client.closed()).await { - panic!("{version:?}: the relay's session closed: {err}"); - } -} - -#[moq_clock::test] -async fn empty_replacement_lite_06() { - empty_replacement_ends_after_the_cached_group("moq-lite-06").await; -} - -#[moq_clock::test] -async fn empty_replacement_lite_07() { - empty_replacement_ends_after_the_cached_group("moq-lite-07-wip").await; -} diff --git a/rs/moq-net/tests/integration/main.rs b/rs/moq-net/tests/integration/main.rs index 99e9941f634..d3610714b54 100644 --- a/rs/moq-net/tests/integration/main.rs +++ b/rs/moq-net/tests/integration/main.rs @@ -19,7 +19,6 @@ mod datagram; mod dial_split_horizon; mod drained_claim; mod empty_group; -mod empty_replacement; mod extensions; mod fetch_cancel; mod fetch_caps; diff --git a/rs/moq-net/tests/integration/route_change.rs b/rs/moq-net/tests/integration/route_change.rs index e0c3605c4d5..77a9b84c540 100644 --- a/rs/moq-net/tests/integration/route_change.rs +++ b/rs/moq-net/tests/integration/route_change.rs @@ -1054,12 +1054,7 @@ async fn producer_replaced(version: &str) { .expect("the track was not requested again") .unwrap(); - // The application owns where the replacement starts: an append would reuse the first - // producer's sequences, so it is refused, and the replacement continues explicitly. - assert!( - matches!(second.append_group(), Err(Error::Duplicate)), - "{version}: the replacement restarted its sequences" - ); + // The application owns where the replacement starts, so it continues explicitly. let mut group = second.create_group(moq_net::group::Info { sequence: 2 }).unwrap(); for frame in 0..FRAMES { group.write_frame(Timestamp::ZERO, payload(2, frame)).unwrap(); From c099d1e32501127e79b4957e92fbea7aa3f3b769 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 08:31:15 -0700 Subject: [PATCH 05/16] test(net): drop an unused mut Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/model/broadcast.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rs/moq-net/src/model/broadcast.rs b/rs/moq-net/src/model/broadcast.rs index cd93344e032..b8e640005d5 100644 --- a/rs/moq-net/src/model/broadcast.rs +++ b/rs/moq-net/src/model/broadcast.rs @@ -1797,7 +1797,7 @@ mod test { let consumer = producer.consume(); let track = consumer.track("media").unwrap().subscribe(None); - let mut first = dynamic.assert_request().accept(crate::track::Info::timed()); + let first = dynamic.assert_request().accept(crate::track::Info::timed()); let mut group = first.create_group(crate::group::Info { sequence: 8 }).unwrap(); group.write_frame(at(1000), b"a".as_slice()).unwrap(); group.finish().unwrap(); From 798deade481fc1dd5c43281c4f3e36d55df388e0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 08:35:01 -0700 Subject: [PATCH 06/16] style(stats): rustfmt the merged imports Co-Authored-By: Claude Opus 5.5 --- rs/moq-stats/src/consume.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rs/moq-stats/src/consume.rs b/rs/moq-stats/src/consume.rs index 570f9a8c3f8..0238a64637a 100644 --- a/rs/moq-stats/src/consume.rs +++ b/rs/moq-stats/src/consume.rs @@ -4,8 +4,8 @@ use moq_net::stats::{Role, Tier}; use moq_net::{AsPath, broadcast}; use crate::{ - Encoding, GroupFrame, PrefixFrame, PresenceFrame, Result, SessionsFrame, TotalsFrame, TrafficFrame, - group_track, prefix_track, presence_track, sessions_track, totals_track, traffic_track, + Encoding, GroupFrame, PrefixFrame, PresenceFrame, Result, SessionsFrame, TotalsFrame, TrafficFrame, group_track, + prefix_track, presence_track, sessions_track, totals_track, traffic_track, }; /// Configuration for a [`Consumer`]. Construct with [`Config::new`] From 4ef184ed872048e3d2d992c7d6dacfe38594c0ec Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 10:12:23 -0700 Subject: [PATCH 07/16] feat(stats): number groups by their publish interval A group's sequence is its timestamp divided by the publish interval, so the interval is the timescale and a recreated track numbers its groups past its predecessor's with nothing kept per name. A track opens at most one group per interval: a snapshot due in an interval that already has one waits for the next, which keeps every track at or behind the clock. This replaces the per-broadcast group counter. Co-Authored-By: Claude Opus 5.5 --- doc/concept/stats.md | 21 ++++-- rs/moq-stats/src/produce.rs | 141 +++++++++++++++++++----------------- 2 files changed, 89 insertions(+), 73 deletions(-) diff --git a/doc/concept/stats.md b/doc/concept/stats.md index 1ec9dba7a1a..512989f4d88 100644 --- a/doc/concept/stats.md +++ b/doc/concept/stats.md @@ -140,8 +140,9 @@ returns within the linger keeps its broadcast and epoch, so viewer churn doesn't unannounce and re-announce it across the mesh, and its totals continue. Once the linger elapses with the group still empty, the broadcast is unannounced and its counters dropped; a group that returns later announces a -new epoch counted from zero. Within one epoch, group numbers keep increasing -across recreated tracks and may have gaps. +new epoch counted from zero. Group numbers count publish intervals (see +[Frames](#frames)), so they keep increasing across recreated tracks and have +gaps. ## Tracks @@ -198,8 +199,8 @@ is empty or refused; a reader that knows a group's name can still subscribe to it while it is listed, and is refused otherwise. A group that returns after its linger is a new instance of its track: its -frames carry a new `epoch` and count from zero, and its group numbers continue -past the old instance's. A frame too large to cache ends only that group's +frames carry a new `epoch` and count from zero, and its group numbers, which +count publish intervals, continue past the old instance's. A frame too large to cache ends only that group's track. ## Frames @@ -216,6 +217,12 @@ frame too large to cache (32 MB) ends its track with a `FRAME_TOO_LARGE` error. The `.z` flavor meets the same limit by its uncompressed size, since a reader inflates the whole group. +A group's number is the time it opened divided by the interval, so the +interval is its timescale, and a track opens at most one group per interval: a +snapshot due in an interval that already has one, such as a joiner's refresh, +waits for the next. A track recreated later numbers its groups past every +group its earlier instance wrote. + ### Totals ```json @@ -279,9 +286,9 @@ anyone holds the track. A track held again starts over. Wire bytes start at zero on every hold, including the first. Other presence fields seed from the root's current live-session counters; they do not recover closed roots from before the hold. Node totals keep the full group epoch's -history. Each recreated track starts a fresh snapshot group using the -broadcast's shared, increasing group sequence, so relay caches can distinguish -it from an earlier hold. Plain and compressed tracks share that allocator. +history. Each recreated track starts a fresh snapshot group numbered by its +publish interval, past any group of an earlier hold, so relay caches can +distinguish the two. ### Prefix diff --git a/rs/moq-stats/src/produce.rs b/rs/moq-stats/src/produce.rs index 73fa913e2de..b61a3e879df 100644 --- a/rs/moq-stats/src/produce.rs +++ b/rs/moq-stats/src/produce.rs @@ -1,7 +1,6 @@ //! The publishing half: drain a [`Registry`] on an interval into stats tracks. use std::collections::{HashMap, HashSet}; -use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::{Arc, Weak}; use std::time::Duration; @@ -44,7 +43,9 @@ pub struct Config { /// nest under a shared region key. An empty path is treated as unset. /// Default none. pub node: Option, - /// How long the publish task waits between drains. Default 1s. + /// How long the publish task waits between drains. Default 1s. Non-zero: + /// it is the timescale of every track's group numbers, so a track opens at + /// most one group per interval. pub interval: Duration, /// How many leading broadcast-path segments to use as a grouping key. /// @@ -500,32 +501,34 @@ impl Serialize for Frame { } } -/// Writes encoded snapshots with group numbers from its group broadcast's -/// allocator. The allocator outlives on-demand tracks, so a recreated track -/// resumes past any group a subscriber has cached, without retaining a -/// tombstone per requested name. A new group broadcast is a new epoch, so its -/// allocator starts over. +/// Writes encoded snapshots, numbering each group by the publish interval it +/// opens in: its timestamp divided by the interval. A recreated track (an +/// on-demand track asked for again, or a group returning after its linger) +/// opens in a later interval than its predecessor's last group, so it resumes +/// past any group a subscriber has cached, with nothing kept for names that +/// left. A track opens at most one group per interval: a snapshot due in an +/// interval that already has one waits for the next. /// -/// The shared allocator is why this wraps the encoder rather than a +/// The numbering is why this wraps the encoder rather than a /// `moq_json::snapshot::Producer`, which numbers its own groups. struct Snapshot { track: track::Producer, encoder: moq_json::snapshot::Encoder, group: Option, - sequence: Arc, + interval: Duration, /// The sequence of the newest group written, which a refresh has to pass. largest: Option, deltas: bool, } impl Snapshot { - fn new(track: track::Producer, config: moq_json::snapshot::Config, sequence: Arc) -> Self { + fn new(track: track::Producer, config: moq_json::snapshot::Config, interval: Duration) -> Self { Self { track, deltas: config.deltas, encoder: moq_json::snapshot::Encoder::new(config), group: None, - sequence, + interval, largest: None, } } @@ -567,16 +570,21 @@ impl Snapshot { Err(err) => return Err(err), }; let len = frame.payload.len(); + let timestamp = moq_net::Timestamp::now(); if frame.keyframe { + let sequence = (std::time::Duration::from(timestamp).as_nanos() / self.interval.as_nanos()) as u64; + // The interval already has its group: the uncommitted frame is encoded again next tick. + if self.largest.is_some_and(|largest| sequence <= largest) { + return Ok(None); + } if let Some(mut group) = self.group.take() { group.finish()?; } - let sequence = self.sequence.fetch_add(1, Ordering::Relaxed); self.group = Some(self.track.create_group(moq_net::group::Info { sequence })?); self.largest = Some(sequence); } let group = self.group.as_mut().expect("a delta follows a snapshot"); - if let Err(err) = group.write_frame(moq_net::Timestamp::now(), frame.payload.clone()) { + if let Err(err) = group.write_frame(timestamp, frame.payload.clone()) { let _ = self.group.take().unwrap().finish(); return Err(err.into()); } @@ -628,21 +636,21 @@ fn compressed_info() -> track::Info { } impl TrackPair { - fn create(broadcast: &broadcast::Producer, name: &str, sequence: Arc) -> Result { + fn create(broadcast: &broadcast::Producer, name: &str, interval: Duration) -> Result { let plain_track = broadcast.create_track(name, info())?; let compressed_track = broadcast.create_track(format!("{name}{COMPRESSED_SUFFIX}").as_str(), compressed_info())?; - Ok(Self::from_tracks(plain_track, compressed_track, sequence)) + Ok(Self::from_tracks(plain_track, compressed_track, interval)) } - fn from_tracks(plain_track: track::Producer, compressed_track: track::Producer, sequence: Arc) -> Self { + fn from_tracks(plain_track: track::Producer, compressed_track: track::Producer, interval: Duration) -> Self { let plain_config = moq_json::snapshot::Config::default().with_deltas(false); let mut compressed_config = moq_json::snapshot::Config::default(); compressed_config.compression = moq_json::Compression::Deflate; Self { - plain: Snapshot::new(plain_track, plain_config, sequence.clone()), - compressed: Snapshot::new(compressed_track, compressed_config, sequence), + plain: Snapshot::new(plain_track, plain_config, interval), + compressed: Snapshot::new(compressed_track, compressed_config, interval), frame: Frame::default(), #[cfg(feature = "bench")] held: 0, @@ -731,7 +739,7 @@ impl PendingPair { self, broadcast: &broadcast::Producer, name: &str, - sequence: Arc, + interval: Duration, ) -> Result, moq_net::Error> { let PendingPair { plain, compressed } = self; let plain_track = match plain { @@ -742,7 +750,7 @@ impl PendingPair { Some(request) => request.accept(compressed_info()), None => broadcast.create_track(format!("{name}{COMPRESSED_SUFFIX}").as_str(), compressed_info())?, }; - Ok(TrackPair::from_tracks(plain_track, compressed_track, sequence)) + Ok(TrackPair::from_tracks(plain_track, compressed_track, interval)) } fn reject(self, err: moq_net::Error) { @@ -771,7 +779,7 @@ impl PendingPair { /// One frame type's live per-tier pairs and the requests parked for them; the /// traffic tracks and the sessions tracks each form one family. struct TrackFamily { - sequence: Arc, + interval: Duration, tracks: HashMap>, /// Valid-shaped requests awaiting quota, keyed by plain name and bounded by /// [`MAX_PARKED_REQUESTS`] across both families. Adopted as the quota @@ -780,9 +788,9 @@ struct TrackFamily { } impl TrackFamily { - fn new(sequence: Arc) -> Self { + fn new(interval: Duration) -> Self { Self { - sequence, + interval, tracks: HashMap::new(), parked: HashMap::new(), } @@ -807,8 +815,8 @@ impl TrackFamily { ) { if !self.tracks.contains_key(name) { let result = match self.parked.remove(name) { - Some(pending) => pending.adopt(broadcast, name, self.sequence.clone()), - None => TrackPair::create(broadcast, name, self.sequence.clone()), + Some(pending) => pending.adopt(broadcast, name, self.interval), + None => TrackPair::create(broadcast, name, self.interval), }; match result { Ok(pair) => { @@ -913,7 +921,7 @@ impl TrackFamily { pending.reject(moq_net::Error::NotFound); return; } - match pending.adopt(broadcast, &plain, self.sequence.clone()) { + match pending.adopt(broadcast, &plain, self.interval) { Ok(mut pair) => { // Hold the subscription open with zeros until the tier records. pair.publish(&plain); @@ -1219,12 +1227,12 @@ fn group_info() -> track::Info { } impl GroupTrack { - fn create(broadcast: &broadcast::Producer, group: &str, sequence: Arc) -> Result { + fn create(broadcast: &broadcast::Producer, group: &str, interval: Duration) -> Result { let name = crate::group_track(group); let track = broadcast.create_track(name.as_str(), group_info())?; Ok(Self { name, - snapshot: Snapshot::new(track, moq_json::snapshot::Config::default(), sequence), + snapshot: Snapshot::new(track, moq_json::snapshot::Config::default(), interval), body: GroupBody { // A new instance of the name: readers keep its counters apart from // an earlier instance's. @@ -1354,9 +1362,9 @@ impl GroupPublisher { }; tracing::debug!(advertised = %advertised, "stats: publishing broadcast"); - let sequence = Arc::new(AtomicU64::new(0)); - let mut traffic = TrackFamily::new(sequence.clone()); - let mut sessions = TrackFamily::new(sequence.clone()); + let interval = task.interval; + let mut traffic = TrackFamily::new(interval); + let mut sessions = TrackFamily::new(interval); let tiers = task.layout == Layout::Broadcast; // The totals and, outside the track layout, the default tier's tracks @@ -1365,7 +1373,7 @@ impl GroupPublisher { let tier = Tier::default(); for role in [Role::Publisher, Role::Subscriber] { let name = traffic_track(&tier, role, Encoding::Plain); - match TrackPair::create(&broadcast, &name, sequence.clone()) { + match TrackPair::create(&broadcast, &name, interval) { Ok(pair) => { traffic.tracks.insert(name, pair); } @@ -1376,7 +1384,7 @@ impl GroupPublisher { } } let name = sessions_track(&tier, Encoding::Plain); - match TrackPair::create(&broadcast, &name, sequence.clone()) { + match TrackPair::create(&broadcast, &name, interval) { Ok(pair) => { sessions.tracks.insert(name, pair); } @@ -1387,7 +1395,7 @@ impl GroupPublisher { } } let name = totals_track(Encoding::Plain); - let totals = match TrackPair::create(&broadcast, &name, sequence) { + let totals = match TrackPair::create(&broadcast, &name, interval) { Ok(pair) => TotalsTrack { pair, sums: Cumulative::new(), @@ -1522,7 +1530,7 @@ impl GroupPublisher { }); if let Some(depth) = *track_depth { let key = group_key(entry.path.as_str(), depth); - if let Some(track) = Self::group_track(broadcast, group_tracks, &traffic.sequence, key) { + if let Some(track) = Self::group_track(broadcast, group_tracks, traffic.interval, key) { track.add_traffic(entry, slots.publisher.moved || slots.subscriber.moved); } } @@ -1547,7 +1555,7 @@ impl GroupPublisher { } if let Some(depth) = *track_depth { let key = group_key(entry.label.as_str(), depth); - if let Some(track) = Self::group_track(broadcast, group_tracks, &traffic.sequence, key) { + if let Some(track) = Self::group_track(broadcast, group_tracks, traffic.interval, key) { track.add_sessions(&entry.label, &entry.tier, entry.presence, entry.closed); } } @@ -1558,11 +1566,11 @@ impl GroupPublisher { fn group_track<'a>( broadcast: &broadcast::Producer, tracks: &'a mut HashMap, - sequence: &Arc, + interval: Duration, key: &str, ) -> Option<&'a mut GroupTrack> { if !tracks.contains_key(key) { - match GroupTrack::create(broadcast, key, sequence.clone()) { + match GroupTrack::create(broadcast, key, interval) { Ok(track) => { tracks.insert(key.to_string(), track); } @@ -1698,7 +1706,7 @@ impl GroupPublisher { return; } let name = presence_track(&label, Encoding::Plain); - let pair = match pending.adopt(&self.broadcast, &name, self.traffic.sequence.clone()) { + let pair = match pending.adopt(&self.broadcast, &name, self.traffic.interval) { Ok(pair) => pair, Err(err) => { tracing::warn!(?err, name, "stats: failed to adopt presence track"); @@ -1733,7 +1741,7 @@ impl GroupPublisher { return; } let name = prefix_track(&path, Encoding::Plain); - let pair = match pending.adopt(&self.broadcast, &name, self.traffic.sequence.clone()) { + let pair = match pending.adopt(&self.broadcast, &name, self.traffic.interval) { Ok(pair) => pair, Err(err) => { tracing::warn!(?err, name, "stats: failed to adopt prefix track"); @@ -1885,17 +1893,15 @@ mod tests { moq_net::track::Info::default().with_timescale(moq_net::Timescale::MILLI) } - /// A joiner once the deltas outweigh the compressed track's ratio gets a fresh snapshot group on - /// the next tick, even of an unchanged value, and deltas resume in it. - #[test] - fn a_joiner_over_the_ratio_gets_a_snapshot_next_tick() { + /// A joiner once the deltas outweigh the compressed track's ratio gets a fresh snapshot group in + /// the next interval, even of an unchanged value, and deltas resume in it. + #[tokio::test(start_paused = true)] + async fn a_joiner_over_the_ratio_gets_a_snapshot_next_interval() { + let interval = Duration::from_secs(1); let broadcast = moq_net::broadcast::Info::new().produce(); let track = broadcast.create_track("t.json.z", compressed_info()).unwrap(); - let mut snapshot = Snapshot::>::new( - track.clone(), - moq_json::snapshot::Config::default(), - Arc::new(AtomicU64::new(0)), - ); + let mut snapshot = + Snapshot::>::new(track.clone(), moq_json::snapshot::Config::default(), interval); let frame = |n| Frame { entries: vec![(PathOwned::from("a"), n)], }; @@ -1905,17 +1911,22 @@ mod tests { for n in 0..10 { snapshot.update(&frame(n)).unwrap(); } - assert_eq!(snapshot.largest, Some(0)); + let first = snapshot.largest.unwrap(); let _late = track.subscribe(None); - assert_eq!(wanted(&track), Some(1), "the join asks for a fresh group"); + assert_eq!(wanted(&track), Some(first + 1), "the join asks for a fresh group"); + + // The interval already has its group, so the snapshot waits for the next one. + assert!(snapshot.update(&frame(9)).unwrap().is_none()); + assert_eq!(snapshot.largest, Some(first)); + tokio::time::advance(interval).await; assert!( snapshot.update(&frame(9)).unwrap().is_some(), "restated though unchanged" ); - assert_eq!(snapshot.largest, Some(1)); + assert_eq!(snapshot.largest, Some(first + 1), "numbered by its interval"); assert!(snapshot.update(&frame(10)).unwrap().is_some()); - assert_eq!(snapshot.largest, Some(1), "the next change is a delta"); + assert_eq!(snapshot.largest, Some(first + 1), "the next change is a delta"); } #[tokio::test(start_paused = true)] @@ -1925,7 +1936,7 @@ mod tests { let broadcast = moq_net::broadcast::Info::new().produce(); let _dynamic = broadcast.dynamic(); let consumer = broadcast.consume(); - let mut family = TrackFamily::::new(Arc::new(AtomicU64::new(0))); + let mut family = TrackFamily::::new(Duration::from_secs(1)); let mut requested = HashSet::new(); let subscribing = consumer.track(name).unwrap().subscribe(None); family.adopt_pair( @@ -1939,6 +1950,8 @@ mod tests { let floor = group.sequence + 1; drop(group); drop(subscriber); + // A drain reclaims before it adopts, so a pair it reclaims last wrote in an earlier one. + tokio::time::advance(Duration::from_secs(1)).await; family.reclaim(&mut requested); assert!(family.tracks.is_empty()); @@ -1977,7 +1990,8 @@ mod tests { pumping_at(drain, Instant::now(), future).await } - /// Drive `drain` at `now` until `future` resolves. + /// Drive `drain` at `now` until `future` resolves, one interval of frame time + /// per drain, as the publish task would. async fn pumping_at(drain: &mut Drain, now: Instant, future: impl std::future::Future) -> T { let mut future = std::pin::pin!(future); for _ in 0..1000 { @@ -1986,7 +2000,7 @@ mod tests { if let Some(result) = futures::FutureExt::now_or_never(future.as_mut()) { return result; } - tokio::task::yield_now().await; + tokio::time::advance(drain.task.interval).await; } panic!("protocol request did not settle while pumping the stats producer"); } @@ -2036,7 +2050,8 @@ mod tests { depth: 0, layout: Layout::Broadcast, linger: DEFAULT_LINGER, - interval: Duration::from_secs(1), + // Hundreds of drains, each an interval apart, stay inside the relays' cache. + interval: Duration::from_millis(10), }) .unwrap(); let (middle, middle_cache) = relay(); @@ -2159,7 +2174,7 @@ mod tests { let broadcast = moq_net::broadcast::Info::new().produce(); let consumer = broadcast.consume(); - let mut pair = TrackPair::::create(&broadcast, "t.json", Arc::new(AtomicU64::new(0))).unwrap(); + let mut pair = TrackPair::::create(&broadcast, "t.json", Duration::from_secs(1)).unwrap(); let mut publish = |key: &str, value: u64| { pair.frame.entries.push((PathOwned::from(key.to_string()), value)); pair.publish("t.json"); @@ -2791,8 +2806,6 @@ mod tests { assert_eq!(read_last_frame(&acme, "publisher.json").await["acme/live"].bytes, 50); assert_eq!(default_totals(&acme).await.publisher.bytes, 150, "totals continue"); assert_eq!(default_totals(&sessions).await.sessions.sessions_started, 2); - let mut track = subscribe(&acme, "publisher.json").await; - let old_sequence = track.next_group().await.expect("ok").expect("group").sequence; // The return re-armed the linger: it unannounces only once it elapses // again with the group still empty. @@ -2835,12 +2848,6 @@ mod tests { 25, "a new epoch's totals count from zero" ); - let mut track = subscribe(&acme, "publisher.json").await; - let group = track.next_group().await.expect("ok").expect("group"); - assert!( - group.sequence < old_sequence, - "a new epoch numbers its groups from zero" - ); } #[tokio::test(start_paused = true)] @@ -3062,6 +3069,7 @@ mod tests { // The label records while held: the same subscription carries it. let _session = registry.tier(Tier::default()).session(""); + tokio::time::advance(Duration::from_secs(1)).await; tick(); assert_eq!(try_presence(&mut track).expect("recorded")[""].active(), 1); @@ -3331,6 +3339,7 @@ mod tests { assert_eq!(last_prefix(&mut track), Some(PrefixFrame::new()), "held at zero"); let _feed = feed(®istry, Tier::default(), "acme/live", true, 1, 7).await; + tokio::time::advance(Duration::from_secs(1)).await; tick(); let frame = last_prefix(&mut track).expect("recorded"); assert_eq!(frame[""].children["live"].publisher.bytes, 7); From 3b7708ec427dab744fa4bd3032e2c13f87ca252b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 10:14:31 -0700 Subject: [PATCH 08/16] style(stats): rustfmt Co-Authored-By: Claude Opus 5.5 --- rs/moq-stats/src/produce.rs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/rs/moq-stats/src/produce.rs b/rs/moq-stats/src/produce.rs index 3cd9b9752da..c0ff7f1630f 100644 --- a/rs/moq-stats/src/produce.rs +++ b/rs/moq-stats/src/produce.rs @@ -1900,8 +1900,7 @@ mod tests { let interval = Duration::from_secs(1); let broadcast = moq_net::broadcast::Info::new().produce(); let track = broadcast.create_track("t.json.z", compressed_info()).unwrap(); - let mut snapshot = - Snapshot::>::new(track.clone(), moq_json::snapshot::Config::default(), interval); + let mut snapshot = Snapshot::>::new(track.clone(), moq_json::snapshot::Config::default(), interval); let frame = |n| Frame { entries: vec![(PathOwned::from("a"), n)], }; From 1a9999c892db4540bb68c31f8912174a4e60ac44 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 11:23:48 -0700 Subject: [PATCH 09/16] fix(stats): number and stamp groups from the drain's instant Every track in one drain now shares the instant Drain::publish was given, instead of reading the clock per frame. The producer_tick bench advances that instant by the interval each tick, so a changed tick opens a group again rather than being held to the previous interval. Co-Authored-By: Claude Opus 5.5 --- rs/moq-stats/src/produce.rs | 91 +++++++++++++++++-------------- rs/moq-stats/src/produce_bench.rs | 12 +++- 2 files changed, 60 insertions(+), 43 deletions(-) diff --git a/rs/moq-stats/src/produce.rs b/rs/moq-stats/src/produce.rs index c0ff7f1630f..d910ced6840 100644 --- a/rs/moq-stats/src/produce.rs +++ b/rs/moq-stats/src/produce.rs @@ -437,7 +437,7 @@ impl Drain { // Serve consumer requests for tracks no drain has created yet: a // tier's tracks appear lazily on its first traffic, and a label's // presence track only while requested. See `serve_requests`. - group.serve_requests(&self.report, depth, self.tick); + group.serve_requests(&self.report, depth, self.tick, now); } } @@ -537,14 +537,15 @@ impl Snapshot { self.track.demand() } - /// Write one frame. `Some(len)` is the payload written; `None` means the - /// value was unchanged, or the track already ended, and nothing was written. + /// Write one frame, stamped and numbered by the drain's `now`. `Some(len)` is the payload + /// written; `None` means the value was unchanged, the interval already has a group for a + /// snapshot, or the track already ended, and nothing was written. /// /// A frame over the cache limit aborts the track with /// [`moq_net::Error::FrameTooLarge`]: skipping it would leave readers on the /// previous snapshot with no sign it stopped tracking. A compressed frame /// meets the same limit by its uncompressed size, which the encoder refuses. - fn update(&mut self, value: &T) -> moq_json::Result> { + fn update(&mut self, value: &T, now: Instant) -> moq_json::Result> { if self.track.is_closed() { return Ok(None); } @@ -570,7 +571,7 @@ impl Snapshot { Err(err) => return Err(err), }; let len = frame.payload.len(); - let timestamp = moq_net::Timestamp::now(); + let timestamp = moq_net::Timestamp::from(now); if frame.keyframe { let sequence = (std::time::Duration::from(timestamp).as_nanos() / self.interval.as_nanos()) as u64; // The interval already has its group: the uncommitted frame is encoded again next tick. @@ -670,16 +671,16 @@ impl TrackPair { /// and clear them for the next drain; moq-json skips unchanged values. A /// flavor that fails ends alone: DEFLATE can shrink a frame too large for /// the plain track, or grow one that just fits. - fn publish(&mut self, name: &str) { + fn publish(&mut self, name: &str, now: Instant) { self.frame.entries.sort_unstable_by(|a, b| a.0.cmp(&b.0)); - let plain = match self.plain.update(&self.frame) { + let plain = match self.plain.update(&self.frame, now) { Ok(len) => len, Err(err) => { tracing::warn!(?err, name, "stats: failed to write frame"); None } }; - let compressed = match self.compressed.update(&self.frame) { + let compressed = match self.compressed.update(&self.frame, now) { Ok(len) => len, Err(err) => { tracing::warn!(?err, name, "stats: failed to write compressed frame"); @@ -838,9 +839,9 @@ impl TrackFamily { /// Publish every pair's pending frame, an empty one when the drain had /// nothing for it, so a track whose last entry closed transitions to `{}` /// exactly once. - fn flush(&mut self) { + fn flush(&mut self, now: Instant) { for (name, pair) in self.tracks.iter_mut() { - pair.publish(name); + pair.publish(name, now); } } @@ -887,7 +888,7 @@ impl TrackFamily { /// later drain, so a valid-shaped request is never terminally rejected /// merely for arriving while the quota was full. Entries whose every /// requester left are dropped instead of adopted. - fn adopt_parked(&mut self, broadcast: &broadcast::Producer, requested: &mut HashSet) { + fn adopt_parked(&mut self, broadcast: &broadcast::Producer, requested: &mut HashSet, now: Instant) { let noop = kio::Waiter::noop(); let mut parked = std::mem::take(&mut self.parked); parked.retain(|plain, pending| { @@ -897,7 +898,7 @@ impl TrackFamily { if requested.len() >= MAX_REQUESTED_TRACKS { return true; } - self.adopt_pair(broadcast, requested, plain.clone(), std::mem::take(pending)); + self.adopt_pair(broadcast, requested, plain.clone(), std::mem::take(pending), now); false }); self.parked = parked; @@ -912,6 +913,7 @@ impl TrackFamily { requested: &mut HashSet, plain: String, pending: PendingPair, + now: Instant, ) { // A request racing the pair's creation is fulfilled by `create_track` // (queued) or adopted by [`Self::push`] (parked), so a live pair here @@ -924,7 +926,7 @@ impl TrackFamily { match pending.adopt(broadcast, &plain, self.interval) { Ok(mut pair) => { // Hold the subscription open with zeros until the tier records. - pair.publish(&plain); + pair.publish(&plain, now); self.tracks.insert(plain.clone(), pair); requested.insert(plain); } @@ -1084,7 +1086,7 @@ struct PresenceTrack { } impl PresenceTrack { - fn publish(&mut self) { + fn publish(&mut self, now: Instant) { self.sums.fill(&mut self.pair.frame); for (tier, presence) in &mut self.pair.frame.entries { if let Some(&(tx, rx)) = self.wire_start.get(tier) { @@ -1092,7 +1094,7 @@ impl PresenceTrack { presence.wire_rx = presence.wire_rx.saturating_sub(rx); } } - self.pair.publish(&self.name); + self.pair.publish(&self.name, now); } } @@ -1159,7 +1161,7 @@ impl PrefixTrack { /// Publish each tier's sums, dropping children this drain carried no row /// for. - fn publish(&mut self, tick: u64) { + fn publish(&mut self, tick: u64, now: Instant) { for (tier, sums) in self.tiers.iter_mut() { sums.children.retain(|_, child| child.seen == tick); self.pair.frame.entries.push(( @@ -1176,7 +1178,7 @@ impl PrefixTrack { }, )); } - self.pair.publish(&self.name); + self.pair.publish(&self.name, now); } } @@ -1312,14 +1314,14 @@ impl GroupTrack { /// Write this drain's frame; moq-json skips an unchanged one. A frame too /// large to cache ends this group's track alone. - fn publish(&mut self) { + fn publish(&mut self, now: Instant) { let tiers = &mut self.body.tiers.entries; tiers.sort_unstable_by(|a, b| a.0.cmp(&b.0)); for (_, tier) in tiers.iter_mut() { tier.paths.entries.sort_unstable_by(|a, b| a.0.cmp(&b.0)); tier.labels.entries.sort_unstable_by(|a, b| a.0.cmp(&b.0)); } - if let Err(err) = self.snapshot.update(&self.body) { + if let Err(err) = self.snapshot.update(&self.body, now) { tracing::warn!(?err, name = self.name, "stats: failed to write group frame"); } } @@ -1586,23 +1588,23 @@ impl GroupPublisher { /// Publish the pending frames, then drop change-detection state for /// entries this drain did not carry (the registry pruned them). fn flush(&mut self, tick: u64, now: Instant, linger: Duration) { - self.traffic.flush(); - self.sessions.flush(); + self.traffic.flush(now); + self.sessions.flush(now); for (_, track) in self.group_tracks.extract_if(|_, track| track.lingered(now, linger)) { track.finish(); } for track in self.group_tracks.values_mut() { - track.publish(); + track.publish(now); } self.totals.sums.fill(&mut self.totals.pair.frame); - self.totals.pair.publish(crate::TOTALS); + self.totals.pair.publish(crate::TOTALS, now); for track in self.presence.values_mut() { - track.publish(); + track.publish(now); } for track in self.prefixes.values_mut() { - track.publish(tick); + track.publish(tick, now); } self.local.retain(|_, tiers| { @@ -1627,7 +1629,7 @@ impl GroupPublisher { /// It is reclaimed once its last consumer leaves, so the held set follows /// live readers. A prefix track follows the same rules for a broadcast /// prefix in this group. Any other name is rejected. - fn serve_requests(&mut self, report: &Report, depth: usize, tick: u64) { + fn serve_requests(&mut self, report: &Report, depth: usize, tick: u64, now: Instant) { // Reclaim before parking and adopting, so a freed slot is usable by // this very drain. self.traffic.reclaim(&mut self.requested); @@ -1686,19 +1688,19 @@ impl GroupPublisher { } } - self.traffic.adopt_parked(&self.broadcast, &mut self.requested); - self.sessions.adopt_parked(&self.broadcast, &mut self.requested); + self.traffic.adopt_parked(&self.broadcast, &mut self.requested, now); + self.sessions.adopt_parked(&self.broadcast, &mut self.requested, now); for (label, pending) in labels { - self.adopt_presence(report, label, pending); + self.adopt_presence(report, label, pending, now); } for (path, pending) in paths { - self.adopt_prefix(report, path, pending, tick); + self.adopt_prefix(report, path, pending, tick, now); } } /// Hold a requested label's presence track, publishing its current counters /// so the subscription resolves at once. Cumulative from here on. - fn adopt_presence(&mut self, report: &Report, label: PathOwned, pending: PendingPair) { + fn adopt_presence(&mut self, report: &Report, label: PathOwned, pending: PendingPair, now: Instant) { // A request for a live track is served by the broadcast itself, so a // live pair here means the requested flavor ended and cannot update. if self.presence.contains_key(&label) { @@ -1728,13 +1730,13 @@ impl GroupPublisher { start.1 += entry.presence.wire_rx; } } - track.publish(); + track.publish(now); self.presence.insert(label, track); } /// Hold a requested prefix track, publishing its current counters so the /// subscription resolves at once. Cumulative from here on. - fn adopt_prefix(&mut self, report: &Report, path: PathOwned, pending: PendingPair, tick: u64) { + fn adopt_prefix(&mut self, report: &Report, path: PathOwned, pending: PendingPair, tick: u64, now: Instant) { // Defensive only, as in `adopt_presence`. if self.prefixes.contains_key(path.as_str()) { pending.reject(moq_net::Error::NotFound); @@ -1761,7 +1763,7 @@ impl GroupPublisher { let child = rest.as_str().split('/').next().filter(|child| !child.is_empty()); track.add(entry, child, tick); } - track.publish(tick); + track.publish(tick, now); self.prefixes.insert(path.as_str().to_string(), track); } @@ -1908,23 +1910,23 @@ mod tests { // A 7 byte snapshot and nine 7 byte deltas: past eight snapshots. for n in 0..10 { - snapshot.update(&frame(n)).unwrap(); + snapshot.update(&frame(n), moq_clock::now()).unwrap(); } let first = snapshot.largest.unwrap(); let _late = track.subscribe(None); assert_eq!(wanted(&track), Some(first + 1), "the join asks for a fresh group"); // The interval already has its group, so the snapshot waits for the next one. - assert!(snapshot.update(&frame(9)).unwrap().is_none()); + assert!(snapshot.update(&frame(9), moq_clock::now()).unwrap().is_none()); assert_eq!(snapshot.largest, Some(first)); tokio::time::advance(interval).await; assert!( - snapshot.update(&frame(9)).unwrap().is_some(), + snapshot.update(&frame(9), moq_clock::now()).unwrap().is_some(), "restated though unchanged" ); assert_eq!(snapshot.largest, Some(first + 1), "numbered by its interval"); - assert!(snapshot.update(&frame(10)).unwrap().is_some()); + assert!(snapshot.update(&frame(10), moq_clock::now()).unwrap().is_some()); assert_eq!(snapshot.largest, Some(first + 1), "the next change is a delta"); } @@ -1943,6 +1945,7 @@ mod tests { &mut requested, "idle/publisher.json".into(), PendingPair::default(), + moq_clock::now(), ); let mut subscriber = subscribing.await.unwrap(); let group = subscriber.recv_group().await.unwrap().unwrap(); @@ -1963,6 +1966,7 @@ mod tests { &mut requested, "idle/publisher.json".into(), PendingPair::default(), + moq_clock::now(), ); let mut subscriber = subscribing.await.unwrap(); let group = subscriber @@ -2176,7 +2180,7 @@ mod tests { let mut pair = TrackPair::::create(&broadcast, "t.json", Duration::from_secs(1)).unwrap(); let mut publish = |key: &str, value: u64| { pair.frame.entries.push((PathOwned::from(key.to_string()), value)); - pair.publish("t.json"); + pair.publish("t.json", moq_clock::now()); }; publish("small", 1); @@ -3794,7 +3798,14 @@ mod tests { let _held = hold.then(|| { let tick = drain.tick; let group = drain.groups.get_mut(key).expect("group"); - group.adopt_prefix(&drain.report, PathOwned::from("room0"), PendingPair::default(), tick); + let now = moq_clock::now(); + group.adopt_prefix( + &drain.report, + PathOwned::from("room0"), + PendingPair::default(), + tick, + now, + ); group.prefixes["room0"].pair.plain.track.subscribe(None) }); drain.collect(); diff --git a/rs/moq-stats/src/produce_bench.rs b/rs/moq-stats/src/produce_bench.rs index abf33eaa8dc..71ce2dee52f 100644 --- a/rs/moq-stats/src/produce_bench.rs +++ b/rs/moq-stats/src/produce_bench.rs @@ -3,6 +3,7 @@ use std::time::Duration; +use moq_clock::Instant; use moq_net::{PathOwned, origin, stats::Registry, track}; use super::{Drain, Layout, PendingPair, Task}; @@ -26,6 +27,8 @@ pub struct FrameBytes { /// A depth-0 stats publisher, driven one drain at a time. pub struct Driver { drain: Drain, + /// The last drain's instant. Each tick is one interval later, so it may open a group. + now: Instant, } impl Driver { @@ -55,13 +58,15 @@ impl Driver { }; Some(Self { drain: Drain::new(task)?, + now: moq_clock::now(), }) } - /// Drain the registry and encode every track once. + /// Drain the registry and encode every track once, one interval after the last tick. pub fn tick(&mut self) -> FrameBytes { + self.now += self.drain.task.interval; self.drain.collect(); - self.drain.publish(moq_clock::now()); + self.drain.publish(self.now); self.drain.publisher_frame_bytes() } @@ -71,7 +76,7 @@ impl Driver { pub fn hold_presence(&mut self, label: &str) -> track::Subscriber { let group = self.drain.groups.get_mut("").expect("the depth 0 broadcast"); let label = PathOwned::from(label.to_string()); - group.adopt_presence(&self.drain.report, label.clone(), PendingPair::default()); + group.adopt_presence(&self.drain.report, label.clone(), PendingPair::default(), self.now); group.presence[&label].pair.plain.track.subscribe(None) } @@ -85,6 +90,7 @@ impl Driver { PathOwned::from(path.to_string()), PendingPair::default(), tick, + self.now, ); group.prefixes[path].pair.plain.track.subscribe(None) } From 7032f4184ad58ee5effa353ab2e1b5bca97b6d2b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 11:45:17 -0700 Subject: [PATCH 10/16] test(stats): a finished group track is unlisted too #6399 reports a finished track as TracksEvent::Finished. Co-Authored-By: Claude Opus 5.5 --- rs/moq-stats/src/produce.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rs/moq-stats/src/produce.rs b/rs/moq-stats/src/produce.rs index d910ced6840..a868c154673 100644 --- a/rs/moq-stats/src/produce.rs +++ b/rs/moq-stats/src/produce.rs @@ -3870,7 +3870,7 @@ mod tests { .expect("live"); let (name, listed) = match event { broadcast::TracksEvent::Added { name, .. } => (name, true), - broadcast::TracksEvent::Removed { name } => (name, false), + broadcast::TracksEvent::Removed { name } | broadcast::TracksEvent::Finished { name } => (name, false), broadcast::TracksEvent::Synced => continue, }; if let Some(group) = crate::parse_group_track(&name) { From c4f17ae531cb60c85c966baa557e5cfc54b72b8d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 12:13:16 -0700 Subject: [PATCH 11/16] fix(stats): number each drain by its scheduled tick A late drain read the clock past an interval boundary, so the next on-time drain landed in the same interval and its plain snapshot was skipped. The closing readout of a pruned row was then lost for good. The ticker's scheduled instants are at least an interval apart, so each drain now numbers and stamps its groups from its own. Co-Authored-By: Claude Opus 5.5 --- doc/concept/stats.md | 10 +++---- rs/moq-stats/src/produce.rs | 54 +++++++++++++++++++++++++++++++++++-- 2 files changed, 57 insertions(+), 7 deletions(-) diff --git a/doc/concept/stats.md b/doc/concept/stats.md index 512989f4d88..96943896267 100644 --- a/doc/concept/stats.md +++ b/doc/concept/stats.md @@ -217,11 +217,11 @@ frame too large to cache (32 MB) ends its track with a `FRAME_TOO_LARGE` error. The `.z` flavor meets the same limit by its uncompressed size, since a reader inflates the whole group. -A group's number is the time it opened divided by the interval, so the -interval is its timescale, and a track opens at most one group per interval: a -snapshot due in an interval that already has one, such as a joiner's refresh, -waits for the next. A track recreated later numbers its groups past every -group its earlier instance wrote. +A group's number is its publish's scheduled time divided by the interval, so +the interval is its timescale. Publishes are scheduled at least an interval +apart, even after a late one, so each opens its groups in an interval of its +own, and a track recreated later numbers its groups past every group its +earlier instance wrote. ### Totals diff --git a/rs/moq-stats/src/produce.rs b/rs/moq-stats/src/produce.rs index a868c154673..f76bf79bebb 100644 --- a/rs/moq-stats/src/produce.rs +++ b/rs/moq-stats/src/produce.rs @@ -290,7 +290,9 @@ impl Task { ticker.set_missed_tick_behavior(moq_clock::MissedTickBehavior::Delay); loop { - ticker.tick().await; + // Scheduled instants are at least an interval apart, even after a late tick, + // so each drain gets an interval of its own to open its groups in. + let scheduled = ticker.tick().await; if weak.upgrade().is_none() { drain.finish(); @@ -298,7 +300,7 @@ impl Task { } drain.collect(); - drain.publish(moq_clock::now()); + drain.publish(scheduled); } } fn node(&self) -> Option<&str> { @@ -2450,6 +2452,54 @@ mod tests { assert_eq!(announced(&origin).await.0, ".stats/node"); } + /// A drain that runs late is numbered by its schedule, so the next on-time + /// drain still opens its own group and publishes its readout on the plain + /// track, which writes every frame as a fresh group. + #[tokio::test(start_paused = true)] + async fn a_late_drain_leaves_the_next_its_own_interval() { + // Schedule the drains a millisecond before an interval boundary. + let phase = Duration::from(Timestamp::from(moq_clock::now())).as_millis() % 1000; + tokio::time::advance(Duration::from_millis(((1999 - phase) % 1000) as u64)).await; + + let origin = produce_origin(); + let producer = node_producer(&origin); + let session = producer.registry().tier(Tier::default()).session("acme/live"); + let counters = moq_net::transport::Wire::new(); + let _wire = session.wire(counters.clone()); + let step = async |millis: u64| { + tokio::time::advance(Duration::from_millis(millis)).await; + for _ in 0..4 { + tokio::task::yield_now().await; + } + }; + // The first drain announces the broadcast. + step(0).await; + let broadcast = origin + .consume() + .request_broadcast(".stats/node/sjc", None) + .await + .expect("resolve"); + let mut totals = subscribe(&broadcast, crate::TOTALS).await; + let mut sent = |bytes: u64| { + counters.set_sent_bytes(bytes); + let mut last = None; + while let Some(frame) = try_next_frame(&mut totals) { + last = Some(frame); + } + last.map(|frame| { + let frame: TotalsFrame = serde_json::from_slice(&frame.payload).expect("json parse"); + frame.get("").expect("the default tier").sessions.wire_tx + }) + }; + + sent(1); + // Two milliseconds late, past the boundary into the next drain's interval. + step(1002).await; + assert_eq!(sent(2), Some(1)); + step(998).await; + assert_eq!(sent(3), Some(2), "the on-time drain publishes its readout"); + } + /// A producer on `origin` publishing as node `sjc` at depth 0. fn node_producer(origin: &origin::Producer) -> Producer { Producer::new( From 637cea1dfa7f65c1cb6efe6d8fd3ab6443bbdef9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 12:22:23 -0700 Subject: [PATCH 12/16] docs: drop stale predecessor-sequence claims Co-Authored-By: Claude Opus 5.5 --- js/net/src/track.ts | 8 ++++---- rs/moq-stats/src/produce.rs | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 199bfa107b1..b62823dca11 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -1372,8 +1372,9 @@ export class Producer { /** * Append a new group with the next sequence number: one past this producer's own writes. * - * Throws if an earlier producer of the name in this broadcast already wrote that sequence. - * A replacement picks where it starts by writing its first group with {@link writeGroup}. + * The broadcast remembers nothing of an earlier producer of the name, so a replacement + * starts at 0. Since a name always means the same content, a replacement continues past + * its predecessor's groups with {@link writeGroup}. * * Its frames must match the track's {@link Info.timescale}: a timestamp on every frame of * a timed track and none on an untimed one, or the write throws {@link TimestampMismatch}. @@ -1450,8 +1451,7 @@ export class Producer { * keep datagram payloads small (e.g. a single audio frame). Datagrams are never delivered * over stream-only transports (the WebSocket fallback). A payload over * 65535 bytes (the QUIC datagram frame ceiling) throws. An origin publisher uses this; a - * relay preserving upstream numbering uses {@link insertDatagram}. Like {@link appendGroup}, - * throws if an earlier producer of the name already wrote that sequence. + * relay preserving upstream numbering uses {@link insertDatagram}. * * Pass `undefined` for the timestamp on an untimed track; a timestamp whose presence * disagrees with the track's {@link Info.timescale} throws {@link TimestampMismatch}. diff --git a/rs/moq-stats/src/produce.rs b/rs/moq-stats/src/produce.rs index f76bf79bebb..dfba315e79c 100644 --- a/rs/moq-stats/src/produce.rs +++ b/rs/moq-stats/src/produce.rs @@ -2881,7 +2881,7 @@ mod tests { ); // A return after the linger is a new epoch counted from zero, with group - // numbers starting over. + // numbers continuing past the old epoch's. let _third = feed(registry, Tier::default(), "acme/live", true, 1, 25).await; drive_tick().await; let mut restarted = take_epochs(&mut events); From 543efdd489f9baff3cf829a07e5eb05719f0477f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 12:51:08 -0700 Subject: [PATCH 13/16] fix(transcode): finish a rung only when its source finished A retired, closed, or source-less rung now aborts instead of finishing. Its name can be served again and that serve starts over, so a finish would declare an end below groups a relay still caches for the name. Co-Authored-By: Claude Opus 5.5 --- rs/moq-transcode/src/catalog.rs | 15 ++--- rs/moq-transcode/src/lib.rs | 96 ++++++++++++++++++++++++-------- rs/moq-transcode/src/pipeline.rs | 4 +- rs/moq-transcode/src/rung.rs | 72 +++++++++++++----------- 4 files changed, 118 insertions(+), 69 deletions(-) diff --git a/rs/moq-transcode/src/catalog.rs b/rs/moq-transcode/src/catalog.rs index ce7a0084f1f..76c8bfcda6d 100644 --- a/rs/moq-transcode/src/catalog.rs +++ b/rs/moq-transcode/src/catalog.rs @@ -35,12 +35,9 @@ impl Resolved { /// The track names a ladder has handed out, so it never hands one out twice. /// -/// A retired rung ends its track for good, and a clean end is terminal: a relay -/// keeps the finished logical track (`broadcast::Consumer::track_inner` only -/// drops an *aborted* one) and serves its EOF to every later subscriber, so a -/// request for that name never reaches the transcoder again. A rung re-resolved -/// under a new picture is therefore published under a name the ladder has never -/// used, rather than reusing the one it just finished. +/// A name always means the same content, so a rung re-resolved under a new +/// picture is published under a name the ladder has never used, rather than +/// reusing the one it just retired: a relay may still cache that name's groups. #[derive(Default)] pub(crate) struct Names { /// The revision last issued per configured short side. The first is the bare @@ -755,9 +752,9 @@ mod tests { assert_eq!(entry.framerate, None); } - /// A name is handed out once and never again: a retired rung's track ends for - /// good, so its replacement has to be a name no subscriber can already hold a - /// finished copy of. + /// A name is handed out once and never again: a retired rung's replacement + /// carries different content, so it needs a name no cache already holds groups + /// of. #[test] fn names_are_never_reused() { let mut names = Names::default(); diff --git a/rs/moq-transcode/src/lib.rs b/rs/moq-transcode/src/lib.rs index 7ab0b48c9ea..19bbbe19630 100644 --- a/rs/moq-transcode/src/lib.rs +++ b/rs/moq-transcode/src/lib.rs @@ -1003,10 +1003,8 @@ mod tests { /// to probe the mode, once when the first subscriber arrives), and a window /// A source that changes aspect ratio keeps every rung height while moving /// every rung width, so a rung retires and its replacement serves the same - /// height. That replacement must not reuse the retired track's name: a clean - /// end is terminal, and a relay keeps the finished logical track (only an - /// *aborted* one is dropped and requested again), so a subscriber asking for - /// the old name would get its EOF forever and never reach the transcoder. + /// height. That replacement must not reuse the retired track's name: it is a + /// different picture, and a name always means the same content. #[tokio::test] async fn a_resized_rung_takes_a_fresh_name() { let mut source = source_catalog(640, 360); @@ -1058,15 +1056,17 @@ mod tests { assert_eq!(replacement.coded_width, Some(160)); assert_eq!(replacement.coded_height, Some(120)); - // The retired name ends cleanly, and the replacement is a track the - // transcoder has never finished, so it serves. + // The retired name aborts rather than finishing: its source did not finish. let ended = tokio::time::timeout(std::time::Duration::from_secs(5), retired.recv_group()) .await - .expect("the retired rung never ended its track") - .expect("the retired rung aborted instead of finishing"); - assert!(ended.is_none(), "expected a clean end, got a group"); + .expect("the retired rung never ended its track"); + assert!( + matches!(ended, Err(moq_net::Error::Cancel)), + "a retired rung aborts, since only a source finish ends it: {:?}", + ended.as_ref().err() + ); - // The replacement is a track the transcoder has never finished, so it serves. + // The replacement serves under its own name. subscribe(&consumer, "video/120p.2").await; transcoder.abort(); @@ -1150,14 +1150,16 @@ mod tests { "the passthrough entry should follow the source" ); - // The retired rung ends its track, so a subscriber reselects the way it - // would on any other rendition going away, rather than stalling or seeing - // an abort it would read as a failure. + // The retired rung ends its track, so a subscriber reselects rather than + // stalling. It aborts, since its source did not finish. let ended = tokio::time::timeout(std::time::Duration::from_secs(5), retired.recv_group()) .await - .expect("the retired rung never ended its track") - .expect("the retired rung aborted instead of finishing"); - assert!(ended.is_none(), "expected a clean end, got a group"); + .expect("the retired rung never ended its track"); + assert!( + matches!(ended, Err(moq_net::Error::Cancel)), + "a retired rung aborts, since only a source finish ends it: {:?}", + ended.as_ref().err() + ); // The rung the new picture still fits keeps serving: its subscriber sees // nothing at all, since the source has no media. @@ -1172,8 +1174,8 @@ mod tests { } /// Retiring a rung stops taking new work, but a group already being produced - /// has to run to a clean end: the consumer reads it out and then sees the - /// track finish, rather than the group going away under it. + /// has to run to its end: the consumer reads it out before the track is + /// aborted, rather than the group going away under it. /// /// The held query is demand, so `live` starts the moment the track is /// queried below and produces group 0 from the still-open source group. The fetch below then resolves straight from the track cache. What @@ -1308,9 +1310,8 @@ mod tests { } /// The other half of the retirement boundary: a fetch that is still opening - /// its decoder has claimed no output group, so retirement must not declare a - /// final sequence until it has. Finishing at retirement instead computes the - /// boundary from the groups produced so far (none) and refuses the very fetch + /// its decoder has claimed no output group, so retirement must not end the + /// track until it has. Ending it at retirement instead refuses the very fetch /// the handler drained its loop to keep. /// /// Reaching the fetch handler takes a group the live path cannot produce. @@ -1852,9 +1853,12 @@ mod tests { let ended = tokio::time::timeout(std::time::Duration::from_secs(5), retired.recv_group()) .await - .expect("the retired rung never ended its track") - .expect("the retired rung aborted instead of finishing"); - assert!(ended.is_none(), "expected a clean end, got a group"); + .expect("the retired rung never ended its track"); + assert!( + matches!(ended, Err(moq_net::Error::Cancel)), + "a retired rung aborts, since only a source finish ends it: {:?}", + ended.as_ref().err() + ); subscribe(&consumer, "video/120p.2").await; transcoder.abort(); @@ -2021,4 +2025,48 @@ mod tests { let result = tokio::time::timeout(std::time::Duration::from_secs(5), transcoder).await; result.expect("run did not shut down within 5s").unwrap().unwrap(); } + + /// A rung finishes only when its source track finishes. Any other end, here + /// the source aborting, aborts the rung: a finish would declare an end that a + /// later serve of the same name, starting over, could fall below. + #[tokio::test(start_paused = true)] + async fn a_rung_finishes_only_with_its_source() { + for finish in [true, false] { + let source = source_catalog(320, 240); + let config = Config { + ladder: Ladder::new([Rung::new(120, moq_net::bandwidth::Rate::from_bps(100_000))]).unwrap(), + encoder: moq_video::encode::Kind::Software, + decoder: moq_video::decode::Kind::Software, + source: None, + ..Default::default() + }; + let output = moq_net::broadcast::Info::default().produce(); + let consumer = output.consume(); + let transcoder = tokio::spawn(run(source.broadcast.consume(), output, config)); + + let mut rung = subscribe(&consumer, "video/120p").await; + let mut group = source._track.create_group(0u64.into()).unwrap(); + write_keyframe(&mut group); + group.finish().unwrap(); + match finish { + true => source._track.finish().unwrap(), + false => source._track.clone().abort(moq_net::Error::Cancel).unwrap(), + } + + let ended = loop { + match rung.recv_group().await { + Ok(Some(_)) => continue, + ended => break ended, + } + }; + match finish { + true => assert!(ended.is_ok(), "a finished source finishes the rung"), + false => assert!( + matches!(ended, Err(moq_net::Error::Cancel)), + "an aborted source aborts the rung" + ), + } + transcoder.abort(); + } + } } diff --git a/rs/moq-transcode/src/pipeline.rs b/rs/moq-transcode/src/pipeline.rs index 0276d9480d2..7a174fcd1da 100644 --- a/rs/moq-transcode/src/pipeline.rs +++ b/rs/moq-transcode/src/pipeline.rs @@ -11,8 +11,8 @@ //! So every source catalog snapshot re-resolves the ladder and diffs it: rungs //! the new picture has no room for retire, rungs it makes room for are added, //! and the rest carry on untouched. A rung whose picture changed retires and its -//! replacement is published under a fresh name, because a retired track ends for -//! good (see [`catalog::Names`]). +//! replacement is published under a fresh name, because a name always means the +//! same content (see [`catalog::Names`]). use std::collections::HashMap; diff --git a/rs/moq-transcode/src/rung.rs b/rs/moq-transcode/src/rung.rs index 32880640330..bc6ab1bab77 100644 --- a/rs/moq-transcode/src/rung.rs +++ b/rs/moq-transcode/src/rung.rs @@ -40,11 +40,11 @@ const MAX_CONCURRENT_FETCHES: usize = 4; /// A rung's retirement signal: the receiving half of a flag the transcoder sets /// when the ladder no longer has room for this rung. /// -/// Retiring is a clean track end rather than a dropped task, so a subscriber -/// sees the same thing it would on any other rendition going away and picks -/// another one, instead of an abort it would read as a failure. The end waits -/// on the fetches still in flight, since the track's final sequence has to sit -/// above every group they are about to claim. +/// Retiring aborts the track at the next group boundary rather than dropping the +/// task mid-group, so a subscriber reads out the group it is in. It aborts rather +/// than finishes because only a source finish ends a rung for good (see [`serve`]). +/// The abort waits on the fetches still in flight, so a group a consumer asked for +/// before retirement still reaches it. #[derive(Clone)] pub(crate) struct Retire(tokio::sync::watch::Receiver); @@ -124,12 +124,12 @@ impl Rung { } } -/// What the live path leaves the output track in. +/// How the live path ended, which decides how the output track ends. enum Ended { - /// Still open: finish it once nothing else can add a group. - Open, - /// Already terminal (aborted), or closed under us. - Closed, + /// The source track finished: finish this one too, once nothing else can add a group. + Finished, + /// Anything else (retired, closed under us, or the source gone without finishing): abort. + Unfinished, } /// Serve one requested rung track until it closes or the source ends. @@ -148,20 +148,25 @@ pub(crate) async fn serve(rung: Rung, request: moq_net::track::Request) -> Resul }; let (live, fetches) = tokio::join!(live, fetches(&rung, dynamic, &mut finishing)); + // Only a source that finished finishes the track; every other ending aborts it. + // A rung retired, closed, or cut off from its source can have its name served + // again, and that serve starts over, so a finish here would declare an end below + // groups a relay still caches for the name. An abort declares no end. + // // Finished here rather than in `live`, because the boundary is only known once - // every fetch has claimed its group. `finish` sets the exclusive boundary one + // no fetch can still claim a group. `finish` sets the exclusive boundary one // past the highest group produced so far (0 when none was), and a group at or // above it is refused. A fetch still opening its decoder has produced nothing - // to count, so finishing then would reject the very fetch retirement drained - // the loop to keep. + // to count, so finishing then would reject the very group it was fetching. let result = match (live, fetches) { - (Ok(Ended::Open), Ok(())) => producer.finish().map_err(Into::into), + (Ok(Ended::Finished), Ok(())) => match producer.finish() { + Ok(()) => return Ok(()), + Err(err) => Err(err.into()), + }, (live, fetches) => live.map(|_| ()).and(fetches), }; - if result.is_err() { - // End the track so subscribers see an error rather than a stall. - let _ = producer.abort(moq_net::Error::Cancel); - } + // Subscribers see an end rather than a stall. + let _ = producer.abort(moq_net::Error::Cancel); result } @@ -179,30 +184,30 @@ async fn until_wanted(demand: &moq_net::track::Demand, wanted: bool) -> moq_net: /// heavy lifting (subscription, decode) is shared with every other active rung /// of this source; only the per-rung resize and encode happen here. /// -/// Reports whether the track is still open rather than finishing it: [`serve`] -/// owns that, once the fetches in flight can no longer add a group. +/// Reports how it ended rather than ending the track: [`serve`] owns that, once +/// the fetches in flight can no longer add a group. async fn live(rung: &Rung, producer: &mut moq_net::track::Producer) -> Result { let demand = producer.demand(); let mut retire = rung.retire.clone(); - // Set once the ladder retires this rung. The track is finished at the next - // group boundary rather than mid-group, so the last thing a subscriber gets - // is a complete group and then a clean end. + // Set once the ladder retires this rung. The track ends at the next group + // boundary rather than mid-group, so the last thing a subscriber gets is a + // complete group. let mut retiring = false; loop { if retiring { - return Ok(Ended::Open); + return Ok(Ended::Unfinished); } tokio::select! { wanted = until_wanted(&demand, true) => if wanted.is_err() { // The output track closed; nothing more to serve. - return Ok(Ended::Closed); + return Ok(Ended::Unfinished); }, () = rung.broadcast.closed() => { // A broadcast end carries no error; stop the idle rung explicitly. producer.clone().abort(moq_net::Error::Cancel)?; - return Ok(Ended::Closed); + return Ok(Ended::Unfinished); } () = retire.fired() => { // Retired while idle, which is the common case: the ladder is @@ -371,7 +376,7 @@ async fn live(rung: &Rung, producer: &mut moq_net::track::Producer) -> Result { // The feed died mid-stream (source or decode error). @@ -379,7 +384,7 @@ async fn live(rung: &Rung, producer: &mut moq_net::track::Producer) -> Result Result Date: Fri, 23 Oct 2026 13:33:34 -0700 Subject: [PATCH 14/16] fix(stats): number each drain's interval at full resolution Timestamp truncates to milliseconds, so drains an interval apart could share a number when the interval is not whole milliseconds, skipping a plain snapshot. The drain now numbers itself once from a timestamp base plus the elapsed instant, and hands every track that Stamp, so tracks no longer carry the interval. Co-Authored-By: Claude Opus 5.5 --- rs/moq-stats/src/produce.rs | 243 ++++++++++++++++++------------ rs/moq-stats/src/produce_bench.rs | 6 +- 2 files changed, 151 insertions(+), 98 deletions(-) diff --git a/rs/moq-stats/src/produce.rs b/rs/moq-stats/src/produce.rs index dfba315e79c..d0d97405bf0 100644 --- a/rs/moq-stats/src/produce.rs +++ b/rs/moq-stats/src/produce.rs @@ -340,6 +340,19 @@ struct Drain { /// Drain counter, stamped on the change-detection state an entry touches /// so state the report no longer carries can be dropped. tick: u64, + /// An instant and its timestamp, which group numbers count intervals from. + base: (Instant, Duration), +} + +/// One drain's instant and the publish interval it falls in, which numbers every +/// group the drain opens. Counted at full resolution, so drains at least an interval +/// apart never share a number. +#[derive(Clone, Copy, Debug)] +struct Stamp { + /// Stamps the drain's frames. + at: Instant, + /// The drain's interval since the timestamp epoch. + group: u64, } impl Drain { @@ -352,15 +365,27 @@ impl Drain { let group = GroupPublisher::create(&task, &Path::empty())?; groups.insert(String::new(), group); } + let now = moq_clock::now(); Some(Self { task, groups, report: Report::default(), refused: Vec::new(), tick: 0, + base: (now, Duration::from(moq_net::Timestamp::from(now))), }) } + /// Stamp a drain at `at`, numbering it by the interval it falls in. + fn stamp(&self, at: Instant) -> Stamp { + let (instant, timestamp) = self.base; + let elapsed = timestamp + at.saturating_duration_since(instant); + Stamp { + at, + group: (elapsed.as_nanos() / self.task.interval.as_nanos()) as u64, + } + } + /// Drain the registry and sort each entry into its group's pending frames, /// creating group broadcasts and tracks on first sight. fn collect(&mut self) { @@ -424,6 +449,7 @@ impl Drain { // group broadcast lives while its group has entries, plus the linger. let depth = self.task.broadcast_depth(); let linger = self.task.linger; + let stamp = self.stamp(now); for (_, group) in self .groups .extract_if(|_, group| depth > 0 && group.lingered(now, linger)) @@ -434,12 +460,12 @@ impl Drain { } for group in self.groups.values_mut() { - group.flush(self.tick, now, linger); + group.flush(self.tick, stamp, linger); // Serve consumer requests for tracks no drain has created yet: a // tier's tracks appear lazily on its first traffic, and a label's // presence track only while requested. See `serve_requests`. - group.serve_requests(&self.report, depth, self.tick, now); + group.serve_requests(&self.report, depth, self.tick, stamp); } } @@ -504,7 +530,7 @@ impl Serialize for Frame { } /// Writes encoded snapshots, numbering each group by the publish interval it -/// opens in: its timestamp divided by the interval. A recreated track (an +/// opens in (see [`Stamp`]). A recreated track (an /// on-demand track asked for again, or a group returning after its linger) /// opens in a later interval than its predecessor's last group, so it resumes /// past any group a subscriber has cached, with nothing kept for names that @@ -517,20 +543,18 @@ struct Snapshot { track: track::Producer, encoder: moq_json::snapshot::Encoder, group: Option, - interval: Duration, /// The sequence of the newest group written, which a refresh has to pass. largest: Option, deltas: bool, } impl Snapshot { - fn new(track: track::Producer, config: moq_json::snapshot::Config, interval: Duration) -> Self { + fn new(track: track::Producer, config: moq_json::snapshot::Config) -> Self { Self { track, deltas: config.deltas, encoder: moq_json::snapshot::Encoder::new(config), group: None, - interval, largest: None, } } @@ -539,7 +563,7 @@ impl Snapshot { self.track.demand() } - /// Write one frame, stamped and numbered by the drain's `now`. `Some(len)` is the payload + /// Write one frame, stamped and numbered by its drain. `Some(len)` is the payload /// written; `None` means the value was unchanged, the interval already has a group for a /// snapshot, or the track already ended, and nothing was written. /// @@ -547,7 +571,7 @@ impl Snapshot { /// [`moq_net::Error::FrameTooLarge`]: skipping it would leave readers on the /// previous snapshot with no sign it stopped tracking. A compressed frame /// meets the same limit by its uncompressed size, which the encoder refuses. - fn update(&mut self, value: &T, now: Instant) -> moq_json::Result> { + fn update(&mut self, value: &T, stamp: Stamp) -> moq_json::Result> { if self.track.is_closed() { return Ok(None); } @@ -573,9 +597,9 @@ impl Snapshot { Err(err) => return Err(err), }; let len = frame.payload.len(); - let timestamp = moq_net::Timestamp::from(now); + let timestamp = moq_net::Timestamp::from(stamp.at); if frame.keyframe { - let sequence = (std::time::Duration::from(timestamp).as_nanos() / self.interval.as_nanos()) as u64; + let sequence = stamp.group; // The interval already has its group: the uncommitted frame is encoded again next tick. if self.largest.is_some_and(|largest| sequence <= largest) { return Ok(None); @@ -639,21 +663,21 @@ fn compressed_info() -> track::Info { } impl TrackPair { - fn create(broadcast: &broadcast::Producer, name: &str, interval: Duration) -> Result { + fn create(broadcast: &broadcast::Producer, name: &str) -> Result { let plain_track = broadcast.create_track(name, info())?; let compressed_track = broadcast.create_track(format!("{name}{COMPRESSED_SUFFIX}").as_str(), compressed_info())?; - Ok(Self::from_tracks(plain_track, compressed_track, interval)) + Ok(Self::from_tracks(plain_track, compressed_track)) } - fn from_tracks(plain_track: track::Producer, compressed_track: track::Producer, interval: Duration) -> Self { + fn from_tracks(plain_track: track::Producer, compressed_track: track::Producer) -> Self { let plain_config = moq_json::snapshot::Config::default().with_deltas(false); let mut compressed_config = moq_json::snapshot::Config::default(); compressed_config.compression = moq_json::Compression::Deflate; Self { - plain: Snapshot::new(plain_track, plain_config, interval), - compressed: Snapshot::new(compressed_track, compressed_config, interval), + plain: Snapshot::new(plain_track, plain_config), + compressed: Snapshot::new(compressed_track, compressed_config), frame: Frame::default(), #[cfg(feature = "bench")] held: 0, @@ -673,16 +697,16 @@ impl TrackPair { /// and clear them for the next drain; moq-json skips unchanged values. A /// flavor that fails ends alone: DEFLATE can shrink a frame too large for /// the plain track, or grow one that just fits. - fn publish(&mut self, name: &str, now: Instant) { + fn publish(&mut self, name: &str, stamp: Stamp) { self.frame.entries.sort_unstable_by(|a, b| a.0.cmp(&b.0)); - let plain = match self.plain.update(&self.frame, now) { + let plain = match self.plain.update(&self.frame, stamp) { Ok(len) => len, Err(err) => { tracing::warn!(?err, name, "stats: failed to write frame"); None } }; - let compressed = match self.compressed.update(&self.frame, now) { + let compressed = match self.compressed.update(&self.frame, stamp) { Ok(len) => len, Err(err) => { tracing::warn!(?err, name, "stats: failed to write compressed frame"); @@ -738,12 +762,7 @@ impl PendingPair { /// queued-request fulfillment cannot reach it; the caller collects both /// flavors' popped requests and this serves each through its actual /// request where one exists. - fn adopt( - self, - broadcast: &broadcast::Producer, - name: &str, - interval: Duration, - ) -> Result, moq_net::Error> { + fn adopt(self, broadcast: &broadcast::Producer, name: &str) -> Result, moq_net::Error> { let PendingPair { plain, compressed } = self; let plain_track = match plain { Some(request) => request.accept(info()), @@ -753,7 +772,7 @@ impl PendingPair { Some(request) => request.accept(compressed_info()), None => broadcast.create_track(format!("{name}{COMPRESSED_SUFFIX}").as_str(), compressed_info())?, }; - Ok(TrackPair::from_tracks(plain_track, compressed_track, interval)) + Ok(TrackPair::from_tracks(plain_track, compressed_track)) } fn reject(self, err: moq_net::Error) { @@ -782,7 +801,6 @@ impl PendingPair { /// One frame type's live per-tier pairs and the requests parked for them; the /// traffic tracks and the sessions tracks each form one family. struct TrackFamily { - interval: Duration, tracks: HashMap>, /// Valid-shaped requests awaiting quota, keyed by plain name and bounded by /// [`MAX_PARKED_REQUESTS`] across both families. Adopted as the quota @@ -791,9 +809,8 @@ struct TrackFamily { } impl TrackFamily { - fn new(interval: Duration) -> Self { + fn new() -> Self { Self { - interval, tracks: HashMap::new(), parked: HashMap::new(), } @@ -818,8 +835,8 @@ impl TrackFamily { ) { if !self.tracks.contains_key(name) { let result = match self.parked.remove(name) { - Some(pending) => pending.adopt(broadcast, name, self.interval), - None => TrackPair::create(broadcast, name, self.interval), + Some(pending) => pending.adopt(broadcast, name), + None => TrackPair::create(broadcast, name), }; match result { Ok(pair) => { @@ -841,9 +858,9 @@ impl TrackFamily { /// Publish every pair's pending frame, an empty one when the drain had /// nothing for it, so a track whose last entry closed transitions to `{}` /// exactly once. - fn flush(&mut self, now: Instant) { + fn flush(&mut self, stamp: Stamp) { for (name, pair) in self.tracks.iter_mut() { - pair.publish(name, now); + pair.publish(name, stamp); } } @@ -890,7 +907,7 @@ impl TrackFamily { /// later drain, so a valid-shaped request is never terminally rejected /// merely for arriving while the quota was full. Entries whose every /// requester left are dropped instead of adopted. - fn adopt_parked(&mut self, broadcast: &broadcast::Producer, requested: &mut HashSet, now: Instant) { + fn adopt_parked(&mut self, broadcast: &broadcast::Producer, requested: &mut HashSet, stamp: Stamp) { let noop = kio::Waiter::noop(); let mut parked = std::mem::take(&mut self.parked); parked.retain(|plain, pending| { @@ -900,7 +917,7 @@ impl TrackFamily { if requested.len() >= MAX_REQUESTED_TRACKS { return true; } - self.adopt_pair(broadcast, requested, plain.clone(), std::mem::take(pending), now); + self.adopt_pair(broadcast, requested, plain.clone(), std::mem::take(pending), stamp); false }); self.parked = parked; @@ -915,7 +932,7 @@ impl TrackFamily { requested: &mut HashSet, plain: String, pending: PendingPair, - now: Instant, + stamp: Stamp, ) { // A request racing the pair's creation is fulfilled by `create_track` // (queued) or adopted by [`Self::push`] (parked), so a live pair here @@ -925,10 +942,10 @@ impl TrackFamily { pending.reject(moq_net::Error::NotFound); return; } - match pending.adopt(broadcast, &plain, self.interval) { + match pending.adopt(broadcast, &plain) { Ok(mut pair) => { // Hold the subscription open with zeros until the tier records. - pair.publish(&plain, now); + pair.publish(&plain, stamp); self.tracks.insert(plain.clone(), pair); requested.insert(plain); } @@ -1088,7 +1105,7 @@ struct PresenceTrack { } impl PresenceTrack { - fn publish(&mut self, now: Instant) { + fn publish(&mut self, stamp: Stamp) { self.sums.fill(&mut self.pair.frame); for (tier, presence) in &mut self.pair.frame.entries { if let Some(&(tx, rx)) = self.wire_start.get(tier) { @@ -1096,7 +1113,7 @@ impl PresenceTrack { presence.wire_rx = presence.wire_rx.saturating_sub(rx); } } - self.pair.publish(&self.name, now); + self.pair.publish(&self.name, stamp); } } @@ -1163,7 +1180,7 @@ impl PrefixTrack { /// Publish each tier's sums, dropping children this drain carried no row /// for. - fn publish(&mut self, tick: u64, now: Instant) { + fn publish(&mut self, tick: u64, stamp: Stamp) { for (tier, sums) in self.tiers.iter_mut() { sums.children.retain(|_, child| child.seen == tick); self.pair.frame.entries.push(( @@ -1180,7 +1197,7 @@ impl PrefixTrack { }, )); } - self.pair.publish(&self.name, now); + self.pair.publish(&self.name, stamp); } } @@ -1231,12 +1248,12 @@ fn group_info() -> track::Info { } impl GroupTrack { - fn create(broadcast: &broadcast::Producer, group: &str, interval: Duration) -> Result { + fn create(broadcast: &broadcast::Producer, group: &str) -> Result { let name = crate::group_track(group); let track = broadcast.create_track(name.as_str(), group_info())?; Ok(Self { name, - snapshot: Snapshot::new(track, moq_json::snapshot::Config::default(), interval), + snapshot: Snapshot::new(track, moq_json::snapshot::Config::default()), body: GroupBody { // A new instance of the name: readers keep its counters apart from // an earlier instance's. @@ -1316,14 +1333,14 @@ impl GroupTrack { /// Write this drain's frame; moq-json skips an unchanged one. A frame too /// large to cache ends this group's track alone. - fn publish(&mut self, now: Instant) { + fn publish(&mut self, stamp: Stamp) { let tiers = &mut self.body.tiers.entries; tiers.sort_unstable_by(|a, b| a.0.cmp(&b.0)); for (_, tier) in tiers.iter_mut() { tier.paths.entries.sort_unstable_by(|a, b| a.0.cmp(&b.0)); tier.labels.entries.sort_unstable_by(|a, b| a.0.cmp(&b.0)); } - if let Err(err) = self.snapshot.update(&self.body, now) { + if let Err(err) = self.snapshot.update(&self.body, stamp) { tracing::warn!(?err, name = self.name, "stats: failed to write group frame"); } } @@ -1366,9 +1383,8 @@ impl GroupPublisher { }; tracing::debug!(advertised = %advertised, "stats: publishing broadcast"); - let interval = task.interval; - let mut traffic = TrackFamily::new(interval); - let mut sessions = TrackFamily::new(interval); + let mut traffic = TrackFamily::new(); + let mut sessions = TrackFamily::new(); let tiers = task.layout == Layout::Broadcast; // The totals and, outside the track layout, the default tier's tracks @@ -1377,7 +1393,7 @@ impl GroupPublisher { let tier = Tier::default(); for role in [Role::Publisher, Role::Subscriber] { let name = traffic_track(&tier, role, Encoding::Plain); - match TrackPair::create(&broadcast, &name, interval) { + match TrackPair::create(&broadcast, &name) { Ok(pair) => { traffic.tracks.insert(name, pair); } @@ -1388,7 +1404,7 @@ impl GroupPublisher { } } let name = sessions_track(&tier, Encoding::Plain); - match TrackPair::create(&broadcast, &name, interval) { + match TrackPair::create(&broadcast, &name) { Ok(pair) => { sessions.tracks.insert(name, pair); } @@ -1399,7 +1415,7 @@ impl GroupPublisher { } } let name = totals_track(Encoding::Plain); - let totals = match TrackPair::create(&broadcast, &name, interval) { + let totals = match TrackPair::create(&broadcast, &name) { Ok(pair) => TotalsTrack { pair, sums: Cumulative::new(), @@ -1534,7 +1550,7 @@ impl GroupPublisher { }); if let Some(depth) = *track_depth { let key = group_key(entry.path.as_str(), depth); - if let Some(track) = Self::group_track(broadcast, group_tracks, traffic.interval, key) { + if let Some(track) = Self::group_track(broadcast, group_tracks, key) { track.add_traffic(entry, slots.publisher.moved || slots.subscriber.moved); } } @@ -1559,7 +1575,7 @@ impl GroupPublisher { } if let Some(depth) = *track_depth { let key = group_key(entry.label.as_str(), depth); - if let Some(track) = Self::group_track(broadcast, group_tracks, traffic.interval, key) { + if let Some(track) = Self::group_track(broadcast, group_tracks, key) { track.add_sessions(&entry.label, &entry.tier, entry.presence, entry.closed); } } @@ -1570,11 +1586,10 @@ impl GroupPublisher { fn group_track<'a>( broadcast: &broadcast::Producer, tracks: &'a mut HashMap, - interval: Duration, key: &str, ) -> Option<&'a mut GroupTrack> { if !tracks.contains_key(key) { - match GroupTrack::create(broadcast, key, interval) { + match GroupTrack::create(broadcast, key) { Ok(track) => { tracks.insert(key.to_string(), track); } @@ -1589,24 +1604,27 @@ impl GroupPublisher { /// Publish the pending frames, then drop change-detection state for /// entries this drain did not carry (the registry pruned them). - fn flush(&mut self, tick: u64, now: Instant, linger: Duration) { - self.traffic.flush(now); - self.sessions.flush(now); + fn flush(&mut self, tick: u64, stamp: Stamp, linger: Duration) { + self.traffic.flush(stamp); + self.sessions.flush(stamp); - for (_, track) in self.group_tracks.extract_if(|_, track| track.lingered(now, linger)) { + for (_, track) in self + .group_tracks + .extract_if(|_, track| track.lingered(stamp.at, linger)) + { track.finish(); } for track in self.group_tracks.values_mut() { - track.publish(now); + track.publish(stamp); } self.totals.sums.fill(&mut self.totals.pair.frame); - self.totals.pair.publish(crate::TOTALS, now); + self.totals.pair.publish(crate::TOTALS, stamp); for track in self.presence.values_mut() { - track.publish(now); + track.publish(stamp); } for track in self.prefixes.values_mut() { - track.publish(tick, now); + track.publish(tick, stamp); } self.local.retain(|_, tiers| { @@ -1631,7 +1649,7 @@ impl GroupPublisher { /// It is reclaimed once its last consumer leaves, so the held set follows /// live readers. A prefix track follows the same rules for a broadcast /// prefix in this group. Any other name is rejected. - fn serve_requests(&mut self, report: &Report, depth: usize, tick: u64, now: Instant) { + fn serve_requests(&mut self, report: &Report, depth: usize, tick: u64, stamp: Stamp) { // Reclaim before parking and adopting, so a freed slot is usable by // this very drain. self.traffic.reclaim(&mut self.requested); @@ -1690,19 +1708,19 @@ impl GroupPublisher { } } - self.traffic.adopt_parked(&self.broadcast, &mut self.requested, now); - self.sessions.adopt_parked(&self.broadcast, &mut self.requested, now); + self.traffic.adopt_parked(&self.broadcast, &mut self.requested, stamp); + self.sessions.adopt_parked(&self.broadcast, &mut self.requested, stamp); for (label, pending) in labels { - self.adopt_presence(report, label, pending, now); + self.adopt_presence(report, label, pending, stamp); } for (path, pending) in paths { - self.adopt_prefix(report, path, pending, tick, now); + self.adopt_prefix(report, path, pending, tick, stamp); } } /// Hold a requested label's presence track, publishing its current counters /// so the subscription resolves at once. Cumulative from here on. - fn adopt_presence(&mut self, report: &Report, label: PathOwned, pending: PendingPair, now: Instant) { + fn adopt_presence(&mut self, report: &Report, label: PathOwned, pending: PendingPair, stamp: Stamp) { // A request for a live track is served by the broadcast itself, so a // live pair here means the requested flavor ended and cannot update. if self.presence.contains_key(&label) { @@ -1710,7 +1728,7 @@ impl GroupPublisher { return; } let name = presence_track(&label, Encoding::Plain); - let pair = match pending.adopt(&self.broadcast, &name, self.traffic.interval) { + let pair = match pending.adopt(&self.broadcast, &name) { Ok(pair) => pair, Err(err) => { tracing::warn!(?err, name, "stats: failed to adopt presence track"); @@ -1732,20 +1750,20 @@ impl GroupPublisher { start.1 += entry.presence.wire_rx; } } - track.publish(now); + track.publish(stamp); self.presence.insert(label, track); } /// Hold a requested prefix track, publishing its current counters so the /// subscription resolves at once. Cumulative from here on. - fn adopt_prefix(&mut self, report: &Report, path: PathOwned, pending: PendingPair, tick: u64, now: Instant) { + fn adopt_prefix(&mut self, report: &Report, path: PathOwned, pending: PendingPair, tick: u64, stamp: Stamp) { // Defensive only, as in `adopt_presence`. if self.prefixes.contains_key(path.as_str()) { pending.reject(moq_net::Error::NotFound); return; } let name = prefix_track(&path, Encoding::Plain); - let pair = match pending.adopt(&self.broadcast, &name, self.traffic.interval) { + let pair = match pending.adopt(&self.broadcast, &name) { Ok(pair) => pair, Err(err) => { tracing::warn!(?err, name, "stats: failed to adopt prefix track"); @@ -1765,7 +1783,7 @@ impl GroupPublisher { let child = rest.as_str().split('/').next().filter(|child| !child.is_empty()); track.add(entry, child, tick); } - track.publish(tick, now); + track.publish(tick, stamp); self.prefixes.insert(path.as_str().to_string(), track); } @@ -1897,14 +1915,21 @@ mod tests { moq_net::track::Info::default().with_timescale(moq_net::Timescale::MILLI) } + /// A drain at the current instant, numbered `group`. + fn stamp(group: u64) -> Stamp { + Stamp { + at: moq_clock::now(), + group, + } + } + /// A joiner once the deltas outweigh the compressed track's ratio gets a fresh snapshot group in /// the next interval, even of an unchanged value, and deltas resume in it. - #[tokio::test(start_paused = true)] - async fn a_joiner_over_the_ratio_gets_a_snapshot_next_interval() { - let interval = Duration::from_secs(1); + #[test] + fn a_joiner_over_the_ratio_gets_a_snapshot_next_interval() { let broadcast = moq_net::broadcast::Info::new().produce(); let track = broadcast.create_track("t.json.z", compressed_info()).unwrap(); - let mut snapshot = Snapshot::>::new(track.clone(), moq_json::snapshot::Config::default(), interval); + let mut snapshot = Snapshot::>::new(track.clone(), moq_json::snapshot::Config::default()); let frame = |n| Frame { entries: vec![(PathOwned::from("a"), n)], }; @@ -1912,24 +1937,23 @@ mod tests { // A 7 byte snapshot and nine 7 byte deltas: past eight snapshots. for n in 0..10 { - snapshot.update(&frame(n), moq_clock::now()).unwrap(); + snapshot.update(&frame(n), stamp(5)).unwrap(); } - let first = snapshot.largest.unwrap(); + assert_eq!(snapshot.largest, Some(5)); let _late = track.subscribe(None); - assert_eq!(wanted(&track), Some(first + 1), "the join asks for a fresh group"); + assert_eq!(wanted(&track), Some(6), "the join asks for a fresh group"); // The interval already has its group, so the snapshot waits for the next one. - assert!(snapshot.update(&frame(9), moq_clock::now()).unwrap().is_none()); - assert_eq!(snapshot.largest, Some(first)); + assert!(snapshot.update(&frame(9), stamp(5)).unwrap().is_none()); + assert_eq!(snapshot.largest, Some(5)); - tokio::time::advance(interval).await; assert!( - snapshot.update(&frame(9), moq_clock::now()).unwrap().is_some(), + snapshot.update(&frame(9), stamp(6)).unwrap().is_some(), "restated though unchanged" ); - assert_eq!(snapshot.largest, Some(first + 1), "numbered by its interval"); - assert!(snapshot.update(&frame(10), moq_clock::now()).unwrap().is_some()); - assert_eq!(snapshot.largest, Some(first + 1), "the next change is a delta"); + assert_eq!(snapshot.largest, Some(6), "numbered by its interval"); + assert!(snapshot.update(&frame(10), stamp(6)).unwrap().is_some()); + assert_eq!(snapshot.largest, Some(6), "the next change is a delta"); } #[tokio::test(start_paused = true)] @@ -1939,7 +1963,7 @@ mod tests { let broadcast = moq_net::broadcast::Info::new().produce(); let _dynamic = broadcast.dynamic(); let consumer = broadcast.consume(); - let mut family = TrackFamily::::new(Duration::from_secs(1)); + let mut family = TrackFamily::::new(); let mut requested = HashSet::new(); let subscribing = consumer.track(name).unwrap().subscribe(None); family.adopt_pair( @@ -1947,7 +1971,7 @@ mod tests { &mut requested, "idle/publisher.json".into(), PendingPair::default(), - moq_clock::now(), + stamp(0), ); let mut subscriber = subscribing.await.unwrap(); let group = subscriber.recv_group().await.unwrap().unwrap(); @@ -1955,7 +1979,6 @@ mod tests { drop(group); drop(subscriber); // A drain reclaims before it adopts, so a pair it reclaims last wrote in an earlier one. - tokio::time::advance(Duration::from_secs(1)).await; family.reclaim(&mut requested); assert!(family.tracks.is_empty()); @@ -1968,7 +1991,7 @@ mod tests { &mut requested, "idle/publisher.json".into(), PendingPair::default(), - moq_clock::now(), + stamp(1), ); let mut subscriber = subscribing.await.unwrap(); let group = subscriber @@ -2179,10 +2202,10 @@ mod tests { let broadcast = moq_net::broadcast::Info::new().produce(); let consumer = broadcast.consume(); - let mut pair = TrackPair::::create(&broadcast, "t.json", Duration::from_secs(1)).unwrap(); + let mut pair = TrackPair::::create(&broadcast, "t.json").unwrap(); let mut publish = |key: &str, value: u64| { pair.frame.entries.push((PathOwned::from(key.to_string()), value)); - pair.publish("t.json", moq_clock::now()); + pair.publish("t.json", stamp(value)); }; publish("small", 1); @@ -2500,6 +2523,34 @@ mod tests { assert_eq!(sent(3), Some(2), "the on-time drain publishes its readout"); } + /// Group numbers count intervals at full resolution: drains an interval apart + /// get consecutive numbers whatever the phase, even for an interval that is not + /// a whole number of the timestamp's milliseconds. + #[tokio::test(start_paused = true)] + async fn drains_an_interval_apart_never_share_a_number() { + let interval = Duration::from_micros(1500); + let drain = Drain::new(Task { + registry: Registry::new(moq_net::stats::Config::new()), + origin: produce_origin(), + prefix: ".stats".into(), + node: None, + depth: 0, + layout: Layout::Broadcast, + linger: DEFAULT_LINGER, + interval, + }) + .unwrap(); + let start = moq_clock::now(); + for phase in (0..1500).step_by(100) { + let at = start + Duration::from_micros(phase); + assert_eq!( + drain.stamp(at + interval).group, + drain.stamp(at).group + 1, + "phase {phase}us" + ); + } + } + /// A producer on `origin` publishing as node `sjc` at depth 0. fn node_producer(origin: &origin::Producer) -> Producer { Producer::new( @@ -3847,14 +3898,14 @@ mod tests { // Hold a prefix with a child per tier, as a reader would. let _held = hold.then(|| { let tick = drain.tick; + let stamp = drain.stamp(moq_clock::now()); let group = drain.groups.get_mut(key).expect("group"); - let now = moq_clock::now(); group.adopt_prefix( &drain.report, PathOwned::from("room0"), PendingPair::default(), tick, - now, + stamp, ); group.prefixes["room0"].pair.plain.track.subscribe(None) }); diff --git a/rs/moq-stats/src/produce_bench.rs b/rs/moq-stats/src/produce_bench.rs index 71ce2dee52f..c80c2ae96e8 100644 --- a/rs/moq-stats/src/produce_bench.rs +++ b/rs/moq-stats/src/produce_bench.rs @@ -74,9 +74,10 @@ impl Driver { /// it. Keep the returned subscriber: dropping it lets a tick reclaim the /// track. pub fn hold_presence(&mut self, label: &str) -> track::Subscriber { + let stamp = self.drain.stamp(self.now); let group = self.drain.groups.get_mut("").expect("the depth 0 broadcast"); let label = PathOwned::from(label.to_string()); - group.adopt_presence(&self.drain.report, label.clone(), PendingPair::default(), self.now); + group.adopt_presence(&self.drain.report, label.clone(), PendingPair::default(), stamp); group.presence[&label].pair.plain.track.subscribe(None) } @@ -84,13 +85,14 @@ impl Driver { /// [`Self::hold_presence`]. pub fn hold_prefix(&mut self, path: &str) -> track::Subscriber { let tick = self.drain.tick; + let stamp = self.drain.stamp(self.now); let group = self.drain.groups.get_mut("").expect("the depth 0 broadcast"); group.adopt_prefix( &self.drain.report, PathOwned::from(path.to_string()), PendingPair::default(), tick, - self.now, + stamp, ); group.prefixes[path].pair.plain.track.subscribe(None) } From 91abf34991c2b8046f4b9899c17196aa21831fba Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 22:15:48 -0700 Subject: [PATCH 15/16] feat(stats)!: make group tracks the only stats layout Drop the per-group broadcast layout: every node publishes one `/node/` broadcast with `totals.json`, a `/group.json` track per depth group, and requested presence and prefix tracks. The per-tier publisher/subscriber/sessions tracks, the `produce::Layout` enum, and the relay's `[stats] layout` option are gone. `aggregate::Consumer` now folds `group.json` tracks across nodes, following each node's track listing and resetting on a new group epoch, and the demo dashboard reads `group.json`. A path's never-moved side is left off the wire, keeping the depth-0 snapshot the size of the old `publisher.json`. Co-Authored-By: Claude Opus 5.5 --- demo/relay/localhost.toml | 2 +- demo/web/src/stats.ts | 82 +- doc/bin/inspect.md | 17 +- doc/bin/relay/config.md | 19 +- doc/concept/stats.md | 203 +-- quest/m2/qos-lag-dashboard.md | 2 +- quest/m2/qos/README.md | 4 +- quest/m2/qos/lag-histogram.md | 2 +- rs/moq-net/src/stats.rs | 23 +- rs/moq-relay/src/config.rs | 12 +- rs/moq-relay/src/settings.rs | 3 - rs/moq-relay/src/stats.rs | 72 +- rs/moq-relay/tests/integration/refusals.rs | 12 +- rs/moq-stats/benches/aggregate.rs | 69 +- rs/moq-stats/benches/decode.rs | 95 +- rs/moq-stats/benches/producer_tick.rs | 53 +- rs/moq-stats/src/aggregate.rs | 1049 ++++++++---- rs/moq-stats/src/consume.rs | 116 +- rs/moq-stats/src/lib.rs | 183 +- rs/moq-stats/src/produce.rs | 1790 ++++---------------- rs/moq-stats/src/produce_bench.rs | 55 +- 21 files changed, 1409 insertions(+), 2454 deletions(-) diff --git a/demo/relay/localhost.toml b/demo/relay/localhost.toml index b68064835af..6296b741266 100644 --- a/demo/relay/localhost.toml +++ b/demo/relay/localhost.toml @@ -30,7 +30,7 @@ public = "**" # # .stats/node/local/host # -# carrying totals.json, publisher.json, subscriber.json, and sessions.json tracks. See +# carrying totals.json and group.json tracks. See # doc/bin/relay/config.md for the field semantics. [stats] enabled = true diff --git a/demo/web/src/stats.ts b/demo/web/src/stats.ts index c912cc70c54..814a3446b92 100644 --- a/demo/web/src/stats.ts +++ b/demo/web/src/stats.ts @@ -8,23 +8,24 @@ * let you drill into one. A restarted relay announces a new epoch at the same * path, which arrives as an end then a start, so its history starts over. * - * Per-node tracks we read: - * publisher.json egress (relay -> downstream viewers) - * subscriber.json ingress (upstream publishers -> relay) - * sessions.json sessions by stats label + * Per node we read `group.json`, the whole node's group at the relay's default + * `depth = 0`: `{ epoch, tiers: { : { paths, labels } } }`, a JSON snapshot + * with merge-patch deltas. From the default tier we take each path's `publisher` + * (egress, relay -> downstream viewers) and `subscriber` (ingress, upstream + * publishers -> relay), and each stats label's sessions. * - * Each frame is `{ "": Snapshot }`. Counters are cumulative; - * "active" = started - ended. The relay includes every entry it still holds - * counters for, idle ones too, so the latest frame is a snapshot of now. We sample the aggregate on an interval - * to derive per-second throughput rates for the charts. A relay built after the - * started/ended rename still writes the legacy names beside the new ones; this - * dashboard prefers the canonical spelling and falls back so it also reads an - * older relay. + * Counters are cumulative; "active" = started - ended. The relay includes every + * entry it still holds counters for, idle ones too, so the latest frame is a + * snapshot of now. We sample the aggregate on an interval to derive per-second + * throughput rates for the charts. A relay built after the started/ended rename + * still writes the legacy names beside the new ones; this dashboard prefers the + * canonical spelling and falls back so it also reads an older relay. */ import "./highlight"; import { Milli, Nano, wall } from "@moq/clock"; import { Net, Signals } from "@moq/hang"; +import * as Json from "@moq/json"; const RELAY_URL = import.meta.env.VITE_RELAY_URL ?? "http://localhost:4443"; @@ -78,9 +79,32 @@ interface SessionCounters { type SessionFrame = Record; interface NodeStats { - egress: BroadcastFrame; // publisher.json - ingress: BroadcastFrame; // subscriber.json - sessions: SessionFrame; // sessions.json + egress: BroadcastFrame; // each path's `publisher` + ingress: BroadcastFrame; // each path's `subscriber` + sessions: SessionFrame; // `labels` +} + +// One tier of a `group.json` frame. +interface GroupTier { + paths?: Record; + labels?: SessionFrame; +} + +interface GroupFrame { + tiers?: Record; +} + +// The default tier's egress, ingress, and sessions from a group frame. +function nodeStatsOf(frame: GroupFrame): NodeStats { + const tier = frame.tiers?.[""] ?? {}; + const egress: BroadcastFrame = {}; + const ingress: BroadcastFrame = {}; + for (const [path, usage] of Object.entries(tier.paths ?? {})) { + // The relay leaves out a side that never moved. + if (usage.publisher) egress[path] = usage.publisher; + if (usage.subscriber) ingress[path] = usage.subscriber; + } + return { egress, ingress, sessions: tier.labels ?? {} }; } // Canonical spelling wins when a frame carries both names. @@ -176,24 +200,18 @@ function subscribeNode(effect: Signals.Effect, origin: Net.Origin.Table, path: N const consumer = request.active.peek(); if (!consumer) return; - const sub = (trackName: string, key: K) => { - const track = consumer.track(trackName).subscribe().ordered(); - effect.cleanup(() => track.close()); - effect.spawn(async () => { - for (;;) { - const data = await effect.race(track.readJson()); - if (data === undefined) break; - nodeStats.mutate((s) => { - const cur = s[node]; - if (cur) cur[key] = (data ?? {}) as NodeStats[K]; - }); - } - }); - }; - - sub("publisher.json", "egress"); - sub("subscriber.json", "ingress"); - sub("sessions.json", "sessions"); + const track = consumer.track("group.json").subscribe(); + effect.cleanup(() => track.close()); + const group = new Json.Snapshot.Consumer({ track }); + effect.spawn(async () => { + for (;;) { + const state = await effect.race(group.latest()); + if (state === undefined) break; + nodeStats.mutate((s) => { + if (s[node]) s[node] = nodeStatsOf(state.value ?? {}); + }); + } + }); } // ---- Aggregation ---------------------------------------------------------- diff --git a/doc/bin/inspect.md b/doc/bin/inspect.md index 94f55ce0f4b..db8ecf3a57a 100644 --- a/doc/bin/inspect.md +++ b/doc/bin/inspect.md @@ -69,15 +69,16 @@ Dial the `.stats` path, which the token must cover: ```bash moq --connect http://localhost:4443/.stats announced -moq --connect http://localhost:4443/.stats --broadcast node/local/host fetch publisher.json | jq -curl http://localhost:4443/fetch/.stats/node/local/host/publisher.json | jq +moq --connect http://localhost:4443/.stats --broadcast node/local/host fetch totals.json | jq +curl http://localhost:4443/fetch/.stats/node/local/host/totals.json | jq ``` The node name (`local/host` here) is `stats.node`. Each fetch returns the newest snapshot of cumulative counters. `totals.json` sums each tier's traffic -and sessions, `publisher.json` is egress per broadcast path, -`subscriber.json` is ingress, and `sessions.json` counts sessions per stats -label. A subscriber may also ask for `/prefix.json`, one broadcast -prefix rolled up with each child, produced only while held. Fetch twice and divide by the interval for a -rate. The `.json.z` twins carry compressed patches, which these tools print as -raw bytes. [Stats](/concept/stats) describes every field. +and sessions. `group.json` (`/group.json` at `stats.depth` 1 or more) +carries a group's traffic per broadcast path and sessions per stats label: its +newest group is a full snapshot followed by merge-patch deltas, one per line. +A subscriber may also ask for `/prefix.json`, one broadcast prefix rolled +up with each child, produced only while held. Fetch twice and divide by the +interval for a rate. The `.json.z` twins carry compressed patches, which these +tools print as raw bytes. [Stats](/concept/stats) describes every field. diff --git a/doc/bin/relay/config.md b/doc/bin/relay/config.md index bb131f22a36..7659f433533 100644 --- a/doc/bin/relay/config.md +++ b/doc/bin/relay/config.md @@ -350,23 +350,22 @@ group of every track is always kept, even past `capacity`. ```toml [stats] enabled = true -prefix = ".stats" # Broadcasts appear under /node/. +prefix = ".stats" # The broadcast appears at /node/. interval = "1s" # Time between snapshots. Default. node = "sjc/1" # Disambiguates relays sharing a cluster. -depth = 1 # Also bucket by the first N path segments (per tenant). -layout = "broadcast" # Or "track": each group is a track of the node's one broadcast. -linger = "5m" # Keep an empty group's broadcast announced this long. Default. +depth = 1 # One group track per first N path segments (per tenant). +linger = "5m" # Keep an empty group's track listed this long. Default. ``` Each node publishes its traffic and session counters as MoQ tracks, split by a **tier** label from the auth server's grant (`--cluster-tier` for links this relay dials and LAN peers it admits), which is what makes billing per customer or per region possible. -Each run, and each group returning after its linger, announces under a fresh -[epoch](/concept/moq-lite#publisher-epochs), so a restart is a new broadcast at -the same path. With `layout = "track"` a group is a -[track](/concept/stats#group-tracks) instead, and a returning one restarts its -counters without an announce. Each group's totals count everything since its epoch began, so -billing reads those; per-path traffic and per-label sessions and HTTP requests are for +Each run announces under a fresh [epoch](/concept/moq-lite#publisher-epochs), +so a restart is a new broadcast at the same path. Each group is a +[track](/concept/stats#group-tracks) of that broadcast, and one returning after +its linger restarts its counters under a new epoch without an announce. The +node's totals count everything since its epoch began, so billing reads those; +a group's per-path traffic and per-label sessions and HTTP requests are for dashboards. Session counters report how close sessions come to the per-session limits, past which one is closed with `TOO_MANY_REQUESTS`. [Stats](/concept/stats) describes the paths, tracks, and encodings; read them diff --git a/doc/concept/stats.md b/doc/concept/stats.md index df626e02e7c..25fba341ec2 100644 --- a/doc/concept/stats.md +++ b/doc/concept/stats.md @@ -99,110 +99,88 @@ source's. `export ts --passthrough` reports it without `mpegts`. ## Broadcasts -Each node publishes under a prefix, `.stats` by default, which +Each node publishes one broadcast under a prefix, `.stats` by default, which [moq-lite](/concept/moq-lite) hides from announce listings that don't ask for it. Traffic under the prefix is never counted, so serving stats doesn't generate more stats. ```text -/node/ depth 0: one broadcast per node -//node/ depth N: one broadcast per group per node -/node/ depth N, track layout: one broadcast per node, - with a /group.json track per group +/node/ ``` - `` tells relays sharing a cluster apart. It may span several segments (`sjc/1`), and is omitted along with its slash when unset: `/node`. -- `` is the first `depth` segments of each broadcast path (for traffic) - or stats label (for sessions), so a consumer can scope an announce to one - tenant. A path shorter than `depth` groups under all of its segments. - The literal `node` segment leaves room for sibling categories under the same - prefix, so a consumer skips any path without `node` where it expects one. A - group segment literally named `node` is ambiguous; don't use one. - -Each announcement carries a fresh [publisher epoch](/concept/moq-lite#publisher-epochs) -on its route, so a restarted node or a group returning from idle is a new -broadcast at the same path: it replaces the old one, and no relay serves it -groups cached under the old epoch. A reader treats each epoch as its own set of -counters. The epoch rides moq-lite 07 announcements, and moq-transport draft 17 -and later with the [epoch extension](/draft/moq-epoch); over anything else a -reader sees the same change as an end and a start at the path. + prefix, so a consumer skips any path without `node` where it expects one. + +The broadcast stays announced for the producer's life. Each announcement +carries a fresh [publisher epoch](/concept/moq-lite#publisher-epochs) on its +route, so a restarted node is a new broadcast at the same path: it replaces the +old one, and no relay serves it groups cached under the old epoch. A reader +treats each epoch as its own set of counters. The epoch rides moq-lite 07 +announcements, and moq-transport draft 17 and later with the +[epoch extension](/draft/moq-epoch); over anything else a reader sees the same +change as an end and a start at the path. An aggregating reader pins each node's subscription to its epoch, and when one publisher is reached through both kinds of link (with and without an epoch), it carries only that path's outgoing counters across the change. -At depth 0 the broadcast stays announced for the producer's life. At depth -1 or more, a group's broadcast is announced while that group has entries, and -for a linger (five minutes by default) after its last one leaves. A group that -returns within the linger keeps its broadcast and epoch, so viewer churn -doesn't unannounce and re-announce it across the mesh, and its totals -continue. Once the linger elapses with the group still empty, the broadcast is -unannounced and its counters dropped; a group that returns later announces a -new epoch counted from zero. Holding a stats track doesn't count as an entry: -every track ends with the broadcast, including presence, prefix, and requested -tier tracks a reader still holds, and the reader follows the group's next -announcement. Group numbers count publish intervals (see [Frames](#frames)), -so they keep increasing across recreated tracks and have gaps. +Every group shares the node's broadcast, so a token scoped to the stats prefix +reads every group's tracks; scope by group with an aggregating reader instead. ## Tracks Traffic is split by **tier**, an arbitrary label (a billing class, a region) the relay takes from the auth grant, or from `--cluster-tier` for links it -dials and LAN peers it admits. The default tier's label is empty. Each track -comes in two encodings: +dials and LAN peers it admits. The default tier's label is empty. Every track +but a group's comes in two encodings: | Track | Frame keyed by | Entry | | --- | --- | --- | -| `totals.json` | tier | [Totals](#totals) for the whole group | -| `[/]publisher.json` | broadcast path | [Traffic](#traffic) this node sent (egress) | -| `[/]subscriber.json` | broadcast path | [Traffic](#traffic) this node received (ingress) | -| `[/]sessions.json` | stats label | [Presence](#presence): sessions connected now | +| `totals.json` | tier | [Totals](#totals) for the whole node | +| `[/]group.json` | | [Group](#group): one group's totals, paths, and labels, every tier | | `[