diff --git a/doc/concept/moq-lite.md b/doc/concept/moq-lite.md index 3f80b89716b..e27b5ba15d6 100644 --- a/doc/concept/moq-lite.md +++ b/doc/concept/moq-lite.md @@ -197,7 +197,7 @@ Each subscription carries the knobs that decide behavior under congestion: | Knob | Effect | | --- | --- | -| **Priority** (0..255) | Higher-priority tracks get bandwidth first. Audio above video, base layer above enhancement. | +| **Priority** (0..255) | Higher-priority tracks get bandwidth first. Audio above video, base layer above enhancement. From a native Rust publisher over QUIC or WebTransport (iroh excepted), subscriptions at the same priority split stream bandwidth evenly by bytes, however many groups each has queued. Datagrams are not part of that split yet. | | **Order** | Which group to send first when several are pending. Newest first for live (the default), oldest first for catch-up. Needs moq-lite 07 or moq-transport; older moq-lite always sends newest first. | | **Max delay** | How long a non-latest group stays worth delivering before it is skipped. Zero means "live edge only", and raising it is also what asks for history. | diff --git a/quest/m1/quic/README.md b/quest/m1/quic/README.md index 0f867694617..2ee4f735ec4 100644 --- a/quest/m1/quic/README.md +++ b/quest/m1/quic/README.md @@ -67,9 +67,6 @@ cache limits remain separate; bitrate pacing composes with this memory cap. - [A reset group keeps its header](/quest/m1/quic/group-reset.md) - a lite publisher resets a group with `RESET_STREAM_AT`, so its subscriber drops the tail grace -- [Hierarchical stream scheduling](/quest/m1/quic/scheduler.md) - strict - subscription priority, fair buckets, and newest-first group order replace - the lossy scalar; retransmits follow the same order - [Schedule datagrams by subscription priority](/quest/m1/quic/datagram-priority.md) - datagrams join their subscription's send group instead of preempting every stream - [Relay peers get wider limits](/quest/m1/quic/peer-limits.md) - MAX_STREAMS diff --git a/quest/m1/quic/datagram-priority.md b/quest/m1/quic/datagram-priority.md index 7d78b89d6d6..55b2d76ed59 100644 --- a/quest/m1/quic/datagram-priority.md +++ b/quest/m1/quic/datagram-priority.md @@ -30,7 +30,7 @@ Decided 2026-10-10 with the maintainer: datagram track starve streams up to the congestion window, which is what every stack does today. - **A datagram queue per send group.** Each send group (one per - subscription, from [the scheduler](/quest/m1/quic/scheduler.md)) owns a + subscription, from `moq-quic`'s `SendGroups`) owns a datagram queue. Datagram bytes spend the group's fair-share credit, so a datagram-heavy subscription buys no extra bandwidth over its equal-priority peers. The datagram send API takes @@ -103,10 +103,8 @@ on the browser's WebTransport, which cannot rank a datagram against a stream, so it ignores the handle and order and keeps its current best-effort send. Wire: none. -## Required - -- [Hierarchical stream scheduling](/quest/m1/quic/scheduler.md) - supplies - the send groups and fair tier the datagram queues join +Docs: the Priority row in `doc/concept/moq-lite.md` says datagrams are not +part of the equal-priority byte split yet; drop that once they are. ## Related diff --git a/quest/m1/quic/receive-memory.md b/quest/m1/quic/receive-memory.md index d0e1f8882da..da985e4eb66 100644 --- a/quest/m1/quic/receive-memory.md +++ b/quest/m1/quic/receive-memory.md @@ -106,6 +106,5 @@ separate guide quest is needed. - [Unbounded groups](/quest/m2/unbounded-groups/README.md) - removes the frame cap; frame payload credit becomes per segment - [Scope track priority](/quest/m1/track-priority-scope.md) - broadcast fairness domains and subscription priority semantics -- [Hierarchical stream scheduling](/quest/m1/quic/scheduler.md) - share priority semantics without requiring sender-side recovery - [QUIC bitrate caps](/quest/m2/rate-quic.md) - pace credit grants inside the independently bounded memory allowance - [Relay peers get wider limits](/quest/m1/quic/peer-limits.md) - client and peer windows must both fit supported units diff --git a/quest/m1/quic/scheduler.md b/quest/m1/quic/scheduler.md deleted file mode 100644 index 9dafe85ccab..00000000000 --- a/quest/m1/quic/scheduler.md +++ /dev/null @@ -1,122 +0,0 @@ -# [XL] Add hierarchical QUIC stream scheduling - -## Goal - -One QUIC connection expresses the MoQ scheduling hierarchy without packing it -into a scalar: higher-priority subscriptions preempt lower-priority ones, -backlogged subscriptions at the same priority receive equal bandwidth within -their scheduling domain, and each subscription serves its own group streams -newest-first. - -This completes [#699](https://github.com/moq-dev/moq/issues/699). In the -reported Alice and Bob case, two priority-4 subscriptions continue to make -progress at the same byte rate even when one has a deeper or faster-growing -group backlog. - -## Plan - -Add a generic send-group primitive to `moq-quic`. A group owns its -strict priority and fair-share scheduler state; each stream owns only its -order within that group. Prefer an owned group handle whose drop removes its -scheduler state and whose priority can be updated without walking every open -stream. If the backend-neutral WebTransport trait cannot carry a handle -without breaking object safety, allocate an opaque group ID from the session. -Do not accept caller-chosen IDs and do not compress the hierarchy into another -integer. - -Use byte-accounted deficit round robin, or an equivalent bounded-quantum -algorithm, between backlogged groups at equal priority. Round robin by stream -count is insufficient because audio, video, and data streams have different -sizes. Within the chosen group, order streams by the subscription's group order -(newest first by default). A blocked stream must not consume -the group's turn, and opening newer groups must not reset its accumulated -fair-share credit. - -Map conventions only at adapters. MoQ's model remains higher value first, the -IETF wire remains lower value first. Native QUIC and qmux use the full three -levels. Browsers (js/net and web-transport-wasm) never create send groups: -browser groups are flat and byte-fair, which would trade strict priority -between subscriptions (audio over video) for fairness nobody on a browser -session needs, so they keep the default group and pack priority and group -order into `sendOrder` (decided 2026-09-26 alongside Firefox 155's -send groups). - -Give every MoQ subscription one native send group. A SUBSCRIBE_UPDATE changes the -group priority atomically. Group streams use their position within the -subscription, never another subscription's sequence. Remove the session-wide -`lite::PriorityQueue` once every enabled backend has an honest implementation -or fallback. - -Add the scheduling-domain tier above subscriptions: on a shared session, -fairness is one deficit-round-robin bucket per domain, and a domain's -subscriptions share that bucket by their own priorities, so a tenant opening -more subscriptions never buys more bandwidth. The domain key and its scope are -[Scope track priority](/quest/m1/track-priority-scope.md)'s; this quest owns -the tier and the weighting. - -### Prototype and compare the hierarchy - -The delivered API is hierarchical send groups. Do not require a separately -published scalar-widening API or drop fairness as an intermediate release. -Prototype the byte-accounted scheduler in `moq-quic`. An existing Quinn prototype can -supply a workload baseline, but a recency field alone does not prove byte -fairness: differently sized writes must spend the group's byte credit, and -requeueing or adding streams must not reset that credit. - -Compare the hierarchy with the existing MoQ queue and two private benchmark -baselines: a scalar `[priority][group sequence]`, and a scalar containing only -priority. The first removes queue coordination but lets unrelated sequence -numbers compete across subscriptions; the second exposes per-stream fairness -but loses newest-first ordering within a subscription. Neither is the public -API this quest ships. - -Keep limitations explicit in the scalar benchmark. A 32-bit key leaves only -24 bits after track priority, so wraparound and sparse sequence values can -invert group order. Browser numeric precision and narrow backend urgency -fields impose different limits. Include these cases and compare audio/video -subscriptions with different group cadences; do not treat scalar ordering as -an exact equivalent of the hierarchy. - -Record queue/lock work, CPU, throughput, latency, and byte fairness on the -same congested workloads. Both equal-priority subscriptions must progress -while a higher-priority subscription preempts them and each subscription -sheds its own old backlog. Measure the full scope of trait and adapter changes -before publishing the API. - -Retransmissions follow the same hierarchy. `moq-quic` already re-queues a lost -range through the stream's priority (`StreamsState::retransmit`), so a lost -video range waits behind new audio at a higher priority. Keep -`retransmit_stream_priority` passing, and add a test proving a retransmission -is never starved indefinitely by an equal-priority group's new data (the -fair tier's byte credit covers retransmits too). - -Tests saturate the sender with differently sized audio and video groups and -prove byte fairness over a bounded window, strict preemption by a higher -priority, newest-first backlog shedding, dynamic priority updates, -blocked-stream handling, sequence wrap and sparse sequence values, and cleanup -on reset. This quest owns native QUIC proof and -reusable scheduling fixtures. [qmux](/quest/m2/quic-qmux.md) owns running those -fixtures through its record writer after adopting the scheduler; native -scheduler completion must not wait for that dependent integration. Preserve -working behavior on backends not yet migrated, and remove queue code only -where the new implementation makes it redundant. - -## Closes - -- [#699](https://github.com/moq-dev/moq/issues/699) - close this issue when the - quest finishes - -## Related - -- [Schedule datagrams by subscription priority](/quest/m1/quic/datagram-priority.md) - - adds a datagram queue to each send group once this lands -- [moq#3320](https://github.com/moq-dev/moq/pull/3320) - removes the current - dense-rank queue from the wide scalar path and records why a scalar cannot - provide this fairness level -- [Ladder controller](/quest/m3/ladder/controller.md) - rendition priority is - a policy consumer of the same hierarchy -- [Scope track priority](/quest/m1/track-priority-scope.md) - owns the - priority semantics this mechanism realizes, including the scheduling-domain - scope and the priority type change; keep the ordering, not just the type -- [noq#816](https://github.com/n0-computer/noq/issues/816) - the send-group proposal to n0; this quest supersedes it -- [noq#817](https://github.com/n0-computer/noq/issues/817) - the 64-bit priority ask; send groups replace the scalar, but the per-stream order within a group must still hold a 64-bit group sequence diff --git a/quest/m1/track-priority-scope.md b/quest/m1/track-priority-scope.md index f43efa89d69..ea657358e09 100644 --- a/quest/m1/track-priority-scope.md +++ b/quest/m1/track-priority-scope.md @@ -119,11 +119,6 @@ Public API impact: `u8` priority fields become `i8` in every language, and in format, but the default byte moves from 127 to 128 on moq-lite and from 128 to 127 on IETF. Report both with the draft change. -## Required - -- [Hierarchical stream scheduling](/quest/m1/quic/scheduler.md) - the fair - tier the broadcast domains and subscription send groups ride on - ## Related - [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - owns the group diff --git a/quest/m2/README.md b/quest/m2/README.md index b955da7d258..0ab57c8fbc2 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -119,6 +119,7 @@ ranked just below deterministic TS legs, which it requires. become resets, and a tail loss probe fires early while there is still time - [qmux on the QUIC stream state machine](/quest/m2/quic-qmux.md) - qmux is a first-class crate over `moq-quic`'s shared stream state machine +- [Drop web-transport-trait](/quest/m2/drop-web-transport-trait.md) - every backend implements moq-net's `transport::poll` traits, and the generic `web_transport_trait` adapters are deleted - [Align BBR loss handling with draft-06](/quest/m2/quic-bbr-loss-parity.md) - losses use their own sample and the draft-05 links go, keeping the spurious-loss undo that already re-enters ProbeUp through Refill - [Measure ECN on the backbone](/quest/m2/quic-ecn-measure.md) - a written verdict on marking versus dropping, and whether Linode and OVH keep marks diff --git a/quest/m2/drop-web-transport-trait.md b/quest/m2/drop-web-transport-trait.md new file mode 100644 index 00000000000..3c05dd0b474 --- /dev/null +++ b/quest/m2/drop-web-transport-trait.md @@ -0,0 +1,47 @@ +# [L] Drop web-transport-trait from the Rust stack + +## Goal + +moq-tokio, moq-uring, moq-relay, moq-ffi and moq-wasm no longer depend on +`web-transport-trait`. The qmux, iroh and wasm backends implement moq-net's +`transport::poll` traits, either directly or through a thin adapter per +backend. The generic adapters over `web_transport_trait` are deleted: +moq-tokio's `transport::Session` with its async-to-poll emulation, and +moq-uring's `transport::Session`. The sealed `Grouped`/`GroupedSend` pair +goes with them. A custom transport implements moq-net's traits, which are +already generic and send-group-aware (`Session::send_group`, +`SendStream::Group`, `Ungrouped`). + +## Plan + +Decided 2026-10-16 in the review of #5711: the extension point for custom +transports is moq-net's `transport::poll`, not `web_transport_trait`. #5711 +sealed the adapters' bound to the crate's own backends, so a foreign +`web_transport_trait::Session` no longer fits them. This quest removes the +layer instead of reopening it. + +- Native QUIC (`moq_tokio::quic::web`, `moq_uring::quic::web`) implements + `poll::Session` itself, keeping its real send groups. It is a candidate for + the first step, since its `web_transport_trait` impl only feeds the adapter. +- qmux and iroh get their own impls. Flat backends use `Ungrouped`. +- The browser backend in moq-wasm and the ffi backend implement the traits + directly. Their adapters already clamp priorities themselves. +- Report the public API impact per crate: deleted generic `Session` + adapters, and new impls on concrete session types. + +Notes from related quests: +- [qmux on moq-quic](/quest/m2/quic-qmux.md) planned to port qmux's + `web-transport-trait` impl. If it lands first, its impl targets moq-net's + `transport::poll` instead. +- [poll_acked](/quest/m2/quic-ack-hook.md) already puts its hook on moq-net's + trait because moq-net has no `web-transport-trait` dependency. That holds + here. +- [Rust request credit](/quest/m1/rs-request-credit.md) prefers adding a + credit query to qmux and iroh upstream in moq-dev/web-transport. Once this + lands, that query belongs on each backend's own moq-net impl. + +## Related + +- [qmux on the QUIC stream state machine](/quest/m2/quic-qmux.md) - the in-tree qmux implements the moq-net traits directly +- [poll_acked on moq-net's send stream](/quest/m2/quic-ack-hook.md) - another hook that lives only on moq-net's traits +- [Rust requests fail fast without stream credit](/quest/m1/rs-request-credit.md) - its credit query moves to each backend's moq-net impl diff --git a/quest/m2/quic-deadline.md b/quest/m2/quic-deadline.md index 66294205c9c..a4509756709 100644 --- a/quest/m2/quic-deadline.md +++ b/quest/m2/quic-deadline.md @@ -51,11 +51,6 @@ raises loss or latency under any profile stays off by default. Decided in the 2026-09-30 audit: moved to m2. No m1 quest consumes it. -## Required - -- [Hierarchical stream scheduling](/quest/m1/quic/scheduler.md) - the - scheduler decides which stream's data a probe carries - ## Related - [QUIC receive-timestamps spike](/quest/m3/quic-gcc.md) - measures whether diff --git a/quest/m2/quic-qmux.md b/quest/m2/quic-qmux.md index 5daecf88dfc..08c4f245f82 100644 --- a/quest/m2/quic-qmux.md +++ b/quest/m2/quic-qmux.md @@ -16,7 +16,8 @@ Decided in the 2026-09-30 plan: - **Scope is the Rust core plus adapters.** Port the prototype core and moq-dev/web-transport's TCP, TLS, WebSocket, and UDS adapters, ALPN negotiation, byte-stream framing, and socket stats (about 2.3k lines) onto - it, along with the `web-transport-trait` impl. The TypeScript `@moq/qmux` + it. Its session implements moq-net's `transport::poll` traits instead of + `web-transport-trait` ([Drop web-transport-trait](/quest/m2/drop-web-transport-trait.md)). The TypeScript `@moq/qmux` and `@moq/web-socket-stream` stay in moq-dev/web-transport with their own state machine and become the interop reference. - **All four wire versions stay**: draft-02, draft-01, draft-00, and the @@ -76,11 +77,6 @@ the uring runtime never unifies two copies of the stream state. Decided in the 2026-09-30 audit: moved to m2, since no m1 quest consumes it; the fork plan the same day kept it there. -## Required - -- [Hierarchical stream scheduling](/quest/m1/quic/scheduler.md) - qmux must - expose the same scheduling contract as native QUIC - ## Related - [noq#812](https://github.com/n0-computer/noq/issues/812) - the qmux proposal to n0 diff --git a/quest/m3/cut-through/README.md b/quest/m3/cut-through/README.md index 819b138c7b1..c10ef675a5a 100644 --- a/quest/m3/cut-through/README.md +++ b/quest/m3/cut-through/README.md @@ -75,6 +75,5 @@ stats field, which its quest documents inline. ## Related -- [Hierarchical stream scheduling](/quest/m1/quic/scheduler.md) - orders streams; offset writes order ranges within one - [QoS](/quest/m2/qos/README.md) - the loss-delay counter follows its per-broadcast ingress row conventions - [Frame slots are cached for free](https://github.com/moq-dev/moq/pull/5239) - changes how the cache charges the in-flight frames a cut-through build would hold diff --git a/quest/m3/ladder/controller.md b/quest/m3/ladder/controller.md index 117a4e2f243..9f61396bc03 100644 --- a/quest/m3/ladder/controller.md +++ b/quest/m3/ladder/controller.md @@ -74,5 +74,3 @@ priority where the subscriber's order wins, and a custom ladder order. ## Related - [Encoder feedback](/quest/m3/stats-encoder-feedback.md) - viewer stalls for `moq import`; its `moq transcode --echo` leg, split out if revived, feeds this controller -- [Hierarchical stream scheduling](/quest/m1/quic/scheduler.md) - supplies the - fair subscription buckets beneath this rendition policy diff --git a/rs/moq-ffi/src/transport/adapter.rs b/rs/moq-ffi/src/transport/adapter.rs index 0ebe75b7b14..bb2e27bfdfb 100644 --- a/rs/moq-ffi/src/transport/adapter.rs +++ b/rs/moq-ffi/src/transport/adapter.rs @@ -77,6 +77,9 @@ impl + Clone + 'st .map_err(Error) .map(|res| res.map(|(send, recv)| (SendStream(send), RecvStream(recv)))) } + fn send_group(&mut self, _: i64) -> Option { + None + } fn poll_send_datagram(&mut self, cx: &mut Context<'_>, payload: &[u8]) -> Poll> { backend::Session::poll_send_datagram(&mut self.0, cx, payload).map_err(Error) } @@ -106,15 +109,20 @@ impl + Clone + 'st pub struct SendStream(S); impl poll::SendStream for SendStream { type Error = Error; + type Group = poll::Ungrouped; fn poll_write(&mut self, cx: &mut Context<'_>, buf: &[u8]) -> Poll> { backend::SendStream::poll_write(&mut self.0, cx, buf).map_err(Error) } fn poll_write_buf(&mut self, cx: &mut Context<'_>, buf: &mut B) -> Poll> { backend::SendStream::poll_write_buf(&mut self.0, cx, buf).map_err(Error) } - fn set_priority(&mut self, order: i32) { + fn set_priority(&mut self, order: i64) { + let order = order.clamp(i32::MIN.into(), i32::MAX.into()) as i32; backend::SendStream::set_priority(&mut self.0, order) } + fn set_group(&mut self, group: &Self::Group) { + match *group {} + } fn finish(&mut self) -> Result<(), Self::Error> { backend::SendStream::finish(&mut self.0).map_err(Error) } diff --git a/rs/moq-net/Cargo.toml b/rs/moq-net/Cargo.toml index 10c304767ba..49b0c680bc9 100644 --- a/rs/moq-net/Cargo.toml +++ b/rs/moq-net/Cargo.toml @@ -131,6 +131,12 @@ name = "priority" harness = false required-features = ["async"] +# Compiles moq-quic's send scheduler from source to compare it with the priority queue. +[[bench]] +name = "schedule" +harness = false +required-features = ["async"] + [[bench]] name = "stats" harness = false diff --git a/rs/moq-net/benches/schedule.rs b/rs/moq-net/benches/schedule.rs new file mode 100644 index 00000000000..0a531ce5427 --- /dev/null +++ b/rs/moq-net/benches/schedule.rs @@ -0,0 +1,489 @@ +//! Send scheduling on a congested link: the transport's send groups against the publishers' +//! session-wide priority queue and two scalar orderings. +//! +//! Each workload runs every scheduler once and prints its byte shares, live-edge lag, and +//! queue work, then criterion times the run. The four schedulers: +//! +//! - `groups`: one send group per subscription at its priority, streams ordered by group +//! sequence inside it (what native QUIC uses). +//! - `queue`: the publishers' `PriorityQueue`, ranking every in-flight group on the session +//! into one `u8` send order, re-applied to each stream whose rank moves. +//! - `scalar-seq`: a 32-bit `[priority][group sequence]` key, which leaves 24 bits for the +//! sequence and compares sequences across subscriptions. +//! - `scalar`: the priority alone, so equal subscriptions share by stream count. +//! +//! `fanout` sweeps subscriptions against groups in flight per subscription, timing one frame, +//! so a scheduling cost that grows with the table rather than the touched path is a slope. + +use std::collections::HashMap; +use std::sync::{Arc, Mutex}; +use std::task::{Wake, Waker}; + +use criterion::{BatchSize, BenchmarkId, Criterion, criterion_group, criterion_main}; + +// Both schedulers compile straight from source, so neither widens its crate's API for this. +#[path = "../src/priority.rs"] +#[allow(dead_code)] +mod priority; + +// The queue names `crate::track::Order`, which is public. +mod track { + pub use moq_net::track::Order; +} + +// Keeps moq-quic's own formatting. +#[rustfmt::skip] +#[path = "../../moq-quic/src/connection/streams/schedule.rs"] +#[allow(dead_code)] +mod schedule; + +use priority::{Priority, PriorityHandle, PriorityQueue}; +use schedule::{Schedule, Slot}; + +/// The most a frame carries, one packet. +const PACKET: usize = 1200; + +/// One subscription's media: `rate` bytes a tick into its open group, a new group every +/// `cadence` ticks, numbered from `first` in steps of `step`. +#[derive(Clone, Copy)] +struct Track { + priority: u8, + rate: usize, + cadence: u64, + first: u64, + step: u64, +} + +impl Track { + const fn new(priority: u8, rate: usize, cadence: u64) -> Self { + Self { + priority, + rate, + cadence, + first: 0, + step: 1, + } + } + + const fn numbered(mut self, first: u64, step: u64) -> Self { + self.first = first; + self.step = step; + self + } +} + +struct Workload { + name: &'static str, + tracks: Vec, + /// Bytes the link takes per tick. + capacity: usize, + ticks: u64, +} + +fn workloads() -> Vec { + vec![ + // The issue: audio and video at one priority, under a higher-priority track that + // takes a quarter of the link. Video has a deep backlog of large groups; audio + // opens a small group every tick. + Workload { + name: "alice-bob", + tracks: vec![ + Track::new(5, 5_000, 2), + Track::new(4, 30_000, 30), + Track::new(4, 15_000, 1), + ], + capacity: 20_000, + ticks: 3_000, + }, + // Group sequences crossing 2^24, where the scalar's 24 bits wrap. + Workload { + name: "wrap", + tracks: vec![ + Track::new(4, 15_000, 2).numbered((1 << 24) - 100, 1), + Track::new(4, 15_000, 2).numbered((1 << 24) - 100, 1), + ], + capacity: 20_000, + ticks: 3_000, + }, + // Groups numbered by a millisecond clock, which wraps 24 bits within hours. + Workload { + name: "sparse", + tracks: vec![ + Track::new(4, 15_000, 2).numbered((1 << 24) - 50_000, 1_000), + Track::new(4, 15_000, 2).numbered(1 << 30, 1_000), + ], + capacity: 20_000, + ticks: 3_000, + }, + // Many equal subscriptions with different cadences sharing one link. + Workload { + name: "crowd", + tracks: (0..64) + .map(|i| Track::new(4, 1_000 + 100 * i, 1 + i as u64 % 8)) + .collect(), + capacity: 50_000, + ticks: 2_000, + }, + ] +} + +#[derive(Clone, Copy, Debug)] +enum Kind { + Groups, + Queue, + ScalarSeq, + Scalar, +} + +const KINDS: [Kind; 4] = [Kind::Groups, Kind::Queue, Kind::ScalarSeq, Kind::Scalar]; + +impl Kind { + fn name(self) -> &'static str { + match self { + Self::Groups => "groups", + Self::Queue => "queue", + Self::ScalarSeq => "scalar-seq", + Self::Scalar => "scalar", + } + } +} + +struct Stream { + track: usize, + seq: u64, + slot: Slot, + priority: i64, + /// Bytes written but not yet sent. + queued: usize, + /// Whether the group is complete, so the stream closes once drained. + finished: bool, + opened: u64, + rank: Option, +} + +/// Collects the streams whose rank moved, the way each group serve is woken. +struct Moved { + id: u64, + list: Arc>>, +} + +impl Wake for Moved { + fn wake(self: Arc) { + self.list.lock().unwrap().push(self.id); + } +} + +#[derive(Default, Clone)] +struct Share { + bytes: u64, + /// The newest group opened and the newest delivered whole. + opened: Option, + delivered: Option, + /// Ticks from a group's completion to its last byte, summed over delivered groups. + delay: u64, + groups: u64, +} + +struct Sim { + kind: Kind, + tracks: Vec, + capacity: usize, + schedule: Schedule, + groups: Vec, + queue: PriorityQueue, + moved: Arc>>, + streams: HashMap, + current: Vec, + next: u64, + tick: u64, + shares: Vec, + /// Stream priority changes the scheduler needed, beyond each stream's first. + work: u64, +} + +impl Sim { + fn new(kind: Kind, workload: &Workload) -> Self { + let mut schedule = Schedule::new(); + let groups = workload + .tracks + .iter() + .map(|track| schedule.open_group(track.priority.into(), None)) + .collect(); + let mut sim = Self { + kind, + tracks: workload.tracks.clone(), + capacity: workload.capacity, + schedule, + groups, + queue: PriorityQueue::default(), + moved: Arc::default(), + streams: HashMap::new(), + current: Vec::new(), + next: 0, + tick: 0, + shares: vec![Share::default(); workload.tracks.len()], + work: 0, + }; + for track in 0..sim.tracks.len() { + let first = sim.tracks[track].first; + let id = sim.open(track, first); + sim.current.push(id); + } + sim + } + + fn open(&mut self, track: usize, seq: u64) -> u64 { + let id = self.next; + self.next += 1; + let priority = self.tracks[track].priority; + let mut stream = Stream { + track, + seq, + slot: Slot::default(), + priority: 0, + queued: 0, + finished: false, + opened: self.tick, + rank: None, + }; + match self.kind { + Kind::Groups => { + self.schedule.join(id, &mut stream.slot, Some(self.groups[track]), 0); + stream.priority = seq as i64; + } + Kind::Queue => { + let mut rank = self.queue.insert(Priority::new(priority, track as u64, seq)); + stream.priority = PriorityHandle::send_order_of(rank.current()).into(); + let waker = Waker::from(Arc::new(Moved { + id, + list: self.moved.clone(), + })); + let _ = rank.poll_next(&kio::Waiter::new(waker)); + stream.rank = Some(rank); + } + Kind::ScalarSeq => stream.priority = (i64::from(priority) << 24) | (seq & 0xff_ffff) as i64, + Kind::Scalar => stream.priority = priority.into(), + } + let share = &mut self.shares[track]; + share.opened = Some(share.opened.map_or(seq, |opened| opened.max(seq))); + self.streams.insert(id, stream); + id + } + + fn close(&mut self, id: u64) { + let mut stream = self.streams.remove(&id).unwrap(); + self.schedule.leave(&mut stream.slot); + let share = &mut self.shares[stream.track]; + share.delivered = Some(share.delivered.map_or(stream.seq, |seq| seq.max(stream.seq))); + share.delay += self.tick - stream.opened; + share.groups += 1; + } + + /// Produce a tick of media, then send a tick's worth of it. + fn step(&mut self) { + self.tick += 1; + for track in 0..self.tracks.len() { + let Track { + rate, cadence, step, .. + } = self.tracks[track]; + if self.tick.is_multiple_of(cadence) { + let old = self.current[track]; + let stream = self.streams.get_mut(&old).unwrap(); + stream.finished = true; + let seq = stream.seq + step; + if stream.queued == 0 { + self.close(old); + } + self.current[track] = self.open(track, seq); + } + let id = self.current[track]; + let stream = self.streams.get_mut(&id).unwrap(); + if stream.queued == 0 { + self.schedule.push(id, &mut stream.slot, stream.priority); + } + stream.queued += rate; + } + + // Apply every rank the queue moved, as each woken group serve would. + let moved = std::mem::take(&mut *self.moved.lock().unwrap()); + for id in moved { + let Some(stream) = self.streams.get_mut(&id) else { + continue; + }; + let rank = stream.rank.as_mut().unwrap(); + let waker = Waker::from(Arc::new(Moved { + id, + list: self.moved.clone(), + })); + let waiter = kio::Waiter::new(waker); + while let std::task::Poll::Ready(rank) = rank.poll_next(&waiter) { + stream.priority = PriorityHandle::send_order_of(rank).into(); + self.work += 1; + } + } + + let mut budget = self.capacity; + while budget > 0 { + let Some(id) = self.schedule.pop() else { + break; + }; + let stream = self.streams.get_mut(&id).unwrap(); + let bytes = stream.queued.min(PACKET).min(budget); + stream.queued -= bytes; + budget -= bytes; + self.shares[stream.track].bytes += bytes as u64; + let requeue = (stream.queued > 0).then_some(stream.priority); + self.schedule.sent(id, &mut stream.slot, bytes, requeue, true); + if stream.queued == 0 && stream.finished { + self.close(id); + } + } + } + + fn run(&mut self, ticks: u64) { + for _ in 0..ticks { + self.step(); + } + } + + fn report(&self, workload: &str) { + let priorities: Vec = { + let mut p: Vec<_> = self.tracks.iter().map(|t| t.priority).collect(); + p.sort(); + p.dedup(); + p + }; + let mut line = format!("{workload:>10} {:>10} work {:>8}", self.kind.name(), self.work); + for priority in priorities.into_iter().rev() { + let bytes: Vec = self + .tracks + .iter() + .zip(&self.shares) + .filter(|(t, _)| t.priority == priority) + .map(|(_, s)| s.bytes as f64) + .collect(); + let sum: f64 = bytes.iter().sum(); + let squares: f64 = bytes.iter().map(|b| b * b).sum(); + let jain = if squares > 0.0 { + sum * sum / (bytes.len() as f64 * squares) + } else { + 0.0 + }; + line += &format!( + " p{priority}: {:.0} KB/tick jain {jain:.3}", + sum / self.tick as f64 / 1e3 + ); + } + let lag: Vec = self + .tracks + .iter() + .zip(&self.shares) + .map(|(track, share)| { + let lag = match (share.opened, share.delivered) { + (Some(opened), Some(delivered)) => (opened.saturating_sub(delivered) / track.step).to_string(), + _ => "-".to_string(), + }; + let delay = match share.groups { + 0 => "-".to_string(), + n => format!("{:.0}", share.delay as f64 / n as f64), + }; + format!("{lag}/{delay}") + }) + .collect(); + line += &format!(" lag/delay {}", lag.join(" ")); + eprintln!("{line}"); + } +} + +fn bench_workloads(c: &mut Criterion) { + eprintln!("byte share and Jain fairness per priority; lag in groups behind the live edge, delay in ticks"); + for workload in workloads() { + let mut group = c.benchmark_group(format!("schedule_{}", workload.name)); + for kind in KINDS { + let mut sim = Sim::new(kind, &workload); + sim.run(workload.ticks); + sim.report(workload.name); + + group.bench_function(kind.name(), |b| { + b.iter_batched( + || Sim::new(kind, &workload), + |mut sim| { + sim.run(workload.ticks); + sim + }, + BatchSize::LargeInput, + ); + }); + } + group.finish(); + } +} + +/// A saturated schedule: `subscriptions` groups at one priority, each with `streams` queued. +fn saturated(subscriptions: usize, streams: usize) -> (Schedule, Vec) { + let mut schedule = Schedule::new(); + let mut slots = Vec::with_capacity(subscriptions * streams); + for _ in 0..subscriptions { + let group = schedule.open_group(0, None); + for seq in 0..streams { + let id = slots.len() as u64; + let mut slot = Slot::default(); + schedule.join(id, &mut slot, Some(group), seq as i64); + schedule.push(id, &mut slot, seq as i64); + slots.push(slot); + } + } + (schedule, slots) +} + +fn bench_fanout(c: &mut Criterion) { + let mut group = c.benchmark_group("schedule_fanout_frame"); + for subscriptions in [1, 16, 256, 4_096] { + for streams in [1, 16, 256] { + if subscriptions * streams > 100_000 { + continue; + } + let (mut schedule, mut slots) = saturated(subscriptions, streams); + group.bench_with_input( + BenchmarkId::new(format!("subs_{subscriptions}"), streams), + &(), + |b, ()| { + b.iter(|| { + let id = schedule.pop().unwrap(); + let slot = &mut slots[id as usize]; + schedule.sent(id, slot, PACKET, Some((id % streams as u64) as i64), true); + }); + }, + ); + } + } + group.finish(); + + // A group opening and closing a stream, as each new MoQ group does. + let mut group = c.benchmark_group("schedule_fanout_churn"); + for subscriptions in [1, 16, 256, 4_096] { + for streams in [1, 16, 256] { + if subscriptions * streams > 100_000 { + continue; + } + // Opening then closing leaves the schedule as it was, so one serves every run. + let (mut schedule, _slots) = saturated(subscriptions, streams); + group.bench_with_input( + BenchmarkId::new(format!("subs_{subscriptions}"), streams), + &(), + |b, ()| { + b.iter(|| { + let mut slot = Slot::default(); + schedule.join(u64::MAX, &mut slot, Some(0), i64::MAX); + schedule.push(u64::MAX, &mut slot, i64::MAX); + schedule.leave(&mut slot); + }); + }, + ); + } + } + group.finish(); +} + +criterion_group!(benches, bench_workloads, bench_fanout); +criterion_main!(benches); diff --git a/rs/moq-net/src/client.rs b/rs/moq-net/src/client.rs index 395fde91f38..1a9d3a8b9e1 100644 --- a/rs/moq-net/src/client.rs +++ b/rs/moq-net/src/client.rs @@ -620,6 +620,10 @@ mod tests { Poll::Pending } + fn send_group(&mut self, _: i64) -> Option { + None + } + fn poll_send_datagram(&mut self, _cx: &mut Context<'_>, _payload: &[u8]) -> Poll> { Poll::Ready(Ok(())) } @@ -678,13 +682,18 @@ mod tests { impl crate::transport::poll::SendStream for FakeSendStream { type Error = FakeError; + type Group = crate::transport::poll::Ungrouped; fn poll_write(&mut self, _cx: &mut Context<'_>, buf: &[u8]) -> Poll> { self.writes.lock().unwrap().put_slice(buf); Poll::Ready(Ok(buf.len())) } - fn set_priority(&mut self, _order: i32) {} + fn set_priority(&mut self, _order: i64) {} + + fn set_group(&mut self, group: &Self::Group) { + match *group {} + } fn finish(&mut self) -> Result<(), Self::Error> { Ok(()) @@ -1006,6 +1015,10 @@ mod tests { }) } + fn send_group(&mut self, _: i64) -> Option { + None + } + fn poll_send_datagram(&mut self, cx: &mut Context<'_>, payload: &[u8]) -> Poll> { self.inner.poll_send_datagram(cx, payload) } @@ -1040,15 +1053,20 @@ mod tests { impl crate::transport::poll::SendStream for LocalSend { type Error = FakeError; + type Group = crate::transport::poll::Ungrouped; fn poll_write(&mut self, cx: &mut Context<'_>, buf: &[u8]) -> Poll> { self.inner.poll_write(cx, buf) } - fn set_priority(&mut self, order: i32) { + fn set_priority(&mut self, order: i64) { self.inner.set_priority(order); } + fn set_group(&mut self, group: &Self::Group) { + match *group {} + } + fn finish(&mut self) -> Result<(), Self::Error> { self.inner.finish() } diff --git a/rs/moq-net/src/coding/stream.rs b/rs/moq-net/src/coding/stream.rs index 1984620c73c..223d64179d5 100644 --- a/rs/moq-net/src/coding/stream.rs +++ b/rs/moq-net/src/coding/stream.rs @@ -67,7 +67,7 @@ impl Stream { V: Clone, { let mut writer = Writer::new(send, version.clone()); - writer.set_priority(CONTROL_SEND_ORDER); + writer.set_priority(CONTROL_SEND_ORDER.into()); Self { writer, reader: Reader::new(recv, version), diff --git a/rs/moq-net/src/coding/writer.rs b/rs/moq-net/src/coding/writer.rs index 037a095676c..d0a52c4e38c 100644 --- a/rs/moq-net/src/coding/writer.rs +++ b/rs/moq-net/src/coding/writer.rs @@ -201,14 +201,20 @@ impl Writer { std::future::poll_fn(|cx| self.poll_closed(cx)).await } - /// Set the stream's send order: streams with HIGHER values are transmitted first. + /// Set the priority within the stream's send group, or session-wide when ungrouped. /// - /// This is the transport trait's convention (matching W3C `sendOrder` and quinn's - /// scheduler) and the model's [`Subscription::priority`](crate::track::Subscription), - /// where higher values preempt lower ones. The lite priority queue's rank is the - /// opposite (0 = most urgent); rank holders convert via `PriorityHandle::send_order`. - pub fn set_priority(&mut self, send_order: u8) { - self.stream.as_mut().unwrap().set_priority(send_order.into()); + /// Streams with HIGHER values are transmitted first. This is the transport trait's + /// convention (matching W3C `sendOrder` and quinn's scheduler) and the model's + /// [`Subscription::priority`](crate::track::Subscription), where higher values preempt + /// lower ones. The lite priority queue's rank is the opposite (0 = most urgent); rank + /// holders convert via `PriorityHandle::send_order`. + pub fn set_priority(&mut self, priority: i64) { + self.stream.as_mut().unwrap().set_priority(priority); + } + + /// Move the stream into a send group from its session. + pub fn set_group(&mut self, group: &S::Group) { + self.stream.as_mut().unwrap().set_group(group); } /// Cast the writer to a different version, used during version negotiation. @@ -265,12 +271,17 @@ mod tests { impl crate::transport::poll::SendStream for Stopped { type Error = Self; + type Group = crate::transport::poll::Ungrouped; fn poll_write(&mut self, _: &mut Context<'_>, _: &[u8]) -> Poll> { Poll::Ready(Err(self.clone())) } - fn set_priority(&mut self, _: i32) {} + fn set_priority(&mut self, _: i64) {} + + fn set_group(&mut self, group: &Self::Group) { + match *group {} + } fn finish(&mut self) -> Result<(), Self::Error> { Err(self.clone()) @@ -323,7 +334,7 @@ mod tests { let mut writer = Writer::new(SinkSend::new(log.clone()), crate::lite::Version::Lite05); for send_order in 0u8..=255 { - writer.set_priority(send_order); + writer.set_priority(send_order.into()); } assert_eq!(log.priorities(), (0u8..=255).collect::>()); diff --git a/rs/moq-net/src/ietf/adapter.rs b/rs/moq-net/src/ietf/adapter.rs index 03ab85c3f76..dc7a2d80aa4 100644 --- a/rs/moq-net/src/ietf/adapter.rs +++ b/rs/moq-net/src/ietf/adapter.rs @@ -364,6 +364,7 @@ impl VirtualSendStream { impl crate::transport::poll::SendStream for VirtualSendStream { type Error = crate::Error; + type Group = crate::transport::poll::Ungrouped; fn poll_write(&mut self, _cx: &mut Context<'_>, buf: &[u8]) -> Poll> { let len = buf.len(); @@ -378,7 +379,11 @@ impl crate::transport::poll::SendStream for VirtualSendStream { Poll::Ready(self.push(chunk).map(|()| len)) } - fn set_priority(&mut self, _order: i32) {} + fn set_priority(&mut self, _order: i64) {} + + fn set_group(&mut self, group: &Self::Group) { + match *group {} + } fn finish(&mut self) -> Result<(), Self::Error> { // Flush any remaining buffered data (e.g. if registration never completed). @@ -406,6 +411,7 @@ pub enum AdapterSend { impl crate::transport::poll::SendStream for AdapterSend { type Error = crate::Error; + type Group = ::Group; fn poll_write(&mut self, cx: &mut Context<'_>, buf: &[u8]) -> Poll> { match self { @@ -421,13 +427,21 @@ impl crate::transport::poll::SendStream for } } - fn set_priority(&mut self, order: i32) { + fn set_priority(&mut self, order: i64) { match self { Self::Real(s) => s.set_priority(order), Self::Virtual(s) => s.set_priority(order), } } + fn set_group(&mut self, group: &Self::Group) { + match self { + Self::Real(s) => s.set_group(group), + // Rides the control stream, which no group schedules. + Self::Virtual(_) => {} + } + } + fn finish(&mut self) -> Result<(), Self::Error> { match self { Self::Real(s) => s.finish().map_err(|_| crate::Error::Closed), @@ -1158,6 +1172,10 @@ impl crate::transport::poll::Session for Con .map(|r| r.map(AdapterRecv::Real).map_err(|_| crate::Error::Closed)) } + fn send_group(&mut self, priority: i64) -> Option<::Group> { + self.inner.send_group(priority) + } + fn poll_send_datagram(&mut self, cx: &mut Context<'_>, payload: &[u8]) -> Poll> { self.inner .poll_send_datagram(cx, payload) diff --git a/rs/moq-net/src/ietf/publisher.rs b/rs/moq-net/src/ietf/publisher.rs index 9f27203e04d..1171af03076 100644 --- a/rs/moq-net/src/ietf/publisher.rs +++ b/rs/moq-net/src/ietf/publisher.rs @@ -11,7 +11,8 @@ use std::{ time::Duration, }; -use crate::transport::poll::SendStream as _; +use crate::transport::poll::GroupOf; +use crate::transport::poll::SendGroup as _; use crate::{ AsPath, DecodeError, Error, Timescale, Timestamp, @@ -517,7 +518,7 @@ pub(super) struct Publisher { // What the peer declared in its SETUP, filled when that stream is read. peer_setup: peer::PeerSetup, // Shared across request handlers; None marks a dispatched subscription still resolving. - joins: kio::Shared>>, + joins: kio::Shared>>>>, version: Version, // Our grant (MoQ Auth): only what it lets us publish is advertised and served, and a // shrink withdraws what it no longer covers. @@ -541,7 +542,7 @@ impl Drop for Serve { /// The snapshot a joining FETCH inherits from its subscription. #[derive(Clone)] -enum Joined { +enum Joined { /// This subscription's filter is not supported by the joining FETCH handler. Unsupported, /// No objects existed when the subscription started. @@ -556,7 +557,7 @@ enum Joined { /// the fetch either. token: bool, /// The subscription's ranking, which the fetch joins. - ranking: Ranking, + ranking: Ranking, }, } @@ -564,12 +565,12 @@ enum Joined { /// /// A fetch that arrives afterwards then finds nothing and is refused, rather than being /// answered from a group that is no longer the edge of anything. -struct Join { - joins: kio::Shared>>, +struct Join { + joins: kio::Shared>>>, request_id: RequestId, } -impl Drop for Join { +impl Drop for Join { fn drop(&mut self) { self.joins.lock().remove(&self.request_id); } @@ -1626,7 +1627,7 @@ where async fn run_fill( &self, fill: FillServe, - ranking: &Ranking, + ranking: &Ranking>, track: track::Consumer, timescale: Option, opened: &AtomicU64, @@ -1945,7 +1946,7 @@ where } /// Register a pending subscription until the returned guard drops. - fn register_join(&self, request_id: RequestId) -> Join { + fn register_join(&self, request_id: RequestId) -> Join> { self.joins.lock().insert(request_id, None); Join { joins: self.joins.clone(), @@ -2090,10 +2091,12 @@ where Err(err) => return self.reject_fetch(stream, msg.request_id, &err, &err.to_string()).await, }; - // Its own entry in the queue, so it competes with group streams by its priority - // alone, its request id breaking ties. Nothing updates a FETCH's priority. + // Its own entry in the queue, or its own send group, so it competes with group + // streams by its priority alone, its request id breaking ties in the queue. + // Nothing updates a FETCH's priority. let ranking = Ranking { queue: self.priority.clone(), + group: self.session.clone().send_group(priority.into()), request_id: msg.request_id, priority: kio::Producer::new(priority).consume(), order: Order::Descending, @@ -2403,7 +2406,10 @@ where &mut self, stream: &mut Stream, subscribe_id: RequestId, - ) -> Result, Error> { + ) -> Result< + Result<(Location, track::Consumer, crate::PathOwned, Ranking>, bool), (Error, &'static str)>, + Error, + > { // Request streams can arrive out of order. Wait on registration, while bounding // the lifetime of a request whose subscription never arrives or resolves. let joined = { @@ -3567,8 +3573,9 @@ struct TrackServe { children: kio::Tasks>, /// The subscriber priority, which every group stream follows while it is open. priority: kio::Producer, - /// Where the group streams are ranked: the session's queue once [`Self::ranked`]. - ranking: Ranking, + /// Where the group streams are ranked: the subscription's send group, or the session's + /// queue once [`Self::ranked`] on a transport without groups. + ranking: Ranking>, /// The track finished: the in-flight group machines drain, then FIN. draining: bool, /// Group streams opened, shared with the group machines. @@ -3607,8 +3614,10 @@ impl TrackServe { // objects on those drafts go out unstamped. let timescale = timescale.filter(|_| ietf::Properties::sends_timescale(version)); let priority = kio::Producer::new(track.subscription().priority); + let mut session = session; let ranking = Ranking { queue: PriorityQueue::default(), + group: session.send_group(track.subscription().priority.into()), request_id, priority: priority.consume(), order: track.subscription().order, @@ -3649,6 +3658,10 @@ impl TrackServe { if let Ok(mut current) = self.priority.write() { *current = priority; } + // One move for the whole subscription; its streams keep their place within it. + if let Some(group) = &mut self.ranking.group { + group.set_priority(priority.into()); + } Ok(()) } @@ -3921,19 +3934,27 @@ impl TrackServe { /// Where one request's streams are ranked: the session's queue, keyed by the request's id /// and following its subscriber priority. #[derive(Clone)] -struct Ranking { +struct Ranking { queue: PriorityQueue, + /// The request's send group, which replaces the queue on a transport that has them. + group: Option, request_id: RequestId, priority: kio::Consumer, /// The request's GROUP_ORDER, fixed for its life: no update carries it. order: Order, } -impl Ranking { +impl Ranking { /// A stream carrying a group from `start` on, ranked among the request's others. - fn rank(&self, start: Location) -> Rank { + fn rank(&self, start: Location) -> Rank { + if let Some(group) = &self.group { + return Rank::Group { + group: group.clone(), + order: crate::priority::in_group(start.group, start.object, self.order), + }; + } let applied = *self.priority.read(); - Rank { + Rank::Queue(Queued { handle: self.queue.insert( Priority::new(applied, self.request_id.0, start.group) .with_order(self.order) @@ -3942,7 +3963,7 @@ impl Ranking { priority: self.priority.clone(), applied, order: self.order, - } + }) } } @@ -3952,17 +3973,17 @@ impl Ranking { /// Every write and close poll applies a move first. A stream parked on its group still /// polls its close to notice the subscriber leaving, so a move reaches the transport /// whatever the stream waits on. -struct Ranked { +struct Ranked { stream: T, /// Absent for a stream that only carries a refusal. - rank: Option, + rank: Option>, park: kio::Park, } impl Ranked { - fn new(mut stream: T, mut rank: Option) -> Self { + fn new(mut stream: T, mut rank: Option>) -> Self { if let Some(rank) = &mut rank { - stream.set_priority(rank.handle.send_order().into()); + rank.place(&mut stream); } Self { stream, @@ -3975,13 +3996,14 @@ impl Ranked { if let Some(rank) = &mut self.rank && let Some(order) = rank.poll_moved(self.park.hold(cx)) { - self.stream.set_priority(order.into()); + self.stream.set_priority(order); } } } impl crate::transport::poll::SendStream for Ranked { type Error = T::Error; + type Group = T::Group; fn poll_write(&mut self, cx: &mut std::task::Context<'_>, buf: &[u8]) -> Poll> { self.follow(cx); @@ -3997,10 +4019,14 @@ impl crate::transport::poll::SendStream f self.stream.poll_write_buf(cx, buf) } - fn set_priority(&mut self, order: i32) { + fn set_priority(&mut self, order: i64) { self.stream.set_priority(order); } + fn set_group(&mut self, group: &Self::Group) { + self.stream.set_group(group); + } + fn finish(&mut self) -> Result<(), Self::Error> { self.stream.finish() } @@ -4027,7 +4053,43 @@ async fn settle(writer: &mut Writer { + /// In the request's send group, at a place fixed for the stream's life: the request's + /// priority moves the whole group. + Group { group: G, order: i64 }, + /// In the session's queue, on a transport without send groups. + Queue(Queued), +} + +impl Rank { + /// A group ranked only against itself, for driving one group machine alone. + #[cfg(test)] + fn solo(priority: kio::Consumer) -> Self { + Self::Queue(Queued::solo(priority)) + } + + /// Put a newly opened stream in its place. + fn place>(&mut self, stream: &mut T) { + match self { + Self::Group { group, order } => { + stream.set_group(group); + stream.set_priority(*order); + } + Self::Queue(queued) => stream.set_priority(queued.handle.send_order().into()), + } + } + + /// A send order for the stream once the subscription priority or the queue moved it. + fn poll_moved(&mut self, waiter: &kio::Waiter) -> Option { + match self { + Self::Group { .. } => None, + Self::Queue(queued) => queued.poll_moved(waiter).map(Into::into), + } + } +} + +/// A stream's entry in the session's queue, following its subscription's priority. +struct Queued { handle: PriorityHandle, /// The subscription's priority, followed while the stream is open. priority: kio::Consumer, @@ -4037,7 +4099,7 @@ struct Rank { order: Order, } -impl Rank { +impl Queued { /// A group ranked only against itself, for driving one group machine alone. #[cfg(test)] fn solo(priority: kio::Consumer) -> Self { @@ -4075,7 +4137,7 @@ struct GroupServe { /// The subscription's count of opened streams, bumped once this one opens. opened: Arc, msg: ietf::GroupHeader, - rank: Rank, + rank: Rank>, group: group::Consumer, timescale: Option, version: Version, @@ -4127,7 +4189,7 @@ impl GroupServe { fn new( session: S, msg: ietf::GroupHeader, - rank: Rank, + rank: Rank>, mut group: group::Consumer, timescale: Option, version: Version, @@ -4190,7 +4252,7 @@ impl GroupServe { self.opened.fetch_add(1, Ordering::Relaxed); let mut stream = stream; // The current rank, which already includes anything `moved` saw. - stream.set_priority(self.rank.handle.send_order().into()); + self.rank.place(&mut stream); moved = None; let mut writer = Writer::new(stream, self.version); @@ -4777,14 +4839,14 @@ mod group_priority_test { } /// A group of `subscribe` ranked in `queue` at `priority`'s current value. - fn rank_in(queue: &PriorityQueue, priority: &kio::Producer, subscribe: u64) -> Rank { + fn rank_in(queue: &PriorityQueue, priority: &kio::Producer, subscribe: u64) -> Rank { let applied = *priority.read(); - Rank { + Rank::Queue(Queued { handle: queue.insert(Priority::new(applied, subscribe, 0)), priority: priority.consume(), applied, order: Order::Descending, - } + }) } /// The model's `Subscription::priority` is higher-first ("higher values preempt @@ -4999,6 +5061,56 @@ mod group_priority_test { ); } + /// On a transport with send groups, a subscription opens one at its priority and its + /// group streams join it, ranked newest first by sequence alone: the session-wide queue + /// takes no part. A priority update moves the group once, not each stream. + #[test] + fn grouped_streams_rank_within_their_subscription() { + let log = crate::lite::test_transport::Log::default(); + let session = SinkSession::new(log.clone()).with_groups().with_unacked_fin(); + let track = track::Producer::new( + std::sync::Arc::new(crate::broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + let subscriber = track.subscribe( + track::Subscription::default() + .with_priority(3) + .with_max_delay(std::time::Duration::from_secs(3600)), + ); + let mut serve = TrackServe::new( + session, + subscriber, + RequestId(0), + Version::Draft14, + ServeRange::default(), + None, + ); + for sequence in 0..2 { + let mut group = track.create_group(group::Info { sequence }).unwrap(); + group + .write_frame(crate::Timestamp::from_millis(sequence).unwrap(), b"x".as_slice()) + .unwrap(); + group.finish().unwrap(); + } + + let waiter = kio::Waiter::noop(); + assert!(serve.poll(&waiter).is_pending()); + assert_eq!(log.groups(), [(0, 3)]); + assert_eq!( + log.joins(), + [(0, 0), (1, 0)], + "each stream joins the subscription's group" + ); + assert_eq!(log.orders(), [(0, 1), (1, 3)], "newer groups first"); + assert!(log.priorities().is_empty(), "the session queue ranks no stream"); + + serve.set_priority(9).unwrap(); + assert!(serve.poll(&waiter).is_pending()); + assert_eq!(log.groups(), [(0, 3), (0, 9)], "one move for the subscription"); + assert_eq!(log.orders().len(), 2, "the streams keep their place in the group"); + } + /// A group with more objects ready than one poll's budget drains over several polls. #[test] fn a_long_group_drains_within_the_budget() { @@ -7415,6 +7527,7 @@ mod serve_tests { let priority = kio::Producer::new(PRIORITY); let ranking = Ranking { queue: queue.clone(), + group: None, request_id: RequestId(REQUEST_ID), priority: priority.consume(), order: Order::Descending, @@ -7469,12 +7582,15 @@ mod serve_tests { let ranking = Ranking { queue: h.publisher.priority.clone(), + group: None, request_id: RequestId(REQUEST_ID), priority: kio::Producer::new(PRIORITY).consume(), order: Order::Descending, }; // The live stream, from object 1 on, queued first so the slab id alone would favor it. - let mut live = ranking.rank(Location { group: 4, object: 1 }); + let Rank::Queue(mut live) = ranking.rank(Location { group: 4, object: 1 }) else { + unreachable!("no send groups") + }; assert_eq!(live.handle.send_order(), 255); let opened = AtomicU64::new(0); diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 75a6cb6fa70..2af6c0377e4 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -20,6 +20,7 @@ use crate::{ lite, priority::{Priority, PriorityHandle, PriorityQueue}, track::Order, + transport::poll::{GroupOf, SendGroup as _}, }; use super::Version; @@ -2003,11 +2004,14 @@ impl Request for SubscribeServe { }; let steer_tx = kio::Producer::new(steer); + let mut session = shared.session.clone(); + let group = session.send_group(msg.priority.into()); let sub = Subscription { - session: shared.session.clone(), + session, id: msg.id, track_name: Arc::from(track.name()), priority: shared.priority.clone(), + group, steer: steer_tx.consume(), steer_seen: steer, version: shared.version, @@ -3141,6 +3145,9 @@ struct Subscription { id: u64, track_name: Arc, priority: PriorityQueue, + /// The subscription's send group, ranking it against the session's others, or `None` + /// on a transport without groups, where `priority` ranks every group stream instead. + group: Option>, steer: kio::Consumer, /// Last steer observed by this clone, so a change only fires once. steer_seen: Steer, @@ -3241,7 +3248,13 @@ impl Subscription { priority: PriorityHandle, group: group::Consumer, ) -> Result<(), Error> { - let mut serve = Box::new(GroupServe::new(self, sequence, frame_start, priority, group)); + let mut serve = Box::new(GroupServe::new( + self, + sequence, + frame_start, + Rank::Queue(priority), + group, + )); kio::wait(move |waiter| serve.poll_serve(waiter)).await } } @@ -3329,6 +3342,10 @@ impl TrackRun { order: upd.order, }; } + // One move for the whole subscription; its streams keep their order within it. + if let Some(group) = &mut self.ctx.group { + group.set_priority(upd.priority.into()); + } // Feed the full update into the model subscriber so the producer's // aggregate reflects it (and a relay re-forwards it upstream). // Read first: `update` replaces these preferences. @@ -3610,23 +3627,35 @@ impl TrackRun { // Use the latest steer for new groups so SUBSCRIBE_UPDATE applies to them too. let steer = self.ctx.steer_current(); - // The subscribe id scopes the group tie-break: one queue serves every - // subscription on the session, and only groups of the same one may be - // ranked against each other by sequence, in that subscription's order. - let handle = self - .ctx - .priority - .insert(Priority::new(steer.priority, self.ctx.id, sequence).with_order(steer.order)); + let rank = match &self.ctx.group { + Some(group) => Rank::Group(group.clone()), + // The subscribe id scopes the group tie-break: one queue serves every + // subscription on the session, and only groups of the same one may be + // ranked against each other by sequence, in that subscription's order. + None => Rank::Queue( + self.ctx + .priority + .insert(Priority::new(steer.priority, self.ctx.id, sequence).with_order(steer.order)), + ), + }; self.children - .push(GroupServe::new(self.ctx.clone(), sequence, frame_start, handle, group)); + .push(GroupServe::new(self.ctx.clone(), sequence, frame_start, rank, group)); } } +/// Where a group stream is ranked. +enum Rank { + /// In the subscription's send group, by its sequence in the subscription's order. + Group(G), + /// In the session-wide queue, on a transport without send groups. + Queue(PriorityHandle), +} + /// Serves one group on its own unidirectional stream: the header, then every /// frame, applying queue and SUBSCRIBE_UPDATE priority changes as they land. struct GroupServe { ctx: Subscription, - priority: PriorityHandle, + rank: Rank>, group: group::Consumer, sequence: u64, frame_start: u64, @@ -3667,13 +3696,13 @@ impl GroupServe { ctx: Subscription, sequence: u64, frame_start: u64, - priority: PriorityHandle, + rank: Rank>, group: group::Consumer, ) -> Self { ctx.opens.pending.fetch_add(1, Ordering::Relaxed); Self { ctx, - priority, + rank, group, sequence, frame_start, @@ -3702,8 +3731,10 @@ impl GroupServe { // (the transport still schedules its retransmissions). The rank is re-read // as a send order when handled, since the two conventions are inverted. if let GroupState::Serve { writer, .. } | GroupState::Closed { writer } = &mut self.state { - while let Poll::Ready(rank) = self.priority.poll_next(waiter) { - writer.set_priority(PriorityHandle::send_order_of(rank)); + if let Rank::Queue(handle) = &mut self.rank { + while let Poll::Ready(rank) = handle.poll_next(waiter) { + writer.set_priority(PriorityHandle::send_order_of(rank).into()); + } } let seen = self.ctx.steer_seen; // A dropped producer just disables this arm, like the queue arm above. @@ -3715,8 +3746,14 @@ impl GroupServe { } }) { self.ctx.steer_seen = value; - let rank = self.priority.steer(value.priority, value.order); - writer.set_priority(PriorityHandle::send_order_of(rank)); + // The run loop moves the send group itself; a stream only follows the order. + let order = match &mut self.rank { + Rank::Group(_) => crate::priority::in_group(self.sequence, self.frame_start, value.order), + Rank::Queue(handle) => { + PriorityHandle::send_order_of(handle.steer(value.priority, value.order)).into() + } + }; + writer.set_priority(order); } } @@ -3738,7 +3775,14 @@ impl GroupServe { }; self.settle_open(true); let mut writer = Writer::new(stream, self.ctx.version); - writer.set_priority(self.priority.send_order()); + match &mut self.rank { + Rank::Group(group) => { + writer.set_group(group); + let order = self.ctx.steer_seen.order; + writer.set_priority(crate::priority::in_group(self.sequence, self.frame_start, order)); + } + Rank::Queue(handle) => writer.set_priority(handle.send_order().into()), + } let msg = lite::Group { subscribe: self.ctx.id, @@ -4070,6 +4114,7 @@ mod serve_group_test { id: 0, track_name: "test".into(), priority: PriorityQueue::default(), + group: None, steer: steer.consume(), steer_seen: Steer::default(), version: Version::Lite06, @@ -4114,6 +4159,7 @@ mod serve_group_test { id: 0, track_name: "test".into(), priority: PriorityQueue::default(), + group: None, steer: steer.consume(), steer_seen: Steer::default(), version: Version::Lite06, @@ -4162,6 +4208,7 @@ mod serve_group_test { id: 0, track_name: "test".into(), priority: PriorityQueue::default(), + group: None, steer: steer.consume(), steer_seen: Steer::default(), version: Version::Lite06, @@ -4228,6 +4275,7 @@ mod serve_group_test { id: 0, track_name: "test".into(), priority: PriorityQueue::default(), + group: None, steer: steer.consume(), steer_seen: Steer::default(), version: Version::Lite06, @@ -4303,6 +4351,7 @@ mod serve_group_test { id: 0, track_name: "test".into(), priority: PriorityQueue::default(), + group: None, steer: steer.consume(), steer_seen: Steer::default(), version: Version::Lite06, @@ -4350,6 +4399,7 @@ mod serve_group_test { id: 0, track_name: "test".into(), priority: PriorityQueue::default(), + group: None, steer: steer.consume(), steer_seen: Steer::default(), version: Version::Lite06, @@ -4396,6 +4446,7 @@ mod serve_group_test { id: 0, track_name: "test".into(), priority: PriorityQueue::default(), + group: None, steer: steer.consume(), steer_seen: Steer::default(), version: Version::Lite06, @@ -4454,6 +4505,7 @@ mod serve_group_test { id: 0, track_name: "test".into(), priority: PriorityQueue::default(), + group: None, steer: steer.consume(), steer_seen: Steer::default(), version, @@ -4478,6 +4530,58 @@ mod serve_group_test { group.finish().unwrap(); } + /// On a transport with send groups, a subscription opens one at its priority and its + /// group streams join it, ranked newest first by sequence alone: the session-wide queue + /// takes no part. A SUBSCRIBE_UPDATE moves the group once, and an order flip re-stamps + /// the open streams. + #[moq_net_sim::test] + async fn grouped_streams_rank_within_their_subscription() { + let mut track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + // Old groups stay worth sending, so both are in flight at once. + let subscriber = track.subscribe(track::Subscription::default().with_max_delay(Duration::from_secs(3600))); + let mut session = SinkSession::new(Log::default()).with_groups().with_unacked_fin(); + let log = session.log.clone(); + let (mut run, mut writer, _) = lite_run(Version::Lite06, session.clone(), subscriber); + run.ctx.group = crate::transport::poll::Session::send_group(&mut session, 3); + + write_group(&mut track, 0, 0); + write_group(&mut track, 1, 1); + { + let mut poll = std::pin::pin!(kio::wait(|waiter| run.poll(&mut writer, waiter))); + assert!(futures::poll!(poll.as_mut()).is_pending()); + } + assert_eq!(log.groups(), [(0, 3)]); + assert_eq!( + log.joins(), + [(0, 0), (1, 0)], + "each stream joins the subscription's group" + ); + assert_eq!(log.orders(), [(0, 1), (1, 3)], "newer groups first"); + assert!(log.priorities().is_empty(), "the session queue ranks no stream"); + + run.update(lite::SubscribeUpdate { + order: Order::Ascending, + priority: 7, + max_delay: Duration::ZERO, + start_group: None, + end_group: None, + start_frame: 0, + end_frame: None, + }); + { + let mut poll = std::pin::pin!(kio::wait(|waiter| run.poll(&mut writer, waiter))); + assert!(futures::poll!(poll.as_mut()).is_pending()); + } + assert_eq!(log.groups(), [(0, 3), (0, 7)], "one move for the subscription"); + let mut flipped = log.orders()[2..].to_vec(); + flipped.sort(); + assert_eq!(flipped, [(0, 1), (1, -1)], "older groups first"); + } + /// A group with more frames ready than one poll's budget drains over several polls, /// so a sibling (the transport's driver, say) runs in between. #[test] @@ -4870,6 +4974,7 @@ mod serve_group_test { id: 0, track_name: "test".into(), priority: PriorityQueue::default(), + group: None, steer: steer.consume(), steer_seen: Steer::default(), version: Version::Lite07, diff --git a/rs/moq-net/src/lite/test_transport.rs b/rs/moq-net/src/lite/test_transport.rs index f47528eec5d..5c32cef64ca 100644 --- a/rs/moq-net/src/lite/test_transport.rs +++ b/rs/moq-net/src/lite/test_transport.rs @@ -50,6 +50,11 @@ pub struct Log { closes: Arc>>, bi_opens: Arc, priorities: Arc>>, + /// Hands each send group its id. + group_ids: Arc, + groups: Arc>>, + joins: Arc>>, + orders: Arc>>, } /// One thing a send stream did, as recorded in [`Log::trail`]. @@ -94,6 +99,23 @@ impl Log { self.priorities.lock().unwrap().clone() } + /// Every send group's priority, as (group, priority) in call order: its opening, then + /// each move. + pub fn groups(&self) -> Vec<(usize, i64)> { + self.groups.lock().unwrap().clone() + } + + /// Every send group a stream joined, as (stream, group) in call order. + pub fn joins(&self) -> Vec<(usize, usize)> { + self.joins.lock().unwrap().clone() + } + + /// Every priority set on a stream in a send group, as (stream, priority) in call + /// order. [`Self::priorities`] keeps those of the streams outside every group. + pub fn orders(&self) -> Vec<(usize, i64)> { + self.orders.lock().unwrap().clone() + } + /// The session-level closes, as (code, reason). A list rather than a last-value so /// a second close racing the first is visible. pub fn closes(&self) -> Vec<(u32, String)> { @@ -123,6 +145,29 @@ pub struct SinkSend { finished: bool, /// A finished stream still parks, like a peer that has not acknowledged the FIN. unacked_fin: bool, + /// The send group the stream joined. + group: Option, +} + +/// A send group of a [`SinkSession::with_groups`], recording its moves in the [`Log`]. +#[derive(Clone)] +pub struct SinkGroup { + id: usize, + log: Log, +} + +impl SinkGroup { + fn open(log: Log, priority: i64) -> Self { + let id = log.group_ids.fetch_add(1, Ordering::Relaxed); + log.groups.lock().unwrap().push((id, priority)); + Self { id, log } + } +} + +impl poll::SendGroup for SinkGroup { + fn set_priority(&mut self, priority: i64) { + self.log.groups.lock().unwrap().push((self.id, priority)); + } } impl SinkSend { @@ -135,6 +180,7 @@ impl SinkSend { park: kio::Park::default(), finished: false, unacked_fin: false, + group: None, } } @@ -148,12 +194,14 @@ impl SinkSend { park: kio::Park::default(), finished: false, unacked_fin: false, + group: None, } } } impl poll::SendStream for SinkSend { type Error = SinkError; + type Group = SinkGroup; fn poll_write(&mut self, cx: &mut Context<'_>, buf: &[u8]) -> Poll> { if self.prefix == 0 @@ -179,11 +227,20 @@ impl poll::SendStream for SinkSend { Poll::Ready(Ok(buf.len())) } - fn set_priority(&mut self, order: i32) { - let order = u8::try_from(order).expect("moq-net sends u8 send orders"); + fn set_priority(&mut self, order: i64) { + if self.group.is_some() { + self.log.orders.lock().unwrap().push((self.id, order)); + return; + } + let order = u8::try_from(order).expect("moq-net sends u8 send orders outside a group"); self.log.priorities.lock().unwrap().push(order); } + fn set_group(&mut self, group: &Self::Group) { + self.group = Some(group.id); + self.log.joins.lock().unwrap().push((self.id, group.id)); + } + fn finish(&mut self) -> Result<(), Self::Error> { self.finished = true; self.log.record(self.id, Sent::Finish); @@ -351,6 +408,10 @@ impl poll::Session for DeadStreamSession { Poll::Ready(Ok(SinkSend::new(self.log.clone()))) } + fn send_group(&mut self, _: i64) -> Option { + None + } + fn poll_send_datagram(&mut self, _cx: &mut Context<'_>, _payload: &[u8]) -> Poll> { Poll::Ready(Ok(())) } @@ -401,6 +462,8 @@ pub struct SinkSession { uni_open_park: kio::Park, /// Set by [`Self::with_unacked_fin`]. unacked_fin: bool, + /// Set by [`Self::with_groups`]. + groups: bool, /// The ALPN to report, for a test that needs a specific negotiated version rather /// than the SETUP-negotiated fallback an absent one selects. protocol: Option<&'static str>, @@ -425,6 +488,12 @@ impl SinkSession { self } + /// Open send groups, as native QUIC does, recording them in the log. + pub fn with_groups(mut self) -> Self { + self.groups = true; + self + } + /// Report these connection statistics, as a real transport would. pub fn with_stats(self, stats: SinkStats) -> Self { self.set_stats(stats); @@ -539,6 +608,10 @@ impl poll::Session for SinkSession { Poll::Ready(Ok(send)) } + fn send_group(&mut self, priority: i64) -> Option { + self.groups.then(|| SinkGroup::open(self.log.clone(), priority)) + } + fn poll_send_datagram(&mut self, _cx: &mut Context<'_>, _payload: &[u8]) -> Poll> { Poll::Ready(Ok(())) } @@ -860,6 +933,10 @@ impl poll::Session for ScriptedSession { Poll::Ready(Ok(SinkSend::new(self.log.clone()))) } + fn send_group(&mut self, _: i64) -> Option { + None + } + fn poll_send_datagram(&mut self, _cx: &mut Context<'_>, payload: &[u8]) -> Poll> { self.log .datagrams diff --git a/rs/moq-net/src/priority.rs b/rs/moq-net/src/priority.rs index 7c8707dad6e..be22c330677 100644 --- a/rs/moq-net/src/priority.rs +++ b/rs/moq-net/src/priority.rs @@ -99,6 +99,22 @@ impl PartialOrd for Priority { } } +/// A stream's send order within its subscription's send group, where higher goes first. +/// +/// The [`Priority`] ranking minus the parts the group itself carries: the group sequence in +/// the subscription's `order`, then a group's first stream ahead of a later `start` in the +/// same group. The low bit only splits a stream from the start of its group (`start == 0`) +/// from the rest, which is the joining prefix beside the live stream; any other two streams +/// of one group share. Sequences are wire varints below 2^62, so doubling one fits. +pub fn in_group(group: u64, start: u64, order: Order) -> i64 { + let group = i64::try_from(group).unwrap_or(i64::MAX).saturating_mul(2); + let group = match order { + Order::Descending => group, + Order::Ascending => -group, + }; + group.saturating_add(i64::from(start == 0)) +} + #[derive(Debug, Clone)] struct PriorityItem { id: usize, diff --git a/rs/moq-net/src/server.rs b/rs/moq-net/src/server.rs index 371e347a161..4e3b2cdbfbf 100644 --- a/rs/moq-net/src/server.rs +++ b/rs/moq-net/src/server.rs @@ -1065,6 +1065,10 @@ mod tests { ) -> std::task::Poll> { std::task::Poll::Pending } + fn send_group(&mut self, _: i64) -> Option { + None + } + fn poll_send_datagram( &mut self, _cx: &mut std::task::Context<'_>, @@ -1102,6 +1106,7 @@ mod tests { struct FakeSend; impl crate::transport::poll::SendStream for FakeSend { type Error = FakeError; + type Group = crate::transport::poll::Ungrouped; fn poll_write( &mut self, _cx: &mut std::task::Context<'_>, @@ -1109,7 +1114,11 @@ mod tests { ) -> std::task::Poll> { std::task::Poll::Ready(Ok(buf.len())) } - fn set_priority(&mut self, _order: i32) {} + fn set_priority(&mut self, _order: i64) {} + fn set_group(&mut self, group: &Self::Group) { + match *group {} + } + fn finish(&mut self) -> Result<(), Self::Error> { Ok(()) } diff --git a/rs/moq-net/src/transport.rs b/rs/moq-net/src/transport.rs index 409ed432e7b..cdd084eee17 100644 --- a/rs/moq-net/src/transport.rs +++ b/rs/moq-net/src/transport.rs @@ -154,6 +154,16 @@ pub mod poll { /// concurrent streams. fn poll_open_bi(&mut self, cx: &mut Context<'_>) -> Poll, Self::Error>>; + /// Open a send group at `priority`, or `None` when the transport orders streams by + /// their own priority alone. + /// + /// Streams that [join](SendStream::set_group) a group share bytes with the session's + /// other groups: a higher priority group preempts every lower one, and groups of equal + /// priority get an equal share of bytes however many streams each has queued. Streams + /// outside every group go before all of them. The group is released once its last + /// handle is dropped and its streams finish. + fn send_group(&mut self, priority: i64) -> Option<::Group>; + /// Poll to send a datagram over the network. /// /// Returns [`Poll::Pending`] while the transport has no room for it, so a caller @@ -295,20 +305,55 @@ pub mod poll { /// boxed drivers (`Send` boxes on native): what the moq-transport path /// requires until it too becomes named machines. Implemented automatically. pub trait Boxable: - Session + MaybeSend + MaybeSync + Session< + SendStream: MaybeSend + SendStream, + RecvStream: MaybeSend, + Error: MaybeSend, + > + MaybeSend + + MaybeSync { } impl Boxable for S where - S: Session + MaybeSend + MaybeSync + S: Session< + SendStream: MaybeSend + SendStream, + RecvStream: MaybeSend, + Error: MaybeSend, + > + MaybeSend + + MaybeSync { } + /// The send group of a session's streams. + #[cfg(feature = "async")] + pub(crate) type GroupOf = <::SendStream as SendStream>::Group; + + /// A transport's send group, from [`Session::send_group`]. + /// + /// Clones share the group. + pub trait SendGroup: Clone + 'static { + /// Change the group's priority: higher preempts lower. + fn set_priority(&mut self, priority: i64); + } + + /// The send group of a transport that has none, which cannot be constructed. + #[derive(Clone, Debug)] + pub enum Ungrouped {} + + impl SendGroup for Ungrouped { + fn set_priority(&mut self, _: i64) { + match *self {} + } + } + /// An outgoing stream with partial writes, priorities, and reset codes. pub trait SendStream: 'static { /// The error type for every operation on this stream. type Error: Error; + /// The transport's send group, [`Ungrouped`] for a transport without them. + type Group: SendGroup; + /// Poll to write some of the buffer to the stream, returning how many bytes were /// written. See [`poll_write_buf`](Self::poll_write_buf) for the partial-write /// contract, which this shares. @@ -338,17 +383,18 @@ pub mod poll { Poll::Ready(Ok(size)) } - /// Set the stream's priority. + /// Set the priority within the stream's send group, or session-wide when ungrouped. /// /// Streams with higher values will be sent first, but are not guaranteed to /// arrive first. This matches the W3C WebTransport `sendOrder` convention (and /// quinn's scheduler). /// - /// The full `i32` range is available so callers can bit-pack a composite ordering - /// (for example a track priority in the high bits and a sequence number in the low - /// bits) into a single value. Backends that cannot express that many distinct - /// levels approximate it, so treat the ordering as best-effort. - fn set_priority(&mut self, order: i32); + /// Backends with a narrower priority saturate or approximate it, so treat the + /// ordering as best-effort. + fn set_priority(&mut self, priority: i64); + + /// Move the stream into `group`, which belongs to this stream's session. + fn set_group(&mut self, group: &Self::Group); /// Mark the stream as finished, erroring on any future writes. /// diff --git a/rs/moq-net/src/transport/memory.rs b/rs/moq-net/src/transport/memory.rs index 1bccc9948bb..c9229b31d8e 100644 --- a/rs/moq-net/src/transport/memory.rs +++ b/rs/moq-net/src/transport/memory.rs @@ -294,6 +294,10 @@ impl poll::Session for Session { Poll::Ready(opened.map(|()| (send, recv))) } + fn send_group(&mut self, _: i64) -> Option { + None + } + fn poll_send_datagram(&mut self, _cx: &mut Context<'_>, payload: &[u8]) -> Poll> { if let Some(err) = closed(&self.conn().close) { return Poll::Ready(Err(err)); @@ -390,6 +394,7 @@ pub struct SendStream { impl poll::SendStream for SendStream { type Error = Error; + type Group = crate::transport::poll::Ungrouped; fn poll_write(&mut self, cx: &mut Context<'_>, buf: &[u8]) -> Poll> { let mut buf = buf; @@ -430,7 +435,11 @@ impl poll::SendStream for SendStream { } /// Ignored: with no link to share, there is nothing to schedule. - fn set_priority(&mut self, _order: i32) {} + fn set_priority(&mut self, _order: i64) {} + + fn set_group(&mut self, group: &Self::Group) { + match *group {} + } fn finish(&mut self) -> Result<(), Error> { if self.done { diff --git a/rs/moq-net/tests/integration/support/mock.rs b/rs/moq-net/tests/integration/support/mock.rs index f9889c45227..d98423ff315 100644 --- a/rs/moq-net/tests/integration/support/mock.rs +++ b/rs/moq-net/tests/integration/support/mock.rs @@ -14,7 +14,7 @@ use std::{ collections::VecDeque, sync::{ Arc, Mutex, - atomic::{AtomicBool, AtomicI32, AtomicUsize, Ordering}, + atomic::{AtomicBool, AtomicI64, AtomicUsize, Ordering}, }, task::{Context, Poll, ready}, time::Duration, @@ -36,7 +36,7 @@ struct Line { /// When the reset reaches the reader. reset: Mutex>, /// The send order last set, for [`MockSession::release_unis_by_priority`]. - priority: AtomicI32, + priority: AtomicI64, /// Opened while its opener held streams back. held: bool, /// Once released, delivers what follows the first write; see [`MockSession::split_unis`]. @@ -212,6 +212,7 @@ impl MockSendStream { impl poll::SendStream for MockSendStream { type Error = memory::Error; + type Group = poll::Ungrouped; fn poll_write(&mut self, cx: &mut Context<'_>, buf: &[u8]) -> Poll> { let written = self.inner.poll_write(cx, buf); @@ -224,11 +225,15 @@ impl poll::SendStream for MockSendStream { self.wrote(first, written) } - fn set_priority(&mut self, order: i32) { + fn set_priority(&mut self, order: i64) { self.line.priority.store(order, Ordering::Relaxed); self.inner.set_priority(order); } + fn set_group(&mut self, group: &Self::Group) { + match *group {} + } + fn finish(&mut self) -> Result<(), Self::Error> { if self .withhold_fin @@ -591,6 +596,10 @@ impl poll::Session for MockSession { Poll::Ready(Ok(send)) } + fn send_group(&mut self, _: i64) -> Option { + None + } + fn poll_send_datagram(&mut self, cx: &mut Context<'_>, payload: &[u8]) -> Poll> { if self.side.lossy.load(Ordering::Relaxed) { return Poll::Ready(Ok(())); diff --git a/rs/moq-quic/README.md b/rs/moq-quic/README.md index 703c5433541..408181655e2 100644 --- a/rs/moq-quic/README.md +++ b/rs/moq-quic/README.md @@ -66,6 +66,7 @@ Changes on top of the upstream commit, besides the renames: - `TransportConfig::handshake_idle_timeout` (default 10s, as msquic's `HandshakeIdleTimeoutMs`) governs the idle timer until the handshake completes, and the negotiated idle timeout takes over once established. A 2s idle timeout otherwise expires 3x the initial PTO in, just before the second Initial probe, leaving a client two flights. A server also drops an Initial left waiting for `accept` past it, and a deadline too far out to represent never fires instead of panicking. Also [moq-dev/noq#34](https://github.com/moq-dev/noq/pull/34), whose third change (the handshake idle timeout picks noq's idle-based PTO interval cap) has nothing to apply to here, since quinn's PTO backoff has no such cap. quinn's `handshake_timeout` test in its async layer expects the handshake to end at the idle timeout, so the driver's copy sets both timeouts. - **Instants on [moq-clock](../moq-clock)**: the crate's `Instant` is `moq_clock::Instant`, so one clock steps every timer, tokio's paused clock and the simulator included. qlog stamps events relative to `QlogConfig::start_time` on that clock rather than the streamer's own `std::time::Instant`. +- **Send groups** (`streams/schedule.rs`, `Connection::send_groups`): streams outside every group go first, in quinn's order; groups in one scheduling domain are served strictly by priority, with equal priorities sharing bytes by deficit round robin; domains share bytes the same way. A stream's priority is an `i64` ordering it only within its group. This replaces quinn's single pending-stream heap, so a cherry-pick touching `PendingStreamsQueue` has to be redone against the schedule. moq-tokio's `quic::driver` and moq-uring's driver wrap a group in a shared `SendGroup` handle that closes on drop. - **`RESET_STREAM_AT`** ([draft-ietf-quic-reliable-stream-reset](https://datatracker.ietf.org/doc/draft-ietf-quic-reliable-stream-reset/)): the `reset_stream_at` transport parameter is always advertised, `Checkpoint::reset` (from `SendStream::checkpoint`) keeps the data written before the checkpoint reliable, holding a plain `RESET_STREAM` until that data is acknowledged when the peer lacks the extension, and a received reset surfaces only once the reliable prefix has been read. Stopping a stream after it was reset no longer credits its unread data twice. - **Draining endpoint event**: `EndpointEvent::is_draining` fires once per connection when it stops being active, before the drained event, so the driver's `Endpoint::wait_all_draining` can end a shutdown without sitting out every closing period. diff --git a/rs/moq-quic/src/connection/mod.rs b/rs/moq-quic/src/connection/mod.rs index 3425f00ba2b..a4ede241751 100644 --- a/rs/moq-quic/src/connection/mod.rs +++ b/rs/moq-quic/src/connection/mod.rs @@ -100,7 +100,8 @@ pub use streams::StreamsState; use streams::StreamsState; pub use streams::{ Checkpoint, Chunks, ClosedStream, Credit, FinishError, ReadError, ReadableError, RecvStream, - SendStream, ShouldTransmit, StreamEvent, Streams, WriteError, Written, + SendDomain, SendGroup, SendGroups, SendStream, ShouldTransmit, StreamEvent, Streams, + WriteError, Written, }; mod timer; @@ -448,6 +449,14 @@ impl Connection { } } + /// Provide control over the send groups that schedule outgoing streams + #[must_use] + pub fn send_groups(&mut self) -> SendGroups<'_> { + SendGroups { + state: &mut self.streams, + } + } + /// Provide control over streams #[must_use] pub fn recv_stream(&mut self, id: StreamId) -> RecvStream<'_> { diff --git a/rs/moq-quic/src/connection/streams/mod.rs b/rs/moq-quic/src/connection/streams/mod.rs index 5148338b7d5..8f862ba66f0 100644 --- a/rs/moq-quic/src/connection/streams/mod.rs +++ b/rs/moq-quic/src/connection/streams/mod.rs @@ -1,7 +1,4 @@ -use std::{ - collections::{BinaryHeap, hash_map}, - io, -}; +use std::{collections::hash_map, io}; use bytes::Bytes; use thiserror::Error; @@ -16,7 +13,12 @@ use crate::{ mod recv; use recv::Recv; + +#[cfg(test)] +mod saturated; +mod schedule; pub use recv::{Chunks, Credit, ReadError, ReadableError}; +use schedule::Schedule; mod send; pub(crate) use send::{ByteSlice, BytesArray}; @@ -286,7 +288,9 @@ impl<'a> SendStream<'a> { self.state.buffered_data += written.bytes as u64; trace!(stream = %self.id, "wrote {} bytes", written.bytes); if !was_pending { - self.state.pending.push_pending(self.id, stream.priority); + self.state + .pending + .push(self.id, &mut stream.slot, stream.priority); } Ok(written) } @@ -317,7 +321,9 @@ impl<'a> SendStream<'a> { let was_pending = stream.is_pending(); stream.finish()?; if !was_pending { - self.state.pending.push_pending(self.id, stream.priority); + self.state + .pending + .push(self.id, &mut stream.slot, stream.priority); } Ok(()) @@ -378,6 +384,10 @@ impl<'a> SendStream<'a> { let buffered = stream.pending.buffered(); let send = stream.reset(error_code, reliable_size, at); self.state.buffered_data -= buffered - stream.pending.buffered(); + // Only data below the Reliable Size is left to send + if !stream.is_pending() { + self.state.pending.remove(&mut stream.slot); + } if let Some(reliable_size) = send { self.pending.reset_stream.push((self.id, reliable_size)); } @@ -386,11 +396,15 @@ impl<'a> SendStream<'a> { Ok(()) } - /// Set the priority of a stream + /// Set the priority within the stream's send group, or session-wide when ungrouped + /// + /// Higher values go first; equal values share bytes round robin. A queued stream moves to + /// its new priority the next time it is requeued, which is after every frame when + /// [`TransportConfig::send_fairness`](crate::TransportConfig::send_fairness) is on. /// /// # Panics /// - when applied to a receive stream - pub fn set_priority(&mut self, priority: i32) -> Result<(), ClosedStream> { + pub fn set_priority(&mut self, priority: i64) -> Result<(), ClosedStream> { let max_send_data = self.state.max_send_data(self.id); let stream = self .state @@ -407,7 +421,7 @@ impl<'a> SendStream<'a> { /// /// # Panics /// - when applied to a receive stream - pub fn priority(&self) -> Result { + pub fn priority(&self) -> Result { let stream = self .state .send @@ -416,6 +430,29 @@ impl<'a> SendStream<'a> { Ok(stream.as_ref().map(|s| s.priority).unwrap_or_default()) } + + /// Move the stream into a send group, or with `None` back to the ungrouped streams + /// + /// Ungrouped streams are sent before any send group. A queued stream keeps its place. + /// + /// # Panics + /// - when applied to a receive stream + /// - when `group` was opened on another connection + pub fn set_group(&mut self, group: Option<&SendGroup>) -> Result<(), ClosedStream> { + let max_send_data = self.state.max_send_data(self.id); + let stream = self + .state + .send + .get_mut(&self.id) + .ok_or(ClosedStream { _private: () })? + .get_or_insert_with(|| Send::new(max_send_data)); + + let group = group.map(|g| g.index(self.state.tag)); + self.state + .pending + .join(self.id, &mut stream.slot, group, stream.priority); + Ok(()) + } } /// The data written to a send stream up to some point, which a reset can still deliver @@ -463,91 +500,108 @@ impl Checkpoint { } } -/// A queue of streams with pending outgoing data, sorted by priority -struct PendingStreamsQueue { - streams: BinaryHeap, - /// The next stream to write out. This is `Some` when `TransportConfig::send_fairness(false)` and writing a stream is - /// interrupted while the stream still has some pending data. See `reinsert_pending()`. - next: Option, - /// A monotonically decreasing counter, used to implement round-robin scheduling for streams of the same priority. - /// Underflowing is not a practical concern, as it is initialized to u64::MAX and only decremented by 1 in `push_pending` - recency: u64, +/// Scheduling of outgoing streams between send groups +/// +/// A send group shares bytes with the other groups of its domain: a higher priority group +/// preempts every lower one, and groups of equal priority get an equal share of bytes no +/// matter how many streams each has queued. Domains share bytes equally whatever their +/// groups' priorities. Streams outside any group go before all of it. +pub struct SendGroups<'a> { + pub(super) state: &'a mut StreamsState, } -impl PendingStreamsQueue { - fn new() -> Self { - Self { - streams: BinaryHeap::new(), - next: None, - recency: u64::MAX, +impl SendGroups<'_> { + /// Open a send group at `priority` in `domain`, or in the default domain + /// + /// # Panics + /// - when `domain` was opened on another connection + pub fn open(&mut self, priority: i64, domain: Option<&SendDomain>) -> SendGroup { + let tag = self.state.tag; + let domain = domain.map(|d| d.index(tag)); + SendGroup { + index: self.state.pending.open_group(priority, domain), + tag, } } - /// Reinsert a stream that was pending and still contains unsent data. - fn reinsert_pending(&mut self, id: StreamId, priority: i32) { - assert!(self.next.is_none()); + /// Release a send group; its streams keep their place until they finish + /// + /// # Panics + /// - when `group` was opened on another connection + pub fn close(&mut self, group: SendGroup) { + let index = group.index(self.state.tag); + self.state.pending.close_group(index); + } - self.next = Some(PendingStream { - priority, - recency: self.recency, // the value here doesn't really matter - id, - }); - } - - /// Push a pending stream ID with the given priority, queued after any already-queued streams for the priority - fn push_pending(&mut self, id: StreamId, priority: i32) { - // Note that in the case where fairness is disabled, if we have a reinserted stream we don't - // bump it even if priority > next.priority. In order to minimize fragmentation we - // always try to complete a stream once part of it has been written. - - // As the recency counter is monotonically decreasing, we know that using its value to sort this stream will queue it - // after all other queued streams of the same priority. - // This is enough to implement round-robin scheduling for streams that are still pending even after being handled, - // as in that case they are removed from the `BinaryHeap`, handled, and then immediately reinserted. - self.recency -= 1; - self.streams.push(PendingStream { - priority, - recency: self.recency, - id, - }); + /// Change a group's priority, moving any data it has queued along with it + /// + /// # Panics + /// - when `group` was opened on another connection + pub fn set_priority(&mut self, group: &SendGroup, priority: i64) { + let index = group.index(self.state.tag); + self.state.pending.set_group_priority(index, priority); } - fn pop(&mut self) -> Option { - self.next.take().or_else(|| self.streams.pop()) + /// The group's priority + /// + /// # Panics + /// - when `group` was opened on another connection + pub fn priority(&self, group: &SendGroup) -> i64 { + self.state + .pending + .group_priority(group.index(self.state.tag)) } - fn clear(&mut self) { - self.next = None; - self.streams.clear(); + /// Open a domain, which gets an equal share of bytes with every other domain + pub fn open_domain(&mut self) -> SendDomain { + SendDomain { + index: self.state.pending.open_domain(), + tag: self.state.tag, + } } - fn iter(&self) -> impl Iterator { - self.next.iter().chain(self.streams.iter()) + /// Release a domain; its groups keep their share until they are released + /// + /// # Panics + /// - when `domain` was opened on another connection + pub fn close_domain(&mut self, domain: SendDomain) { + let index = domain.index(self.state.tag); + self.state.pending.close_domain(index); } +} - #[cfg(test)] - fn len(&self) -> usize { - self.streams.len() + self.next.is_some() as usize +/// A send group, from [`SendGroups::open`] +/// +/// Release it with [`SendGroups::close`] on the connection that opened it. +#[derive(Debug, PartialEq, Eq)] +#[must_use = "a send group lives until it is closed"] +pub struct SendGroup { + index: usize, + tag: ConnectionTag, +} + +impl SendGroup { + fn index(&self, tag: ConnectionTag) -> usize { + assert_eq!(self.tag, tag, "send group used on another connection"); + self.index } } -/// The [`StreamId`] of a stream with pending data queued, ordered by its priority and recency -#[derive(Clone, PartialEq, Eq, PartialOrd, Ord)] -struct PendingStream { - /// The priority of the stream - // Note that this field should be kept above the `recency` field, in order for the `Ord` derive to be correct - // (See https://doc.rust-lang.org/stable/std/cmp/trait.Ord.html#derivable) - priority: i32, - /// A tie-breaker for streams of the same priority, used to improve fairness by implementing round-robin scheduling: - /// Larger values are prioritized, so it is initialised to `u64::MAX`, and when a stream writes data, we know - /// that it currently has the highest recency value, so it is deprioritized by setting its recency to 1 less than the - /// previous lowest recency value, such that all other streams of this priority will get processed once before we get back - /// round to this one - recency: u64, - /// The ID of the stream - // The way this type is used ensures that every instance has a unique `recency` value, so this field should be kept below - // the `priority` and `recency` fields, so that it does not interfere with the behaviour of the `Ord` derive - id: StreamId, +/// A domain of send groups, from [`SendGroups::open_domain`] +/// +/// Release it with [`SendGroups::close_domain`] on the connection that opened it. +#[derive(Debug, PartialEq, Eq)] +#[must_use = "a domain lives until it is closed"] +pub struct SendDomain { + index: usize, + tag: ConnectionTag, +} + +impl SendDomain { + fn index(&self, tag: ConnectionTag) -> usize { + assert_eq!(self.tag, tag, "send domain used on another connection"); + self.index + } } /// Application events about streams diff --git a/rs/moq-quic/src/connection/streams/saturated.rs b/rs/moq-quic/src/connection/streams/saturated.rs new file mode 100644 index 00000000000..fb561e5fb6e --- /dev/null +++ b/rs/moq-quic/src/connection/streams/saturated.rs @@ -0,0 +1,495 @@ +//! Scheduling proofs through the real send path: streams written into send groups and +//! drained into packets by `write_stream_frames`, with loss and resets applied the way +//! the connection applies them. + +use std::collections::HashMap; + +use super::state::StreamsState; +use super::{SendDomain, SendGroup, SendGroups, SendStream, Streams}; +use crate::{ + Dir, Side, StreamId, VarInt, + connection::{State as ConnState, spaces::Retransmits}, + frame::StreamMeta, + transport_parameters::TransportParameters, +}; + +const PACKET: usize = 1200; + +/// How far two equally entitled groups may drift apart: each round robin may +/// overdraw by a frame, at both the group and the domain tier. +const SLACK: u64 = 2 * PACKET as u64; + +/// A sender with more queued than the path takes, drained a packet at a time. +struct Sender { + state: StreamsState, + pending: Retransmits, + conn: ConnState, + /// Payload bytes each stream got out. + sent: HashMap, + /// Bytes each stream's frames took on the wire, headers included, which is what + /// the round robins share. + wire: HashMap, +} + +impl Sender { + fn new() -> Self { + Self::limited(VarInt::MAX) + } + + /// With each stream's flow control window at `window` bytes. + fn limited(window: VarInt) -> Self { + let mut state = StreamsState::new( + Side::Server, + 0u32.into(), + 0u32.into(), + u64::MAX, + VarInt::MAX, + VarInt::MAX, + ); + state.set_params(&TransportParameters { + initial_max_streams_uni: VarInt::MAX, + initial_max_data: VarInt::MAX, + initial_max_stream_data_uni: window, + ..TransportParameters::default() + }); + Self { + state, + pending: Retransmits::default(), + conn: ConnState::Established, + sent: HashMap::new(), + wire: HashMap::new(), + } + } + + fn groups(&mut self) -> SendGroups<'_> { + SendGroups { + state: &mut self.state, + } + } + + fn group(&mut self, priority: i64) -> SendGroup { + self.groups().open(priority, None) + } + + fn send(&mut self, id: StreamId) -> SendStream<'_> { + SendStream { + id, + state: &mut self.state, + pending: &mut self.pending, + conn_state: &self.conn, + } + } + + /// Open a stream in `group` at `priority` and queue `bytes` on it. + fn open(&mut self, group: Option<&SendGroup>, priority: i64, bytes: usize) -> StreamId { + let id = Streams { + state: &mut self.state, + conn_state: &self.conn, + } + .open(Dir::Uni) + .unwrap(); + let mut stream = self.send(id); + stream.set_group(group).unwrap(); + stream.set_priority(priority).unwrap(); + self.write(id, bytes); + id + } + + fn write(&mut self, id: StreamId, bytes: usize) { + assert_eq!(self.send(id).write(&vec![0; bytes]), Ok(bytes)); + } + + /// Send one packet, returning its frames. + fn packet(&mut self) -> Vec { + let mut buf = Vec::with_capacity(PACKET); + let frames = self.state.write_stream_frames(&mut buf, PACKET, true); + + // Every frame carries its length but perhaps the last, which can run to the + // end of the packet instead. + let mut sizes: Vec<_> = frames + .iter() + .map(|frame| { + let mut header = Vec::new(); + frame.encode(true, &mut header); + (header.len() as u64, frame.offsets.end - frame.offsets.start) + }) + .collect(); + if sizes.iter().map(|(h, p)| h + p).sum::() != buf.len() as u64 { + let (header, payload) = sizes.last_mut().unwrap(); + *header -= VarInt::from_u64(*payload).unwrap().size() as u64; + } + assert_eq!( + sizes.iter().map(|(h, p)| h + p).sum::(), + buf.len() as u64 + ); + + for (frame, (header, payload)) in frames.iter().zip(sizes) { + *self.sent.entry(frame.id).or_default() += payload; + *self.wire.entry(frame.id).or_default() += header + payload; + } + frames.into_iter().collect() + } + + /// Send `packets` packets, returning the stream of every frame in order. + fn drain(&mut self, packets: usize) -> Vec { + (0..packets) + .flat_map(|_| self.packet()) + .map(|frame| frame.id) + .collect() + } + + fn bytes(&self, streams: &[StreamId]) -> u64 { + streams + .iter() + .map(|id| self.sent.get(id).copied().unwrap_or(0)) + .sum() + } + + fn wire(&self, streams: &[StreamId]) -> u64 { + streams + .iter() + .map(|id| self.wire.get(id).copied().unwrap_or(0)) + .sum() + } + + fn clear(&mut self) { + self.sent.clear(); + self.wire.clear(); + } +} + +fn assert_fair(a: u64, b: u64) { + assert!(a.abs_diff(b) <= SLACK, "unfair split: {a} vs {b}"); + assert!(a + b > 0, "nothing sent"); +} + +/// The issue's Alice and Bob: two subscriptions at one priority, an audio track of +/// many small groups and a video track with a deep backlog of large ones. Each gets +/// half the bytes, not half the streams. +#[test] +fn equal_groups_share_bytes_not_streams() { + let mut sender = Sender::new(); + let audio = sender.group(4); + let video = sender.group(4); + let audio: Vec<_> = (0..1_000) + .map(|seq| sender.open(Some(&audio), seq, 300)) + .collect(); + let video: Vec<_> = (0..4) + .map(|seq| sender.open(Some(&video), seq, 1_000_000)) + .collect(); + + sender.drain(400); + assert_fair(sender.wire(&audio), sender.wire(&video)); +} + +/// A group that keeps opening newer streams gains nothing over one draining a single +/// stream: new streams spend the group's credit rather than earning more. +#[test] +fn newer_streams_do_not_reset_a_groups_credit() { + let mut sender = Sender::new(); + let busy = sender.group(0); + let steady = sender.group(0); + let mut busy_streams = vec![sender.open(Some(&busy), 0, 1_000_000)]; + let steady = [sender.open(Some(&steady), 0, 1_000_000)]; + + for seq in 1..400 { + busy_streams.push(sender.open(Some(&busy), seq, 2_000)); + sender.packet(); + } + assert_fair(sender.wire(&busy_streams), sender.wire(&steady)); +} + +#[test] +fn higher_priority_preempts() { + let mut sender = Sender::new(); + let high = sender.group(1); + let low = sender.group(0); + let low = sender.open(Some(&low), 0, 50_000); + let high = sender.open(Some(&high), 0, 50_000); + + let order = sender.drain(100); + let first_low = order.iter().position(|&id| id == low).unwrap(); + assert!(order[..first_low].iter().all(|&id| id == high)); + assert!(order[first_low..].iter().all(|&id| id == low)); + assert_eq!(sender.bytes(&[high, low]), 100_000); +} + +#[test] +fn priority_updates_apply_to_queued_data() { + let mut sender = Sender::new(); + let high = sender.group(1); + let low = sender.group(0); + let low_stream = sender.open(Some(&low), 0, 50_000); + let high_stream = sender.open(Some(&high), 0, 50_000); + assert!(sender.drain(5).iter().all(|&id| id == high_stream)); + + sender.groups().set_priority(&low, 2); + assert_eq!(sender.groups().priority(&low), 2); + assert!(sender.drain(5).iter().all(|&id| id == low_stream)); + + sender.groups().set_priority(&low, 1); + sender.clear(); + sender.drain(20); + assert_fair(sender.wire(&[low_stream]), sender.wire(&[high_stream])); +} + +/// Each group sends its own newest stream first, whatever the other groups' numbering. +#[test] +fn each_group_sends_in_its_own_order() { + let mut sender = Sender::new(); + let descending = sender.group(0); + let ascending = sender.group(0); + let newest: Vec<_> = [3, 1, 4, 2] + .into_iter() + .map(|seq| (seq, sender.open(Some(&descending), seq, 3_000))) + .collect(); + // An oldest-first subscription negates its sequences. + let oldest: Vec<_> = [7, 5, 6] + .into_iter() + .map(|seq| (seq, sender.open(Some(&ascending), -seq, 3_000))) + .collect(); + + let order = sender.drain(100); + let finished = |streams: &[(i64, StreamId)]| { + let mut last: Vec<_> = streams + .iter() + .map(|&(seq, id)| (order.iter().rposition(|&x| x == id).unwrap(), seq)) + .collect(); + last.sort(); + last.into_iter().map(|(_, seq)| seq).collect::>() + }; + assert_eq!(finished(&newest), [4, 3, 2, 1]); + assert_eq!(finished(&oldest), [5, 6, 7]); +} + +/// Group order takes full 62-bit group sequences and sparse jumps between them, +/// which a 32-bit scalar shared with the track priority would truncate or wrap. +#[test] +fn group_order_holds_wide_and_sparse_sequences() { + let mut sender = Sender::new(); + let group = sender.group(0); + let seqs = [ + 0, + 1 << 40, + (1 << 62) - 1, + 7, + 1 << 33, + (1 << 32) + 1, + 1 << 32, + ]; + let mut streams: Vec<_> = seqs + .into_iter() + .map(|seq| (seq, sender.open(Some(&group), seq, 100))) + .collect(); + + let order = sender.drain(10); + streams.sort_by_key(|&(seq, _)| std::cmp::Reverse(seq)); + let expected: Vec<_> = streams.into_iter().map(|(_, id)| id).collect(); + assert_eq!(order, expected); +} + +/// A stream blocked by flow control holds no turn and saves up no credit while it +/// waits, so its group neither stalls the other nor bursts once unblocked. +#[test] +fn blocked_streams_hold_no_turn() { + let mut sender = Sender::limited(5_000u32.into()); + let blocked = sender.group(0); + let open = sender.group(0); + let blocked = sender.open(Some(&blocked), 0, 5_000); + let open = sender.open(Some(&open), 0, 5_000); + let unlimited = 1 << 60; + sender + .state + .received_max_stream_data(open, unlimited) + .unwrap(); + sender.write(open, 1_000_000); + + sender.drain(50); + assert_eq!(sender.bytes(&[blocked]), 5_000); + assert!(sender.bytes(&[open]) > 50_000); + + sender + .state + .received_max_stream_data(blocked, unlimited) + .unwrap(); + sender.write(blocked, 1_000_000); + sender.clear(); + sender.drain(100); + assert_fair(sender.wire(&[blocked]), sender.wire(&[open])); +} + +/// A lost frame retransmits through its group, so new data from an equal group +/// delays it by a turn at most. +#[test] +fn retransmits_are_not_starved_by_equal_groups() { + let mut sender = Sender::new(); + let lossy = sender.group(0); + let busy = sender.group(0); + let lossy = sender.open(Some(&lossy), 0, 1_000); + let busy = sender.open(Some(&busy), 0, 10_000_000); + + let lost = sender + .packet() + .into_iter() + .find(|frame| frame.id == lossy) + .unwrap(); + sender.drain(10); + sender.state.retransmit(lost); + sender.clear(); + sender.drain(2); + assert_eq!(sender.bytes(&[lossy]), 1_000); + assert!(sender.bytes(&[busy]) > 0); +} + +/// Streams outside every group, such as control streams, go before all of them. +#[test] +fn ungrouped_streams_preempt_groups() { + let mut sender = Sender::new(); + let group = sender.group(i64::MAX); + let grouped = sender.open(Some(&group), i64::MAX, 10_000); + let control = sender.open(None, i64::MIN, 10_000); + + let order = sender.drain(20); + let first = order.iter().position(|&id| id == grouped).unwrap(); + assert!(order[..first].iter().all(|&id| id == control)); + assert_eq!(sender.bytes(&[control]), 10_000); +} + +/// Domains split bytes evenly however many groups each holds and whatever their +/// priorities, while priority still orders the groups inside one domain. +#[test] +fn domains_share_bytes_and_confine_priority() { + let mut sender = Sender::new(); + let crowded = sender.groups().open_domain(); + let quiet = sender.groups().open_domain(); + + let crowded_groups: Vec<_> = (0..10) + .map(|priority| sender.groups().open(priority, Some(&crowded))) + .collect(); + let crowded_streams: Vec<_> = crowded_groups + .iter() + .map(|group| sender.open(Some(group), 0, 1_000_000)) + .collect(); + let quiet_group = sender.groups().open(-100, Some(&quiet)); + let quiet_stream = sender.open(Some(&quiet_group), 0, 1_000_000); + + sender.drain(200); + assert_fair(sender.wire(&crowded_streams), sender.wire(&[quiet_stream])); + // Only the crowded domain's top group sent anything. + assert_eq!( + sender.bytes(&crowded_streams[9..]), + sender.bytes(&crowded_streams) + ); +} + +/// Groups outside any explicit domain share the default domain, which takes one share +/// of the domain round robin. +#[test] +fn the_default_domain_is_one_share() { + let mut sender = Sender::new(); + let domain = sender.groups().open_domain(); + let inside = sender.groups().open(0, Some(&domain)); + let outside: Vec = (0..3).map(|_| sender.group(0)).collect(); + let inside = sender.open(Some(&inside), 0, 1_000_000); + let outside: Vec<_> = outside + .iter() + .map(|group| sender.open(Some(group), 0, 1_000_000)) + .collect(); + + sender.drain(200); + assert_fair(sender.wire(&[inside]), sender.wire(&outside)); +} + +/// A released group lives until its last stream is gone, here by a reset, and then +/// leaves nothing behind in the schedule. +#[test] +fn reset_frees_a_released_group() { + let mut sender = Sender::new(); + let domain: SendDomain = sender.groups().open_domain(); + let group = sender.groups().open(0, Some(&domain)); + let reset = sender.open(Some(&group), 0, 10_000); + let finished = sender.open(Some(&group), 1, 100); + sender.send(finished).finish().unwrap(); + sender.groups().close(group); + sender.groups().close_domain(domain); + + assert_eq!(sender.drain(1)[0], finished); + sender.send(reset).reset(0u32.into()).unwrap(); + assert!(!sender.state.can_send_stream_data()); + assert!(sender.drain(1).is_empty()); + + sender.state.received_ack_of(StreamMeta { + id: finished, + offsets: 0..100, + fin: true, + }); + assert!(!sender.state.pending.is_clean()); + sender.state.reset_acked(reset, 0); + assert!(sender.state.pending.is_clean()); +} + +/// A reset that still delivers its first bytes keeps the stream in its group for just +/// that prefix, and the released group is freed once the reset is acknowledged. +#[test] +fn reliable_reset_sends_its_prefix_then_frees_its_group() { + let mut sender = Sender::new(); + let group = sender.group(0); + let reset = sender.open(Some(&group), 0, 10_000); + let finished = sender.open(Some(&group), 0, 10_000); + sender.send(finished).finish().unwrap(); + sender.groups().close(group); + + sender.send(reset).reset_to(0u32.into(), 3_000).unwrap(); + assert!(sender.state.can_send_stream_data()); + sender.drain(100); + assert_eq!(sender.bytes(&[reset]), 3_000, "only the reliable prefix"); + assert_eq!(sender.bytes(&[finished]), 10_000); + assert!(!sender.state.can_send_stream_data()); + + sender.state.received_ack_of(StreamMeta { + id: finished, + offsets: 0..10_000, + fin: true, + }); + sender.state.received_ack_of(StreamMeta { + id: reset, + offsets: 0..3_000, + fin: false, + }); + assert!( + !sender.state.pending.is_clean(), + "the reset is unacknowledged" + ); + sender.state.reset_acked(reset, 3_000); + assert!(sender.state.pending.is_clean()); +} + +/// STOP_SENDING abandons the prefix a reset still owed, so the stream leaves the +/// schedule rather than linger in it. +#[test] +fn stop_sending_drops_a_reliable_prefix_from_the_schedule() { + let mut sender = Sender::new(); + let group = sender.group(0); + let reset = sender.open(Some(&group), 0, 10_000); + sender.send(reset).reset_to(0u32.into(), 3_000).unwrap(); + assert!(sender.state.can_send_stream_data()); + + assert_eq!( + sender.state.received_stop_sending(reset, 0u32.into()), + Some(0) + ); + assert!(!sender.state.can_send_stream_data()); + assert!(sender.drain(1).is_empty()); +} + +/// A stream can leave its group for the ungrouped queue, keeping its data queued. +#[test] +fn streams_can_leave_their_group() { + let mut sender = Sender::new(); + let group = sender.group(0); + let id = sender.open(Some(&group), 0, 100); + sender.send(id).set_group(None).unwrap(); + sender.groups().close(group); + assert!(sender.state.pending.is_clean()); + assert_eq!(sender.drain(1), [id]); +} diff --git a/rs/moq-quic/src/connection/streams/schedule.rs b/rs/moq-quic/src/connection/streams/schedule.rs new file mode 100644 index 00000000000..7d316452984 --- /dev/null +++ b/rs/moq-quic/src/connection/streams/schedule.rs @@ -0,0 +1,660 @@ +//! The order streams with queued data are sent in. +//! +//! Streams outside any send group go first, in quinn's order: highest priority first, round +//! robin between equal priorities. Everything else is a hierarchy: +//! +//! 1. Domains share the remaining bytes by deficit round robin. +//! 2. Inside a domain, a higher-priority send group preempts every lower one. +//! 3. Send groups of equal priority share their domain's bytes by deficit round robin. +//! 4. Inside a group, streams go in their own priority order, round robin between ties. +//! +//! Both round robins charge the bytes actually written, so a group of many small streams and a +//! group of one large stream get the same rate. A turn may overdraw its credit by one frame; +//! the debt carries into the next turn, so the error stays bounded by a frame. +//! +//! Self-contained (std and slab only) so the scheduler benchmark can compile it from source. + +use std::collections::{BTreeMap, VecDeque}; + +use slab::Slab; + +/// Bytes a domain or group may send per turn of its round robin before it yields. +/// +/// One minimum-size QUIC packet, so equal groups interleave about once per packet. +const QUANTUM: i64 = 1200; + +/// The index of the domain every group outside an explicit domain belongs to. +const DEFAULT_DOMAIN: usize = 0; + +/// A stream's place in the schedule, stored with the stream. +#[derive(Debug, Default)] +pub(super) struct Slot { + /// The send group the stream belongs to, or `None` for the ungrouped queue. + group: Option, + /// Where the stream is queued, always in its own group's queue. + queued: Option, +} + +#[derive(Debug, Clone, Copy)] +enum Queued { + /// The stream a group resumes first, when fairness between its streams is off. + Next, + At(Key), +} + +/// A queued stream's priority, then a decreasing counter so ties go round robin. +type Key = (i64, u64); + +#[derive(Debug)] +struct Queue { + streams: BTreeMap, + next: Option, +} + +impl Queue { + fn new() -> Self { + Self { + streams: BTreeMap::new(), + next: None, + } + } + + fn pop(&mut self) -> Option { + self.next + .take() + .or_else(|| self.streams.pop_last().map(|(_, id)| id)) + } + + fn remove(&mut self, queued: Queued) { + match queued { + Queued::Next => self.next = None, + Queued::At(key) => { + self.streams.remove(&key); + } + } + } + + fn is_empty(&self) -> bool { + self.next.is_none() && self.streams.is_empty() + } + + #[cfg(test)] + fn len(&self) -> usize { + self.streams.len() + self.next.is_some() as usize + } +} + +#[derive(Debug)] +struct Group { + domain: usize, + priority: i64, + queue: Queue, + /// Bytes this group may still send in its current turn; negative after an overdraw. + deficit: i64, + /// Whether the group has queued streams, or is being served one. It stays set between + /// [`Schedule::pop`] and [`Schedule::sent`], so requeueing the stream is not a new turn. + active: bool, + /// Whether an entry for this group sits in its level's round robin. Usually equal to + /// having queued streams, but a group that runs dry away from the front of the round + /// robin keeps its entry until the round robin reaches it. + ringed: bool, + /// Streams that name this group, queued or not. + members: usize, + /// Whether the owner released the group, which is freed once no stream names it. + closed: bool, +} + +#[derive(Debug)] +struct Domain { + /// The round robin of groups at each priority. Only the highest level with a backlogged + /// group is served. + levels: BTreeMap>, + deficit: i64, + /// Whether an entry for this domain sits in the domain round robin, as for groups. + ringed: bool, + /// Groups in this domain with queued streams. + backlogged: usize, + /// Groups in this domain that are not yet freed. + groups: usize, + closed: bool, +} + +impl Domain { + fn new() -> Self { + Self { + levels: BTreeMap::new(), + deficit: 0, + ringed: false, + backlogged: 0, + groups: 0, + closed: false, + } + } +} + +/// Every stream with data to send, in the order to send it. +#[derive(Debug)] +pub(super) struct Schedule { + ungrouped: Queue, + groups: Slab>, + domains: Slab, + /// The domain round robin. + ring: VecDeque, + /// Domains with a backlogged group. + backlogged: usize, + /// Decreases with every push, so a requeued stream goes behind its equal-priority peers. + recency: u64, +} + +impl Schedule { + pub(super) fn new() -> Self { + let mut domains = Slab::new(); + assert_eq!(domains.insert(Domain::new()), DEFAULT_DOMAIN); + Self { + ungrouped: Queue::new(), + groups: Slab::new(), + domains, + ring: VecDeque::new(), + backlogged: 0, + recency: u64::MAX, + } + } + + /// Whether no stream is queued. + pub(super) fn is_empty(&self) -> bool { + self.ungrouped.is_empty() && self.backlogged == 0 + } + + #[cfg(test)] + pub(super) fn len(&self) -> usize { + self.ungrouped.len() + + self + .groups + .iter() + .map(|(_, g)| g.queue.len()) + .sum::() + } + + /// Whether every group and domain the application released has been freed. + #[cfg(test)] + pub(super) fn is_clean(&self) -> bool { + self.groups.iter().all(|(_, g)| !g.closed) && self.domains.iter().all(|(_, d)| !d.closed) + } + + /// Create a domain, sharing bytes equally with every other domain. + pub(super) fn open_domain(&mut self) -> usize { + self.domains.insert(Domain::new()) + } + + /// Release a domain; it is freed once its last group is. + pub(super) fn close_domain(&mut self, domain: usize) { + assert_ne!(domain, DEFAULT_DOMAIN, "the default domain is never closed"); + self.domains[domain].closed = true; + self.free_domain(domain); + } + + /// Create a send group in `domain`, or the default domain. + pub(super) fn open_group(&mut self, priority: i64, domain: Option) -> usize { + let domain = domain.unwrap_or(DEFAULT_DOMAIN); + let state = &mut self.domains[domain]; + assert!(!state.closed, "group opened in a closed domain"); + state.groups += 1; + self.groups.insert(Group { + domain, + priority, + queue: Queue::new(), + deficit: 0, + active: false, + ringed: false, + members: 0, + closed: false, + }) + } + + /// Release a group; it is freed once no stream names it. + pub(super) fn close_group(&mut self, group: usize) { + self.groups[group].closed = true; + self.free_group(group); + } + + pub(super) fn group_priority(&self, group: usize) -> i64 { + self.groups[group].priority + } + + /// Move a group to another priority level, keeping its credit. + pub(super) fn set_group_priority(&mut self, group: usize, priority: i64) { + let state = &mut self.groups[group]; + if state.priority == priority { + return; + } + + let old = std::mem::replace(&mut state.priority, priority); + let domain = state.domain; + if state.ringed { + Self::unring_group(&mut self.domains[domain], group, old); + let state = &mut self.groups[group]; + state.ringed = !state.queue.is_empty(); + if state.ringed { + self.domains[domain] + .levels + .entry(priority) + .or_default() + .push_back(group); + } + } + } + + /// Move a stream to `group`, or out of every group, keeping its place in line if queued. + pub(super) fn join(&mut self, id: K, slot: &mut Slot, group: Option, priority: i64) { + if slot.group == group { + return; + } + if let Some(group) = group { + assert!(!self.groups[group].closed, "stream joined a closed group"); + } + + let queued = slot.queued.is_some(); + self.remove(slot); + self.release(slot); + slot.group = group; + if let Some(group) = group { + self.groups[group].members += 1; + } + if queued { + self.push(id, slot, priority); + } + } + + /// Forget a stream that is being freed. + pub(super) fn leave(&mut self, slot: &mut Slot) { + self.remove(slot); + self.release(slot); + } + + /// Queue a stream behind every queued stream of its group at the same priority. + /// + /// A stream already queued keeps its place. + pub(super) fn push(&mut self, id: K, slot: &mut Slot, priority: i64) { + if slot.queued.is_some() { + return; + } + self.recency -= 1; + let key = (priority, self.recency); + slot.queued = Some(Queued::At(key)); + self.queue(slot.group).streams.insert(key, id); + self.activate(slot.group); + } + + /// Queue a stream ahead of every other stream of its group, to finish what it started. + pub(super) fn reinsert(&mut self, id: K, slot: &mut Slot) { + debug_assert!(slot.queued.is_none(), "stream queued twice"); + let queue = self.queue(slot.group); + assert!(queue.next.is_none()); + queue.next = Some(id); + slot.queued = Some(Queued::Next); + self.activate(slot.group); + } + + /// Dequeue a stream that has nothing left to send, such as one being reset. + pub(super) fn remove(&mut self, slot: &mut Slot) { + let Some(queued) = slot.queued.take() else { + return; + }; + self.queue(slot.group).remove(queued); + self.deactivate(slot.group); + } + + /// Take the next stream to send from. + /// + /// The caller writes from it and then reports back with [`Self::sent`], which requeues + /// it and charges the bytes to its group. + pub(super) fn pop(&mut self) -> Option { + if let Some(id) = self.ungrouped.pop() { + return Some(id); + } + + loop { + let &domain = self.ring.front()?; + let state = &mut self.domains[domain]; + if state.backlogged == 0 { + self.ring.pop_front(); + state.ringed = false; + continue; + } + if state.deficit <= 0 { + state.deficit += QUANTUM; + self.ring.rotate_left(1); + continue; + } + + let mut level = state + .levels + .last_entry() + .expect("backlogged domain has a level"); + let Some(&group) = level.get().front() else { + level.remove(); + continue; + }; + let group_state = &mut self.groups[group]; + if !group_state.active { + level.get_mut().pop_front(); + group_state.ringed = false; + continue; + } + if group_state.deficit <= 0 { + group_state.deficit += QUANTUM; + level.get_mut().rotate_left(1); + continue; + } + + return Some(group_state.queue.pop().expect("checked non-empty")); + } + } + + /// Report the `bytes` written from a stream [`Self::pop`] returned, requeueing it at + /// `priority` if it has more to send. + /// + /// With `fair`, it goes behind its group's equal-priority streams; without, its group + /// resumes it before any other. + pub(super) fn sent( + &mut self, + id: K, + slot: &mut Slot, + bytes: usize, + requeue: Option, + fair: bool, + ) { + slot.queued = None; + if let Some(group) = slot.group { + let bytes = bytes as i64; + let state = &mut self.groups[group]; + state.deficit -= bytes; + self.domains[state.domain].deficit -= bytes; + } + + match requeue { + Some(priority) if fair => self.push(id, slot, priority), + Some(_) => self.reinsert(id, slot), + None => self.deactivate(slot.group), + } + } + + fn queue(&mut self, group: Option) -> &mut Queue { + match group { + Some(group) => &mut self.groups[group].queue, + None => &mut self.ungrouped, + } + } + + /// Give a group that just got its first queued stream a turn. + fn activate(&mut self, group: Option) { + let Some(group) = group else { + return; + }; + let state = &mut self.groups[group]; + if state.active { + return; + } + state.active = true; + + // Debt carries over, but unused credit does not, so going idle saves nothing up. + state.deficit = state.deficit.min(0) + QUANTUM; + let domain = &mut self.domains[state.domain]; + if !state.ringed { + state.ringed = true; + domain + .levels + .entry(state.priority) + .or_default() + .push_back(group); + } + + domain.backlogged += 1; + if domain.backlogged > 1 { + return; + } + + self.backlogged += 1; + domain.deficit = domain.deficit.min(0) + QUANTUM; + if !domain.ringed { + domain.ringed = true; + self.ring.push_back(self.groups[group].domain); + } + } + + /// Account for a group whose queue may have run dry. + /// + /// A group or domain heading its round robin, as one just served does, leaves it now. + /// Anywhere else its entry stays until the round robin reaches it, since finding it + /// would mean a scan. + fn deactivate(&mut self, group: Option) { + let Some(group) = group else { + return; + }; + let state = &mut self.groups[group]; + if !state.active || !state.queue.is_empty() { + return; + } + state.active = false; + + let domain = &mut self.domains[state.domain]; + if let Some(level) = domain.levels.get_mut(&state.priority) + && level.front() == Some(&group) + { + level.pop_front(); + if level.is_empty() { + domain.levels.remove(&state.priority); + } + state.ringed = false; + } + + domain.backlogged -= 1; + if domain.backlogged > 0 { + return; + } + + self.backlogged -= 1; + if self.ring.front() == Some(&state.domain) { + self.ring.pop_front(); + domain.ringed = false; + } + } + + /// Drop a stream's membership of its group. + fn release(&mut self, slot: &mut Slot) { + if let Some(group) = slot.group.take() { + self.groups[group].members -= 1; + self.free_group(group); + } + } + + fn free_group(&mut self, group: usize) { + let state = &self.groups[group]; + if !state.closed || state.members > 0 { + return; + } + + // No members means nothing queued, but a stale round-robin entry may remain. + let state = self.groups.remove(group); + let domain = &mut self.domains[state.domain]; + if state.ringed { + Self::unring_group(domain, group, state.priority); + } + domain.groups -= 1; + self.free_domain(state.domain); + } + + fn free_domain(&mut self, domain: usize) { + let state = &self.domains[domain]; + if !state.closed || state.groups > 0 { + return; + } + + if self.domains.remove(domain).ringed { + let index = self.ring.iter().position(|&d| d == domain); + self.ring + .remove(index.expect("ringed domain is in the ring")); + } + } + + fn unring_group(domain: &mut Domain, group: usize, priority: i64) { + let level = domain + .levels + .get_mut(&priority) + .expect("ringed group has a level"); + let index = level.iter().position(|&g| g == group); + level.remove(index.expect("ringed group is in its level")); + if level.is_empty() { + domain.levels.remove(&priority); + } + } +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + + use super::*; + + /// Streams with a fixed number of bytes each, drained a frame at a time. + struct Sim { + schedule: Schedule, + streams: HashMap, + next: u64, + fair: bool, + } + + struct Stream { + slot: Slot, + priority: i64, + remaining: usize, + } + + impl Sim { + fn new() -> Self { + Self { + schedule: Schedule::new(), + streams: HashMap::new(), + next: 0, + fair: true, + } + } + + fn stream(&mut self, group: Option, priority: i64, bytes: usize) -> u64 { + let id = self.next; + self.next += 1; + let mut stream = Stream { + slot: Slot::default(), + priority, + remaining: bytes, + }; + self.schedule.join(id, &mut stream.slot, group, priority); + self.schedule.push(id, &mut stream.slot, priority); + self.streams.insert(id, stream); + id + } + + /// Send one frame of up to `frame` bytes, returning the stream it came from. + fn step(&mut self, frame: usize) -> Option { + let id = self.schedule.pop()?; + let stream = self.streams.get_mut(&id).unwrap(); + let bytes = frame.min(stream.remaining); + stream.remaining -= bytes; + let requeue = (stream.remaining > 0).then_some(stream.priority); + self.schedule + .sent(id, &mut stream.slot, bytes, requeue, self.fair); + if requeue.is_none() { + let mut stream = self.streams.remove(&id).unwrap(); + self.schedule.leave(&mut stream.slot); + } + Some(id) + } + } + + #[test] + fn ungrouped_streams_go_first() { + let mut sim = Sim::new(); + let group = sim.schedule.open_group(i64::MAX, None); + let grouped = sim.stream(Some(group), i64::MAX, 100); + let ungrouped = sim.stream(None, i64::MIN, 100); + assert_eq!(sim.step(1000), Some(ungrouped)); + assert_eq!(sim.step(1000), Some(grouped)); + assert_eq!(sim.step(1000), None); + assert!(sim.schedule.is_empty()); + } + + #[test] + fn a_dry_group_away_from_the_front_is_skipped() { + let mut sim = Sim::new(); + let a = sim.schedule.open_group(0, None); + let b = sim.schedule.open_group(0, None); + let first = sim.stream(Some(a), 0, 100_000); + let second = sim.stream(Some(b), 0, 100_000); + + // `b` runs dry while `a` heads the round robin. + let mut stream = sim.streams.remove(&second).unwrap(); + sim.schedule.leave(&mut stream.slot); + for _ in 0..10 { + assert_eq!(sim.step(1000), Some(first)); + } + + // And rejoins with a fresh turn. + let second = sim.stream(Some(b), 0, 100_000); + let mut seen = [0usize; 2]; + for _ in 0..100 { + let id = sim.step(1000).unwrap(); + seen[(id == second) as usize] += 1; + } + assert!(seen[0].abs_diff(seen[1]) <= 2, "{seen:?}"); + } + + #[test] + fn closed_groups_and_domains_free_with_their_last_stream() { + let mut sim = Sim::new(); + let domain = sim.schedule.open_domain(); + let group = sim.schedule.open_group(0, Some(domain)); + let id = sim.stream(Some(group), 0, 10); + sim.schedule.close_group(group); + sim.schedule.close_domain(domain); + assert!(!sim.schedule.is_clean()); + + assert_eq!(sim.step(1000), Some(id)); + assert!(sim.schedule.is_clean()); + assert!(sim.schedule.groups.is_empty()); + assert_eq!(sim.schedule.domains.len(), 1); + assert!(sim.schedule.ring.is_empty()); + } + + #[test] + fn moving_a_queued_stream_keeps_it_queued() { + let mut sim = Sim::new(); + let low = sim.schedule.open_group(0, None); + let high = sim.schedule.open_group(1, None); + let a = sim.stream(Some(low), 0, 10); + let b = sim.stream(Some(low), 0, 10); + + let stream = sim.streams.get_mut(&b).unwrap(); + sim.schedule.join(b, &mut stream.slot, Some(high), 0); + assert_eq!(sim.step(1000), Some(b)); + assert_eq!(sim.step(1000), Some(a)); + assert!(sim.schedule.is_empty()); + assert!(sim.schedule.ring.is_empty()); + } + + #[test] + fn priority_changes_move_a_backlogged_group() { + let mut sim = Sim::new(); + let a = sim.schedule.open_group(1, None); + let b = sim.schedule.open_group(0, None); + let first = sim.stream(Some(a), 0, 100_000); + let second = sim.stream(Some(b), 0, 100_000); + assert_eq!(sim.step(1000), Some(first)); + + sim.schedule.set_group_priority(b, 2); + assert_eq!(sim.step(1000), Some(second)); + sim.schedule.set_group_priority(a, 2); + // Equal again, so the two alternate. + let ids: Vec<_> = (0..4).map(|_| sim.step(1000).unwrap()).collect(); + assert!(ids.contains(&first) && ids.contains(&second), "{ids:?}"); + } +} diff --git a/rs/moq-quic/src/connection/streams/send.rs b/rs/moq-quic/src/connection/streams/send.rs index 729e39e6811..0b59adabd1b 100644 --- a/rs/moq-quic/src/connection/streams/send.rs +++ b/rs/moq-quic/src/connection/streams/send.rs @@ -1,6 +1,7 @@ use bytes::Bytes; use thiserror::Error; +use super::schedule::Slot; use crate::{StreamId, VarInt, connection::send_buffer::SendBuffer, frame}; #[derive(Debug)] @@ -8,7 +9,8 @@ pub(super) struct Send { pub(super) max_data: u64, pub(super) state: SendState, pub(super) pending: SendBuffer, - pub(super) priority: i32, + pub(super) priority: i64, + pub(super) slot: Slot, /// Whether a frame containing a FIN bit must be transmitted, even if we don't have any new data pub(super) fin_pending: bool, /// Whether this stream is in the `connection_blocked` list of `Streams` @@ -28,6 +30,7 @@ impl Send { state: SendState::Ready, pending: SendBuffer::new(), priority: 0, + slot: Slot::default(), fin_pending: false, connection_blocked: false, data_blocked_limit: None, diff --git a/rs/moq-quic/src/connection/streams/state.rs b/rs/moq-quic/src/connection/streams/state.rs index dfc8b4ce196..5f767b35b37 100644 --- a/rs/moq-quic/src/connection/streams/state.rs +++ b/rs/moq-quic/src/connection/streams/state.rs @@ -9,8 +9,8 @@ use rustc_hash::FxHashMap; use tracing::{debug, trace}; use super::{ - Credit, PendingStreamsQueue, Recv, Retransmits, Send, SendState, ShouldTransmit, StreamEvent, - StreamHalf, ThinRetransmits, + Credit, Recv, Retransmits, Schedule, Send, SendState, ShouldTransmit, StreamEvent, StreamHalf, + ThinRetransmits, }; use crate::{ Dir, MAX_STREAM_COUNT, Side, StreamId, TransportError, VarInt, @@ -101,8 +101,8 @@ pub struct StreamsState { /// This differs from `self.send.len()` in that it does not include streams that the peer is /// permitted to open but which have not yet been opened. pub(super) send_streams: usize, - /// Streams with outgoing data queued, sorted by priority - pub(super) pending: PendingStreamsQueue, + /// Streams with outgoing data queued, in the order to send them + pub(super) pending: Schedule, events: VecDeque, /// Streams blocked on connection-level flow control or stream window space @@ -182,7 +182,7 @@ impl StreamsState { opened: [false, false], next_reported_remote: [0, 0], send_streams: 0, - pending: PendingStreamsQueue::new(), + pending: Schedule::new(), events: VecDeque::new(), connection_blocked: Vec::new(), max_data: 0, @@ -241,7 +241,9 @@ impl StreamsState { // We don't bother calling `stream_freed` here because we explicitly reset affected // counters below. let id = StreamId::new(self.side, dir, i); - self.send.remove(&id).unwrap(); + if let Some(mut stream) = self.send.remove(&id).unwrap() { + self.pending.leave(&mut stream.slot); + } if let Dir::Bi = dir { self.recv.remove(&id).unwrap(); } @@ -256,7 +258,6 @@ impl StreamsState { } self.generation += 1; - self.pending.clear(); self.send_streams = 0; self.data_sent = 0; self.buffered_data = 0; @@ -401,6 +402,9 @@ impl StreamsState { let buffered = stream.pending.buffered(); let reset = stream.abandon_reliable(); self.buffered_data -= buffered - stream.pending.buffered(); + if !stream.is_pending() { + self.pending.remove(&mut stream.slot); + } reset } @@ -415,21 +419,16 @@ impl StreamsState { .as_mut() .is_some_and(|s| s.reset_acked(reliable_size)) { - e.remove_entry(); + if let (_, Some(mut stream)) = e.remove_entry() { + self.pending.leave(&mut stream.slot); + } self.stream_freed(id, StreamHalf::Send); } } /// Whether any stream data is queued, regardless of control frames pub(crate) fn can_send_stream_data(&self) -> bool { - // Reset streams may linger in the pending stream list, but only produce stream frames - // for data below their Reliable Size - self.pending.iter().any(|stream| { - self.send - .get(&stream.id) - .and_then(|s| s.as_ref()) - .is_some_and(|s| !s.is_reset() || s.is_pending()) - }) + !self.pending.is_empty() } /// Whether MAX_STREAM_DATA frames could be sent for stream `id` @@ -655,25 +654,17 @@ impl StreamsState { break; } - // Pop the stream of the highest priority that currently has pending data - // If the stream still has some pending data left after writing, it will be reinserted, otherwise not - let Some(stream) = self.pending.pop() else { + // Pop the next stream to send from. A stream leaves the schedule once it has + // nothing left to send, so a reset one stays only for data below its Reliable Size. + let Some(id) = self.pending.pop() else { break; }; - - let id = stream.id; - - let Some(stream) = self.send.get_mut(&id).and_then(|s| s.as_mut()) else { - // Stream was reset with pending data and the reset was acknowledged - continue; - }; - - // Reset streams aren't removed from the pending list and still exist while the peer - // hasn't acknowledged the reset, but only generate STREAM frames for data below their - // Reliable Size, so we need to check for them explicitly. - if stream.is_reset() && !stream.is_pending() { - continue; - } + let stream = self + .send + .get_mut(&id) + .and_then(|s| s.as_mut()) + .expect("scheduled stream is open"); + let start = buf.len(); // Now that we know the `StreamId`, we can better account for how many bytes // are required to encode it. @@ -685,18 +676,6 @@ impl StreamsState { stream.fin_pending = false; } - if stream.is_pending() { - // If the stream still has pending data, reinsert it, possibly with an updated priority value - // Fairness with other streams is achieved by implementing round-robin scheduling, - // so that the other streams will have a chance to write data - // before we touch this stream again. - if fair { - self.pending.push_pending(id, stream.priority); - } else { - self.pending.reinsert_pending(id, stream.priority); - } - } - let meta = frame::StreamMeta { id, offsets, fin }; trace!(id = %meta.id, off = meta.offsets.start, len = meta.offsets.end - meta.offsets.start, fin = meta.fin, "STREAM"); meta.encode(encode_length, buf); @@ -710,6 +689,13 @@ impl StreamsState { offsets.start += data.len() as u64; buf.put_slice(data); } + + // Requeue a stream with data left, possibly at an updated priority. With fairness, + // it goes behind its equal-priority peers so they get a turn first. The frame's + // bytes count against its group's share either way. + let requeue = stream.is_pending().then_some(stream.priority); + self.pending + .sent(id, &mut stream.slot, buf.len() - start, requeue, fair); stream_frames.push(meta); } @@ -745,7 +731,9 @@ impl StreamsState { // A reset is not reported as finished, like one without a Reliable Size let reset = stream.is_reset(); - entry.remove_entry(); + if let (_, Some(mut stream)) = entry.remove_entry() { + self.pending.leave(&mut stream.slot); + } self.stream_freed(id, StreamHalf::Send); if !reset { self.events.push_back(StreamEvent::Finished { id }); @@ -763,7 +751,8 @@ impl StreamsState { stream.fin_pending |= frame.fin && !stream.is_reset(); stream.pending.retransmit(frame.offsets); if !was_pending && stream.is_pending() { - self.pending.push_pending(frame.id, stream.priority); + self.pending + .push(frame.id, &mut stream.slot, stream.priority); } } @@ -780,7 +769,7 @@ impl StreamsState { continue; } if !stream.is_pending() { - self.pending.push_pending(id, stream.priority); + self.pending.push(id, &mut stream.slot, stream.priority); } stream.pending.retransmit_all_for_0rtt(); } @@ -3221,6 +3210,28 @@ mod tests { assert_eq!(client.local_max_data, initial + 16); } + /// A group token indexes its own connection's schedule, so another connection refuses it + /// rather than reach whichever of its own groups shares the index. + #[test] + #[should_panic(expected = "send group used on another connection")] + fn send_group_from_another_connection() { + let mut a = make(Side::Client); + let mut b = make(Side::Client); + let group = crate::SendGroups { state: &mut a }.open(0, None); + let _other = crate::SendGroups { state: &mut b }.open(0, None); + crate::SendGroups { state: &mut b }.set_priority(&group, 1); + } + + #[test] + #[should_panic(expected = "send domain used on another connection")] + fn send_domain_from_another_connection() { + let mut a = make(Side::Client); + let mut b = make(Side::Client); + let domain = crate::SendGroups { state: &mut a }.open_domain(); + let _other = crate::SendGroups { state: &mut b }.open_domain(); + let _ = crate::SendGroups { state: &mut b }.open(0, Some(&domain)); + } + /// Another connection holding at least as much credit still refuses it. #[test] #[should_panic(expected = "released credit from another connection")] diff --git a/rs/moq-quic/src/lib.rs b/rs/moq-quic/src/lib.rs index 7ceb2ad916b..c5915b3f8a9 100644 --- a/rs/moq-quic/src/lib.rs +++ b/rs/moq-quic/src/lib.rs @@ -46,8 +46,8 @@ mod connection; pub use crate::connection::{ Checkpoint, Chunk, Chunks, ClosedStream, Connection, ConnectionError, ConnectionStats, Credit, CreditStats, Datagrams, Event, FinishError, FrameStats, PathStats, ReadError, ReadableError, - RecvStream, RttEstimator, SendDatagramError, SendStream, ShouldTransmit, StreamEvent, Streams, - UdpStats, WriteError, Written, + RecvStream, RttEstimator, SendDatagramError, SendDomain, SendGroup, SendGroups, SendStream, + ShouldTransmit, StreamEvent, Streams, UdpStats, WriteError, Written, }; #[cfg(feature = "qlog")] pub use connection::qlog::QlogStream; diff --git a/rs/moq-tokio/src/quic/driver/connection.rs b/rs/moq-tokio/src/quic/driver/connection.rs index 17ab2f70ea3..d5e80f4bfd1 100644 --- a/rs/moq-tokio/src/quic/driver/connection.rs +++ b/rs/moq-tokio/src/quic/driver/connection.rs @@ -25,7 +25,7 @@ use tracing::{Instrument, Span, debug_span}; use crate::quic::driver::{ ConnectionEvent, Duration, Instant, VarInt, mutex::Mutex, recv_stream::RecvStream, - runtime::UdpSender, send_stream::SendStream, udp_transmit, + runtime::UdpSender, send_group::SendGroup, send_stream::SendStream, udp_transmit, }; use moq_quic::{ ConnectionError, ConnectionHandle, ConnectionStats, Dir, EndpointEvent, Side, StreamEvent, @@ -346,6 +346,13 @@ impl Connection { } } + /// Open a send group at `priority`, for streams to join with [`SendStream::set_group`]. + /// + /// [`SendStream::set_group`]: crate::quic::driver::SendStream::set_group + pub fn send_group(&self, priority: i64) -> SendGroup { + SendGroup::new(self.0.0.clone(), priority) + } + /// Initiate a new outgoing bidirectional stream. /// /// Streams are cheap and instantaneous to open unless blocked by flow control. As a diff --git a/rs/moq-tokio/src/quic/driver/mod.rs b/rs/moq-tokio/src/quic/driver/mod.rs index 7a8b6a68cf1..b248a9fe098 100644 --- a/rs/moq-tokio/src/quic/driver/mod.rs +++ b/rs/moq-tokio/src/quic/driver/mod.rs @@ -47,6 +47,7 @@ mod incoming; mod mutex; mod recv_stream; mod runtime; +mod send_group; mod send_stream; mod work_limiter; @@ -78,6 +79,7 @@ pub use crate::quic::driver::recv_stream::{ #[cfg(test)] pub(crate) use crate::quic::driver::runtime::wrap_udp_socket; pub use crate::quic::driver::runtime::{AsyncUdpSocket, UdpSender}; +pub use crate::quic::driver::send_group::SendGroup; pub use crate::quic::driver::send_stream::{Checkpoint, SendStream, StoppedError, WriteError}; #[cfg(test)] diff --git a/rs/moq-tokio/src/quic/driver/send_group.rs b/rs/moq-tokio/src/quic/driver/send_group.rs new file mode 100644 index 00000000000..41cd2e0ab44 --- /dev/null +++ b/rs/moq-tokio/src/quic/driver/send_group.rs @@ -0,0 +1,63 @@ +use std::sync::Arc; + +use crate::quic::driver::connection::ConnectionInner; + +/// A send group on a [`Connection`](crate::quic::driver::Connection) +/// +/// Streams that [join](crate::quic::driver::SendStream::set_group) a group share bytes with +/// the connection's other groups: a higher priority group preempts every lower one, and groups +/// of equal priority get an equal share of bytes however many streams each has queued. A +/// stream's own priority orders it only against the group's other streams. Streams outside any +/// group go before every group. See [`moq_quic::SendGroups`]. +/// +/// Clones share one group, released once the last clone is dropped and its streams finish. A +/// group does not keep its connection open. +#[derive(Debug, Clone)] +pub struct SendGroup(Arc); + +#[derive(Debug)] +struct Inner { + conn: Arc, + group: Option, +} + +impl SendGroup { + pub(crate) fn new(conn: Arc, priority: i64) -> Self { + let group = conn + .state + .lock("SendGroup::new") + .inner + .send_groups() + .open(priority, None); + Self(Arc::new(Inner { + conn, + group: Some(group), + })) + } + + /// Change the group's priority, moving any data it has queued along with it + pub fn set_priority(&self, priority: i64) { + let mut conn = self.0.conn.state.lock("SendGroup::set_priority"); + conn.inner.send_groups().set_priority(self.id(), priority); + } + + /// Get the group's priority + pub fn priority(&self) -> i64 { + let mut conn = self.0.conn.state.lock("SendGroup::priority"); + conn.inner.send_groups().priority(self.id()) + } + + /// The group as the connection knows it, which refuses it on any other connection. + pub(crate) fn id(&self) -> &moq_quic::SendGroup { + self.0.group.as_ref().expect("only taken on drop") + } +} + +impl Drop for Inner { + fn drop(&mut self) { + if let Some(group) = self.group.take() { + let mut conn = self.conn.state.lock("SendGroup::drop"); + conn.inner.send_groups().close(group); + } + } +} diff --git a/rs/moq-tokio/src/quic/driver/send_stream.rs b/rs/moq-tokio/src/quic/driver/send_stream.rs index 7701d6817d2..63401220104 100644 --- a/rs/moq-tokio/src/quic/driver/send_stream.rs +++ b/rs/moq-tokio/src/quic/driver/send_stream.rs @@ -10,7 +10,7 @@ use moq_quic::{ClosedStream, ConnectionError, FinishError, StreamId, Written}; use thiserror::Error; use crate::quic::driver::{ - VarInt, + SendGroup, VarInt, connection::{ConnectionRef, State}, }; @@ -225,25 +225,36 @@ impl SendStream { } } - /// Set the priority of the send stream + /// Set the priority within the stream's send group, or session-wide when ungrouped /// /// Every send stream has an initial priority of 0. Locally buffered data from streams with /// higher priority will be transmitted before data from streams with lower priority. Changing /// the priority of a stream with pending data may only take effect after that data has been /// transmitted. Using many different priority levels per connection may have a negative /// impact on performance. - pub fn set_priority(&self, priority: i32) -> Result<(), ClosedStream> { + pub fn set_priority(&self, priority: i64) -> Result<(), ClosedStream> { let mut conn = self.conn.state.lock("SendStream::set_priority"); conn.inner.send_stream(self.stream).set_priority(priority)?; Ok(()) } /// Get the priority of the send stream - pub fn priority(&self) -> Result { + pub fn priority(&self) -> Result { let mut conn = self.conn.state.lock("SendStream::priority"); conn.inner.send_stream(self.stream).priority() } + /// Move the stream into a send group, or with `None` back to the ungrouped streams, which + /// go before every group + /// + /// # Panics + /// - when `group` belongs to another connection + pub fn set_group(&self, group: Option<&SendGroup>) -> Result<(), ClosedStream> { + let mut conn = self.conn.state.lock("SendStream::set_group"); + let group = group.map(SendGroup::id); + conn.inner.send_stream(self.stream).set_group(group) + } + /// Completes when the peer stops the stream or reads the stream to completion /// /// Yields `Some` with the stop error code if the peer stops the stream. Yields `None` if the diff --git a/rs/moq-tokio/src/quic/web/send.rs b/rs/moq-tokio/src/quic/web/send.rs index a6c040a045f..3c807c0fc7a 100644 --- a/rs/moq-tokio/src/quic/web/send.rs +++ b/rs/moq-tokio/src/quic/web/send.rs @@ -104,14 +104,22 @@ impl SendStream { self.stream.finish().map_err(Into::into) } - pub fn set_priority(&self, order: i32) -> Result<(), ClosedStream> { - self.stream.set_priority(order).map_err(Into::into) + /// Set the priority within the stream's send group, or session-wide when ungrouped. + pub fn set_priority(&self, priority: i64) -> Result<(), ClosedStream> { + self.stream.set_priority(priority).map_err(Into::into) } - pub fn priority(&self) -> Result { + /// Get the stream's priority. See [`crate::quic::driver::SendStream::priority`]. + pub fn priority(&self) -> Result { self.stream.priority().map_err(Into::into) } + /// Move the stream into a send group, or with `None` back out. See + /// [`crate::quic::driver::SendStream::set_group`]. + pub fn set_group(&self, group: Option<&crate::quic::driver::SendGroup>) -> Result<(), ClosedStream> { + self.stream.set_group(group).map_err(Into::into) + } + /// Return the underlying QUIC stream ID. /// /// > **Warning** @@ -144,7 +152,7 @@ impl web_transport_trait::SendStream for SendStream { type Error = WriteError; fn set_priority(&mut self, order: i32) { - Self::set_priority(self, order).ok(); + Self::set_priority(self, order.into()).ok(); } fn reset(&mut self, code: u32) { diff --git a/rs/moq-tokio/src/quic/web/session.rs b/rs/moq-tokio/src/quic/web/session.rs index c84ecfc0305..0fdcb2fcc9d 100644 --- a/rs/moq-tokio/src/quic/web/session.rs +++ b/rs/moq-tokio/src/quic/web/session.rs @@ -269,7 +269,7 @@ impl Session { // Otherwise the application could write data with lower priority than the header, resulting // in queuing. Also the header is very important for determining the session ID // without reliable reset. - send.set_priority(i32::MAX).ok(); + send.set_priority(i64::MAX).ok(); Self::write_full(&mut send, &self.header_uni) .await .map_err(|e| self.map_error(e))?; @@ -288,7 +288,7 @@ impl Session { // Otherwise the application could write data with lower priority than the header, resulting // in queuing. Also the header is very important for determining the session ID // without reliable reset. - send.set_priority(i32::MAX).ok(); + send.set_priority(i64::MAX).ok(); Self::write_full(&mut send, &self.header_bi) .await .map_err(|e| self.map_error(e))?; @@ -372,6 +372,14 @@ impl Session { Ok(()) } + /// Open a send group at `priority`, for streams to join with [`SendStream::set_group`]. + /// + /// See [`crate::quic::driver::SendGroup`]. The HTTP/3 streams stay outside every group, + /// so they go first. + pub fn send_group(&self, priority: i64) -> crate::quic::driver::SendGroup { + self.conn.send_group(priority) + } + /// Computes the maximum size of datagrams that may be passed to /// [`send_datagram`](Self::send_datagram), or 0 when the peer does not accept datagrams. pub fn max_datagram_size(&self) -> usize { diff --git a/rs/moq-tokio/src/server.rs b/rs/moq-tokio/src/server.rs index 44bcb1893e6..a46c5e425c2 100644 --- a/rs/moq-tokio/src/server.rs +++ b/rs/moq-tokio/src/server.rs @@ -966,7 +966,7 @@ async fn accept_request( deadline: Option, ) -> crate::Result<(PendingRequest>, Option)> where - S: web_transport_trait::Session, + S: crate::transport::Grouped, crate::transport::Session: moq_net::transport::poll::Boxable, as moq_net::transport::poll::Session>::SendStream: moq_net::transport::MaybeSync, as moq_net::transport::poll::Session>::RecvStream: moq_net::transport::MaybeSync, @@ -2178,6 +2178,18 @@ mod tests { } } + #[cfg(feature = "websocket")] + impl crate::transport::GroupedSend for ScriptedSend { + type Group = moq_net::transport::poll::Ungrouped; + + fn set_group(&mut self, group: &Self::Group) { + match *group {} + } + } + + #[cfg(feature = "websocket")] + impl crate::transport::Grouped for Scripted {} + /// Delivers `data` once, then nothing more. #[cfg(feature = "websocket")] struct ScriptedRecv { diff --git a/rs/moq-tokio/src/transport.rs b/rs/moq-tokio/src/transport.rs index 5ab97f52a9e..06e6b0652e3 100644 --- a/rs/moq-tokio/src/transport.rs +++ b/rs/moq-tokio/src/transport.rs @@ -16,7 +16,9 @@ use bytes::Bytes; use futures::FutureExt; use web_transport_trait::poll as wt_poll; +mod grouped; mod owned; +pub(crate) use grouped::{Grouped, GroupedSend}; pub use owned::Error; /// The network transport carrying a MoQ session, on either side of it. @@ -157,7 +159,7 @@ impl Clone for Session { impl wt_poll::Session for Session where - S: web_transport_trait::Session, + S: Grouped, S::SendStream: 'static, S::RecvStream: 'static, { @@ -242,13 +244,14 @@ where /// The send half produced by [`Session`]: an async-interface stream adapted to the /// poll interface. See [`Session`] for the deferred `finish`/`reset` behavior. -pub struct SendStream { +pub struct SendStream { // Same never-locked Mutex as [`Session`]'s slots: `get_mut` is free on an // exclusive borrow, and the wrapper keeps the boxed future from stripping // `Sync` off the containing session types. state: std::sync::Mutex>>, // Deferred actions, applied when the in-flight operation settles. - priority: Option, + group: Option, + priority: Option, finish: bool, reset: Option, /// Bytes already transmitted but not yet reported to the caller: a write the @@ -266,7 +269,7 @@ pub struct SendStream { interrupt: Option>, } -enum SendState { +enum SendState { Idle(S), /// A write in flight; `chunk` is the copied bytes, kept (refcounted, no /// extra copy) so a completion absorbed by `poll_closed` can verify the @@ -282,11 +285,12 @@ enum SendState { Closing(#[allow(clippy::type_complexity)] OpBox<(S, Option>)>), } -impl SendStream { +impl SendStream { /// Wrap an async-interface send stream. pub fn new(stream: S) -> Self { Self { state: std::sync::Mutex::new(Some(SendState::Idle(stream))), + group: None, priority: None, finish: false, reset: None, @@ -297,8 +301,11 @@ impl SendStream { /// Apply actions deferred while an operation was in flight. fn settle(&mut self, stream: &mut S) { - if let Some(order) = self.priority.take() { - stream.set_priority(order); + if let Some(group) = self.group.take() { + stream.set_group(&group); + } + if let Some(priority) = self.priority.take() { + GroupedSend::set_priority(stream, priority); } if let Some(code) = self.reset.take() { self.finish = false; @@ -312,7 +319,42 @@ impl SendStream { } } -impl wt_poll::SendStream for SendStream { +impl SendStream { + /// Set the full-width priority, deferred while an operation holds the stream. + fn prioritize(&mut self, priority: i64) { + match self.state.get_mut().unwrap().as_mut() { + Some(SendState::Idle(stream)) => GroupedSend::set_priority(stream, priority), + // A finished stream still retransmits under its priority until the FIN is + // acknowledged, so reclaim it from the watch rather than wait that out. + Some(SendState::Closing(_)) => { + self.priority = Some(priority); + self.reclaim(); + } + _ => self.priority = Some(priority), + } + } + + /// Join `group`, deferred like [`prioritize`](Self::prioritize). + fn join(&mut self, group: &S::Group) { + match self.state.get_mut().unwrap().as_mut() { + Some(SendState::Idle(stream)) => stream.set_group(group), + Some(SendState::Closing(_)) => { + self.group = Some(group.clone()); + self.reclaim(); + } + _ => self.group = Some(group.clone()), + } + } + + /// Interrupt the closed() watch so a deferred action applies now. + fn reclaim(&mut self) { + if let Some(tx) = self.interrupt.take() { + let _ = tx.send(()); + } + } +} + +impl wt_poll::SendStream for SendStream { type Error = S::Error; fn poll_write(&mut self, cx: &mut Context<'_>, buf: &[u8]) -> Poll> { @@ -411,18 +453,7 @@ impl wt_poll::SendStream for SendS } fn set_priority(&mut self, order: i32) { - match self.state.get_mut().unwrap().as_mut() { - Some(SendState::Idle(stream)) => stream.set_priority(order), - // A finished stream still retransmits under its priority until the FIN is - // acknowledged, so reclaim it from the watch rather than wait that out. - Some(SendState::Closing(_)) => { - self.priority = Some(order); - if let Some(tx) = self.interrupt.take() { - let _ = tx.send(()); - } - } - _ => self.priority = Some(order), - } + self.prioritize(order.into()); } fn finish(&mut self) -> Result<(), Self::Error> { @@ -527,7 +558,7 @@ impl wt_poll::SendStream for SendS } } -impl Drop for SendStream { +impl Drop for SendStream { fn drop(&mut self) { match self.state.get_mut().unwrap().take() { // Apply a deferred reset if the stream is idle. @@ -823,6 +854,58 @@ impl Drop for RecvStream { } } +#[cfg(feature = "quic")] +impl Grouped for crate::quic::web::Session { + fn send_group(&self, priority: i64) -> Option { + Some(Self::send_group(self, priority)) + } +} + +#[cfg(feature = "quic")] +impl GroupedSend for crate::quic::web::SendStream { + type Group = crate::quic::driver::SendGroup; + + fn set_group(&mut self, group: &Self::Group) { + // A closed stream has nothing left to schedule. + Self::set_group(self, Some(group)).ok(); + } + + fn set_priority(&mut self, priority: i64) { + crate::quic::web::SendStream::set_priority(self, priority).ok(); + } +} + +#[cfg(feature = "quic")] +impl moq_net::transport::poll::SendGroup for crate::quic::driver::SendGroup { + fn set_priority(&mut self, priority: i64) { + Self::set_priority(self, priority) + } +} + +#[cfg(feature = "iroh")] +impl Grouped for web_transport_iroh::Session {} + +#[cfg(feature = "iroh")] +impl GroupedSend for web_transport_iroh::SendStream { + type Group = moq_net::transport::poll::Ungrouped; + + fn set_group(&mut self, group: &Self::Group) { + match *group {} + } +} + +#[cfg(any(feature = "websocket", feature = "tcp"))] +impl Grouped for qmux::Session {} + +#[cfg(any(feature = "websocket", feature = "tcp"))] +impl GroupedSend for qmux::SendStream { + type Group = moq_net::transport::poll::Ungrouped; + + fn set_group(&mut self, group: &Self::Group) { + match *group {} + } +} + #[cfg(test)] mod tests { use std::sync::{ @@ -857,6 +940,7 @@ mod tests { writes: Arc>>, blocked: Arc, priorities: Arc>>, + groups: Arc>>, finished: Arc, resets: Arc>>, /// The peer never acknowledges the FIN: closed() stays pending forever. @@ -907,6 +991,22 @@ mod tests { } } + /// A send group that only names itself, so a test can see which one a stream joined. + #[derive(Clone)] + struct FakeGroup(u32); + + impl moq_net::transport::poll::SendGroup for FakeGroup { + fn set_priority(&mut self, _: i64) {} + } + + impl GroupedSend for FakeSend { + type Group = FakeGroup; + + fn set_group(&mut self, group: &FakeGroup) { + self.groups.lock().unwrap().push(group.0); + } + } + /// An async-interface receive stream replaying canned chunks, then EOF. struct FakeRecv { chunks: std::collections::VecDeque, @@ -1008,6 +1108,34 @@ mod tests { assert_eq!(fake.priorities.lock().unwrap().as_slice(), &[7]); } + // A group joined while a write holds the stream applies when it settles, and one + // joined during the FIN watch reclaims the stream, which still retransmits. + #[test] + fn send_defers_a_group_across_an_inflight_write() { + use moq_net::transport::poll; + let fake = FakeSend::default(); + fake.blocked.store(true, Ordering::SeqCst); + let mut send = SendStream::new(fake.clone()); + let mut cx = cx(); + + assert!(send.poll_write(&mut cx, b"queued").is_pending()); + poll::SendStream::set_group(&mut send, &FakeGroup(1)); + poll::SendStream::set_priority(&mut send, i64::MAX); + assert!(fake.groups.lock().unwrap().is_empty(), "deferred while in flight"); + + fake.blocked.store(false, Ordering::SeqCst); + assert_eq!(send.poll_write(&mut cx, b"queued"), Poll::Ready(Ok(6))); + assert_eq!(fake.groups.lock().unwrap().as_slice(), &[1]); + assert_eq!(fake.priorities.lock().unwrap().as_slice(), &[i32::MAX], "saturated"); + + fake.never_ack.store(true, Ordering::SeqCst); + send.finish().unwrap(); + assert!(send.poll_closed(&mut cx).is_pending()); + poll::SendStream::set_group(&mut send, &FakeGroup(2)); + assert!(send.poll_closed(&mut cx).is_pending()); + assert_eq!(fake.groups.lock().unwrap().as_slice(), &[1, 2]); + } + // The cluster-failover corruption: a write held Pending by backpressure is // driven to completion by the peer-close check (poll_closed). Its result must // reach the retrying poll_write; discarding it made the retry put the same @@ -1282,6 +1410,8 @@ mod tests { #[derive(Clone)] struct FakeSession; + impl Grouped for FakeSession {} + impl web_transport_trait::Session for FakeSession { type SendStream = FakeSend; type RecvStream = FakeRecv; diff --git a/rs/moq-tokio/src/transport/grouped.rs b/rs/moq-tokio/src/transport/grouped.rs new file mode 100644 index 00000000000..acdf01cddf8 --- /dev/null +++ b/rs/moq-tokio/src/transport/grouped.rs @@ -0,0 +1,34 @@ +//! Send-group hooks for the async backends. +//! +//! The traits are `pub` in this private module, so other crates can neither name nor +//! implement them: `pub(crate)` is refused (E0446) because the public adapter's +//! `type Group` projects through [`GroupedSend`]. + +/// A backend session whose send streams may join send groups, as native QUIC's do. +/// +/// The async interface has no groups, so moq-net's +/// [`send_group`](moq_net::transport::poll::Session::send_group) reaches a backend through +/// this. A backend without groups implements it empty. +pub trait Grouped: web_transport_trait::Session { + /// Open a send group at `priority`, or `None` when the backend has none. + fn send_group(&self, priority: i64) -> Option<::Group> { + let _ = priority; + None + } +} + +/// A backend send stream that may join its [`Grouped`] session's send groups. +pub trait GroupedSend: web_transport_trait::SendStream { + /// The backend's send group, [`Ungrouped`](moq_net::transport::poll::Ungrouped) when it has none. + type Group: moq_net::transport::poll::SendGroup + Send + Sync; + + /// Move the stream into `group`. + fn set_group(&mut self, group: &Self::Group); + + /// Set the priority within the stream's send group, or session-wide when ungrouped. + /// + /// Saturated into the async interface's narrower priority by default. + fn set_priority(&mut self, priority: i64) { + web_transport_trait::SendStream::set_priority(self, priority.clamp(i32::MIN.into(), i32::MAX.into()) as i32); + } +} diff --git a/rs/moq-tokio/src/transport/owned.rs b/rs/moq-tokio/src/transport/owned.rs index 44d98ef0d7e..0c322037389 100644 --- a/rs/moq-tokio/src/transport/owned.rs +++ b/rs/moq-tokio/src/transport/owned.rs @@ -44,7 +44,7 @@ impl moq::Stats for Stats { self.0.estimated_send_rate() } } -impl> poll::Session for super::Session { +impl> poll::Session for super::Session { type SendStream = super::SendStream; type RecvStream = super::RecvStream; type Error = Error; @@ -60,6 +60,9 @@ impl> poll::Session for fn poll_open_bi(&mut self, cx: &mut Context<'_>) -> Poll, Self::Error>> { backend::Session::poll_open_bi(self, cx).map_err(Error) } + fn send_group(&mut self, priority: i64) -> Option<::Group> { + self.session.send_group(priority) + } fn poll_send_datagram(&mut self, cx: &mut Context<'_>, payload: &[u8]) -> Poll> { backend::Session::poll_send_datagram(self, cx, payload).map_err(Error) } @@ -85,16 +88,20 @@ impl> poll::Session for Stats(backend::Session::stats(self)) } } -impl poll::SendStream for super::SendStream { +impl poll::SendStream for super::SendStream { type Error = Error; + type Group = S::Group; fn poll_write(&mut self, cx: &mut Context<'_>, buf: &[u8]) -> Poll> { backend::SendStream::poll_write(self, cx, buf).map_err(Error) } fn poll_write_buf(&mut self, cx: &mut Context<'_>, buf: &mut B) -> Poll> { backend::SendStream::poll_write_buf(self, cx, buf).map_err(Error) } - fn set_priority(&mut self, order: i32) { - backend::SendStream::set_priority(self, order) + fn set_priority(&mut self, priority: i64) { + self.prioritize(priority) + } + fn set_group(&mut self, group: &Self::Group) { + self.join(group) } fn finish(&mut self) -> Result<(), Self::Error> { backend::SendStream::finish(self).map_err(Error) diff --git a/rs/moq-uring/src/quic/driver/connection.rs b/rs/moq-uring/src/quic/driver/connection.rs index 2b7c834b89e..6f3721eef9d 100644 --- a/rs/moq-uring/src/quic/driver/connection.rs +++ b/rs/moq-uring/src/quic/driver/connection.rs @@ -315,6 +315,12 @@ impl Connection { &self.shared.owner } + /// Open a send group at `priority`, for streams to join with + /// [`SendStream::set_group`](super::SendStream::set_group). + pub fn send_group(&self, priority: i64) -> super::SendGroup { + super::SendGroup::new(self.shared.clone(), priority) + } + /// The peer's certificate chain in DER, leaf first, or `None` if it /// presented none. /// diff --git a/rs/moq-uring/src/quic/driver/group.rs b/rs/moq-uring/src/quic/driver/group.rs new file mode 100644 index 00000000000..a90b1e2e97c --- /dev/null +++ b/rs/moq-uring/src/quic/driver/group.rs @@ -0,0 +1,66 @@ +//! Send groups: a shared handle around one of moq-quic's. + +use std::rc::Rc; + +use super::Shared; + +/// A send group on a [`Connection`](super::Connection). +/// +/// Streams that [join](super::SendStream::set_group) a group share bytes with the +/// connection's other groups: a higher priority group preempts every lower one, and groups of +/// equal priority get an equal share of bytes however many streams each has queued. A stream's +/// own priority orders it only against the group's other streams. Streams outside any group go +/// before every group. See [`moq_quic::SendGroups`]. +/// +/// Clones share one group, released once the last clone is dropped and its streams finish. +#[derive(Clone)] +pub struct SendGroup(Rc); + +struct Inner { + shared: Shared, + group: Option, +} + +impl SendGroup { + pub(crate) fn new(shared: Shared, priority: i64) -> Self { + let group = shared.conn.borrow_mut().send_groups().open(priority, None); + Self(Rc::new(Inner { + shared, + group: Some(group), + })) + } + + /// Change the group's priority, moving any data it has queued along with it. + pub fn set_priority(&self, priority: i64) { + self.0 + .shared + .conn + .borrow_mut() + .send_groups() + .set_priority(self.id(), priority); + } + + /// The group's priority. + pub fn priority(&self) -> i64 { + self.0.shared.conn.borrow_mut().send_groups().priority(self.id()) + } + + /// The group as the connection knows it, which refuses it on any other connection. + pub(crate) fn id(&self) -> &moq_quic::SendGroup { + self.0.group.as_ref().expect("only taken on drop") + } +} + +impl Drop for Inner { + fn drop(&mut self) { + if let Some(group) = self.group.take() { + self.shared.conn.borrow_mut().send_groups().close(group); + } + } +} + +impl std::fmt::Debug for SendGroup { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("SendGroup").finish_non_exhaustive() + } +} diff --git a/rs/moq-uring/src/quic/driver/mod.rs b/rs/moq-uring/src/quic/driver/mod.rs index c0a2d1bbe52..1a4b7ca578f 100644 --- a/rs/moq-uring/src/quic/driver/mod.rs +++ b/rs/moq-uring/src/quic/driver/mod.rs @@ -11,10 +11,12 @@ mod connection; mod endpoint; +mod group; mod stream; pub use connection::Connection; pub use endpoint::Endpoint; +pub use group::SendGroup; pub use stream::{Credit, RecvStream, SendStream}; pub(crate) use connection::{End, Shared}; diff --git a/rs/moq-uring/src/quic/driver/stream.rs b/rs/moq-uring/src/quic/driver/stream.rs index d6150c51159..06baf6a7634 100644 --- a/rs/moq-uring/src/quic/driver/stream.rs +++ b/rs/moq-uring/src/quic/driver/stream.rs @@ -56,6 +56,28 @@ impl SendStream { self.shared.conn.borrow_mut().send_stream(self.id).checkpoint() } + /// Set the priority within the stream's send group, or session-wide when ungrouped. + /// + /// Higher goes first. + pub fn set_priority(&mut self, priority: i64) { + let _ = self + .shared + .conn + .borrow_mut() + .send_stream(self.id) + .set_priority(priority); + } + + /// Move the stream into a send group, or with `None` back to the + /// ungrouped streams, which go before every group. + /// + /// # Panics + /// When `group` belongs to another connection. + pub fn set_group(&mut self, group: Option<&super::SendGroup>) { + let group = group.map(super::SendGroup::id); + let _ = self.shared.conn.borrow_mut().send_stream(self.id).set_group(group); + } + /// [`reset`](web_transport_trait::poll::SendStream::reset) with a /// full-width code, for the WebTransport HTTP/3 error mapping, that still /// delivers the data before `checkpoint`, one this stream took. @@ -109,7 +131,7 @@ impl web_transport_trait::poll::SendStream for SendStream { fn set_priority(&mut self, order: i32) { // The trait (like W3C sendOrder) sends HIGHER values first, and so // does moq-quic. - let _ = self.shared.conn.borrow_mut().send_stream(self.id).set_priority(order); + SendStream::set_priority(self, order.into()); } fn finish(&mut self) -> Result<(), Self::Error> { diff --git a/rs/moq-uring/src/quic/mod.rs b/rs/moq-uring/src/quic/mod.rs index b8ccd230f33..55c02f2031c 100644 --- a/rs/moq-uring/src/quic/mod.rs +++ b/rs/moq-uring/src/quic/mod.rs @@ -30,7 +30,7 @@ pub mod web; mod driver; -pub use driver::{Connection, Credit, RecvStream, SendStream}; +pub use driver::{Connection, Credit, RecvStream, SendGroup, SendStream}; pub use endpoint::Endpoint; /// The QUIC payload size every full datagram in a GSO train uses, and the diff --git a/rs/moq-uring/src/quic/web.rs b/rs/moq-uring/src/quic/web.rs index b6854e5b715..02222e8c6e7 100644 --- a/rs/moq-uring/src/quic/web.rs +++ b/rs/moq-uring/src/quic/web.rs @@ -443,6 +443,13 @@ pub struct Session { } impl Session { + /// Open a send group at `priority`, for streams to join with + /// [`SendStream::set_group`]. The HTTP/3 streams stay outside every + /// group, so they go first. + pub fn send_group(&self, priority: i64) -> super::SendGroup { + self.conn.send_group(priority) + } + /// Wrap a raw-QUIC connection in a session, with no /// WebTransport layering: streams and datagrams pass through untouched. pub fn raw(conn: Connection) -> Self { @@ -822,6 +829,17 @@ impl SendStream { } } + /// Set the priority within the stream's send group, or session-wide when ungrouped. + pub fn set_priority(&mut self, priority: i64) { + self.inner.set_priority(priority); + } + + /// Move the stream into a send group, or with `None` back out. See + /// [`super::SendStream::set_group`]. + pub fn set_group(&mut self, group: Option<&super::SendGroup>) { + self.inner.set_group(group); + } + /// Write the WebTransport header of a stream we opened, checkpointing it /// once it is all written. fn poll_header(&mut self, cx: &mut Context<'_>, header: &mut Bytes) -> Poll> { @@ -866,7 +884,7 @@ impl web_transport_trait::poll::SendStream for SendStream { } fn set_priority(&mut self, order: i32) { - web_transport_trait::poll::SendStream::set_priority(&mut self.inner, order); + self.inner.set_priority(order.into()); } fn finish(&mut self) -> Result<(), Self::Error> { diff --git a/rs/moq-uring/src/transport.rs b/rs/moq-uring/src/transport.rs index 6a26f1d66b7..f4492304b33 100644 --- a/rs/moq-uring/src/transport.rs +++ b/rs/moq-uring/src/transport.rs @@ -1,3 +1,30 @@ //! Adapt backend poll sessions to moq-net without changing thread affinity. mod adapter; pub use adapter::{Error, RecvStream, SendStream, Session}; +use adapter::{Grouped, GroupedSend}; + +use crate::quic; + +impl Grouped for quic::web::Session { + fn send_group(&mut self, priority: i64) -> Option { + Some(quic::web::Session::send_group(self, priority)) + } +} + +impl GroupedSend for quic::web::SendStream { + type Group = quic::SendGroup; + + fn set_group(&mut self, group: &Self::Group) { + quic::web::SendStream::set_group(self, Some(group)); + } + + fn set_priority(&mut self, priority: i64) { + quic::web::SendStream::set_priority(self, priority); + } +} + +impl moq_net::transport::poll::SendGroup for quic::SendGroup { + fn set_priority(&mut self, priority: i64) { + quic::SendGroup::set_priority(self, priority) + } +} diff --git a/rs/moq-uring/src/transport/adapter.rs b/rs/moq-uring/src/transport/adapter.rs index 0a751d76718..a09b5e417a8 100644 --- a/rs/moq-uring/src/transport/adapter.rs +++ b/rs/moq-uring/src/transport/adapter.rs @@ -44,6 +44,39 @@ impl moq::Stats for Stats { self.0.estimated_send_rate() } } +/// A backend session whose send streams may join send groups, as native QUIC's do. +/// +/// `pub` only because `pub(crate)` is refused (E0446): the adapter's `type Group` projects +/// through [`GroupedSend`]. The module is private and the trait is not re-exported, so +/// other crates can neither name nor implement it. +/// +/// The backend interface has no groups, so moq-net's +/// [`send_group`](poll::Session::send_group) reaches a backend through this. A backend +/// without groups implements it empty. +pub trait Grouped: backend::Session { + /// Open a send group at `priority`, or `None` when the backend has none. + fn send_group(&mut self, priority: i64) -> Option<::Group> { + let _ = priority; + None + } +} + +/// A backend send stream that may join its [`Grouped`] session's send groups. +pub trait GroupedSend: backend::SendStream { + /// The backend's send group, [`Ungrouped`](poll::Ungrouped) when it has none. + type Group: poll::SendGroup; + + /// Move the stream into `group`. + fn set_group(&mut self, group: &Self::Group); + + /// Set the priority within the stream's send group, or session-wide when ungrouped. + /// + /// Saturated into the backend's narrower priority by default. + fn set_priority(&mut self, priority: i64) { + backend::SendStream::set_priority(self, priority.clamp(i32::MIN.into(), i32::MAX.into()) as i32); + } +} + /// A WebTransport session adapted to moq-net. #[derive(Clone)] pub struct Session(S, moq::Binding); @@ -53,7 +86,7 @@ impl Session { Self(session, binding) } } -impl + Clone + 'static> poll::Session for Session { +impl + Clone + 'static> poll::Session for Session { type SendStream = SendStream; type RecvStream = RecvStream; type Error = Error; @@ -77,6 +110,9 @@ impl + Clone + 'st .map_err(Error) .map(|res| res.map(|(send, recv)| (SendStream(send), RecvStream(recv)))) } + fn send_group(&mut self, priority: i64) -> Option<::Group> { + Grouped::send_group(&mut self.0, priority) + } fn poll_send_datagram(&mut self, cx: &mut Context<'_>, payload: &[u8]) -> Poll> { backend::Session::poll_send_datagram(&mut self.0, cx, payload).map_err(Error) } @@ -104,16 +140,20 @@ impl + Clone + 'st } /// An outgoing WebTransport stream adapted to moq-net. pub struct SendStream(S); -impl poll::SendStream for SendStream { +impl poll::SendStream for SendStream { type Error = Error; + type Group = S::Group; fn poll_write(&mut self, cx: &mut Context<'_>, buf: &[u8]) -> Poll> { backend::SendStream::poll_write(&mut self.0, cx, buf).map_err(Error) } fn poll_write_buf(&mut self, cx: &mut Context<'_>, buf: &mut B) -> Poll> { backend::SendStream::poll_write_buf(&mut self.0, cx, buf).map_err(Error) } - fn set_priority(&mut self, order: i32) { - backend::SendStream::set_priority(&mut self.0, order) + fn set_priority(&mut self, priority: i64) { + GroupedSend::set_priority(&mut self.0, priority) + } + fn set_group(&mut self, group: &Self::Group) { + GroupedSend::set_group(&mut self.0, group) } fn finish(&mut self) -> Result<(), Self::Error> { backend::SendStream::finish(&mut self.0).map_err(Error) @@ -203,6 +243,13 @@ mod tests { Poll::Ready(Err(Failure)) } } + impl GroupedSend for Stream { + type Group = poll::Ungrouped; + fn set_group(&mut self, group: &Self::Group) { + match *group {} + } + } + impl Grouped for Backend {} struct Sample; impl wt::Stats for Sample { fn rtt(&self) -> Option { diff --git a/rs/moq-wasm/src/transport/adapter.rs b/rs/moq-wasm/src/transport/adapter.rs index 0ebe75b7b14..bb2e27bfdfb 100644 --- a/rs/moq-wasm/src/transport/adapter.rs +++ b/rs/moq-wasm/src/transport/adapter.rs @@ -77,6 +77,9 @@ impl + Clone + 'st .map_err(Error) .map(|res| res.map(|(send, recv)| (SendStream(send), RecvStream(recv)))) } + fn send_group(&mut self, _: i64) -> Option { + None + } fn poll_send_datagram(&mut self, cx: &mut Context<'_>, payload: &[u8]) -> Poll> { backend::Session::poll_send_datagram(&mut self.0, cx, payload).map_err(Error) } @@ -106,15 +109,20 @@ impl + Clone + 'st pub struct SendStream(S); impl poll::SendStream for SendStream { type Error = Error; + type Group = poll::Ungrouped; fn poll_write(&mut self, cx: &mut Context<'_>, buf: &[u8]) -> Poll> { backend::SendStream::poll_write(&mut self.0, cx, buf).map_err(Error) } fn poll_write_buf(&mut self, cx: &mut Context<'_>, buf: &mut B) -> Poll> { backend::SendStream::poll_write_buf(&mut self.0, cx, buf).map_err(Error) } - fn set_priority(&mut self, order: i32) { + fn set_priority(&mut self, order: i64) { + let order = order.clamp(i32::MIN.into(), i32::MAX.into()) as i32; backend::SendStream::set_priority(&mut self.0, order) } + fn set_group(&mut self, group: &Self::Group) { + match *group {} + } fn finish(&mut self) -> Result<(), Self::Error> { backend::SendStream::finish(&mut self.0).map_err(Error) }