From b82cff679cbf7454c8571d4e133699540e43aae2 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 22 Oct 2026 09:51:04 -0700 Subject: [PATCH 1/3] quest: plan qmux receive leases and the pre-draft-20 join after a quick finish Two m2 follow-ups: qmux leasing receive memory natively after #5878, weighing the per-session window cost, and the publisher answering a joining FETCH that arrives after its subscription finished, after #6348. Co-Authored-By: Claude Opus 5.5 --- quest/m2/README.md | 2 + quest/m2/ietf-join-after-finish.md | 74 ++++++++++++++++++++++++++++++ quest/m2/qmux-receive-lease.md | 73 +++++++++++++++++++++++++++++ 3 files changed, 149 insertions(+) create mode 100644 quest/m2/ietf-join-after-finish.md create mode 100644 quest/m2/qmux-receive-lease.md diff --git a/quest/m2/README.md b/quest/m2/README.md index c6dbabf0107..7517b7118cb 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -103,6 +103,7 @@ ranked just below deterministic TS legs, which it requires. - [Bindings in the interop matrix](/quest/m2/interop-bindings.md) - Kotlin, Swift, and Dart publish and subscribe against every other interop client - [WebGPU HDR](/quest/m2/webgpu-hdr.md) - HDR renditions play as HDR where the browser and display can show it, and tone-map to SDR elsewhere - [One port](/quest/m2/one-port/README.md) - one UDP port carries QUIC, SRT, and WebRTC media through an embedder hook, and one TCP port carries HTTP, RTMP, and RTMPS +- [qmux receive leases](/quest/m2/qmux-receive-lease.md) - WebSocket, TCP, and WebRTC sessions bound incomplete messages by their receive window, at a per-session cost the maintainer picks - [QUIC caps](/quest/m2/rate-quic.md) - paced MAX_DATA credit in and a capped pacer out hold a QUIC session to its token's bitrate - [WebSocket caps](/quest/m2/rate-websocket.md) - paced reads and writes over bounded socket buffers hold WebSocket to the same caps - [Publishers learn their cap](/quest/m2/rate-grant.md) - the AUTH grant carries the caps and publishers clamp their encoder to them @@ -195,6 +196,7 @@ ranked just below deterministic TS legs, which it requires. - [Relay metrics through prometheus-client](/quest/m2/relay-prometheus-client.md) - `/metrics` renders through the maintained client with the same series - [IETF request order](/quest/m2/ietf-request-order.md) - d14/15 requests take their ID after opening, legacy `control.sending()` races are gone, and SUBSCRIBE_NAMESPACE never closes over its own close - [JS IETF request streams](/quest/m2/js-request-stream.md) - `RequestStream` owns deadlines and cancellation, and TRACK trailing bytes close the session like Rust +- [Join after a quick finish](/quest/m2/ietf-join-after-finish.md) - a pre-draft-20 joining FETCH that arrives after its subscription finished is still answered, within a bound the peer already negotiates - [Error cleanups](/quest/m2/error-cleanups.md) - delete the unused `Error::SessionClosed`; raw transport failures already map to `Unroutable` - [FromStr parsers](/quest/m2/from-str-trait.md) - hang, moq-auth, and moq-msf implement `FromStr` instead of inherent `from_str` - [Drop dead items](/quest/m2/drop-dead-items.md) - the audit's uncalled public items and test helpers are removed diff --git a/quest/m2/ietf-join-after-finish.md b/quest/m2/ietf-join-after-finish.md new file mode 100644 index 00000000000..f8242f6da43 --- /dev/null +++ b/quest/m2/ietf-join-after-finish.md @@ -0,0 +1,74 @@ +# [M] A quick finish keeps its pre-draft-20 join + +## Goal + +On drafts 14 to 19, a joining FETCH that reaches our publisher after its +subscription finished cleanly is still answered from the subscription's +saved Largest group. A track that ends within about one RTT of SUBSCRIBE_OK +then finishes cleanly at the subscriber, head included, instead of failing. +What the publisher retains for this is bounded by design (the session, the +subscription's request, or a limit the peer already negotiates), never by a +timer constant. + +## Plan + +Starts after [moq-dev/moq#6348](https://github.com/moq-dev/moq/pull/6348) +merges (open on 2026-10-22). It makes every pre-draft-20 join a SUBSCRIBE +followed, after SUBSCRIBE_OK, by a `RelativeJoining{0}` FETCH, and fails the +subscription when that FETCH is refused. If it is abandoned, re-judge this +quest. + +State: `rs/moq-net/src/ietf/publisher.rs` registers a `Join` entry in +`Publisher::joins` when it dispatches the SUBSCRIBE, fills it with a `Joined` +snapshot as SUBSCRIBE_OK goes out, and drops it when the subscription ends, +before PUBLISH_DONE. A joining FETCH that arrives later finds no entry, waits +out the registration wait in `joined()`, and is refused as an unknown +subscription. Before #6348 the subscriber swallowed that refusal and lost the +head silently; after it, the subscriber errors. Short or already-finished +tracks hit this often. + +Fix it at the publisher. Accepting the refusal at the subscriber after a clean +PUBLISH_DONE would hide the lost head again, which is what #6348 fixed, and +other publishers may refuse the same way. + +A finished entry holds a `track::Consumer`, which keeps the track's cache +alive, so retaining it costs memory a peer controls. Candidate bounds: + +- **The request.** Drafts 17 to 19 give each SUBSCRIBE its own request + stream; keep the entry until the subscriber closes its side, with our + subscriber holding it open until its joining FETCH resolves. Drafts 14 to 16 + share the control stream and have no such signal. +- **The subscription limit.** A finished entry stays charged to the peer's + `session::Limits::subscriptions`, released when its FETCH takes it, the peer + cancels, or a new subscription needs the room (oldest finished entry first). + This works on every draft, and a peer that never sends the FETCH costs no + more than an open subscription. +- **The session alone.** Grows with every subscription a long-lived relay + session ever makes, so probably not bounded enough. + +Recommendation: the subscription limit, plus the request close where the +draft has one. Keep only entries that can be answered (`Joined::Group`); an +unsubscribed or refused subscription still drops its entry at once. A head +evicted from the cache meanwhile stays a loud refusal. The existing wait in +`joined()` bounds a FETCH whose subscription has not arrived yet; do not +stretch it to cover this. + +Tests, on mocked time with no sleeps, for each of drafts 14 to 19: + +- A track that finishes right after SUBSCRIBE_OK, with the joining FETCH + delivered an RTT after PUBLISH_DONE: the FETCH returns the Largest group's + head and the subscriber's track finishes cleanly with it. Fails today. +- More finished subscriptions than the limit with no FETCH: retained entries + stay at the limit, the oldest are dropped, and their late FETCHes are + refused. New subscriptions are still admitted. +- On drafts 17 to 19, closing the subscription's request drops its entry. +- An unsubscribed subscription's FETCH is refused, and session close frees + every entry. + +Public API: none. Wire: none; a late joining FETCH is answered where it was +refused. + +## Related + +- [Interop covers late join and FETCH](/quest/m1/interop-coverage/late-join.md) - the cross-implementation joining FETCH case +- [IETF request order](/quest/m2/ietf-request-order.md) - d14/15 request IDs and their ordering races diff --git a/quest/m2/qmux-receive-lease.md b/quest/m2/qmux-receive-lease.md new file mode 100644 index 00000000000..a8da74fcf29 --- /dev/null +++ b/quest/m2/qmux-receive-lease.md @@ -0,0 +1,73 @@ +# [M] qmux leases receive memory + +## Goal + +qmux sessions (WebSocket, `tcp://`, `tls://`, `unix://`, and WebRTC) bound +what a peer's incomplete messages hold by the session's receive window, and +evict the lowest-priority holder when that window is stuck, as native QUIC +does after [Bound message assembly](/quest/m1/quic/receive-memory.md). The +per-session memory a peer can make us hold is a deliberate, documented number, +chosen after weighing its cost against today's. + +## Plan + +Starts after [moq-dev/moq#5878](https://github.com/moq-dev/moq/pull/5878) +merges (open on 2026-10-22). Re-read the landed code before designing, since +the lease API may still move. + +State with #5878: qmux's `RecvStream` wraps its reader in moq-net's +`poll::Buffered`, which returns connection credit as it reads. `max_data` +then bounds only unread bytes. Each stream's incomplete message is bounded by +the largest message alone, so nothing bounds them in aggregate and nothing is +evicted. moq-tokio's `qmux_config` grants 10,000 streams per direction and +keeps qmux's defaults of 1 MiB `max_data` and 256 KiB `max_stream_data`. +qmux already runs on moq-quic's mux `Connection`, whose stream state carries +the lease and eviction machinery the tokio and io_uring drivers use, so the +transport side should be wiring more than new machinery. + +The cost: a leasing transport must offer at least +`moq_net::transport::MIN_RECEIVE_WINDOW`, the lite-01/02 `AnnounceInit` cap +(64 MiB) plus 16 bytes of prefixes, which also covers the 32 MiB frame cap. +Native QUIC defaults to 128 MiB. qmux keeps reading its byte stream to see +control frames, so the window, not the socket, is what bounds the bytes a +slow reader's peer parks in userspace. Going from 1 MiB to 64 MiB multiplies +that by every browser fallback, TCP, and WebRTC session a relay serves. + +Bring the options to the maintainer with a recommendation before building: + +- **The native floor.** Lease at `MIN_RECEIVE_WINDOW` or native's 128 MiB. + Simplest, and one behavior across transports, at the largest allowance. +- **A smaller qmux floor.** Only lite-01/02 needs the 64 MiB term, and + [Segmented frames](/quest/m2/unbounded-groups/segmented-frames.md) removes + the frame term. MAX_DATA can only grow, so a session can start at the floor + its version needs and extend once SETUP (or a WebSocket subprotocol) names a + legacy version. A floor below some supported unit must refuse it loudly. +- **Grow on demand.** Advertise a small window and extend MAX_DATA only when + leases block progress, up to the floor, evicting only at the floor. Typical + sessions stay small. Check that ordinary media adds no round trips. + +Bounding `Buffered` in aggregate with a cap that only refuses is not an +option: it gives no backpressure. Lowering a protocol or frame size limit to +shrink the floor is a separate maintainer decision. + +Validate the configured windows against whatever floor is chosen and fail +loudly, as moq-tokio's `ReceiveWindowTooSmall` does, and align moq-tokio's +`qmux_config` and WebRTC config with it. Cover the native receive-credit +cases over qmux: many partial messages filling the window, eviction by +priority, and progress after eviction, with a JS sender over WebSocket. +Measure retained bytes per session and throughput across many sessions for +the chosen window. + +Public API: qmux's `RecvStream` leases natively instead of through +`Buffered`, and its window defaults or validation change (MIGRATE.md). Wire: +QMux MAX_DATA and STOP_SENDING only, no MoQ format change. + +## Required + +- [Bound message assembly](/quest/m1/quic/receive-memory.md) - native leasing and `MIN_RECEIVE_WINDOW`, in #5878 + +## Related + +- [Segmented frames](/quest/m2/unbounded-groups/segmented-frames.md) - frame credit returns per segment, shrinking the floor's frame term +- [WebSocket caps](/quest/m2/rate-websocket.md) - paced reads on the same sessions, inside the window +- [Transport x version benches](/quest/m2/bench-transport-sweep.md) - where a per-session memory and throughput sweep belongs From d37b7bf309954b25bef4c2d3f363b2a78acff872 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 22 Oct 2026 09:56:59 -0700 Subject: [PATCH 2/3] quest: a retained join ends with the subscription's authorization Co-Authored-By: Claude Opus 5.5 --- quest/m2/ietf-join-after-finish.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/quest/m2/ietf-join-after-finish.md b/quest/m2/ietf-join-after-finish.md index f8242f6da43..075167a9c1e 100644 --- a/quest/m2/ietf-join-after-finish.md +++ b/quest/m2/ietf-join-after-finish.md @@ -48,7 +48,10 @@ alive, so retaining it costs memory a peer controls. Candidate bounds: Recommendation: the subscription limit, plus the request close where the draft has one. Keep only entries that can be answered (`Joined::Group`); an -unsubscribed or refused subscription still drops its entry at once. A head +unsubscribed or refused subscription still drops its entry at once. The +entry must not outlive what authorized the subscription: when its request +token's grant expires or is revoked, or the session grant no longer covers +the track, drop it, since a late FETCH reads the cache on that authority. A head evicted from the cache meanwhile stays a loud refusal. The existing wait in `joined()` bounds a FETCH whose subscription has not arrived yet; do not stretch it to cover this. @@ -62,6 +65,8 @@ Tests, on mocked time with no sleeps, for each of drafts 14 to 19: stay at the limit, the oldest are dropped, and their late FETCHes are refused. New subscriptions are still admitted. - On drafts 17 to 19, closing the subscription's request drops its entry. +- A request token that expires (by mocked time) or is revoked after the + finish drops the entry, and the late FETCH is refused. - An unsubscribed subscription's FETCH is refused, and session close frees every entry. From 2aa16d02e7e60b29041e51d319ebf0883d0d976f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 22 Oct 2026 10:28:13 -0700 Subject: [PATCH 3/3] quest: point the qmux lease quest at #5878 instead of the quest it deletes Co-Authored-By: Claude Opus 5.5 --- quest/m2/qmux-receive-lease.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/quest/m2/qmux-receive-lease.md b/quest/m2/qmux-receive-lease.md index a8da74fcf29..935b59fa059 100644 --- a/quest/m2/qmux-receive-lease.md +++ b/quest/m2/qmux-receive-lease.md @@ -5,7 +5,7 @@ qmux sessions (WebSocket, `tcp://`, `tls://`, `unix://`, and WebRTC) bound what a peer's incomplete messages hold by the session's receive window, and evict the lowest-priority holder when that window is stuck, as native QUIC -does after [Bound message assembly](/quest/m1/quic/receive-memory.md). The +does after [moq-dev/moq#5878](https://github.com/moq-dev/moq/pull/5878). The per-session memory a peer can make us hold is a deliberate, documented number, chosen after weighing its cost against today's. @@ -62,10 +62,6 @@ Public API: qmux's `RecvStream` leases natively instead of through `Buffered`, and its window defaults or validation change (MIGRATE.md). Wire: QMux MAX_DATA and STOP_SENDING only, no MoQ format change. -## Required - -- [Bound message assembly](/quest/m1/quic/receive-memory.md) - native leasing and `MIN_RECEIVE_WINDOW`, in #5878 - ## Related - [Segmented frames](/quest/m2/unbounded-groups/segmented-frames.md) - frame credit returns per segment, shrinking the floor's frame term