Repository navigation
quest: qmux receive leases and the pre-draft-20 join after a quick finish #6352
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
b82cff6
quest: plan qmux receive leases and the pre-draft-20 join after a qui…
kixelated d37b7bf
quest: a retained join ends with the subscription's authorization
kixelated 2aa16d0
quest: point the qmux lease quest at #5878 instead of the quest it de…
kixelated File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| # [M] A quick finish keeps its pre-draft-20 join | ||
|
|
||
| ## Goal | ||
|
|
||
| On drafts 14 to 19, a joining FETCH that reaches our publisher after its | ||
| subscription finished cleanly is still answered from the subscription's | ||
| saved Largest group. A track that ends within about one RTT of SUBSCRIBE_OK | ||
| then finishes cleanly at the subscriber, head included, instead of failing. | ||
| What the publisher retains for this is bounded by design (the session, the | ||
| subscription's request, or a limit the peer already negotiates), never by a | ||
| timer constant. | ||
|
|
||
| ## Plan | ||
|
|
||
| Starts after [moq-dev/moq#6348](https://github.com/moq-dev/moq/pull/6348) | ||
| merges (open on 2026-10-22). It makes every pre-draft-20 join a SUBSCRIBE | ||
| followed, after SUBSCRIBE_OK, by a `RelativeJoining{0}` FETCH, and fails the | ||
| subscription when that FETCH is refused. If it is abandoned, re-judge this | ||
| quest. | ||
|
|
||
| State: `rs/moq-net/src/ietf/publisher.rs` registers a `Join` entry in | ||
| `Publisher::joins` when it dispatches the SUBSCRIBE, fills it with a `Joined` | ||
| snapshot as SUBSCRIBE_OK goes out, and drops it when the subscription ends, | ||
| before PUBLISH_DONE. A joining FETCH that arrives later finds no entry, waits | ||
| out the registration wait in `joined()`, and is refused as an unknown | ||
| subscription. Before #6348 the subscriber swallowed that refusal and lost the | ||
| head silently; after it, the subscriber errors. Short or already-finished | ||
| tracks hit this often. | ||
|
|
||
| Fix it at the publisher. Accepting the refusal at the subscriber after a clean | ||
| PUBLISH_DONE would hide the lost head again, which is what #6348 fixed, and | ||
| other publishers may refuse the same way. | ||
|
|
||
| A finished entry holds a `track::Consumer`, which keeps the track's cache | ||
| alive, so retaining it costs memory a peer controls. Candidate bounds: | ||
|
|
||
| - **The request.** Drafts 17 to 19 give each SUBSCRIBE its own request | ||
| stream; keep the entry until the subscriber closes its side, with our | ||
| subscriber holding it open until its joining FETCH resolves. Drafts 14 to 16 | ||
| share the control stream and have no such signal. | ||
| - **The subscription limit.** A finished entry stays charged to the peer's | ||
| `session::Limits::subscriptions`, released when its FETCH takes it, the peer | ||
| cancels, or a new subscription needs the room (oldest finished entry first). | ||
| This works on every draft, and a peer that never sends the FETCH costs no | ||
| more than an open subscription. | ||
| - **The session alone.** Grows with every subscription a long-lived relay | ||
| session ever makes, so probably not bounded enough. | ||
|
|
||
| Recommendation: the subscription limit, plus the request close where the | ||
| draft has one. Keep only entries that can be answered (`Joined::Group`); an | ||
| unsubscribed or refused subscription still drops its entry at once. The | ||
| entry must not outlive what authorized the subscription: when its request | ||
| token's grant expires or is revoked, or the session grant no longer covers | ||
| the track, drop it, since a late FETCH reads the cache on that authority. A head | ||
| evicted from the cache meanwhile stays a loud refusal. The existing wait in | ||
| `joined()` bounds a FETCH whose subscription has not arrived yet; do not | ||
| stretch it to cover this. | ||
|
|
||
| Tests, on mocked time with no sleeps, for each of drafts 14 to 19: | ||
|
|
||
| - A track that finishes right after SUBSCRIBE_OK, with the joining FETCH | ||
| delivered an RTT after PUBLISH_DONE: the FETCH returns the Largest group's | ||
| head and the subscriber's track finishes cleanly with it. Fails today. | ||
| - More finished subscriptions than the limit with no FETCH: retained entries | ||
| stay at the limit, the oldest are dropped, and their late FETCHes are | ||
| refused. New subscriptions are still admitted. | ||
| - On drafts 17 to 19, closing the subscription's request drops its entry. | ||
| - A request token that expires (by mocked time) or is revoked after the | ||
| finish drops the entry, and the late FETCH is refused. | ||
| - An unsubscribed subscription's FETCH is refused, and session close frees | ||
| every entry. | ||
|
|
||
| Public API: none. Wire: none; a late joining FETCH is answered where it was | ||
| refused. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Interop covers late join and FETCH](/quest/m1/interop-coverage/late-join.md) - the cross-implementation joining FETCH case | ||
| - [IETF request order](/quest/m2/ietf-request-order.md) - d14/15 request IDs and their ordering races | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| # [M] qmux leases receive memory | ||
|
|
||
| ## Goal | ||
|
|
||
| qmux sessions (WebSocket, `tcp://`, `tls://`, `unix://`, and WebRTC) bound | ||
| what a peer's incomplete messages hold by the session's receive window, and | ||
| evict the lowest-priority holder when that window is stuck, as native QUIC | ||
| does after [moq-dev/moq#5878](https://github.com/moq-dev/moq/pull/5878). The | ||
| per-session memory a peer can make us hold is a deliberate, documented number, | ||
| chosen after weighing its cost against today's. | ||
|
|
||
| ## Plan | ||
|
|
||
| Starts after [moq-dev/moq#5878](https://github.com/moq-dev/moq/pull/5878) | ||
| merges (open on 2026-10-22). Re-read the landed code before designing, since | ||
| the lease API may still move. | ||
|
|
||
| State with #5878: qmux's `RecvStream` wraps its reader in moq-net's | ||
| `poll::Buffered`, which returns connection credit as it reads. `max_data` | ||
| then bounds only unread bytes. Each stream's incomplete message is bounded by | ||
| the largest message alone, so nothing bounds them in aggregate and nothing is | ||
| evicted. moq-tokio's `qmux_config` grants 10,000 streams per direction and | ||
| keeps qmux's defaults of 1 MiB `max_data` and 256 KiB `max_stream_data`. | ||
| qmux already runs on moq-quic's mux `Connection`, whose stream state carries | ||
| the lease and eviction machinery the tokio and io_uring drivers use, so the | ||
| transport side should be wiring more than new machinery. | ||
|
|
||
| The cost: a leasing transport must offer at least | ||
| `moq_net::transport::MIN_RECEIVE_WINDOW`, the lite-01/02 `AnnounceInit` cap | ||
| (64 MiB) plus 16 bytes of prefixes, which also covers the 32 MiB frame cap. | ||
| Native QUIC defaults to 128 MiB. qmux keeps reading its byte stream to see | ||
| control frames, so the window, not the socket, is what bounds the bytes a | ||
| slow reader's peer parks in userspace. Going from 1 MiB to 64 MiB multiplies | ||
| that by every browser fallback, TCP, and WebRTC session a relay serves. | ||
|
|
||
| Bring the options to the maintainer with a recommendation before building: | ||
|
|
||
| - **The native floor.** Lease at `MIN_RECEIVE_WINDOW` or native's 128 MiB. | ||
| Simplest, and one behavior across transports, at the largest allowance. | ||
| - **A smaller qmux floor.** Only lite-01/02 needs the 64 MiB term, and | ||
| [Segmented frames](/quest/m2/unbounded-groups/segmented-frames.md) removes | ||
| the frame term. MAX_DATA can only grow, so a session can start at the floor | ||
| its version needs and extend once SETUP (or a WebSocket subprotocol) names a | ||
| legacy version. A floor below some supported unit must refuse it loudly. | ||
| - **Grow on demand.** Advertise a small window and extend MAX_DATA only when | ||
| leases block progress, up to the floor, evicting only at the floor. Typical | ||
| sessions stay small. Check that ordinary media adds no round trips. | ||
|
|
||
| Bounding `Buffered` in aggregate with a cap that only refuses is not an | ||
| option: it gives no backpressure. Lowering a protocol or frame size limit to | ||
| shrink the floor is a separate maintainer decision. | ||
|
|
||
| Validate the configured windows against whatever floor is chosen and fail | ||
| loudly, as moq-tokio's `ReceiveWindowTooSmall` does, and align moq-tokio's | ||
| `qmux_config` and WebRTC config with it. Cover the native receive-credit | ||
| cases over qmux: many partial messages filling the window, eviction by | ||
| priority, and progress after eviction, with a JS sender over WebSocket. | ||
| Measure retained bytes per session and throughput across many sessions for | ||
| the chosen window. | ||
|
|
||
| Public API: qmux's `RecvStream` leases natively instead of through | ||
| `Buffered`, and its window defaults or validation change (MIGRATE.md). Wire: | ||
| QMux MAX_DATA and STOP_SENDING only, no MoQ format change. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Segmented frames](/quest/m2/unbounded-groups/segmented-frames.md) - frame credit returns per segment, shrinking the floor's frame term | ||
| - [WebSocket caps](/quest/m2/rate-websocket.md) - paced reads on the same sessions, inside the window | ||
| - [Transport x version benches](/quest/m2/bench-transport-sweep.md) - where a per-session memory and throughput sweep belongs |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When a SUBSCRIBE is admitted by a request-borne token, retaining its
Joined::Groupafter the subscription completes also outlives theRequestGrantthat enforces expiry and revocation. This is especially unbounded on drafts 14–16, where there is no per-request close signal: the later joining FETCH uses the retained entry and aGate::named()session-grant check, so it can read cached data after the authorizing request token has expired or been revoked. Include request-grant lifetime among the removal conditions and add a mocked-time expiry/revocation test.AGENTS.md reference: AGENTS.md:L17-L18
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in d37b7bf: the plan now drops a retained entry when the subscription's request-token grant expires or is revoked, or the session grant stops covering the track, and the test plan adds a mocked-time expiry and a revocation case.
(Written by Claude Opus 5.5)