Skip to content

feat(quic)!: schedule datagrams in their send group - #5770

Merged
kixelated merged 57 commits into
mainfrom
quest/m1/quic/datagram-priority
Oct 17, 2026
Merged

kixelated merged 57 commits into
mainfrom
quest/m1/quic/datagram-priority

Conversation

@kixelated

@kixelated kixelated commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Quest: m1/quic/datagram-priority. Follows #5711.

Datagrams used to go first in every packet, ahead of every stream. A low-priority datagram flood could starve a high-priority stream. Now each moq-quic send group queues its own datagrams, and they are scheduled with the group's streams.

What changed

  • moq-quic scheduler: a datagram is an item in its group's queue, keyed by its in-group order (the same value a stream uses for priority). On a tie, a datagram goes before a stream. Its frame bytes are charged to the group's and the domain's DRR credit. Ungrouped datagrams go in the ungrouped queue, ahead of every group, like ungrouped streams. The datagrams-first loop in populate_packet is gone; write_frames writes streams and datagrams in one pass.
  • Never blocks: datagram_send_buffer_size is one connection-wide budget counted in frame bytes. When a datagram doesn't fit, the scheduler evicts from the domain holding the most datagram bytes (oldest on a tie). Within that domain it takes the lowest-priority group's oldest datagram, and the new datagram is dropped if it would be the victim. SendDatagramError::Blocked, Event::DatagramsUnblocked, the drop flag, and send_datagram_wait are removed.
  • Packet fit: a datagram that doesn't fit ends the packet if the packet already holds something. In an empty packet (a clamped loss probe or GSO segment) it is deferred so smaller frames can still go, and it keeps its place for the next packet.
  • Drivers: moq_tokio::quic::driver::Connection::send_datagram(data, group, order), the matching quic::web::Session methods, and the moq-uring equivalents.
  • moq-net: transport::poll::Session::poll_send_datagram becomes a synchronous send_datagram(payload, group: Option<&Group>, order: i64). The lite and IETF publishers send each datagram in its subscription's send group at priority::in_group(sequence, 0, order). Backends without groups (web-transport adapters, moq-ffi, moq-wasm) ignore the group and order, and drop a datagram they have no room for.
  • js/net is unchanged.

Public API impact (breaking)

  • moq-quic: Datagrams::send(data, group: Option<&SendGroup>, order: i64) replaces send(data, drop: bool). Removed: SendDatagramError::Blocked and Event::DatagramsUnblocked.
  • moq-tokio: send_datagram takes group and order. Removed: send_datagram_wait and SendDatagram.
  • moq-uring: Connection::send_datagram and web::Session::send_datagram are added (inherent, with group and order).
  • moq-net: poll::Session::send_datagram replaces poll_send_datagram.
  • moq-quic fuzz harness (hidden fuzz feature): the datagram op carries an order instead of drop.

Wire impact

None on the wire; the draft adds only scheduling guidance (SHOULD).

Tests

  • moq-quic Pair tests, using mocked time, in one congestion-limited flight:
    • low_priority_datagrams_wait_behind_streams (none of the datagrams make it into the flight);
    • high_priority_datagrams_preempt_streams (all of them do).
      With the old datagrams-first rule, the low-priority test sees all 8 datagrams in the first flight.
  • moq-quic scheduler tests (saturated.rs): datagrams merge into their group's order, share a group's credit, share bytes across saturated domains, preempt groups when ungrouped, lead the next packet when they don't fit, fit what they can in a clamped packet, and still send from a released group.
  • Schedule unit tests for ties, eviction (lowest-priority oldest, the new datagram as victim, busiest domain, tied domains), deferral, and retain_datagrams.
  • moq-net: datagrams_rank_within_their_subscription for both the lite and IETF publishers.

Benchmark

rs/moq-net/benches/schedule.rs adds bench_datagram, which sweeps groups (or domains) {1, 16, 256, 4096} against datagrams queued per group {1, 16, 256}. Times are ns per op, short run:

bench 1/1 4096/16 256/256
schedule_datagram_evict_groups 59 227 254
schedule_datagram_evict_domains 59 211 267
schedule_datagram_send 67 241 230

Cost grows logarithmically with the table. Stream fanout keeps its shape (subs_1/1 25, subs_4096/16 42, subs_256/256 58).

Decided

  • The moq-lite draft replaces "send a datagram only if the congestion controller can transmit it immediately" with "SHOULD drop a datagram rather than block when the transport has no room for it", and gains a SHOULD in Prioritization, with a changelog bullet under moq-lite-07: a publisher orders a datagram among the subscription's streams as if it were its Group's stream, ahead of that stream when both exist. doc/concept/moq-lite.md says the same for users, and its Priority row now counts datagrams in the equal-priority byte split for native moq-quic transports (browser publishers and transports without send groups are not covered). The moq-hang draft says nothing about datagram scheduling.
  • A datagram that doesn't fit a non-empty packet ends the packet, even if that wastes the tail.
  • Setting a datagram aside in a clamped packet may gift its group up to one quantum of credit.
  • The budget counts frame bytes, with no per-item overhead, and ungrouped datagrams are evicted last.

Follow-ups

  • A work-conserving skip for a datagram that doesn't fit, without gifting credit.

🤖 Generated with Claude Code

(Written by Opus 5.5)

kixelated and others added 30 commits October 15, 2026 19:35
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…batim

Unmodified copies, not yet compiled, so a reviewer can diff the adaptation
that follows against upstream:

- quinn-rs/quinn 7616e6b2782722f3ce4a1b181ef08817d7f545e4 quinn/src ->
  rs/moq-tokio/src/quic/driver (lib.rs -> mod.rs), the commit moq-quic
  forked from.
- moq-dev/noq e27c0ce00741d81c8620b98e345d4f2c88e0768c web-transport-moq/src ->
  rs/moq-tokio/src/quic/web (lib.rs -> mod.rs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop every moq-noq dependency. moq-tokio's QUIC backend now drives moq-quic
through quinn's async layer (`quic::driver`) with web-transport-moq folded in
as `quic::web`; moq-uring's sans-IO driver moves from moq-noq-proto to
moq-quic. Upstream noq stays only behind moq-tokio's `iroh` feature.

- The `noq` cargo feature is `quic` everywhere it was forwarded.
- `moq_tokio::noq::Error` is `moq_tokio::quic::Error`; `Error::Noq` is
  `Error::Quic`.
- moq-quic takes `moq_clock::Instant` and raises a once-per-connection
  Draining endpoint event, which backs the driver's `wait_all_draining`.
- qlog writes one trace per endpoint (per dial on moq-uring's connect side),
  since moq-quic holds one sink per transport config.
- Session stats report the congestion controller's bandwidth estimate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The capsule reader maps a lost stream to UnexpectedEnd, so every ordinary
session close warned "failed to read capsule". A read error while the
connection is closed is now debug; one on a live connection still warns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Relay peak RSS on the fanout workloads matches moq-noq on origin/main, which
was the last open part of the Goal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Carry #5638's steering changes onto the renamed moq-quic drivers: the
shard CID comes from moq_sock::shard::cid, and moq-uring stops greasing
the QUIC bit on sharded endpoints.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Endpoint::close now refuses attempts still queued for accept, and
wait_all_draining waits for accepts partway through Incoming::accept, so
a server shutdown cannot release the socket before those peers hear back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace quinn's single pending-stream heap with a hierarchical schedule:

- Streams outside every group go first, in quinn's order (priority, then
  round robin), so control and HTTP/3 streams are never queued behind media.
- Groups belong to a scheduling domain (one default). Inside a domain the
  highest backlogged priority is served strictly, and equal priorities
  share bytes by deficit round robin, charged in wire bytes. Domains share
  bytes the same way, so opening more groups buys a domain nothing.
- A stream's priority widens to i64 and orders it only within its group.
  Debt carries across a group going idle; unused credit does not, and a
  blocked stream holds no turn.

Connection::send_groups opens, moves and closes groups and domains as
owned tokens. moq-tokio's and moq-uring's drivers wrap a group in a shared
SendGroup handle that closes on drop, and their streams gain set_group.

Retransmits re-queue through the same schedule, so a lost range keeps its
group's share. Tests drive the real send path with differently sized
streams: byte fairness within a bounded window, preemption, per-group
order, priority updates, sparse sequences, blocked streams, retransmits,
domains, resets and leaving a group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The poll transport gains send groups:

- Session::send_group(priority) opens one, or returns None on a transport
  without them.
- SendStream::Group names the transport's group (Ungrouped, uninhabited,
  for flat transports), SendStream::set_group joins one, and
  SendStream::set_priority widens to i64.
- SendGroup::set_priority moves a whole group.

The lite and IETF publishers open one group per subscription (and per
standalone FETCH; a joining FETCH shares its subscription's) at the
subscriber priority. Group streams join it ranked by sequence in the
subscription's group order, with a group's first stream ahead of the
rest of it. SUBSCRIBE_UPDATE moves the group once instead of reranking
every stream, and an order flip re-stamps the open streams. On a
transport without groups the session-wide PriorityQueue still ranks
every stream, as before.

moq-tokio's and moq-uring's adapters reach backend groups through new
Grouped and GroupedSend extension traits: native QUIC and WebTransport
open real groups; iroh, qmux, ffi and wasm stay flat. The moq-tokio
adapter defers a join made while a write or FIN watch holds the stream,
like a priority change.

A benchmark compares the groups with the PriorityQueue and two scalar
packings on congested audio and video workloads, and sweeps
subscriptions against in-flight streams.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hierarchical send groups landed in moq-quic and every subscription uses one, so drop the quest and the links that waited on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The server socket retried every failed recv, so a persistent error spun
the endpoint task while tokio kept readiness asserted. Only WouldBlock
retries now, matching the driver's tokio runtime.

Also cap moq_quic at info like noq was, fix the driver doc example, and
point the moq-uring README at the just recipe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the async publishers name it, so a model-only build flagged it as dead code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop GroupedSend::set_order: the adapters use a single set_priority(i64),
documented everywhere as the priority within the stream's send group, or
session-wide when ungrouped.

Grouped and GroupedSend leave the public API. pub(crate) is refused
(E0446) because the public adapters' `type Group` projects through
GroupedSend, so both are `pub` in a private module and never re-exported:
other crates can neither name nor implement them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	rs/moq-quic/README.md
#	rs/moq-tokio/src/quic/driver/connection.rs
#	rs/moq-tokio/src/quic/driver/mod.rs
#	rs/moq-tokio/src/quic/driver/send_stream.rs
#	rs/moq-tokio/src/quic/web/send.rs
#	rs/moq-tokio/src/quic/web/session.rs
#	rs/moq-uring/src/quic/driver/stream.rs
#	rs/moq-uring/src/quic/mod.rs
# Conflicts:
#	quest/m1/quic/scheduler.md
#	quest/m2/quic-deadline.md
Brings in receive credit (#5675). Its ConnectionTag now also tags the
send group and domain tokens, so moq-quic refuses a token from another
connection as its docs promised, instead of indexing whichever of its own
groups shares the slot. The drivers' pointer checks became redundant and
are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…duler

# Conflicts:
#	quest/m3/cut-through/README.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
moq-quic SendGroups::{open, set_priority, priority}, the tokio and uring
driver handles and send_group, and moq-net's poll::Session::send_group
and poll::SendGroup::set_priority now take i64. Group and stream
priorities share one width, so no layer converts between them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated and others added 6 commits October 16, 2026 15:38
Browser WebTransport from Wasm has no send groups, and Lite datagrams
bypass the stream scheduler, so neither gets the byte-fair split.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Datagram priority will move datagrams into the send-group hierarchy;
its quest now owns updating the Priority row when that lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from quest/m1/quic/scheduler to main October 17, 2026 01:01
kixelated and others added 3 commits October 16, 2026 18:04
main carries #5711 squashed, so the merge takes main's changes relative to
the scheduler tip this branch already merged (ddeb523).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review October 17, 2026 01:56
@coderabbitai

coderabbitai Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 25 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: cc83c93c-9dbf-4a6d-8289-49897f7c425c
📥 Commits

Reviewing files that changed from the base of the PR and between 3789ee2 and 2ad974d.

📒 Files selected for processing: 39
  • doc/concept/moq-lite.md
  • drafts/draft-lcurley-moq-lite.md
  • rs/moq-ffi/src/transport/adapter.rs
  • rs/moq-net/benches/schedule.rs
  • rs/moq-net/src/client.rs
  • rs/moq-net/src/ietf/adapter.rs
  • rs/moq-net/src/ietf/publisher.rs
  • rs/moq-net/src/lite/publisher.rs
  • rs/moq-net/src/lite/test_transport.rs
  • rs/moq-net/src/server.rs
  • rs/moq-net/src/transport.rs
  • rs/moq-net/src/transport/memory.rs
  • rs/moq-net/tests/integration/support/mock.rs
  • rs/moq-quic/README.md
  • rs/moq-quic/benches/wire.rs
  • rs/moq-quic/src/config/transport.rs
  • rs/moq-quic/src/connection/datagrams.rs
  • rs/moq-quic/src/connection/mod.rs
  • rs/moq-quic/src/connection/streams/mod.rs
  • rs/moq-quic/src/connection/streams/saturated.rs
  • rs/moq-quic/src/connection/streams/schedule.rs
  • rs/moq-quic/src/connection/streams/state.rs
  • rs/moq-quic/src/fuzz/connection.rs
  • rs/moq-quic/src/tests/mod.rs
  • rs/moq-tokio/src/quic/driver/connection.rs
  • rs/moq-tokio/src/quic/driver/mod.rs
  • rs/moq-tokio/src/quic/driver/send_group.rs
  • rs/moq-tokio/src/quic/driver/tests.rs
  • rs/moq-tokio/src/quic/web/session.rs
  • rs/moq-tokio/src/transport.rs
  • rs/moq-tokio/src/transport/grouped.rs
  • rs/moq-tokio/src/transport/owned.rs
  • rs/moq-uring/src/quic/driver/connection.rs
  • rs/moq-uring/src/quic/driver/group.rs
  • rs/moq-uring/src/quic/web.rs
  • rs/moq-uring/src/transport.rs
  • rs/moq-uring/src/transport/adapter.rs
  • rs/moq-uring/tests/integration/web.rs
  • rs/moq-wasm/src/transport/adapter.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-17T02:45:25.642277Z 33fc34e New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d93eb5e9f5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread rs/moq-net/src/lite/publisher.rs Outdated
kixelated and others added 3 commits October 16, 2026 19:38
A SUBSCRIBE_UPDATE refreshed the steer only when a group was served next, so
datagrams before then kept the old order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 33fc34e350

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread rs/moq-quic/src/connection/streams/state.rs Outdated
kixelated and others added 2 commits October 16, 2026 19:58
write_frames judged emptiness by its own frames, so a top-ranked datagram that
did not fit behind an ACK or other control frame stepped aside and let
lower-ranked data take the packet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated added this pull request to the merge queue Oct 17, 2026
Merged via the queue into main with commit 79005b0 Oct 17, 2026
14 of 15 checks passed
@kixelated
kixelated deleted the quest/m1/quic/datagram-priority branch October 17, 2026 05:13
kixelated added a commit that referenced this pull request Oct 17, 2026
#5770 schedules datagrams in their send group, so ungrouped datagrams now
go before equal-priority stream data, and one that does not fit ends the
packet instead of letting streams fill it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated added a commit that referenced this pull request Oct 17, 2026
…ames

Also covers @moq/hang's reserved catalog `version` from #5806.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated added a commit that referenced this pull request Oct 19, 2026
Implemented in #5770 and refined in #5872, with publisher integration, saturation and eviction regressions, fanout benchmarks, and draft coverage on main.

Co-Authored-By: GPT-6.1-Sol <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant