quic: add support for HTTP/3 datagrams

[pimterry]

With nghttp3 released and updated here, we can now cleanly add proper HTTP/3 datagram support to fix #63891. This PR:

• Adds stream.sendDatagram and stream.ondatagram support on HTTP/3 streams.
• Makes the session equivalents throw for HTTP/3 sessions.

The resulting API is a little awkward (both H3 & QUIC methods present everywhere, but throwing in different cases) due to the mixed QUIC/H3 API as already discussed, but I think it's worthwhile opening in the meantime anyway so people can start playing with this, and so we can review it standalone.

The API will conflict with the HTTP/3 split PR that restructures related areas, but the internals should be the same in any case (just moved around) so it'll be easy to slide into whatever final structure we end up with there.

This doesn't support HTTP/3 datagrams for 0RTT on the client side. The RFC would require us to persist the HTTP/3 SETTINGS client-side to validate datagrams are supported to do so, but we currently don't persist HTTP/3 SETTINGS at all. Right now for all HTTP/3 0RTT we're just falling back to the HTTP/3 defaults instead (as allowed by the spec) but the datagram default is off. We could explore that later if we want, but completely rejigging HTTP/3 session resumption to enable this seemed more effort than it's worth for now. Server acceptance of inbound 0RTT datagrams from non-Node clients should work fine but can't be covered in the tests due to this.

[nodejs-github-bot]

Review requested:

• @nodejs/quic

[metcoder95]

lgtm

[pimterry]

Now updated. This now buffer datagrams for pending streams that couldn't be immediately opened (thanks @martenrichter for the suggestion). Returns the id as normal when sending, reports statuses as normal, including ABANDONED if it wasn't possible to eventually unbuffer the datagrams.

Queues up to highWatermark to match the normal write behaviour, then explicitly rejects future writes. We could consider drop-oldest instead, but that would mean there's no way to detect this behaviour at all since there's no backpressure mechanism here. Seems unlikely that stream opening will queue for long or that datagram traffic will get huge, so this seems reasonable for now but happy to debate.

[jasnell]

It is unfortunate that regular QUIC datagrams and HTTP/3 datagrams appear to have these different semantics. I'm not convinced that having two separate mechanisms session.sendDatagram and stream.sendDatagram is the best approach, but I also really don't want HTTP/3 specific APIs introduced as we've discussed before. I wonder if taking a different approach would work...

const datagrams = session.datagrams; // Returns null if datagrams
                                     // at the session-level are
                                     // not supported

const datagrams = stream.datagrams;  // Returns null if datagrams
                                     // at the stream-level are
                                     // not supported

// Same high-level API for both
datagrams.send(...);
datagrams.ondatagram = (...) => { /* ... */ };

Further, in the QUIC docs, saying things like "Only applies to HTTP/3", etc is too specific. I would word it like, "Session-level datagrams may not be supported by all QUIC applications" and "Stream-level datagrams may not be supported by all QUIC applications", etc.

Thinking more about it...

[jasnell]

See my comments here #64234 (comment)

[jasnell]

We should keep application-specific handling into the Application subclass as much as possible.

[jasnell]

Nit: we should expose the type enum constants via the binding so we don't have to worry about keeping anything in sync.

[jasnell]

Do we really need to introduce a new callback? Alternatively, we could introduce a third argument to the existing callback that indicates that the datagram is associated with a specific stream.

[jasnell]

Nit: the naming is a bit confusing. An application may support datagrams but not need custom datagram handling. Perhaps something like SupportsCustomDatagrams or something similar?

[jasnell]

Alternatively, we could just route all datagrams through the application ReceiveDatagram and just let the application handle custom formats transparently, without having to advertise via SupportsDatagrams

[jasnell]

Should this be in options_ instead?

[jasnell]

Is this pending outbound or pending inbound? A comment to clarify here would be helpful.

[jasnell]

Nit: just defensively, I'd copy the contents of pending_datagram_queue_ out to a temporary local. You then don't need to clear it below.

[jasnell]

Instead of having EnqueuePendingDatagram return a boolean which then needs to be checked below, it could just return 0 when the datagram is not enqueued.

[jasnell]

Let's remove the mentions of HTTP/3 here. It sends a datagram associated with this stream, only works if the application supports stream-level datagrams, etc.

[jasnell]

There's a bit of a TOCTOU issue here. You're reserving the datagram id, but enqueuing it below may not accept it. We can end up burning datagram ids that are rejected.

[jasnell]

A comment here explaining the difference between DatagramReceived and DeliverRawDatagram would be helpful

[jasnell]

Maybe we should have the Session level SendDatagram (for both session and stream level) defer to the Application and make the Stream* argument optional. I can foresee future QUIC applications applying their own semantics to datagrams even at the session level, so adjusting for it now makes sense.

[pimterry]

It is unfortunate that regular QUIC datagrams and HTTP/3 datagrams appear to have these different semantics. I'm not convinced that having two separate mechanisms session.sendDatagram and stream.sendDatagram is the best approach, but I also really don't want HTTP/3 specific APIs introduced as we've discussed before.

Happy to cover the other comments, but quite a few here are clearly focused on providing a single QUIC+HTTP datagram API.

I really feel the whole HTTP/3 API doesn't currently work well without some kind of explicit API split between QUIC & HTTP/3. This PR is designed to easily refactor into that split model later once we have it, so I don't really want to redesign it to fit a single combined API in the meantime.

We need to talk about this and resolve it, here or in #63995 or directly, it's blocking a few things now.

I understand you prefer to have a single pair of classes (Session/Stream) act as the API for both separate protocols at the same time, but I'm not clear why, and there isn't any more context or explanation behind that which I can see anywhere.

Can you explain when & why that API is preferable to the separate QuicSession/Stream + Http3Session/Stream alternative?

For reference, I think my position is captured well in the 'Background' of the PR description for #63995.

[jasnell]

... but quite a few here are clearly focused on providing a single QUIC+HTTP datagram API

I'm focused on providing a QUIC API with affordances to implement a unified http api around it.

We already have a mess with having separate http, https, and http2 APIs all separate and distinct from each other. I don't want to add Yet-Another-HTTP-API-That-Is-Specific-To-HTTP3 on top for HTTP/3. I want to see us put effort into a single http API that can sit on top of http1, 2 or 3 and is capable of replacing the existing legacy http1 and http2 APIs.

[pimterry]

I don't want to add Yet-Another-HTTP-API-That-Is-Specific-To-HTTP3 on top for HTTP/3.

This is what the current API already does though.

Today we have an HTTP/3 specific API, it's just smooshed in on top of the QUIC API. All the methods and events and fields are there and exposed right now. It's actively inconsistent with the other HTTP APIs, and separate from the existing internals (e.g. totally separate TLS config), and the API is entirely unique & specific for HTTP/3. We're not currently avoiding this problem.

I want to reduce that gap and make this more maintainable: aligning more closely, and making the split explicit via QUIC + HTTP/3 classes you compose (just like net.Server/tls.Server/http.Server) rather than one implementation with implicit combined behaviour.

I'm not suggesting we change from the API features exposed today, or change the internals beyond moving things around. I do understand you'd rather not have an actual http3 module, that's fine.

If you really want to go this direction though, we should drop HTTP/3 here entirely, and go QUIC-only. If we have to choose, I'd prefer this - we could then implement HTTP/3 primitives in userspace at least, which is impossible with the current API. There are arguments in this direction, e.g. QUIC-only is the current state of Deno (https://docs.deno.com/examples/quic/).

Would you prefer that?

Personally, I'd like have a proper API for HTTP/3 in Node itself, but doing pure QUIC for now would be better than mixing them all together in a way they can't be used independently like today. We could drop HTTP/3 and potentially explore as a separate topic later on.

I want to see us put effort into a single http API

I'd love this, fully agreed, happy to work on it directly and I've had other discussions about it. But they aren't mutually exclusive. The opposite: to build a unified HTTP API, we need a good HTTP/3 API somewhere (at least internally) to build it on. The current API would be difficult even for the universal API internal use case, since it diverges so much from the HTTP/1 & HTTP/2 & TLS modules.

Personally though, I have some limited need for a unified API, and a very strong need every day for direct control over HTTP/3 and QUIC. Most simple servers & clients would appreciate a unified version, but serious networking libraries/tools/infra need to touch specific protocol primitives. The differences do matter. There are clearly large numbers of people doing low-level networking (lower than 'generic HTTP') on top of Node who are interested in specific protocol support (see the interest on #38478). Given that we must implement the primitives anyway, I think we should expose them in a way people can use them.

More philosophically: I think it's more important for Node to provide the core primitives than unifying broader features (ofc, in a perfect world we do both). Userspace can implement the latter - it's much much harder to externally implement the former.

Is there a route through here? Happy to explore the specifics of how we do this in #63995, I'm very flexible on the details within this. I haven't updated that from other discussion or rebased on the other PRs but I can this week if you'd be interested. Or would you prefer to drop HTTP/3 from QUIC entirely for now? That would solve the concern in practice and work for me. Or are we stuck, and this is a topic for a TSC debate?

[martenrichter]

Or would you prefer to drop HTTP/3 from QUIC entirely for now?

Please don't. The http/3 support is already very close to be mature.
And it is c implmentation and very thin and fast. Doing it in user land in js would not be as nice, and doing a binary addon is always a lot of work maintaning the compile infrastructure. (That is what my package currently does on top of the UDPSocket, and you may guess what motivates me to work on node.js itself for the same feature).

But I agree, that sometimes you want to access the primitives. What about an option to listenetc., that one wants nghttp/3 not to handle http3, so that one can opt out?

Regarding having more control about http/3 while nghttp/3 is invoked is relatively tough, I think,
as the lib itself has limited points to tap in. But so far nghttp/3 was very cooperative if a callback etc. was missing.

(And regarding whether the objects needs to be separated: Yes having http/3 methods already on the quic object is a bit awkward, but node.js has a lot of historic artifacts and people that want to go low level, usually can handle it. I would suspect, once the api is stable that enough packages will appear that put some sugar around the low level api....)

[pimterry]

Yes having http/3 methods already on the quic object is a bit awkward, but node.js has a lot of historic artifacts and people that want to go low level, usually can handle it.

These awkward artifacts are mistakes though, and we have a chance to avoid adding another. We can now freely change things to improve the design now, but only before this goes stable and then we live with it forever.

Agree we should offer low-level primitives people can wrap elsewhere (and internally, for a universal HTTP API) but we need primitives that match the protocols they represent. The current API doesn't give clear primitives for each protocol, it mixes them together, in some ways wrappers can't fix (like welding protocol to ALPN negotiation).

But I agree, that sometimes you want to access the primitives. What about an option to listenetc., that one wants nghttp/3 not to handle http3, so that one can opt out?

That would help a bit, but just for one part I'd like to improve in the design. The main problems I see are:

1. No control over which protocol implementation is used.
   Right now it's controlled automatically, by the ALPN selection of the remote peer - you can't pick the implementation yourself. In practice you'd want to be able to do it dynamically per-connection (using ALPN & SNI info etc) - just like you can today with TCP & TLS. This blocks various low-level use cases like network tooling, testing & advanced server configurations, and you can't build a wrapper to fix it.
   An endpoint option would help, like { http3: h3Options } to opt in to automatic HTTP/3, instead of the listenQuic()/listenHttp3() or node:http3 split. Dynamic negotiation still needs a way to upgrade QUIC to H3 at session open, but there's a path there. @jasnell would you prefer that approach?
2. The API has a single shared methods/events/etc API for both protocols, even though they're fundamentally different.
   We both agree it's awkward, and this creates footguns (some methods can't be used, or take different args, based on internal state) - e.g. all sessions have createUnidirectionalStream, but if an HTTP/3 client calls it they violate the protocol (with headers in release builds - in debug it crashes on an nghttp3 assert) or create a stream that buffers forever. This gets worse with WebTransport if we mix 3 protocols in one API (WT can do uni streams, QUIC can do uni streams, HTTP/3 can't - which one you're using is invisible, depending on both ALPN & SETTINGS negotiations). The API has hit a few cases like this because of the design itself, and it'll keep happening. An alternative design removes the entire class of issues.
   TypeScript usage is hard. You can't type a function to only take an HTTP stream or a WT stream, or validate method calls - all 3 protocols are represented by the same thing.
   Per-protocol APIs would let us make things clearer: easy sugar like h3Session.request instead of quicSession.createBidirectionalStream and clearer signatures everywhere (e.g. headers as an required arg) and we can drop e.g. createUnidirectionalStream from HTTP/3 but keep it for QUIC & WT.
3. Combining the protocol implementations makes everything harder to maintain, and makes the API harder to evolve.
   When we want to add QUIC-only or HTTP/3 only features, it's easier to do if the code for each is encapsulated & independent of the other's internals, and easier to design nice APIs if we don't have to worry about conflicts between them (sendDatagram here is exactly that).
   Today we have session/stream creation methods that have to validate all QUIC + HTTP/3 + WebTransport options in the same place and then manage the state for all of them - every method has to validate all possible states at once in the same function.

We can fix these remarkably easily, and I'm happy to do the work. I'm proposing a separated design, without significant internal changes or exposing new functionality - just regrouping existing code. We break up the existing implementation into separate classes:

• QuicStream/Http3Stream
• QuicSession/Http3Session
• listenQuic()/listenHttp3() (or one method + option, if we prefer) and connect equivalent.

Clean separated types for each use case, with their methods etc matching the corresponding protocol. Each Http3 instance wraps a Quic instance - automatically, if you use listenHttp3, or manually for protocol negotiation via new Http3Session(quicSession) from a session (up front - before app protocol activity on the session).

This does other nice things, like open easy space for WebTransport pooling: separate WebTransportSession instances coming from http3Session.connectWebTransport(options) or inbound events. Each created WT session is independent, backed by a single Http3Session, but with its own per-wt-session API. Streams emit from the owning WT session - so users never have to disambiguate HTTP & WT streams from the same event manually, and API impl just handles one protocol. WT & HTTP/3 multiplexing is a really core feature that comes out very nicely if you have separate instances to represent everything.

This is easy because it's how ngtcp2+nghttp3 work as well (and the rest of the QUIC/H3 ecosystem). They're separate components with no direct relationship that can be combined together on demand. It's a good design, we can expose that here too. There's a lot of wins if we do, and we don't even need to change much (the internals are maturing great as-is) we just need to regroup things a bit.

[jasnell]

Because it's relevant to this discussion ;-) ... https://www.jasnell.me/posts/fetch-is-not-enough

[martenrichter]

@pimterry
Regarding to 1:
What about a callback to choose if native http/3 is used.

to 2:
Well, all the three cases (native Quic, Http/3, and WebTransport) are basically QuicStream. For http/3 there is some additional framing, and WebTransport has some additional headers in its data streams at the beginning. So it is a bit like a QuicStream which has different bidirectional transforms (currently called applications, but it is not the same). So what about an object property on the QuicStream, that gives information about the used transport and that is a different type derived from a base type. This or another property may also carry an object with different transport-specific callbacks and methods? That is just an idea to make a kind of compromise. It is also an idea, so that one could tap in, though it is not necessary.

to 3: Actually Webtransport does not add much to the mix, it is not much. I will see if I get the tests all running, I may add today or tomorrow some tests.

@jasnell Great article! Btw. my webtransport package adds W3C interfaces to the server side, and I get often people why it does not connect to quic from deno or bun, as no one gets the subtile difference between webtransport over http/3 and pure quic, as the interface looks the same.