From 275865b7c4bb9023f42af3f0c5fdb69af9317e38 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 16 Oct 2026 19:21:39 -0700 Subject: [PATCH 01/31] quest: claim js-group-handover From 79949509239611aa5f3221b2f4cb1f4c9769a853 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 16 Oct 2026 20:06:15 -0700 Subject: [PATCH 02/31] feat(js/net): resume track subscriptions across same-epoch route swaps A request's broadcast handle stays the same when another route of its epoch takes over: a logical front re-sources each track, and every reader resumes on the new route's copy from the first group it lacks. Mirrors model::resume in rs/moq-net, including per-reader delivered records and group recovery. Co-Authored-By: Claude Opus 5.5 --- doc/lib/js/net.md | 6 +- js/net/src/connection/connect.test.ts | 51 ++ js/net/src/connection/connect.ts | 82 +- js/net/src/delivered.test.ts | 55 ++ js/net/src/delivered.ts | 81 ++ js/net/src/group.ts | 194 ++++- js/net/src/internal.ts | 150 +++- js/net/src/origin-demand.test.ts | 5 +- js/net/src/origin.test.ts | 29 +- js/net/src/origin.ts | 24 +- js/net/src/resume.test-util.ts | 23 + js/net/src/resume.test.ts | 292 +++++++ js/net/src/resume.ts | 844 ++++++++++++++++++++ js/net/src/track.test.ts | 84 ++ js/net/src/track.ts | 335 +++++--- test/drain/README.md | 28 +- test/drain/drain.ts | 135 ++-- test/drain/relay.toml | 7 + test/interop/README.md | 2 +- test/interop/clients/js-native/subscribe.ts | 2 +- 20 files changed, 2148 insertions(+), 281 deletions(-) create mode 100644 js/net/src/delivered.test.ts create mode 100644 js/net/src/delivered.ts create mode 100644 js/net/src/resume.test-util.ts create mode 100644 js/net/src/resume.test.ts create mode 100644 js/net/src/resume.ts diff --git a/doc/lib/js/net.md b/doc/lib/js/net.md index cb895538573..356c5b05e48 100644 --- a/doc/lib/js/net.md +++ b/doc/lib/js/net.md @@ -36,16 +36,16 @@ broadcast.announce(); - **The origin holds the broadcasts, not the connection.** Closing a session unannounces them and leaves them created for the next one. A broadcast is invisible, locally and remotely, until `announce()`. `dynamic(prefix, route)` claims a prefix and yields each requested path to accept or reject. `accept` throws on a broadcast a session delivered, since JS apps do not proxy; copy its tracks into a broadcast you produce instead. - **Requests can pin an epoch.** `consumer.request(path, { epoch })` resolves only through a route announcing that [publisher epoch](/concept/moq-lite#publisher-epochs) and reports `unroutable` otherwise. The resolved broadcast's `epoch` names the route it came through, pinned or not. - **Rank several CDNs.** Connect to each with its own `consume: origin.withPreference(Origin.Preference(rank))` (higher wins; 0 is neutral and a negative rank demotes), all on one origin. Requests use the preferred link that has a route and fail over to the next; see [multiple CDNs](/concept/moq-lite#multiple-cdns). -- **Follow restarts.** `origin.request(path, { announced: true })` follows whichever publisher instance serves the path: its `active` changes to a fresh broadcast on a restart, clears on an end, and resolves again on the next start, so read it reactively and close the request when done. Instances are never stitched, so subscribe again to each new broadcast. A refusal ends the request. Add `epoch` to pin one instance; a plain `request(path)` keeps its accepted instance, can resolve blindly, and ends once that instance stops serving. To list what is live, iterate `announced(scope)`. +- **Follow restarts.** `origin.request(path, { announced: true })` follows whichever publisher instance serves the path: its `active` changes to a fresh broadcast on a restart, clears on an end, and resolves again on the next start, so read it reactively and close the request when done. Instances are never stitched, so subscribe again to each new broadcast. Another route of the same epoch taking over keeps `active` as is, and each track subscription resumes there from the first group it lacks. A refusal ends the request. Add `epoch` to pin one instance; a plain `request(path)` keeps its accepted instance, can resolve blindly, and ends once that instance stops serving. To list what is live, iterate `announced(scope)`. - **Discovery follows what you watch.** A `consume` origin asks the relay for announcements only under the prefixes its readers watch with `announced(scope)` or `request(path, { announced: true })`, stops asking about a prefix 2s after nothing at or beneath it is watched, and asks again on reconnect. A plain `request(path)` asks nothing and subscribes blind, but once it resolves through a discovered route it watches its path until closed, so it outlives the listing that found it. A relay that refuses an announce request turns that session's announced requests blind too. - **One connection per URL, unless you opt out.** `new Connection({ url })` pools and reconnects with backoff, which the elements use. Supplying your own transport options or origin selects a private loop. Receive estimates are on demand: observe `connection.probe` reactively or subscribe while you need them; `peek()` alone opens no PROBE stream. -- **GOAWAY moves the session.** The connection dials the replacement at once while the old session keeps serving its groups, up to `goaway.handover` (default 10s, or the relay's deadline when that is sooner). New requests keep opening on the old session until the replacement's route outranks it, which is a restart without an epoch: an unpinned announced request follows the replacement, while a blind or epoch-pinned request stays on its accepted instance. An empty GOAWAY redials the same URL. A redirect stays on the same host unless `goaway.redirect` is `"follow"`. +- **GOAWAY moves the session.** The connection dials the replacement at once while the old session keeps serving its groups, up to `goaway.handover` (default 10s, or the relay's deadline when that is sooner). New requests keep opening on the old session until the replacement's route outranks it, which is a restart without an epoch: an unpinned announced request follows the replacement, while a blind or epoch-pinned request stays on its accepted instance. With an epoch the replacement serves the same broadcast, so open subscriptions move across without missing a group. An empty GOAWAY redials the same URL. A redirect stays on the same host unless `goaway.redirect` is `"follow"`. - **One send estimate per connection.** `Bandwidth.Allocator` divides it by track priority, max-min fair within a tier. An idle track claims nothing. Publishers reserve against it so their targets sum to the estimate. - **One track per name.** Concurrent subscriptions share one request and producer. `createTrack` on a name with a request still queued answers it. The application owns sequence numbers: a track that replaces an ended one, created or requested, throws on an append that would reuse a sequence the name already wrote, so it writes its first group with an explicit sequence. Only a new broadcast starts over. - **Timing is per track, with no default.** A track that declares a `timescale` carries a `timestamp` on every frame; one without is [untimed](/concept/moq-lite#subscriptions) and its frames carry none. Omitting it means untimed, as in Rust. Nothing stamps a frame for you: the publisher passes the timestamp, and the `Timed` type (`{ value, at }`) carries one into the `@moq/json` and `@moq/flate` producers. - **Subscriber staleness is `maxDelay`.** It counts from a group's successor, by media time and by wall clock, whichever reaches it first. Passing the old `maxAge` key throws a `TypeError` naming `maxDelay`. Publisher retention is the separate `Track.Info.maxAge`. - **Hidden paths** stay out of discovery unless the announce request opts in. See [hidden broadcasts](/concept/moq-lite#hidden-broadcasts). -- **Authorization is in band.** On moq-lite 07 (offered only when `moq-lite-07-wip` is listed in `webtransport.protocols`; the WebSocket fallback cannot offer it) and on moq-transport draft-17+ with the [MoQ Auth extension](/draft/moq-auth), `auth.grant` watches what the relay lets this side publish and subscribe to, and `auth.add(token)` presents another token without reconnecting. Publishing outside the grant closes the connection with `SessionCode.Unauthorized`, naming the path. On moq-lite, a subscription the grant stops covering resets with `StreamCode.Unauthorized` and the session stays up. +- **Authorization is in band.** On moq-lite 07 (offered only when `moq-lite-07-wip` is listed in `webtransport.protocols` or `websocket.protocols`) and on moq-transport draft-17+ with the [MoQ Auth extension](/draft/moq-auth), `auth.grant` watches what the relay lets this side publish and subscribe to, and `auth.add(token)` presents another token without reconnecting. Publishing outside the grant closes the connection with `SessionCode.Unauthorized`, naming the path. On moq-lite, a subscription the grant stops covering resets with `StreamCode.Unauthorized` and the session stays up. - **A graceful close waits.** `await connection.close()` withdraws announcements and drains for up to one second. Application delivery requires explicit `moq-lite-07-wip`; older Lite versions wait only for transport ACKs. IETF media streams are not drained. `abort()` ends immediately. Examples: diff --git a/js/net/src/connection/connect.test.ts b/js/net/src/connection/connect.test.ts index 1ebf9f65845..cae8f363b98 100644 --- a/js/net/src/connection/connect.test.ts +++ b/js/net/src/connection/connect.test.ts @@ -292,3 +292,54 @@ test("SETUP preserves a delayed session close code", async () => { jest.useRealTimers(); } }); + +// The WebSocket subprotocols one connect() attempt asks a server for. +async function websocketOffer(props: Omit): Promise { + const original = globalThis.WebTransport; + // No WebTransport, so the WebSocket attempt goes at once. + globalThis.WebTransport = undefined as unknown as typeof WebTransport; + let offer: string[] = []; + const server = Bun.serve({ + port: 0, + fetch(request) { + offer = (request.headers.get("sec-websocket-protocol") ?? "").split(",").map((protocol) => protocol.trim()); + return new Response("no", { status: 400 }); + }, + }); + const controller = new AbortController(); + try { + const attempt = connect(new URL(`http://127.0.0.1:${server.port}/`), { + signal: controller.signal, + ...props, + }).catch(() => undefined); + for (let i = 0; i < 100 && offer.length === 0; i++) await settle(); + controller.abort(); + await attempt; + return offer; + } finally { + globalThis.WebTransport = original; + server.stop(true); + } +} + +test("the WebSocket fallback offers lite-07 only when asked", async () => { + const defaults = await websocketOffer({}); + expect(defaults.some((protocol) => protocol.endsWith(ALPN_06))).toBe(true); + expect(defaults.some((protocol) => protocol.endsWith(ALPN_07_WIP))).toBe(false); + + const asked = await websocketOffer({ websocket: { protocols: [ALPN_07_WIP, ALPN_06] } }); + expect(asked[0]?.endsWith(ALPN_07_WIP)).toBe(true); + expect(asked.some((protocol) => protocol.endsWith(ALPN_05))).toBe(false); +}); + +test("the WebSocket fallback refuses an unsupported protocol", async () => { + const original = globalThis.WebTransport; + globalThis.WebTransport = undefined as unknown as typeof WebTransport; + try { + await expect(connect(url, { websocket: { protocols: ["moq-lite-99"] } })).rejects.toThrow( + "unsupported WebSocket protocol", + ); + } finally { + globalThis.WebTransport = original; + } +}); diff --git a/js/net/src/connection/connect.ts b/js/net/src/connection/connect.ts index ff07a31c128..5b28030826b 100644 --- a/js/net/src/connection/connect.ts +++ b/js/net/src/connection/connect.ts @@ -33,6 +33,13 @@ export interface WebSocketProps { * If WebSocket won the previous race for a given URL, this is 0. */ delay?: Milli; + + /** + * The protocols to offer, most preferred first, as `webtransport.protocols` does for + * WebTransport. Defaults to every supported protocol except the work-in-progress + * `moq-lite-07-wip`; an unsupported one throws. + */ + protocols?: string[]; } /** @@ -180,6 +187,9 @@ async function connectInner(url: URL, props: Omit, abort: P return connectTransport(url, transport, wiring); } + // Resolved before racing, so a misconfigured protocol fails the call rather than one transport. + const versions = props.websocket?.enabled !== false ? websocketVersions(props.websocket?.protocols) : undefined; + // Stop transports after one connects or the caller aborts. const { promise: raced, resolve: done } = Promise.withResolvers(); const cancel = Promise.race([raced, abort]); @@ -190,10 +200,7 @@ async function connectInner(url: URL, props: Omit, abort: P // NOTE that QUIC should be faster because it involves 1/2 fewer RTTs. const headstart = !webtransport || websocketWon.has(url.toString()) ? 0 : (props.websocket?.delay ?? DEFAULT_WEBSOCKET_DELAY_MS); - const websocket = - props.websocket?.enabled !== false - ? connectWebSocket(props.websocket?.url ?? url, headstart, cancel) - : undefined; + const websocket = versions ? connectWebSocket(props.websocket?.url ?? url, headstart, cancel, versions) : undefined; if (!websocket && !webtransport) { throw new Error("no transport available; WebTransport not supported and WebSocket is disabled"); @@ -519,35 +526,56 @@ async function connectWebTransport( return quic; } +// moq-transport-18 and newer are pinned to qmux-01. Every other ALPN we +// support is currently negotiated as `qmux-00.{alpn}` on the wire, but we +// don't want to lock that in: set the value to `null` so the polyfill +// advertises every QMux draft it knows about and the server picks one. +// Insertion order is the negotiation preference on the wire. +const WEBSOCKET_VERSIONS = { + [Lite.ALPN_06]: null, + [Lite.ALPN_05]: null, + [Lite.ALPN_04]: null, + [Lite.ALPN_03]: null, + [Lite.ALPN]: null, + [Ietf.ALPN.DRAFT_22]: "qmux-01", + [Ietf.ALPN.DRAFT_21]: "qmux-01", + [Ietf.ALPN.DRAFT_20]: "qmux-01", + [Ietf.ALPN.DRAFT_19]: "qmux-01", + [Ietf.ALPN.DRAFT_18]: "qmux-01", + [Ietf.ALPN.DRAFT_17]: null, + [Ietf.ALPN.DRAFT_16]: null, + [Ietf.ALPN.DRAFT_15]: null, +} as const satisfies Record; + +// The QMux versions to offer per ALPN: `protocols` in their order, or the defaults. Lite-07 is +// work-in-progress, so it is offered only when asked for. +function websocketVersions(protocols?: string[]): Record { + if (!protocols) return WEBSOCKET_VERSIONS; + const supported: Record = { + [Lite.ALPN_07_WIP]: null, + ...WEBSOCKET_VERSIONS, + }; + const versions: Record = {}; + for (const protocol of protocols) { + const version = supported[protocol]; + if (version === undefined) throw new Error(`unsupported WebSocket protocol: ${protocol}`); + versions[protocol] = version; + } + return versions; +} + // TODO accept arguments to control the port/path used. -async function connectWebSocket(url: URL, delay: number, cancel: Promise): Promise { +async function connectWebSocket( + url: URL, + delay: number, + cancel: Promise, + versions: Record, +): Promise { const timer = sleep(Nano.fromMilli(Milli(delay))); const active = await Promise.race([cancel, timer.then(() => true)]); if (!active) return undefined; - // moq-transport-18 and newer are pinned to qmux-01. Every other ALPN we - // support is currently negotiated as `qmux-00.{alpn}` on the wire, but we - // don't want to lock that in: set the value to `null` so the polyfill - // advertises every QMux draft it knows about and the server picks one. - // Insertion order is the negotiation preference on the wire. - const versions = { - // Lite.ALPN_07_WIP omitted on purpose: lite-07 is work-in-progress, not advertised by default. - [Lite.ALPN_06]: null, - [Lite.ALPN_05]: null, - [Lite.ALPN_04]: null, - [Lite.ALPN_03]: null, - [Lite.ALPN]: null, - [Ietf.ALPN.DRAFT_22]: "qmux-01", - [Ietf.ALPN.DRAFT_21]: "qmux-01", - [Ietf.ALPN.DRAFT_20]: "qmux-01", - [Ietf.ALPN.DRAFT_19]: "qmux-01", - [Ietf.ALPN.DRAFT_18]: "qmux-01", - [Ietf.ALPN.DRAFT_17]: null, - [Ietf.ALPN.DRAFT_16]: null, - [Ietf.ALPN.DRAFT_15]: null, - } as const satisfies Record; - // The default (`requireProtocol: false`) also advertises bare `qmux-01`, // `qmux-00`, and `webtransport` so we still interop with relays that only // know a wire-format version (today's moq-relay only accepts bare diff --git a/js/net/src/delivered.test.ts b/js/net/src/delivered.test.ts new file mode 100644 index 00000000000..30cc2319b61 --- /dev/null +++ b/js/net/src/delivered.test.ts @@ -0,0 +1,55 @@ +import { expect, test } from "bun:test"; +import { Delivered } from "./delivered.ts"; + +test("add joins neighbouring runs", () => { + const delivered = new Delivered(); + expect(delivered.add(5)).toBe(true); + expect(delivered.add(7)).toBe(true); + expect(delivered.runs).toBe(2); + expect(delivered.add(6)).toBe(true); + expect(delivered.runs).toBe(1); + expect(delivered.add(6)).toBe(false); + expect(delivered.add(4)).toBe(true); + expect(delivered.add(8)).toBe(true); + expect(delivered.runs).toBe(1); + expect(delivered.has(4) && delivered.has(8)).toBe(true); + expect(delivered.has(3) || delivered.has(9)).toBe(false); +}); + +test("the extremes are sequences too", () => { + const delivered = new Delivered(); + const max = Number.MAX_SAFE_INTEGER; + expect(delivered.add(0)).toBe(true); + expect(delivered.add(max)).toBe(true); + expect(delivered.add(max)).toBe(false); + expect(delivered.runs).toBe(2); + delivered.forgetBelow(max); + expect(delivered.runs).toBe(1); + expect(delivered.aboveGap()).toBeUndefined(); +}); + +test("forgetBelow closes the gaps under the floor", () => { + const delivered = new Delivered(); + for (const sequence of [3, 5, 6, 9, 12]) delivered.add(sequence); + expect(delivered.aboveGap()).toBe(3); + delivered.forgetBelow(0); + expect(delivered.runs).toBe(4); + + // A floor at a run's start takes the run along. + delivered.forgetBelow(5); + expect(delivered.runs).toBe(3); + expect(delivered.aboveGap()).toBe(9); + expect(delivered.has(4)).toBe(true); + expect(delivered.has(7)).toBe(false); + + // A floor inside a gap keeps the run above it. + delivered.forgetBelow(11); + expect(delivered.runs).toBe(2); + expect(delivered.has(10)).toBe(true); + expect(delivered.aboveGap()).toBe(12); + delivered.forgetBelow(13); + expect(delivered.runs).toBe(1); + expect(delivered.aboveGap()).toBeUndefined(); + expect(delivered.add(13)).toBe(true); + expect(delivered.runs).toBe(1); +}); diff --git a/js/net/src/delivered.ts b/js/net/src/delivered.ts new file mode 100644 index 00000000000..713b8392b9b --- /dev/null +++ b/js/net/src/delivered.ts @@ -0,0 +1,81 @@ +/** + * The group sequences a reader handed out, so it never hands one out twice. Mirrors + * `model::delivered` in `rs/moq-net`. + * + * @module + */ + +/** + * The group sequences a reader handed out, as runs of consecutive sequences. + * + * A reader stops expecting a group once it is past its budget, and counts everything below it + * as handed out (see {@link forgetBelow}). The run starting at 0 is that floor, so only the + * gaps above it cost memory. + */ +export class Delivered { + // First and last sequence of each run, inclusive, sorted and never adjacent to each other. + #runs: [number, number][] = []; + + // The index of the run containing or preceding `sequence`, or -1 when none starts at or below it. + #below(sequence: number): number { + let lo = 0; + let hi = this.#runs.length; + while (lo < hi) { + const mid = (lo + hi) >>> 1; + if (this.#runs[mid][0] <= sequence) lo = mid + 1; + else hi = mid; + } + return lo - 1; + } + + /** Whether `sequence` was handed out, or is below the floor. */ + has(sequence: number): boolean { + const index = this.#below(sequence); + return index >= 0 && sequence <= this.#runs[index][1]; + } + + /** Record `sequence` as handed out; `false` when it already was. */ + add(sequence: number): boolean { + const index = this.#below(sequence); + const run = this.#runs[index]; + if (run && sequence <= run[1]) return false; + const next = this.#runs[index + 1]; + const joinsBelow = run !== undefined && run[1] + 1 === sequence; + const joinsAbove = next !== undefined && next[0] === sequence + 1; + if (joinsBelow && joinsAbove) { + run[1] = next[1]; + this.#runs.splice(index + 1, 1); + } else if (joinsBelow) { + run[1] = sequence; + } else if (joinsAbove) { + next[0] = sequence; + } else { + this.#runs.splice(index + 1, 0, [sequence, sequence]); + } + return true; + } + + /** Count every sequence below `floor` as handed out. */ + forgetBelow(floor: number): void { + if (floor <= 0) return; + // Every run starting at or below the floor joins it, including one starting right at it. + const index = this.#below(floor); + const last = Math.max(floor - 1, index >= 0 ? this.#runs[index][1] : 0); + this.#runs.splice(0, index + 1, [0, last]); + } + + /** + * The lowest sequence handed out above a gap: once it is past the reader's budget, so is + * the gap, since group timestamps on a track never rewind. + */ + aboveGap(): number | undefined { + const first = this.#runs[0]; + if (!first) return undefined; + return first[0] === 0 ? this.#runs[1]?.[0] : first[0]; + } + + /** How many runs it holds: one per gap, plus the floor. */ + get runs(): number { + return this.#runs.length; + } +} diff --git a/js/net/src/group.ts b/js/net/src/group.ts index f4b32d38f66..fdc1c6a7ace 100644 --- a/js/net/src/group.ts +++ b/js/net/src/group.ts @@ -4,9 +4,9 @@ * @module */ import { Instant, now } from "@moq/clock"; -import { type Dispose, type GetPromise, type Getter, Once, Signal } from "@moq/signals"; +import { type Dispose, type GetPromise, type Getter, Once, race, Signal } from "@moq/signals"; import { FrameTooLarge, GroupTooLarge, ProtocolViolation, TimestampMismatch, TooFarBehind } from "./error.ts"; -import { hooks, type ReadGroupFrame } from "./internal.ts"; +import { type Expiry, type GroupRecover, hooks, type ReadGroupFrame, type Replacement } from "./internal.ts"; import type { Timestamp } from "./time.ts"; /** Maximum bytes of frames in a group. A write that would exceed this aborts with {@link GroupTooLarge}. */ @@ -314,43 +314,57 @@ export class Producer { let makeConsumer: (state: GroupState) => Consumer; +// A replacement copy asked for on behalf of one consumer, settled or still out. +interface Attempt { + abort: AbortController; + result: Promise; + settled: boolean; + replacement?: Replacement; +} + // A consumer can finish from its own expiry verdict or from the shared group state. Keep // one stable public handle while letting peek() observe either source synchronously, before -// signal subscribers run in the next microtask. +// signal subscribers run in the next microtask. The source is re-read on every check, since a +// recovered group moves to another copy's state, and `final` says whether its value ends the +// consumer: a failure recovery can still continue does not. class CombinedClosed implements GetPromise { #preferred: Once; - #source: Once; - #sourceCleanReady: () => boolean; + #source: () => Once; + #final: (value: Error | null) => boolean; #value = new Once(); - #dispose?: Dispose; - - constructor(preferred: Once, source: Once, sourceCleanReady: () => boolean) { + #watched?: Once; + #dispose: Dispose[] = []; + + constructor( + preferred: Once, + source: () => Once, + final: (value: Error | null) => boolean, + ) { this.#preferred = preferred; this.#source = source; - this.#sourceCleanReady = sourceCleanReady; + this.#final = final; if (this.#sync()) return; - - const dispose = [preferred.changed(() => this.#sync()), source.changed(() => this.#sync())]; - this.#dispose = () => { - for (const close of dispose) close(); - }; + this.#dispose.push(preferred.changed(() => this.#sync())); } #sync(): boolean { if (this.#value.peek() !== undefined) return true; const preferred = this.#preferred.peek(); - const source = this.#source.peek(); + const source = this.#source(); + const current = source.peek(); const value = - preferred !== undefined - ? preferred - : source instanceof Error || this.#sourceCleanReady() - ? source - : undefined; - if (value === undefined) return false; + preferred !== undefined ? preferred : current !== undefined && this.#final(current) ? current : undefined; + if (value === undefined) { + if (this.#watched !== source) { + this.#watched = source; + this.#dispose.push(source.changed(() => this.#sync())); + } + return false; + } this.#value.set(value); - this.#dispose?.(); - this.#dispose = undefined; + for (const dispose of this.#dispose) dispose(); + this.#dispose = []; return true; } @@ -398,7 +412,13 @@ export class Consumer { readonly sequence: number; #state: GroupState; - #expiry?: { expired: () => boolean; changed: readonly Getter[] }; + #expiry?: Expiry; + // Carries the group across its logical track's route changes; see `resume.ts`. + #recover?: GroupRecover; + // A replacement asked for while the copy being read stalls, raced against that copy. + #attempt?: Attempt; + // The frame a replacement copy resumes at: frames below it were read from an earlier copy. + #skip = 0; // Sticky verdicts, set once. `#ended` is a drained group the budget gave up on, // which is indistinguishable from one that ended: nothing was lost, so it reads as // the end of the group. `#terminal` is a failure, including a budget that gave up @@ -422,12 +442,20 @@ export class Consumer { get closed(): GetPromise { this.#closed ??= new CombinedClosed( this.#verdict, - this.#state.closed, - () => this.#state.frames.peek().length === 0 && this.#pendingFrames === 0, + () => this.#state.closed, + (value) => + value instanceof Error + ? !this.#recovers(value) + : this.#state.frames.peek().length === 0 && this.#pendingFrames === 0, ); return this.#closed; } + // Whether a failure of the copy being read may still be continued from another route. + #recovers(failed: Error): boolean { + return !this.#terminal && !this.#ended && this.#recover?.wants(failed) === true; + } + static { makeConsumer = (state) => new Consumer(state); hooks.groupTimestamp = (group) => group.#state.timestamp; @@ -439,6 +467,9 @@ export class Consumer { hooks.expireGroup = (group, expiry) => { group.#expiry = expiry; }; + hooks.recoverGroup = (group, recover) => { + group.#recover = recover; + }; hooks.groupExpired = (group) => group.#ended; hooks.guardGroup = (group, operation) => group.#guard(operation); hooks.readGroupFrame = (group, from) => group.#readFramePosition(true, from); @@ -523,15 +554,85 @@ export class Consumer { } async #changed(): Promise { - if (!this.#expiry) { - await Signal.race(this.#state.frames, this.#state.closed); + await this.#changes(); + } + + // The next change a parked read waits for. + #changes(): GetPromise { + const changed: Getter[] = [this.#state.frames, this.#state.closed]; + if (this.#expiry) changed.push(...this.#expiry.changed); + if (this.#recover) changed.push(this.#recover.changed); + return Signal.race(...changed); + } + + // Continue from the serving route's copy once the one being read failed with `failed`, or + // stalled (no `failed`) while a newer route serves. Returns whether the read should look + // again; throws the error that ends the group once nothing can continue it. + async #resume(failed: Error | undefined): Promise { + const recover = this.#recover; + if (!recover?.wants(failed)) return false; + let attempt = this.#attempt; + if (!attempt) { + const abort = new AbortController(); + const started: Attempt = { abort, result: Promise.resolve(), settled: false }; + started.result = recover.replace(this.#state.start, failed, abort.signal).then( + (replacement) => { + started.settled = true; + started.replacement = replacement; + }, + (err: unknown) => { + started.settled = true; + if (abort.signal.aborted) return; + this.#terminal ??= err instanceof Error ? err : new Error(String(err)); + if (this.#verdict.peek() === undefined) this.#verdict.set(this.#terminal); + }, + ); + attempt = started; + this.#attempt = attempt; + } + // A stalled copy may still deliver first; a failed one never will. + if (failed) await attempt.result; + else await race([attempt.result, this.#changes()]); + // The copy being read moved first: look again, keeping the attempt out. + if (!attempt.settled) return true; + if (this.#attempt === attempt) this.#attempt = undefined; + const replacement = attempt.replacement; + if (!replacement) return true; + if (!failed && this.#state.frames.peek().length > 0) { + // The copy being read delivered meanwhile; keep reading it. + replacement.group.close(); + return true; + } + this.#adopt(replacement); + return true; + } + + // Read the rest of the group from `replacement`, from the frame this one reached. + #adopt(replacement: Replacement): void { + const next = replacement.group.#state; + const index = this.#state.start; + if (next.start > index) { + replacement.group.close(); + this.#terminal ??= new Error(`group ${this.sequence} cannot resume at frame ${index}`); + if (this.#verdict.peek() === undefined) this.#verdict.set(this.#terminal); return; } - await Signal.race(this.#state.frames, this.#state.closed, ...this.#expiry.changed); + replacement.adopt(); + // The copy left behind stops teeing into the state this consumer read. + this.#state.closed.peek() ?? this.#state.closed.set(null); + this.#state = next; + this.#expiry = replacement.expiry; + this.#skip = index; + this.#closed?.refresh(); } #readBufferedFrame(pending = false): ReadGroupFrame | undefined { const frames = this.#state.frames.peek(); + // A resumed copy repeats the frames an earlier one already delivered. + while (this.#state.start < this.#skip && frames.length > 0) { + frames.shift(); + this.#state.start++; + } const buffered = frames.shift(); if (!buffered) return undefined; @@ -553,15 +654,19 @@ export class Consumer { /** True once no further frames can be read: the group has closed and every buffered frame is read. */ get done(): boolean { return ( - this.#terminal !== undefined || - this.#ended || - (this.#state.frames.peek().length === 0 && this.#state.closed.peek() !== undefined) + this.#terminal !== undefined || this.#ended || (this.#state.frames.peek().length === 0 && this.#settled()) ); } /** True once the group has been closed, regardless of whether buffered frames remain unread. Synchronous complement to the {@link closed} promise. */ get isClosed(): boolean { - return this.#terminal !== undefined || this.#ended || this.#state.closed.peek() !== undefined; + return this.#terminal !== undefined || this.#ended || this.#settled(); + } + + // Whether the copy being read ended, for good: a failure recovery may continue is not an end. + #settled(): boolean { + const closed = this.#state.closed.peek(); + return closed !== undefined && !(closed instanceof Error && this.#recovers(closed)); } /** @@ -602,11 +707,17 @@ export class Consumer { for (;;) { if (this.#terminal || this.#ended) return; if (this.#state.frames.peek().length > 0) return; - if (this.#state.closed.peek() !== undefined) { + const closed = this.#state.closed.peek(); + if (closed instanceof Error && (await this.#resume(closed))) continue; + if (closed !== undefined) { this.#closed?.refresh(); return; } if (this.#expire()) return; + if (this.#recover?.wants(undefined)) { + await this.#resume(undefined); + continue; + } await this.#changed(); } } @@ -635,7 +746,10 @@ export class Consumer { } const closed = this.#state.closed.peek(); - if (closed instanceof Error) throw closed; + if (closed instanceof Error) { + if (await this.#resume(closed)) continue; + throw closed; + } if (closed !== undefined) { this.#closed?.refresh(); return; @@ -644,6 +758,11 @@ export class Consumer { // Nothing buffered and the group is still open: this wait is the stall the // drift budget bounds, and the only place it applies. if (this.#expire()) continue; + // A stall while a newer route serves looks there too. + if (this.#recover?.wants(undefined)) { + await this.#resume(undefined); + continue; + } await this.#changed(); } } @@ -678,6 +797,11 @@ export class Consumer { /** Closes the group, optionally with an error to abort readers. Idempotent. */ close(abort?: Error) { + this.#attempt?.abort.abort(); + this.#attempt = undefined; + this.#recover = undefined; + // Closing is terminal for this reader, even with frames still unread. + if (this.#verdict.peek() === undefined) this.#verdict.set(abort ?? null); if (this.#state.closed.peek() !== undefined) return; // already closed this.#state.closed.set(abort ?? null); } diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index cc549267203..25cbb791a0e 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -6,14 +6,16 @@ * @module */ import type { Instant, Milli } from "@moq/clock"; -import type { Dispose, Getter } from "@moq/signals"; +import type { Dispose, GetPromise, Getter } from "@moq/signals"; import type { Consumer as BroadcastConsumer, Producer as BroadcastProducer } from "./broadcast.ts"; +import type { Datagram } from "./datagram.ts"; +import type { Delivered } from "./delivered.ts"; import type * as Epoch from "./epoch.ts"; import type { Frame, Consumer as GroupConsumer, Producer as GroupProducer } from "./group.ts"; import { isAnonymous, type Route } from "./hop.ts"; import * as Path from "./path.ts"; import type { Timestamp } from "./time.ts"; -import type { Groups, Location, Producer, Request, Subscriber } from "./track.ts"; +import type { Groups, Info, Location, Producer, Request, Subscriber, Subscription } from "./track.ts"; import type { Advertised, Advertisements } from "./wire.ts"; /** Normalize public group bounds into an inclusive start and exclusive end. */ @@ -157,6 +159,91 @@ export class TrackSequences { } } +/** What a resumed reader's {@link ResumeSource} sees of the subscriber it serves. */ +export interface ResumeShell { + /** The reader's own preferences, mirrored into every copy it subscribes. */ + readonly subscription: Getter; + /** The reader's group cursor: an inclusive start and an exclusive end. */ + readonly cursor: Getter<{ start: number; end?: number }>; + /** Settles once the logical track ends. */ + readonly closed: GetPromise; + /** The properties the first copy committed, if any yet. */ + info(): Info | undefined; + /** Commit the logical track's properties, once. */ + commit(info: Info): void; + /** End the logical track: cleanly, or with `abort`. */ + close(abort?: Error): void; +} + +/** A drift policy a handed-out group keeps applying; see `hooks.expireGroup`. */ +export interface Expiry { + expired: () => boolean; + changed: readonly Getter[]; +} + +/** + * Carries a handed-out group across its logical track's route changes. Consulted only once + * the copy it reads fails, or stalls while a newer route serves, so a passthrough read never + * touches it. The counterpart of the Rust `resume::Recover`. + */ +export interface GroupRecover { + /** + * Whether to look past the copy being read: it failed with `failed` and that is no verdict + * on the group, or (with none) it stalled while a newer route serves. + */ + wants(failed: Error | undefined): boolean; + /** Wakes a stalled read once the route changes. */ + readonly changed: Getter; + /** + * The serving route's copy of the group, to read from frame `index` on. Rejects with the + * error that ends the group once nothing can continue it; abandoned once `signal` aborts. + */ + replace(index: number, failed: Error | undefined, signal: AbortSignal): Promise; +} + +/** A candidate copy of a group, committed only once the read can use it. */ +export interface Replacement { + readonly group: GroupConsumer; + /** The drift policy of the copy it came from. */ + readonly expiry: Expiry; + /** The group reads this copy from now on. */ + adopt(): void; +} + +/** A datagram read that does not wait. */ +export type RecvDatagram = + | { kind: "datagram"; datagram: Datagram } + | { kind: "idle" } + | { kind: "done" } + | { kind: "error"; error: Error }; + +/** + * The reads a resumed {@link Subscriber} delegates: it reads its routes' copies rather than a + * buffer of its own. Implemented by `resume.ts`, the counterpart of the Rust `Inner::Resume`. + */ +export interface ResumeSource { + /** The next group not in `delivered`, in arrival or sequence order, without waiting. */ + tryGroup(ordered: boolean, delivered: Delivered): Recv; + /** The next datagram in arrival order, without waiting. */ + tryDatagram(): RecvDatagram; + /** Wake once a read may go differently; fires at most once. */ + changed(fn: () => void): Dispose; + /** Forget the gaps in `delivered` below each group past the budget, judged on the newest copy. */ + forgetPastBudget(delivered: Delivered): void; + /** The newest sequence any copy has seen. */ + latest(): number | undefined; + /** The serving copy's declared exclusive end. */ + end(): Location | undefined; + /** Resolve with the logical track's exclusive end once known. */ + finished(): Promise; + /** The serving copy's newest cached frame. */ + largest(): Location | undefined; + /** An independent subscriber to the serving copy. */ + fork(options?: Subscription): Subscriber; + /** Let go of every copy. */ + close(abort?: Error): void; +} + /** Inputs for creating a package-internal track request. */ export interface TrackRequestOptions { /** The requested track name. */ @@ -197,6 +284,28 @@ export const hooks: { * {@link groupChanged} when it reports `idle` or `boundary`. */ tryRecvGroup: (subscriber: Subscriber) => Recv; + /** Take the next group in sequence order without waiting, as {@link tryRecvGroup} does in arrival order. */ + tryNextGroup: (subscriber: Subscriber) => Recv; + /** Wake once a subscriber's datagram cursor may read differently; fires at most once. */ + datagramChanged: (subscriber: Subscriber, fn: () => void) => Dispose; + /** Take the next datagram in arrival order without waiting. */ + tryRecvDatagram: (subscriber: Subscriber) => RecvDatagram; + /** Whether a buffered group in `[start, end)` that `delivered` lacks still waits to be read. */ + hasUnreadGroup: (subscriber: Subscriber, start: number, end: number | undefined, delivered: Delivered) => boolean; + /** How many runs of handed-out groups a subscriber remembers: one per gap, plus the floor. Tests only. */ + deliveredRuns: (subscriber: Subscriber) => number; + /** Forget the gaps in another reader's `delivered` below each group past this subscriber's budget. */ + forgetPastBudget: (subscriber: Subscriber, delivered: Delivered) => void; + /** Whether this subscriber's budget gives up on `sequence`, judged by its successor; see {@link expireGroup}. */ + staleAt: (subscriber: Subscriber, sequence: number) => Expiry; + /** A group the subscriber's track still caches, as a fresh read handle. */ + cachedGroup: (subscriber: Subscriber, sequence: number) => GroupConsumer | undefined; + /** A subscriber that reads its routes' copies through `source` instead of a buffer of its own. */ + resumeSubscriber: ( + name: string, + subscription: Subscription | undefined, + source: (shell: ResumeShell) => ResumeSource, + ) => Subscriber; /** Wake once a subscriber's group cursor may read differently; assigned by `track.ts`. */ groupChanged: (subscriber: Subscriber, fn: () => void) => Dispose; /** Apply a FETCH reader budget without inheriting SUBSCRIBE's zero default. */ @@ -225,10 +334,9 @@ export const hooks: { /** Record when a track's mirror of a group joined that track at this hop. */ arriveGroup: (group: GroupConsumer, at: Instant) => void; /** Keep applying a subscription's drift policy after it hands a group out. */ - expireGroup: ( - group: GroupConsumer, - expiry: { expired: () => boolean; changed: readonly Getter[] }, - ) => void; + expireGroup: (group: GroupConsumer, expiry: Expiry) => void; + /** Carry a handed-out group across its logical track's route changes. */ + recoverGroup: (group: GroupConsumer, recover: GroupRecover) => void; /** Whether a drift budget gave up on a drained group, which reads as a clean end. */ groupExpired: (group: GroupConsumer) => boolean; /** Start a group operation unless the handed-out group has expired, and stop it if the group expires mid-flight. */ @@ -265,6 +373,33 @@ export const hooks: { tryRecvGroup: () => { throw new Error("track.ts not loaded"); }, + tryNextGroup: () => { + throw new Error("track.ts not loaded"); + }, + forgetPastBudget: () => { + throw new Error("track.ts not loaded"); + }, + tryRecvDatagram: () => { + throw new Error("track.ts not loaded"); + }, + deliveredRuns: () => { + throw new Error("track.ts not loaded"); + }, + datagramChanged: () => { + throw new Error("track.ts not loaded"); + }, + hasUnreadGroup: () => { + throw new Error("track.ts not loaded"); + }, + staleAt: () => { + throw new Error("track.ts not loaded"); + }, + cachedGroup: () => { + throw new Error("track.ts not loaded"); + }, + resumeSubscriber: () => { + throw new Error("track.ts not loaded"); + }, groupChanged: () => { throw new Error("track.ts not loaded"); }, @@ -295,6 +430,9 @@ export const hooks: { expireGroup: () => { throw new Error("group.ts not loaded"); }, + recoverGroup: () => { + throw new Error("group.ts not loaded"); + }, groupExpired: () => { throw new Error("group.ts not loaded"); }, diff --git a/js/net/src/origin-demand.test.ts b/js/net/src/origin-demand.test.ts index 3f1291c18c8..79236cf59e7 100644 --- a/js/net/src/origin-demand.test.ts +++ b/js/net/src/origin-demand.test.ts @@ -14,6 +14,7 @@ import { Version as LiteVersion } from "./lite/version.ts"; import { createMockTransportPair } from "./mock.ts"; import { Producer } from "./origin.ts"; import * as Path from "./path.ts"; +import { readsFrom } from "./resume.test-util"; import { Stream } from "./stream.ts"; import { wireOf } from "./wire.ts"; @@ -381,7 +382,7 @@ test.each(["answer", "release"] as const)( if (action === "release") expect(close).toHaveBeenCalledTimes(1); else { expect(close).not.toHaveBeenCalled(); - expect(held.active.peek()?.closed).toBe(late.closed); + expect(readsFrom(held.active.peek(), late)).toBe(true); } } finally { close.mockRestore(); @@ -439,7 +440,7 @@ test.each([0, 1])( expect(close).toHaveBeenCalledTimes(1); pending[1 - released].accept(replacement); await wait; - expect(entries[1 - released].held.active.peek()?.closed).toBe(replacement.closed); + expect(readsFrom(entries[1 - released].held.active.peek(), replacement)).toBe(true); } finally { close.mockRestore(); for (const entry of entries) entry.held.close(); diff --git a/js/net/src/origin.test.ts b/js/net/src/origin.test.ts index 74e7aaa5286..132044695d0 100644 --- a/js/net/src/origin.test.ts +++ b/js/net/src/origin.test.ts @@ -10,6 +10,7 @@ import { spreadHash } from "./internal.ts"; import type { Consumer, Table } from "./origin.ts"; import { Preference, Producer } from "./origin.ts"; import * as Path from "./path.ts"; +import { readsFrom } from "./resume.test-util"; import { Timescale } from "./time.ts"; import { overrideBroadcastWire, wireOf } from "./wire.ts"; @@ -1123,12 +1124,12 @@ test("a request prefers a cheaper received route over an announced local broadca const announced = consumer.announced(); expect(await announced.next()).toMatchObject({ prefix: path, route: { hops: [PEER] } }); - // Re-priced below it, the local broadcast of the same instance wins at once, and the remote - // front it replaced closes. + // Re-priced below it, the local broadcast of the same instance wins at once: the request's + // broadcast stays, and its tracks read the local one. local.announce({ epoch: EPOCH, cost: 0n }); expect(await announced.next()).toMatchObject({ prefix: path, kind: "update", route: { epoch: EPOCH, hops: [] } }); - expect(request.active.peek()).not.toBe(remote); - expect(remote?.closed.peek()).not.toBeUndefined(); + expect(request.active.peek()).toBe(remote); + expect(readsFrom(remote, local)).toBe(true); announced.close(); request.close(); @@ -1543,13 +1544,13 @@ test("an outranked route keeps serving until its replacement answers", async () await settle(); expect(request.active.peek()).toBe(first); - // Once it answers, the request swaps straight across. + // Once it answers, the request's tracks read it, under the same broadcast. const replacement = new BroadcastProducer(); const { value: req } = await asked; req?.accept(replacement.consume()); await settle(); - expect(request.active.peek()).toBeDefined(); - expect(request.active.peek()).not.toBe(first); + expect(request.active.peek()).toBe(first); + expect(readsFrom(first, replacement)).toBe(true); expect(gaps).toBe(0); stop(); @@ -1662,10 +1663,9 @@ test("a request on a replaced epoch fails over to another route of it", async () disposePrimary(); await settle(); expect(request.closed.peek()).toBeUndefined(); - const failed = request.active.peek(); - expect(failed).toBeDefined(); - expect(failed).not.toBe(first); - expect(failed?.epoch).toBe(EPOCH); + expect(request.active.peek()).toBe(first); + expect(first?.epoch).toBe(EPOCH); + expect(readsFrom(first, replica)).toBe(true); const fresh = consumer.request(path); await settle(); @@ -1711,10 +1711,9 @@ test("a request on a replaced epoch fails over to a local replica of it", async disposePrimary(); await settle(); expect(request.closed.peek()).toBeUndefined(); - const failed = request.active.peek(); - expect(failed).toBeDefined(); - expect(failed).not.toBe(first); - expect(failed?.epoch).toBe(EPOCH); + expect(request.active.peek()).toBe(first); + expect(first?.epoch).toBe(EPOCH); + expect(readsFrom(first, local)).toBe(true); const fresh = consumer.request(path); await settle(); diff --git a/js/net/src/origin.ts b/js/net/src/origin.ts index 1061cfcd81c..e179133ca95 100644 --- a/js/net/src/origin.ts +++ b/js/net/src/origin.ts @@ -29,6 +29,7 @@ import { spreadHash, } from "./internal.ts"; import * as Path from "./path.ts"; +import { Front } from "./resume.ts"; import { type Advertised, type Advertisements, @@ -1533,7 +1534,9 @@ export class Requesting { /** * The current broadcast, or undefined while nothing provides the path. * Announced requests without an epoch follow publisher restarts. Blind and epoch-pinned - * requests retain their accepted instance. Closing this handle affects only this request. + * requests retain their accepted instance. Another route of the same epoch taking over keeps + * this handle: its tracks resume on that route from the first group each reader lacks. + * Closing this handle affects only this request. */ readonly active: Getter; @@ -1837,20 +1840,33 @@ export class Consumer { // Memoized on the route's identity: the same front resolving again returns the handle // we already made, and only a real swap clones a new one (cloning before closing the // old, so a broadcast that both routes share never briefly loses its last handle). + // Under an epoch the handle is a logical front instead: another route of the same epoch + // serves the same broadcast, so the front re-sources its tracks and the handle stays. let released = false; let source: Resolution | undefined; let handle: broadcast.Consumer | undefined; + let logical: Front | undefined; const own = (resolution: Resolution | undefined): broadcast.Consumer | undefined => { if (released) return undefined; const front = epoch === undefined || resolution?.epoch === epoch ? resolution?.front : undefined; + if (logical && source?.epoch !== undefined && resolution?.epoch === source.epoch) { + // Between routes of the instance, readers wait for the next one to answer. + if (front) logical.serve(front); + else logical.park(); + return handle; + } if (front !== source?.front || resolution?.epoch !== source?.epoch) { const previous = handle; + const concluded = logical; source = front ? { front, epoch: resolution?.epoch } : undefined; - handle = front?.clone(); + logical = front && resolution?.epoch !== undefined ? new Front() : undefined; + logical?.serve(front as broadcast.Consumer); + handle = logical ? logical.consumer : front?.clone(); if (handle) { hooks.stampPath(handle, relative); hooks.stampEpoch(handle, resolution?.epoch); } + concluded?.conclude(); previous?.close(); } return handle; @@ -1886,13 +1902,15 @@ export class Consumer { (resolution, answerers, ended) => ended !== undefined || (epoch !== undefined && resolution?.epoch !== epoch) || - (!resolution?.front && answerers === 0), + (!own(resolution) && answerers === 0), ); return makeRequesting(relative, active, unroutable, closed, () => { // Releases this request's handle; the route itself belongs to the table. released = true; unsubscribe(); + logical?.conclude(); + logical = undefined; handle?.close(); handle = undefined; source = undefined; diff --git a/js/net/src/resume.test-util.ts b/js/net/src/resume.test-util.ts new file mode 100644 index 00000000000..8599606f7fb --- /dev/null +++ b/js/net/src/resume.test-util.ts @@ -0,0 +1,23 @@ +import type { Consumer as BroadcastConsumer, Producer as BroadcastProducer } from "./broadcast"; + +let probes = 0; + +/** Whether `active` reads its tracks from `source` now: a group cached there arrives through it. @internal */ +export function readsFrom(active: BroadcastConsumer | undefined, source: BroadcastProducer): boolean { + if (!active) return false; + const name = `probe-${probes++}`; + const track = source.createTrack(name); + const group = track.appendGroup(); + group.writeFrame({ payload: new Uint8Array([1]) }); + group.close(); + const sub = active.track(name).subscribe(); + try { + return sub.tryRecvGroup() !== undefined; + } catch { + return false; + } finally { + sub.close(); + track.close(); + source.removeTrack(name); + } +} diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts new file mode 100644 index 00000000000..fe4e2450447 --- /dev/null +++ b/js/net/src/resume.test.ts @@ -0,0 +1,292 @@ +import { expect, test } from "bun:test"; +import { Producer as BroadcastProducer } from "./broadcast.ts"; +import * as Epoch from "./epoch.ts"; +import { StreamCode, StreamError } from "./error.ts"; +import { Producer as GroupProducer } from "./group.ts"; +import { Route } from "./hop.ts"; +import { Producer } from "./origin.ts"; +import * as Path from "./path.ts"; +import { Front } from "./resume.ts"; +import type * as track from "./track.ts"; +import { wireOf } from "./wire.ts"; + +const EPOCH = Epoch.parse("01900000-0000-7000-8000-000000000001"); +const path = Path.from("live"); +const enc = new TextEncoder(); +const dec = new TextDecoder(); + +async function settle() { + await new Promise((resolve) => setTimeout(resolve, 0)); +} + +/** A stand-in session announcing `path` under the epoch at `cost`, serving `source`. */ +function serve(origin: Producer, source: BroadcastProducer, cost: bigint) { + const handle = wireOf(origin).receive(path, Route.normalize({ epoch: EPOCH, cost })); + void (async () => { + for await (const request of handle.requested()) request.accept(source.consume()); + })(); + return () => handle.close(); +} + +/** A copy of the logical track: every route's copy holds the same groups under the same sequences. */ +function copy(source: BroadcastProducer) { + const producer = source.createTrack("seq"); + return { + producer, + write(sequence: number, frames = 1, close = true) { + const group = new GroupProducer(sequence); + producer.writeGroup(group); + for (let frame = 0; frame < frames; frame++) + group.writeFrame({ payload: enc.encode(`${sequence}.${frame}`) }); + if (close) group.close(); + return group; + }, + }; +} + +/** Every group sequence readable without waiting. */ +function drain(sub: track.Subscriber): number[] { + const sequences: number[] = []; + for (let group = sub.tryRecvGroup(); group; group = sub.tryRecvGroup()) { + sequences.push(group.sequence); + group.close(); + } + return sequences; +} + +async function setup() { + const origin = new Producer(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const disposeA = serve(origin, a, 5n); + const request = origin.consume().request(path); + await settle(); + const active = request.active.peek(); + if (!active) throw new Error("expected the path to resolve"); + return { + origin, + a, + b, + disposeA, + request, + active, + // The cheaper route takes over once it answers. + swap: async () => { + const disposeB = serve(origin, b, 1n); + await settle(); + await settle(); + return disposeB; + }, + close: () => { + request.close(); + disposeA(); + a.close(); + b.close(); + origin.close(); + }, + }; +} + +test("a subscription reads on across a same-epoch route swap, each group once", async () => { + const { a, b, active, request, swap, close } = await setup(); + const ta = copy(a); + const tb = copy(b); + const sub = active.track("seq").subscribe({ maxDelay: 10_000 as track.Subscription["maxDelay"] }); + ta.write(0); + expect(drain(sub)).toEqual([0]); + + // Both routes deliver group 1; the reader takes it once. + ta.write(1); + tb.write(0); + tb.write(1); + await swap(); + expect(request.active.peek()).toBe(active); + tb.write(2); + expect(drain(sub)).toEqual([1, 2]); + + // The replaced route's later groups are the new route's to deliver. + ta.write(3); + expect(drain(sub)).toEqual([]); + tb.write(3); + expect(drain(sub)).toEqual([3]); + + sub.close(); + close(); +}); + +test("a stalled group continues from the replacement at the next frame", async () => { + const { a, b, swap, active, close } = await setup(); + const ta = copy(a); + const tb = copy(b); + const sub = active.track("seq").subscribe(); + ta.write(0, 1, false); + const group = sub.tryRecvGroup(); + if (!group) throw new Error("expected group 0"); + expect(dec.decode((await group.readFrame())?.payload)).toBe("0.0"); + + // The new route has the rest of it; the old one stalls. + tb.write(0, 3); + await swap(); + expect(dec.decode((await group.readFrame())?.payload)).toBe("0.1"); + expect(dec.decode((await group.readFrame())?.payload)).toBe("0.2"); + expect(await group.readFrame()).toBeUndefined(); + + sub.close(); + close(); +}); + +test("a group whose route fails continues from the replacement", async () => { + const { a, b, swap, active, disposeA, close } = await setup(); + const ta = copy(a); + const tb = copy(b); + const sub = active.track("seq").subscribe(); + const stalled = ta.write(0, 1, false); + const group = sub.tryRecvGroup(); + if (!group) throw new Error("expected group 0"); + expect(dec.decode((await group.readFrame())?.payload)).toBe("0.0"); + + tb.write(0, 2); + await swap(); + // The old route's session goes, failing what it was still delivering. + stalled.close(new StreamError(StreamCode.SessionClosed)); + disposeA(); + expect(dec.decode((await group.readFrame())?.payload)).toBe("0.1"); + expect(await group.readFrame()).toBeUndefined(); + + sub.close(); + close(); +}); + +test("a group no route continues is given up once a newer one supersedes it", async () => { + const { a, b, swap, active, close } = await setup(); + const ta = copy(a); + const tb = copy(b); + const sub = active.track("seq").subscribe(); + const stalled = ta.write(0, 1, false); + const group = sub.tryRecvGroup(); + if (!group) throw new Error("expected group 0"); + await group.readFrame(); + + // The new route never had group 0, and its group 1 is past a zero budget's reach for it. + tb.write(1); + await swap(); + stalled.close(new StreamError(StreamCode.SessionClosed)); + await expect(group.readFrame()).rejects.toBeInstanceOf(StreamError); + expect(drain(sub)).toEqual([1]); + + sub.close(); + close(); +}); + +test("a replacement claiming different track properties is refused once the old copy runs out", async () => { + const { a, b, swap, active, close } = await setup(); + const old = a.createTrack("seq", { priority: 1 }); + b.createTrack("seq", { priority: 2 }); + const sub = active.track("seq").subscribe(); + expect((await sub.info()).priority).toBe(1); + await swap(); + expect(sub.closed.peek()).toBeUndefined(); + + // The old copy failing with its route is what ends the track. + old.close(new StreamError(StreamCode.SessionClosed)); + await settle(); + expect(() => sub.tryRecvGroup()).toThrow("changed its properties"); + + sub.close(); + close(); +}); + +test("datagrams a new copy replays are handed out once", async () => { + const { a, b, swap, active, close } = await setup(); + const ta = copy(a); + const tb = copy(b); + const sub = active.track("seq").subscribe(); + ta.producer.appendDatagram(undefined, enc.encode("a0")); + expect(dec.decode((await sub.recvDatagram())?.payload)).toBe("a0"); + + tb.producer.appendDatagram(undefined, enc.encode("b0")); + await swap(); + tb.producer.appendDatagram(undefined, enc.encode("b1")); + expect(dec.decode((await sub.recvDatagram())?.payload)).toBe("b1"); + + sub.close(); + close(); +}); + +test("a route that goes for good ends the track as its last copy does", async () => { + const { a, active, disposeA, request, close } = await setup(); + const ta = copy(a); + const sub = active.track("seq").subscribe(); + ta.write(0); + expect(drain(sub)).toEqual([0]); + + ta.producer.close(new StreamError(StreamCode.SessionClosed)); + disposeA(); + await settle(); + expect(request.active.peek()).toBeUndefined(); + await expect(sub.recvGroup()).rejects.toBeInstanceOf(StreamError); + + close(); +}); + +test("between routes a reader keeps reading the copy it has", async () => { + const { origin, a, b, active, request, disposeA, close } = await setup(); + const ta = copy(a); + const tb = copy(b); + const sub = active.track("seq").subscribe(); + ta.write(0); + expect(drain(sub)).toEqual([0]); + + // The next route has not answered when the old one is retracted. + const next = wireOf(origin).receive(path, Route.normalize({ epoch: EPOCH, cost: 1n })); + const asked = next.requested(); + disposeA(); + await settle(); + expect(request.active.peek()).toBe(active); + expect(drain(sub)).toEqual([]); + ta.write(1); + expect(drain(sub)).toEqual([1]); + + const { value: pending } = await asked.next(); + pending?.accept(b.consume()); + await settle(); + await settle(); + tb.write(1); + tb.write(2); + expect(drain(sub)).toEqual([2]); + + sub.close(); + next.close(); + close(); +}); + +test("a parked front keeps its readers on the copy they have", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + ta.write(0); + expect(drain(sub)).toEqual([0]); + + // Nothing still reads group 0 from it. + await settle(); + front.park(); + expect(drain(sub)).toEqual([]); + ta.write(1); + expect(drain(sub)).toEqual([1]); + + // Spliced in once its copy proves compatible. + front.serve(b.consume()); + await settle(); + tb.write(1); + tb.write(2); + expect(drain(sub)).toEqual([2]); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts new file mode 100644 index 00000000000..905f5814323 --- /dev/null +++ b/js/net/src/resume.ts @@ -0,0 +1,844 @@ +/** + * Serve a front's logical track straight from its routes' copies. Mirrors `model::resume` in + * `rs/moq-net`. + * + * A {@link Front} is one broadcast served through one route at a time: the origin re-sources it + * when another route of the same epoch takes over, since routes with one epoch serve one + * broadcast and so hold the same groups and frames. A reader of a logical track reads the + * serving route's copy directly, through its own subscription: nothing is copied, and with one + * route it is a passthrough. + * + * When the front switches, each reader subscribes to the new copy from the newest group it + * handed out. It keeps replaced copies for unread groups they already had and groups still being + * read, bounds their subscriptions after their newest group at replacement, and drops them once + * neither needs them. A group carries on across the switch through its {@link Recover}: once its + * copy fails with its route, or stalls while a newer route serves, it continues from the serving + * route's copy of the same group at the same frame, found in that copy's cache or fetched from it. + * + * A group is handed out once however many copies deliver it: the reader skips any sequence its + * subscriber already handed out, the same record that keeps a direct reader from taking a + * re-created group twice. + * + * @module + */ +import { type Dispose, type Getter, race, Signal } from "@moq/signals"; +import * as broadcast from "./broadcast.ts"; +import type { Delivered } from "./delivered.ts"; +import { NotFound, StreamCode, StreamError } from "./error.ts"; +import type { Consumer as GroupConsumer } from "./group.ts"; +import { + type GroupRecover, + groupBounds, + hooks, + type Recv, + type RecvDatagram, + type Replacement, + type ResumeShell, + type ResumeSource, +} from "./internal.ts"; +import type * as track from "./track.ts"; +import { type Broadcast as BroadcastWire, registerWire, wireOf } from "./wire.ts"; + +// Stream codes that report the route failing rather than refusing: another route may still +// deliver what it failed. Anything that is not a stream reset (a session or transport error) is +// one too. +const ROUTE_FAILURES: ReadonlySet = new Set([ + StreamCode.Internal, + StreamCode.Cancel, + StreamCode.SessionClosed, + StreamCode.GoingAway, + StreamCode.ControlTimeout, +]); + +/** Whether `err` is the route failing rather than refusing: another route may still deliver what it failed. */ +function routeFailed(err: Error): boolean { + return !(err instanceof StreamError) || ROUTE_FAILURES.has(err.code); +} + +/** Whether `err` is a route's verdict on a group it read: too old, evicted, or the application's. */ +function verdict(err: Error): boolean { + if (!(err instanceof StreamError)) return false; + return ( + err.code === StreamCode.Old || + err.code === StreamCode.Evicted || + err.code === StreamCode.TooFarBehind || + err.code >= 64 + ); +} + +/** Two copies claiming the same content must agree on what it is. */ +function compatible(a: track.Info, b: track.Info): boolean { + return a.timescale === b.timescale && a.maxAge === b.maxAge && a.priority === b.priority; +} + +/** One route's broadcast, held while the front serves it or a reader still reads a copy of it. */ +class Source { + readonly wire: BroadcastWire; + /** The table's handle this was served from, so serving it again is a no-op. */ + readonly from: broadcast.Consumer; + #consumer: broadcast.Consumer; + #holds = 0; + + constructor(from: broadcast.Consumer) { + this.from = from; + this.#consumer = from.clone(); + this.wire = wireOf(this.#consumer); + } + + retain(): void { + this.#holds++; + } + + release(): void { + if (--this.#holds === 0) this.#consumer.close(); + } +} + +/** What the front says about one logical track: the copy to read now, and how it ended. */ +class Route { + readonly name: string; + /** Bumped whenever `source` changes, so a reader can tell a replacement apart. */ + generation = 0; + /** The serving route's broadcast; undefined between routes or once the front let go. */ + source?: Source; + /** No front is left to replace the copy: readers follow it to its end. */ + concluded = false; + /** The front refused the replacement: the track ends with this once the serving copy runs out. */ + refusal?: Error; + /** Bumped on every change above, waking parked reads. */ + readonly changed = new Signal(0); + readonly readers = new Set(); + /** Called once the last reader leaves. */ + idle: () => void = () => {}; + + constructor(name: string) { + this.name = name; + } + + #bump(): void { + this.changed.update((n) => n + 1); + } + + /** Serve the track from `source` from now on, subscribing every reader to it. */ + serve(source: Source): void { + if (this.concluded || this.source === source) return; + this.generation++; + this.source = source; + this.#bump(); + for (const reader of [...this.readers]) reader.sync(); + } + + /** Between routes: readers wait for the next one, keeping what they have. */ + park(): void { + if (!this.source) return; + this.generation++; + this.source = undefined; + this.#bump(); + } + + /** End the track once the serving copy runs out, rather than replacing it. */ + refuse(err: Error): void { + this.refusal ??= err; + this.#bump(); + } + + /** + * No front will replace the serving copy. Same generation: a reader on the copy keeps it as + * the serving one, while a reader arriving later finds nothing to read. + */ + conclude(): void { + if (this.concluded) return; + this.concluded = true; + this.source = undefined; + this.#bump(); + } + + leave(reader: Reader): void { + if (this.readers.delete(reader) && this.readers.size === 0) this.idle(); + } +} + +/** A route's copy, as one reader reads it. */ +interface Copy { + generation: number; + source: Source; + sub: track.Subscriber; + /** Where the subscription was asked to start, so demand updates never ask for less. */ + floor?: number; + /** + * Replaced by a newer route: nothing past this group, the newest it had then, is taken from + * it. `null` when it had none. + */ + until?: number | null; + /** The group channel ran out: cleanly (`null`) or failed. */ + done?: Error | null; + /** The datagram channel ran out. */ + datagramsDone?: Error | null; + /** The newest datagram handed out before this copy was subscribed, which it may replay. */ + datagramsSeen?: number; + /** Groups handed out from this copy and still read, which keep a replaced copy subscribed. */ + lease: number; +} + +/** The first group past where `copy` was replaced: 0 for one replaced before any group. */ +function copyEnd(copy: Copy): number | undefined { + if (copy.until === undefined) return undefined; + return copy.until === null ? 0 : copy.until + 1; +} + +/** One reader's subscription to a logical track; see the module docs. */ +class Reader implements ResumeSource { + readonly route: Route; + #shell: ResumeShell; + /** The route generation last read. */ + #generation?: number; + /** Oldest first: the last one is the serving route's, when it is subscribed. */ + #copies: Copy[] = []; + /** The newest group handed out: where a new copy is asked to start. */ + #newest?: number; + /** The newest datagram handed out. */ + #datagram?: number; + /** Bumped when this reader subscribes a new copy, waking its parked reads. */ + #changed = new Signal(0); + #dispose: Dispose[] = []; + #closed = false; + + constructor(route: Route, shell: ResumeShell) { + this.route = route; + this.#shell = shell; + route.readers.add(this); + this.#dispose.push( + shell.subscription.subscribe(() => { + for (const copy of this.#copies) this.#update(copy); + }), + shell.cursor.subscribe(() => { + for (const copy of this.#copies) this.#bound(copy); + }), + ); + // Subscribed to the serving copy at once, so groups it has now are the reader's even if + // the route changes before the first read, and to each copy after. + this.sync(); + } + + /** Follow the route: subscribe to a new copy and bound the ones it replaced. */ + sync(): void { + if (this.#closed) return; + const route = this.route; + if (this.#generation === route.generation) return; + this.#generation = route.generation; + const source = route.source; + if (!source) return; + // The replaced copies finish the groups they had when replaced, and nothing newer. + for (const copy of this.#copies) { + if (copy.until !== undefined) continue; + const latest = copy.sub.latest(); + const newest = this.#newest === undefined ? latest : Math.max(this.#newest, latest ?? 0); + copy.until = newest ?? null; + this.#update(copy); + this.#bound(copy); + } + const copy: Copy = { + generation: route.generation, + source, + floor: this.#newest, + datagramsSeen: this.#datagram, + lease: 0, + } as Copy; + // The wire request carries the floor from the first message: some sessions read the + // demand only once. + copy.sub = source.wire.subscribe(route.name, this.#wire(copy)); + source.retain(); + this.#bound(copy); + this.#copies.push(copy); + const shell = this.#shell; + copy.sub.info().then( + (info) => shell.commit(info), + () => {}, + ); + this.#changed.update((n) => n + 1); + } + + /** What `copy` is asked for: the reader's own preferences, from its floor and through its replacement. */ + #wire(copy: Copy): track.Subscription { + const preferences = this.#shell.subscription.peek() ?? {}; + const bounds = groupBounds(preferences.groups); + const start = Math.max(bounds.start, copy.floor ?? 0); + const replaced = copyEnd(copy); + const end = replaced === undefined ? bounds.end : Math.min(bounds.end ?? replaced, replaced); + return { + ...preferences, + groups: { + start: start > 0 ? { included: start } : undefined, + end: end === undefined ? undefined : { excluded: end }, + }, + }; + } + + #update(copy: Copy): void { + if (copy.sub.closed.peek() === undefined) copy.sub.update(this.#wire(copy)); + } + + /** + * Apply the reader's group cursor to `copy`, never below the copy's own resume point: groups + * under it were handed out by an earlier route, or never owed. + */ + #bound(copy: Copy): void { + const cursor = this.#shell.cursor.peek(); + const start = Math.max(cursor.start, copy.floor ?? 0); + const replaced = copyEnd(copy); + const end = replaced === undefined ? cursor.end : Math.min(cursor.end ?? replaced, replaced); + hooks.replaceGroups(copy.sub, { + start: { included: start }, + end: end === undefined ? undefined : { excluded: end }, + }); + } + + /** The reader's copy of the route at `generation`, if it subscribed one. */ + copy(generation: number): Copy | undefined { + return this.#copies.find((copy) => copy.generation === generation); + } + + #deliverable(copy: Copy, sequence: number, delivered: Delivered): boolean { + const end = copyEnd(copy); + return (end === undefined || sequence < end) && !delivered.has(sequence); + } + + tryGroup(ordered: boolean, delivered: Delivered): Recv { + this.sync(); + // The serving copy first: it is the one that keeps delivering. + for (let index = this.#copies.length - 1; index >= 0; index--) { + const copy = this.#copies[index]; + if (copy.done !== undefined) continue; + for (;;) { + const recv = ordered ? hooks.tryNextGroup(copy.sub) : hooks.tryRecvGroup(copy.sub); + if (recv.kind === "group") { + if (!this.#deliverable(copy, recv.group.sequence, delivered)) { + recv.group.close(); + continue; + } + return { kind: "group", group: this.#handOut(copy, recv.group) }; + } + // Drained: the track ended as the copy did, which is not cleanly when its + // session went away short of the declared end. + if (recv.kind === "done") copy.done = copy.sub.closed.peek() ?? null; + else if (recv.kind === "error") copy.done = recv.error; + break; + } + } + this.#letGo(delivered); + return this.#end((copy) => copy.done); + } + + tryDatagram(): RecvDatagram { + this.sync(); + for (let index = this.#copies.length - 1; index >= 0; index--) { + const copy = this.#copies[index]; + // A replaced copy's datagrams are the serving one's to deliver. + if (copy.datagramsDone !== undefined || copy.until !== undefined) continue; + for (;;) { + const recv = hooks.tryRecvDatagram(copy.sub); + if (recv.kind === "datagram") { + const sequence = recv.datagram.sequence; + if (copy.datagramsSeen !== undefined && sequence <= copy.datagramsSeen) continue; + this.#datagram = Math.max(this.#datagram ?? 0, sequence); + return recv; + } + if (recv.kind === "done") copy.datagramsDone = copy.sub.closed.peek() ?? null; + else if (recv.kind === "error") copy.datagramsDone = recv.error; + break; + } + } + const end = this.#end((copy) => copy.datagramsDone); + if (end.kind === "error" || end.kind === "done") return end; + return { kind: "idle" }; + } + + /** + * Whether the track ended, once the serving copy's channel (as `done` reads it) ran out: + * cleanly, or with a verdict a route failure does not excuse. + */ + #end(done: (copy: Copy) => Error | null | undefined): Recv & { kind: "idle" | "done" | "error" } { + const route = this.route; + const last = this.#copies.at(-1); + const serving = last?.generation === this.#generation ? last : undefined; + const ended = serving ? done(serving) : undefined; + // The serving copy drained a complete track: that is its end, whichever route would + // serve it next. + if (serving && ended === null) return this.#finish({ kind: "done" }); + if (ended instanceof Error) { + // A refusal from a route that still serves is the track's answer. + if (!routeFailed(ended)) return this.#finish({ kind: "error", error: ended }); + if (route.refusal) return this.#finish({ kind: "error", error: route.refusal }); + } + if (route.concluded) { + // No front is left to replace the copy: the track ends as it does. + if (!serving) return this.#finish({ kind: "error", error: dropped() }); + if (ended instanceof Error) return this.#finish({ kind: "error", error: ended }); + } + return { kind: "idle" }; + } + + #finish(end: T): T { + this.#shell.close(end.kind === "error" ? end.error : undefined); + return end; + } + + /** + * Drop replaced copies only once neither a handed-out nor an unread group needs them. One not + * yet replaced stays, even between routes: it is still the newest copy there is. + */ + #letGo(delivered: Delivered): void { + const cursor = this.#shell.cursor.peek(); + this.#copies = this.#copies.filter((copy) => { + if (copy.until === undefined || copy.lease > 0) return true; + if (copy.done === undefined && hooks.hasUnreadGroup(copy.sub, cursor.start, copyEnd(copy), delivered)) { + return true; + } + this.#drop(copy); + return false; + }); + } + + #drop(copy: Copy): void { + copy.sub.close(); + copy.source.release(); + } + + #handOut(copy: Copy, group: GroupConsumer): GroupConsumer { + this.#newest = Math.max(this.#newest ?? 0, group.sequence); + hooks.recoverGroup(group, new Recover(this, group, copy)); + return group; + } + + changed(fn: () => void): Dispose { + let fired = false; + const disposes: Dispose[] = []; + const dispose = () => { + fired = true; + for (const close of disposes) close(); + }; + const once = () => { + if (fired) return; + dispose(); + fn(); + }; + disposes.push(this.route.changed.changed(once), this.#changed.changed(once)); + for (const copy of this.#copies) + disposes.push(hooks.groupChanged(copy.sub, once), hooks.datagramChanged(copy.sub, once)); + return dispose; + } + + forgetPastBudget(delivered: Delivered): void { + const newest = this.#copies.at(-1); + if (newest) hooks.forgetPastBudget(newest.sub, delivered); + } + + latest(): number | undefined { + let latest: number | undefined; + for (const copy of this.#copies) { + const sequence = copy.sub.latest(); + if (sequence !== undefined && (latest === undefined || sequence > latest)) latest = sequence; + } + return latest; + } + + #serving(): Copy | undefined { + const last = this.#copies.at(-1); + return last?.generation === this.#generation ? last : undefined; + } + + end(): track.Location | undefined { + this.sync(); + return this.#serving()?.sub.end(); + } + + async finished(): Promise { + for (;;) { + this.sync(); + const serving = this.#serving(); + const end = serving?.sub.end(); + if (end) return end; + const failed = serving?.sub.closed.peek(); + // A copy that failed is only the end of the track if no front replaces it. + if (failed instanceof Error && (this.route.concluded || !routeFailed(failed))) throw failed; + if (!serving && this.route.concluded) throw dropped(); + const closed = this.#shell.closed.peek(); + if (closed instanceof Error) throw closed; + if (closed !== undefined) throw new Error("track closed before its end was known"); + await new Promise((resolve) => { + const dispose = this.changed(() => { + dispose(); + resolve(); + }); + }); + } + } + + largest(): track.Location | undefined { + this.sync(); + return this.#serving()?.sub.largest(); + } + + fork(options?: track.Subscription): track.Subscriber { + this.sync(); + const serving = this.#serving(); + if (!serving) throw new Error("track has no serving route to fork from"); + return serving.sub.fork(options); + } + + close(): void { + if (this.#closed) return; + this.#closed = true; + for (const dispose of this.#dispose) dispose(); + this.#dispose = []; + for (const copy of this.#copies) this.#drop(copy); + this.#copies = []; + this.route.leave(this); + } +} + +/** A fetch of a group's rest from one route generation. */ +interface Fetch { + generation: number; + /** Set once the fetch settles. */ + outcome?: { group: GroupConsumer } | { error: Error }; + /** Fires once it settles. */ + settled: Signal; +} + +function startFetch( + wire: BroadcastWire, + name: string, + sequence: number, + generation: number, + signal: AbortSignal, +): Fetch { + const fetch: Fetch = { generation, settled: new Signal(false) }; + wire.fetchGroup(name, sequence, { signal }) + .then( + (group) => { + fetch.outcome = { group }; + }, + (err: unknown) => { + fetch.outcome = { error: err instanceof Error ? err : new Error(String(err)) }; + }, + ) + .finally(() => fetch.settled.set(true)); + return fetch; +} + +/** Resolves once `signal` aborts. */ +function aborted(signal: AbortSignal): Promise { + return new Promise((resolve) => { + if (signal.aborted) resolve(); + else signal.addEventListener("abort", () => resolve(), { once: true }); + }); +} + +function dropped(): Error { + return new StreamError(StreamCode.Cancel, { message: "no route serves the track" }); +} + +/** Carries a group across route changes; see the module docs. */ +class Recover implements GroupRecover { + #reader: Reader; + #sequence: number; + /** The copy being read, whose subscription this group keeps alive while it is read. */ + #copy?: Copy; + + constructor(reader: Reader, group: GroupConsumer, copy: Copy) { + this.#reader = reader; + this.#sequence = group.sequence; + this.#copy = copy; + copy.lease++; + void group.closed.then(() => this.#release()); + } + + #release(): void { + if (this.#copy) this.#copy.lease--; + this.#copy = undefined; + } + + get changed(): Getter { + return this.#reader.route.changed; + } + + wants(failed: Error | undefined): boolean { + const copy = this.#copy; + if (!copy) return false; + // Only a route that serves can refuse a group on purpose. + if (failed && (!verdict(failed) || copy.sub.closed.peek() instanceof Error)) return true; + const route = this.#reader.route; + return route.generation !== copy.generation || route.concluded; + } + + async replace(_index: number, failed: Error | undefined, signal: AbortSignal): Promise { + const reader = this.#reader; + const route = reader.route; + const abort = aborted(signal); + let fetch: Fetch | undefined; + for (;;) { + signal.throwIfAborted(); + // Subscribe the reader to a new route from its newest group, which is this one or + // later, so that route's subscription may deliver the rest of it. + reader.sync(); + const current = this.#copy; + if (!current) throw new Error("group closed"); + const generation = route.generation; + const serving = route.source ? reader.copy(generation) : undefined; + const waits: Getter[] = [route.changed]; + if (!serving) { + // Nothing serves: a failed copy waits for a route, unless none will come. + if (failed && route.concluded) throw failed; + } else if (generation !== current.generation || failed) { + // A newer copy's track caches it: continue from there. + if (generation !== current.generation) { + const cached = hooks.cachedGroup(serving.sub, this.#sequence); + if (cached) return this.#replacement(cached, serving); + } + // Otherwise ask for it. The subscription may still deliver it first. + if (fetch?.generation !== generation) + fetch = startFetch(serving.source.wire, route.name, this.#sequence, generation, signal); + const stale = hooks.staleAt(serving.sub, this.#sequence); + if (stale.expired()) throw new StreamError(StreamCode.Old, { message: "group is past the budget" }); + waits.push(...stale.changed, ...(fetch.outcome ? [] : [fetch.settled])); + const outcome = fetch.outcome; + if (outcome && "group" in outcome) return this.#replacement(outcome.group, serving); + // That route failing too waits for the next one, and a refusal is a verdict only for + // a copy that failed. + if (outcome && failed && !routeFailed(outcome.error) && !(serving.sub.closed.peek() instanceof Error)) { + throw outcome.error; + } + } + await race([Signal.race(...waits), abort]); + } + } + + #replacement(group: GroupConsumer, copy: Copy): Replacement { + return { + group, + expiry: hooks.staleAt(copy.sub, this.#sequence), + adopt: () => { + // A later switch must keep this copy subscribed while the recovered group still + // reads it, just like a group handed out from that copy directly. + this.#release(); + this.#copy = copy; + copy.lease++; + }, + }; + } +} + +/** + * One broadcast served through one route at a time; see the module docs. + * + * Its {@link consumer} is the broadcast handed out: every track read through it follows the + * front's route. The origin calls {@link serve} as routes of the same epoch take over, and + * {@link conclude} once none will: readers then follow the last copy to its end. + */ +export class Front { + /** The logical broadcast; closing its last handle concludes the front. */ + readonly consumer: broadcast.Consumer; + #source?: Source; + #routes = new Map(); + /** Each track's properties as first committed, which every later copy must match. */ + #info = new Map(); + /** Bumped whenever the source changes, so an answer from a replaced one is dropped. */ + #changed = new Signal(0); + #concluded = false; + + constructor() { + this.consumer = makeFrontConsumer(this); + void this.consumer.closed.then(() => this.conclude()); + } + + /** Serve every track from `from` from now on, once each copy proves compatible. */ + serve(from: broadcast.Consumer): void { + if (this.#concluded || this.#source?.from === from) return; + const source = new Source(from); + source.retain(); + const previous = this.#source; + this.#source = source; + this.#changed.update((n) => n + 1); + for (const route of this.#routes.values()) this.#splice(route, source); + previous?.release(); + } + + /** Between routes: readers keep what they have and wait for the next one. */ + park(): void { + if (this.#concluded || !this.#source) return; + const previous = this.#source; + this.#source = undefined; + this.#changed.update((n) => n + 1); + for (const route of this.#routes.values()) route.park(); + previous.release(); + } + + /** No route will replace the serving one: readers follow it to its end. Idempotent. */ + conclude(): void { + if (this.#concluded) return; + this.#concluded = true; + for (const route of this.#routes.values()) route.conclude(); + this.#source?.release(); + this.#source = undefined; + this.#changed.update((n) => n + 1); + } + + /** Whether the serving broadcast came from a session; see {@link BroadcastWire.fromSession}. */ + get fromSession(): boolean { + return this.#source?.wire.fromSession ?? false; + } + + // Splice `source`'s copy of a track in, once it claims the same content as the copies before. + #splice(route: Route, source: Source): void { + const expected = this.#info.get(route.name); + if (!expected) { + route.serve(source); + return; + } + // The replaced copy keeps feeding the track while the new one answers. + source.retain(); + source.wire + .resolveTrackInfo(route.name) + .then( + (info) => { + if (this.#source !== source) return; + // A copy claiming the same content with different metadata is refused rather + // than reinterpreted. + if (compatible(expected, info)) route.serve(source); + else route.refuse(new Error(`track ${route.name} changed its properties across routes`)); + }, + (err: unknown) => { + if (this.#source !== source) return; + const error = err instanceof Error ? err : new Error(String(err)); + // A route on its way out is no verdict: the next one is asked. + if (!routeFailed(error)) route.refuse(error); + }, + ) + .finally(() => source.release()); + } + + #route(name: string): Route { + let route = this.#routes.get(name); + if (route) return route; + route = new Route(name); + const created = route; + created.idle = () => { + if (this.#routes.get(name) === created) this.#routes.delete(name); + }; + this.#routes.set(name, created); + if (this.#concluded) created.conclude(); + else if (this.#source) created.serve(this.#source); + return created; + } + + #record(name: string, info: track.Info): void { + if (!this.#info.has(name)) this.#info.set(name, info); + } + + subscribe(name: string, options?: track.Subscription): track.Subscriber { + if (this.consumer.closed.peek() !== undefined) throw new Error("broadcast is closed"); + const route = this.#route(name); + return hooks.resumeSubscriber( + name, + options, + (shell) => + new Reader(route, { + ...shell, + commit: (info) => { + this.#record(name, info); + shell.commit(info); + }, + }), + ); + } + + // Wait for the source to change, or `signal` to abort. + async #next(signal?: AbortSignal): Promise { + signal?.throwIfAborted(); + await race(signal ? [this.#changed.changed(), aborted(signal)] : [this.#changed.changed()]); + signal?.throwIfAborted(); + } + + async resolveTrackInfo(name: string, hold?: AbortSignal): Promise { + for (;;) { + const source = this.#source; + if (!source) { + if (this.#concluded) throw dropped(); + await this.#next(hold); + continue; + } + source.retain(); + try { + const info = await source.wire.resolveTrackInfo(name, hold); + this.#record(name, info); + return info; + } catch (err) { + const error = err instanceof Error ? err : new Error(String(err)); + // The route failed: ask the next one, unless none will come. + if (!routeFailed(error) || this.#concluded || hold?.aborted) throw error; + if (this.#source === source) await this.#next(hold); + } finally { + source.release(); + } + } + } + + largest(name: string): track.Location | undefined { + return this.#source?.wire.largest(name); + } + + async fetchGroup(name: string, sequence: number, options?: track.FetchGroupOptions): Promise { + for (;;) { + const source = this.#source; + if (!source) { + if (this.#concluded) throw new NotFound(`group ${sequence}`); + await this.#next(options?.signal); + continue; + } + source.retain(); + try { + return await source.wire.fetchGroup(name, sequence, options); + } catch (err) { + const error = err instanceof Error ? err : new Error(String(err)); + // The route failed: ask the next one, unless none will come. + if (!routeFailed(error) || this.#concluded || options?.signal?.aborted) throw error; + if (this.#source === source) await this.#next(options?.signal); + } finally { + source.release(); + } + } + } +} + +let makeFrontConsumer: (front: Front) => FrontConsumer; + +/** A front's logical broadcast: a consumer handle whose tracks follow the front's route. */ +class FrontConsumer extends broadcast.Consumer { + #front: Front; + + private constructor(front: Front, state?: never) { + super(state); + this.#front = front; + const view = wireOf(this); + registerWire(this, { + requested: view.requested, + subscribe: (name, options) => front.subscribe(name, options), + resolveTrackInfo: (name, hold) => front.resolveTrackInfo(name, hold), + largest: (name) => front.largest(name), + fetchGroup: (name, sequence, options) => front.fetchGroup(name, sequence, options), + get fromSession() { + return front.fromSession; + }, + }); + } + + static { + makeFrontConsumer = (front) => new FrontConsumer(front); + } + + // Preserve the subclass, and the front behind it, when a handle is shared. + override clone(): FrontConsumer { + return new FrontConsumer(this.#front, this.shareState()); + } +} diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index 06197d83b3e..125898b6b1f 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -2443,3 +2443,87 @@ test.each([true, false])("a mid-group end caps later writes (declared first: %s) expect(() => producer.writeGroup(overfull)).toThrow(ProtocolViolation); producer.close(); }); + +function writeTimed(producer: TrackProducer, sequence: number, ms = sequence) { + const group = new GroupProducer(sequence); + producer.writeGroup(group); + group.writeFrame({ payload: enc.encode(`${sequence}`), timestamp: Timestamp.fromMillis(ms) }); + group.close(); + return group; +} + +function drainGroups(subscriber: ReturnType): number[] { + const sequences: number[] = []; + for (let group = subscriber.tryRecvGroup(); group; group = subscriber.tryRecvGroup()) + sequences.push(group.sequence); + return sequences; +} + +test("a re-created group is handed out once", () => { + const producer = new TrackProducer("test").accept({ timescale: Timescale.MILLI, maxAge: Milli(30_000) }); + const subscriber = producer.subscribe({ maxDelay: Milli(30_000) }); + writeTimed(producer, 0); + const aborted = new GroupProducer(1); + producer.writeGroup(aborted); + expect(drainGroups(subscriber)).toEqual([0, 1]); + + aborted.close(new Error("aborted")); + writeTimed(producer, 1); + writeTimed(producer, 2); + expect(drainGroups(subscriber)).toEqual([2]); +}); + +// A gap is remembered until a group above it is past the budget, max delay or max age alike. +// Then every group below that one counts as handed out. +test("a gap is forgotten once a group above it is past the budget", () => { + const time = manualTime(1_000); + try { + const readers = [ + { info: { timescale: Timescale.MILLI, maxAge: Milli(30_000) }, maxDelay: Milli(10) }, + { info: { timescale: Timescale.MILLI, maxAge: Milli(10) }, maxDelay: Milli(3_600_000) }, + ]; + for (const { info, maxDelay } of readers) { + const producer = new TrackProducer("test").accept(info); + const subscriber = producer.subscribe({ maxDelay }); + // Group 3 never arrives. + for (const sequence of [0, 1, 2, 4, 5, 6]) writeTimed(producer, sequence); + expect(drainGroups(subscriber)).toEqual([0, 1, 2, 4, 5, 6]); + // Group 4 reaches until group 5 presents at 5ms, so it is past a 10ms budget once the + // edge presents at 15ms. + for (let sequence = 7; sequence <= 14; sequence++) { + writeTimed(producer, sequence); + expect(drainGroups(subscriber)).toEqual([sequence]); + expect(hooks.deliveredRuns(subscriber)).toBe(2); + } + writeTimed(producer, 15); + expect(drainGroups(subscriber)).toEqual([15]); + expect(hooks.deliveredRuns(subscriber)).toBe(1); + } + } finally { + time.restore(); + } +}); + +test("an endless track with gaps remembers only those within the budget", () => { + const time = manualTime(1_000); + try { + const producer = new TrackProducer("test").accept({ timescale: Timescale.MILLI, maxAge: Milli(30_000) }); + // Groups present 1ms apart, so the budget spans ten of them. + const subscriber = producer.subscribe({ maxDelay: Milli(10) }); + let most = 0; + for (let sequence = 0; sequence < 3000; sequence++) { + // Every third group never arrives. + if (sequence % 3 === 1) continue; + writeTimed(producer, sequence); + expect(drainGroups(subscriber)).toEqual([sequence]); + most = Math.max(most, hooks.deliveredRuns(subscriber)); + } + expect(most).toBe(5); + + // A late unseen group within the budget is still delivered. + writeTimed(producer, 2995); + expect(drainGroups(subscriber)).toEqual([2995]); + } finally { + time.restore(); + } +}); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 8ee50e9bbd0..cedf4fbb168 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -7,6 +7,7 @@ import { Instant, Milli, Nano, now, sleepUntil } from "@moq/clock"; import { type Dispose, type GetPromise, type Getter, Once, Signal } from "@moq/signals"; import type { Datagram } from "./datagram.ts"; +import { Delivered } from "./delivered.ts"; import { GroupTooLarge, ProtocolViolation, TimestampMismatch, TooFarBehind } from "./error.ts"; import { type Frame, type Consumer as GroupConsumer, Producer as GroupProducer } from "./group.ts"; import { @@ -14,6 +15,8 @@ import { groupBounds, hooks, type Recv, + type RecvDatagram, + type ResumeSource, type TrackRequestOptions, type TrackSequence, type TrackSequences, @@ -464,6 +467,8 @@ async function resolveInfo(state: TrackState): Promise { type CachedGroup = { group: GroupProducer; mirrors: Map; arrived: Instant }; let bindProducerSequence: (producer: Producer, sequence: TrackSequence) => void; +// A fresh read handle on a group the producer still caches; assigned in its static block. +let cachedMirror: (producer: Producer, sequence: number) => GroupConsumer | undefined; // The sequence-order cursor lives inside `Subscriber` (it shares the group buffer and the // drift anchor with the arrival cursor), so `Ordered` reaches it through this bridge, @@ -605,6 +610,13 @@ export class Producer { }; hooks.bindSequence = bindProducerSequence; hooks.cachedLargest = (producer) => producer.#largest(); + cachedMirror = (producer, sequence) => { + const entry = producer.#cached.get(sequence); + if (!entry || entry.group.closed.peek() instanceof Error) return undefined; + const mirror = entry.group.mirror(); + hooks.arriveGroup(mirror, entry.arrived); + return mirror; + }; } /** @@ -1166,6 +1178,13 @@ export class Producer { } } +// One read pass's view of the budget and the live edge; see Subscriber.#drift. +type Drift = { + budget: number; + presentation?: { sequence: number; timestamp: Timestamp }; + end?: number; +}; + /** * The read side of a live track subscription, mirroring the Rust `Subscriber`. * @@ -1178,6 +1197,12 @@ export class Subscriber { readonly name: string; #state: TrackState; + // Reads its routes' copies instead of `#state`'s buffer: the counterpart of the Rust + // `Inner::Resume`. `#state` then only carries the reader's own preferences, info, and close. + #resume?: ResumeSource; + // Every group sequence this reader handed out, so a re-created group or one a second route + // delivers is handed out once. + #delivered = new Delivered(); #nextSequence = 0; #cursor = new Signal<{ start: number; end?: number }>({ start: 0 }); // A FETCH's own budget in place of the subscription's; unlike SUBSCRIBE's zero @@ -1199,11 +1224,7 @@ export class Subscriber { at: Number.POSITIVE_INFINITY, }; - #drift(): { - budget: number; - presentation?: { sequence: number; timestamp: Timestamp }; - end?: number; - } { + #drift(): Drift { const { end } = this.#cursor.peek(); const timeline = this.#state.timeline; let presentation: { sequence: number; timestamp: Timestamp } | undefined; @@ -1269,18 +1290,14 @@ export class Subscriber { // successor that never presents cannot pin a superseded group. The successor is re-read, // not latched, like the reach. The newest group has no successor, so neither clock ever // convicts it. `wake` arms a timer for the wall deadline, for a held read that waits on it. - #isStale( - group: GroupConsumer, - drift: { - budget: number; - presentation?: { sequence: number; timestamp: Timestamp }; - end?: number; - }, - wake = false, - ): boolean { + #isStale(group: GroupConsumer, drift: Drift, wake = false): boolean { if (this.#state.timeline[timelineIndex(this.#state.timeline, group.sequence)] !== group) return false; + return this.#staleAt(group.sequence, drift, wake); + } - const successor = this.#successor(group.sequence, drift.end); + // The drift verdict for `sequence` by its successor alone, whether or not this track holds it. + #staleAt(sequence: number, drift: Drift, wake = false): boolean { + const successor = this.#successor(sequence, drift.end); if (!successor) return false; // Plain arithmetic: an unenforced budget is infinite, which no duration brand holds. const deadline = hooks.groupArrived(successor) + drift.budget * 1_000_000; @@ -1290,7 +1307,7 @@ export class Subscriber { const reach = hooks.groupTimestamp(successor)?.asMillis(); return ( drift.presentation !== undefined && - drift.presentation.sequence > group.sequence && + drift.presentation.sequence > sequence && reach !== undefined && drift.presentation.timestamp.asMillis() - reach >= drift.budget ); @@ -1320,19 +1337,45 @@ export class Subscriber { #guard(group: GroupConsumer): GroupConsumer { hooks.expireGroup(group, { expired: () => this.#isStale(group, this.#drift(), true), - changed: [ - this.#state.groups, - this.#state.timelineChanged, - this.#wall.revision, - this.#state.update, - this.#state.info, - this.#cursor, - this.#state.closed, - ], + changed: this.#expiryChanged(), }); return group; } + // What can change a drift verdict. + #expiryChanged(): Getter[] { + return [ + this.#state.groups, + this.#state.timelineChanged, + this.#wall.revision, + this.#state.update, + this.#state.info, + this.#cursor, + this.#state.closed, + ]; + } + + // Record `sequence` as handed out. An ordered read never goes back, and any reader stops + // expecting what is past its budget, so only the gaps within it cost memory. + #handOut(sequence: number, ordered: boolean): void { + if (ordered) this.#delivered.forgetBelow(sequence); + this.#delivered.add(sequence); + if (this.#resume) this.#resume.forgetPastBudget(this.#delivered); + else this.#forgetPastBudget(this.#delivered); + } + + // Forget the gaps in `delivered` below each group past this subscriber's budget now, whether + // this track holds it or not: one landing there would be written off. + #forgetPastBudget(delivered: Delivered): void { + if (delivered.aboveGap() === undefined) return; + // One edge for every group judged, so forgetting N gaps never resolves it N times. + const drift = this.#drift(); + for (let oldest = delivered.aboveGap(); oldest !== undefined; oldest = delivered.aboveGap()) { + if (!this.#staleAt(oldest, drift)) break; + delivered.forgetBelow(oldest); + } + } + private constructor(name: string, state: TrackState) { this.name = name; this.#state = state; @@ -1346,6 +1389,58 @@ export class Subscriber { static { makeSubscriber = (name, state) => new Subscriber(name, state); hooks.tryRecvGroup = (subscriber) => subscriber.#tryRecvGroup(); + hooks.tryNextGroup = (subscriber) => subscriber.#tryNextGroup(); + hooks.forgetPastBudget = (subscriber, delivered) => subscriber.#forgetPastBudget(delivered); + hooks.tryRecvDatagram = (subscriber) => subscriber.#tryDatagram(); + hooks.deliveredRuns = (subscriber) => subscriber.#delivered.runs; + hooks.datagramChanged = (subscriber, fn) => { + if (subscriber.#resume) return subscriber.#groupChanged(fn); + const state = subscriber.#state; + const dispose = [ + state.datagrams.changed(fn), + state.closed.changed(fn), + state.end.changed(fn), + state.received.changed(fn), + ]; + return () => { + for (const close of dispose) close(); + }; + }; + hooks.hasUnreadGroup = (subscriber, start, end, delivered) => + subscriber.#state.groups + .peek() + .some( + (group) => + group.sequence >= start && + (end === undefined || group.sequence < end) && + !delivered.has(group.sequence) && + !(group.closed.peek() instanceof Error), + ); + hooks.staleAt = (subscriber, sequence) => ({ + expired: () => subscriber.#staleAt(sequence, subscriber.#drift(), true), + changed: subscriber.#expiryChanged(), + }); + hooks.cachedGroup = (subscriber, sequence) => { + const producer = subscriber.#state.producer; + return producer && cachedMirror(producer, sequence); + }; + hooks.resumeSubscriber = (name, subscription, source) => { + const state = new TrackState(subscription); + const subscriber = new Subscriber(name, state); + subscriber.#resume = source({ + subscription: state.update, + cursor: subscriber.#cursor, + closed: state.closed, + info: () => state.info.peek(), + commit: (info) => { + if (state.info.peek() === undefined) state.info.set(info); + }, + close: (abort) => { + closeTrackState(state, abort); + }, + }); + return subscriber; + }; hooks.groupChanged = (subscriber, fn) => subscriber.#groupChanged(fn); hooks.fetchBudget = (subscriber, maxDelay) => { subscriber.#fetchBudget = maxDelay ?? Number.POSITIVE_INFINITY; @@ -1393,7 +1488,7 @@ export class Subscriber { /** Return the latest group sequence observed on this track, if any. */ latest(): number | undefined { - return this.#state.latest; + return this.#resume ? this.#resume.latest() : this.#state.latest; } /** @@ -1404,7 +1499,7 @@ export class Subscriber { * Undefined until declared, and after an abort that declared none. Mirrors the Rust `end`. */ end(): Location | undefined { - return this.#state.end.peek(); + return this.#resume ? this.#resume.end() : this.#state.end.peek(); } /** @@ -1415,6 +1510,7 @@ export class Subscriber { * `undefined` for that. Rejects with the abort, or if the track closes without an end. */ async finished(): Promise { + if (this.#resume) return this.#resume.finished(); for (;;) { const end = this.#state.end.peek(); if (end !== undefined) return end; @@ -1436,6 +1532,7 @@ export class Subscriber { * left can no longer say where the edge is. */ largest(): Location | undefined { + if (this.#resume) return this.#resume.largest(); const latest = this.#state.latest; if (latest === undefined) return undefined; @@ -1462,6 +1559,7 @@ export class Subscriber { * @internal Wire layers only. */ fork(options?: Subscription): Subscriber { + if (this.#resume) return this.#resume.fork(options); const producer = this.#state.producer; if (!producer) throw new Error("track has no producer to fork from"); return producer.subscribe(options); @@ -1500,6 +1598,7 @@ export class Subscriber { // producer close, so the subscriber leaving is what must release them: closing // and clearing wakes a pending read to observe the end instead of hanging. closeTrackState(this.#state, abort); + this.#resume?.close(abort); this.#state.groups.mutate((groups) => { for (const group of groups) group.close(abort); groups.length = 0; @@ -1544,19 +1643,24 @@ export class Subscriber { } // Idle, or parked at the boundary waiting for the cap to rise. - await Signal.race( - this.#state.groups, - this.#cursor, - this.#state.closed, - this.#state.end, - this.#state.received, - ); + await this.#wait(); } } + // Park until a group read may go differently. + #wait(): Promise { + return new Promise((resolve) => { + const dispose = this.#groupChanged(() => { + dispose(); + resolve(); + }); + }); + } + // Package-internal synchronous half of recvGroup. The lite publisher uses this so applying // control state, popping the group, and positioning its frames are one JavaScript turn. #tryRecvGroup(): Recv { + if (this.#resume) return this.#tryResumed(false); const groups = this.#state.groups.peek(); const { start, end } = this.#cursor.peek(); while (groups.length > 0 && groups[0].sequence < start) groups.shift()?.close(); @@ -1568,10 +1672,11 @@ export class Subscriber { const group = groups[0]; if (!group || (end !== undefined && group.sequence >= end)) break; groups.shift(); - if (this.#isStale(group, drift)) { + if (this.#delivered.has(group.sequence) || this.#isStale(group, drift)) { group.close(); continue; } + this.#handOut(group.sequence, false); return { kind: "group", group: this.#guard(group) }; } @@ -1584,6 +1689,17 @@ export class Subscriber { return group ? { kind: "boundary" } : { kind: "done" }; } + // A group from the routes' copies; see #resume. + #tryResumed(ordered: boolean): Recv { + const resume = this.#resume as ResumeSource; + const closed = this.#state.closed.peek(); + if (closed instanceof Error) return { kind: "error", error: closed }; + if (closed !== undefined) return { kind: "done" }; + const recv = resume.tryGroup(ordered, this.#delivered); + if (recv.kind === "group") this.#handOut(recv.group.sequence, ordered); + return recv; + } + // Reader completion does not settle the tail: missing lower groups and open streams // remain accounted for by the wire layer, and an abort still wins on the next read. #complete(): boolean { @@ -1595,6 +1711,12 @@ export class Subscriber { // the caller disposes the losers after whichever source wakes it. A declared end wakes it // too, so a publisher can forward the end before the live edge reaches it. #groupChanged(fn: () => void): Dispose { + if (this.#resume) { + const resumed = [this.#resume.changed(fn), this.#cursor.changed(fn), this.#state.closed.changed(fn)]; + return () => { + for (const close of resumed) close(); + }; + } const dispose = [ this.#state.groups.changed(fn), this.#cursor.changed(fn), @@ -1644,74 +1766,107 @@ export class Subscriber { // choice of group order says nothing about it. async #recvDatagram(): Promise { for (;;) { - const datagrams = this.#state.datagrams.peek(); - - // The group range bounds datagrams too, judged when each is read. One outside it is - // gone for good: nothing holds it for a later raise, unlike a group past the cap. - const { start, end } = this.#cursor.peek(); - for (let datagram = datagrams.shift(); datagram; datagram = datagrams.shift()) { - if (datagram.sequence >= start && (end === undefined || datagram.sequence < end)) return datagram; + const recv = this.#tryDatagram(); + switch (recv.kind) { + case "datagram": + return recv.datagram; + case "done": + return undefined; + case "error": + throw recv.error; } + if (this.#resume) await this.#wait(); + else await Signal.race(this.#state.datagrams, this.#state.closed, this.#state.end, this.#state.received); + } + } - const closed = this.#state.closed.peek(); - if (closed instanceof Error) throw closed; - if (closed !== undefined || this.#complete()) return undefined; - - await Signal.race(this.#state.datagrams, this.#state.closed, this.#state.end, this.#state.received); + // The synchronous half of #recvDatagram. + #tryDatagram(): RecvDatagram { + const closed = this.#state.closed.peek(); + if (this.#resume) { + if (closed instanceof Error) return { kind: "error", error: closed }; + if (closed !== undefined) return { kind: "done" }; + } + // The group range bounds datagrams too, judged when each is read. One outside it is + // gone for good: nothing holds it for a later raise, unlike a group past the cap. + const { start, end } = this.#cursor.peek(); + const inRange = (datagram: Datagram) => + datagram.sequence >= start && (end === undefined || datagram.sequence < end); + if (this.#resume) { + for (;;) { + const recv = this.#resume.tryDatagram(); + if (recv.kind !== "datagram" || inRange(recv.datagram)) return recv; + } } + const datagrams = this.#state.datagrams.peek(); + for (let datagram = datagrams.shift(); datagram; datagram = datagrams.shift()) { + if (inRange(datagram)) return { kind: "datagram", datagram }; + } + if (closed instanceof Error) return { kind: "error", error: closed }; + if (closed !== undefined || this.#complete()) return { kind: "done" }; + return { kind: "idle" }; } // The sequence cursor behind {@link Ordered}, which owns the only public door to it. async #nextGroup(): Promise { for (;;) { - const groups = this.#state.groups.peek(); - const cursor = this.#cursor.peek(); - const start = Math.max(cursor.start, this.#nextSequence); - while (groups.length > 0 && groups[0].sequence < start) groups.shift()?.close(); - // One anchor for the whole pass, so walking a backlog off stays linear. - const drift = this.#drift(); - - for (;;) { - const group = groups[0]; - if (!group || (cursor.end !== undefined && group.sequence >= cursor.end)) break; - groups.shift(); - this.#nextSequence = group.sequence + 1; - // Every frame this group could still hold is past the budget, so keep - // scanning: one pass walks a whole backlog off rather than replaying it. - if (this.#isStale(group, drift)) { - group.close(); - continue; - } - // One cursor: the frame helpers must not keep draining a group this - // read just moved past, or interleaved reads would run backwards. - if (this.#frameGroup && this.#frameGroup.sequence < group.sequence) { - this.#frameGroup.close(); - this.#frameGroup = undefined; - } - return this.#guard(group); + const recv = this.#tryNextGroup(); + switch (recv.kind) { + case "group": + // One cursor: the frame helpers must not keep draining a group this + // read just moved past, or interleaved reads would run backwards. + if (this.#frameGroup && this.#frameGroup.sequence < recv.group.sequence) { + this.#frameGroup.close(); + this.#frameGroup = undefined; + } + return recv.group; + case "done": + return undefined; + case "error": + throw recv.error; } + await this.#wait(); + } + } - const closed = this.#state.closed.peek(); - if (closed instanceof Error) throw closed; - // A group parked above the cap stays deliverable even after a clean close - // (its frames remain buffered), so keep waiting for a cap raise. Only a - // drained track reports finished. - const end = this.#state.end.peek(); - if ( - !groups[0] && - (closed !== undefined || this.#complete() || (end !== undefined && start >= finalSequence(end))) - ) { - return undefined; + // The synchronous half of #nextGroup. + #tryNextGroup(): Recv { + if (this.#resume) return this.#tryResumed(true); + const groups = this.#state.groups.peek(); + const cursor = this.#cursor.peek(); + const start = Math.max(cursor.start, this.#nextSequence); + while (groups.length > 0 && groups[0].sequence < start) groups.shift()?.close(); + // One anchor for the whole pass, so walking a backlog off stays linear. + const drift = this.#drift(); + + for (;;) { + const group = groups[0]; + if (!group || (cursor.end !== undefined && group.sequence >= cursor.end)) break; + groups.shift(); + this.#nextSequence = group.sequence + 1; + // Every frame this group could still hold is past the budget, so keep + // scanning: one pass walks a whole backlog off rather than replaying it. + if (this.#isStale(group, drift)) { + group.close(); + continue; } + this.#handOut(group.sequence, true); + return { kind: "group", group: this.#guard(group) }; + } - await Signal.race( - this.#state.groups, - this.#cursor, - this.#state.closed, - this.#state.end, - this.#state.received, - ); + const closed = this.#state.closed.peek(); + if (closed instanceof Error) return { kind: "error", error: closed }; + // A group parked above the cap stays deliverable even after a clean close + // (its frames remain buffered), so keep waiting for a cap raise. Only a + // drained track reports finished. + const end = this.#state.end.peek(); + if ( + !groups[0] && + (closed !== undefined || this.#complete() || (end !== undefined && start >= finalSequence(end))) + ) { + return { kind: "done" }; } + return { kind: "idle" }; } /** diff --git a/test/drain/README.md b/test/drain/README.md index 882519b2069..3c6f921f247 100644 --- a/test/drain/README.md +++ b/test/drain/README.md @@ -32,25 +32,25 @@ deletes its own. - **The publisher** sits on B and writes one group every 100 ms, each carrying its own sequence number. A pulls the track through the cluster, so both relays serve the same groups. -- **The viewer** is `@moq/net` over WebSocket, following whichever broadcast the - path routes to the way a player does: subscribe to the new one, drop the old. - Without an epoch the other relay's route is another publisher instance, so the - viewer requests again when that route is announced as a restart. +- **The viewer** is `@moq/net` over WebSocket. It requests the path once and + reads one subscription for the whole run. The publisher mints an epoch, so + B's route serves the same broadcast as A's, and the origin resumes the track + there from the first group the viewer lacks. Epochs travel on the + work-in-progress moq-lite-07 only, so the relays, A's cluster dial, and both + JS sessions offer it. Once the viewer has read groups through A, the driver withdraws A from the name and sends it SIGTERM, which fires the same trigger an embedder's drain hook does. The run passes when: -- the viewer reads fresh groups through B within a few seconds, well inside A's - 20 s drain window, having dialed A exactly once; -- every group from the first read through the last arrived, from one relay or - the other; +- the viewer reads fresh groups through B within a few seconds of leaving A, + well inside A's 20 s drain window, having dialed A exactly once; +- every group from the first read through the last arrived exactly once, from + one relay or the other, on the one subscription, whose broadcast handle never + changed; - the viewer leaves A at its 2 s handover cap, and A then exits on its own, logging that every session left rather than that its deadline forced one out. -The viewer subscribes with a 1 s latency budget, as the interop subscribers do. -After the swap, B has to subscribe upstream afresh once A drops its pull, and the -budget is what reaches back to a group in flight across the swap. With no budget, -a group boundary that lands inside the swap loses that group. -[JS group-boundary handover](/quest/m1/js-group-handover.md) removes the -budget by carrying the subscription across the swap. +The viewer subscribes with no latency budget: only the live edge. B serves the +track while A still does, and each group the viewer has not read yet comes +from one route or the other, so none is lost across the swap. diff --git a/test/drain/drain.ts b/test/drain/drain.ts index b80cf7065ef..eb4f5d225a0 100644 --- a/test/drain/drain.ts +++ b/test/drain/drain.ts @@ -10,7 +10,11 @@ * * bun drain.ts --a-port 4470 --b-port 4471 --proxy-port 4472 --a-pid 1234 --timeout 60 * - * Exits 0 once the viewer has migrated, read groups on both relays with none missing in + * The viewer subscribes once, at no latency budget, and reads the same subscription throughout: + * the publisher mints an epoch, so B's route serves the same broadcast and the origin resumes the + * track there from the first group the viewer lacks. + * + * Exits 0 once the viewer has migrated, read groups through both relays with none missing in * between, and left A on its own; 1 otherwise. * * @module @@ -48,14 +52,10 @@ const GROUP_INTERVAL_MS = 100; // How long the viewer's old session may keep serving after the GOAWAY. Well inside the relay's // 20s drain window, so the relay's exit log can prove the viewer left on its own. const HANDOVER = Milli(2000); -// The viewer's latency budget, as a player sets one (the interop subscribers use the same). -// Following the route means resubscribing on B, and B has to subscribe upstream afresh once A -// drops its pull; the budget is what lets that resubscribe reach back to a group that was in -// flight across the swap instead of starting at the next one. With none, a group boundary -// landing inside the swap drops that group. -const MAX_DELAY = Milli(1000); const path = Moq.Path.from("drain"); +// Publisher epochs travel on the work-in-progress moq-lite-07 only, which nothing offers by default. +const websocket = { protocols: ["moq-lite-07-wip"] }; const trackName = "seq"; function log(message: string) { @@ -111,8 +111,13 @@ await new Promise((resolve, reject) => { const published = new Moq.Origin.Producer(); const broadcast = published.createBroadcast(path); const track = broadcast.createTrack(trackName, { timescale: Moq.Time.Timescale.MILLI }); -broadcast.announce(); -const publisher = new Moq.Connection({ url: new URL(`http://127.0.0.1:${bPort}/`), publish: published.consume() }); +// The epoch is what says B's route serves the same broadcast as A's. +broadcast.announce({ epoch: Moq.Epoch.mint() }); +const publisher = new Moq.Connection({ + url: new URL(`http://127.0.0.1:${bPort}/`), + publish: published.consume(), + websocket, +}); let lastPublished = -1; const ticker = setInterval(() => { @@ -126,112 +131,75 @@ const ticker = setInterval(() => { }, GROUP_INTERVAL_MS); // ── viewer through the name ─────────────────────────────────────────────────── -/** Which relay delivered each group: the generation of the broadcast it was read from. */ -const seen = new Map>(); -let generation = -1; +/** How many times each group was read: once, through whichever relay served it then. */ +const seen = new Map(); /** A group whose payload is not its sequence number: fails the run however the swap goes. */ let corrupt: string | undefined; +/** Why the subscription ended, if it did: it must outlive the migration. */ +let ended: string | undefined; const watched = new Moq.Origin.Producer(); const viewer = new Moq.Connection({ url: new URL(`http://127.0.0.1:${proxyPort}/`), consume: watched, goaway: { handover: HANDOVER }, + websocket, }); -async function read(sub: Moq.Track.Subscriber, gen: number): Promise { + +// One request and one subscription, at the default zero budget, read for the whole run: the +// route moving from A to B re-sources the track underneath. +const request = watched.request(path, { announced: true }); +await until("the broadcast to resolve", () => request.active.peek() !== undefined); +const active = request.active.peek() as Moq.Broadcast.Consumer; +log("watching"); +const sub = active.track(trackName).subscribe(); +void (async () => { try { for (;;) { const group = await sub.recvGroup(); - if (!group) return; + if (!group) { + ended = "the subscription finished"; + return; + } const text = await group.readString(); if (text !== String(group.sequence)) { - corrupt ??= `generation ${gen}: group ${group.sequence} carried ${text}`; + corrupt ??= `group ${group.sequence} carried ${text}`; return; } - let gens = seen.get(group.sequence); - if (!gens) { - gens = new Set(); - seen.set(group.sequence, gens); - } - gens.add(gen); + seen.set(group.sequence, (seen.get(group.sequence) ?? 0) + 1); } } catch (err) { - // The broadcast it was read from went away; its successor picks up from here. - log(`generation ${gen} ended: ${err instanceof Error ? err.message : String(err)}`); - } -} - -// Follow whichever broadcast the path routes to, as a player does: subscribe to the new one and -// drop the old one each time the route changes. A request stays on the publisher instance it -// resolved, and without an epoch the other relay's route is another instance, so a player -// requests again whenever the path is announced (a start or a restart), keeping its request when -// the fresh one resolves the same broadcast, and never because its request ended. -let current: { broadcast: Moq.Broadcast.Consumer; sub: Moq.Track.Subscriber } | undefined; -const follow = (active: Moq.Broadcast.Consumer | undefined) => { - if (!active || active === current?.broadcast) return; - generation++; - log(`watching generation ${generation}`); - const sub = active.track(trackName).subscribe({ maxDelay: MAX_DELAY }); - void read(sub, generation); - current?.sub.close(); - current = { broadcast: active, sub }; -}; -let request = watched.request(path, { announced: true }); -// Subscribed before the first peek, so no route change can land between reading it and listening. -let unfollow = request.active.subscribe(follow); -follow(request.active.peek()); - -const announced = watched.announced(Moq.Path.Pattern.literal(path), { hidden: true }); -void (async () => { - for (;;) { - const entry = await announced.next(); - if (!entry) break; - if (entry.kind === "end" || entry.kind === "update") continue; - if (!Moq.Path.hasPrefix(entry.prefix, path)) continue; - const fresh = watched.request(path, { announced: true }); - const same = - request.closed.peek() === undefined && request.active.peek()?.closed === fresh.active.peek()?.closed; - if (same) { - fresh.close(); - continue; - } - log(`path ${entry.kind}: requesting again`); - const old = request; - unfollow(); - request = fresh; - unfollow = request.active.subscribe(follow); - follow(request.active.peek()); - old.close(); + ended = err instanceof Error ? err.message : String(err); } })(); -const readOn = (gen: number) => [...seen.values()].filter((gens) => gens.has(gen)).length; -const newestOn = (gen: number) => Math.max(-1, ...[...seen].filter(([, gens]) => gens.has(gen)).map(([seq]) => seq)); +const newest = () => Math.max(-1, ...seen.keys()); let failure: Error | undefined; try { - await until(`${GROUPS_PER_RELAY} groups through relay A`, () => generation === 0 && readOn(0) >= GROUPS_PER_RELAY); + await until(`${GROUPS_PER_RELAY} groups through relay A`, () => seen.size >= GROUPS_PER_RELAY); if (a.dialed !== 1 || b.dialed !== 0) throw new Error(`expected one dial to A, saw A=${a.dialed} B=${b.dialed}`); log("withdrawing A from the name, then draining it"); resolved = b; process.kill(aPid, "SIGTERM"); - // Counted past the last group A delivered: B's first answer can include recent groups the - // viewer already had, which prove nothing about B being live. - // Bounded well inside A's 20s drain window: a viewer that only reconnects once A cuts it off - // at the deadline would otherwise pass, just late. + // The old session leaves at the handover cap, before A's own deadline would cut it. + await until("the viewer to leave relay A", () => a.open === 0, HANDOVER * 3); + // Counted past the newest group when A went: only B can have delivered these. Bounded well + // inside A's 20s drain window: a viewer that only reconnects once A cuts it off at the + // deadline would otherwise pass, just late. + const left = newest(); await until( `${GROUPS_PER_RELAY} new groups through relay B`, - () => generation >= 1 && newestOn(generation) >= newestOn(0) + GROUPS_PER_RELAY, + () => newest() >= left + GROUPS_PER_RELAY, HANDOVER * 3, ); - // The old session leaves at the handover cap, before A's own deadline would cut it. - await until("the viewer to leave relay A", () => a.open === 0, HANDOVER * 3); - if (generation !== 1) throw new Error(`expected exactly one migration, saw ${generation}`); if (a.dialed !== 1) throw new Error(`the viewer redialed the withdrawn relay (${a.dialed} dials)`); if (b.dialed < 1) throw new Error("the viewer never dialed relay B"); + if (request.active.peek() !== active) throw new Error("the request swapped its broadcast across the migration"); + if (ended) throw new Error(`the subscription ended across the migration: ${ended}`); const sequences = [...seen.keys()].sort((x, y) => x - y); const first = sequences[0] ?? 0; @@ -240,17 +208,16 @@ try { for (let seq = first; seq <= last; seq++) { if (!seen.has(seq)) missing.push(seq); } - const both = sequences.filter((seq) => seen.get(seq)?.size === 2).length; - log(`read groups ${first}..${last} (published up to ${lastPublished}); ${both} arrived from both relays`); + const repeated = sequences.filter((seq) => (seen.get(seq) ?? 0) > 1); + log(`read groups ${first}..${last} (published up to ${lastPublished}); left A after ${left}`); if (missing.length > 0) throw new Error(`dropped groups across the migration: ${missing.join(", ")}`); - if (corrupt) throw new Error(`corrupt group from ${corrupt}`); + if (repeated.length > 0) throw new Error(`groups handed out twice: ${repeated.join(", ")}`); + if (corrupt) throw new Error(`corrupt group: ${corrupt}`); log("migrated without a dropped group"); } catch (err) { failure = err instanceof Error ? err : new Error(String(err)); } finally { - announced.close(); - unfollow(); - current?.sub.close(); + sub.close(); clearInterval(ticker); request.close(); viewer.close(); diff --git a/test/drain/relay.toml b/test/drain/relay.toml index f64e9090294..2376a9e58e0 100644 --- a/test/drain/relay.toml +++ b/test/drain/relay.toml @@ -16,6 +16,13 @@ level = "info" # QUIC on UDP. 127.0.0.1 avoids IPv6 flakiness on CI runners. bind = "127.0.0.1:4470" tls.generate = ["localhost", "127.0.0.1"] +# Publisher epochs travel on lite-07 only, and the epoch is what lets the viewer's subscription +# resume on the other relay instead of ending with its route. +version = ["moq-lite-07-wip", "moq-lite-06"] + +[connect] +# A's cluster dial to B carries the epoch on to the viewer, so it needs lite-07 too. +version = ["moq-lite-07-wip", "moq-lite-06"] [web.http] # HTTP and WebSocket on TCP, also serving /certificate.sha256, which is how the diff --git a/test/interop/README.md b/test/interop/README.md index e8be61a2a6b..4a3ed0f2687 100644 --- a/test/interop/README.md +++ b/test/interop/README.md @@ -111,7 +111,7 @@ until moq-ffi exposes one, and the browser's shared connection keeps its session private, so their cells check media alone. AUTH is only on the work-in-progress moq-lite-07, so the printing clients dial `moq-lite-07-wip` alone and a printing client that reports nothing never got its grant. The native JS subscribers dial -WebTransport alone, since the WebSocket fallback cannot offer it. The rest keep +WebTransport alone and offer it there. The rest keep their defaults, so the matrix also crosses versions through the relay, which accepts both. diff --git a/test/interop/clients/js-native/subscribe.ts b/test/interop/clients/js-native/subscribe.ts index 1b637a0286e..f118dc76445 100644 --- a/test/interop/clients/js-native/subscribe.ts +++ b/test/interop/clients/js-native/subscribe.ts @@ -66,7 +66,7 @@ async function video(bc: Moq.Broadcast.Consumer): Promise<[string, Catalog.Video async function run(): Promise { const origin = new Moq.Origin.Producer(); // The grant arrives over AUTH, which only the work-in-progress moq-lite-07 carries and no - // client offers by default. The WebSocket fallback cannot offer it, so it stays off. + // client offers by default. It is offered over WebTransport alone, so the fallback stays off. const connection = await Moq.Connection.connect({ url: new URL(url as string), consume: origin, From 2b3d42db386bfbb7801e794ce113b7cfca5f1e99 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 16 Oct 2026 20:12:21 -0700 Subject: [PATCH 03/31] feat(js/net): fetch recovery, resume benchmark, and quest cleanup Co-Authored-By: Claude Opus 5.5 --- js/justfile | 2 +- js/net/bench/resume.ts | 125 +++++++++++++++++++++++++++++ js/net/src/group.ts | 18 ++--- js/net/src/resume.test.ts | 22 +++++ js/net/src/resume.ts | 78 +++++++++++++++--- quest/m1/README.md | 1 - quest/m1/ietf-epochs.md | 1 - quest/m1/ietf-legacy-updates.md | 4 - quest/m1/js-group-handover.md | 78 ------------------ quest/m1/public-model/deferred.md | 5 +- quest/m1/refuse-rewinds.md | 1 - quest/m2/p2p/cost-scopes.md | 1 - quest/m2/route-health/selection.md | 1 - sh/gh/nightly.sh | 2 +- 14 files changed, 225 insertions(+), 114 deletions(-) create mode 100644 js/net/bench/resume.ts delete mode 100644 quest/m1/js-group-handover.md diff --git a/js/justfile b/js/justfile index e4fecd8189e..22fcc3927ad 100644 --- a/js/justfile +++ b/js/justfile @@ -47,7 +47,7 @@ test: test-changed list: python3 ../sh/js/workspace.py test '{{ list }}' -# Run one net benchmark: broadcasts, forward, requests, reader, frames, track, interests, observed, varint, lite-varint, priority, or request-ids. +# Run one net benchmark: broadcasts, forward, requests, reader, frames, track, resume, interests, observed, varint, lite-varint, priority, or request-ids. bench name: bun install --frozen-lockfile bun net/bench/{{ name }}.ts diff --git a/js/net/bench/resume.ts b/js/net/bench/resume.ts new file mode 100644 index 00000000000..e5f8da67b4c --- /dev/null +++ b/js/net/bench/resume.ts @@ -0,0 +1,125 @@ +/** + * Sweep readers and route swaps for groups read through a logical front, against the same + * groups read straight from the track. + * + * A front reads its serving route's copy directly, so with one route it should cost about what a + * direct read does, and each swap should leave nothing behind once its groups are read. + */ + +import { Producer as BroadcastProducer } from "../src/broadcast.ts"; +import { Producer as GroupProducer } from "../src/group.ts"; +import { Front } from "../src/resume.ts"; +import type { Subscriber } from "../src/track.ts"; + +const readerCounts = [1, 4, 16]; +const swapCounts = [0, 1, 16]; +const groups = 400; +const reps = 7; +// A reader through a front must not cost more per group as readers or swaps grow. The margin is +// loose because the machine is noisy. +const maxSlope = 3; +const payload = new Uint8Array(80); +let checksum = 0; + +// Let every woken reader and pending splice run. +const flush = () => new Promise((resolve) => setImmediate(resolve)); + +function read(subscribers: Subscriber[]) { + for (const subscriber of subscribers) { + const group = subscriber.tryRecvGroup(); + if (!group) throw new Error("group was not delivered"); + const frame = group.tryReadFrame(); + if (!frame) throw new Error("frame was not delivered"); + checksum += frame.payload.byteLength; + group.close(); + // Read until idle, as a reader parked on the next group does. + if (subscriber.tryRecvGroup()) throw new Error("group delivered twice"); + } +} + +/** Microseconds per group per reader, reading through a front after `swaps` route swaps, or directly. */ +async function measure(readers: number, swaps: number | undefined): Promise { + let sequence = 0; + const sources: { broadcast: BroadcastProducer; write: () => void }[] = []; + const source = () => { + const broadcast = new BroadcastProducer(); + const track = broadcast.createTrack("seq"); + // Every route's copy holds the same groups under the same sequences. + const write = () => { + const group = new GroupProducer(sequence); + track.writeGroup(group); + group.writeFrame({ payload }); + group.close(); + }; + sources.push({ broadcast, write }); + return broadcast; + }; + const publish = () => { + for (const { write } of sources) write(); + sequence++; + }; + + let subscribers: Subscriber[]; + let front: Front | undefined; + if (swaps === undefined) { + const broadcast = source().consume(); + subscribers = Array.from({ length: readers }, () => broadcast.track("seq").subscribe()); + } else { + front = new Front(); + front.serve(source().consume()); + const consumer = front.consumer; + subscribers = Array.from({ length: readers }, () => consumer.track("seq").subscribe()); + for (let swap = 0; swap < swaps; swap++) { + publish(); + read(subscribers); + // Only the serving route publishes from here, as a replaced one would stop. + sources.splice(0, sources.length - 1); + front.serve(source().consume()); + await flush(); + } + } + publish(); + read(subscribers); + await flush(); + + const start = performance.now(); + for (let index = 0; index < groups; index++) { + publish(); + read(subscribers); + } + const elapsed = performance.now() - start; + + for (const subscriber of subscribers) subscriber.close(); + front?.consumer.close(); + return (elapsed * 1000) / groups / readers; +} + +// Warm the JIT so the first row isn't the slowest. +await measure(4, 1); +await measure(4, undefined); + +console.log("readers,swaps,direct_us,front_us"); +let baseline: number | undefined; +for (const swaps of swapCounts) { + for (const readers of readerCounts) { + const direct: number[] = []; + const front: number[] = []; + // Interleaved, so a noisy stretch of the machine lands on both columns alike. + for (let rep = 0; rep < reps; rep++) { + direct.push(await measure(readers, undefined)); + front.push(await measure(readers, swaps)); + } + // The fastest run is the one least disturbed by the rest of the machine. + const directUs = Math.min(...direct); + const frontUs = Math.min(...front); + console.log(`${readers},${swaps},${directUs.toFixed(2)},${frontUs.toFixed(2)}`); + + baseline ??= frontUs; + if (frontUs > baseline * maxSlope) { + throw new Error( + `${readers} readers after ${swaps} swaps cost ${frontUs.toFixed(2)} us/group, over ${maxSlope}x ${baseline.toFixed(2)}`, + ); + } + } +} +if (checksum === 0) throw new Error("benchmark did no work"); diff --git a/js/net/src/group.ts b/js/net/src/group.ts index fdc1c6a7ace..a40277ed847 100644 --- a/js/net/src/group.ts +++ b/js/net/src/group.ts @@ -428,6 +428,8 @@ export class Consumer { #verdict = new Once(); #closed?: CombinedClosed; #pendingFrames = 0; + // Closed by its reader, which ends it whatever is left unread. + #released = false; private constructor(state: GroupState) { this.#state = state; @@ -446,7 +448,7 @@ export class Consumer { (value) => value instanceof Error ? !this.#recovers(value) - : this.#state.frames.peek().length === 0 && this.#pendingFrames === 0, + : this.#released || (this.#state.frames.peek().length === 0 && this.#pendingFrames === 0), ); return this.#closed; } @@ -553,12 +555,8 @@ export class Consumer { this.#state.frames.set([]); } - async #changed(): Promise { - await this.#changes(); - } - // The next change a parked read waits for. - #changes(): GetPromise { + #changed(): GetPromise { const changed: Getter[] = [this.#state.frames, this.#state.closed]; if (this.#expiry) changed.push(...this.#expiry.changed); if (this.#recover) changed.push(this.#recover.changed); @@ -592,7 +590,7 @@ export class Consumer { } // A stalled copy may still deliver first; a failed one never will. if (failed) await attempt.result; - else await race([attempt.result, this.#changes()]); + else await race([attempt.result, this.#changed()]); // The copy being read moved first: look again, keeping the attempt out. if (!attempt.settled) return true; if (this.#attempt === attempt) this.#attempt = undefined; @@ -801,8 +799,8 @@ export class Consumer { this.#attempt = undefined; this.#recover = undefined; // Closing is terminal for this reader, even with frames still unread. - if (this.#verdict.peek() === undefined) this.#verdict.set(abort ?? null); - if (this.#state.closed.peek() !== undefined) return; // already closed - this.#state.closed.set(abort ?? null); + this.#released = true; + if (this.#state.closed.peek() === undefined) this.#state.closed.set(abort ?? null); + this.#closed?.refresh(); } } diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index fe4e2450447..7ace5a239c7 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -290,3 +290,25 @@ test("a parked front keeps its readers on the copy they have", async () => { a.close(); b.close(); }); + +test("a fetched group continues from the next route once its own fails", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + const stalled = ta.write(5, 1, false); + front.serve(a.consume()); + const group = await front.consumer.track("seq").fetchGroup(5); + expect(dec.decode((await group.readFrame())?.payload)).toBe("5.0"); + + tb.write(5, 2); + front.serve(b.consume()); + stalled.close(new StreamError(StreamCode.SessionClosed)); + expect(dec.decode((await group.readFrame())?.payload)).toBe("5.1"); + expect(await group.readFrame()).toBeUndefined(); + + front.consumer.close(); + a.close(); + b.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 905f5814323..dc051090962 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -36,7 +36,7 @@ import { type ResumeShell, type ResumeSource, } from "./internal.ts"; -import type * as track from "./track.ts"; +import * as track from "./track.ts"; import { type Broadcast as BroadcastWire, registerWire, wireOf } from "./wire.ts"; // Stream codes that report the route failing rather than refusing: another route may still @@ -180,8 +180,19 @@ interface Copy { lease: number; } +/** Subscribe to a route's copy, or to a failed stand-in when that route can no longer serve. */ +function subscribe(wire: BroadcastWire, name: string, subscription: track.Subscription): track.Subscriber { + try { + return wire.subscribe(name, subscription); + } catch (err) { + const failed = new track.Producer(name); + failed.close(err instanceof Error ? err : new Error(String(err))); + return failed.subscribe(subscription); + } +} + /** The first group past where `copy` was replaced: 0 for one replaced before any group. */ -function copyEnd(copy: Copy): number | undefined { +function copyEnd(copy: Pick): number | undefined { if (copy.until === undefined) return undefined; return copy.until === null ? 0 : copy.until + 1; } @@ -237,16 +248,17 @@ class Reader implements ResumeSource { this.#update(copy); this.#bound(copy); } + const floor = this.#newest; const copy: Copy = { generation: route.generation, source, - floor: this.#newest, + // The wire request carries the floor from the first message: some sessions read the + // demand only once. + sub: subscribe(source.wire, route.name, this.#wire({ floor })), + floor, datagramsSeen: this.#datagram, lease: 0, - } as Copy; - // The wire request carries the floor from the first message: some sessions read the - // demand only once. - copy.sub = source.wire.subscribe(route.name, this.#wire(copy)); + }; source.retain(); this.#bound(copy); this.#copies.push(copy); @@ -259,7 +271,7 @@ class Reader implements ResumeSource { } /** What `copy` is asked for: the reader's own preferences, from its floor and through its replacement. */ - #wire(copy: Copy): track.Subscription { + #wire(copy: Pick): track.Subscription { const preferences = this.#shell.subscription.peek() ?? {}; const bounds = groupBounds(preferences.groups); const start = Math.max(bounds.start, copy.floor ?? 0); @@ -789,23 +801,65 @@ export class Front { } async fetchGroup(name: string, sequence: number, options?: track.FetchGroupOptions): Promise { + const fetched = await this.#fetch(name, sequence, options); + let current = fetched.source; + void fetched.group.closed.then(() => current.release()); + // Carried across route changes like a subscribed group, but by fetching again: a fetch has + // no copy of its own to find the rest in. + hooks.recoverGroup(fetched.group, { + changed: this.#changed, + wants: (failed) => { + // Too old and evicted judge a subscription's delivery, not whether the group + // exists, so only the application's refusal is a verdict on a fetch. + if (failed) return !(failed instanceof StreamError && failed.code >= 64); + return this.#source !== undefined && this.#source !== current; + }, + replace: async (_index, failed, signal) => { + // A stalled fetch only moves to another route; a failed one may ask its own again. + const next = await this.#fetch(name, sequence, { ...options, signal }, failed ? undefined : current); + let adopted = false; + void next.group.closed.then(() => { + if (!adopted) next.source.release(); + }); + return { + group: next.group, + // The route applies the fetch's own budget. + expiry: { expired: () => false, changed: [] }, + adopt: () => { + adopted = true; + current.release(); + current = next.source; + }, + }; + }, + }); + return fetched.group; + } + + // Fetch a group from whichever route serves the track, asking the next one when a route + // fails, and never `avoid`. The source comes back retained for as long as the group is read. + async #fetch( + name: string, + sequence: number, + options: track.FetchGroupOptions | undefined, + avoid?: Source, + ): Promise<{ group: GroupConsumer; source: Source }> { for (;;) { const source = this.#source; - if (!source) { + if (!source || source === avoid) { if (this.#concluded) throw new NotFound(`group ${sequence}`); await this.#next(options?.signal); continue; } source.retain(); try { - return await source.wire.fetchGroup(name, sequence, options); + return { group: await source.wire.fetchGroup(name, sequence, options), source }; } catch (err) { + source.release(); const error = err instanceof Error ? err : new Error(String(err)); // The route failed: ask the next one, unless none will come. if (!routeFailed(error) || this.#concluded || options?.signal?.aborted) throw error; if (this.#source === source) await this.#next(options?.signal); - } finally { - source.release(); } } } diff --git a/quest/m1/README.md b/quest/m1/README.md index 5df3713e3e0..e6ded856330 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -96,7 +96,6 @@ lands before the cut. - [Rust PROBE on demand](/quest/m1/probe-on-demand-rs.md) - a Rust client opens PROBE only while a `recv_bandwidth()` consumer is held, and `Stats` drops `estimated_recv_rate` - [Session outcomes](/quest/m1/session-outcomes.md) - the relay's session stats count refusals by reason and ends by kind, per root and tier, including refusals an embedder records on its stats handle - [Typed refusal reason](/quest/m1/refusal-reason.md) - an auth server's 403 names the reason (including `expired`) and the root and tier, so the relay's session outcomes count and attribute refusals -- [JS track handover](/quest/m1/js-group-handover.md) - a JS track subscription resumes across a route swap from the first frame it lacks, so `test/drain` passes at zero latency budget - [Scope track priority](/quest/m1/track-priority-scope.md) - priority orders one owner's streams, and a shared cluster session is fair across tenants - [Audio jitter target](/quest/m1/audio-jitter-target/README.md) - the playout target is default in both languages; a manual browser proof and a native trace replay remain - [A/V clock](/quest/m1/av-clock.md) - the audio playhead drives Sync.reference while audio plays, through per-track sync handles diff --git a/quest/m1/ietf-epochs.md b/quest/m1/ietf-epochs.md index a59017af587..44b26fed3f3 100644 --- a/quest/m1/ietf-epochs.md +++ b/quest/m1/ietf-epochs.md @@ -79,6 +79,5 @@ extension; no changes to unnegotiated published versions or broadcast paths. - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - current lite-07 identity and legacy restart behavior - [Announcement takeover](/quest/m1/announce-takeover.md) - shared continuity model and lite-07 operation - [IETF takeover](/quest/m1/ietf-takeover.md) - independently advertised continuation of a claim-served instance -- [JS track handover](/quest/m1/js-group-handover.md) - stable JS readers across same-epoch route changes - [Catalog references](/quest/m0/broadcast-epoch/catalog-references.md) - optional requested epochs become verifiable on this negotiated transport; epochless fallback remains best effort - [IETF fill timeout](/quest/m1/ietf-fill-timeout.md) - native wait-budget support is separate from epoch negotiation and adds no age extension diff --git a/quest/m1/ietf-legacy-updates.md b/quest/m1/ietf-legacy-updates.md index 32f505589a8..cf9f9096e63 100644 --- a/quest/m1/ietf-legacy-updates.md +++ b/quest/m1/ietf-legacy-updates.md @@ -24,7 +24,3 @@ subscription with INTERNAL_ERROR. Remaining (maintainer, 2026-10-07, from Test: an update keeps a live subscription open and a later UNSUBSCRIBE still ends it, on d14, d15 and d16, in both languages. - -## Related - -- [JS group handover](/quest/m1/js-group-handover.md) - JS also lacks Rust's route-failure splice diff --git a/quest/m1/js-group-handover.md b/quest/m1/js-group-handover.md deleted file mode 100644 index 32a01a3de19..00000000000 --- a/quest/m1/js-group-handover.md +++ /dev/null @@ -1,78 +0,0 @@ -# [L] JS track handover - -## Goal - -A `js/net` track subscription survives its broadcast's route swapping to -another provider with the same epoch, such as a relay migration after a -GOAWAY, and resumes from the new provider at the first frame it has not -delivered. A viewer at the live -edge with no latency budget never loses a group across the swap, and never -has to notice the swap to keep reading. - -Done when `test/drain` passes with the viewer's `MAX_DELAY` at zero, the -viewer subscribes once instead of following `request.active`, and the run is -stable enough for the nightly. - -## Plan - -Today the origin (`js/net/src/origin.ts`) holds the outranked route until the -new one answers, then closes the old front. Every track read through that -front ends, and the app has to subscribe again on the new broadcast. Once the -old relay drops its upstream pull, the new relay subscribes upstream from -scratch at the live edge. A group boundary that lands inside that window loses -the group in flight: about 1 run in 5 at 100 ms groups. - -Current Rust (`model/resume.rs`) reads route copies directly through each -reader. It retains replaced copies for unread and in-flight groups, bounds -them at replacement, and recovers a group from the new copy at the first -missing frame and byte. It does not copy through the earlier single-writer -pump or immediately cancel every old copy. Decided in the 2026-10-10 audit: -mirror the current identity, reader, recovery and lifetime contract, including -the late-duplicate correction, rather than porting the obsolete pump. -Decided 2026-10-13: every track reader, direct or resumed, hands out each group -sequence at most once, including a group aborted and re-created and one a -second route delivers, so no consumer ever dedupes itself. The record is a -per-reader interval set of delivered sequences on the track subscriber, shared -by direct and resumed reads, with no fixed-size history. Once a group is past -the reader's budget (max delay, clamped by max age), the reader stops expecting -every group below it and counts them as delivered, so only gaps within the -budget cost memory. A genuinely unseen late group within the budget is still -delivered. Group timestamps on a track never rewind; which late groups are -delivered after a rewind is unspecified, so the floor is judged by the -delivered group above a gap rather than the gap itself. -Things to settle along the way: - -- Whether the request's `active` broadcast stays the same object across a swap, - with its tracks re-sourced underneath. That is the Rust behavior and the - simplest for players. It is also a behavior change for code that watches - `active` to resubscribe. -- The resumed subscription names where it left off (group and frame) so the - new provider serves the rest from its cache or upstream instead of starting - at its own live edge. -- Failover compatibility: Rust refuses to resume onto a source whose track - properties differ (timescale, retention, priority, order). Match it. -- Resume only between routes with the same epoch. Any other swap is a new - broadcast that announce consumers see as a `Restart`, and players follow - it with a fresh subscription. A route without an epoch never resumes, as - in Rust, so `drain.ts`'s publisher mints an epoch. -- `js/watch` and `js/hang` consumers that re-subscribe on `active` changes. - Check whether they still need to. -- Giving up a resumed group no route continues. Mirror Rust's rule, which - js/net's `#isStale` already applies to subscriptions: give it up once its - wall-clock age since its successor arrived, or its media-time drift, - reaches the reader's budget. - -Add unit coverage at the origin level against stand-in sessions, then flip -`test/drain` to zero budget (drop the resubscribe loop in `drain.ts` and the -budget note in its README). - -Decided 2026-10-08: Restart and One max_age meaning land first, so this -mirrors their final shape instead of chasing it. - -Public API: likely a behavior change to `Origin.Requesting.active` and track -subscriptions across a swap. Report it in the PR. - -## Related - -- [Retired requests](https://github.com/moq-dev/moq/pull/5294) - old handles cannot open unpinned requests after an identity change -- [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - a terminal group disposition stops recovery even for the newest group diff --git a/quest/m1/public-model/deferred.md b/quest/m1/public-model/deferred.md index 42f37a217e7..64d0a36a2b6 100644 --- a/quest/m1/public-model/deferred.md +++ b/quest/m1/public-model/deferred.md @@ -30,8 +30,8 @@ the prerequisites will have deleted or renamed most of these. caller, bring the shape back to the maintainer rather than publishing peeks. - JS `Group.Producer.mirror` (lite subscriber fetch coalescing): remove the - need through Model ranges and JS track handover, which owns the JS fan-out - copy. + need through Model ranges. JS track handover landed without a fan-out copy + of its own: it reads each route's copy directly. - JS track info and fetch answering (`wire.ts` `overrideBroadcastWire`): answered through the public `Track.Request` and `Track.Dynamic` range requests that Model ranges defines, completing the answering quest's @@ -47,7 +47,6 @@ built-in sessions moved onto the public calls. - [Model ranges](/quest/m1/subscribe-ranges/model.md) - replaces per-group fetch requests, the peeks, and `fork` - [Live largest location](/quest/m1/largest.md) - the public largest-location stream that replaces `peek_latest` and `is_live` on the IETF Largest Object path -- [JS track handover](/quest/m1/js-group-handover.md) - owns the JS fan-out copy behind `Group.Producer.mirror` - [JS pending tail](/quest/m1/js-pending-tail.md) - the JS received-end hold behind `hooks.endTrack` - [Public ingest](/quest/m1/public-model/ingest.md) - settles the received-end shape JS mirrors - [JS ranges](/quest/m1/subscribe-ranges/js.md) - the JS `Track.Request`/`Track.Dynamic` range requests that answer track info and fetches diff --git a/quest/m1/refuse-rewinds.md b/quest/m1/refuse-rewinds.md index 852c6c6c725..d2c446b097a 100644 --- a/quest/m1/refuse-rewinds.md +++ b/quest/m1/refuse-rewinds.md @@ -135,6 +135,5 @@ code 0x39 broadens. ## Related - [#5446](https://github.com/moq-dev/moq/pull/5446) - where the maintainer decided rewinds are never supported -- [JS track handover](/quest/m1/js-group-handover.md) - mirrors Rust's resume contract, which relies on starts never falling - [Same-epoch importers](/quest/m2/hop-aligned-import.md) - importers already refuse a rewind at the container layer - [Epoch anchor](/quest/m2/epoch-anchor.md) - a rewind stays fatal to an importer, which restarts under a successor epoch diff --git a/quest/m2/p2p/cost-scopes.md b/quest/m2/p2p/cost-scopes.md index 248c8db7c46..056d35cc9a8 100644 --- a/quest/m2/p2p/cost-scopes.md +++ b/quest/m2/p2p/cost-scopes.md @@ -42,7 +42,6 @@ in Peers and native configuration. Wire: none beyond shared routing and existing ## Required -- [JS track handover](/quest/m1/js-group-handover.md) - same-epoch browser migration - [Routes and announces](/quest/m1/cluster-routing/routes.md) - common identity and route ordering - [Signaling and policy](/quest/m2/p2p/signal.md) - explicit application peer selection - [moq-cli joins](/quest/m2/p2p/cli.md) - matching native controls diff --git a/quest/m2/route-health/selection.md b/quest/m2/route-health/selection.md index 16d6c229e8d..341799d109a 100644 --- a/quest/m2/route-health/selection.md +++ b/quest/m2/route-health/selection.md @@ -38,4 +38,3 @@ report any required surface change. No independent wire extension planned. - [Health design](/quest/m2/route-health/design.md) - the proven selection boundary - [Routes and announces](/quest/m1/cluster-routing/routes.md) - shared candidate and identity model -- [JS track handover](/quest/m1/js-group-handover.md) - matching logical-track resume behavior diff --git a/sh/gh/nightly.sh b/sh/gh/nightly.sh index 43f3bb46592..6f9acd85815 100755 --- a/sh/gh/nightly.sh +++ b/sh/gh/nightly.sh @@ -50,7 +50,7 @@ case "${1:-}" in run just rs test -p moq-cli -E 'test(benchmark_stream_import)' --run-ignored only --success-output immediate ;; js) - for benchmark in broadcasts request-ids requests forward reader frames varint lite-varint track interests priority; do + for benchmark in broadcasts request-ids requests forward reader frames varint lite-varint track resume interests priority; do run just js bench "$benchmark" done run just js catalog-bench From 43cb52974236586468b0d125d967b722d5dd605d Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 16 Oct 2026 20:56:35 -0700 Subject: [PATCH 04/31] fix(js/net): check a replacement against a first copy that has not answered yet Co-Authored-By: Claude Opus 5.5 --- js/net/src/resume.test.ts | 23 +++++++++++++++++++++++ js/net/src/resume.ts | 36 ++++++++++++++++++++++++++++-------- 2 files changed, 51 insertions(+), 8 deletions(-) diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 7ace5a239c7..1bf9cbf36a9 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -312,3 +312,26 @@ test("a fetched group continues from the next route once its own fails", async ( a.close(); b.close(); }); + +test("a replacement spliced before the first copy answers is still checked against it", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const old = a.createTrack("seq", { priority: 1 }); + b.createTrack("seq", { priority: 2 }); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + // Swapped in the same turn, before the first copy's info committed anywhere. + front.serve(b.consume()); + await settle(); + expect((await sub.info()).priority).toBe(1); + + old.close(new StreamError(StreamCode.SessionClosed)); + await settle(); + expect(() => sub.tryRecvGroup()).toThrow("changed its properties"); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index dc051090962..c7be68e4c87 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -101,6 +101,8 @@ class Route { generation = 0; /** The serving route's broadcast; undefined between routes or once the front let go. */ source?: Source; + /** The last broadcast served, kept between routes: what a replacement must match. */ + served?: Source; /** No front is left to replace the copy: readers follow it to its end. */ concluded = false; /** The front refused the replacement: the track ends with this once the serving copy runs out. */ @@ -124,6 +126,7 @@ class Route { if (this.concluded || this.source === source) return; this.generation++; this.source = source; + this.served = source; this.#bump(); for (const reader of [...this.readers]) reader.sync(); } @@ -703,21 +706,35 @@ export class Front { // Splice `source`'s copy of a track in, once it claims the same content as the copies before. #splice(route: Route, source: Source): void { - const expected = this.#info.get(route.name); - if (!expected) { + const recorded = this.#info.get(route.name); + const previous = route.served; + // The first copy of a track has nothing to match. + if (!recorded && !previous) { route.serve(source); return; } - // The replaced copy keeps feeding the track while the new one answers. + // The replaced copy keeps feeding the track while the new one answers. One that has not + // answered yet is asked too, so a swap racing its first answer is still checked. source.retain(); - source.wire - .resolveTrackInfo(route.name) + previous?.retain(); + const expected = recorded + ? Promise.resolve(recorded) + : (previous as Source).wire.resolveTrackInfo(route.name).then( + (info) => { + this.#record(route.name, info); + return info; + }, + () => undefined, + ); + Promise.all([expected, source.wire.resolveTrackInfo(route.name)]) .then( - (info) => { + ([expected, info]) => { if (this.#source !== source) return; + // The copy it replaces never answered, so this one says what the track is. + if (!expected) this.#record(route.name, info); // A copy claiming the same content with different metadata is refused rather // than reinterpreted. - if (compatible(expected, info)) route.serve(source); + if (!expected || compatible(expected, info)) route.serve(source); else route.refuse(new Error(`track ${route.name} changed its properties across routes`)); }, (err: unknown) => { @@ -727,7 +744,10 @@ export class Front { if (!routeFailed(error)) route.refuse(error); }, ) - .finally(() => source.release()); + .finally(() => { + source.release(); + previous?.release(); + }); } #route(name: string): Route { From 4d4c727b40478fcca17bbc54ae9fd175b0fd0496 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 16 Oct 2026 21:45:30 -0700 Subject: [PATCH 05/31] fix(js/net): ordered resumed reads take the lowest group across copies Also reject inherited names in websocket.protocols and state lite-07 as the condition for epoch handover in the JS docs. Co-Authored-By: Claude Opus 5.5 --- doc/lib/js/net.md | 4 ++-- js/net/src/connection/connect.test.ts | 8 ++++--- js/net/src/connection/connect.ts | 5 ++--- js/net/src/internal.ts | 5 +++++ js/net/src/resume.test.ts | 28 +++++++++++++++++++++++++ js/net/src/resume.ts | 30 +++++++++++++++++++++++++++ js/net/src/track.ts | 14 +++++++++++++ 7 files changed, 86 insertions(+), 8 deletions(-) diff --git a/doc/lib/js/net.md b/doc/lib/js/net.md index 356c5b05e48..563f69e2690 100644 --- a/doc/lib/js/net.md +++ b/doc/lib/js/net.md @@ -36,10 +36,10 @@ broadcast.announce(); - **The origin holds the broadcasts, not the connection.** Closing a session unannounces them and leaves them created for the next one. A broadcast is invisible, locally and remotely, until `announce()`. `dynamic(prefix, route)` claims a prefix and yields each requested path to accept or reject. `accept` throws on a broadcast a session delivered, since JS apps do not proxy; copy its tracks into a broadcast you produce instead. - **Requests can pin an epoch.** `consumer.request(path, { epoch })` resolves only through a route announcing that [publisher epoch](/concept/moq-lite#publisher-epochs) and reports `unroutable` otherwise. The resolved broadcast's `epoch` names the route it came through, pinned or not. - **Rank several CDNs.** Connect to each with its own `consume: origin.withPreference(Origin.Preference(rank))` (higher wins; 0 is neutral and a negative rank demotes), all on one origin. Requests use the preferred link that has a route and fail over to the next; see [multiple CDNs](/concept/moq-lite#multiple-cdns). -- **Follow restarts.** `origin.request(path, { announced: true })` follows whichever publisher instance serves the path: its `active` changes to a fresh broadcast on a restart, clears on an end, and resolves again on the next start, so read it reactively and close the request when done. Instances are never stitched, so subscribe again to each new broadcast. Another route of the same epoch taking over keeps `active` as is, and each track subscription resumes there from the first group it lacks. A refusal ends the request. Add `epoch` to pin one instance; a plain `request(path)` keeps its accepted instance, can resolve blindly, and ends once that instance stops serving. To list what is live, iterate `announced(scope)`. +- **Follow restarts.** `origin.request(path, { announced: true })` follows whichever publisher instance serves the path: its `active` changes to a fresh broadcast on a restart, clears on an end, and resolves again on the next start, so read it reactively and close the request when done. Instances are never stitched, so subscribe again to each new broadcast. Another route of the same epoch taking over (epochs travel only on `moq-lite-07-wip`) keeps `active` as is, and each track subscription resumes there from the first group it lacks. A refusal ends the request. Add `epoch` to pin one instance; a plain `request(path)` keeps its accepted instance, can resolve blindly, and ends once that instance stops serving. To list what is live, iterate `announced(scope)`. - **Discovery follows what you watch.** A `consume` origin asks the relay for announcements only under the prefixes its readers watch with `announced(scope)` or `request(path, { announced: true })`, stops asking about a prefix 2s after nothing at or beneath it is watched, and asks again on reconnect. A plain `request(path)` asks nothing and subscribes blind, but once it resolves through a discovered route it watches its path until closed, so it outlives the listing that found it. A relay that refuses an announce request turns that session's announced requests blind too. - **One connection per URL, unless you opt out.** `new Connection({ url })` pools and reconnects with backoff, which the elements use. Supplying your own transport options or origin selects a private loop. Receive estimates are on demand: observe `connection.probe` reactively or subscribe while you need them; `peek()` alone opens no PROBE stream. -- **GOAWAY moves the session.** The connection dials the replacement at once while the old session keeps serving its groups, up to `goaway.handover` (default 10s, or the relay's deadline when that is sooner). New requests keep opening on the old session until the replacement's route outranks it, which is a restart without an epoch: an unpinned announced request follows the replacement, while a blind or epoch-pinned request stays on its accepted instance. With an epoch the replacement serves the same broadcast, so open subscriptions move across without missing a group. An empty GOAWAY redials the same URL. A redirect stays on the same host unless `goaway.redirect` is `"follow"`. +- **GOAWAY moves the session.** The connection dials the replacement at once while the old session keeps serving its groups, up to `goaway.handover` (default 10s, or the relay's deadline when that is sooner). New requests keep opening on the old session until the replacement's route outranks it, which is a restart without an epoch: an unpinned announced request follows the replacement, while a blind or epoch-pinned request stays on its accepted instance. With an epoch, which travels only when `moq-lite-07-wip` is offered, the replacement serves the same broadcast, so open subscriptions move across without missing a group. An empty GOAWAY redials the same URL. A redirect stays on the same host unless `goaway.redirect` is `"follow"`. - **One send estimate per connection.** `Bandwidth.Allocator` divides it by track priority, max-min fair within a tier. An idle track claims nothing. Publishers reserve against it so their targets sum to the estimate. - **One track per name.** Concurrent subscriptions share one request and producer. `createTrack` on a name with a request still queued answers it. The application owns sequence numbers: a track that replaces an ended one, created or requested, throws on an append that would reuse a sequence the name already wrote, so it writes its first group with an explicit sequence. Only a new broadcast starts over. - **Timing is per track, with no default.** A track that declares a `timescale` carries a `timestamp` on every frame; one without is [untimed](/concept/moq-lite#subscriptions) and its frames carry none. Omitting it means untimed, as in Rust. Nothing stamps a frame for you: the publisher passes the timestamp, and the `Timed` type (`{ value, at }`) carries one into the `@moq/json` and `@moq/flate` producers. diff --git a/js/net/src/connection/connect.test.ts b/js/net/src/connection/connect.test.ts index cae8f363b98..ed99c924154 100644 --- a/js/net/src/connection/connect.test.ts +++ b/js/net/src/connection/connect.test.ts @@ -336,9 +336,11 @@ test("the WebSocket fallback refuses an unsupported protocol", async () => { const original = globalThis.WebTransport; globalThis.WebTransport = undefined as unknown as typeof WebTransport; try { - await expect(connect(url, { websocket: { protocols: ["moq-lite-99"] } })).rejects.toThrow( - "unsupported WebSocket protocol", - ); + for (const protocol of ["moq-lite-99", "toString"]) { + await expect(connect(url, { websocket: { protocols: [protocol] } })).rejects.toThrow( + "unsupported WebSocket protocol", + ); + } } finally { globalThis.WebTransport = original; } diff --git a/js/net/src/connection/connect.ts b/js/net/src/connection/connect.ts index 5b28030826b..3b7dd386227 100644 --- a/js/net/src/connection/connect.ts +++ b/js/net/src/connection/connect.ts @@ -557,9 +557,8 @@ function websocketVersions(protocols?: string[]): Record = {}; for (const protocol of protocols) { - const version = supported[protocol]; - if (version === undefined) throw new Error(`unsupported WebSocket protocol: ${protocol}`); - versions[protocol] = version; + if (!Object.hasOwn(supported, protocol)) throw new Error(`unsupported WebSocket protocol: ${protocol}`); + versions[protocol] = supported[protocol]; } return versions; } diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index 25cbb791a0e..9bb40e7c3a6 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -284,6 +284,8 @@ export const hooks: { * {@link groupChanged} when it reports `idle` or `boundary`. */ tryRecvGroup: (subscriber: Subscriber) => Recv; + /** The sequence {@link tryNextGroup} would take next, without taking it. */ + peekNextGroup: (subscriber: Subscriber) => number | undefined; /** Take the next group in sequence order without waiting, as {@link tryRecvGroup} does in arrival order. */ tryNextGroup: (subscriber: Subscriber) => Recv; /** Wake once a subscriber's datagram cursor may read differently; fires at most once. */ @@ -376,6 +378,9 @@ export const hooks: { tryNextGroup: () => { throw new Error("track.ts not loaded"); }, + peekNextGroup: () => { + throw new Error("track.ts not loaded"); + }, forgetPastBudget: () => { throw new Error("track.ts not loaded"); }, diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 1bf9cbf36a9..49a3b687e35 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -335,3 +335,31 @@ test("a replacement spliced before the first copy answers is still checked again a.close(); b.close(); }); + +test("an ordered reader takes a replaced copy's buffered group before the new copy's later one", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + front.serve(a.consume()); + const ordered = front.consumer + .track("seq") + .subscribe({ maxDelay: 10_000 as track.Subscription["maxDelay"] }) + .ordered(); + ta.write(0); + expect((await ordered.nextGroup())?.sequence).toBe(0); + + // The old copy holds group 1 unread; the new one only has group 2. + ta.write(1); + tb.write(2); + front.serve(b.consume()); + await settle(); + expect((await ordered.nextGroup())?.sequence).toBe(1); + expect((await ordered.nextGroup())?.sequence).toBe(2); + + ordered.close(); + front.consumer.close(); + a.close(); + b.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index c7be68e4c87..6682737c00b 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -320,6 +320,15 @@ class Reader implements ResumeSource { tryGroup(ordered: boolean, delivered: Delivered): Recv { this.sync(); + // In sequence order, the lowest group any copy holds next, so a replaced copy's buffered + // group is not skipped for a later one the serving copy already has. + if (ordered) { + const next = this.#lowest(delivered); + if (next) { + const recv = hooks.tryNextGroup(next.sub); + if (recv.kind === "group") return { kind: "group", group: this.#handOut(next, recv.group) }; + } + } // The serving copy first: it is the one that keeps delivering. for (let index = this.#copies.length - 1; index >= 0; index--) { const copy = this.#copies[index]; @@ -344,6 +353,27 @@ class Reader implements ResumeSource { return this.#end((copy) => copy.done); } + // The copy whose next group in sequence order is the lowest deliverable one, dropping any it + // holds next that are not. + #lowest(delivered: Delivered): Copy | undefined { + let lowest: { copy: Copy; sequence: number } | undefined; + for (const copy of this.#copies) { + if (copy.done !== undefined) continue; + for (;;) { + const sequence = hooks.peekNextGroup(copy.sub); + if (sequence === undefined) break; + if (this.#deliverable(copy, sequence, delivered)) { + if (!lowest || sequence < lowest.sequence) lowest = { copy, sequence }; + break; + } + const recv = hooks.tryNextGroup(copy.sub); + if (recv.kind !== "group") break; + recv.group.close(); + } + } + return lowest?.copy; + } + tryDatagram(): RecvDatagram { this.sync(); for (let index = this.#copies.length - 1; index >= 0; index--) { diff --git a/js/net/src/track.ts b/js/net/src/track.ts index cedf4fbb168..3eb25e8973f 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -1390,6 +1390,7 @@ export class Subscriber { makeSubscriber = (name, state) => new Subscriber(name, state); hooks.tryRecvGroup = (subscriber) => subscriber.#tryRecvGroup(); hooks.tryNextGroup = (subscriber) => subscriber.#tryNextGroup(); + hooks.peekNextGroup = (subscriber) => subscriber.#peekNextGroup(); hooks.forgetPastBudget = (subscriber, delivered) => subscriber.#forgetPastBudget(delivered); hooks.tryRecvDatagram = (subscriber) => subscriber.#tryDatagram(); hooks.deliveredRuns = (subscriber) => subscriber.#delivered.runs; @@ -1829,6 +1830,19 @@ export class Subscriber { } } + // The sequence #tryNextGroup would take next: the first buffered group in range the budget spares. + #peekNextGroup(): number | undefined { + const cursor = this.#cursor.peek(); + const start = Math.max(cursor.start, this.#nextSequence); + const drift = this.#drift(); + for (const group of this.#state.groups.peek()) { + if (group.sequence < start) continue; + if (cursor.end !== undefined && group.sequence >= cursor.end) return undefined; + if (!this.#isStale(group, drift)) return group.sequence; + } + return undefined; + } + // The synchronous half of #nextGroup. #tryNextGroup(): Recv { if (this.#resume) return this.#tryResumed(true); From 9715f0b6df28c5417fb9c7c9e059e8babdf0523e Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 16 Oct 2026 22:17:51 -0700 Subject: [PATCH 06/31] fix(js/net): check late routes against recorded info, and stop superseded fetches Track info is recorded with the source that gave it, so a route created after info() is still compared before serving. A recovery fetch a later route supersedes is aborted and any group it returns closed, and waits no longer leave abort listeners behind. Co-Authored-By: Claude Opus 5.5 --- js/net/src/resume.test.ts | 59 ++++++++++++++++++ js/net/src/resume.ts | 127 +++++++++++++++++++++++++++----------- 2 files changed, 150 insertions(+), 36 deletions(-) diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 49a3b687e35..bd4a0583653 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -363,3 +363,62 @@ test("an ordered reader takes a replaced copy's buffered group before the new co a.close(); b.close(); }); + +test("a late subscriber's first route is checked against properties an earlier source gave", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + a.createTrack("seq", { priority: 1 }); + b.createTrack("seq", { priority: 2 }); + front.serve(a.consume()); + expect((await front.consumer.track("seq").info()).priority).toBe(1); + + // Nobody reads the track when the route moves, so nothing has been spliced yet. + front.serve(b.consume()); + const sub = front.consumer.track("seq").subscribe(); + await settle(); + expect(() => sub.tryRecvGroup()).toThrow("changed its properties"); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); +}); + +test("a recovery fetch a later route supersedes is abandoned", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const c = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + const tc = copy(c); + const stalled = ta.write(0, 1, false); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + const group = sub.tryRecvGroup(); + if (!group) throw new Error("expected group 0"); + expect(dec.decode((await group.readFrame())?.payload)).toBe("0.0"); + + // The next route does not have the group yet: recovery fetches it there and waits. + front.serve(b.consume()); + await settle(); + stalled.close(new StreamError(StreamCode.SessionClosed)); + const read = group.readFrame(); + await settle(); + const fetching = tb.producer.demand().used.peek(); + + // Another route has it cached: the group continues there, and the fetch from b stops. + tc.write(0, 2); + front.serve(c.consume()); + expect(dec.decode((await read)?.payload)).toBe("0.1"); + sub.close(); + await settle(); + expect(fetching).toBe(true); + expect(tb.producer.demand().used.peek()).toBe(false); + + front.consumer.close(); + a.close(); + b.close(); + c.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 6682737c00b..6e3b5643fc0 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -216,10 +216,13 @@ class Reader implements ResumeSource { #changed = new Signal(0); #dispose: Dispose[] = []; #closed = false; + /** Records the properties a copy committed, and which source it came from. */ + #record: (info: track.Info, source: Source) => void; - constructor(route: Route, shell: ResumeShell) { + constructor(route: Route, shell: ResumeShell, record: (info: track.Info, source: Source) => void) { this.route = route; this.#shell = shell; + this.#record = record; route.readers.add(this); this.#dispose.push( shell.subscription.subscribe(() => { @@ -266,8 +269,12 @@ class Reader implements ResumeSource { this.#bound(copy); this.#copies.push(copy); const shell = this.#shell; + const record = this.#record; copy.sub.info().then( - (info) => shell.commit(info), + (info) => { + record(info, source); + shell.commit(info); + }, () => {}, ); this.#changed.update((n) => n + 1); @@ -415,6 +422,8 @@ class Reader implements ResumeSource { if (!routeFailed(ended)) return this.#finish({ kind: "error", error: ended }); if (route.refusal) return this.#finish({ kind: "error", error: route.refusal }); } + // Refused before any copy fed it: nothing will. + if (!last && route.refusal) return this.#finish({ kind: "error", error: route.refusal }); if (route.concluded) { // No front is left to replace the copy: the track ends as it does. if (!serving) return this.#finish({ kind: "error", error: dropped() }); @@ -549,6 +558,8 @@ interface Fetch { outcome?: { group: GroupConsumer } | { error: Error }; /** Fires once it settles. */ settled: Signal; + /** Abandon it, closing a group it returns, unless the caller kept that group. */ + cancel(): void; } function startFetch( @@ -558,26 +569,55 @@ function startFetch( generation: number, signal: AbortSignal, ): Fetch { - const fetch: Fetch = { generation, settled: new Signal(false) }; - wire.fetchGroup(name, sequence, { signal }) + // Its own controller, so a fetch a later route supersedes stops without ending the recovery. + const controller = new AbortController(); + const onAbort = () => controller.abort(signal.reason); + signal.addEventListener("abort", onAbort, { once: true }); + const fetch: Fetch = { + generation, + settled: new Signal(false), + cancel: () => { + signal.removeEventListener("abort", onAbort); + controller.abort(); + if (fetch.outcome && "group" in fetch.outcome) fetch.outcome.group.close(); + }, + }; + wire.fetchGroup(name, sequence, { signal: controller.signal }) .then( (group) => { fetch.outcome = { group }; + // Abandoned while it was out: nobody reads it. + if (controller.signal.aborted) group.close(); }, (err: unknown) => { fetch.outcome = { error: err instanceof Error ? err : new Error(String(err)) }; }, ) - .finally(() => fetch.settled.set(true)); + .finally(() => { + signal.removeEventListener("abort", onAbort); + fetch.settled.set(true); + }); return fetch; } -/** Resolves once `signal` aborts. */ -function aborted(signal: AbortSignal): Promise { - return new Promise((resolve) => { - if (signal.aborted) resolve(); - else signal.addEventListener("abort", () => resolve(), { once: true }); +/** Wait for `change`, or throw once `signal` aborts, leaving no listener behind either way. */ +async function until(change: PromiseLike, signal?: AbortSignal): Promise { + signal?.throwIfAborted(); + if (!signal) { + await race([change]); + return; + } + let onAbort = () => {}; + const abort = new Promise((resolve) => { + onAbort = resolve; + signal.addEventListener("abort", onAbort, { once: true }); }); + try { + await race([change, abort]); + } finally { + signal.removeEventListener("abort", onAbort); + } + signal.throwIfAborted(); } function dropped(): Error { @@ -618,9 +658,29 @@ class Recover implements GroupRecover { } async replace(_index: number, failed: Error | undefined, signal: AbortSignal): Promise { + let fetch: Fetch | undefined; + let kept: GroupConsumer | undefined; + try { + const replacement = await this.#replace(failed, signal, (next) => { + fetch?.cancel(); + fetch = next; + }); + kept = replacement.group; + return replacement; + } finally { + // A fetch nothing continues from is abandoned, along with whatever it returns. + if (fetch?.outcome === undefined || !("group" in fetch.outcome) || fetch.outcome.group !== kept) + fetch?.cancel(); + } + } + + async #replace( + failed: Error | undefined, + signal: AbortSignal, + started: (fetch: Fetch) => void, + ): Promise { const reader = this.#reader; const route = reader.route; - const abort = aborted(signal); let fetch: Fetch | undefined; for (;;) { signal.throwIfAborted(); @@ -642,8 +702,10 @@ class Recover implements GroupRecover { if (cached) return this.#replacement(cached, serving); } // Otherwise ask for it. The subscription may still deliver it first. - if (fetch?.generation !== generation) + if (fetch?.generation !== generation) { fetch = startFetch(serving.source.wire, route.name, this.#sequence, generation, signal); + started(fetch); + } const stale = hooks.staleAt(serving.sub, this.#sequence); if (stale.expired()) throw new StreamError(StreamCode.Old, { message: "group is past the budget" }); waits.push(...stale.changed, ...(fetch.outcome ? [] : [fetch.settled])); @@ -655,7 +717,7 @@ class Recover implements GroupRecover { throw outcome.error; } } - await race([Signal.race(...waits), abort]); + await until(Signal.race(...waits), signal); } } @@ -686,8 +748,8 @@ export class Front { readonly consumer: broadcast.Consumer; #source?: Source; #routes = new Map(); - /** Each track's properties as first committed, which every later copy must match. */ - #info = new Map(); + /** Each track's properties as first committed, and the source that said so: every later copy must match. */ + #info = new Map(); /** Bumped whenever the source changes, so an answer from a replaced one is dropped. */ #changed = new Signal(0); #concluded = false; @@ -738,8 +800,9 @@ export class Front { #splice(route: Route, source: Source): void { const recorded = this.#info.get(route.name); const previous = route.served; - // The first copy of a track has nothing to match. - if (!recorded && !previous) { + // The first copy of a track has nothing to match, and the source that said what it is + // matches itself. + if ((!recorded && !previous) || recorded?.source === source) { route.serve(source); return; } @@ -748,10 +811,10 @@ export class Front { source.retain(); previous?.retain(); const expected = recorded - ? Promise.resolve(recorded) + ? Promise.resolve(recorded.info) : (previous as Source).wire.resolveTrackInfo(route.name).then( (info) => { - this.#record(route.name, info); + this.#record(route.name, info, previous as Source); return info; }, () => undefined, @@ -761,7 +824,7 @@ export class Front { ([expected, info]) => { if (this.#source !== source) return; // The copy it replaces never answered, so this one says what the track is. - if (!expected) this.#record(route.name, info); + if (!expected) this.#record(route.name, info, source); // A copy claiming the same content with different metadata is refused rather // than reinterpreted. if (!expected || compatible(expected, info)) route.serve(source); @@ -789,13 +852,14 @@ export class Front { if (this.#routes.get(name) === created) this.#routes.delete(name); }; this.#routes.set(name, created); + // A track whose properties another source already gave is checked like any replacement. if (this.#concluded) created.conclude(); - else if (this.#source) created.serve(this.#source); + else if (this.#source) this.#splice(created, this.#source); return created; } - #record(name: string, info: track.Info): void { - if (!this.#info.has(name)) this.#info.set(name, info); + #record(name: string, info: track.Info, source: Source): void { + if (!this.#info.has(name)) this.#info.set(name, { info, source }); } subscribe(name: string, options?: track.Subscription): track.Subscriber { @@ -804,22 +868,13 @@ export class Front { return hooks.resumeSubscriber( name, options, - (shell) => - new Reader(route, { - ...shell, - commit: (info) => { - this.#record(name, info); - shell.commit(info); - }, - }), + (shell) => new Reader(route, shell, (info, source) => this.#record(name, info, source)), ); } // Wait for the source to change, or `signal` to abort. - async #next(signal?: AbortSignal): Promise { - signal?.throwIfAborted(); - await race(signal ? [this.#changed.changed(), aborted(signal)] : [this.#changed.changed()]); - signal?.throwIfAborted(); + #next(signal?: AbortSignal): Promise { + return until(Signal.race(this.#changed), signal); } async resolveTrackInfo(name: string, hold?: AbortSignal): Promise { @@ -833,7 +888,7 @@ export class Front { source.retain(); try { const info = await source.wire.resolveTrackInfo(name, hold); - this.#record(name, info); + this.#record(name, info, source); return info; } catch (err) { const error = err instanceof Error ? err : new Error(String(err)); From d147647fdd3612f3e22fc6868636c07498db5bd0 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 16 Oct 2026 23:05:46 -0700 Subject: [PATCH 07/31] fix(js/net): one info answer per logical track, datagram readers let go, sticky provenance Co-Authored-By: Claude Opus 5.5 --- js/net/src/resume.test.ts | 53 ++++++++++++++++++++++++++++++++++++++- js/net/src/resume.ts | 39 ++++++++++++++++++++++------ 2 files changed, 83 insertions(+), 9 deletions(-) diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index bd4a0583653..357e351d5fb 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -8,7 +8,7 @@ import { Producer } from "./origin.ts"; import * as Path from "./path.ts"; import { Front } from "./resume.ts"; import type * as track from "./track.ts"; -import { wireOf } from "./wire.ts"; +import { overrideBroadcastWire, wireOf } from "./wire.ts"; const EPOCH = Epoch.parse("01900000-0000-7000-8000-000000000001"); const path = Path.from("live"); @@ -422,3 +422,54 @@ test("a recovery fetch a later route supersedes is abandoned", async () => { b.close(); c.close(); }); + +test("track info keeps one answer across routes", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + a.createTrack("seq", { priority: 1 }); + b.createTrack("seq", { priority: 2 }); + front.serve(a.consume()); + expect((await front.consumer.track("seq").info()).priority).toBe(1); + front.serve(b.consume()); + await expect(front.consumer.track("seq").info()).rejects.toThrow("changed its properties"); + + front.consumer.close(); + a.close(); + b.close(); +}); + +test("a datagram reader lets go of the copies a swap replaced", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + copy(b); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + expect(ta.producer.demand().used.peek()).toBe(true); + front.serve(b.consume()); + await settle(); + // A read with nothing to deliver is the pass that lets go of a's copy. + void sub.recvDatagram(); + await settle(); + expect(ta.producer.demand().used.peek()).toBe(false); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); +}); + +test("a front remembers it served a session's broadcast between routes", () => { + const front = new Front(); + const a = new BroadcastProducer(); + const session = a.consume(); + overrideBroadcastWire(session, {}); + front.serve(session); + expect(front.fromSession).toBe(true); + front.park(); + expect(front.fromSession).toBe(true); + front.consumer.close(); + a.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 6e3b5643fc0..d9b00089fb1 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -71,16 +71,23 @@ function compatible(a: track.Info, b: track.Info): boolean { return a.timescale === b.timescale && a.maxAge === b.maxAge && a.priority === b.priority; } +function incompatible(name: string): Error { + return new Error(`track ${name} changed its properties across routes`); +} + /** One route's broadcast, held while the front serves it or a reader still reads a copy of it. */ class Source { readonly wire: BroadcastWire; /** The table's handle this was served from, so serving it again is a no-op. */ readonly from: broadcast.Consumer; + /** Whether a session delivered it, read from the handle served rather than the clone. */ + readonly fromSession: boolean; #consumer: broadcast.Consumer; #holds = 0; constructor(from: broadcast.Consumer) { this.from = from; + this.fromSession = wireOf(from).fromSession === true; this.#consumer = from.clone(); this.wire = wireOf(this.#consumer); } @@ -400,6 +407,7 @@ class Reader implements ResumeSource { break; } } + this.#letGo(); const end = this.#end((copy) => copy.datagramsDone); if (end.kind === "error" || end.kind === "done") return end; return { kind: "idle" }; @@ -441,11 +449,15 @@ class Reader implements ResumeSource { * Drop replaced copies only once neither a handed-out nor an unread group needs them. One not * yet replaced stays, even between routes: it is still the newest copy there is. */ - #letGo(delivered: Delivered): void { + #letGo(delivered?: Delivered): void { const cursor = this.#shell.cursor.peek(); this.#copies = this.#copies.filter((copy) => { if (copy.until === undefined || copy.lease > 0) return true; - if (copy.done === undefined && hooks.hasUnreadGroup(copy.sub, cursor.start, copyEnd(copy), delivered)) { + if ( + copy.done === undefined && + delivered && + hooks.hasUnreadGroup(copy.sub, cursor.start, copyEnd(copy), delivered) + ) { return true; } this.#drop(copy); @@ -753,6 +765,8 @@ export class Front { /** Bumped whenever the source changes, so an answer from a replaced one is dropped. */ #changed = new Signal(0); #concluded = false; + /** Whether any source it served came from a session, kept between routes. */ + #fromSession = false; constructor() { this.consumer = makeFrontConsumer(this); @@ -764,6 +778,7 @@ export class Front { if (this.#concluded || this.#source?.from === from) return; const source = new Source(from); source.retain(); + this.#fromSession ||= source.fromSession; const previous = this.#source; this.#source = source; this.#changed.update((n) => n + 1); @@ -791,9 +806,12 @@ export class Front { this.#changed.update((n) => n + 1); } - /** Whether the serving broadcast came from a session; see {@link BroadcastWire.fromSession}. */ + /** + * Whether a broadcast it served came from a session; see {@link BroadcastWire.fromSession}. + * Sticky, since the front can move back onto a session's route at any time. + */ get fromSession(): boolean { - return this.#source?.wire.fromSession ?? false; + return this.#fromSession; } // Splice `source`'s copy of a track in, once it claims the same content as the copies before. @@ -828,7 +846,7 @@ export class Front { // A copy claiming the same content with different metadata is refused rather // than reinterpreted. if (!expected || compatible(expected, info)) route.serve(source); - else route.refuse(new Error(`track ${route.name} changed its properties across routes`)); + else route.refuse(incompatible(route.name)); }, (err: unknown) => { if (this.#source !== source) return; @@ -886,18 +904,23 @@ export class Front { continue; } source.retain(); + let info: track.Info; try { - const info = await source.wire.resolveTrackInfo(name, hold); - this.#record(name, info, source); - return info; + info = await source.wire.resolveTrackInfo(name, hold); } catch (err) { const error = err instanceof Error ? err : new Error(String(err)); // The route failed: ask the next one, unless none will come. if (!routeFailed(error) || this.#concluded || hold?.aborted) throw error; if (this.#source === source) await this.#next(hold); + continue; } finally { source.release(); } + const recorded = this.#info.get(name); + // One logical track has one set of properties, whichever route answers. + if (recorded && !compatible(recorded.info, info)) throw incompatible(name); + this.#record(name, info, source); + return recorded?.info ?? info; } } From ca964b3c3517d1ec6dbda21847e6d3363eef42d8 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 16 Oct 2026 23:50:42 -0700 Subject: [PATCH 08/31] fix(js/net): end recovery on a refused route, and abandon it when the group ends Co-Authored-By: Claude Opus 5.5 --- js/net/src/group.test.ts | 29 +++++++++++++++++++++++++++++ js/net/src/group.ts | 35 +++++++++++++++++++++++++++++------ js/net/src/resume.test.ts | 27 +++++++++++++++++++++++++++ js/net/src/resume.ts | 2 ++ 4 files changed, 87 insertions(+), 6 deletions(-) diff --git a/js/net/src/group.test.ts b/js/net/src/group.test.ts index 33b199e1107..d06b54e7fe0 100644 --- a/js/net/src/group.test.ts +++ b/js/net/src/group.test.ts @@ -1,6 +1,8 @@ import { expect, test } from "bun:test"; +import { Signal } from "@moq/signals"; import { FrameTooLarge, GroupTooLarge } from "./error.ts"; import { MAX_GROUP_CACHE_BYTES, MAX_GROUP_FRAMES, Producer } from "./group.ts"; +import { hooks } from "./internal.ts"; import { textFrame } from "./mock.ts"; import { Timestamp } from "./time.ts"; @@ -227,3 +229,30 @@ test("a frame larger than the cache is rejected rather than silently dropped", ( producer.close(); expect(consumer.tryReadFrame()).toBeUndefined(); }); + +test("a group that ends on its own abandons the replacement it asked for", async () => { + const producer = new Producer(0); + const consumer = producer.mirror(); + const route = new Signal(0); + let signal: AbortSignal | undefined; + hooks.recoverGroup(consumer, { + changed: route, + // A stall looks past the copy once the route moved. + wants: (failed) => failed !== undefined || route.peek() > 0, + replace: (_index, _failed, abort) => { + signal = abort; + return new Promise(() => {}); + }, + }); + producer.writeFrame({ payload: new Uint8Array([0]) }); + await consumer.readFrame(); + + route.set(1); + const read = consumer.readFrame(); + await new Promise((resolve) => setTimeout(resolve, 0)); + expect(signal?.aborted).toBe(false); + + producer.close(); + expect(await read).toBeUndefined(); + expect(signal?.aborted).toBe(true); +}); diff --git a/js/net/src/group.ts b/js/net/src/group.ts index a40277ed847..27ae5684cd5 100644 --- a/js/net/src/group.ts +++ b/js/net/src/group.ts @@ -576,7 +576,9 @@ export class Consumer { started.result = recover.replace(this.#state.start, failed, abort.signal).then( (replacement) => { started.settled = true; - started.replacement = replacement; + // Abandoned while it was out: nobody reads it. + if (abort.signal.aborted) replacement.group.close(); + else started.replacement = replacement; }, (err: unknown) => { started.settled = true; @@ -605,6 +607,15 @@ export class Consumer { return true; } + // Give up a replacement still being asked for, or one nobody adopted: the group ended without it. + #abandon(): void { + const attempt = this.#attempt; + if (!attempt) return; + this.#attempt = undefined; + attempt.abort.abort(); + attempt.replacement?.group.close(); + } + // Read the rest of the group from `replacement`, from the frame this one reached. #adopt(replacement: Replacement): void { const next = replacement.group.#state; @@ -703,11 +714,15 @@ export class Consumer { /** Resolves once {@link readFrame} would not block. */ async readable(): Promise { for (;;) { - if (this.#terminal || this.#ended) return; + if (this.#terminal || this.#ended) { + this.#abandon(); + return; + } if (this.#state.frames.peek().length > 0) return; const closed = this.#state.closed.peek(); if (closed instanceof Error && (await this.#resume(closed))) continue; if (closed !== undefined) { + this.#abandon(); this.#closed?.refresh(); return; } @@ -730,8 +745,14 @@ export class Consumer { async #readFramePosition(pending = false, from = 0): Promise { for (;;) { - if (this.#terminal) throw this.#terminal; - if (this.#ended) return; + if (this.#terminal) { + this.#abandon(); + throw this.#terminal; + } + if (this.#ended) { + this.#abandon(); + return; + } const read = this.#readBufferedFrame(pending); if (read) { @@ -746,9 +767,12 @@ export class Consumer { const closed = this.#state.closed.peek(); if (closed instanceof Error) { if (await this.#resume(closed)) continue; + this.#abandon(); throw closed; } if (closed !== undefined) { + // The copy finished the group: a replacement asked for meanwhile is moot. + this.#abandon(); this.#closed?.refresh(); return; } @@ -795,8 +819,7 @@ export class Consumer { /** Closes the group, optionally with an error to abort readers. Idempotent. */ close(abort?: Error) { - this.#attempt?.abort.abort(); - this.#attempt = undefined; + this.#abandon(); this.#recover = undefined; // Closing is terminal for this reader, even with frames still unread. this.#released = true; diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 357e351d5fb..8819f7a1586 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -473,3 +473,30 @@ test("a front remembers it served a session's broadcast between routes", () => { front.consumer.close(); a.close(); }); + +test("a failed group on a route whose replacement was refused ends with the refusal", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = a.createTrack("seq", { priority: 1 }); + b.createTrack("seq", { priority: 2 }); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + const stalled = new GroupProducer(0); + ta.writeGroup(stalled); + stalled.writeFrame({ payload: enc.encode("0.0") }); + const group = sub.tryRecvGroup(); + if (!group) throw new Error("expected group 0"); + await group.readFrame(); + await sub.info(); + + front.serve(b.consume()); + await settle(); + stalled.close(new StreamError(StreamCode.SessionClosed)); + await expect(group.readFrame()).rejects.toThrow("changed its properties"); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index d9b00089fb1..0c32a4a481d 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -701,6 +701,8 @@ class Recover implements GroupRecover { reader.sync(); const current = this.#copy; if (!current) throw new Error("group closed"); + // The front refused the route that would continue it: nothing will. + if (failed && route.refusal) throw route.refusal; const generation = route.generation; const serving = route.source ? reader.copy(generation) : undefined; const waits: Getter[] = [route.changed]; From 1c5b482681464dc0c1cfc16ef13f1c37f29c4bcf Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Sat, 17 Oct 2026 00:22:41 -0700 Subject: [PATCH 09/31] fix(js/net): end group and datagram channels apart, keep copies group reads need, verify fetch sources Co-Authored-By: Claude Opus 5.5 --- js/net/src/internal.ts | 7 ++-- js/net/src/resume.test.ts | 61 ++++++++++++++++++++++++++++++++ js/net/src/resume.ts | 73 +++++++++++++++++++++++++++++---------- js/net/src/track.ts | 5 ++- 4 files changed, 124 insertions(+), 22 deletions(-) diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index 9bb40e7c3a6..0fc578c6b0b 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -224,8 +224,11 @@ export type RecvDatagram = export interface ResumeSource { /** The next group not in `delivered`, in arrival or sequence order, without waiting. */ tryGroup(ordered: boolean, delivered: Delivered): Recv; - /** The next datagram in arrival order, without waiting. */ - tryDatagram(): RecvDatagram; + /** + * The next datagram in arrival order, without waiting. `delivered` is the group record once + * groups are read too, so a copy that still holds one for them is kept. + */ + tryDatagram(delivered: Delivered | undefined): RecvDatagram; /** Wake once a read may go differently; fires at most once. */ changed(fn: () => void): Dispose; /** Forget the gaps in `delivered` below each group past the budget, judged on the newest copy. */ diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 8819f7a1586..1894a6752fd 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -500,3 +500,64 @@ test("a failed group on a route whose replacement was refused ends with the refu a.close(); b.close(); }); + +test("draining groups to their end leaves buffered datagrams readable", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const ta = copy(a); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + ta.write(0); + ta.producer.appendDatagram(undefined, enc.encode("d")); + ta.producer.close(); + expect(drain(sub)).toEqual([0]); + expect(await sub.recvGroup()).toBeUndefined(); + expect(dec.decode((await sub.recvDatagram())?.payload)).toBe("d"); + + sub.close(); + front.consumer.close(); + a.close(); +}); + +test("a datagram read keeps a replaced copy a group reader still needs", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + copy(b); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe({ maxDelay: 10_000 as track.Subscription["maxDelay"] }); + ta.write(0); + expect(drain(sub)).toEqual([0]); + ta.write(1); + + front.serve(b.consume()); + await settle(); + void sub.recvDatagram(); + expect(drain(sub)).toEqual([1]); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); +}); + +test("a fetch never reads a replacement refused for its track properties", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + a.createTrack("seq", { priority: 1 }); + const tb = b.createTrack("seq", { priority: 2 }); + const group = new GroupProducer(0); + tb.writeGroup(group); + group.writeFrame({ payload: enc.encode("b") }); + group.close(); + front.serve(a.consume()); + await front.consumer.track("seq").info(); + front.serve(b.consume()); + await expect(front.consumer.track("seq").fetchGroup(0)).rejects.toThrow("changed its properties"); + + front.consumer.close(); + a.close(); + b.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 0c32a4a481d..0cd293fb0c0 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -50,8 +50,17 @@ const ROUTE_FAILURES: ReadonlySet = new Set([ StreamCode.ControlTimeout, ]); +/** A copy claiming the same content with different metadata: refused rather than reinterpreted. */ +class Incompatible extends Error { + constructor(name: string) { + super(`track ${name} changed its properties across routes`); + this.name = "Incompatible"; + } +} + /** Whether `err` is the route failing rather than refusing: another route may still deliver what it failed. */ function routeFailed(err: Error): boolean { + if (err instanceof Incompatible) return false; return !(err instanceof StreamError) || ROUTE_FAILURES.has(err.code); } @@ -71,10 +80,6 @@ function compatible(a: track.Info, b: track.Info): boolean { return a.timescale === b.timescale && a.maxAge === b.maxAge && a.priority === b.priority; } -function incompatible(name: string): Error { - return new Error(`track ${name} changed its properties across routes`); -} - /** One route's broadcast, held while the front serves it or a reader still reads a copy of it. */ class Source { readonly wire: BroadcastWire; @@ -223,6 +228,8 @@ class Reader implements ResumeSource { #changed = new Signal(0); #dispose: Dispose[] = []; #closed = false; + /** The channels that ran out cleanly. */ + #drained = new Set<"groups" | "datagrams">(); /** Records the properties a copy committed, and which source it came from. */ #record: (info: track.Info, source: Source) => void; @@ -364,7 +371,7 @@ class Reader implements ResumeSource { } } this.#letGo(delivered); - return this.#end((copy) => copy.done); + return this.#end("groups", (copy) => copy.done); } // The copy whose next group in sequence order is the lowest deliverable one, dropping any it @@ -388,7 +395,7 @@ class Reader implements ResumeSource { return lowest?.copy; } - tryDatagram(): RecvDatagram { + tryDatagram(delivered: Delivered | undefined): RecvDatagram { this.sync(); for (let index = this.#copies.length - 1; index >= 0; index--) { const copy = this.#copies[index]; @@ -407,8 +414,8 @@ class Reader implements ResumeSource { break; } } - this.#letGo(); - const end = this.#end((copy) => copy.datagramsDone); + this.#letGo(delivered); + const end = this.#end("datagrams", (copy) => copy.datagramsDone); if (end.kind === "error" || end.kind === "done") return end; return { kind: "idle" }; } @@ -417,31 +424,41 @@ class Reader implements ResumeSource { * Whether the track ended, once the serving copy's channel (as `done` reads it) ran out: * cleanly, or with a verdict a route failure does not excuse. */ - #end(done: (copy: Copy) => Error | null | undefined): Recv & { kind: "idle" | "done" | "error" } { + #end( + channel: "groups" | "datagrams", + done: (copy: Copy) => Error | null | undefined, + ): Recv & { kind: "idle" | "done" | "error" } { const route = this.route; const last = this.#copies.at(-1); const serving = last?.generation === this.#generation ? last : undefined; const ended = serving ? done(serving) : undefined; // The serving copy drained a complete track: that is its end, whichever route would // serve it next. - if (serving && ended === null) return this.#finish({ kind: "done" }); + if (serving && ended === null) return this.#finish({ kind: "done" }, channel); if (ended instanceof Error) { // A refusal from a route that still serves is the track's answer. - if (!routeFailed(ended)) return this.#finish({ kind: "error", error: ended }); - if (route.refusal) return this.#finish({ kind: "error", error: route.refusal }); + if (!routeFailed(ended)) return this.#finish({ kind: "error", error: ended }, channel); + if (route.refusal) return this.#finish({ kind: "error", error: route.refusal }, channel); } // Refused before any copy fed it: nothing will. - if (!last && route.refusal) return this.#finish({ kind: "error", error: route.refusal }); + if (!last && route.refusal) return this.#finish({ kind: "error", error: route.refusal }, channel); if (route.concluded) { // No front is left to replace the copy: the track ends as it does. - if (!serving) return this.#finish({ kind: "error", error: dropped() }); - if (ended instanceof Error) return this.#finish({ kind: "error", error: ended }); + if (!serving) return this.#finish({ kind: "error", error: dropped() }, channel); + if (ended instanceof Error) return this.#finish({ kind: "error", error: ended }, channel); } return { kind: "idle" }; } - #finish(end: T): T { - this.#shell.close(end.kind === "error" ? end.error : undefined); + // End the logical track on a failure, or cleanly once both channels drained: groups and + // datagrams end apart, and one running out leaves the other's backlog readable. + #finish(end: T, channel: "groups" | "datagrams"): T { + if (end.kind === "error") { + this.#shell.close(end.error); + return end; + } + this.#drained.add(channel); + if (this.#drained.size === 2) this.#shell.close(); return end; } @@ -767,6 +784,8 @@ export class Front { /** Bumped whenever the source changes, so an answer from a replaced one is dropped. */ #changed = new Signal(0); #concluded = false; + /** The tracks each source was checked against the recorded properties for; see #verify. */ + #verified = new WeakMap>(); /** Whether any source it served came from a session, kept between routes. */ #fromSession = false; @@ -848,7 +867,7 @@ export class Front { // A copy claiming the same content with different metadata is refused rather // than reinterpreted. if (!expected || compatible(expected, info)) route.serve(source); - else route.refuse(incompatible(route.name)); + else route.refuse(new Incompatible(route.name)); }, (err: unknown) => { if (this.#source !== source) return; @@ -878,6 +897,21 @@ export class Front { return created; } + // Check `source` claims the same properties for `name` as recorded, once per source: a fetch + // never reads a copy a subscription would refuse. + async #verify(name: string, source: Source): Promise { + const recorded = this.#info.get(name); + if (!recorded || recorded.source === source) return; + let verified = this.#verified.get(source); + if (verified?.has(name)) return; + // A one-shot query: a hold would keep the track wanted for as long as the fetch's signal. + const info = await source.wire.resolveTrackInfo(name); + if (!compatible(recorded.info, info)) throw new Incompatible(name); + verified ??= new Set(); + verified.add(name); + this.#verified.set(source, verified); + } + #record(name: string, info: track.Info, source: Source): void { if (!this.#info.has(name)) this.#info.set(name, { info, source }); } @@ -920,7 +954,7 @@ export class Front { } const recorded = this.#info.get(name); // One logical track has one set of properties, whichever route answers. - if (recorded && !compatible(recorded.info, info)) throw incompatible(name); + if (recorded && !compatible(recorded.info, info)) throw new Incompatible(name); this.#record(name, info, source); return recorded?.info ?? info; } @@ -983,6 +1017,7 @@ export class Front { } source.retain(); try { + await this.#verify(name, source); return { group: await source.wire.fetchGroup(name, sequence, options), source }; } catch (err) { source.release(); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 3eb25e8973f..f5fe497f0fd 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -1203,6 +1203,8 @@ export class Subscriber { // Every group sequence this reader handed out, so a re-created group or one a second route // delivers is handed out once. #delivered = new Delivered(); + // Whether a resumed reader has read groups, so a datagram read keeps copies they still need. + #groupsRead = false; #nextSequence = 0; #cursor = new Signal<{ start: number; end?: number }>({ start: 0 }); // A FETCH's own budget in place of the subscription's; unlike SUBSCRIBE's zero @@ -1693,6 +1695,7 @@ export class Subscriber { // A group from the routes' copies; see #resume. #tryResumed(ordered: boolean): Recv { const resume = this.#resume as ResumeSource; + this.#groupsRead = true; const closed = this.#state.closed.peek(); if (closed instanceof Error) return { kind: "error", error: closed }; if (closed !== undefined) return { kind: "done" }; @@ -1795,7 +1798,7 @@ export class Subscriber { datagram.sequence >= start && (end === undefined || datagram.sequence < end); if (this.#resume) { for (;;) { - const recv = this.#resume.tryDatagram(); + const recv = this.#resume.tryDatagram(this.#groupsRead ? this.#delivered : undefined); if (recv.kind !== "datagram" || inRange(recv.datagram)) return recv; } } From 47e442cd11cdb07b2c1c64c73e1918567b643684 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Sat, 17 Oct 2026 00:54:24 -0700 Subject: [PATCH 10/31] fix(js/net): end rather than adopt at a racing clean end, abort fetch checks, record first fetch info Co-Authored-By: Claude Opus 5.5 --- js/net/src/group.test.ts | 36 ++++++++++++++++++++++++++++++- js/net/src/group.ts | 6 +++--- js/net/src/resume.test.ts | 45 +++++++++++++++++++++++++++++++++++++++ js/net/src/resume.ts | 9 +++++--- 4 files changed, 89 insertions(+), 7 deletions(-) diff --git a/js/net/src/group.test.ts b/js/net/src/group.test.ts index d06b54e7fe0..91297de661f 100644 --- a/js/net/src/group.test.ts +++ b/js/net/src/group.test.ts @@ -2,7 +2,7 @@ import { expect, test } from "bun:test"; import { Signal } from "@moq/signals"; import { FrameTooLarge, GroupTooLarge } from "./error.ts"; import { MAX_GROUP_CACHE_BYTES, MAX_GROUP_FRAMES, Producer } from "./group.ts"; -import { hooks } from "./internal.ts"; +import { hooks, type Replacement } from "./internal.ts"; import { textFrame } from "./mock.ts"; import { Timestamp } from "./time.ts"; @@ -256,3 +256,37 @@ test("a group that ends on its own abandons the replacement it asked for", async expect(await read).toBeUndefined(); expect(signal?.aborted).toBe(true); }); + +test("a group that ends as its replacement arrives ends rather than adopting it", async () => { + const producer = new Producer(0); + const consumer = producer.mirror(); + const route = new Signal(0); + const replacement = new Producer(0); + let resolve: ((value: Replacement) => void) | undefined; + let adopted = false; + hooks.recoverGroup(consumer, { + changed: route, + wants: (failed) => failed !== undefined || route.peek() > 0, + replace: () => + new Promise((r) => { + resolve = r; + }), + }); + producer.writeFrame({ payload: new Uint8Array([0]) }); + await consumer.readFrame(); + + route.set(1); + const read = consumer.readFrame(); + await new Promise((r) => setTimeout(r, 0)); + // Both land before the read looks again. + resolve?.({ + group: replacement.mirror(), + expiry: { expired: () => false, changed: [] }, + adopt: () => { + adopted = true; + }, + }); + producer.close(); + expect(await read).toBeUndefined(); + expect(adopted).toBe(false); +}); diff --git a/js/net/src/group.ts b/js/net/src/group.ts index 27ae5684cd5..a2b969606fc 100644 --- a/js/net/src/group.ts +++ b/js/net/src/group.ts @@ -598,8 +598,8 @@ export class Consumer { if (this.#attempt === attempt) this.#attempt = undefined; const replacement = attempt.replacement; if (!replacement) return true; - if (!failed && this.#state.frames.peek().length > 0) { - // The copy being read delivered meanwhile; keep reading it. + if (!failed && (this.#state.frames.peek().length > 0 || this.#state.closed.peek() !== undefined)) { + // The copy being read delivered or ended meanwhile; keep reading it. replacement.group.close(); return true; } @@ -628,7 +628,7 @@ export class Consumer { } replacement.adopt(); // The copy left behind stops teeing into the state this consumer read. - this.#state.closed.peek() ?? this.#state.closed.set(null); + if (this.#state.closed.peek() === undefined) this.#state.closed.set(null); this.#state = next; this.#expiry = replacement.expiry; this.#skip = index; diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 1894a6752fd..65215aa289f 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -561,3 +561,48 @@ test("a fetch never reads a replacement refused for its track properties", async a.close(); b.close(); }); + +test("the first fetch of a track says what the track is for later routes", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + for (const [source, priority] of [ + [a, 1], + [b, 2], + ] as const) { + const producer = source.createTrack("seq", { priority }); + const group = new GroupProducer(0); + producer.writeGroup(group); + group.writeFrame({ payload: enc.encode(`${priority}`) }); + group.close(); + } + front.serve(a.consume()); + (await front.consumer.track("seq").fetchGroup(0)).close(); + front.serve(b.consume()); + await expect(front.consumer.track("seq").fetchGroup(0)).rejects.toThrow("changed its properties"); + + front.consumer.close(); + a.close(); + b.close(); +}); + +test("aborting a fetch stops it while its route's properties are still being checked", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + a.createTrack("seq"); + // b serves tracks on demand but never answers, so its properties never arrive. + void wireOf(b).requested(); + front.serve(a.consume()); + await front.consumer.track("seq").info(); + front.serve(b.consume()); + const abort = new AbortController(); + const fetching = front.consumer.track("seq").fetchGroup(0, { signal: abort.signal }); + await settle(); + abort.abort(new Error("cancelled")); + await expect(fetching).rejects.toThrow("cancelled"); + + front.consumer.close(); + a.close(); + b.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 0cd293fb0c0..2823517825c 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -901,12 +901,15 @@ export class Front { // never reads a copy a subscription would refuse. async #verify(name: string, source: Source): Promise { const recorded = this.#info.get(name); - if (!recorded || recorded.source === source) return; + if (recorded?.source === source) return; let verified = this.#verified.get(source); if (verified?.has(name)) return; // A one-shot query: a hold would keep the track wanted for as long as the fetch's signal. const info = await source.wire.resolveTrackInfo(name); - if (!compatible(recorded.info, info)) throw new Incompatible(name); + // The first answer says what the track is; later sources have to match it. + const expected = this.#info.get(name); + if (!expected) this.#record(name, info, source); + else if (!compatible(expected.info, info)) throw new Incompatible(name); verified ??= new Set(); verified.add(name); this.#verified.set(source, verified); @@ -1017,7 +1020,7 @@ export class Front { } source.retain(); try { - await this.#verify(name, source); + await until(this.#verify(name, source), options?.signal); return { group: await source.wire.fetchGroup(name, sequence, options), source }; } catch (err) { source.release(); From eeac660f918865c76dd53c957c58b274fffd3157 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Sat, 17 Oct 2026 01:19:43 -0700 Subject: [PATCH 11/31] fix(js/net): a refused route stays refused, and recovery drops stale closed watchers Co-Authored-By: Claude Opus 5.5 --- js/net/src/group.test.ts | 32 ++++++++++++++++++++++++++++++++ js/net/src/group.ts | 8 +++++++- js/net/src/resume.test.ts | 26 ++++++++++++++++++++++++++ js/net/src/resume.ts | 3 ++- 4 files changed, 67 insertions(+), 2 deletions(-) diff --git a/js/net/src/group.test.ts b/js/net/src/group.test.ts index 91297de661f..d86c49a6d64 100644 --- a/js/net/src/group.test.ts +++ b/js/net/src/group.test.ts @@ -290,3 +290,35 @@ test("a group that ends as its replacement arrives ends rather than adopting it" expect(await read).toBeUndefined(); expect(adopted).toBe(false); }); + +test("a group recovered twice reads on and settles", async () => { + const producer = new Producer(0); + const consumer = producer.mirror(); + const route = new Signal(0); + const replacements = [new Producer(0), new Producer(0)]; + let next = 0; + hooks.recoverGroup(consumer, { + changed: route, + wants: (failed) => failed !== undefined, + replace: async () => ({ + group: replacements[next++].mirror(), + expiry: { expired: () => false, changed: [] }, + adopt: () => {}, + }), + }); + void consumer.closed; + producer.writeFrame({ payload: new Uint8Array([0]) }); + await consumer.readFrame(); + + // Two failures in a row: each moves the group to the next copy. + producer.close(new Error("route failed")); + replacements[0].writeFrame({ payload: new Uint8Array([0]) }); + replacements[0].writeFrame({ payload: new Uint8Array([1]) }); + expect((await consumer.readFrame())?.payload[0]).toBe(1); + replacements[0].close(new Error("route failed")); + for (const payload of [0, 1, 2]) replacements[1].writeFrame({ payload: new Uint8Array([payload]) }); + replacements[1].close(); + expect((await consumer.readFrame())?.payload[0]).toBe(2); + expect(await consumer.readFrame()).toBeUndefined(); + expect(await consumer.closed).toBeNull(); +}); diff --git a/js/net/src/group.ts b/js/net/src/group.ts index a2b969606fc..cd7c0fcbff6 100644 --- a/js/net/src/group.ts +++ b/js/net/src/group.ts @@ -333,6 +333,8 @@ class CombinedClosed implements GetPromise { #final: (value: Error | null) => boolean; #value = new Once(); #watched?: Once; + // The watcher on `#watched`, replaced when a recovery moves to another copy's state. + #unwatch?: Dispose; #dispose: Dispose[] = []; constructor( @@ -356,8 +358,9 @@ class CombinedClosed implements GetPromise { preferred !== undefined ? preferred : current !== undefined && this.#final(current) ? current : undefined; if (value === undefined) { if (this.#watched !== source) { + this.#unwatch?.(); this.#watched = source; - this.#dispose.push(source.changed(() => this.#sync())); + this.#unwatch = source.changed(() => this.#sync()); } return false; } @@ -365,6 +368,9 @@ class CombinedClosed implements GetPromise { this.#value.set(value); for (const dispose of this.#dispose) dispose(); this.#dispose = []; + this.#unwatch?.(); + this.#unwatch = undefined; + this.#watched = undefined; return true; } diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 65215aa289f..98bf24d4eeb 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -606,3 +606,29 @@ test("aborting a fetch stops it while its route's properties are still being che a.close(); b.close(); }); + +test("a refused route is not served again by a later compatible source", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const c = new BroadcastProducer(); + const ta = copy(a); + b.createTrack("seq", { priority: 2 }); + const tc = copy(c); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + ta.write(0); + expect(drain(sub)).toEqual([0]); + front.serve(b.consume()); + await settle(); + front.serve(c.consume()); + await settle(); + tc.write(1); + expect(drain(sub)).toEqual([]); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); + c.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 2823517825c..3191fc4df2d 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -135,7 +135,8 @@ class Route { /** Serve the track from `source` from now on, subscribing every reader to it. */ serve(source: Source): void { - if (this.concluded || this.source === source) return; + // A refusal ends the track: no later source continues it, compatible or not. + if (this.concluded || this.refusal || this.source === source) return; this.generation++; this.source = source; this.served = source; From c996cb7f67ce9e6ba166fa919cc5c258e023a921 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Sat, 17 Oct 2026 02:20:46 -0700 Subject: [PATCH 12/31] ci: retrigger checks Co-Authored-By: Claude Opus 5.5 From 07599189e674d614f1add4d07778eefedccf9dca Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Sat, 17 Oct 2026 08:01:42 -0700 Subject: [PATCH 13/31] fix(net): bound a replaced route's copy only once the new copy delivers A replaced copy kept only the groups it had at the swap, so a group the old route got after the swap, already superseded when the new route's subscription started, reached nobody at a zero budget. Now a replaced copy keeps delivering until the serving copy delivers its first group F, then is bounded at max(newest handed out, F - 1). Same rule in rs/moq-net and js/net, with the deterministic regression test in both. Also plans the Rust port of JS's ordered-read fix as quest/m1/resume-ordered.md. Co-Authored-By: Claude Opus 5.5 --- js/net/src/resume.test.ts | 31 ++++++++++++ js/net/src/resume.ts | 60 ++++++++++++++++------- quest/m1/README.md | 1 + quest/m1/resume-ordered.md | 37 ++++++++++++++ rs/moq-net/src/model/resume.rs | 89 ++++++++++++++++++++++++++++------ 5 files changed, 185 insertions(+), 33 deletions(-) create mode 100644 quest/m1/resume-ordered.md diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 98bf24d4eeb..39b836bd9fc 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -632,3 +632,34 @@ test("a refused route is not served again by a later compatible source", async ( b.close(); c.close(); }); + +// A replaced copy is bounded only once the new copy delivers its first group: one the old route +// gets after the swap, which the new route never has, still arrives. +test("a group the old route gets after the swap, which the new route never has, is delivered", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + ta.write(0); + expect(drain(sub)).toEqual([0]); + front.serve(b.consume()); + await settle(); + // Group 1 reaches the old route after the swap; the new route joins at group 2. + ta.write(1); + expect(drain(sub)).toEqual([1]); + tb.write(2); + expect(drain(sub)).toEqual([2]); + // Bounded at the group before the new copy's first: later groups come from the new route. + ta.write(3); + expect(drain(sub)).toEqual([]); + tb.write(3); + expect(drain(sub)).toEqual([3]); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 8222c9d66bc..401ff731fd6 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -9,9 +9,10 @@ * route it is a passthrough. * * When the front switches, each reader subscribes to the new copy from the newest group it - * handed out. It keeps replaced copies for unread groups they already had and groups still being - * read, bounds their subscriptions after their newest group at replacement, and drops them once - * neither needs them. A group carries on across the switch through its {@link Recover}: once its + * handed out. A replaced copy keeps delivering until the new copy delivers its first group, then + * is bounded just below it (or at the newest group handed out, if later), so a group only the old + * route got after the switch is not lost. It is kept for unread groups it had and groups still + * being read, and dropped once neither needs it. A group carries on across the switch through its {@link Recover}: once its * copy fails with its route, or stalls while a newer route serves, it continues from the serving * route's copy of the same group at the same frame, found in that copy's cache or fetched from it. * @@ -181,9 +182,12 @@ interface Copy { sub: track.Subscriber; /** Where the subscription was asked to start, so demand updates never ask for less. */ floor?: number; + /** Replaced by a newer route: its datagrams are the newer copy's to deliver. */ + replaced: boolean; /** - * Replaced by a newer route: nothing past this group, the newest it had then, is taken from - * it. `null` when it had none. + * Bounded once the newer copy delivered its first group: nothing past this group is taken + * from it. `null` when that leaves nothing. Unset while it is the only copy still feeding + * groups the newer one may never have. */ until?: number | null; /** The group channel ran out: cleanly (`null`) or failed. */ @@ -252,7 +256,7 @@ class Reader implements ResumeSource { this.sync(); } - /** Follow the route: subscribe to a new copy and bound the ones it replaced. */ + /** Follow the route: subscribe to a new copy and mark the ones it replaced. */ sync(): void { if (this.#closed) return; const route = this.route; @@ -260,15 +264,9 @@ class Reader implements ResumeSource { this.#generation = route.generation; const source = route.source; if (!source) return; - // The replaced copies finish the groups they had when replaced, and nothing newer. - for (const copy of this.#copies) { - if (copy.until !== undefined) continue; - const latest = copy.sub.latest(); - const newest = this.#newest === undefined ? latest : Math.max(this.#newest, latest ?? 0); - copy.until = newest ?? null; - this.#update(copy); - this.#bound(copy); - } + // The replaced copies keep delivering until the new one delivers its first group: a + // group they get meanwhile may be one the new route never has. See #observe. + for (const copy of this.#copies) copy.replaced = true; const floor = this.#newest; const copy: Copy = { generation: route.generation, @@ -278,6 +276,7 @@ class Reader implements ResumeSource { sub: subscribe(source.wire, route.name, this.#wire({ floor })), floor, datagramsSeen: this.#datagram, + replaced: false, lease: 0, }; source.retain(); @@ -335,6 +334,22 @@ class Reader implements ResumeSource { return this.#copies.find((copy) => copy.generation === generation); } + /** + * `copy` delivered `sequence`. The serving copy's first group bounds the copies it replaced + * just below it: what they hold under it may be groups the new route never has, and what + * is at or above it the new route delivers. Never below a group already handed out. + */ + #observe(copy: Copy, sequence: number): void { + if (copy.generation !== this.#generation) return; + const bound = Math.max(this.#newest ?? -1, sequence - 1); + for (const replaced of this.#copies) { + if (replaced === copy || !replaced.replaced || replaced.until !== undefined) continue; + replaced.until = bound < 0 ? null : bound; + this.#update(replaced); + this.#bound(replaced); + } + } + #deliverable(copy: Copy, sequence: number, delivered: Delivered): boolean { const end = copyEnd(copy); return (end === undefined || sequence < end) && !delivered.has(sequence); @@ -348,7 +363,10 @@ class Reader implements ResumeSource { const next = this.#lowest(delivered); if (next) { const recv = hooks.tryNextGroup(next.sub); - if (recv.kind === "group") return { kind: "group", group: this.#handOut(next, recv.group) }; + if (recv.kind === "group") { + this.#observe(next, recv.group.sequence); + return { kind: "group", group: this.#handOut(next, recv.group) }; + } } } // The serving copy first: it is the one that keeps delivering. @@ -358,6 +376,7 @@ class Reader implements ResumeSource { for (;;) { const recv = ordered ? hooks.tryNextGroup(copy.sub) : hooks.tryRecvGroup(copy.sub, false); if (recv.kind === "group") { + this.#observe(copy, recv.group.sequence); if (!this.#deliverable(copy, recv.group.sequence, delivered)) { recv.group.close(); continue; @@ -390,6 +409,7 @@ class Reader implements ResumeSource { } const recv = hooks.tryNextGroup(copy.sub); if (recv.kind !== "group") break; + this.#observe(copy, recv.group.sequence); recv.group.close(); } } @@ -401,7 +421,7 @@ class Reader implements ResumeSource { for (let index = this.#copies.length - 1; index >= 0; index--) { const copy = this.#copies[index]; // A replaced copy's datagrams are the serving one's to deliver. - if (copy.datagramsDone !== undefined || copy.until !== undefined) continue; + if (copy.datagramsDone !== undefined || copy.replaced) continue; for (;;) { const recv = hooks.tryRecvDatagram(copy.sub); if (recv.kind === "datagram") { @@ -470,7 +490,11 @@ class Reader implements ResumeSource { #letGo(delivered?: Delivered): void { const cursor = this.#shell.cursor.peek(); this.#copies = this.#copies.filter((copy) => { - if (copy.until === undefined || copy.lease > 0) return true; + // The serving copy, or one between routes with nothing replacing it yet. + if (!copy.replaced || copy.lease > 0) return true; + // Not yet bounded, it may still feed a group the new copy never has, unless nobody + // reads groups at all. + if (copy.until === undefined && delivered) return true; if ( copy.done === undefined && delivered && diff --git a/quest/m1/README.md b/quest/m1/README.md index 7f4d930fb61..3aec919e80a 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -106,6 +106,7 @@ lands before the cut. - [TS fidelity and recovery](/quest/m1/ts-reliability/README.md) - both modes have explicit profiles, automated validation, loss tests, and a dedicated guide - [Draft 14-16 updates](/quest/m1/ietf-legacy-updates.md) - Rust and JS apply and answer moq-transport 14-16 request updates without ending or leaking the request - [Resume reorder](/quest/m1/resume-reorder.md) - a group whose stream header arrives after its successor's still resumes on failover; only a dropped or long-gone group fails +- [Rust resume ordered reads](/quest/m1/resume-ordered.md) - a Rust ordered reader takes the lowest group across route copies, as JS does - [Refuse rewinds](/quest/m1/refuse-rewinds.md) - a timed track refuses a group whose first frame is earlier than the previous group's, in Rust and js/net, on every hop - [JS pending tail](/quest/m1/js-pending-tail.md) - a JS reader holds for a track's pending tail like Rust - [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS diff --git a/quest/m1/resume-ordered.md b/quest/m1/resume-ordered.md new file mode 100644 index 00000000000..0bd62d25621 --- /dev/null +++ b/quest/m1/resume-ordered.md @@ -0,0 +1,37 @@ +# [S] Rust ordered reads take the lowest group across route copies + +## Goal + +A Rust reader reading a logical track in sequence order (`poll_next_group`) +across a same-epoch route swap never skips a group a replaced route's copy +already holds just because the serving copy offered a later one first. This +matches JS since #5836. + +## Plan + +Decided 2026-10-17 by the maintainer on #5836: JS keeps reading the lowest next +group across copies, and Rust follows rather than JS reverting. + +Today `Reader::poll_group_once` in `rs/moq-net/src/model/resume.rs` polls the +serving copy first in both orders. In sequence order, `track::Subscriber::hand_out` +then forgets every sequence below the group it returns. So a replaced copy's +buffered N+1 is dropped when the serving copy offers N+2 first, a group a direct +ordered reader would have delivered. + +- Mirror JS `Reader.#lowest`: in sequence order, peek each live copy's next + deliverable group without moving its cursor, drop what is not deliverable, + and take from the copy with the lowest one. Arrival order keeps serving-first. +- The peek needs a non-consuming counterpart of `poll_next_group` on + `track::Subscriber` that applies the same floor, cap, and drift budget. +- Feed the taken group through `Reader::observe` as the other paths do, so the + deferred bound on replaced copies still applies. +- Regression test: an ordered reader across a swap where the old copy holds + group 1 unread and the new copy only has group 2 reads 1 then 2, as JS + `an ordered reader takes a replaced copy's buffered group before the new + copy's later one` does. + +Public API: behavior only. Wire: none. + +## Related + +- [Resume reorder](/quest/m1/resume-reorder.md) - another resume verdict that reads a passed group too eagerly diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 08d72cb3164..1314520576e 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -7,9 +7,10 @@ //! [`Subscriber`]: nothing is copied, and with one route it is a passthrough. //! //! When the front switches, each reader subscribes to the new copy from the newest group -//! it handed out. It keeps replaced copies for unread groups they already had and groups -//! still being read, bounds their subscriptions after their newest group at replacement, -//! and drops them once neither needs them. A group carries on across the switch through +//! it handed out. A replaced copy keeps delivering until the new copy delivers its first +//! group, then is bounded just below it (or at the newest group handed out, if later), so a +//! group only the old route got after the switch is not lost. It is kept for unread groups it +//! had and groups still being read, and dropped once neither needs it. A group carries on across the switch through //! its [`Recover`]: once its copy fails with its route, or stalls while a newer route //! serves, it continues from the serving route's copy of the same group at the same frame //! (and byte), found in that copy's cache or fetched from it. @@ -153,6 +154,7 @@ impl Consumer { newest: None, datagram: None, groups: (0, Cap::from(..)), + groups_read: false, }) }); // Subscribed to the serving copy at once, so groups it has now are the reader's @@ -240,8 +242,10 @@ struct Copy { sub: Sub, /// Where the subscription was asked to start, so demand updates never ask for less. floor: Option, - /// Replaced by a newer route: nothing past this group, the newest it had then, is - /// taken from it. + /// Replaced by a newer route: its datagrams are the newer copy's to deliver. + replaced: bool, + /// Bounded once the newer copy delivered its first group: nothing past this group is + /// taken from it. Unset while it may still feed a group the newer copy never has. until: Option>, /// The group channel ran out. done: Option>, @@ -271,7 +275,7 @@ impl Copy { if let Some(end) = self.end().map(Position::group) { wire.end = Some(wire.end.map_or(end, |requested| requested.min(end))); } - // A replaced copy only finishes what it had, so a new group from it never arrives. + // A bounded copy only finishes what it had, so a new group from it never arrives. if self.until.is_some() { wire.new_group = None; } @@ -463,10 +467,12 @@ struct Reader { datagram: Option<(u64, u64)>, /// The local group filter; see [`track::Subscriber::set_groups`]. groups: (u64, Cap), + /// Whether groups were read, so a datagram read keeps the copies they still need. + groups_read: bool, } impl Reader { - /// Follow the route: subscribe to a new copy, bound the ones it replaced, and mirror + /// Follow the route: subscribe to a new copy, mark the ones it replaced, and mirror /// the reader's preferences into each. fn sync(&mut self, waiter: &kio::Waiter) { // Snapshot and re-arm together, including when a change was observed: another @@ -497,12 +503,10 @@ impl Reader { if self.copies.last().is_some_and(|copy| copy.generation == generation) { return; } - // The replaced copies finish the groups they had when replaced, and nothing newer. + // The replaced copies keep delivering until the new one delivers its first group: a + // group they get meanwhile may be one the new route never has. See `observe`. for copy in &mut self.copies { - if copy.until.is_none() { - copy.until = Some(self.newest.max(copy.track.latest())); - copy.update(&self.mirrored); - } + copy.replaced = true; } let floor = self.newest.map(Position::group); // The wire request carries the floor from the first message: some sessions read @@ -516,6 +520,7 @@ impl Reader { sub: Sub::Pending(track.subscribe(wire)), track, floor, + replaced: false, until: None, done: None, datagrams_done: None, @@ -528,8 +533,15 @@ impl Reader { fn let_go(&mut self, delivered: &Delivered) { let serving = self.generation; let groups = self.groups; + let groups_read = self.groups_read; self.copies.retain_mut(|copy| { - if copy.needed(Some(copy.generation) == serving) { + // The serving copy, or one between routes with nothing replacing it yet. + if !copy.replaced || copy.needed(Some(copy.generation) == serving) { + return true; + } + // Not yet bounded, it may still feed a group the new copy never has, unless + // nobody reads groups at all. + if copy.until.is_none() && groups_read { return true; } if copy.done.is_some() { @@ -550,6 +562,23 @@ impl Reader { }); } + /// The copy at `index` delivered `sequence`. The serving copy's first group bounds the + /// copies it replaced just below it: what they hold under it may be groups the new route + /// never has, and what is at or above it the new route delivers. Never below a group + /// already handed out. + fn observe(&mut self, index: usize, sequence: u64) { + if Some(self.copies[index].generation) != self.generation { + return; + } + let bound = self.newest.max(sequence.checked_sub(1)); + for copy in &mut self.copies[..index] { + if copy.replaced && copy.until.is_none() { + copy.until = Some(bound); + copy.update(&self.mirrored); + } + } + } + fn deliverable(copy: &Copy, sequence: u64, delivered: &Delivered) -> bool { let replaced = copy .until @@ -595,6 +624,7 @@ impl Reader { waiter: &kio::Waiter, ) -> Option>>> { self.sync(waiter); + self.groups_read = true; let groups = self.groups; // The serving copy first: it is the one that keeps delivering. for index in (0..self.copies.len()).rev() { @@ -619,6 +649,7 @@ impl Reader { match res { Poll::Pending => break, Poll::Ready(Ok(Some(group))) => { + self.observe(index, group.sequence); if !Self::deliverable(&self.copies[index], group.sequence, delivered) { continue; } @@ -707,7 +738,7 @@ impl Reader { for index in (0..self.copies.len()).rev() { loop { let copy = &mut self.copies[index]; - if copy.datagrams_done.is_some() || copy.until.is_some() { + if copy.datagrams_done.is_some() || copy.replaced { break; } let sub = match copy.poll_ready(groups, waiter) { @@ -1376,6 +1407,34 @@ mod test { } } + /// A replaced copy is bounded only once the new copy delivers its first group: a group + /// the old route gets after the swap, which the new route never has, still arrives. + #[test] + fn a_group_the_old_route_gets_after_the_swap_is_delivered() { + for ordered in [false, true] { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::ZERO); + write(&a, 0); + assert_eq!(drain(&mut sub, ordered), [0]); + + let b = copy(); + routes.serve(b.consume()); + // Group 1 reaches the old route after the swap; the new route joins at group 2. + write(&a, 1); + assert_eq!(drain(&mut sub, ordered), [1]); + write(&b, 2); + assert_eq!(drain(&mut sub, ordered), [2]); + // Bounded below the new copy's first group: later groups come from the new route. + write(&a, 3); + assert_eq!(drain(&mut sub, ordered), Vec::::new()); + write(&b, 3); + assert_eq!(drain(&mut sub, ordered), [3]); + } + } + /// A copy delivering a group long after the reader moved past it does not hand it out /// again, however many groups came between; one never handed out is still delivered. #[test] @@ -1613,7 +1672,7 @@ mod test { #[test] #[cfg(feature = "async")] - fn a_replaced_route_finishes_only_the_groups_it_had() { + fn a_replaced_route_yields_to_the_serving_copy() { let routes = Producer::new(); let logical = logical(&routes); let a = copy(); From e10d351b4beb1de53a1533256e5ce2ddd643d89f Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Sat, 17 Oct 2026 08:51:26 -0700 Subject: [PATCH 14/31] fix(js/net): recover an already-failed group, retire superseded info lookups, gate fetches on the read copy A group handed out already failed now recovers: the recover hook is attached before its closed state is read, a latched failure is re-derived once it is, recovery resumes past the frames an adopted copy repeats, and a concurrent read never adopts the same replacement twice. Info lookups and splice checks are abandoned when their route is superseded, track info moves to the new route, and a fetch lets the copy being read say what the track is first. Co-Authored-By: Claude Opus 5.5 --- js/net/src/group.ts | 15 +++++- js/net/src/resume.test.ts | 99 +++++++++++++++++++++++++++++++++++++++ js/net/src/resume.ts | 68 +++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 10 deletions(-) diff --git a/js/net/src/group.ts b/js/net/src/group.ts index cd7c0fcbff6..d826891db4f 100644 --- a/js/net/src/group.ts +++ b/js/net/src/group.ts @@ -477,6 +477,10 @@ export class Consumer { }; hooks.recoverGroup = (group, recover) => { group.#recover = recover; + // A track's drift scan may have read `closed` already, latching a failure recovery can + // now continue: derive it again with the recovery in view. + if (group.#closed?.peek() instanceof Error && group.#verdict.peek() === undefined) + group.#closed = undefined; }; hooks.groupExpired = (group) => group.#ended; hooks.guardGroup = (group, operation) => group.#guard(operation); @@ -579,7 +583,7 @@ export class Consumer { if (!attempt) { const abort = new AbortController(); const started: Attempt = { abort, result: Promise.resolve(), settled: false }; - started.result = recover.replace(this.#state.start, failed, abort.signal).then( + started.result = recover.replace(this.#index(), failed, abort.signal).then( (replacement) => { started.settled = true; // Abandoned while it was out: nobody reads it. @@ -602,7 +606,9 @@ export class Consumer { // The copy being read moved first: look again, keeping the attempt out. if (!attempt.settled) return true; if (this.#attempt === attempt) this.#attempt = undefined; + // Taken once: a concurrent read awaiting the same attempt looks again instead. const replacement = attempt.replacement; + attempt.replacement = undefined; if (!replacement) return true; if (!failed && (this.#state.frames.peek().length > 0 || this.#state.closed.peek() !== undefined)) { // The copy being read delivered or ended meanwhile; keep reading it. @@ -613,6 +619,11 @@ export class Consumer { return true; } + // The next frame this consumer reads: past what it read, and what a copy it adopted repeats. + #index(): number { + return Math.max(this.#state.start, this.#skip); + } + // Give up a replacement still being asked for, or one nobody adopted: the group ended without it. #abandon(): void { const attempt = this.#attempt; @@ -625,7 +636,7 @@ export class Consumer { // Read the rest of the group from `replacement`, from the frame this one reached. #adopt(replacement: Replacement): void { const next = replacement.group.#state; - const index = this.#state.start; + const index = this.#index(); if (next.start > index) { replacement.group.close(); this.#terminal ??= new Error(`group ${this.sequence} cannot resume at frame ${index}`); diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 39b836bd9fc..96d12842ea0 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -663,3 +663,102 @@ test("a group the old route gets after the swap, which the new route never has, a.close(); b.close(); }); + +test("a group handed out already failed with its route still continues on the next one", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + // The old route fails group 0 before the reader takes it. + const failed = ta.write(0, 1, false); + failed.close(new StreamError(StreamCode.SessionClosed)); + const group = sub.tryRecvGroup(); + if (!group) throw new Error("expected group 0"); + expect(dec.decode((await group.readFrame())?.payload)).toBe("0.0"); + const read = group.readFrame(); + + tb.write(0, 2); + front.serve(b.consume()); + expect(dec.decode((await read)?.payload)).toBe("0.1"); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); +}); + +test("track info asks the next route when the current one never answers", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + // a serves tracks on demand but never answers. + void wireOf(a).requested(); + b.createTrack("seq", { priority: 2 }); + front.serve(a.consume()); + const info = front.consumer.track("seq").info(); + await settle(); + front.serve(b.consume()); + expect((await info).priority).toBe(2); + + front.consumer.close(); + a.close(); + b.close(); +}); + +test("a route superseded while its properties are checked stops being asked", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const c = new BroadcastProducer(); + copy(a); + copy(c); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + await sub.info(); + // b never answers what the track is, so its splice waits on it. + void wireOf(b).requested(); + front.serve(b.consume()); + await settle(); + expect(b.demand().used.peek()).toBe(true); + + front.serve(c.consume()); + await settle(); + expect(b.demand().used.peek()).toBe(false); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); + c.close(); +}); + +test("a fetch waits for the copy being read to say what the track is", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + // a answers what the track is only when told to. + const requests = wireOf(a).requested(); + const tb = b.createTrack("seq", { priority: 2 }); + const group = new GroupProducer(0); + tb.writeGroup(group); + group.writeFrame({ payload: enc.encode("b") }); + group.close(); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + front.serve(b.consume()); + const fetching = front.consumer.track("seq").fetchGroup(0); + fetching.catch(() => {}); + await settle(); + + const request = await requests; + request?.accept({ priority: 1 }); + await expect(fetching).rejects.toThrow("changed its properties"); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 401ff731fd6..7bdf65bf66b 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -81,6 +81,22 @@ function compatible(a: track.Info, b: track.Info): boolean { return a.timescale === b.timescale && a.maxAge === b.maxAge && a.priority === b.priority; } +/** + * Ask `source` what track `name` is, abandoning the question once `retired` aborts. A caller's + * `hold` keeps the track wanted until it or `retired` aborts; without one, only until the answer. + */ +function ask(source: Source, name: string, retired: AbortSignal, hold?: AbortSignal): Promise { + if (hold) return source.wire.resolveTrackInfo(name, AbortSignal.any([hold, retired])); + const query = new AbortController(); + const onRetire = () => query.abort(retired.reason); + if (retired.aborted) onRetire(); + else retired.addEventListener("abort", onRetire, { once: true }); + return source.wire.resolveTrackInfo(name, query.signal).finally(() => { + retired.removeEventListener("abort", onRetire); + query.abort(); + }); +} + /** One route's broadcast, held while the front serves it or a reader still reads a copy of it. */ class Source { readonly wire: BroadcastWire; @@ -514,7 +530,9 @@ class Reader implements ResumeSource { #handOut(copy: Copy, group: GroupConsumer): GroupConsumer { this.#newest = Math.max(this.#newest ?? 0, group.sequence); - hooks.recoverGroup(group, new Recover(this, group, copy)); + const recover = new Recover(this, group, copy); + hooks.recoverGroup(group, recover); + recover.watch(group); return group; } @@ -639,6 +657,13 @@ function startFetch( wire.fetchGroup(name, sequence, { signal: controller.signal }) .then( (group) => { + // A copy that already failed continues nothing: that is the route failing again. + const failed = group.closed.peek(); + if (failed instanceof Error) { + group.close(); + fetch.outcome = { error: failed }; + return; + } fetch.outcome = { group }; // Abandoned while it was out: nobody reads it. if (controller.signal.aborted) group.close(); @@ -690,6 +715,13 @@ class Recover implements GroupRecover { this.#sequence = group.sequence; this.#copy = copy; copy.lease++; + } + + /** + * Release the copy once the group ends. Called only after the group carries this recovery: + * reading `closed` before then would settle it on a failure recovery could still continue. + */ + watch(group: GroupConsumer): void { void group.closed.then(() => this.#release()); } @@ -808,6 +840,8 @@ export class Front { #info = new Map(); /** Bumped whenever the source changes, so an answer from a replaced one is dropped. */ #changed = new Signal(0); + /** Aborted once the current source is replaced, retiring what was asked of it. */ + #retire = new AbortController(); #concluded = false; /** The tracks each source was checked against the recorded properties for; see #verify. */ #verified = new WeakMap>(); @@ -827,7 +861,7 @@ export class Front { this.#fromSession ||= source.fromSession; const previous = this.#source; this.#source = source; - this.#changed.update((n) => n + 1); + this.#retired(); for (const route of this.#routes.values()) this.#splice(route, source); previous?.release(); } @@ -837,7 +871,7 @@ export class Front { if (this.#concluded || !this.#source) return; const previous = this.#source; this.#source = undefined; - this.#changed.update((n) => n + 1); + this.#retired(); for (const route of this.#routes.values()) route.park(); previous.release(); } @@ -849,7 +883,14 @@ export class Front { for (const route of this.#routes.values()) route.conclude(); this.#source?.release(); this.#source = undefined; + this.#retired(); + } + + // The source changed: wake what waits on it, and retire what was asked of the old one. + #retired(): void { this.#changed.update((n) => n + 1); + this.#retire.abort(); + this.#retire = new AbortController(); } /** @@ -874,16 +915,18 @@ export class Front { // answered yet is asked too, so a swap racing its first answer is still checked. source.retain(); previous?.retain(); + // Abandoned once another route supersedes this one, releasing both. + const retired = this.#retire.signal; const expected = recorded ? Promise.resolve(recorded.info) - : (previous as Source).wire.resolveTrackInfo(route.name).then( + : ask(previous as Source, route.name, retired).then( (info) => { this.#record(route.name, info, previous as Source); return info; }, () => undefined, ); - Promise.all([expected, source.wire.resolveTrackInfo(route.name)]) + Promise.all([expected, ask(source, route.name, retired)]) .then( ([expected, info]) => { if (this.#source !== source) return; @@ -929,8 +972,14 @@ export class Front { if (recorded?.source === source) return; let verified = this.#verified.get(source); if (verified?.has(name)) return; - // A one-shot query: a hold would keep the track wanted for as long as the fetch's signal. - const info = await source.wire.resolveTrackInfo(name); + const retired = this.#retire.signal; + // A copy a reader still reads says what the track is before any fetch does. + const served = this.#routes.get(name)?.served; + if (!recorded && served && served !== source) { + const info = await ask(served, name, retired).catch(() => undefined); + if (info) this.#record(name, info, served); + } + const info = await ask(source, name, retired); // The first answer says what the track is; later sources have to match it. const expected = this.#info.get(name); if (!expected) this.#record(name, info, source); @@ -968,10 +1017,13 @@ export class Front { continue; } source.retain(); + const retired = this.#retire.signal; let info: track.Info; try { - info = await source.wire.resolveTrackInfo(name, hold); + info = await ask(source, name, retired, hold); } catch (err) { + // Another route took over before this one answered: ask it instead. + if (retired.aborted && !hold?.aborted && !this.#concluded) continue; const error = err instanceof Error ? err : new Error(String(err)); // The route failed: ask the next one, unless none will come. if (!routeFailed(error) || this.#concluded || hold?.aborted) throw error; From 8dcd935a36b8a2ed3bf759b5c81cd2c8ee82f9d7 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Sat, 17 Oct 2026 09:44:49 -0700 Subject: [PATCH 15/31] fix(js/net): arrival reads take the lowest buffered group across copies, fetches follow the route, stalls learn of failures Co-Authored-By: Claude Opus 5.5 --- js/net/src/group.test.ts | 21 ++++++++++++++++++ js/net/src/group.ts | 6 +++++- js/net/src/internal.ts | 6 +++--- js/net/src/resume.test.ts | 44 ++++++++++++++++++++++++++++++++++++++ js/net/src/resume.ts | 44 ++++++++++++++++++++++---------------- js/net/src/track.ts | 11 +++++----- quest/m1/README.md | 2 +- quest/m1/resume-ordered.md | 36 ++++++++++++++++++------------- 8 files changed, 127 insertions(+), 43 deletions(-) diff --git a/js/net/src/group.test.ts b/js/net/src/group.test.ts index d86c49a6d64..13393544194 100644 --- a/js/net/src/group.test.ts +++ b/js/net/src/group.test.ts @@ -322,3 +322,24 @@ test("a group recovered twice reads on and settles", async () => { expect(await consumer.readFrame()).toBeUndefined(); expect(await consumer.closed).toBeNull(); }); + +test("a stalled group whose copy then fails asks again with the failure", async () => { + const producer = new Producer(0); + const consumer = producer.mirror(); + const route = new Signal(0); + const verdict = new Error("refused"); + hooks.recoverGroup(consumer, { + changed: route, + wants: (failed) => failed !== undefined || route.peek() > 0, + // A stall waits for the route; a failure gets the route's verdict. + replace: (_index, failed) => (failed ? Promise.reject(verdict) : new Promise(() => {})), + }); + producer.writeFrame({ payload: new Uint8Array([0]) }); + await consumer.readFrame(); + + route.set(1); + const read = consumer.readFrame(); + await new Promise((resolve) => setTimeout(resolve, 0)); + producer.close(new Error("route failed")); + await expect(read).rejects.toBe(verdict); +}); diff --git a/js/net/src/group.ts b/js/net/src/group.ts index d826891db4f..8b66f00faf6 100644 --- a/js/net/src/group.ts +++ b/js/net/src/group.ts @@ -316,6 +316,8 @@ let makeConsumer: (state: GroupState) => Consumer; // A replacement copy asked for on behalf of one consumer, settled or still out. interface Attempt { + /** Asked for a copy that failed, rather than one that stalled. */ + failed: boolean; abort: AbortController; result: Promise; settled: boolean; @@ -579,10 +581,12 @@ export class Consumer { async #resume(failed: Error | undefined): Promise { const recover = this.#recover; if (!recover?.wants(failed)) return false; + // A stall's attempt learns nothing of a failure since: ask again with it. + if (failed && this.#attempt && !this.#attempt.failed) this.#abandon(); let attempt = this.#attempt; if (!attempt) { const abort = new AbortController(); - const started: Attempt = { abort, result: Promise.resolve(), settled: false }; + const started: Attempt = { failed: failed !== undefined, abort, result: Promise.resolve(), settled: false }; started.result = recover.replace(this.#index(), failed, abort.signal).then( (replacement) => { started.settled = true; diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index 988b7682aac..38193a21b50 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -335,8 +335,8 @@ export const hooks: { * the cursor gives up is reported once as a `drop`. */ tryRecvGroup: (subscriber: Subscriber, dispositions: boolean) => Recv; - /** The sequence {@link tryNextGroup} would take next, without taking it. */ - peekNextGroup: (subscriber: Subscriber) => number | undefined; + /** The sequence {@link tryNextGroup} (`ordered`) or {@link tryRecvGroup} would take next, without taking it. */ + peekGroup: (subscriber: Subscriber, ordered: boolean) => number | undefined; /** Take the next group in sequence order without waiting, as {@link tryRecvGroup} does in arrival order. */ tryNextGroup: (subscriber: Subscriber) => Recv; /** Wake once a subscriber's datagram cursor may read differently; fires at most once. */ @@ -432,7 +432,7 @@ export const hooks: { tryNextGroup: () => { throw new Error("track.ts not loaded"); }, - peekNextGroup: () => { + peekGroup: () => { throw new Error("track.ts not loaded"); }, forgetPastBudget: () => { diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 96d12842ea0..1e3138198b4 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -762,3 +762,47 @@ test("a fetch waits for the copy being read to say what the track is", async () a.close(); b.close(); }); + +test("an arrival read takes a replaced copy's buffered group before the new copy's later one", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe({ maxDelay: 10_000 as track.Subscription["maxDelay"] }); + ta.write(0); + expect(drain(sub)).toEqual([0]); + // Both buffered when the read starts: the lower one comes first. + ta.write(1); + tb.write(2); + front.serve(b.consume()); + await settle(); + expect(drain(sub)).toEqual([1, 2]); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); +}); + +test("a fetch still out on the old route moves to the new one", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + // a has the track but not group 5 yet, so its fetch waits. + a.createTrack("seq"); + const tb = copy(b); + tb.write(5); + front.serve(a.consume()); + const fetching = front.consumer.track("seq").fetchGroup(5); + await settle(); + front.serve(b.consume()); + const group = await fetching; + expect(dec.decode((await group.readFrame())?.payload)).toBe("5.0"); + + group.close(); + front.consumer.close(); + a.close(); + b.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 7bdf65bf66b..c8f62e3f2c9 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -227,6 +227,11 @@ function subscribe(wire: BroadcastWire, name: string, subscription: track.Subscr } } +/** Take a copy's next group, in sequence or arrival order. */ +function take(copy: Copy, ordered: boolean): Recv { + return ordered ? hooks.tryNextGroup(copy.sub) : hooks.tryRecvGroup(copy.sub, false); +} + /** The first group past where `copy` was replaced: 0 for one replaced before any group. */ function copyEnd(copy: Pick): number | undefined { if (copy.until === undefined) return undefined; @@ -373,16 +378,15 @@ class Reader implements ResumeSource { tryGroup(ordered: boolean, delivered: Delivered): Recv { this.sync(); - // In sequence order, the lowest group any copy holds next, so a replaced copy's buffered - // group is not skipped for a later one the serving copy already has. - if (ordered) { - const next = this.#lowest(delivered); - if (next) { - const recv = hooks.tryNextGroup(next.sub); - if (recv.kind === "group") { - this.#observe(next, recv.group.sequence); - return { kind: "group", group: this.#handOut(next, recv.group) }; - } + // The lowest group any copy holds next, in either order: a replaced copy's buffered group + // comes before a later one the serving copy already has. Arrival order still delivers a + // lower group that truly arrives later. + const next = this.#lowest(ordered, delivered); + if (next) { + const recv = take(next, ordered); + if (recv.kind === "group") { + this.#observe(next, recv.group.sequence); + return { kind: "group", group: this.#handOut(next, recv.group) }; } } // The serving copy first: it is the one that keeps delivering. @@ -390,7 +394,7 @@ class Reader implements ResumeSource { const copy = this.#copies[index]; if (copy.done !== undefined) continue; for (;;) { - const recv = ordered ? hooks.tryNextGroup(copy.sub) : hooks.tryRecvGroup(copy.sub, false); + const recv = take(copy, ordered); if (recv.kind === "group") { this.#observe(copy, recv.group.sequence); if (!this.#deliverable(copy, recv.group.sequence, delivered)) { @@ -410,20 +414,20 @@ class Reader implements ResumeSource { return this.#end("groups", (copy) => copy.done); } - // The copy whose next group in sequence order is the lowest deliverable one, dropping any it - // holds next that are not. - #lowest(delivered: Delivered): Copy | undefined { + // The copy whose next group is the lowest deliverable one, dropping any it holds next that + // are not. + #lowest(ordered: boolean, delivered: Delivered): Copy | undefined { let lowest: { copy: Copy; sequence: number } | undefined; for (const copy of this.#copies) { if (copy.done !== undefined) continue; for (;;) { - const sequence = hooks.peekNextGroup(copy.sub); + const sequence = hooks.peekGroup(copy.sub, ordered); if (sequence === undefined) break; if (this.#deliverable(copy, sequence, delivered)) { if (!lowest || sequence < lowest.sequence) lowest = { copy, sequence }; break; } - const recv = hooks.tryNextGroup(copy.sub); + const recv = take(copy, ordered); if (recv.kind !== "group") break; this.#observe(copy, recv.group.sequence); recv.group.close(); @@ -1096,11 +1100,15 @@ export class Front { continue; } source.retain(); + // Abandoned once another route takes over, and asked of that one instead. + const retired = this.#retire.signal; + const signal = options?.signal ? AbortSignal.any([options.signal, retired]) : retired; try { - await until(this.#verify(name, source), options?.signal); - return { group: await source.wire.fetchGroup(name, sequence, options), source }; + await until(this.#verify(name, source), signal); + return { group: await source.wire.fetchGroup(name, sequence, { ...options, signal }), source }; } catch (err) { source.release(); + if (retired.aborted && !options?.signal?.aborted && !this.#concluded) continue; const error = err instanceof Error ? err : new Error(String(err)); // The route failed: ask the next one, unless none will come. if (!routeFailed(error) || this.#concluded || options?.signal?.aborted) throw error; diff --git a/js/net/src/track.ts b/js/net/src/track.ts index d2c94ff4c98..161af83718e 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -1438,7 +1438,7 @@ export class Subscriber { makeSubscriber = (name, state) => new Subscriber(name, state); hooks.tryRecvGroup = (subscriber, started) => subscriber.#tryRecvGroup(started); hooks.tryNextGroup = (subscriber) => subscriber.#tryNextGroup(); - hooks.peekNextGroup = (subscriber) => subscriber.#peekNextGroup(); + hooks.peekGroup = (subscriber, ordered) => subscriber.#peekGroup(ordered); hooks.forgetPastBudget = (subscriber, delivered) => subscriber.#forgetPastBudget(delivered); hooks.tryRecvDatagram = (subscriber) => subscriber.#tryDatagram(); hooks.deliveredRuns = (subscriber) => subscriber.#delivered.runs; @@ -1930,15 +1930,16 @@ export class Subscriber { } } - // The sequence #tryNextGroup would take next: the first buffered group in range the budget spares. - #peekNextGroup(): number | undefined { + // The sequence #tryNextGroup (`ordered`) or #tryRecvGroup would take next: the lowest buffered + // group in range, not yet handed out, that the budget spares. + #peekGroup(ordered: boolean): number | undefined { const cursor = this.#cursor.peek(); - const start = Math.max(cursor.start, this.#nextSequence); + const start = ordered ? Math.max(cursor.start, this.#nextSequence) : cursor.start; const drift = this.#drift(); for (const group of this.#state.groups.peek()) { if (group.sequence < start) continue; if (cursor.end !== undefined && group.sequence >= cursor.end) return undefined; - if (!this.#isStale(group, drift)) return group.sequence; + if (!this.#delivered.has(group.sequence) && !this.#isStale(group, drift)) return group.sequence; } return undefined; } diff --git a/quest/m1/README.md b/quest/m1/README.md index 3aec919e80a..cdf2eb7afe8 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -106,7 +106,7 @@ lands before the cut. - [TS fidelity and recovery](/quest/m1/ts-reliability/README.md) - both modes have explicit profiles, automated validation, loss tests, and a dedicated guide - [Draft 14-16 updates](/quest/m1/ietf-legacy-updates.md) - Rust and JS apply and answer moq-transport 14-16 request updates without ending or leaking the request - [Resume reorder](/quest/m1/resume-reorder.md) - a group whose stream header arrives after its successor's still resumes on failover; only a dropped or long-gone group fails -- [Rust resume ordered reads](/quest/m1/resume-ordered.md) - a Rust ordered reader takes the lowest group across route copies, as JS does +- [Rust resumed reads](/quest/m1/resume-ordered.md) - a Rust reader takes the lowest group across route copies in both orders, as JS does - [Refuse rewinds](/quest/m1/refuse-rewinds.md) - a timed track refuses a group whose first frame is earlier than the previous group's, in Rust and js/net, on every hop - [JS pending tail](/quest/m1/js-pending-tail.md) - a JS reader holds for a track's pending tail like Rust - [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS diff --git a/quest/m1/resume-ordered.md b/quest/m1/resume-ordered.md index 0bd62d25621..7270a4feb09 100644 --- a/quest/m1/resume-ordered.md +++ b/quest/m1/resume-ordered.md @@ -1,16 +1,20 @@ -# [S] Rust ordered reads take the lowest group across route copies +# [S] Rust resumed reads take the lowest group across route copies ## Goal -A Rust reader reading a logical track in sequence order (`poll_next_group`) -across a same-epoch route swap never skips a group a replaced route's copy -already holds just because the serving copy offered a later one first. This -matches JS since #5836. +A Rust reader of a logical track across a same-epoch route swap takes the +lowest buffered group across every route copy, in both orders. In sequence +order (`poll_next_group`) it never skips a group a replaced copy already holds +because the serving copy offered a later one first. In arrival order +(`poll_recv_group`) it hands out the lowest buffered group first, as a direct +reader does. This matches JS since #5836. ## Plan Decided 2026-10-17 by the maintainer on #5836: JS keeps reading the lowest next -group across copies, and Rust follows rather than JS reverting. +group across copies, and Rust follows rather than JS reverting. A later review +round on #5836 extended the JS rule to arrival reads, so this covers both +orders. Today `Reader::poll_group_once` in `rs/moq-net/src/model/resume.rs` polls the serving copy first in both orders. In sequence order, `track::Subscriber::hand_out` @@ -18,17 +22,19 @@ then forgets every sequence below the group it returns. So a replaced copy's buffered N+1 is dropped when the serving copy offers N+2 first, a group a direct ordered reader would have delivered. -- Mirror JS `Reader.#lowest`: in sequence order, peek each live copy's next - deliverable group without moving its cursor, drop what is not deliverable, - and take from the copy with the lowest one. Arrival order keeps serving-first. -- The peek needs a non-consuming counterpart of `poll_next_group` on - `track::Subscriber` that applies the same floor, cap, and drift budget. +- Mirror JS `Reader.#lowest`: peek each live copy's next deliverable group + without moving its cursor, drop what is not deliverable, and take from the + copy with the lowest one, in either order. Arrival order still delivers a + lower group that truly arrives later. +- The peek needs a non-consuming counterpart of `poll_next_group` and + `poll_recv_group` on `track::Subscriber` that applies the same floor, cap, + and drift budget (JS `hooks.peekGroup`). - Feed the taken group through `Reader::observe` as the other paths do, so the deferred bound on replaced copies still applies. -- Regression test: an ordered reader across a swap where the old copy holds - group 1 unread and the new copy only has group 2 reads 1 then 2, as JS - `an ordered reader takes a replaced copy's buffered group before the new - copy's later one` does. +- Regression test, in both orders: a reader across a swap where the old copy + holds group 1 unread and the new copy only has group 2 reads 1 then 2, as the + JS `... takes a replaced copy's buffered group before the new copy's later + one` tests do. Public API: behavior only. Wire: none. From c16124b904e46cf179e08475fc947019d5aeaa75 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 17 Oct 2026 20:46:10 -0700 Subject: [PATCH 16/31] docs(net): reflow the resume module doc Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/model/resume.rs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index ba0c63ec9fc..fdce1dae6a0 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -10,10 +10,10 @@ //! it handed out. A replaced copy keeps delivering until the new copy delivers its first //! group, then is bounded just below it (or at the newest group handed out, if later), so a //! group only the old route got after the switch is not lost. It is kept for unread groups it -//! had and groups still being read, and dropped once neither needs it. A group carries on across the switch through -//! its [`Recover`]: once its copy fails with its route, or stalls while a newer route -//! serves, it continues from the serving route's copy of the same group at the same frame -//! (and byte), found in that copy's cache or fetched from it. +//! had and groups still being read, and dropped once neither needs it. A group carries on +//! across the switch through its [`Recover`]: once its copy fails with its route, or stalls +//! while a newer route serves, it continues from the serving route's copy of the same group +//! at the same frame (and byte), found in that copy's cache or fetched from it. //! //! A group is handed out once however many copies deliver it: the reader skips any //! sequence its [`track::Subscriber`] already handed out, the same record that keeps a From 0c00bffb6ba8241d17d076a90519a2af78bd8d89 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 17 Oct 2026 21:51:11 -0700 Subject: [PATCH 17/31] fix(net): a replayed group never bounds a replaced copy; a failed fetch waits for the next route A replaced copy was bounded by the new copy's first group even when that group was a replay already handed out, hiding a group only the old copy held. Bound only on a group the reader hands out, in Rust and JS. A replaced copy whose group channel ran out is let go at once. A JS fetch returned an already-failed group, so recovery of a fetched group on the still-serving route re-adopted the same failed copy and spun. It now treats that as the route failing and waits for the next. Found by a local Codex adversarial review. Co-Authored-By: Claude Opus 5.5 --- js/net/src/resume.test.ts | 51 ++++++++++++++++++++++++++++++++++ js/net/src/resume.ts | 33 ++++++++++++---------- rs/moq-net/src/model/resume.rs | 39 +++++++++++++++++++++----- 3 files changed, 102 insertions(+), 21 deletions(-) diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 1e3138198b4..1370c0d66a2 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -664,6 +664,57 @@ test("a group the old route gets after the swap, which the new route never has, b.close(); }); +test("a group the new route replays bounds nothing, so the old route's next group still arrives", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + ta.write(0); + expect(drain(sub)).toEqual([0]); + ta.write(1); + front.serve(b.consume()); + await settle(); + // The new route only replays group 0, which was already handed out. + tb.write(0); + expect(drain(sub)).toEqual([1]); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); +}); + +test("a fetched group that fails on the route still serving waits for the next route", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + const failing = ta.write(5, 1, false); + front.serve(a.consume()); + const group = await front.consumer.track("seq").fetchGroup(5); + expect(dec.decode((await group.readFrame())?.payload)).toBe("5.0"); + + // Its route stays current, so its only copy of group 5 is the failed one. + failing.close(new StreamError(StreamCode.SessionClosed)); + const read = group.readFrame(); + // Timers still run: recovery does not spin on the failed copy. + await settle(); + await settle(); + + tb.write(5, 2); + front.serve(b.consume()); + expect(dec.decode((await read)?.payload)).toBe("5.1"); + expect(await group.readFrame()).toBeUndefined(); + + front.consumer.close(); + a.close(); + b.close(); +}); + test("a group handed out already failed with its route still continues on the next one", async () => { const front = new Front(); const a = new BroadcastProducer(); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index c8f62e3f2c9..eafb6b711e0 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -356,9 +356,10 @@ class Reader implements ResumeSource { } /** - * `copy` delivered `sequence`. The serving copy's first group bounds the copies it replaced + * `copy` handed out `sequence`. The serving copy's first group bounds the copies it replaced * just below it: what they hold under it may be groups the new route never has, and what - * is at or above it the new route delivers. Never below a group already handed out. + * is at or above it the new route delivers. Never below a group already handed out. A group + * it replays, already handed out, bounds nothing: the replaced copies may still hold the next. */ #observe(copy: Copy, sequence: number): void { if (copy.generation !== this.#generation) return; @@ -396,11 +397,11 @@ class Reader implements ResumeSource { for (;;) { const recv = take(copy, ordered); if (recv.kind === "group") { - this.#observe(copy, recv.group.sequence); if (!this.#deliverable(copy, recv.group.sequence, delivered)) { recv.group.close(); continue; } + this.#observe(copy, recv.group.sequence); return { kind: "group", group: this.#handOut(copy, recv.group) }; } // Drained: the track ended as the copy did, which is not cleanly when its @@ -429,7 +430,6 @@ class Reader implements ResumeSource { } const recv = take(copy, ordered); if (recv.kind !== "group") break; - this.#observe(copy, recv.group.sequence); recv.group.close(); } } @@ -512,16 +512,14 @@ class Reader implements ResumeSource { this.#copies = this.#copies.filter((copy) => { // The serving copy, or one between routes with nothing replacing it yet. if (!copy.replaced || copy.lease > 0) return true; - // Not yet bounded, it may still feed a group the new copy never has, unless nobody - // reads groups at all. - if (copy.until === undefined && delivered) return true; - if ( - copy.done === undefined && - delivered && - hooks.hasUnreadGroup(copy.sub, cursor.start, copyEnd(copy), delivered) - ) { - return true; + // Its group channel ran out, or nobody reads groups at all: it feeds nothing more. + if (copy.done !== undefined || !delivered) { + this.#drop(copy); + return false; } + // Not yet bounded, it may still feed a group the new copy never has. + if (copy.until === undefined) return true; + if (hooks.hasUnreadGroup(copy.sub, cursor.start, copyEnd(copy), delivered)) return true; this.#drop(copy); return false; }); @@ -1105,7 +1103,14 @@ export class Front { const signal = options?.signal ? AbortSignal.any([options.signal, retired]) : retired; try { await until(this.#verify(name, source), signal); - return { group: await source.wire.fetchGroup(name, sequence, { ...options, signal }), source }; + const group = await source.wire.fetchGroup(name, sequence, { ...options, signal }); + // A copy that already failed continues nothing: that is the route failing again. + const failed = group.closed.peek(); + if (failed instanceof Error) { + group.close(); + throw failed; + } + return { group, source }; } catch (err) { source.release(); if (retired.aborted && !options?.signal?.aborted && !this.#concluded) continue; diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index fdce1dae6a0..0ecb6c3c86e 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -530,14 +530,15 @@ impl Reader { if !copy.replaced || copy.needed(Some(copy.generation) == serving) { return true; } + // Its group channel ran out: it feeds nothing more. + if copy.done.is_some() { + return false; + } // Not yet bounded, it may still feed a group the new copy never has, unless // nobody reads groups at all. if copy.until.is_none() && groups_read { return true; } - if copy.done.is_some() { - return false; - } let end = copy.end(); let Poll::Ready(Ok(sub)) = copy.poll_ready(groups, &kio::Waiter::noop()) else { return false; @@ -553,10 +554,11 @@ impl Reader { }); } - /// The copy at `index` delivered `sequence`. The serving copy's first group bounds the + /// The copy at `index` hands out `sequence`. The serving copy's first group bounds the /// copies it replaced just below it: what they hold under it may be groups the new route /// never has, and what is at or above it the new route delivers. Never below a group - /// already handed out. + /// already handed out. A group it replays, already handed out, bounds nothing: the + /// replaced copies may still hold the next. fn observe(&mut self, index: usize, sequence: u64) { if Some(self.copies[index].generation) != self.generation { return; @@ -640,10 +642,10 @@ impl Reader { match res { Poll::Pending => break, Poll::Ready(Ok(Some(group))) => { - self.observe(index, group.sequence); if !Self::deliverable(&self.copies[index], group.sequence, delivered) { continue; } + self.observe(index, group.sequence); let group = self.hand_out(index, group); return Some(Poll::Ready(Ok(Some(group)))); } @@ -1426,6 +1428,27 @@ mod test { } } + /// A group the new copy replays, already handed out, does not bound the replaced copy: + /// the old route's next group still arrives. + #[test] + fn a_group_the_new_route_replays_bounds_nothing() { + for ordered in [false, true] { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::ZERO); + write(&a, 0); + assert_eq!(drain(&mut sub, ordered), [0]); + write(&a, 1); + + let b = copy(); + routes.serve(b.consume()); + write(&b, 0); + assert_eq!(drain(&mut sub, ordered), [1]); + } + } + /// A copy delivering a group long after the reader moved past it does not hand it out /// again, however many groups came between; one never handed out is still delivered. #[test] @@ -2726,7 +2749,9 @@ mod test { let waiter = kio::Waiter::new(wake.clone().into()); assert!(reading.poll(&waiter).is_pending()); if recovered { - // Retire the original copy so its preferences stop being mirrored. + // Retire the original copy so its preferences stop being mirrored: its + // route goes away, since a replayed group alone never bounds it. + a.clone().abort(Error::Dropped).unwrap(); assert!(sub.recv_group().now_or_never().is_none()); assert!(a.subscription().is_none()); } From 89a3cd1fad421236042e2c12c0eba85736d4aafb Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 17 Oct 2026 22:01:04 -0700 Subject: [PATCH 18/31] fix(net): a group every JS route holds comes from the serving copy Copies tied on their next group handed it out from the oldest, so the serving copy's duplicate was discarded without bounding the copies it replaced, and every replaced subscription stayed held. Ties now go to the newer copy, as Rust's serving-first poll already does. Found by a second local Codex adversarial review. Co-Authored-By: Claude Opus 5.5 --- js/net/src/resume.test.ts | 36 ++++++++++++++++++++++++++++++++++++ js/net/src/resume.ts | 4 +++- 2 files changed, 39 insertions(+), 1 deletion(-) diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 1370c0d66a2..97974d2036a 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -687,6 +687,42 @@ test("a group the new route replays bounds nothing, so the old route's next grou b.close(); }); +test.each([true, false])( + "a group every route holds comes from the serving one, so replaced copies go (ordered: %s)", + async (ordered) => { + const front = new Front(); + const sources = Array.from({ length: 4 }, () => new BroadcastProducer()); + const copies = sources.map(copy); + front.serve(sources[0].consume()); + const sub = front.consumer.track("seq").subscribe(); + const view = ordered ? sub.ordered() : undefined; + const next = async () => { + const group = await (view ? view.nextGroup() : sub.recvGroup()); + group?.close(); + return group?.sequence; + }; + copies[0].write(0); + expect(await next()).toBe(0); + let pending = next(); + for (let route = 1; route < sources.length; route++) { + front.serve(sources[route].consume()); + await settle(); + // Every route so far holds the next group. + for (const held of copies.slice(0, route + 1)) held.write(route); + expect(await pending).toBe(route); + // The serving copy's group bounded the previous route's copy, so the next read lets it go. + pending = next(); + await settle(); + expect(copies[route - 1].producer.demand().used.peek()).toBe(false); + } + + view?.close(); + sub.close(); + front.consumer.close(); + for (const source of sources) source.close(); + }, +); + test("a fetched group that fails on the route still serving waits for the next route", async () => { const front = new Front(); const a = new BroadcastProducer(); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index eafb6b711e0..c34cca7fa66 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -425,7 +425,9 @@ class Reader implements ResumeSource { const sequence = hooks.peekGroup(copy.sub, ordered); if (sequence === undefined) break; if (this.#deliverable(copy, sequence, delivered)) { - if (!lowest || sequence < lowest.sequence) lowest = { copy, sequence }; + // A tie goes to the newer copy, so the serving one hands out its first group and + // bounds the copies it replaced. + if (!lowest || sequence <= lowest.sequence) lowest = { copy, sequence }; break; } const recv = take(copy, ordered); From ef562a0c9821f08908615721967360804cd54b57 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 18 Oct 2026 22:40:55 -0700 Subject: [PATCH 19/31] fix(net): a failed group continues from any copy still holding it A group whose serving copy failed (for example SessionClosed) while a replaced copy holds it whole now continues from that copy, in Rust and JS, instead of waiting on a fetch of the same failed group. Handing a group out leases the other copies that hold it, so reading on does not let them go while it is read; JS wakes parked reads once such a lease ends. Also from the hosted Codex review: JS track info and fetches ask the copy being read first, and its refusal is the track's answer for readers and fetches alike; a Rust reader that read no groups bounds replaced copies at what they have. Co-Authored-By: Claude Opus 5.5 --- js/net/src/internal.ts | 5 ++ js/net/src/resume.test.ts | 101 ++++++++++++++++++++++- js/net/src/resume.ts | 87 ++++++++++++++++---- js/net/src/track.ts | 10 +++ rs/moq-net/src/model/resume.rs | 145 ++++++++++++++++++++++++++++++--- 5 files changed, 321 insertions(+), 27 deletions(-) diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index e9a186aadf7..1d1ce6f402c 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -392,6 +392,8 @@ export const hooks: { staleAt: (subscriber: Subscriber, sequence: number) => Expiry; /** A group the subscriber's track still caches, as a fresh read handle. */ cachedGroup: (subscriber: Subscriber, sequence: number) => GroupConsumer | undefined; + /** Whether {@link cachedGroup} would find the group, without making a read handle. */ + holdsGroup: (subscriber: Subscriber, sequence: number) => boolean; /** A subscriber's largest location now, and whether it can still advance. */ peekLargest: (subscriber: Subscriber) => { largest?: Location; ended: boolean }; /** Wake once a subscriber's largest location or its end may have changed; fires at most once. */ @@ -529,6 +531,9 @@ export const hooks: { cachedGroup: () => { throw new Error("track.ts not loaded"); }, + holdsGroup: () => { + throw new Error("track.ts not loaded"); + }, peekLargest: () => { throw new Error("track.ts not loaded"); }, diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 5ba73e6ef8d..bcd8700a395 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -1,7 +1,7 @@ import { expect, test } from "bun:test"; import { Producer as BroadcastProducer } from "./broadcast.ts"; import * as Epoch from "./epoch.ts"; -import { StreamCode, StreamError } from "./error.ts"; +import { NotFound, StreamCode, StreamError } from "./error.ts"; import { Producer as GroupProducer } from "./group.ts"; import { Route } from "./hop.ts"; import { Producer } from "./origin.ts"; @@ -723,6 +723,54 @@ test.each([true, false])( }, ); +test.each([ + [true, true], + [false, true], + [true, false], + [false, false], +])( + "a group the serving copy failed continues from a replaced copy that holds it (ordered: %p, failed before handout: %p)", + async (ordered, early) => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + const view = ordered ? sub.ordered() : undefined; + const next = () => (view ? view.nextGroup() : sub.recvGroup()); + ta.write(0); + (await next())?.close(); + front.serve(b.consume()); + await settle(); + + // Both routes hold group 1: the old one whole, the serving one failing after a frame. + ta.write(1, 2); + const failing = tb.write(1, 1, false); + const fail = () => failing.close(new StreamError(StreamCode.SessionClosed)); + if (early) fail(); + const group = await next(); + if (group?.sequence !== 1) throw new Error("expected group 1"); + // Reading on must not let go of the replaced copy while the group still needs it. + const later = next(); + await settle(); + if (!early) fail(); + expect(dec.decode((await group.readFrame())?.payload)).toBe("1.0"); + expect(dec.decode((await group.readFrame())?.payload)).toBe("1.1"); + expect(await group.readFrame()).toBeUndefined(); + + group.close(); + tb.write(2); + expect((await later)?.sequence).toBe(2); + view?.close(); + sub.close(); + front.consumer.close(); + a.close(); + b.close(); + }, +); + test("a fetched group that fails on the route still serving waits for the next route", async () => { const front = new Front(); const a = new BroadcastProducer(); @@ -822,6 +870,57 @@ test("a route superseded while its properties are checked stops being asked", as c.close(); }); +test("track info waits for the copy being read to say what the track is", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + // a answers what the track is only when told to. + const requests = wireOf(a).requested(); + b.createTrack("seq", { priority: 2 }); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + front.serve(b.consume()); + const info = front.consumer.track("seq").info(); + info.catch(() => {}); + await settle(); + + const request = await requests; + request?.accept({ priority: 1 }); + await expect(info).rejects.toThrow("changed its properties"); + expect((await sub.info()).priority).toBe(1); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); +}); + +test("a refusal from the copy being read is the track's answer, for its readers and fetches alike", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + // a refuses the track only when told to. + const requests = wireOf(a).requested(); + const tb = copy(b); + tb.write(0); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + front.serve(b.consume()); + const fetching = front.consumer.track("seq").fetchGroup(0); + fetching.catch(() => {}); + await settle(); + + const request = await requests; + request?.reject(new NotFound("seq")); + await expect(fetching).rejects.toBeInstanceOf(NotFound); + await expect(sub.recvGroup()).rejects.toBeInstanceOf(NotFound); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); +}); + test("a fetch waits for the copy being read to say what the track is", async () => { const front = new Front(); const a = new BroadcastProducer(); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 354f68c5688..865b5aeb8e9 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -12,9 +12,11 @@ * handed out. A replaced copy keeps delivering until the new copy delivers its first group, then * is bounded just below it (or at the newest group handed out, if later), so a group only the old * route got after the switch is not lost. It is kept for unread groups it had and groups still - * being read, and dropped once neither needs it. A group carries on across the switch through its {@link Recover}: once its - * copy fails with its route, or stalls while a newer route serves, it continues from the serving - * route's copy of the same group at the same frame, found in that copy's cache or fetched from it. + * being read (from it, or from another copy while it holds them too), and dropped once neither + * needs it. A group carries on across the switch through its {@link Recover}: once its copy + * stalls while a newer route serves, it continues from the serving route's copy of the same group + * at the same frame, found in that copy's cache or fetched from it. Once its copy fails, any other + * copy still holding the group continues it first, since a group's name fixes its content. * * A group is handed out once however many copies deliver it: the reader skips any sequence its * subscriber already handed out, the same record that keeps a direct reader from taking a @@ -212,7 +214,10 @@ interface Copy { datagramsDone?: Error | null; /** The newest datagram handed out before this copy was subscribed, which it may replay. */ datagramsSeen?: number; - /** Groups handed out from this copy and still read, which keep a replaced copy subscribed. */ + /** + * Groups still read that came from this copy, or that it held too when handed out, which keep + * a replaced copy subscribed. + */ lease: number; } @@ -250,7 +255,7 @@ class Reader implements ResumeSource { #newest?: number; /** The newest datagram handed out. */ #datagram?: number; - /** Bumped when this reader subscribes a new copy, waking its parked reads. */ + /** Bumped when this reader subscribes a new copy or a copy's last lease ends, waking its parked reads. */ #changed = new Signal(0); #dispose: Dispose[] = []; #closed = false; @@ -350,11 +355,26 @@ class Reader implements ResumeSource { }); } + /** Wake parked reads, so they look again and let go of what nothing needs. */ + wake(): void { + this.#changed.update((n) => n + 1); + } + /** The reader's copy of the route at `generation`, if it subscribed one. */ copy(generation: number): Copy | undefined { return this.#copies.find((copy) => copy.generation === generation); } + /** The copies other than `except` whose track still holds group `sequence` unfailed, newest first. */ + holding(sequence: number, except: Copy): Copy[] { + const holding: Copy[] = []; + for (let index = this.#copies.length - 1; index >= 0; index--) { + const copy = this.#copies[index]; + if (copy !== except && hooks.holdsGroup(copy.sub, sequence)) holding.push(copy); + } + return holding; + } + /** * `copy` handed out `sequence`. The serving copy's first group bounds the copies it replaced * just below it: what they hold under it may be groups the new route never has, and what @@ -534,7 +554,10 @@ class Reader implements ResumeSource { #handOut(copy: Copy, group: GroupConsumer): GroupConsumer { this.#newest = Math.max(this.#newest ?? 0, group.sequence); - const recover = new Recover(this, group, copy); + // Another copy holding the group is where a failure continues it, so it stays while the + // group is read. Only while copies overlap: a passthrough read has none. + const backups = this.#copies.length > 1 ? this.holding(group.sequence, copy) : []; + const recover = new Recover(this, group, copy, backups); hooks.recoverGroup(group, recover); recover.watch(group); return group; @@ -741,12 +764,16 @@ class Recover implements GroupRecover { #sequence: number; /** The copy being read, whose subscription this group keeps alive while it is read. */ #copy?: Copy; + /** Other copies that held the group when it was handed out, kept for a failure to continue from. */ + #backups: Copy[]; - constructor(reader: Reader, group: GroupConsumer, copy: Copy) { + constructor(reader: Reader, group: GroupConsumer, copy: Copy, backups: Copy[] = []) { this.#reader = reader; this.#sequence = group.sequence; this.#copy = copy; + this.#backups = backups; copy.lease++; + for (const backup of backups) backup.lease++; } /** @@ -758,8 +785,13 @@ class Recover implements GroupRecover { } #release(): void { - if (this.#copy) this.#copy.lease--; + const released = this.#copy ? [this.#copy, ...this.#backups] : this.#backups; this.#copy = undefined; + this.#backups = []; + let freed = false; + for (const copy of released) if (--copy.lease === 0 && copy.replaced) freed = true; + // A parked read lets go of a replaced copy nothing needs any more. + if (freed) this.#reader.wake(); } get changed(): Getter { @@ -812,12 +844,20 @@ class Recover implements GroupRecover { const generation = route.generation; const serving = route.source ? reader.copy(generation) : undefined; const waits: Getter[] = [route.changed]; + // A failed group continues from any copy still holding it, newest first: its name fixes + // its content. A stall only moves to the serving copy, so two stalled copies never trade + // the group back and forth. + if (failed) { + const held = reader.holding(this.#sequence, current)[0]; + const cached = held && hooks.cachedGroup(held.sub, this.#sequence); + if (cached) return this.#replacement(cached, held); + } if (!serving) { // Nothing serves: a failed copy waits for a route, unless none will come. if (failed && route.concluded) throw failed; } else if (generation !== current.generation || failed) { // A newer copy's track caches it: continue from there. - if (generation !== current.generation) { + if (!failed && generation !== current.generation) { const cached = hooks.cachedGroup(serving.sub, this.#sequence); if (cached) return this.#replacement(cached, serving); } @@ -956,7 +996,11 @@ export class Front { this.#record(route.name, info, previous as Source); return info; }, - () => undefined, + // Its route failing is no answer; a refusal is the track's. + (err: unknown) => { + if (routeFailed(err instanceof Error ? err : new Error(String(err)))) return undefined; + throw err; + }, ); Promise.all([expected, ask(source, route.name, retired)]) .then( @@ -1005,12 +1049,7 @@ export class Front { let verified = this.#verified.get(source); if (verified?.has(name)) return; const retired = this.#retire.signal; - // A copy a reader still reads says what the track is before any fetch does. - const served = this.#routes.get(name)?.served; - if (!recorded && served && served !== source) { - const info = await ask(served, name, retired).catch(() => undefined); - if (info) this.#record(name, info, served); - } + await this.#askServed(name, source, retired); const info = await ask(source, name, retired); // The first answer says what the track is; later sources have to match it. const expected = this.#info.get(name); @@ -1021,6 +1060,21 @@ export class Front { this.#verified.set(source, verified); } + // A copy a reader still reads says what the track is before `source` does: one logical track + // has one set of properties. Its route failing is no answer; a refusal is the track's. + async #askServed(name: string, source: Source, retired: AbortSignal): Promise { + const served = this.#routes.get(name)?.served; + if (this.#info.has(name) || !served || served === source) return; + let info: track.Info; + try { + info = await ask(served, name, retired); + } catch (err) { + if (routeFailed(err instanceof Error ? err : new Error(String(err)))) return; + throw err; + } + this.#record(name, info, served); + } + #record(name: string, info: track.Info, source: Source): void { if (!this.#info.has(name)) this.#info.set(name, { info, source }); } @@ -1052,6 +1106,7 @@ export class Front { const retired = this.#retire.signal; let info: track.Info; try { + await this.#askServed(name, source, retired); info = await ask(source, name, retired, hold); } catch (err) { // Another route took over before this one answered: ask it instead. diff --git a/js/net/src/track.ts b/js/net/src/track.ts index b0b3c75b08e..14aa45e18da 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -533,6 +533,8 @@ function binary(value: number): [bigint, number] { let bindProducerSequence: (producer: Producer, sequence: TrackSequence) => void; // A fresh read handle on a group the producer still caches; assigned in its static block. let cachedMirror: (producer: Producer, sequence: number) => GroupConsumer | undefined; +// Whether the producer still caches a group that has not failed; assigned in its static block. +let cachedHolds: (producer: Producer, sequence: number) => boolean; // The sequence-order cursor lives inside `Subscriber` (it shares the group buffer and the // drift anchor with the arrival cursor), so `Ordered` reaches it through this bridge, @@ -699,6 +701,10 @@ export class Producer { hooks.bindSequence = bindProducerSequence; hooks.cachedLargest = (producer) => producer.#cachedLargest(); hooks.reportedLargest = (producer, largest) => producer.#advanceLargest(largest); + cachedHolds = (producer, sequence) => { + const entry = producer.#cached.get(sequence); + return entry !== undefined && !(entry.group.closed.peek() instanceof Error); + }; cachedMirror = (producer, sequence) => { const entry = producer.#cached.get(sequence); if (!entry || entry.group.closed.peek() instanceof Error) return undefined; @@ -1676,6 +1682,10 @@ export class Subscriber { const producer = subscriber.#state.producer; return producer && cachedMirror(producer, sequence); }; + hooks.holdsGroup = (subscriber, sequence) => { + const producer = subscriber.#state.producer; + return producer !== undefined && cachedHolds(producer, sequence); + }; hooks.resumeSubscriber = (name, subscription, source) => { const state = new TrackState(subscription); const subscriber = new Subscriber(name, state); diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index c9bf0668d23..987f00bffa8 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -10,10 +10,12 @@ //! it handed out. A replaced copy keeps delivering until the new copy delivers its first //! group, then is bounded just below it (or at the newest group handed out, if later), so a //! group only the old route got after the switch is not lost. It is kept for unread groups it -//! had and groups still being read, and dropped once neither needs it. A group carries on -//! across the switch through its [`Recover`]: once its copy fails with its route, or stalls -//! while a newer route serves, it continues from the serving route's copy of the same group -//! at the same frame (and byte), found in that copy's cache or fetched from it. +//! had and groups still being read (from it, or from another copy while it holds them too), +//! and dropped once neither needs it. A group carries on across the switch through its +//! [`Recover`]: once its copy stalls while a newer route serves, it continues from the +//! serving route's copy of the same group at the same frame (and byte), found in that copy's +//! cache or fetched from it. Once its copy fails, any other copy still holding the group +//! continues it first, since a group's name fixes its content. //! //! A group is handed out once however many copies deliver it: the reader skips any //! sequence its [`track::Subscriber`] already handed out, the same record that keeps a @@ -276,8 +278,8 @@ struct Copy { /// The newest datagram handed out before this copy was subscribed: a copy that /// buffers datagrams replays them, and those were already handed out. datagrams_seen: Option<(u64, u64)>, - /// Held by every group handed out from this copy, so a replaced copy stays - /// subscribed while one is still read. + /// Held by every group handed out from this copy, or that it held too when handed + /// out, so a replaced copy stays subscribed while one is still read. lease: Arc<()>, } @@ -555,6 +557,8 @@ impl Reader { let serving = self.generation; let groups = self.groups; let groups_read = self.groups_read; + let newest = self.newest; + let mirrored = &self.mirrored; self.copies.retain_mut(|copy| { // The serving copy, or one between routes with nothing replacing it yet. if !copy.replaced || copy.needed(Some(copy.generation) == serving) { @@ -564,10 +568,15 @@ impl Reader { if copy.done.is_some() { return false; } - // Not yet bounded, it may still feed a group the new copy never has, unless - // nobody reads groups at all. - if copy.until.is_none() && groups_read { - return true; + if copy.until.is_none() { + // Not yet bounded, it may still feed a group the new copy never has. + if groups_read { + return true; + } + // Nobody read groups yet, so no new copy's group will bound it: it finishes + // the groups it has, and nothing newer. + copy.until = Some(newest.max(copy.track.latest_sequence())); + copy.update(mirrored); } let end = copy.end(); let Poll::Ready(Ok(sub)) = copy.poll_ready(groups, &kio::Waiter::noop()) else { @@ -612,6 +621,15 @@ impl Reader { fn hand_out(&mut self, index: usize, group: group::Consumer) -> group::Consumer { let sequence = group.sequence; self.newest = self.newest.max(Some(sequence)); + // Another copy holding the group is where a failure continues it, so it stays while + // the group is read. Only while copies overlap: a passthrough read has none. + let backups = match self.copies.len() { + 1 => Vec::new(), + _ => (self.copies.iter().enumerate()) + .filter(|(other, copy)| *other != index && copy.track.peek_group(sequence).is_some()) + .map(|(_, copy)| copy.lease.clone()) + .collect(), + }; let copy = &self.copies[index]; group.with_recover(Recover { sequence, @@ -621,6 +639,7 @@ impl Reader { generation: copy.generation, copy: copy.track.clone(), lease: Some(copy.lease.clone()), + backups, fetch: None, }) } @@ -911,6 +930,9 @@ pub(super) struct Recover { copy: track::Consumer, /// Keeps a replaced copy subscribed while this group is read from it. lease: Option>, + /// Keeps the other copies that held this group when handed out subscribed, for a + /// failure to continue from. + backups: Vec>, /// A fetch of the rest of the group from the serving route, by its generation. fetch: Option<(u64, Fetch)>, } @@ -931,6 +953,7 @@ impl Clone for Recover { generation: self.generation, copy: self.copy.clone(), lease: self.lease.clone(), + backups: self.backups.clone(), // A clone reads in parallel, so it asks for itself. fetch: None, } @@ -989,6 +1012,14 @@ impl Recover { if let Some(Err(err)) = &end { return Poll::Ready(Err(err.clone())); } + // A failed group continues from any copy still holding it, newest first: its name + // fixes its content. A stall only moves to the serving copy, so two stalled copies + // never trade the group back and forth. + if failed.is_some() + && let Some(replacement) = self.held(index) + { + return Poll::Ready(Ok(replacement)); + } let Some(serving) = serving else { // Nothing serves: a failed copy waits for a route, unless none will come. return match (failed, closed || end.is_some()) { @@ -1021,6 +1052,25 @@ impl Recover { } } + /// The group from the newest of the reader's other copies that still holds it unfailed, + /// positioned at frame `index`. + fn held(&self, index: u64) -> Option { + let reader = self.reader.as_ref()?.upgrade()?; + let reader = reader.lock().unwrap_or_else(PoisonError::into_inner); + reader.copies.iter().rev().find_map(|copy| { + if copy.generation == self.generation { + return None; + } + let mut group = copy.track.peek_group(self.sequence)?; + group.start_at(index); + (group.index() == index).then(|| Replacement { + group, + generation: copy.generation, + copy: copy.track.clone(), + }) + }) + } + /// [`Self::poll`] once a newer route serves, through its copy `serving`. fn poll_serving( &mut self, @@ -1185,6 +1235,7 @@ impl kio::Task for Fetching { generation, copy: serving.clone(), lease: None, + backups: Vec::new(), fetch: None, }))), // The route failed: ask the next one, unless none will come. @@ -1536,6 +1587,54 @@ mod test { } } + /// A group the serving copy failed continues from a replaced copy that holds it whole, + /// in either read order, whether it failed before or after it was handed out, and even + /// once the reader read on past it. + #[test] + #[cfg(feature = "async")] + fn a_group_the_serving_copy_failed_continues_from_a_replaced_copy() { + for (ordered, early) in [(false, true), (true, true), (false, false), (true, false)] { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, HOUR); + write(&a, 0); + assert_eq!(drain(&mut sub, ordered), [0]); + let b = copy(); + routes.serve(b.consume()); + + // Both routes hold group 1: the old one whole, the serving one failing after a frame. + let mut whole = a.create_group(group::Info { sequence: 1 }).unwrap(); + whole.write_frame(ts(1), b"a".as_ref()).unwrap(); + whole.write_frame(ts(2), b"b".as_ref()).unwrap(); + whole.finish().unwrap(); + let mut failing = b.create_group(group::Info { sequence: 1 }).unwrap(); + failing.write_frame(ts(1), b"a".as_ref()).unwrap(); + let mut failing = Some(failing); + if early { + failing.take().unwrap().abort(Error::SessionClosed).unwrap(); + } + let noop = kio::Waiter::noop(); + let res = match ordered { + true => sub.poll_next_group(&noop), + false => sub.poll_recv_group(&noop), + }; + let Poll::Ready(Ok(Some(mut reading))) = res else { + panic!("expected group 1"); + }; + assert_eq!(reading.sequence, 1); + // Reading on must not let go of the replaced copy while the group still needs it. + assert_eq!(drain(&mut sub, ordered), Vec::::new()); + if let Some(failing) = failing { + failing.abort(Error::SessionClosed).unwrap(); + } + assert_eq!(read(&mut reading), Some(b"a".to_vec())); + assert_eq!(read(&mut reading), Some(b"b".to_vec())); + assert_eq!(read(&mut reading), None); + } + } + /// A group the new copy replays, already handed out, does not bound the replaced copy: /// the old route's next group still arrives. #[test] @@ -2750,6 +2849,32 @@ mod test { assert_eq!(reader.lock().unwrap().copies.len(), 1); } + /// A reader that read no groups bounds a replaced copy at what it has, so the copy stops + /// taking new groups: no new copy's group handed out will ever bound it. + #[test] + #[cfg(feature = "async")] + fn datagram_readers_bound_replaced_copies() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(10)); + write(&a, 0); + let b = copy(); + routes.serve(b.consume()); + assert!(sub.recv_datagram().now_or_never().is_none()); + write(&a, 1); + { + let readers = routes.readers.lock().unwrap(); + let reader = readers[0].upgrade().unwrap(); + let reader = reader.lock().unwrap(); + assert_eq!(reader.copies.len(), 2, "its buffered group may still be read"); + assert_eq!(reader.copies[0].until, Some(Some(0))); + } + // A group read later still gets what it had, and nothing it took after. + assert_eq!(drain(&mut sub, false), [0]); + } + #[test] fn ending_the_front_releases_a_pending_fetch() { let routes = Producer::new(); From 949876acc8dd370840f6dc7db2cb7e0bf404fe58 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 08:25:54 -0700 Subject: [PATCH 20/31] fix(js/net): finished() ends with a refused replacement; a fetch watches closed only once it can recover finished() now rejects with the route's refusal once the old copy fails with its route, as group reads do. fetchGroup reads the group's closed only after attaching its recovery, so a route failure in between no longer releases the route early and again on adopt. Pins that a failed group continues from a failed route's finished group (the track drops its open ones), and plans the group-only close as quest/m1/resume-closed.md. Co-Authored-By: Claude Opus 5.5 --- js/net/src/resume.test.ts | 69 ++++++++++++++++++++++++++++++++++ js/net/src/resume.ts | 13 +++++-- quest/m1/README.md | 1 + quest/m1/resume-closed.md | 36 ++++++++++++++++++ rs/moq-net/src/model/resume.rs | 38 +++++++++++++++++++ 5 files changed, 154 insertions(+), 3 deletions(-) create mode 100644 quest/m1/resume-closed.md diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index bcd8700a395..5c09c03b48e 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -196,6 +196,75 @@ test("a replacement claiming different track properties is refused once the old close(); }); +test("waiting for the end of a track whose replacement was refused ends with the refusal", async () => { + const { a, b, swap, active, close } = await setup(); + const old = a.createTrack("seq", { priority: 1 }); + b.createTrack("seq", { priority: 2 }); + const sub = active.track("seq").subscribe(); + expect((await sub.info()).priority).toBe(1); + await swap(); + const finished = sub.finished(); + finished.catch(() => {}); + + // The old copy failing with its route leaves only the refusal. + old.close(new StreamError(StreamCode.SessionClosed)); + await expect(finished).rejects.toThrow("changed its properties"); + + sub.close(); + close(); +}); + +test("a fetched group failing as it is handed out releases its route once", async () => { + // The failure lands at each turn around the hand-out, so one lands between the fetch + // returning and its recovery being attached. + for (let turns = 0; turns < 24; turns++) { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + front.serve(a.consume()); + // A reader on another track of the same route holds the route's broadcast too. + const sub = front.consumer.track("seq").subscribe(); + ta.write(0); + expect(drain(sub)).toEqual([0]); + + const fa = a.createTrack("fetch"); + const failing = new GroupProducer(5); + fa.writeGroup(failing); + failing.writeFrame({ payload: enc.encode("5.0") }); + const fetching = front.consumer.track("fetch").fetchGroup(5); + for (let turn = 0; turn < turns; turn++) await Promise.resolve(); + failing.close(new StreamError(StreamCode.SessionClosed)); + await settle(); + + // The next route holds the fetched group whole; the old one still feeds the reader a + // group the new route never has. + const fb = b.createTrack("fetch"); + const whole = new GroupProducer(5); + fb.writeGroup(whole); + whole.writeFrame({ payload: enc.encode("5.0") }); + whole.writeFrame({ payload: enc.encode("5.1") }); + whole.close(); + b.createTrack("seq"); + front.serve(b.consume()); + const group = await fetching; + const frames: string[] = []; + for (let frame = await group.readFrame(); frame; frame = await group.readFrame()) + frames.push(dec.decode(frame.payload)); + expect(frames).toEqual(["5.0", "5.1"]); + group.close(); + await settle(); + + ta.write(1); + expect(drain(sub)).toEqual([1]); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); + } +}); + test("datagrams a new copy replays are handed out once", async () => { const { a, b, swap, active, close } = await setup(); const ta = copy(a); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 865b5aeb8e9..c3b5ce013ea 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -612,8 +612,13 @@ class Reader implements ResumeSource { const end = serving?.sub.end(); if (end) return end; const failed = serving?.sub.closed.peek(); - // A copy that failed is only the end of the track if no front replaces it. - if (failed instanceof Error && (this.route.concluded || !routeFailed(failed))) throw failed; + // A copy that failed is only the end of the track if no front replaces it, and a + // refused replacement leaves the refusal as the answer. + if (failed instanceof Error) { + if (!routeFailed(failed)) throw failed; + if (this.route.refusal) throw this.route.refusal; + if (this.route.concluded) throw failed; + } if (!serving && this.route.concluded) throw dropped(); const closed = this.#shell.closed.peek(); if (closed instanceof Error) throw closed; @@ -1134,7 +1139,6 @@ export class Front { async fetchGroup(name: string, sequence: number, options?: track.FetchGroupOptions): Promise { const fetched = await this.#fetch(name, sequence, options); let current = fetched.source; - void fetched.group.closed.then(() => current.release()); // Carried across route changes like a subscribed group, but by fetching again: a fetch has // no copy of its own to find the rest in. hooks.recoverGroup(fetched.group, { @@ -1164,6 +1168,9 @@ export class Front { }; }, }); + // Only once the group carries its recovery: reading `closed` before then would settle it + // on a failure recovery could still continue, releasing the route while it is read. + void fetched.group.closed.then(() => current.release()); return fetched.group; } diff --git a/quest/m1/README.md b/quest/m1/README.md index 5b5840fa747..96016410b0d 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -109,6 +109,7 @@ lands before the cut. - [TS fidelity and recovery](/quest/m1/ts-reliability/README.md) - both modes have explicit profiles, automated validation, loss tests, and a dedicated guide - [Resume reorder](/quest/m1/resume-reorder.md) - a group whose stream header arrives after its successor's still resumes on failover; only a dropped or long-gone group fails - [Rust resumed reads](/quest/m1/resume-ordered.md) - a Rust reader takes the lowest group across route copies in both orders, as JS does +- [Resumed close](/quest/m1/resume-closed.md) - a resumed JS track settles `closed` on a clean end even when only one read channel is polled - [Received timestamps are media time](/quest/m1/lite-time-provenance.md) - a received timed track's timestamps are media time, so the rewind check can run across hops - [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS - [HLS export drains after End](/quest/m1/hls-export-end-drain.md) - a cleanly ended broadcast still exports its retained segments diff --git a/quest/m1/resume-closed.md b/quest/m1/resume-closed.md new file mode 100644 index 00000000000..e031b8db701 --- /dev/null +++ b/quest/m1/resume-closed.md @@ -0,0 +1,36 @@ +# [S] A resumed track closes without both channels drained + +## Goal + +A resumed JS track subscriber (`Track.Subscriber` reading a front's route +copies) settles `closed` once its logical track ends cleanly, whether the +application reads groups, datagrams, or both. A group-only reader that reaches +`recvGroup()` EOF on a finite track sees `closed` settle and releases its +`Reader`, copies, and route sources, while a reader of the other channel still +drains what is buffered. + +## Plan + +Found by a hosted Codex review of #5836 (`resume.ts` `Reader.#finish`). Today a +clean end adds the channel that ran out to `#drained`, and the shell closes only +once both `"groups"` and `"datagrams"` are there. Nothing adds `"datagrams"` for +a reader that never calls `recvDatagram()`, so `closed` stays pending and the +route keeps the reader. Errors already close at once. + +- Settle `closed` when the serving copy's track ends cleanly, from the copy's + own end (its `closed` and declared end), instead of waiting for both read + channels to be polled to EOF. +- Keep each channel's buffered backlog readable after that: one channel running + out must not cut off the other's unread data, as `draining groups to their end + leaves buffered datagrams readable` checks today. +- Check whether a direct (non-resumed) `Track.Subscriber` and the Rust + `resume::Subscriber` already settle this way, and match them. +- Regression test: a group-only reader of a finite resumed track sees `closed` + settle and the route release its copy. + +Public API: behavior only (`closed` settles for a group-only resumed reader). +Wire: none. + +## Related + +- [Rust resumed reads](/quest/m1/resume-ordered.md) - the other resume follow-up from #5836 diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index a28df95f872..00c75724fd9 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -1602,6 +1602,44 @@ mod test { } } + /// A copy whose route failed still continues a failed group it finished: a failed track + /// keeps its finished groups and drops its open ones, so only a whole group is taken. + #[test] + #[cfg(feature = "async")] + fn a_failed_group_continues_from_a_failed_route_that_finished_it() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, HOUR); + write(&a, 0); + assert_eq!(drain(&mut sub, false), [0]); + let b = copy(); + routes.serve(b.consume()); + let c = copy(); + routes.serve(c.consume()); + + // Every route holds group 1: a whole, b cut off by its route failing, c failing after a frame. + let mut whole = a.create_group(group::Info { sequence: 1 }).unwrap(); + whole.write_frame(ts(1), b"a".as_ref()).unwrap(); + whole.write_frame(ts(2), b"b".as_ref()).unwrap(); + whole.finish().unwrap(); + let mut partial = b.create_group(group::Info { sequence: 1 }).unwrap(); + partial.write_frame(ts(1), b"a".as_ref()).unwrap(); + b.clone().abort(Error::Dropped).unwrap(); + let mut failing = c.create_group(group::Info { sequence: 1 }).unwrap(); + failing.write_frame(ts(1), b"a".as_ref()).unwrap(); + let mut reading = recv(&mut sub); + assert_eq!(reading.sequence, 1); + // a's route fails too, after it finished the group. + a.clone().abort(Error::Dropped).unwrap(); + failing.abort(Error::SessionClosed).unwrap(); + assert_eq!(read(&mut reading), Some(b"a".to_vec())); + assert_eq!(read(&mut reading), Some(b"b".to_vec())); + assert_eq!(read(&mut reading), None); + drop(partial); + } + /// A group the serving copy failed continues from a replaced copy that holds it whole, /// in either read order, whether it failed before or after it was handed out, and even /// once the reader read on past it. From dedefa9f0735746e9d197ae67718188427b77489 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 10:39:15 -0700 Subject: [PATCH 21/31] fix(net): a replaced copy whose route ended still backs a group it holds A Rust reader dropped a replaced copy once its group channel ran out, even while a handed-out group still leased it as a backup. A later failure of the serving copy's group then found nothing to continue from and waited forever. Keep a leased copy whatever its channel did; JS already does, and gets the same regression test. Co-Authored-By: Claude Opus 5.5 --- js/net/src/resume.test.ts | 42 +++++++++++++++++++++++++++++++ rs/moq-net/src/model/resume.rs | 45 ++++++++++++++++++++++++++++++++-- 2 files changed, 85 insertions(+), 2 deletions(-) diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 5c09c03b48e..2ce0f391d76 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -840,6 +840,48 @@ test.each([ }, ); +test.each([true, false])( + "a replaced copy whose route fails still backs a group handed out while it held it (ordered: %p)", + async (ordered) => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + const view = ordered ? sub.ordered() : undefined; + const next = () => (view ? view.nextGroup() : sub.recvGroup()); + ta.write(0); + (await next())?.close(); + front.serve(b.consume()); + await settle(); + + ta.write(1, 2); + const failing = tb.write(1, 1, false); + const group = await next(); + if (group?.sequence !== 1) throw new Error("expected group 1"); + expect(dec.decode((await group.readFrame())?.payload)).toBe("1.0"); + + // The old route goes away, and the reader reads on and sees its copy end. + ta.producer.close(new StreamError(StreamCode.SessionClosed)); + const later = next(); + await settle(); + failing.close(new StreamError(StreamCode.SessionClosed)); + expect(dec.decode((await group.readFrame())?.payload)).toBe("1.1"); + expect(await group.readFrame()).toBeUndefined(); + + group.close(); + tb.write(2); + expect((await later)?.sequence).toBe(2); + view?.close(); + sub.close(); + front.consumer.close(); + a.close(); + b.close(); + }, +); + test("a fetched group that fails on the route still serving waits for the next route", async () => { const front = new Front(); const a = new BroadcastProducer(); diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 165237ca63d..7f123e437b6 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -335,9 +335,10 @@ impl Copy { Poll::Ready(Ok(sub)) } - /// Whether this copy is still needed: it serves, or a group read from it is still out. + /// Whether this copy is still needed: it serves, or a group read from or backed by it is + /// still out. Even once its channel ran out, since a failure continues from what it holds. fn needed(&self, serving: bool) -> bool { - serving || (self.done.is_none() && Arc::strong_count(&self.lease) > 1) + serving || Arc::strong_count(&self.lease) > 1 } } @@ -1688,6 +1689,46 @@ mod test { } } + /// A replaced copy whose route fails still backs a group handed out while it held it + /// whole, even once the reader polled on and saw that copy end. + #[test] + #[cfg(feature = "async")] + fn a_failed_group_continues_from_a_backup_whose_route_ended() { + for ordered in [false, true] { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, HOUR); + write(&a, 0); + assert_eq!(drain(&mut sub, ordered), [0]); + let b = copy(); + routes.serve(b.consume()); + + let mut whole = a.create_group(group::Info { sequence: 1 }).unwrap(); + whole.write_frame(ts(1), b"a".as_ref()).unwrap(); + whole.write_frame(ts(2), b"b".as_ref()).unwrap(); + whole.finish().unwrap(); + let mut failing = b.create_group(group::Info { sequence: 1 }).unwrap(); + failing.write_frame(ts(1), b"a".as_ref()).unwrap(); + let res = match ordered { + true => sub.poll_next_group(&kio::Waiter::noop()), + false => sub.poll_recv_group(&kio::Waiter::noop()), + }; + let Poll::Ready(Ok(Some(mut reading))) = res else { + panic!("expected group 1"); + }; + assert_eq!(read(&mut reading), Some(b"a".to_vec())); + + // The old route goes away, and the reader polls on and sees its copy end. + a.clone().abort(Error::Dropped).unwrap(); + assert_eq!(drain(&mut sub, ordered), Vec::::new()); + failing.abort(Error::SessionClosed).unwrap(); + assert_eq!(read(&mut reading), Some(b"b".to_vec())); + assert_eq!(read(&mut reading), None); + } + } + /// A group the new copy replays, already handed out, does not bound the replaced copy: /// the old route's next group still arrives. #[test] From 71271f335916fc97b773e3d90b5f3998ab05f340 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 12:41:32 -0700 Subject: [PATCH 22/31] fix(net): a fetch's replacement route is released once, adopted or not A fetched group's recovery released the replacement's route when that group's `closed` settled, unless it had been adopted. An empty, already closed replacement settles before adoption, so the route was released there and again when the adopted group ended, and its broadcast closed under readers still holding it. A replacement is now either adopted or discarded, exactly once: the group discards every replacement it does not adopt, and a fetch's discard is what gives its route back. A route released past zero now throws. Plan the other open #5836 finding, a replaced copy let go with a lower group still in flight, in quest/m1/resume-in-flight.md. Co-Authored-By: Claude Opus 5.5 --- js/net/src/group.test.ts | 6 +++++ js/net/src/group.ts | 8 +++---- js/net/src/internal.ts | 2 ++ js/net/src/resume.test.ts | 46 ++++++++++++++++++++++++++++++++++++ js/net/src/resume.ts | 12 ++++++---- quest/m1/README.md | 1 + quest/m1/resume-in-flight.md | 39 ++++++++++++++++++++++++++++++ 7 files changed, 105 insertions(+), 9 deletions(-) create mode 100644 quest/m1/resume-in-flight.md diff --git a/js/net/src/group.test.ts b/js/net/src/group.test.ts index bc1baabea0f..4c6e9fd287d 100644 --- a/js/net/src/group.test.ts +++ b/js/net/src/group.test.ts @@ -264,6 +264,7 @@ test("a group that ends as its replacement arrives ends rather than adopting it" const replacement = new Producer(0); let resolve: ((value: Replacement) => void) | undefined; let adopted = false; + let discarded = false; hooks.recoverGroup(consumer, { changed: route, wants: (failed) => failed !== undefined || route.peek() > 0, @@ -285,10 +286,14 @@ test("a group that ends as its replacement arrives ends rather than adopting it" adopt: () => { adopted = true; }, + discard: () => { + discarded = true; + }, }); producer.close(); expect(await read).toBeUndefined(); expect(adopted).toBe(false); + expect(discarded).toBe(true); }); test("a group recovered twice reads on and settles", async () => { @@ -304,6 +309,7 @@ test("a group recovered twice reads on and settles", async () => { group: replacements[next++].mirror(), expiry: { expired: () => false, changed: [] }, adopt: () => {}, + discard: () => {}, }), }); void consumer.closed; diff --git a/js/net/src/group.ts b/js/net/src/group.ts index c221db422cd..6c9981499ce 100644 --- a/js/net/src/group.ts +++ b/js/net/src/group.ts @@ -628,7 +628,7 @@ export class Consumer { (replacement) => { started.settled = true; // Abandoned while it was out: nobody reads it. - if (abort.signal.aborted) replacement.group.close(); + if (abort.signal.aborted) replacement.discard(); else started.replacement = replacement; }, (err: unknown) => { @@ -653,7 +653,7 @@ export class Consumer { if (!replacement) return true; if (!failed && (this.#state.frames.peek().length > 0 || this.#state.closed.peek() !== undefined)) { // The copy being read delivered or ended meanwhile; keep reading it. - replacement.group.close(); + replacement.discard(); return true; } this.#adopt(replacement); @@ -671,7 +671,7 @@ export class Consumer { if (!attempt) return; this.#attempt = undefined; attempt.abort.abort(); - attempt.replacement?.group.close(); + attempt.replacement?.discard(); } // Read the rest of the group from `replacement`, from the frame this one reached. @@ -679,7 +679,7 @@ export class Consumer { const next = replacement.group.#state; const index = this.#index(); if (next.start > index) { - replacement.group.close(); + replacement.discard(); this.#terminal ??= new Error(`group ${this.sequence} cannot resume at frame ${index}`); if (this.#verdict.peek() === undefined) this.#verdict.set(this.#terminal); return; diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index dce0302b9f6..3a17fead7d4 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -267,6 +267,8 @@ export interface Replacement { readonly expiry: Expiry; /** The group reads this copy from now on. */ adopt(): void; + /** Nobody reads this copy: close it and give back what it holds. */ + discard(): void; } /** A datagram read that does not wait. */ diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 90a0d9f2202..7a02492714e 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -215,6 +215,52 @@ test("waiting for the end of a track whose replacement was refused ends with the close(); }); +test("a fetched group continued from an empty, already-closed copy releases that route once", async () => { + const errors: unknown[] = []; + const onError = (err: unknown) => errors.push(err); + process.on("unhandledRejection", onError); + try { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const c = new BroadcastProducer(); + copy(a); + copy(b); + copy(c); + front.serve(a.consume()); + // A reader on another track holds each route's broadcast too. + const sub = front.consumer.track("seq").subscribe(); + + // Group 5 is empty: the first route fails it, the next holds it closed. + const failing = new GroupProducer(5); + a.createTrack("fetch").writeGroup(failing); + const group = await front.consumer.track("fetch").fetchGroup(5); + const read = group.readFrame(); + const whole = new GroupProducer(5); + b.createTrack("fetch").writeGroup(whole); + whole.close(); + front.serve(b.consume()); + await settle(); + failing.close(new StreamError(StreamCode.SessionClosed)); + expect(await read).toBeUndefined(); + group.close(); + await settle(); + + // Moving off the route and closing everything gives back every hold, and no more. + front.serve(c.consume()); + await settle(); + sub.close(); + front.consumer.close(); + await settle(); + a.close(); + b.close(); + c.close(); + } finally { + process.off("unhandledRejection", onError); + } + expect(errors).toEqual([]); +}); + test("a fetched group failing as it is handed out releases its route once", async () => { // The failure lands at each turn around the hand-out, so one lands between the fetch // returning and its recovery being attached. diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 3fbf7e7f43c..2fc0b0fc443 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -121,6 +121,7 @@ class Source { } release(): void { + if (this.#holds === 0) throw new Error("route released more often than retained"); if (--this.#holds === 0) this.#consumer.close(); } } @@ -903,6 +904,7 @@ class Recover implements GroupRecover { this.#copy = copy; copy.lease++; }, + discard: () => group.close(), }; } } @@ -1158,19 +1160,19 @@ export class Front { replace: async (_index, failed, signal) => { // A stalled fetch only moves to another route; a failed one may ask its own again. const next = await this.#fetch(name, sequence, { ...options, signal }, failed ? undefined : current); - let adopted = false; - void next.group.closed.then(() => { - if (!adopted) next.source.release(); - }); + // The fetch's hold on its route passes to the group on adoption, or goes with the discard. return { group: next.group, // The route applies the fetch's own budget. expiry: { expired: () => false, changed: [] }, adopt: () => { - adopted = true; current.release(); current = next.source; }, + discard: () => { + next.group.close(); + next.source.release(); + }, }; }, }); diff --git a/quest/m1/README.md b/quest/m1/README.md index 2badd629258..a0be8196c14 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -111,6 +111,7 @@ lands before the cut. - [Resume reorder](/quest/m1/resume-reorder.md) - a group whose stream header arrives after its successor's still resumes on failover; only a dropped or long-gone group fails - [Rust resumed reads](/quest/m1/resume-ordered.md) - a Rust reader takes the lowest group across route copies in both orders, as JS does - [Resumed close](/quest/m1/resume-closed.md) - a resumed JS track settles `closed` on a clean end even when only one read channel is polled +- [Resumed in-flight groups](/quest/m1/resume-in-flight.md) - a replaced route copy stays while a group below its bound is still in flight - [Received timestamps are media time](/quest/m1/lite-time-provenance.md) - a received timed track's timestamps are media time, so the rewind check can run across hops - [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS - [HLS export drains after End](/quest/m1/hls-export-end-drain.md) - a cleanly ended broadcast still exports its retained segments diff --git a/quest/m1/resume-in-flight.md b/quest/m1/resume-in-flight.md new file mode 100644 index 00000000000..4a5cfd974a4 --- /dev/null +++ b/quest/m1/resume-in-flight.md @@ -0,0 +1,39 @@ +# [M] A replaced copy stays while a group below its bound is in flight + +## Goal + +After a same-epoch route swap, a replaced copy bounded below the new copy's +first group stays subscribed until every group under that bound it may still +deliver has arrived, been dropped by its publisher, or fallen outside the +reader's budget. An arrival-order reader then gets a group the old route was +still sending when the new route took over, in Rust and JS alike. + +## Plan + +Found by a hosted Codex review of #5836. The new copy handing out group F +bounds the old copy at F-1, and the old copy is then kept only while a group +under the bound is *buffered* unread: JS `Reader.#letGo` in +`js/net/src/resume.ts` asks `hooks.hasUnreadGroup`, and Rust `Reader::let_go` +in `rs/moq-net/src/model/resume.rs` asks `has_unread_group`. A read before +F-1's stream header lands finds nothing buffered and lets the copy go, so F-1 +never arrives. Main bounds the old copy at the swap itself and never delivers +that group in either order, so this completes #5836's handover rather than +fixing a regression. An ordered reader past F already skips F-1, so only +arrival order gains. + +- Keep a bounded copy while a gap under its bound is still open, using the rule + [Resume reorder](/quest/m1/resume-reorder.md) settles for a passed group (a + gap stays open for the subscription's grace) rather than a second rule. +- The copy goes once its group channel ends, every sequence under the bound has + arrived or been dropped, or the reader's budget has expired the gap. Never + wait on a peer past that: relays report what they have. +- Regression test in both languages, with mocked time: the old copy writes F-1 + only after the new copy handed out F and the reader polled, and an + arrival-order reader still gets F-1. A dropped or expired F-1 lets the copy go. + +Public API: behavior only. Wire: none. + +## Related + +- [Resume reorder](/quest/m1/resume-reorder.md) - the gap rule this reuses +- [Rust resumed reads](/quest/m1/resume-ordered.md) - another resume follow-up from #5836 From e108157a596494339daf8250422c715ee66d1524 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 13:06:40 -0700 Subject: [PATCH 23/31] quest: a replaced copy that gets a handed-out group late backs it Plan the #5836 review finding that hand-out leases only the copies holding a group at that moment, so a replaced copy that finishes it afterwards is let go before a serving-group failure can continue from it. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 2 +- quest/m1/resume-in-flight.md | 62 +++++++++++++++++++++++++----------- 2 files changed, 44 insertions(+), 20 deletions(-) diff --git a/quest/m1/README.md b/quest/m1/README.md index a0be8196c14..c405376ea68 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -111,7 +111,7 @@ lands before the cut. - [Resume reorder](/quest/m1/resume-reorder.md) - a group whose stream header arrives after its successor's still resumes on failover; only a dropped or long-gone group fails - [Rust resumed reads](/quest/m1/resume-ordered.md) - a Rust reader takes the lowest group across route copies in both orders, as JS does - [Resumed close](/quest/m1/resume-closed.md) - a resumed JS track settles `closed` on a clean end even when only one read channel is polled -- [Resumed in-flight groups](/quest/m1/resume-in-flight.md) - a replaced route copy stays while a group below its bound is still in flight +- [Resumed in-flight groups](/quest/m1/resume-in-flight.md) - a replaced route copy stays while a group below its bound is in flight or it backs a handed-out group late - [Received timestamps are media time](/quest/m1/lite-time-provenance.md) - a received timed track's timestamps are media time, so the rewind check can run across hops - [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS - [HLS export drains after End](/quest/m1/hls-export-end-drain.md) - a cleanly ended broadcast still exports its retained segments diff --git a/quest/m1/resume-in-flight.md b/quest/m1/resume-in-flight.md index 4a5cfd974a4..292ea22cb86 100644 --- a/quest/m1/resume-in-flight.md +++ b/quest/m1/resume-in-flight.md @@ -1,25 +1,32 @@ -# [M] A replaced copy stays while a group below its bound is in flight +# [M] A replaced copy stays while a group it may still carry is needed ## Goal -After a same-epoch route swap, a replaced copy bounded below the new copy's -first group stays subscribed until every group under that bound it may still -deliver has arrived, been dropped by its publisher, or fallen outside the -reader's budget. An arrival-order reader then gets a group the old route was -still sending when the new route took over, in Rust and JS alike. +After a same-epoch route swap, a replaced copy is let go only once nothing it +may still carry is needed, in Rust and JS alike: + +- A copy bounded below the new copy's first group stays subscribed until every + group under that bound it may still deliver has arrived, been dropped by its + publisher, or fallen outside the reader's budget. An arrival-order reader then + gets a group the old route was still sending when the new route took over. +- A copy that receives a handed-out group after hand-out backs that group as if + it had held it then, so a later failure of the serving copy's group continues + from it. ## Plan -Found by a hosted Codex review of #5836. The new copy handing out group F -bounds the old copy at F-1, and the old copy is then kept only while a group -under the bound is *buffered* unread: JS `Reader.#letGo` in -`js/net/src/resume.ts` asks `hooks.hasUnreadGroup`, and Rust `Reader::let_go` -in `rs/moq-net/src/model/resume.rs` asks `has_unread_group`. A read before -F-1's stream header lands finds nothing buffered and lets the copy go, so F-1 -never arrives. Main bounds the old copy at the swap itself and never delivers -that group in either order, so this completes #5836's handover rather than -fixing a regression. An ordered reader past F already skips F-1, so only -arrival order gains. +Found by hosted Codex reviews of #5836. + +**In flight below the bound.** The new copy handing out group F bounds the old +copy at F-1, and the old copy is then kept only while a group under the bound +is *buffered* unread: JS `Reader.#letGo` in `js/net/src/resume.ts` asks +`hooks.hasUnreadGroup`, and Rust `Reader::let_go` in +`rs/moq-net/src/model/resume.rs` asks `has_unread_group`. A read before F-1's +stream header lands finds nothing buffered and lets the copy go, so F-1 never +arrives. Main bounds the old copy at the swap itself and never delivers that +group in either order, so this completes #5836's handover rather than fixing a +regression. An ordered reader past F already skips F-1, so only arrival order +gains. - Keep a bounded copy while a gap under its bound is still open, using the rule [Resume reorder](/quest/m1/resume-reorder.md) settles for a passed group (a @@ -27,9 +34,26 @@ arrival order gains. - The copy goes once its group channel ends, every sequence under the bound has arrived or been dropped, or the reader's budget has expired the gap. Never wait on a peer past that: relays report what they have. -- Regression test in both languages, with mocked time: the old copy writes F-1 - only after the new copy handed out F and the reader polled, and an - arrival-order reader still gets F-1. A dropped or expired F-1 lets the copy go. + +**Late backups.** A handed-out group leases the other copies holding it as +backups, but only those holding it at hand-out (JS `Reader.#handOut` via +`holding`, Rust `Reader::hand_out` via `peek_group`). A replaced copy that gets +group F afterwards holds no lease, and F is above its F-1 bound, so the next +poll lets it go. A failure of the serving copy's F then waits for another +route, though an identical whole copy was there. + +- Decide between leasing late arrivals while a handed-out group is out (the + reader knows which groups are out and can lease a copy when one lands) and + keeping every replaced copy until the groups handed out above its bound end. + Prefer whichever keeps one lease rule for both paths. + +Regression tests in both languages, with mocked time: + +- The old copy writes F-1 only after the new copy handed out F and the reader + polled, and an arrival-order reader still gets F-1. A dropped or expired F-1 + lets the copy go. +- The old copy finishes F after the new copy handed F out and the reader polled + on. The serving copy's F then fails, and the read continues from the old copy. Public API: behavior only. Wire: none. From 9b511bff26b64358e94b5c4faf3c0bac74d925ac Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 13:13:06 -0700 Subject: [PATCH 24/31] quest: a forked resumed subscriber follows route swaps Plan the #5836 review finding that `fork()` on a resumed JS subscriber ties the fork to the copy serving at the time, so a draft-20 FILL read ends with the old route instead of continuing on the new one. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 1 + quest/m1/resume-fork.md | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 34 insertions(+) create mode 100644 quest/m1/resume-fork.md diff --git a/quest/m1/README.md b/quest/m1/README.md index c405376ea68..d4995dc74a7 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -112,6 +112,7 @@ lands before the cut. - [Rust resumed reads](/quest/m1/resume-ordered.md) - a Rust reader takes the lowest group across route copies in both orders, as JS does - [Resumed close](/quest/m1/resume-closed.md) - a resumed JS track settles `closed` on a clean end even when only one read channel is polled - [Resumed in-flight groups](/quest/m1/resume-in-flight.md) - a replaced route copy stays while a group below its bound is in flight or it backs a handed-out group late +- [Resumed forks](/quest/m1/resume-fork.md) - a fork of a resumed JS subscriber follows route swaps like its parent - [Received timestamps are media time](/quest/m1/lite-time-provenance.md) - a received timed track's timestamps are media time, so the rewind check can run across hops - [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS - [HLS export drains after End](/quest/m1/hls-export-end-drain.md) - a cleanly ended broadcast still exports its retained segments diff --git a/quest/m1/resume-fork.md b/quest/m1/resume-fork.md new file mode 100644 index 00000000000..4b7c9fefb24 --- /dev/null +++ b/quest/m1/resume-fork.md @@ -0,0 +1,33 @@ +# [S] A forked resumed subscriber follows route swaps + +## Goal + +A fork of a resumed JS track subscriber (`Track.Subscriber.fork()` on a +subscriber reading a front's route copies) is itself a logical reader. It +follows the same-epoch route swaps its parent does, so a read it has open +continues on the new route instead of ending with the old one. + +## Plan + +Found by a hosted Codex review of #5836. `ResumeSource.fork` in +`js/net/src/resume.ts` returns `serving.sub.fork(options)`, a subscriber tied +to the copy serving at fork time. The IETF draft-20 FILL path in +`js/net/src/ietf/publisher.ts` forks the track to drain cached and arriving +content. If the old route fails while the fill reads an incomplete group or +waits for an announced one, the parent recovers through the new route and the +fill ends on the old one. + +- Make a resumed `fork` open another logical reader on the same `Route`, as + `hooks.resumeSubscriber` does for the parent, with the fork's own + subscription, instead of delegating to the serving copy. +- Rust has no subscriber fork today, so nothing to mirror. Check that the IETF + FILL path in `rs/moq-net` reads through the logical track. +- Regression test: fork a resumed subscriber, swap routes while the fork reads + an open group, fail the old route, and the fork reads the rest of the group + from the new one. + +Public API: behavior only. Wire: none. + +## Related + +- [Resumed in-flight groups](/quest/m1/resume-in-flight.md) - another resume follow-up from #5836 From 530d19d65f2325634f0e60bda0bf34b1bd4eef78 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 13:25:01 -0700 Subject: [PATCH 25/31] quest: bound the replaced copies a datagram-only reader keeps Plan the #5836 review finding that a reader that never reads groups keeps every bounded replaced copy holding an unread group, one per route swap. Main does the same, so it goes with the other let-go rules in resume-in-flight. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 2 +- quest/m1/resume-in-flight.md | 15 +++++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/quest/m1/README.md b/quest/m1/README.md index afc6ac5e58e..66f3ee1fee0 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -110,7 +110,7 @@ lands before the cut. - [Resume reorder](/quest/m1/resume-reorder.md) - a group whose stream header arrives after its successor's still resumes on failover; only a dropped or long-gone group fails - [Rust resumed reads](/quest/m1/resume-ordered.md) - a Rust reader takes the lowest group across route copies in both orders, as JS does - [Resumed close](/quest/m1/resume-closed.md) - a resumed JS track settles `closed` on a clean end even when only one read channel is polled -- [Resumed in-flight groups](/quest/m1/resume-in-flight.md) - a replaced route copy stays while a group below its bound is in flight or it backs a handed-out group late +- [Resumed in-flight groups](/quest/m1/resume-in-flight.md) - a replaced route copy stays while a group below its bound is in flight or it backs a handed-out group late, and no longer than the reader's budget - [Resumed forks](/quest/m1/resume-fork.md) - a fork of a resumed JS subscriber follows route swaps like its parent - [Received timestamps are media time](/quest/m1/lite-time-provenance.md) - a received timed track's timestamps are media time, so the rewind check can run across hops - [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS diff --git a/quest/m1/resume-in-flight.md b/quest/m1/resume-in-flight.md index 292ea22cb86..6787e85b33e 100644 --- a/quest/m1/resume-in-flight.md +++ b/quest/m1/resume-in-flight.md @@ -47,6 +47,21 @@ route, though an identical whole copy was there. keeping every replaced copy until the groups handed out above its bound end. Prefer whichever keeps one lease rule for both paths. +**Bounded retention.** A reader that never reads groups (datagrams only) keeps +every bounded replaced copy that still has an unread group, since nothing ever +reads it. Rust `Reader::let_go` does this on main too. A peer that keeps +migrating the route then leaves one subscription and cache per swap. Keeping +copies longer, as above, must not make that worse. + +- A reader may read groups later (`datagram_reads_keep_unread_groups_on_a_replaced_copy`), + so an unread group still keeps its copy. Bound that by what the reader could + still take: a copy goes once its unread groups fall outside the reader's + budget. Decide what an unbounded budget keeps, for example the unread + backlog consolidated into one retained cache instead of one copy per swap, + rather than a count cap. +- Regression test: a datagram-only Rust reader across many swaps, each copy + holding an unread group, keeps retained copies bounded by its budget. + Regression tests in both languages, with mocked time: - The old copy writes F-1 only after the new copy handed out F and the reader From 3a062c10b449235577d9aba06f5f0a6f26b91575 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 13:37:35 -0700 Subject: [PATCH 26/31] quest: a resumed track's declared end and largest hold across copies Plan two #5836 review findings in resume-closed: a declared end regresses to unknown when a copy that hasn't learned it takes over, and `largest()` waits forever once a finite serving copy ends cleanly with the front still open. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 2 +- quest/m1/resume-closed.md | 62 ++++++++++++++++++++++++++++++--------- 2 files changed, 49 insertions(+), 15 deletions(-) diff --git a/quest/m1/README.md b/quest/m1/README.md index 66f3ee1fee0..69742483be8 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -109,7 +109,7 @@ lands before the cut. - [TS fidelity and recovery](/quest/m1/ts-reliability/README.md) - both modes have explicit profiles, automated validation, loss tests, and a dedicated guide - [Resume reorder](/quest/m1/resume-reorder.md) - a group whose stream header arrives after its successor's still resumes on failover; only a dropped or long-gone group fails - [Rust resumed reads](/quest/m1/resume-ordered.md) - a Rust reader takes the lowest group across route copies in both orders, as JS does -- [Resumed close](/quest/m1/resume-closed.md) - a resumed JS track settles `closed` on a clean end even when only one read channel is polled +- [Resumed close](/quest/m1/resume-closed.md) - a resumed JS track keeps its declared end across copies, and its `closed` and `largest()` settle on a clean end - [Resumed in-flight groups](/quest/m1/resume-in-flight.md) - a replaced route copy stays while a group below its bound is in flight or it backs a handed-out group late, and no longer than the reader's budget - [Resumed forks](/quest/m1/resume-fork.md) - a fork of a resumed JS subscriber follows route swaps like its parent - [Received timestamps are media time](/quest/m1/lite-time-provenance.md) - a received timed track's timestamps are media time, so the rewind check can run across hops diff --git a/quest/m1/resume-closed.md b/quest/m1/resume-closed.md index e031b8db701..aa6b7854063 100644 --- a/quest/m1/resume-closed.md +++ b/quest/m1/resume-closed.md @@ -1,17 +1,27 @@ -# [S] A resumed track closes without both channels drained +# [M] A resumed track's end holds across copies and settles every reader ## Goal A resumed JS track subscriber (`Track.Subscriber` reading a front's route -copies) settles `closed` once its logical track ends cleanly, whether the -application reads groups, datagrams, or both. A group-only reader that reaches -`recvGroup()` EOF on a finite track sees `closed` settle and releases its -`Reader`, copies, and route sources, while a reader of the other channel still -drains what is buffered. +copies) ends the way a direct one does: + +- It settles `closed` once its logical track ends cleanly, whether the + application reads groups, datagrams, or both. A group-only reader that reaches + `recvGroup()` EOF on a finite track sees `closed` settle and releases its + `Reader`, copies, and route sources, while a reader of the other channel still + drains what is buffered. +- A declared end, once any copy reported it, stays the logical track's end + across route swaps: `end()` and `finished()` never regress to unknown because + the new copy hasn't learned it yet, and a copy reporting a different end is + refused. +- `largest()` resolves `undefined` once the serving copy ends cleanly at its + declared end, even while the front stays open for other tracks. ## Plan -Found by a hosted Codex review of #5836 (`resume.ts` `Reader.#finish`). Today a +Found by hosted Codex reviews of #5836. + +**Closed with one channel polled** (`resume.ts` `Reader.#finish`). Today a clean end adds the channel that ran out to `#drained`, and the shell closes only once both `"groups"` and `"datagrams"` are there. Nothing adds `"datagrams"` for a reader that never calls `recvDatagram()`, so `closed` stays pending and the @@ -23,14 +33,38 @@ route keeps the reader. Errors already close at once. - Keep each channel's buffered backlog readable after that: one channel running out must not cut off the other's unread data, as `draining groups to their end leaves buffered datagrams readable` checks today. -- Check whether a direct (non-resumed) `Track.Subscriber` and the Rust - `resume::Subscriber` already settle this way, and match them. -- Regression test: a group-only reader of a finite resumed track sees `closed` - settle and the route release its copy. -Public API: behavior only (`closed` settles for a group-only resumed reader). -Wire: none. +**A declared end across swaps** (`Reader.end()` and `finished()`). Both read the +serving copy, so when copy A has declared the end and copy B takes over before +its subscription learns it, `end()` returns `undefined` and `finished()` waits +on B. Wire publishers can then advertise or wait on the wrong boundary. + +- Record the first end any copy declares on the reader and answer from it. + Check each later copy's end against it, as `resolveTrackInfo` checks track + properties, and refuse a mismatch. + +**Largest after a clean end** (`Reader.largest(seen)`). It returns only once +the route or shell concluded, or the route was refused. A finite serving copy +that reaches its declared end cleanly sets `view.ended` with `closed` `null`, +neither of which ends the wait, so `largest()` waits on a stream that has ended. + +- Treat a clean end of the serving copy as terminal there, while a copy that + failed with its route still waits for the next route. + +For all three, check whether a direct (non-resumed) `Track.Subscriber` and the +Rust `resume::Subscriber` already behave this way, and match them. + +Regression tests: + +- A group-only reader of a finite resumed track sees `closed` settle and the + route release its copy. +- A swap after copy A declared the end keeps `end()` at that end, and + `finished()` resolves with it before B learns it. +- `largest()` on a finite resumed track resolves `undefined` once its serving + copy ends cleanly, with the front still open. + +Public API: behavior only. Wire: none. ## Related -- [Rust resumed reads](/quest/m1/resume-ordered.md) - the other resume follow-up from #5836 +- [Rust resumed reads](/quest/m1/resume-ordered.md) - another resume follow-up from #5836 From fddf61d64f4bf909f9501fea9232295ebf4d6a3f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 14:03:15 -0700 Subject: [PATCH 27/31] quest: a resumed track's clean end waits for retained copies; recovered readiness skips replays Plan the last two #5836 review findings: a new copy's clean end before its first group cuts off a group the still-retained old copy may deliver, and a recovered group's `readable()` resolves on a replayed frame the read then skips. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 1 + quest/m1/resume-in-flight.md | 13 +++++++++++++ quest/m1/resume-readable.md | 28 ++++++++++++++++++++++++++++ 3 files changed, 42 insertions(+) create mode 100644 quest/m1/resume-readable.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 69742483be8..bf340b45d62 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -112,6 +112,7 @@ lands before the cut. - [Resumed close](/quest/m1/resume-closed.md) - a resumed JS track keeps its declared end across copies, and its `closed` and `largest()` settle on a clean end - [Resumed in-flight groups](/quest/m1/resume-in-flight.md) - a replaced route copy stays while a group below its bound is in flight or it backs a handed-out group late, and no longer than the reader's budget - [Resumed forks](/quest/m1/resume-fork.md) - a fork of a resumed JS subscriber follows route swaps like its parent +- [Recovered group readiness](/quest/m1/resume-readable.md) - a recovered JS group's `readable()` ignores frames its replacement replays - [Received timestamps are media time](/quest/m1/lite-time-provenance.md) - a received timed track's timestamps are media time, so the rewind check can run across hops - [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS - [HLS export drains after End](/quest/m1/hls-export-end-drain.md) - a cleanly ended broadcast still exports its retained segments diff --git a/quest/m1/resume-in-flight.md b/quest/m1/resume-in-flight.md index 6787e85b33e..81dbd84bda3 100644 --- a/quest/m1/resume-in-flight.md +++ b/quest/m1/resume-in-flight.md @@ -47,6 +47,17 @@ route, though an identical whole copy was there. keeping every replaced copy until the groups handed out above its bound end. Prefer whichever keeps one lease rule for both paths. +**A clean end with a copy still retained.** When the new copy B reaches clean +EOF before delivering any group, JS `Reader.#end` (in `js/net/src/resume.ts`) +and Rust `Reader::poll_end` treat the serving copy's clean end as the track's +end at once. The old copy A, still unbounded because B never delivered, may yet +deliver a final group, and that group is lost once `recvGroup()` has returned +`undefined`. + +- Defer the clean end until every retained copy has drained or been bounded, + under the same gap rule as above, so the end never cuts off a group a kept + copy still carries. + **Bounded retention.** A reader that never reads groups (datagrams only) keeps every bounded replaced copy that still has an unread group, since nothing ever reads it. Rust `Reader::let_go` does this on main too. A peer that keeps @@ -69,6 +80,8 @@ Regression tests in both languages, with mocked time: lets the copy go. - The old copy finishes F after the new copy handed F out and the reader polled on. The serving copy's F then fails, and the read continues from the old copy. +- The new copy ends cleanly before delivering a group, the old copy then writes + its last group, and the reader gets that group before the end. Public API: behavior only. Wire: none. diff --git a/quest/m1/resume-readable.md b/quest/m1/resume-readable.md new file mode 100644 index 00000000000..073b4222b62 --- /dev/null +++ b/quest/m1/resume-readable.md @@ -0,0 +1,28 @@ +# [S] A recovered group is readable only past its replayed frames + +## Goal + +`group.Consumer.readable()` on a JS group that adopted a replacement copy +resolves only once the next `readFrame()` would return without waiting. Frames +the replacement replays below where the reader already is never count as +readable. + +## Plan + +Found by a hosted Codex review of #5836. After a stalled group delivered frame +0 and adopted an open replacement that so far buffers only its replay of frame +0, `readable()` in `js/net/src/group.ts` sees a non-empty buffer and resolves. +The replay is dropped only later, in `#readBufferedFrame`'s skip below +`#skip`, so the next `readFrame()` still waits for frame 1. + +- Drop the frames below `#skip` in one place that both `readable()` and the + read path use before they look at the buffer, so readiness and the read agree. +- Regression test, in `group.test.ts` with the recovery hooks: adopt a + replacement holding only the replayed frame 0, and `readable()` stays + pending until frame 1 arrives. + +Public API: behavior only. Wire: none. + +## Related + +- [Resumed close](/quest/m1/resume-closed.md) - another resume follow-up from #5836 From 2d532ae0c8de9de2243270f8f986162e915fef10 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 14:41:43 -0700 Subject: [PATCH 28/31] quest: datagram reads before group reads keep a replaced copy's unread group Co-Authored-By: Claude Opus 5.5 --- quest/m1/resume-in-flight.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/quest/m1/resume-in-flight.md b/quest/m1/resume-in-flight.md index 81dbd84bda3..323459d99d0 100644 --- a/quest/m1/resume-in-flight.md +++ b/quest/m1/resume-in-flight.md @@ -73,6 +73,22 @@ copies longer, as above, must not make that worse. - Regression test: a datagram-only Rust reader across many swaps, each copy holding an unread group, keeps retained copies bounded by its budget. +**Datagrams read before groups.** JS errs the other way. The subscriber's +`#tryDatagram` in `js/net/src/track.ts` passes `delivered` to +`Reader.tryDatagram` only once `#groupsRead` is set, and `Reader.#letGo` +drops every replaced copy when `delivered` is undefined. A `recvDatagram()` +after a swap but before the first `recvGroup()` therefore lets the old copy go +with an unread group the new copy lacks, and that group is never delivered. +Rust keeps it (`datagram_reads_keep_unread_groups_on_a_replaced_copy`). + +- Pass `delivered` regardless of `#groupsRead`, so a datagram poll asks + `hasUnreadGroup` like a group poll does. The budget rule + above then bounds what a reader that never reads groups keeps, in both + languages alike. +- Regression test: port the Rust test to `js/net/src/resume.test.ts`, polling + `recvDatagram()` between the swap and the first `recvGroup()`, which must + return the old copy's group. + Regression tests in both languages, with mocked time: - The old copy writes F-1 only after the new copy handed out F and the reader From 23d31e94c691b97a43531ba29319f091b85a7b36 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 14:58:25 -0700 Subject: [PATCH 29/31] quest: a resumed subscriber's abort reaches its handed-out groups Co-Authored-By: Claude Opus 5.5 --- quest/m1/resume-closed.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/quest/m1/resume-closed.md b/quest/m1/resume-closed.md index aa6b7854063..b7073ab9b3c 100644 --- a/quest/m1/resume-closed.md +++ b/quest/m1/resume-closed.md @@ -16,6 +16,8 @@ copies) ends the way a direct one does: refused. - `largest()` resolves `undefined` once the serving copy ends cleanly at its declared end, even while the front stays open for other tracks. +- `close(abort)` reaches every group already handed out, so a pending frame + read rejects with the abort rather than ending cleanly. ## Plan @@ -51,7 +53,18 @@ neither of which ends the wait, so `largest()` waits on a stream that has ended. - Treat a clean end of the serving copy as terminal there, while a copy that failed with its route still waits for the next route. -For all three, check whether a direct (non-resumed) `Track.Subscriber` and the +**An abort reaches the copies** (`Reader.close()`). `Subscriber.close(abort)` +in `js/net/src/track.ts` calls `this.#resume?.close(abort)`, but +`Reader.close()` takes no argument and `#drop` clean-closes every copy. A group +already handed out is backed by a copy rather than the outer buffer, so its +pending frame read can end with clean EOF instead of the abort a direct +subscriber gives. + +- Take the abort in `Reader.close(abort?)` and close each copy with it, so a + handed-out group and its `Recover` end with the same error. A recovery in + flight must settle with the abort too, not adopt a replacement. + +For all four, check whether a direct (non-resumed) `Track.Subscriber` and the Rust `resume::Subscriber` already behave this way, and match them. Regression tests: @@ -62,6 +75,8 @@ Regression tests: `finished()` resolves with it before B learns it. - `largest()` on a finite resumed track resolves `undefined` once its serving copy ends cleanly, with the front still open. +- `close(abort)` on a resumed subscriber with a group handed out and a frame + read pending rejects that read with the abort. Public API: behavior only. Wire: none. From 2622b0fc35c41acbba16dcdcf0f65c10a8a2e78f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 15:05:14 -0700 Subject: [PATCH 30/31] fix(net): a failed group continued from a held copy lends it the head The branch's held() path continues a failed group from another copy before poll_serving runs, so #5958's lend_head never ran for it and that copy's cache missed the head the failed route received. Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/model/resume.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 7b95be1a52c..f325a0f6d59 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -1098,6 +1098,7 @@ impl Recover { return None; } let mut group = copy.track.peek_group(self.sequence)?; + lend_head(&self.handoff, self.sequence, ©.track, &group); group.start_at(index); (group.index() == index).then(|| Replacement { group, From 27be34e01977b6fe5d92c0cf47e9bec6d94f2d58 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 15:20:25 -0700 Subject: [PATCH 31/31] quest: a resumed track skips only datagrams it already delivered Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 1 + quest/m1/resume-datagrams.md | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+) create mode 100644 quest/m1/resume-datagrams.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 34c5eda6fb2..b2c4c0a1a9c 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -112,6 +112,7 @@ lands before the cut. - [Resumed in-flight groups](/quest/m1/resume-in-flight.md) - a replaced route copy stays while a group below its bound is in flight or it backs a handed-out group late, and no longer than the reader's budget - [Resumed forks](/quest/m1/resume-fork.md) - a fork of a resumed JS subscriber follows route swaps like its parent - [Recovered group readiness](/quest/m1/resume-readable.md) - a recovered JS group's `readable()` ignores frames its replacement replays +- [Resumed datagrams](/quest/m1/resume-datagrams.md) - a resumed track skips only datagrams it already delivered, not every lower sequence - [Received timestamps are media time](/quest/m1/lite-time-provenance.md) - a received timed track's timestamps are media time, so the rewind check can run across hops - [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS - [HLS export drains after End](/quest/m1/hls-export-end-drain.md) - a cleanly ended broadcast still exports its retained segments diff --git a/quest/m1/resume-datagrams.md b/quest/m1/resume-datagrams.md new file mode 100644 index 00000000000..dce17aafe8e --- /dev/null +++ b/quest/m1/resume-datagrams.md @@ -0,0 +1,32 @@ +# [S] A resumed track skips only datagrams it already delivered + +## Goal + +After a same-epoch route swap, a resumed track drops a datagram from the new +copy only if the reader already handed out that same datagram. A datagram +first emitted after the swap is delivered even when its sequence is below the +newest one handed out, since `insertDatagram` allows out-of-order sequences and +`recvDatagram()` delivers in arrival order. + +## Plan + +Found by a hosted Codex review of #5836. Each copy records the newest datagram +the reader handed out before that copy subscribed (JS `Copy.datagramsSeen` in +`js/net/src/resume.ts`, Rust `datagrams_seen` in +`rs/moq-net/src/model/resume.rs`), and JS `Reader.tryDatagram` and Rust +`Reader::poll_datagram_once` drop everything at or below it as a replay. A relay forwarding datagram 5 after 10 +therefore loses 5 on the new route, though the old route would have delivered +it. + +- Skip by identity: record which datagram sequences the reader delivered, as + `Delivered` does for groups, and drop only those. Prune the record with the + reader's budget so it stays bounded, rather than with a count cap. +- Apply the same rule in both languages, so the JS and Rust readers agree. +- Regression test in both: route A delivers datagram 10, route B takes over and + delivers a replay of 10 and a new 5. The reader gets 5 once, and 10 only once. + +Public API: behavior only. Wire: none. + +## Related + +- [Resumed in-flight groups](/quest/m1/resume-in-flight.md) - the group side of what a replaced copy still delivers