From 8686b13f9309769671978d140af567d92d8d95d6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 25 Oct 2026 20:44:25 -0700 Subject: [PATCH 1/4] quest: plan the studio's MoQ-native and OBS-polish backlog Adds the viewer-side composition epic (source tracks, a layout track, and browser, server, and studio hosts), the scene model and undo, live encode control, health, failover, replay, HTML/CSS overlays through Blitz, animation, effects, snapping, and guided setup. Folds per-app audio into the capture quests and extends mixes and audio filters. Co-Authored-By: Claude Opus 5.5 --- quest/m2/README.md | 5 +-- quest/m2/capture-app-audio-macos.md | 20 ++++++++++++ quest/m2/capture-linux.md | 5 +++ quest/m2/capture-windows.md | 4 +++ quest/m3/studio/README.md | 11 +++++++ quest/m3/studio/animation.md | 23 +++++++++++++ quest/m3/studio/audio-filters.md | 8 +++-- quest/m3/studio/compose/README.md | 50 +++++++++++++++++++++++++++++ quest/m3/studio/compose/browser.md | 25 +++++++++++++++ quest/m3/studio/compose/controls.md | 20 ++++++++++++ quest/m3/studio/compose/layout.md | 21 ++++++++++++ quest/m3/studio/compose/server.md | 20 ++++++++++++ quest/m3/studio/compose/sources.md | 18 +++++++++++ quest/m3/studio/effects.md | 24 ++++++++++++++ quest/m3/studio/encode-control.md | 26 +++++++++++++++ quest/m3/studio/failover.md | 19 +++++++++++ quest/m3/studio/guided-setup.md | 17 ++++++++++ quest/m3/studio/health.md | 21 ++++++++++++ quest/m3/studio/mixes.md | 4 ++- quest/m3/studio/overlays.md | 39 ++++++++++++++++++++++ quest/m3/studio/replay.md | 23 +++++++++++++ quest/m3/studio/scene-model.md | 26 +++++++++++++++ quest/m3/studio/snapping.md | 18 +++++++++++ 23 files changed, 442 insertions(+), 5 deletions(-) create mode 100644 quest/m2/capture-app-audio-macos.md create mode 100644 quest/m3/studio/animation.md create mode 100644 quest/m3/studio/compose/README.md create mode 100644 quest/m3/studio/compose/browser.md create mode 100644 quest/m3/studio/compose/controls.md create mode 100644 quest/m3/studio/compose/layout.md create mode 100644 quest/m3/studio/compose/server.md create mode 100644 quest/m3/studio/compose/sources.md create mode 100644 quest/m3/studio/effects.md create mode 100644 quest/m3/studio/encode-control.md create mode 100644 quest/m3/studio/failover.md create mode 100644 quest/m3/studio/guided-setup.md create mode 100644 quest/m3/studio/health.md create mode 100644 quest/m3/studio/overlays.md create mode 100644 quest/m3/studio/replay.md create mode 100644 quest/m3/studio/scene-model.md create mode 100644 quest/m3/studio/snapping.md diff --git a/quest/m2/README.md b/quest/m2/README.md index 238937e5305..623c031b3c6 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -179,8 +179,9 @@ lite-07; bitrate enforcement is not part of this release. - [Keep-alive from the idle timeout](/quest/m2/quic-keep-alive.md) - the keep-alive interval defaults to a fraction of the negotiated idle timeout; an explicit `quic.keep_alive` overrides - [Socket close](/quest/m2/quic-socket-close.md) - moq-tokio's QUIC endpoint releases its socket on close, so it drops its wrapper - [Windows.Graphics.Capture](/quest/m2/capture-wgc.md) - the WGC display and window backend verified on real Windows hardware -- [Windows capture parity](/quest/m2/capture-windows.md) - system audio and a settled app-capture policy -- [Linux capture parity](/quest/m2/capture-linux.md) - Wayland window/system-audio capture with explicit display-selection and app-capture limits +- [Windows capture parity](/quest/m2/capture-windows.md) - system and per-app audio and a settled app-capture policy +- [Linux capture parity](/quest/m2/capture-linux.md) - Wayland window/system-audio capture, per-app audio, and explicit display-selection and app-capture limits +- [Per-app audio on macOS](/quest/m2/capture-app-audio-macos.md) - one application's audio through ScreenCaptureKit - [cpal loads libasound at runtime](/quest/m2/cpal-alsa-runtime.md) - condition: a cpal release whose Linux build carries no load-time libasound requirement - [Audio capture without ALSA link](/quest/m2/capture-alsa-link.md) - moq-audio capture and playback build on Linux without linking libasound, and capture becomes a default feature - [Ship capture and playback](/quest/m2/cli-packaging.md) - a released moq binary can capture and play, which no distribution currently enables diff --git a/quest/m2/capture-app-audio-macos.md b/quest/m2/capture-app-audio-macos.md new file mode 100644 index 00000000000..eb34170ea77 --- /dev/null +++ b/quest/m2/capture-app-audio-macos.md @@ -0,0 +1,20 @@ +# [S] Per-app audio on macOS + +## Goal + +`moq_audio::capture` captures one application's audio on macOS, so a creator +can stream a game without their music. The Linux and Windows halves live in +their capture parity quests. + +## Plan + +Decided 2026-10-26 (maintainer). ScreenCaptureKit already delivers system +audio here (`Source::System`), and an `SCContentFilter` that includes only one +application's windows limits the audio to that app. Apps are listed from +`SCShareableContent`, so only a running app can be chosen. It needs the +Screen Recording permission, like system audio. + +## Related + +- [Linux capture parity](/quest/m2/capture-linux.md) - per-app audio through PipeWire nodes +- [Windows capture parity](/quest/m2/capture-windows.md) - per-app audio through WASAPI process loopback diff --git a/quest/m2/capture-linux.md b/quest/m2/capture-linux.md index 09c8b56d67c..d267359f854 100644 --- a/quest/m2/capture-linux.md +++ b/quest/m2/capture-linux.md @@ -31,6 +31,11 @@ unimplemented, because xdg-desktop-portal owns source selection: - **Applications** have no portal concept behind them, so decide explicitly whether Linux offers app capture rather than leaving a variant that errors at runtime. +- **Per-app audio** (decided 2026-10-26, maintainer: yes) captures one + application's output from its PipeWire stream node, so a creator can stream + a game without their music. Apps are listed from the nodes present, so only + a running app can be chosen. The Windows and macOS halves share the same + `moq_audio::capture` source. ## Related diff --git a/quest/m2/capture-windows.md b/quest/m2/capture-windows.md index 5e3f4a14043..19e20618b74 100644 --- a/quest/m2/capture-windows.md +++ b/quest/m2/capture-windows.md @@ -17,6 +17,10 @@ including the cursor, move to exists but is macOS-only and returns `Unsupported` elsewhere. Unlike macOS this does not go through the screen-capture API, so it is an independent path and does not inherit the Screen Recording permission coupling. +- **Per-app audio** (decided 2026-10-26, maintainer: yes) uses WASAPI process + loopback (`AUDIOCLIENT_ACTIVATION_TYPE_PROCESS_LOOPBACK`) to capture one + process tree's output, through the same `moq_audio::capture` source as the + Linux and macOS halves. WGC does not answer app capture. It captures one monitor or one window per item and has no per-process source, so the per-window-composition or diff --git a/quest/m3/studio/README.md b/quest/m3/studio/README.md index 4b7de4afbee..e1e17b82fdc 100644 --- a/quest/m3/studio/README.md +++ b/quest/m3/studio/README.md @@ -70,6 +70,17 @@ surface and zero-copy texture import, land before the cut. - [Zero-copy program encode](/quest/m1/studio-gpu/encode.md) - the program reaches the encoder as a GPU surface instead of a CPU readback - [Zero-copy source import](/quest/m1/studio-gpu/import.md) - captured and decoded frames reach the compositor as GPU textures instead of a CPU upload - [Studio on macOS and Windows](/quest/m3/studio/app.md) - the studio runs on macOS and Windows, not only Linux +- [Viewer-side composition](/quest/m3/studio/compose/README.md) - sources as their own tracks plus a layout track, composed by viewers, with a server-composed program as the fallback +- [Undo and the scene model](/quest/m3/studio/scene-model.md) - undo history, linked and cloned sources, copy and paste, a downstream keyer, and per-item filters +- [Live encode control](/quest/m3/studio/encode-control.md) - simulcast rungs that follow the bandwidth grant, live bitrate and resolution, keyframes on request +- [Stream health](/quest/m3/studio/health.md) - per-track delivery, viewer reports, and latency, with causes named +- [Relay failover](/quest/m3/studio/failover.md) - the same epoch on two relays, so a relay failure does not drop the stream +- [Instant replay and DVR](/quest/m3/studio/replay.md) - replays and clips from held groups, and viewer scrubbing through the archive +- [HTML/CSS overlays](/quest/m3/studio/overlays.md) - script-free HTML and CSS graphics through Blitz natively and the DOM in the browser +- [Animation and transitions](/quest/m3/studio/animation.md) - move, slide, wipe, and stinger transitions, and keyframed item properties +- [Video effects](/quest/m3/studio/effects.md) - a curated effect set with CSS meanings, and ML effects baked into their source +- [Snapping and guides](/quest/m3/studio/snapping.md) - preview moves snap to edges, centers, and other items +- [Guided setup](/quest/m3/studio/guided-setup.md) - first launch detects devices, measures the uplink, and proposes settings - [Destinations](/quest/m3/studio/destinations.md) - RTMP, RTMPS, and WHIP push with per-destination codecs and shared encoders, on a UI-free egress library a hosted service can reuse - [Multiple canvases](/quest/m3/studio/multi-canvas.md) - landscape and portrait canvases composite at once from shared sources - [Guests](/quest/m3/studio/guests.md) - remote guests and phone cameras join through a moq-room link or QR code and appear as sources diff --git a/quest/m3/studio/animation.md b/quest/m3/studio/animation.md new file mode 100644 index 00000000000..36fdbe99a7d --- /dev/null +++ b/quest/m3/studio/animation.md @@ -0,0 +1,23 @@ +# [M] Animation and transitions + +## Goal + +Scene changes in `moq-studio` use transitions beyond a cut and fade: move +(items glide between their positions in the two scenes), slide, wipe, and +stinger (a video clip with a cut point). Any item property can be keyframed +with easing, so an item can fly in or a camera can push in, and the same +animation plays in every composition host. + +## Plan + +Move is OBS's most-installed plugin (Move Transition), and stingers are +standard in broadcast. Animate in moq-scene on the media clock and carry the +transition and keyframes in the layout track, so viewers compose them too. +Overlays animate through their own CSS. No Lottie runtime for now (decided +2026-10-26): the open ones either lack text (velato) or add a C++ build +(ThorVG). A stinger clip is a media source like any other. + +## Required + +- [Undo and the scene model](/quest/m3/studio/scene-model.md) - keyframes are scene model state +- [Layout track](/quest/m3/studio/compose/layout.md) - where transitions and keyframes travel diff --git a/quest/m3/studio/audio-filters.md b/quest/m3/studio/audio-filters.md index 99a5e7ed53c..e4f4af73b97 100644 --- a/quest/m3/studio/audio-filters.md +++ b/quest/m3/studio/audio-filters.md @@ -4,10 +4,14 @@ Each audio source has an ordered processing chain in moq-audio: gain, noise suppression, noise gate, compressor, and limiter. Each filter is configured -from the studio and saved with the scene collection. +from the studio and saved with the scene collection. Ducking lowers one source +(music) when another (the microphone) is active. Third-party VST3 and CLAP +plugins load into the chain. ## Plan Prefer maintained crates, such as `nnnoiseless` for RNNoise suppression, over hand-rolled DSP. Benchmark the chain per source so its cost per added source -shows up as a slope. +shows up as a slope. Host plugins out of process, so a crashing or hung plugin +drops that filter rather than the audio bus, and use a maintained host crate +(such as `clack` for CLAP) rather than hand-rolling the plugin ABIs. diff --git a/quest/m3/studio/compose/README.md b/quest/m3/studio/compose/README.md new file mode 100644 index 00000000000..fe5d91f52f5 --- /dev/null +++ b/quest/m3/studio/compose/README.md @@ -0,0 +1,50 @@ +# Viewer-side composition + +## Goal + +A `moq-studio` broadcast carries each source as its own track plus a layout +track describing the live scene. Viewers compose the picture themselves: +they prioritize sources and toggle elements on and off. A composited program +remains for protocols and devices that cannot compose (RTMP and WHIP +destinations, HLS, legacy players), produced by the studio or by a server. + +This README's own work is the end-to-end test: a studio broadcast is composed +by the browser, by `moq compose`, and by the studio itself, and all three +pictures match. + +## Plan + +Decided 2026-10-26 (maintainer), the studio's flagship and first in its epic: + +- **Delivery:** MoQ's value is per-track delivery, so the studio embraces + viewer composition. The program is the fallback, not the default. +- **One compositor, three hosts:** moq-scene renders in the studio, headless + on a server (`moq compose` in moq-cli), and in the browser, so all three + pictures come from one implementation. +- **Parity rule:** every scene feature (transitions, filters, effects, + overlays) renders in all three hosts with the same layout, timing, and + colors, within a tolerance. Each renderer keeps its own text antialiasing + and blur, so the browser can draw overlays as real DOM. A feature that + cannot meet this, such as a CEF browser source or an ML effect, is baked + into its source's own track before publishing rather than living in the + layout. +- **Wire layout:** local sources are tracks in the studio's one broadcast. + Guests stay their own broadcasts, named through + [contribution references](/quest/m2/contribution-ref.md). +- **Layout:** the live scene state (items, transforms, transitions, and each + element's priority and toggleable flag) as a hang JSON track on the media + clock, so any host composites the same frame the studio shows. +- **Browser build:** whichever line [Rust-backed @moq/net](/quest/m2/rs2ts/README.md) + lands (WASM first, then JS built from Rust). Not fixed to WASM. +- **Cost:** every source and the program follow MoQ's pull model and are + encoded only while something subscribes, so a fallback nobody uses costs + nothing. + +## Required + +- [Source tracks](/quest/m3/studio/compose/sources.md) - each source publishes as its own track in the studio broadcast, encoded only while subscribed +- [Layout track](/quest/m3/studio/compose/layout.md) - the live scene state as a hang JSON track on the media clock, specified in the hang draft +- [Browser composition](/quest/m3/studio/compose/browser.md) - `@moq/watch` composes a broadcast with a layout track through moq-scene's browser build +- [Server composition](/quest/m3/studio/compose/server.md) - `moq compose` renders the program from sources and layout, headless +- [Viewer controls](/quest/m3/studio/compose/controls.md) - viewers toggle elements and the layout's priorities steer what they fetch +- [Hang contribution references](/quest/m2/contribution-ref.md) - guests composed by reference diff --git a/quest/m3/studio/compose/browser.md b/quest/m3/studio/compose/browser.md new file mode 100644 index 00000000000..1e3e32c4112 --- /dev/null +++ b/quest/m3/studio/compose/browser.md @@ -0,0 +1,25 @@ +# [L] Browser composition + +## Goal + +`@moq/watch` composes a broadcast that carries a layout track: it fetches the +tracks the layout shows and draws them through moq-scene's browser build, +matching the studio's picture. A broadcast without one plays its program as +today, so embedders change nothing. + +## Plan + +The browser build comes from whichever line +[Rust-backed @moq/net](/quest/m2/rs2ts/README.md) lands (decided 2026-10-26), +WASM first and JS built from Rust later, so this quest does not fix the +mechanism. A thin package (e.g. `@moq/scene`) wraps it and `@moq/watch` uses +it. Frames reach it as WebCodecs `VideoFrame`s through the same browser import +path the [web studio](/quest/m3/studio/web/video-web.md) adds. + +Prove parity with a fixture: the same broadcast composited here and by the +studio, compared within a stated tolerance. + +## Required + +- [Layout track](/quest/m3/studio/compose/layout.md) - what the browser composes +- [moq-video in the browser](/quest/m3/studio/web/video-web.md) - browser frames into wgpu diff --git a/quest/m3/studio/compose/controls.md b/quest/m3/studio/compose/controls.md new file mode 100644 index 00000000000..6eecca38dab --- /dev/null +++ b/quest/m3/studio/compose/controls.md @@ -0,0 +1,20 @@ +# [M] Viewer controls + +## Goal + +A viewer composing a studio broadcast can switch off any element the layout +marks toggleable (a facecam, a chat overlay), and stops fetching its track. +The layout's per-element priorities steer delivery, so under congestion the +least important sources degrade first. The creator sets both per element in +the studio. + +## Plan + +Map element priority onto the subscription priorities moq-lite already has +(`doc/concept/moq-lite.md`), rather than a second mechanism. A toggled-off +element is unsubscribed, so it costs the viewer and the relay nothing, and +the pull model may stop its encode. + +## Required + +- [Browser composition](/quest/m3/studio/compose/browser.md) - the viewer that exposes the toggles diff --git a/quest/m3/studio/compose/layout.md b/quest/m3/studio/compose/layout.md new file mode 100644 index 00000000000..40fce8833a2 --- /dev/null +++ b/quest/m3/studio/compose/layout.md @@ -0,0 +1,21 @@ +# [M] Layout track + +## Goal + +A studio broadcast carries a layout track: moq-scene's live scene state +(items, the tracks they show, transforms, transitions, and each element's +priority and whether a viewer may toggle it) as a hang JSON track on the +media clock. Any host that reads it composites the same frame the studio +shows. The hang catalog and `drafts/draft-lcurley-moq-hang.md` specify it. + +## Plan + +Serialize moq-scene's own model rather than a second layout schema, so the +studio, server, and browser cannot drift. Version it and refuse an unknown +version loudly. Keep it free of local paths and credentials: items name +tracks and contribution references, never files. A late joiner needs the +current state, so use the JSON track's snapshot mode. + +## Required + +- [Source tracks](/quest/m3/studio/compose/sources.md) - the tracks a layout names diff --git a/quest/m3/studio/compose/server.md b/quest/m3/studio/compose/server.md new file mode 100644 index 00000000000..e4bf1b60e17 --- /dev/null +++ b/quest/m3/studio/compose/server.md @@ -0,0 +1,20 @@ +# [M] Server composition + +## Goal + +`moq compose` in moq-cli subscribes to a studio broadcast's sources and +layout track and publishes the composited program, headless (a GPU, or +lavapipe without one). It can push that program to RTMP and WHIP through the +studio's egress library, so a hosted service makes the fallback composite and +the creator saves the upload. + +## Plan + +Decided 2026-10-26 (maintainer): a moq-cli subcommand, not a mode of +`moq transcode`. A hosted service embeds the same library. It publishes its +own broadcast, never the studio's names: each publisher produces its own +stream (AGENTS.md). Update `doc/bin/cli.md`. + +## Required + +- [Layout track](/quest/m3/studio/compose/layout.md) - what the server composes diff --git a/quest/m3/studio/compose/sources.md b/quest/m3/studio/compose/sources.md new file mode 100644 index 00000000000..b8d1055b905 --- /dev/null +++ b/quest/m3/studio/compose/sources.md @@ -0,0 +1,18 @@ +# [M] Source tracks + +## Goal + +Each local source in a `moq-studio` scene (camera, display, window, image, +microphone) publishes as its own track in the studio's broadcast, listed in +its catalog, alongside the program. Each is encoded only while something +subscribes to it. + +## Plan + +Follow the pull model the program already uses: an unwatched source costs +no encode and no upload. Static sources (images, colors) publish once rather +than as a video stream. Consumer GPUs cap concurrent hardware encode +sessions, so when the hardware refuses one, that source's track fails loudly +with the reason shown rather than silently degrading. +Track names stay stable for a source's life, and an edited source that +changes what it shows gets a new name. diff --git a/quest/m3/studio/effects.md b/quest/m3/studio/effects.md new file mode 100644 index 00000000000..0b57ef79b60 --- /dev/null +++ b/quest/m3/studio/effects.md @@ -0,0 +1,24 @@ +# [M] Video effects + +## Goal + +Sources and scene items in `moq-studio` take a curated set of video effects: +color correction, `.cube` LUTs, blur, crop and masks, chroma and luma key, +and blend modes. They render the same in every composition host. ML effects +such as background removal are offered too, applied before the source +publishes. + +## Plan + +Write the effects in WGSL to the meanings the CSS Filter Effects and +Compositing specs define (color matrices, blur, drop-shadow, blend modes), so +an effect means the same thing as the CSS overlays use. The layout track +carries effect names and parameters, never shader code, since a shader from +the network costs every viewer's GPU and can lose the device. An effect the +browser cannot reproduce, like an ML model, is baked into its source's track +per the [composition epic](/quest/m3/studio/compose/README.md). Prefer an +existing model crate for ML effects over training one. + +## Required + +- [Undo and the scene model](/quest/m3/studio/scene-model.md) - per-item filters this extends diff --git a/quest/m3/studio/encode-control.md b/quest/m3/studio/encode-control.md new file mode 100644 index 00000000000..4ff102202cd --- /dev/null +++ b/quest/m3/studio/encode-control.md @@ -0,0 +1,26 @@ +# [L] Live encode control + +## Goal + +`moq-studio` adapts its encode live without a restart: the program and each +source publish simulcast rungs that follow the relay's bandwidth grant, a rung +nobody watches stops encoding, and bitrate and resolution change mid-stream +through catalog updates. OBS's multitrack output does +part of this only on Windows with specific GPUs; this works on every +platform. + +## Plan + +Today the studio publishes one H.264 rendition at fixed settings and never +reads `bandwidth::Allocator`. Reuse what exists rather than adding a studio +path: the grant follower in `moq_video::encode` and `moq_mux::rate`, the +catalog's in-place updates and new renditions (`doc/concept/hang.md`). The +encode producer already cuts a keyframe when a subscriber joins +(`encode/cuts.rs`), so each rung inherits that and can run a long GOP. Show +each rung's grant, bitrate, and viewers in the Output panel. + +## Related + +- [Simulcast rung disable](/quest/m1/simulcast-rung-disable.md) - grant-driven rung disable this builds on +- [moq-audio follows the grant](/quest/m2/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md) - the audio half +- [Scene editing](/quest/m3/studio/scene-editing.md) - canvas resize through a new rendition diff --git a/quest/m3/studio/failover.md b/quest/m3/studio/failover.md new file mode 100644 index 00000000000..3d2eec81828 --- /dev/null +++ b/quest/m3/studio/failover.md @@ -0,0 +1,19 @@ +# [M] Relay failover + +## Goal + +`moq-studio` publishes the same broadcast epoch to two relays at once, so +when one relay or the path to it fails, viewers resume from the other +mid-group and the stream does not drop. + +## Plan + +Publisher epochs already support this: the same epoch on several relays +resumes mid-group, and a restart is a newer epoch (`doc/concept/moq-lite.md`, +Publisher epochs and Multiple CDNs). The studio dials one relay today. Encode +once and publish to both. The second relay is configured, never inferred. Show +each relay's state in the health panel. + +## Related + +- [Stream health](/quest/m3/studio/health.md) - where each relay's state shows diff --git a/quest/m3/studio/guided-setup.md b/quest/m3/studio/guided-setup.md new file mode 100644 index 00000000000..bb7b893fecc --- /dev/null +++ b/quest/m3/studio/guided-setup.md @@ -0,0 +1,17 @@ +# [M] Guided setup + +## Goal + +The first launch of `moq-studio` gets a creator live without reading +settings: it detects cameras, microphones, and displays, measures the uplink, +probes which encoders work on this machine, and proposes a scene and output +settings in plain language. Every setting it chooses can be changed later and +explains itself. + +## Plan + +Twitch Studio did this before it was shut down, and "settings are too +complicated" is a top OBS complaint. Reuse the probes that already exist +(the encoder choice `moq_video::render::export` makes at setup, the bandwidth +grant) rather than a separate benchmark. A probe that fails is reported, not +hidden. diff --git a/quest/m3/studio/health.md b/quest/m3/studio/health.md new file mode 100644 index 00000000000..11e50e6a1a9 --- /dev/null +++ b/quest/m3/studio/health.md @@ -0,0 +1,21 @@ +# [M] Stream health + +## Goal + +`moq-studio` shows how its stream is actually delivered, per track: viewers, +the bandwidth granted, what viewers report receiving and dropping, and +end-to-end latency, rather than only encoder counters. A problem names its +cause in plain words (the uplink is congested, a source stalled, the encoder +is behind). + +## Plan + +Read what the stack already publishes: the catalog's `stats` and `echo` +entries and `hang::{stats,echo}::Snapshot` (`doc/concept/stats.md`), and the +relay's stats broadcasts. OBS reports "0 dropped frames" while the bitrate sits +at zero, which is the confusion to avoid. + +## Related + +- [Stats](/quest/m2/stats/README.md) - the stats tracks this reads +- [Latency ledger](/quest/m2/latency-ledger.md) - per-stage playback delay a viewer reports diff --git a/quest/m3/studio/mixes.md b/quest/m3/studio/mixes.md index a9aef7338ba..d3da7dca72f 100644 --- a/quest/m3/studio/mixes.md +++ b/quest/m3/studio/mixes.md @@ -6,7 +6,9 @@ The program bus in moq-audio feeds several mixes. For example, a stream mix excludes the creator's music, a recording mix keeps it, and a monitor mix plays to headphones. Each source routes to any subset of mixes, and each output or recording picks a mix. Every source and mix shows a level meter. -Per-source sync offsets fix cameras that lag their microphones. +Per-source sync offsets fix cameras that lag their microphones, and a clap +or flash in front of the camera measures the offset instead of guessing it. +Each guest hears a mix-minus: the program without their own voice. ## Plan diff --git a/quest/m3/studio/overlays.md b/quest/m3/studio/overlays.md new file mode 100644 index 00000000000..318aa06c7c1 --- /dev/null +++ b/quest/m3/studio/overlays.md @@ -0,0 +1,39 @@ +# [L] HTML/CSS overlays + +## Goal + +Titles, lower thirds, tickers, and other graphics in `moq-studio` are written +as HTML and CSS without JavaScript, travel in the layout track, and render in +all three composition hosts: through Blitz in the studio and `moq compose`, +and as real DOM over the video in the browser. CSS animations and transitions +run on the media clock, so an overlay animates on the same frame everywhere. + +## Plan + +Decided 2026-10-26 (maintainer): + +- **Format:** HTML and CSS, no JavaScript, so creators author in a format they + know and nothing executes in a viewer. +- **Native renderer:** Blitz (Stylo, Taffy, Parley) drawing through Vello into + the studio's wgpu device, with its animation clock driven by the media + clock. Servo was passed over: it outputs OpenGL rather than wgpu, bundles a + JavaScript engine, and has no clock injection. +- **Browser:** the DOM layered over the composited video, so no HTML-in-Canvas + and it works in every browser today. +- **Subset:** the layout track accepts only the CSS Blitz renders and refuses + the rest at setup with the property named. Blitz is beta (no blend modes or + `text-shadow`, filters limited to blur and drop-shadow), and the subset + grows as it does. +- **Parity:** layout parity per the [composition epic](/quest/m3/studio/compose/README.md); + the fixture compares boxes and timing, and pixels only loosely. + +An overlay is hostile input to every viewer, so it fetches nothing from the +network: fonts and images travel with the broadcast or are bundled. First check +whether Blitz can draw into a caller's wgpu texture without a readback, and +upstream that if it cannot. Arbitrary web pages and JavaScript graphics stay +[browser sources](/quest/m3/studio/browser-source.md), baked into their own +track. + +## Required + +- [Layout track](/quest/m3/studio/compose/layout.md) - where overlays travel diff --git a/quest/m3/studio/replay.md b/quest/m3/studio/replay.md new file mode 100644 index 00000000000..78d1f24f155 --- /dev/null +++ b/quest/m3/studio/replay.md @@ -0,0 +1,23 @@ +# [M] Instant replay and DVR + +## Goal + +`moq-studio` replays the last seconds of any source or the program on demand: +several lengths, slow motion, and a one-click clip saved to disk, with no +separate always-on recording. Viewers can scrub back through the live +broadcast as far as the relay or archive holds it. + +## Plan + +The groups are already on hand: the studio's own recent groups and the +relay's cache, addressed by presentation time through subscription ranges +(`doc/concept/moq-lite.md`). A replay plays from those rather than a RAM-only +frame buffer like OBS's. Retention past the relay cache is the archive's job. +Replay published to viewers is a new source in the scene, so it composes like +any other. + +## Related + +- [Archive time ranges](/quest/m2/archive-time-ranges.md) - ranges reaching past the cache +- [Archive DVR](/quest/m2/archive-dvr.md) - viewer scrubbing beyond the relay cache +- [Recording](/quest/m3/studio/recording.md) - clips use the same file writer diff --git a/quest/m3/studio/scene-model.md b/quest/m3/studio/scene-model.md new file mode 100644 index 00000000000..545c89f2fe3 --- /dev/null +++ b/quest/m3/studio/scene-model.md @@ -0,0 +1,26 @@ +# [L] Undo and the scene model + +## Goal + +Every edit in `moq-studio` can be undone and redone from a history panel, and +the history survives autosave and restarts. The scene model distinguishes a +linked source (one capture shown in several scenes) from a clone with its own +filters, items and filters copy and paste between scenes and collections, +each canvas has a top layer that stays on across scene switches (a +downstream keyer), and filters can be set per scene item rather than only per +source. + +## Plan + +These are the most-voted OBS gaps and the most-installed plugins (Source +Clone, Source Copy, Downstream Keyer), and they all shape moq-scene's data +model, which the [layout track](/quest/m3/studio/compose/layout.md) also +serializes. So land them right after viewer composition, before more features +build on the model. Route every edit through one command type so undo, +autosave, and the layout track see the same changes. Update +`doc/bin/studio.md`. + +## Related + +- [Autosave](/quest/m3/studio/autosave.md) - the history must survive it +- [Scene editing](/quest/m3/studio/scene-editing.md) - rename, reorder, and resize go through the same commands diff --git a/quest/m3/studio/snapping.md b/quest/m3/studio/snapping.md new file mode 100644 index 00000000000..b64c9bd938e --- /dev/null +++ b/quest/m3/studio/snapping.md @@ -0,0 +1,18 @@ +# [S] Snapping and guides + +## Goal + +Moving and resizing items in `moq-studio`'s preview snaps to the canvas +edges and center, to other items' edges and centers, and to an optional grid, +with smart guides and equal-spacing hints drawn while dragging. A modifier key +turns snapping off. + +## Plan + +OBS only now adds axis snapping. Snapping is a preview interaction, so it +lives in the studio, not in moq-scene or the layout track. Moves still go +through the scene model's commands so they undo. + +## Related + +- [Undo and the scene model](/quest/m3/studio/scene-model.md) - snapped moves are ordinary commands From 685ae7ebc4b307da233524c78f96e200b37cffb0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 25 Oct 2026 23:08:07 -0700 Subject: [PATCH 2/4] quest: arm replay as a local subscriber so unwatched tracks still replay Co-Authored-By: Claude Opus 5.5 --- quest/m3/studio/replay.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/quest/m3/studio/replay.md b/quest/m3/studio/replay.md index 78d1f24f155..a0a0fcc6d86 100644 --- a/quest/m3/studio/replay.md +++ b/quest/m3/studio/replay.md @@ -13,6 +13,16 @@ The groups are already on hand: the studio's own recent groups and the relay's cache, addressed by presentation time through subscription ranges (`doc/concept/moq-lite.md`). A replay plays from those rather than a RAM-only frame buffer like OBS's. Retention past the relay cache is the archive's job. + +Sources and the program encode only while subscribed +([source tracks](/quest/m3/studio/compose/sources.md)), so a track nobody +watched has no groups to replay. Arming replay for a source or the program +is a local subscriber holding a retention window, so that track keeps +encoding with no external subscriber, and its encode cost shows like any +other subscriber's. An unarmed track replays only what was encoded, and the +gaps show rather than being skipped. Acceptance covers the case where every +viewer composes sources themselves and nothing outside the studio subscribes +to the program. Replay published to viewers is a new source in the scene, so it composes like any other. From d6a9e56630b49610a2b54e948ee4b4694512c9f4 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 25 Oct 2026 23:15:26 -0700 Subject: [PATCH 3/4] quest: settle the review's gaps in the studio backlog plans Co-Authored-By: Claude Opus 5.5 --- quest/m2/capture-app-audio-macos.md | 4 +++- quest/m2/capture-linux.md | 7 +++++-- quest/m2/capture-windows.md | 5 ++++- quest/m3/studio/audio-filters.md | 5 ++++- quest/m3/studio/compose/controls.md | 4 +++- quest/m3/studio/compose/layout.md | 3 ++- quest/m3/studio/effects.md | 3 ++- quest/m3/studio/failover.md | 5 ++++- quest/m3/studio/guided-setup.md | 6 ++++-- quest/m3/studio/mixes.md | 3 ++- quest/m3/studio/overlays.md | 5 ++++- quest/m3/studio/scene-model.md | 4 +++- quest/m3/studio/snapping.md | 4 ++-- 13 files changed, 42 insertions(+), 16 deletions(-) diff --git a/quest/m2/capture-app-audio-macos.md b/quest/m2/capture-app-audio-macos.md index eb34170ea77..4195fb855da 100644 --- a/quest/m2/capture-app-audio-macos.md +++ b/quest/m2/capture-app-audio-macos.md @@ -12,7 +12,9 @@ Decided 2026-10-26 (maintainer). ScreenCaptureKit already delivers system audio here (`Source::System`), and an `SCContentFilter` that includes only one application's windows limits the audio to that app. Apps are listed from `SCShareableContent`, so only a running app can be chosen. It needs the -Screen Recording permission, like system audio. +Screen Recording permission, like system audio. Per-app audio is its own +source variant naming the application; `Source::System` keeps capturing all +desktop audio through its display filter. ## Related diff --git a/quest/m2/capture-linux.md b/quest/m2/capture-linux.md index 59357185876..d1dd50e9555 100644 --- a/quest/m2/capture-linux.md +++ b/quest/m2/capture-linux.md @@ -35,8 +35,11 @@ unimplemented, because xdg-desktop-portal owns source selection: - **Per-app audio** (decided 2026-10-26, maintainer: yes) captures one application's output from its PipeWire stream node, so a creator can stream a game without their music. Apps are listed from the nodes present, so only - a running app can be chosen. The Windows and macOS halves share the same - `moq_audio::capture` source. + a running app can be chosen. An app is every stream node with its + `application.process.id` (one game may open several), mixed together, and + capture follows nodes as the app opens and closes them, matching the + Windows process tree and the macOS application. The Windows and macOS halves + share the same `moq_audio::capture` source. ## Related diff --git a/quest/m2/capture-windows.md b/quest/m2/capture-windows.md index 19e20618b74..7680587411f 100644 --- a/quest/m2/capture-windows.md +++ b/quest/m2/capture-windows.md @@ -20,7 +20,10 @@ including the cursor, move to - **Per-app audio** (decided 2026-10-26, maintainer: yes) uses WASAPI process loopback (`AUDIOCLIENT_ACTIVATION_TYPE_PROCESS_LOOPBACK`) to capture one process tree's output, through the same `moq_audio::capture` source as the - Linux and macOS halves. + Linux and macOS halves. Microsoft documents build 20348 as the floor, though + it works from Windows 10 2004 (build 19041) in practice. Activate it at + setup and refuse per-app audio with the reason where activation fails, + rather than gating on a build number. WGC does not answer app capture. It captures one monitor or one window per item and has no per-process source, so the per-window-composition or diff --git a/quest/m3/studio/audio-filters.md b/quest/m3/studio/audio-filters.md index e4f4af73b97..cc91eb55f47 100644 --- a/quest/m3/studio/audio-filters.md +++ b/quest/m3/studio/audio-filters.md @@ -13,5 +13,8 @@ plugins load into the chain. Prefer maintained crates, such as `nnnoiseless` for RNNoise suppression, over hand-rolled DSP. Benchmark the chain per source so its cost per added source shows up as a slope. Host plugins out of process, so a crashing or hung plugin -drops that filter rather than the audio bus, and use a maintained host crate +drops that filter rather than the audio bus: the audio thread never blocks on +the plugin process, a block the plugin misses its deadline for passes through +unfiltered, and a plugin that keeps missing is bypassed and shown as failed. +Use a maintained host crate (such as `clack` for CLAP) rather than hand-rolling the plugin ABIs. diff --git a/quest/m3/studio/compose/controls.md b/quest/m3/studio/compose/controls.md index 6eecca38dab..c6816831247 100644 --- a/quest/m3/studio/compose/controls.md +++ b/quest/m3/studio/compose/controls.md @@ -13,7 +13,9 @@ the studio. Map element priority onto the subscription priorities moq-lite already has (`doc/concept/moq-lite.md`), rather than a second mechanism. A toggled-off element is unsubscribed, so it costs the viewer and the relay nothing, and -the pull model may stop its encode. +the pull model may stop its encode. Subscriptions are per track, not per +element: a linked source shown by several items stays subscribed while any +of them is on, at the highest priority among them. ## Required diff --git a/quest/m3/studio/compose/layout.md b/quest/m3/studio/compose/layout.md index 40fce8833a2..fc4e0291794 100644 --- a/quest/m3/studio/compose/layout.md +++ b/quest/m3/studio/compose/layout.md @@ -13,7 +13,8 @@ shows. The hang catalog and `drafts/draft-lcurley-moq-hang.md` specify it. Serialize moq-scene's own model rather than a second layout schema, so the studio, server, and browser cannot drift. Version it and refuse an unknown version loudly. Keep it free of local paths and credentials: items name -tracks and contribution references, never files. A late joiner needs the +tracks and contribution references, never files. A viewer that refuses a +layout version plays the program instead and says why. A late joiner needs the current state, so use the JSON track's snapshot mode. ## Required diff --git a/quest/m3/studio/effects.md b/quest/m3/studio/effects.md index 0b57ef79b60..3c8ab494335 100644 --- a/quest/m3/studio/effects.md +++ b/quest/m3/studio/effects.md @@ -16,7 +16,8 @@ an effect means the same thing as the CSS overlays use. The layout track carries effect names and parameters, never shader code, since a shader from the network costs every viewer's GPU and can lose the device. An effect the browser cannot reproduce, like an ML model, is baked into its source's track -per the [composition epic](/quest/m3/studio/compose/README.md). Prefer an +per the [composition epic](/quest/m3/studio/compose/README.md), so ML effects +apply to a source, never to one scene item. Prefer an existing model crate for ML effects over training one. ## Required diff --git a/quest/m3/studio/failover.md b/quest/m3/studio/failover.md index 3d2eec81828..5e909dbaba5 100644 --- a/quest/m3/studio/failover.md +++ b/quest/m3/studio/failover.md @@ -11,7 +11,10 @@ mid-group and the stream does not drop. Publisher epochs already support this: the same epoch on several relays resumes mid-group, and a restart is a newer epoch (`doc/concept/moq-lite.md`, Publisher epochs and Multiple CDNs). The studio dials one relay today. Encode -once and publish to both. The second relay is configured, never inferred. Show +once and publish to both. The second relay is configured, never inferred. +Both links must negotiate a version with publisher epochs; a relay that does +not is refused at setup with the reason, rather than failing over as a +restart. Show each relay's state in the health panel. ## Related diff --git a/quest/m3/studio/guided-setup.md b/quest/m3/studio/guided-setup.md index bb7b893fecc..ced4e99ce0e 100644 --- a/quest/m3/studio/guided-setup.md +++ b/quest/m3/studio/guided-setup.md @@ -13,5 +13,7 @@ explains itself. Twitch Studio did this before it was shut down, and "settings are too complicated" is a top OBS complaint. Reuse the probes that already exist (the encoder choice `moq_video::render::export` makes at setup, the bandwidth -grant) rather than a separate benchmark. A probe that fails is reported, not -hidden. +grant) rather than a separate benchmark. The grant alone does not measure +the uplink, so setup runs a short probe publish to the configured relay and +reads the congestion controller's bandwidth estimate. A probe that fails is +reported, not hidden. diff --git a/quest/m3/studio/mixes.md b/quest/m3/studio/mixes.md index d3da7dca72f..dd0b89a3983 100644 --- a/quest/m3/studio/mixes.md +++ b/quest/m3/studio/mixes.md @@ -7,7 +7,8 @@ excludes the creator's music, a recording mix keeps it, and a monitor mix plays to headphones. Each source routes to any subset of mixes, and each output or recording picks a mix. Every source and mix shows a level meter. Per-source sync offsets fix cameras that lag their microphones, and a clap -or flash in front of the camera measures the offset instead of guessing it. +in front of the camera, seen and heard, measures the offset instead of +guessing it. Each guest hears a mix-minus: the program without their own voice. ## Plan diff --git a/quest/m3/studio/overlays.md b/quest/m3/studio/overlays.md index 318aa06c7c1..902418829c2 100644 --- a/quest/m3/studio/overlays.md +++ b/quest/m3/studio/overlays.md @@ -28,7 +28,10 @@ Decided 2026-10-26 (maintainer): the fixture compares boxes and timing, and pixels only loosely. An overlay is hostile input to every viewer, so it fetches nothing from the -network: fonts and images travel with the broadcast or are bundled. First check +network: fonts and images travel with the broadcast or are bundled. The +browser enforces it rather than trusting the studio: the overlay renders in a +sandboxed frame with scripts off and a content security policy that loads +only those carried resources. First check whether Blitz can draw into a caller's wgpu texture without a readback, and upstream that if it cannot. Arbitrary web pages and JavaScript graphics stay [browser sources](/quest/m3/studio/browser-source.md), baked into their own diff --git a/quest/m3/studio/scene-model.md b/quest/m3/studio/scene-model.md index 545c89f2fe3..b6c2a90e148 100644 --- a/quest/m3/studio/scene-model.md +++ b/quest/m3/studio/scene-model.md @@ -17,7 +17,9 @@ Clone, Source Copy, Downstream Keyer), and they all shape moq-scene's data model, which the [layout track](/quest/m3/studio/compose/layout.md) also serializes. So land them right after viewer composition, before more features build on the model. Route every edit through one command type so undo, -autosave, and the layout track see the same changes. Update +autosave, and the layout track see the same changes. Autosave persists the +history beside the scene file, bounded by a count, and launch restores it; +an entry the loader cannot replay drops it and everything older. Update `doc/bin/studio.md`. ## Related diff --git a/quest/m3/studio/snapping.md b/quest/m3/studio/snapping.md index b64c9bd938e..71f7ac8479f 100644 --- a/quest/m3/studio/snapping.md +++ b/quest/m3/studio/snapping.md @@ -10,8 +10,8 @@ turns snapping off. ## Plan OBS only now adds axis snapping. Snapping is a preview interaction, so it -lives in the studio, not in moq-scene or the layout track. Moves still go -through the scene model's commands so they undo. +lives in the studio, not in moq-scene or the layout track. Moves and resizes +still go through the scene model's commands so they undo and autosave. ## Related From ef0f5f558d5b2996fcfb95ec24ce5a8ec03a8afd Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 25 Oct 2026 23:22:40 -0700 Subject: [PATCH 4/4] quest: failover links negotiate the same draft Co-Authored-By: Claude Opus 5.5 --- quest/m3/studio/failover.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/quest/m3/studio/failover.md b/quest/m3/studio/failover.md index 5e909dbaba5..74fbae37901 100644 --- a/quest/m3/studio/failover.md +++ b/quest/m3/studio/failover.md @@ -12,10 +12,10 @@ Publisher epochs already support this: the same epoch on several relays resumes mid-group, and a restart is a newer epoch (`doc/concept/moq-lite.md`, Publisher epochs and Multiple CDNs). The studio dials one relay today. Encode once and publish to both. The second relay is configured, never inferred. -Both links must negotiate a version with publisher epochs; a relay that does -not is refused at setup with the reason, rather than failing over as a -restart. Show -each relay's state in the health panel. +Both links must negotiate the same protocol and draft, one with publisher +epochs, since a copy whose track info differs ends the track; a relay that +does not match is refused at setup with the reason, rather than failing over +as a restart. Show each relay's state in the health panel. ## Related