diff --git a/quest/m2/README.md b/quest/m2/README.md index d6056522bbe..2997a22447d 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -182,8 +182,9 @@ lite-07; bitrate enforcement is not part of this release. - [Keep-alive from the idle timeout](/quest/m2/quic-keep-alive.md) - the keep-alive interval defaults to a fraction of the negotiated idle timeout; an explicit `quic.keep_alive` overrides - [Socket close](/quest/m2/quic-socket-close.md) - moq-tokio's QUIC endpoint releases its socket on close, so it drops its wrapper - [Windows.Graphics.Capture](/quest/m2/capture-wgc.md) - the WGC display and window backend verified on real Windows hardware -- [Windows capture parity](/quest/m2/capture-windows.md) - system audio and a settled app-capture policy -- [Linux capture parity](/quest/m2/capture-linux.md) - Wayland window/system-audio capture with explicit display-selection and app-capture limits +- [Windows capture parity](/quest/m2/capture-windows.md) - system and per-app audio and a settled app-capture policy +- [Linux capture parity](/quest/m2/capture-linux.md) - Wayland window/system-audio capture, per-app audio, and explicit display-selection and app-capture limits +- [Per-app audio on macOS](/quest/m2/capture-app-audio-macos.md) - one application's audio through ScreenCaptureKit - [cpal loads libasound at runtime](/quest/m2/cpal-alsa-runtime.md) - condition: a cpal release whose Linux build carries no load-time libasound requirement - [Audio capture without ALSA link](/quest/m2/capture-alsa-link.md) - moq-audio capture and playback build on Linux without linking libasound, and capture becomes a default feature - [Ship capture and playback](/quest/m2/cli-packaging.md) - a released moq binary can capture and play, which no distribution currently enables diff --git a/quest/m2/capture-app-audio-macos.md b/quest/m2/capture-app-audio-macos.md new file mode 100644 index 00000000000..4195fb855da --- /dev/null +++ b/quest/m2/capture-app-audio-macos.md @@ -0,0 +1,22 @@ +# [S] Per-app audio on macOS + +## Goal + +`moq_audio::capture` captures one application's audio on macOS, so a creator +can stream a game without their music. The Linux and Windows halves live in +their capture parity quests. + +## Plan + +Decided 2026-10-26 (maintainer). ScreenCaptureKit already delivers system +audio here (`Source::System`), and an `SCContentFilter` that includes only one +application's windows limits the audio to that app. Apps are listed from +`SCShareableContent`, so only a running app can be chosen. It needs the +Screen Recording permission, like system audio. Per-app audio is its own +source variant naming the application; `Source::System` keeps capturing all +desktop audio through its display filter. + +## Related + +- [Linux capture parity](/quest/m2/capture-linux.md) - per-app audio through PipeWire nodes +- [Windows capture parity](/quest/m2/capture-windows.md) - per-app audio through WASAPI process loopback diff --git a/quest/m2/capture-linux.md b/quest/m2/capture-linux.md index d6083901346..d1dd50e9555 100644 --- a/quest/m2/capture-linux.md +++ b/quest/m2/capture-linux.md @@ -32,6 +32,14 @@ unimplemented, because xdg-desktop-portal owns source selection: - **Applications** have no portal concept behind them, so decide explicitly whether Linux offers app capture rather than leaving a variant that errors at runtime. +- **Per-app audio** (decided 2026-10-26, maintainer: yes) captures one + application's output from its PipeWire stream node, so a creator can stream + a game without their music. Apps are listed from the nodes present, so only + a running app can be chosen. An app is every stream node with its + `application.process.id` (one game may open several), mixed together, and + capture follows nodes as the app opens and closes them, matching the + Windows process tree and the macOS application. The Windows and macOS halves + share the same `moq_audio::capture` source. ## Related diff --git a/quest/m2/capture-windows.md b/quest/m2/capture-windows.md index 5e3f4a14043..7680587411f 100644 --- a/quest/m2/capture-windows.md +++ b/quest/m2/capture-windows.md @@ -17,6 +17,13 @@ including the cursor, move to exists but is macOS-only and returns `Unsupported` elsewhere. Unlike macOS this does not go through the screen-capture API, so it is an independent path and does not inherit the Screen Recording permission coupling. +- **Per-app audio** (decided 2026-10-26, maintainer: yes) uses WASAPI process + loopback (`AUDIOCLIENT_ACTIVATION_TYPE_PROCESS_LOOPBACK`) to capture one + process tree's output, through the same `moq_audio::capture` source as the + Linux and macOS halves. Microsoft documents build 20348 as the floor, though + it works from Windows 10 2004 (build 19041) in practice. Activate it at + setup and refuse per-app audio with the reason where activation fails, + rather than gating on a build number. WGC does not answer app capture. It captures one monitor or one window per item and has no per-process source, so the per-window-composition or diff --git a/quest/m3/studio/README.md b/quest/m3/studio/README.md index 8a71d19fcd8..95aa53f03b7 100644 --- a/quest/m3/studio/README.md +++ b/quest/m3/studio/README.md @@ -69,10 +69,21 @@ surface and zero-copy texture import, land before the cut. - [Zero-copy source import](/quest/m1/studio-gpu/import.md) - captured and decoded frames reach the compositor as GPU textures instead of a CPU upload - [Studio on macOS and Windows](/quest/m3/studio/app.md) - the studio runs on macOS and Windows, not only Linux +- [Viewer-side composition](/quest/m3/studio/compose/README.md) - sources as their own tracks plus a layout track, composed by viewers, with a server-composed program as the fallback +- [Undo and the scene model](/quest/m3/studio/scene-model.md) - undo history, linked and cloned sources, copy and paste, a downstream keyer, and per-item filters - [Autosave](/quest/m3/studio/autosave.md) - every edit is written to the scene file, with no save step or quit prompt - [Scene editing](/quest/m3/studio/scene-editing.md) - rename and reorder scenes, and resize the canvas while live through a new rendition - [Broadcast sources play in sync](/quest/m3/studio/broadcast-sync.md) - subscribed broadcasts and guests play audio and video on the publisher's timestamps, with `moq play`'s delay - [Wayland display and window sources](/quest/m3/studio/wayland-sources.md) - portal-picked sources that reopen without the picker +- [Live encode control](/quest/m3/studio/encode-control.md) - simulcast rungs that follow the bandwidth grant, live bitrate and resolution, keyframes on request +- [Stream health](/quest/m3/studio/health.md) - per-track delivery, viewer reports, and latency, with causes named +- [Relay failover](/quest/m3/studio/failover.md) - the same epoch on two relays, so a relay failure does not drop the stream +- [Instant replay and DVR](/quest/m3/studio/replay.md) - replays and clips from held groups, and viewer scrubbing through the archive +- [HTML/CSS overlays](/quest/m3/studio/overlays.md) - script-free HTML and CSS graphics through Blitz natively and the DOM in the browser +- [Animation and transitions](/quest/m3/studio/animation.md) - move, slide, wipe, and stinger transitions, and keyframed item properties +- [Video effects](/quest/m3/studio/effects.md) - a curated effect set with CSS meanings, and ML effects baked into their source +- [Snapping and guides](/quest/m3/studio/snapping.md) - preview moves snap to edges, centers, and other items +- [Guided setup](/quest/m3/studio/guided-setup.md) - first launch detects devices, measures the uplink, and proposes settings - [Destinations](/quest/m3/studio/destinations.md) - RTMP, RTMPS, and WHIP push with per-destination codecs and shared encoders, on a UI-free egress library a hosted service can reuse - [Multiple canvases](/quest/m3/studio/multi-canvas.md) - landscape and portrait canvases composite at once from shared sources - [Guests](/quest/m3/studio/guests.md) - remote guests and phone cameras join through a moq-room link or QR code and appear as sources diff --git a/quest/m3/studio/animation.md b/quest/m3/studio/animation.md new file mode 100644 index 00000000000..36fdbe99a7d --- /dev/null +++ b/quest/m3/studio/animation.md @@ -0,0 +1,23 @@ +# [M] Animation and transitions + +## Goal + +Scene changes in `moq-studio` use transitions beyond a cut and fade: move +(items glide between their positions in the two scenes), slide, wipe, and +stinger (a video clip with a cut point). Any item property can be keyframed +with easing, so an item can fly in or a camera can push in, and the same +animation plays in every composition host. + +## Plan + +Move is OBS's most-installed plugin (Move Transition), and stingers are +standard in broadcast. Animate in moq-scene on the media clock and carry the +transition and keyframes in the layout track, so viewers compose them too. +Overlays animate through their own CSS. No Lottie runtime for now (decided +2026-10-26): the open ones either lack text (velato) or add a C++ build +(ThorVG). A stinger clip is a media source like any other. + +## Required + +- [Undo and the scene model](/quest/m3/studio/scene-model.md) - keyframes are scene model state +- [Layout track](/quest/m3/studio/compose/layout.md) - where transitions and keyframes travel diff --git a/quest/m3/studio/audio-filters.md b/quest/m3/studio/audio-filters.md index 99a5e7ed53c..cc91eb55f47 100644 --- a/quest/m3/studio/audio-filters.md +++ b/quest/m3/studio/audio-filters.md @@ -4,10 +4,17 @@ Each audio source has an ordered processing chain in moq-audio: gain, noise suppression, noise gate, compressor, and limiter. Each filter is configured -from the studio and saved with the scene collection. +from the studio and saved with the scene collection. Ducking lowers one source +(music) when another (the microphone) is active. Third-party VST3 and CLAP +plugins load into the chain. ## Plan Prefer maintained crates, such as `nnnoiseless` for RNNoise suppression, over hand-rolled DSP. Benchmark the chain per source so its cost per added source -shows up as a slope. +shows up as a slope. Host plugins out of process, so a crashing or hung plugin +drops that filter rather than the audio bus: the audio thread never blocks on +the plugin process, a block the plugin misses its deadline for passes through +unfiltered, and a plugin that keeps missing is bypassed and shown as failed. +Use a maintained host crate +(such as `clack` for CLAP) rather than hand-rolling the plugin ABIs. diff --git a/quest/m3/studio/compose/README.md b/quest/m3/studio/compose/README.md new file mode 100644 index 00000000000..fe5d91f52f5 --- /dev/null +++ b/quest/m3/studio/compose/README.md @@ -0,0 +1,50 @@ +# Viewer-side composition + +## Goal + +A `moq-studio` broadcast carries each source as its own track plus a layout +track describing the live scene. Viewers compose the picture themselves: +they prioritize sources and toggle elements on and off. A composited program +remains for protocols and devices that cannot compose (RTMP and WHIP +destinations, HLS, legacy players), produced by the studio or by a server. + +This README's own work is the end-to-end test: a studio broadcast is composed +by the browser, by `moq compose`, and by the studio itself, and all three +pictures match. + +## Plan + +Decided 2026-10-26 (maintainer), the studio's flagship and first in its epic: + +- **Delivery:** MoQ's value is per-track delivery, so the studio embraces + viewer composition. The program is the fallback, not the default. +- **One compositor, three hosts:** moq-scene renders in the studio, headless + on a server (`moq compose` in moq-cli), and in the browser, so all three + pictures come from one implementation. +- **Parity rule:** every scene feature (transitions, filters, effects, + overlays) renders in all three hosts with the same layout, timing, and + colors, within a tolerance. Each renderer keeps its own text antialiasing + and blur, so the browser can draw overlays as real DOM. A feature that + cannot meet this, such as a CEF browser source or an ML effect, is baked + into its source's own track before publishing rather than living in the + layout. +- **Wire layout:** local sources are tracks in the studio's one broadcast. + Guests stay their own broadcasts, named through + [contribution references](/quest/m2/contribution-ref.md). +- **Layout:** the live scene state (items, transforms, transitions, and each + element's priority and toggleable flag) as a hang JSON track on the media + clock, so any host composites the same frame the studio shows. +- **Browser build:** whichever line [Rust-backed @moq/net](/quest/m2/rs2ts/README.md) + lands (WASM first, then JS built from Rust). Not fixed to WASM. +- **Cost:** every source and the program follow MoQ's pull model and are + encoded only while something subscribes, so a fallback nobody uses costs + nothing. + +## Required + +- [Source tracks](/quest/m3/studio/compose/sources.md) - each source publishes as its own track in the studio broadcast, encoded only while subscribed +- [Layout track](/quest/m3/studio/compose/layout.md) - the live scene state as a hang JSON track on the media clock, specified in the hang draft +- [Browser composition](/quest/m3/studio/compose/browser.md) - `@moq/watch` composes a broadcast with a layout track through moq-scene's browser build +- [Server composition](/quest/m3/studio/compose/server.md) - `moq compose` renders the program from sources and layout, headless +- [Viewer controls](/quest/m3/studio/compose/controls.md) - viewers toggle elements and the layout's priorities steer what they fetch +- [Hang contribution references](/quest/m2/contribution-ref.md) - guests composed by reference diff --git a/quest/m3/studio/compose/browser.md b/quest/m3/studio/compose/browser.md new file mode 100644 index 00000000000..1e3e32c4112 --- /dev/null +++ b/quest/m3/studio/compose/browser.md @@ -0,0 +1,25 @@ +# [L] Browser composition + +## Goal + +`@moq/watch` composes a broadcast that carries a layout track: it fetches the +tracks the layout shows and draws them through moq-scene's browser build, +matching the studio's picture. A broadcast without one plays its program as +today, so embedders change nothing. + +## Plan + +The browser build comes from whichever line +[Rust-backed @moq/net](/quest/m2/rs2ts/README.md) lands (decided 2026-10-26), +WASM first and JS built from Rust later, so this quest does not fix the +mechanism. A thin package (e.g. `@moq/scene`) wraps it and `@moq/watch` uses +it. Frames reach it as WebCodecs `VideoFrame`s through the same browser import +path the [web studio](/quest/m3/studio/web/video-web.md) adds. + +Prove parity with a fixture: the same broadcast composited here and by the +studio, compared within a stated tolerance. + +## Required + +- [Layout track](/quest/m3/studio/compose/layout.md) - what the browser composes +- [moq-video in the browser](/quest/m3/studio/web/video-web.md) - browser frames into wgpu diff --git a/quest/m3/studio/compose/controls.md b/quest/m3/studio/compose/controls.md new file mode 100644 index 00000000000..c6816831247 --- /dev/null +++ b/quest/m3/studio/compose/controls.md @@ -0,0 +1,22 @@ +# [M] Viewer controls + +## Goal + +A viewer composing a studio broadcast can switch off any element the layout +marks toggleable (a facecam, a chat overlay), and stops fetching its track. +The layout's per-element priorities steer delivery, so under congestion the +least important sources degrade first. The creator sets both per element in +the studio. + +## Plan + +Map element priority onto the subscription priorities moq-lite already has +(`doc/concept/moq-lite.md`), rather than a second mechanism. A toggled-off +element is unsubscribed, so it costs the viewer and the relay nothing, and +the pull model may stop its encode. Subscriptions are per track, not per +element: a linked source shown by several items stays subscribed while any +of them is on, at the highest priority among them. + +## Required + +- [Browser composition](/quest/m3/studio/compose/browser.md) - the viewer that exposes the toggles diff --git a/quest/m3/studio/compose/layout.md b/quest/m3/studio/compose/layout.md new file mode 100644 index 00000000000..fc4e0291794 --- /dev/null +++ b/quest/m3/studio/compose/layout.md @@ -0,0 +1,22 @@ +# [M] Layout track + +## Goal + +A studio broadcast carries a layout track: moq-scene's live scene state +(items, the tracks they show, transforms, transitions, and each element's +priority and whether a viewer may toggle it) as a hang JSON track on the +media clock. Any host that reads it composites the same frame the studio +shows. The hang catalog and `drafts/draft-lcurley-moq-hang.md` specify it. + +## Plan + +Serialize moq-scene's own model rather than a second layout schema, so the +studio, server, and browser cannot drift. Version it and refuse an unknown +version loudly. Keep it free of local paths and credentials: items name +tracks and contribution references, never files. A viewer that refuses a +layout version plays the program instead and says why. A late joiner needs the +current state, so use the JSON track's snapshot mode. + +## Required + +- [Source tracks](/quest/m3/studio/compose/sources.md) - the tracks a layout names diff --git a/quest/m3/studio/compose/server.md b/quest/m3/studio/compose/server.md new file mode 100644 index 00000000000..e4bf1b60e17 --- /dev/null +++ b/quest/m3/studio/compose/server.md @@ -0,0 +1,20 @@ +# [M] Server composition + +## Goal + +`moq compose` in moq-cli subscribes to a studio broadcast's sources and +layout track and publishes the composited program, headless (a GPU, or +lavapipe without one). It can push that program to RTMP and WHIP through the +studio's egress library, so a hosted service makes the fallback composite and +the creator saves the upload. + +## Plan + +Decided 2026-10-26 (maintainer): a moq-cli subcommand, not a mode of +`moq transcode`. A hosted service embeds the same library. It publishes its +own broadcast, never the studio's names: each publisher produces its own +stream (AGENTS.md). Update `doc/bin/cli.md`. + +## Required + +- [Layout track](/quest/m3/studio/compose/layout.md) - what the server composes diff --git a/quest/m3/studio/compose/sources.md b/quest/m3/studio/compose/sources.md new file mode 100644 index 00000000000..b8d1055b905 --- /dev/null +++ b/quest/m3/studio/compose/sources.md @@ -0,0 +1,18 @@ +# [M] Source tracks + +## Goal + +Each local source in a `moq-studio` scene (camera, display, window, image, +microphone) publishes as its own track in the studio's broadcast, listed in +its catalog, alongside the program. Each is encoded only while something +subscribes to it. + +## Plan + +Follow the pull model the program already uses: an unwatched source costs +no encode and no upload. Static sources (images, colors) publish once rather +than as a video stream. Consumer GPUs cap concurrent hardware encode +sessions, so when the hardware refuses one, that source's track fails loudly +with the reason shown rather than silently degrading. +Track names stay stable for a source's life, and an edited source that +changes what it shows gets a new name. diff --git a/quest/m3/studio/effects.md b/quest/m3/studio/effects.md new file mode 100644 index 00000000000..3c8ab494335 --- /dev/null +++ b/quest/m3/studio/effects.md @@ -0,0 +1,25 @@ +# [M] Video effects + +## Goal + +Sources and scene items in `moq-studio` take a curated set of video effects: +color correction, `.cube` LUTs, blur, crop and masks, chroma and luma key, +and blend modes. They render the same in every composition host. ML effects +such as background removal are offered too, applied before the source +publishes. + +## Plan + +Write the effects in WGSL to the meanings the CSS Filter Effects and +Compositing specs define (color matrices, blur, drop-shadow, blend modes), so +an effect means the same thing as the CSS overlays use. The layout track +carries effect names and parameters, never shader code, since a shader from +the network costs every viewer's GPU and can lose the device. An effect the +browser cannot reproduce, like an ML model, is baked into its source's track +per the [composition epic](/quest/m3/studio/compose/README.md), so ML effects +apply to a source, never to one scene item. Prefer an +existing model crate for ML effects over training one. + +## Required + +- [Undo and the scene model](/quest/m3/studio/scene-model.md) - per-item filters this extends diff --git a/quest/m3/studio/encode-control.md b/quest/m3/studio/encode-control.md new file mode 100644 index 00000000000..4ff102202cd --- /dev/null +++ b/quest/m3/studio/encode-control.md @@ -0,0 +1,26 @@ +# [L] Live encode control + +## Goal + +`moq-studio` adapts its encode live without a restart: the program and each +source publish simulcast rungs that follow the relay's bandwidth grant, a rung +nobody watches stops encoding, and bitrate and resolution change mid-stream +through catalog updates. OBS's multitrack output does +part of this only on Windows with specific GPUs; this works on every +platform. + +## Plan + +Today the studio publishes one H.264 rendition at fixed settings and never +reads `bandwidth::Allocator`. Reuse what exists rather than adding a studio +path: the grant follower in `moq_video::encode` and `moq_mux::rate`, the +catalog's in-place updates and new renditions (`doc/concept/hang.md`). The +encode producer already cuts a keyframe when a subscriber joins +(`encode/cuts.rs`), so each rung inherits that and can run a long GOP. Show +each rung's grant, bitrate, and viewers in the Output panel. + +## Related + +- [Simulcast rung disable](/quest/m1/simulcast-rung-disable.md) - grant-driven rung disable this builds on +- [moq-audio follows the grant](/quest/m2/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md) - the audio half +- [Scene editing](/quest/m3/studio/scene-editing.md) - canvas resize through a new rendition diff --git a/quest/m3/studio/failover.md b/quest/m3/studio/failover.md new file mode 100644 index 00000000000..74fbae37901 --- /dev/null +++ b/quest/m3/studio/failover.md @@ -0,0 +1,22 @@ +# [M] Relay failover + +## Goal + +`moq-studio` publishes the same broadcast epoch to two relays at once, so +when one relay or the path to it fails, viewers resume from the other +mid-group and the stream does not drop. + +## Plan + +Publisher epochs already support this: the same epoch on several relays +resumes mid-group, and a restart is a newer epoch (`doc/concept/moq-lite.md`, +Publisher epochs and Multiple CDNs). The studio dials one relay today. Encode +once and publish to both. The second relay is configured, never inferred. +Both links must negotiate the same protocol and draft, one with publisher +epochs, since a copy whose track info differs ends the track; a relay that +does not match is refused at setup with the reason, rather than failing over +as a restart. Show each relay's state in the health panel. + +## Related + +- [Stream health](/quest/m3/studio/health.md) - where each relay's state shows diff --git a/quest/m3/studio/guided-setup.md b/quest/m3/studio/guided-setup.md new file mode 100644 index 00000000000..ced4e99ce0e --- /dev/null +++ b/quest/m3/studio/guided-setup.md @@ -0,0 +1,19 @@ +# [M] Guided setup + +## Goal + +The first launch of `moq-studio` gets a creator live without reading +settings: it detects cameras, microphones, and displays, measures the uplink, +probes which encoders work on this machine, and proposes a scene and output +settings in plain language. Every setting it chooses can be changed later and +explains itself. + +## Plan + +Twitch Studio did this before it was shut down, and "settings are too +complicated" is a top OBS complaint. Reuse the probes that already exist +(the encoder choice `moq_video::render::export` makes at setup, the bandwidth +grant) rather than a separate benchmark. The grant alone does not measure +the uplink, so setup runs a short probe publish to the configured relay and +reads the congestion controller's bandwidth estimate. A probe that fails is +reported, not hidden. diff --git a/quest/m3/studio/health.md b/quest/m3/studio/health.md new file mode 100644 index 00000000000..11e50e6a1a9 --- /dev/null +++ b/quest/m3/studio/health.md @@ -0,0 +1,21 @@ +# [M] Stream health + +## Goal + +`moq-studio` shows how its stream is actually delivered, per track: viewers, +the bandwidth granted, what viewers report receiving and dropping, and +end-to-end latency, rather than only encoder counters. A problem names its +cause in plain words (the uplink is congested, a source stalled, the encoder +is behind). + +## Plan + +Read what the stack already publishes: the catalog's `stats` and `echo` +entries and `hang::{stats,echo}::Snapshot` (`doc/concept/stats.md`), and the +relay's stats broadcasts. OBS reports "0 dropped frames" while the bitrate sits +at zero, which is the confusion to avoid. + +## Related + +- [Stats](/quest/m2/stats/README.md) - the stats tracks this reads +- [Latency ledger](/quest/m2/latency-ledger.md) - per-stage playback delay a viewer reports diff --git a/quest/m3/studio/mixes.md b/quest/m3/studio/mixes.md index a9aef7338ba..dd0b89a3983 100644 --- a/quest/m3/studio/mixes.md +++ b/quest/m3/studio/mixes.md @@ -6,7 +6,10 @@ The program bus in moq-audio feeds several mixes. For example, a stream mix excludes the creator's music, a recording mix keeps it, and a monitor mix plays to headphones. Each source routes to any subset of mixes, and each output or recording picks a mix. Every source and mix shows a level meter. -Per-source sync offsets fix cameras that lag their microphones. +Per-source sync offsets fix cameras that lag their microphones, and a clap +in front of the camera, seen and heard, measures the offset instead of +guessing it. +Each guest hears a mix-minus: the program without their own voice. ## Plan diff --git a/quest/m3/studio/overlays.md b/quest/m3/studio/overlays.md new file mode 100644 index 00000000000..902418829c2 --- /dev/null +++ b/quest/m3/studio/overlays.md @@ -0,0 +1,42 @@ +# [L] HTML/CSS overlays + +## Goal + +Titles, lower thirds, tickers, and other graphics in `moq-studio` are written +as HTML and CSS without JavaScript, travel in the layout track, and render in +all three composition hosts: through Blitz in the studio and `moq compose`, +and as real DOM over the video in the browser. CSS animations and transitions +run on the media clock, so an overlay animates on the same frame everywhere. + +## Plan + +Decided 2026-10-26 (maintainer): + +- **Format:** HTML and CSS, no JavaScript, so creators author in a format they + know and nothing executes in a viewer. +- **Native renderer:** Blitz (Stylo, Taffy, Parley) drawing through Vello into + the studio's wgpu device, with its animation clock driven by the media + clock. Servo was passed over: it outputs OpenGL rather than wgpu, bundles a + JavaScript engine, and has no clock injection. +- **Browser:** the DOM layered over the composited video, so no HTML-in-Canvas + and it works in every browser today. +- **Subset:** the layout track accepts only the CSS Blitz renders and refuses + the rest at setup with the property named. Blitz is beta (no blend modes or + `text-shadow`, filters limited to blur and drop-shadow), and the subset + grows as it does. +- **Parity:** layout parity per the [composition epic](/quest/m3/studio/compose/README.md); + the fixture compares boxes and timing, and pixels only loosely. + +An overlay is hostile input to every viewer, so it fetches nothing from the +network: fonts and images travel with the broadcast or are bundled. The +browser enforces it rather than trusting the studio: the overlay renders in a +sandboxed frame with scripts off and a content security policy that loads +only those carried resources. First check +whether Blitz can draw into a caller's wgpu texture without a readback, and +upstream that if it cannot. Arbitrary web pages and JavaScript graphics stay +[browser sources](/quest/m3/studio/browser-source.md), baked into their own +track. + +## Required + +- [Layout track](/quest/m3/studio/compose/layout.md) - where overlays travel diff --git a/quest/m3/studio/replay.md b/quest/m3/studio/replay.md new file mode 100644 index 00000000000..a0a0fcc6d86 --- /dev/null +++ b/quest/m3/studio/replay.md @@ -0,0 +1,33 @@ +# [M] Instant replay and DVR + +## Goal + +`moq-studio` replays the last seconds of any source or the program on demand: +several lengths, slow motion, and a one-click clip saved to disk, with no +separate always-on recording. Viewers can scrub back through the live +broadcast as far as the relay or archive holds it. + +## Plan + +The groups are already on hand: the studio's own recent groups and the +relay's cache, addressed by presentation time through subscription ranges +(`doc/concept/moq-lite.md`). A replay plays from those rather than a RAM-only +frame buffer like OBS's. Retention past the relay cache is the archive's job. + +Sources and the program encode only while subscribed +([source tracks](/quest/m3/studio/compose/sources.md)), so a track nobody +watched has no groups to replay. Arming replay for a source or the program +is a local subscriber holding a retention window, so that track keeps +encoding with no external subscriber, and its encode cost shows like any +other subscriber's. An unarmed track replays only what was encoded, and the +gaps show rather than being skipped. Acceptance covers the case where every +viewer composes sources themselves and nothing outside the studio subscribes +to the program. +Replay published to viewers is a new source in the scene, so it composes like +any other. + +## Related + +- [Archive time ranges](/quest/m2/archive-time-ranges.md) - ranges reaching past the cache +- [Archive DVR](/quest/m2/archive-dvr.md) - viewer scrubbing beyond the relay cache +- [Recording](/quest/m3/studio/recording.md) - clips use the same file writer diff --git a/quest/m3/studio/scene-model.md b/quest/m3/studio/scene-model.md new file mode 100644 index 00000000000..b6c2a90e148 --- /dev/null +++ b/quest/m3/studio/scene-model.md @@ -0,0 +1,28 @@ +# [L] Undo and the scene model + +## Goal + +Every edit in `moq-studio` can be undone and redone from a history panel, and +the history survives autosave and restarts. The scene model distinguishes a +linked source (one capture shown in several scenes) from a clone with its own +filters, items and filters copy and paste between scenes and collections, +each canvas has a top layer that stays on across scene switches (a +downstream keyer), and filters can be set per scene item rather than only per +source. + +## Plan + +These are the most-voted OBS gaps and the most-installed plugins (Source +Clone, Source Copy, Downstream Keyer), and they all shape moq-scene's data +model, which the [layout track](/quest/m3/studio/compose/layout.md) also +serializes. So land them right after viewer composition, before more features +build on the model. Route every edit through one command type so undo, +autosave, and the layout track see the same changes. Autosave persists the +history beside the scene file, bounded by a count, and launch restores it; +an entry the loader cannot replay drops it and everything older. Update +`doc/bin/studio.md`. + +## Related + +- [Autosave](/quest/m3/studio/autosave.md) - the history must survive it +- [Scene editing](/quest/m3/studio/scene-editing.md) - rename, reorder, and resize go through the same commands diff --git a/quest/m3/studio/snapping.md b/quest/m3/studio/snapping.md new file mode 100644 index 00000000000..71f7ac8479f --- /dev/null +++ b/quest/m3/studio/snapping.md @@ -0,0 +1,18 @@ +# [S] Snapping and guides + +## Goal + +Moving and resizing items in `moq-studio`'s preview snaps to the canvas +edges and center, to other items' edges and centers, and to an optional grid, +with smart guides and equal-spacing hints drawn while dragging. A modifier key +turns snapping off. + +## Plan + +OBS only now adds axis snapping. Snapping is a preview interaction, so it +lives in the studio, not in moq-scene or the layout track. Moves and resizes +still go through the scene model's commands so they undo and autosave. + +## Related + +- [Undo and the scene model](/quest/m3/studio/scene-model.md) - snapped moves are ordinary commands