Repository navigation
feat(studio)!: moq-studio, moq-scene, and a moq-audio program bus - #6797
Conversation
Quest-Claim: 9bb42c9a3f0daa9f
Adds the UI-free half of moq-studio: a versioned TOML scene collection, live sources (capture, image, color, subscribed broadcast), a compositor with preview, program, cut, and fade, and a pooled program readback. Adds moq-audio's program bus with per-source strips, and moves Level to the crate root so the bus and capture share it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One wgpu device serves the sources, the compositor, and the window. A program clock draws the canvas and mixes the bus, and the program is read back and encoded only while a viewer watches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…emains Adds doc/bin/studio.md, the repository map and AGENTS.md rows, and the moq-audio migration note for Level. The app quest now holds only the zero-copy program encode and the macOS and Windows runs, and the studio quests that only needed the app are unblocked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… frame The canvas is black until a source delivers its first picture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated
changed the title
Collection::validate already refuses a negative or non-finite gain from a scene file, so only a caller bug reaches Strip::set_gain; fail loud there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
app.md keeps only the macOS and Windows runs. Zero-copy program encode and zero-copy source import become quest/m2/studio-gpu, ranked at the top of m2 and first in the studio epic, with a dated note in m1 and the studio README that they drive moq-video improvements ahead of the cut without gating it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WalkthroughThe change adds an audio bus, a scene library with GPU composition and readback, and the MoQ Studio desktop application. Studio can edit scene collections, preview and program scenes, manage sources and audio, and optionally publish to a relay. The change also adds build and test wiring, Studio documentation, and updated GPU and cross-platform quest plans. Priority: Merge Risk:
 |
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage |  |
Docstring coverage is 55.45% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 202 functions across 16 files. (19 skippe… | Write docstrings for the functions missing them to satisfy the coverage threshold. |
-
Check name Status Explanation Title check The title clearly identifies the three primary changes: the new studio application, scene crate, and audio program bus. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Description check The pull request description directly describes the new moq-audio bus, moq-scene crate, moq-studio application, documentation and quest changes, GPU work split, breaking API change, tests, and remaini… Full details: Description check
-
Explanation
The pull request description directly describes the new moq-audio bus, moq-scene crate, moq-studio application, documentation and quest changes, GPU work split, breaking API change, tests, and remaining platform work. It is clearly related to the changeset.
-
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 55.45% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 202 functions across 16 files. (19 skipped: 19 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
Finishing Touches
-
✨ Simplify code
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
kixelated
left a comment
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit: c9a4ffd
Two actionable findings:
- P2 — Follow catalog updates within a broadcast (rs/moq-scene/src/source.rs:295–306). Only the first catalog is read. Joining while a camera is disabled means enabling it never starts video; switching codecs/tracks likewise leaves the original subscription frozen or ended until the broadcast restarts. Keep polling catalogs and rebuild affected consumers. Test enabling and track replacement without changing broadcast epoch.
- P2 — Clear cached pictures when replacing a source (rs/moq-scene/src/compositor.rs:431–438). Sources restart when their configuration changes, but GPU layers survive by name alone. Replacing a visible source with an offline broadcast or missing image therefore keeps the previous source's picture on program indefinitely. Invalidate its layer when feed identity changes, while retaining last-frame behavior for the same stalled feed.
Overall direction is sound: separating scene composition from the UI, bounded readback, and caller-owned audio controls keep the implementation understandable.
Verification: static inspection of scene/source lifecycle, composition, audio bus, publisher integration and build wiring. No tests or GPU execution performed.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c9a4ffd1c5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| pub async fn run(self) { | ||
| let status = self.status.clone(); | ||
| let result = tokio::select! { | ||
| _ = status.unused() => return, | ||
| result = self.produce() => result, | ||
| }; |
There was a problem hiding this comment.
rs/AGENTS.md requires new library logic to expose poll_* operations with async helpers and specifically directs libraries away from Tokio select!. The new source driver instead implements cancellation and production entirely through async futures here and another tokio::select! in broadcast, leaving no poll interface for callers that drive the UI-free scene library synchronously. Implement the driver state machine with kio primitives and make run its async wrapper.
AGENTS.md reference: rs/AGENTS.md:L21-L24
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Declining. The driver composes moq-video's capture and decode consumers and moq-audio's capture, which are async-only today; a poll state machine here would have to pin and poll those futures inside kio::wait, the pattern rs/AGENTS.md names a smell. tokio::select! over them matches the sibling media crates (moq-video capture and decode, moq-audio capture). Poll APIs belong in those crates first, and moq-scene can follow once they exist.
(Written by Claude Opus 5.5)
| for entry in entries.iter_mut() { | ||
| entry.cons.read_interleaved(scratch, false); |
There was a problem hiding this comment.
The new mixer performs resampling, gain processing, metering, and summation for every channel on each audio tick, alongside a new per-frame compositor and readback path. Searching the repository's benchmark code and Cargo manifests finds no benchmark exercising these additions; the supplied tests establish correctness but do not measure whether these paths meet their real-time budgets as source and item counts grow. Root AGENTS.md requires benchmarks for performance-sensitive changes, so add measured coverage for these paths before landing.
AGENTS.md reference: AGENTS.md:L53-L53
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Partly fixed in b299e60009: rs/moq-audio/benches/bus.rs sweeps the bus over 1, 4, and 16 channels, at 48 kHz and resampled 44.1 kHz (about 3.4 us per channel per 10 ms block here, linear). Declining a compositor and readback bench for now: their cost is GPU time, and the only GPU in CI is lavapipe, whose timings say nothing about real hardware. The zero-copy encode and import quests (quest/m1/studio-gpu/) count readback and upload in the studio's stats, which is where that cost is tracked.
(Written by Claude Opus 5.5)
moq-video's GPU encode and import API changes land before the cut. The m1 quests are scoped to the API shape and the backends provable now (NVENC, VideoToolbox, Media Foundation, DMA-BUF, Metal); AMD Vulkan encode, Intel VA-API import, and Direct3D11 render import stay m2 follow-ups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- moq-studio enters its runtime before building the QUIC client, which refused to start outside one, so --connect works from the binary. - A failed video or audio output aborts its track with the cause instead of dropping it unfinished, and a clean stop finishes the video. - A broadcast source follows catalog updates, switching a rendition when a better one appears instead of keeping the first catalog's choice. - The compositor forgets an edited source's last picture, so a replaced source never shows the old one's frame. - A criterion bench sweeps the program bus over channel count and resampling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Reply to the automated OpenAI review of c9a4ffd:
(Written by Claude Opus 5.5) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 428a7565b3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| while let Some(read) = consumer.read().await? { | ||
| if let moq_audio::decode::Read::Frame(frame) = read { | ||
| channel.write(&frame.data)?; |
There was a problem hiding this comment.
When a broadcast delivers buffered audio containing a timestamp hole, this loop discards both the Read::Gap event and each frame's timestamp, then appends the PCM on either side contiguously. moq_audio::decode::Consumer::read returns available samples without pacing them, and the bus only buffers sample data, so silence is not inserted merely because the publisher's timeline contains a gap. For example, packets delivered together after a network stall can have their missing interval compressed away, shifting the source's audio relative to its video. Map decoded timestamps onto the program timeline and preserve gaps before writing samples to the bus.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Declining. The studio plays every broadcast at its live edge, not on the publisher's timeline: a picture shows the moment it decodes (one-slot mailbox), and audio goes through the bus's bounded ring on the program clock. During a stall the bus already plays silence for that source in real time, so the hole is heard while it happens. Replaying the publisher's timeline after the burst would insert that hole a second time, putting the audio behind the live edge and behind its own video, which is already showing the newest frame. A burst of late audio is bounded by the bus's buffer (50 ms target, 200 ms capacity, trimmed back by its overflow correction). Timeline-accurate A/V sync per source would need a jitter buffer for video too, which belongs with the per-source sync work rather than here. The comment in play_audio now says this in b206186024.
(Written by Claude Opus 5.5)
kixelated
left a comment
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit: 428a756
Both earlier findings are fixed: broadcast catalog updates are followed, and changed source definitions invalidate cached pictures.
P2 — Preserve playback across metadata-only catalog updates (rs/moq-scene/src/source.rs:329–332, likewise 317–320). Comparing entire rendition configurations recreates the decoder and audio bus channel when only jitter, delay, or descriptive metadata changes. Publisher estimates can republish once per second; each restart discards buffered audio and rejoins at Start::Latest, introducing discontinuities despite unchanged media. Compare playback identity/decoder-relevant fields separately and add a jitter-only update regression test.
The independently reported timestamp-gap loss #6797 (comment) also remains relevant; no duplicate comment added.
Direction remains sound. Runtime initialization and explicit producer aborts address the corresponding independent findings; the bus benchmark adds useful coverage.
Verification: static comparison against c9a4ffd and source/test inspection, accounting for merged base changes. Tests and benchmarks were not run.
kixelated
left a comment
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit: b206186
The resolved-track comparison finding is fixed: Reference::track(config, id) now controls rebinding, with a regression covering stable-ID replacement. Metadata-only updates still preserve playback.
The shared catalog clock, continued preview rendering after encoder failure, explicit source errors and non-finite PCM rejection are sensible improvements. No new actionable bugs found in this update.
The earlier timestamp-gap concern #6797 (comment) is unchanged, not fixed; the author explicitly chooses arrival-clock playback and defers timeline-accurate source synchronization.
Verification: static comparison against 551bef6 and surrounding code/tests, accounting for base advancement. Tests, GPU execution and benchmarks were not run.
|
Landing
(Written by Claude Opus 5.5) |
# Conflicts: # quest/m1/README.md
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3b598fe099
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ture Display and window sources map to moq-video's X11 capture, so a Wayland screen or window is unreachable until a portal-backed source exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Re-landing after the queue dropped it for a
(Written by Claude Opus 5.5) |
Also registers moq-audio's bus bench in the rust-extra Nightly leg, which main's bench coverage test now requires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated
left a comment
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit: 4b96152
No new actionable bugs found in the PR-specific update. Registering moq-audio's bus benchmark in the rust-extra Nightly invocation is a focused coverage fix: the named target exists, uses Criterion, and its required bus feature is enabled by default. The merged benchmark-coverage check now includes it.
The studio/audio implementation and previously fixed resolved-track rebinding are unchanged. The timestamp-gap concern #6797 (comment) remains intentionally unchanged, not fixed; arrival-clock playback is still the chosen design.
Verification: static base-aware comparison against the last published review and intervening equivalent merges, plus Nightly registration, target manifest and benchmark inspection. No builds, tests, benchmarks or GPU execution performed.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4b96152401
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A pick held across removing its source, or reloading a collection without it, added a dangling item that failed every later save. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated
left a comment
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit: 98fb1be
The independent stale-source selection finding #6797 (comment) is fixed: the Items panel clears a selection absent from the current video-capable sources before it can be inserted. The same state lock covers validation and insertion, so source removal or reload cannot interleave between them.
No new actionable bugs found. Reusing the existing eligible-source list is a simple, focused fix. Earlier resolved-track fixes and the intentionally deferred timestamp-gap behavior are unchanged.
Verification: static inspection of the four-line delta and surrounding add/remove/reload flow against 4b96152; base unchanged. No regression test was added in this delta, and no tests, builds or GUI execution were run.
|
Re-landing after the queue failed
(Written by Claude Opus 5.5) |
Works on quest/m3/studio/app.md and rewrites it to the remaining work: running the studio on macOS and Windows. The GPU work moves to a new m1 epic,
quest/m1/studio-gpu/(zero-copy program encode and zero-copy source import), so moq-video's GPU encode and import API changes land before the cut.Problem
The studio epic needs its first consumer: a native app that composes scenes on the GPU and goes live over MoQ. It also needs the UI-free crate (
rs/moq-scene) and moq-audio's program bus that later quests build on.Approach
moq-audio::bus(defaultbusfeature, pure Rust):BusmixesChannels into one interleaved f32 feed, read on the caller's clock.fixed-resample, and remixed to the bus layout.Strip(gain with a per-block ramp, mute, level meter) is caller-owned, so controls outlive a source restart.Channel::writerefuses a NaN or infinite sample, which would poison the whole mix through any gain.Bus::capture(withcapture) feeds a microphone or system audio into the bus until the device fails or stops, returning its error.rs/moq-scene(new,publish = false):Collectionis a versioned TOML scene file withdeny_unknown_fields. An unknown version or a dangling reference fails loudly, and the file never holds credentials.source::Sourcesruns camera, display, window, microphone, and system-audio capture (behindcapture), image, color, and subscribed-broadcast sources. Pictures go to a one-slot mailbox and sound goes to the bus. Editing a source restarts only that source. A broadcast source restarts a rendition only when the track its catalog entry resolves to changes, and fails on a catalog whose renditions all live in another broadcast.Compositordraws preview and program into textures on the caller'swgpudevice. It handles cut and fade, blends straight alpha in gamma space, and repeats a stalled source's last picture. A picture it cannot draw is reported throughCompositor::error.Readbackis a pooled, non-blocking program copy for encoders that cannot take a GPU surface. A full pool drops frames rather than queuing them.rs/moq-studio(new binary,publish = false):WgpuSetup::Existing), with preview and program registered as native egui textures, so neither is read back for display.--broadcastwith a minted epoch. Both are stamped on the catalog's clock, so its advertised wall time is when the program drew them. Broadcast sources subscribe over the same connection.Impact
moq_audio::encode::Level(capturefeature) is nowmoq_audio::Level. Migrate by changing the import path. Noted inrs/moq-audio/MIGRATE.md.moq_audio::bus::{Bus, Channel, Strip, Config, Capture}behind the default-onbusfeature (adds thefixed-resampledependency).moq-sceneandmoq-studio. No wire changes.just rs render-sw(nightly) also runs the moq-scene compositor tests and a moq-studio end-to-end test on lavapipe. Thecapturelint and test pass now covers moq-scene and moq-studio.Tests
benches/bus.rssweeps the mix over 1, 4, and 16 channels at 48 kHz and resampled 44.1 kHz (about 3.4 us per channel per 10 ms block, linear).Review round (/quest-iterate)
--connectfailed from the binary because the QUIC client was built outside the runtime; fixed. A failed or stopped output dropped its tracks unfinished (viewers sawDropped); it now aborts with the cause or finishes. Bus benchmark added. Declined: a poll-based source driver (its capture and decode inputs are async-only, and the sibling media crates useselect!the same way) and a compositor bench (lavapipe timing is not meaningful; the GPU quests count readback and upload instead).551bef62f3), fixed in b206186024:tokio::select!in moq-scene, and hoistingbytemuck.quest/m1/README.mdconflict. Codex then noted thatpipewireadvertised Wayland screen capture the scene sources never reach; the claim is corrected in 39277bb6a0 (PipeWire cameras and audio; display and window are X11 and XWayland only), and a portal-backed source is a follow-up.moq-audio'sbusbench now runs in therust-extraleg. Codex: an Items pick whose source was removed or reloaded away added a dangling item; fixed in 98fb1be6f8.Bus::captureno longer reopens a failed device with backoff, a catalog of only foreign renditions fails the source instead of playing nothing, and a picture the compositor cannot draw shows as a failed source instead of only a log line. GPU import and export paths are in the stacked moq-dev/moq#6805 and moq-dev/moq#6806.Alternatives
wgpu. That is a separate, independently landable step, nowquest/m1/studio-gpu/encode.md.Decisions
Settled with the maintainer (/quest-iterate, recommendations accepted):
✅ Ship CPU readback now, zero-copy later. Every program frame is read back for the encoder (counted in the Output panel). The GPU work is split out of
app.md(maintainer, 2026-10-25) into a new m1 epic,quest/m1/studio-gpu/README.md, so moq-video's GPU encode and import API changes land before the cut:encode.md[L]: the program reaches the encoder as a GPU surface on NVENC (Vulkan slot), VideoToolbox (IOSurfaceCVPixelBuffer), and Media Foundation (shared D3D11), exported in moq-video or moq-scene. The slot is shaped so VA-API and AMD Vulkan Video can take it later without an API change.import.md[M]: moq-video's renderer reports import or upload per frame, and captured and decoded frames reach the compositor as imported textures on Linux (DMA-BUF) and Apple (Metal), with uploads counted. The renderer already imports both; the gaps are that report, moq-scene not enablingpipewire, and no upload counter.quest/m1/README.mdRequired right after the API audit, and both quests leadquest/m3/studio/README.mdRequired. A dated Plan note inquest/m1/README.mdand the studio README records why they are in m1.app.md: each path is tested on its own platform's CI, NVENC byjust rs gpu. No m2 quest moved. AMD Vulkan Video encode (vulkan-encode.md), Intel VA-API on an external Vulkan image (vaapi-vulkan-import.md), and Direct3D11 render import (render-d3d11.md) stay in m2 as Related follow-ups, as the m1 README already defers them; until they land, those backends take the counted CPU path.app.mdis retitled "Studio on macOS and Windows" and holds only the platform runs.✅ Quest alignment. Nine studio quests that only needed the app to exist are unblocked: destinations, multi-canvas, guests, mixes, audio filters, recording, browser source, Linux vcam, and web video. Packaging and the macOS and Windows vcams still require
app.mdfor the platform runs; none of them needs the GPU quests.✅ Every audio source is always in the program mix, whatever scene is live, muted per source, until the audio mixes quest owns routing. OBS ties scene-local audio to the active scene.
✅ TOML scene file, version 1. Audio gain and mute live in a separate
[audio.<name>]table, so editing them never restarts a source. A canvas change takes a restart.✅
capturestays opt-in for moq-studio, as for moq-cli, because ALSA links at build time on Linux. A default build has image, color, and broadcast sources only. Revisit in packaging.✅ No silent runtime fallbacks (maintainer, 2026-10-25): a path that cannot run is an error at setup or on use, never a quiet degrade with strikes or retries. Consequence here: an unplugged microphone fails its source with the reason, and the user re-adds or edits it to reopen, as with a camera.
Chosen without asking:
Strip::set_gainpanics on a negative or non-finite gain, likeDuration::from_secs_f32, instead of silently keeping the old one.Collection::validaterefuses such a gain from a scene file, so only a caller bug reaches it.Channel::writerefuses non-finite samples withError::Unsupportedrather than replacing them with silence, per "fail loud on malformed input".Follow-ups
app.md(rewritten): running the studio on macOS and Windows. Only Linux was built and run here.quest/m1/studio-gpu/encode.md(new): the program reaches the encoder as a GPU surface instead of a CPU readback.quest/m1/studio-gpu/import.md(new): sources reach the compositor as imported GPU textures, with uploads counted.quest/m2/capture-linux.md).vulkan-encode.md,vaapi-vulkan-import.md,render-d3d11.md.(Written by Claude Opus 5.5)
🤖 Generated with Claude Code