feat(runtime-v2): add synchronized audio output

[jmccaffrey-nv]

Summary

Extracted from #523 as the model-neutral prerequisite for the MiniMax H3 V2 integration. PR #523 will be stacked on this exact branch head.

• Add typed normalized-PCM audio to the V2 StepResult and SessionDesc contracts.
• Forward each model chunk audio payload exactly once through the default presentation path.
• Publish synchronized H.264/AAC-LC MP4 using external host ffmpeg and ffprobe executables.
• Align audio to the exact written-frame timeline, including forward gaps, padding, and truncation.
• Make MP4 publication transactional and preserve existing targets on inference, encode, mux, close, or cleanup failures.
• Bound mux timeouts and cleanup retries, retain unreaped child ownership, and keep Python 3.10 exception cleanup compatible.
• Reject audio sessions from WebRTC until that sink supports the declared contract.

This PR contains no MiniMax H3 or Diffusers code, adds no workspace package, and changes no dependency lock data.

Validation

• Focused V2 audio/runtime suite with real external-ffmpeg round trips: 174 passed.
• Exact repository CPU tier: 1,975 passed, 2 skipped, 340 deselected.
• Full pre-commit gate: passed, including Ruff check/format, uv lock validation, package-version sync, and full-workspace ty.
• Standalone FlashDreams sdist and wheel build: passed.
• DCO: 2 / 2 commits signed off.

Stack

• Extracted from feat(minimax-h3): integrate native video-audio apps with V2 #523.
• feat(minimax-h3): integrate native video-audio apps with V2 #523 remains the H3 integration and will use this branch as its temporary base until this prerequisite merges.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[jmccaffrey-nv]

Review provenance: this prerequisite contains the model-neutral portions of the Claude review originally posted on #523, including Python 3.10-safe exception notes, duration-scaled mux timeout and terminate/kill recovery, once-only application close, per-sink transaction semantics, exact FPS/audio alignment, and retained cleanup ownership. The original disposition matrix remains at #523 (comment). H3-specific and media-input findings remain on #523.

[jmccaffrey-nv]

Split validation complete on head 264840e. Local: focused V2 audio/runtime 174 passed; full ci_cpu 1,975 passed, 2 skipped, 340 deselected; every pre-commit hook passed; standalone FlashDreams sdist/wheel built. Fresh GitHub checks: CPU passed in 8m38s, GPU passed in 17m8s, docs/REUSE/OSRB passed. The PR remains draft and mergeable.

[jmccaffrey-nv]

Thanks — this discussion helped separate the model-result contract from the behavior of individual transports. My proposed disposition for #526 is:

1. Keep AudioOutput as a dedicated typed field. It represents decoded, raw normalized PCM, not an arbitrary tensor and not model-selected compressed bytes. The current contract is a floating-point [channels, samples] tensor in [-1, 1], a sample rate, and an absolute sample_offset. Mono/stereo channel count is declared in SessionDesc. AAC/Opus/WebM are sink concerns, so there is intentionally no codec field in AudioOutput; if we later need channel layouts beyond implicit mono or stereo L/R, that should be an explicit contract extension.

2. Treat each payload as a segment, not necessarily a complete track. One model StepResult may carry one PCM segment for the media interval produced by that model step. A bidirectional model that generates a whole clip in one step can return one whole-clip segment; an autoregressive model can return many short segments. The V2 audio contract does not impose a universal chunk size: the model/integration decides how many frames and samples one IModelLoop.step() produces, and presentation forwards each audio payload exactly once rather than coalescing several payloads.

3. Use the session timeline as the synchronization authority. sample_offset places the first sample on an absolute audio timeline. Producers should emit audio for the same logical interval as their model video chunk, within the model's frame/audio alignment. Exact per-result equality is not always possible from sample_count == frame_count * sample_rate / fps, and the default UI also presents a multi-frame model chunk one frame at a time while forwarding its audio once with the first presented frame. For that reason the absolute sample interval is authoritative. The MP4 sink rejects backward/overlapping segments, fills forward gaps with silence, and at close pads or truncates the accumulated PCM to round(written_frames * sample_rate / fps).

4. Be precise about what feat(runtime-v2): add synchronized audio output #526 validates today. It validates typed PCM segments, once-only forwarding, multiple incremental sink writes, offsets/gaps/overlap rejection, final A/V alignment, and real H.264/AAC-LC MP4 round trips. The MP4 sink ingests audio incrementally but publishes the final file atomically only after close; it is not a progressive-download/fMP4 implementation. It resolves a host-installed ffmpeg executable and performs the exact AAC-LC preflight before inference. WebRTC currently rejects an audio session during open() rather than silently dropping audio.

5. Validate live streaming in a transport follow-up, independently of H3. MiniMax H3 currently produces one bidirectional whole-clip result (roughly 5–15 seconds), so it proves synchronized file output but is not a good low-latency streaming oracle. The useful test is a deterministic synthetic/autoregressive producer that emits one short video+PCM chunk at a time, with no transport-side coalescing. The WebRTC follow-up should add an audio track and verify:

   • each chunk is enqueued as soon as its StepResult arrives;
   • RTP/audio timestamps are monotonic and derived from the sample timeline;
   • forward gaps become silence and overlaps are rejected;
   • the browser starts playback before generation completes;
   • steady-state A/V skew and final-tail alignment stay within an agreed bound; and
   • backpressure/drop behavior is explicit for synchronized audio and video.

So I believe the AudioOutput shape is suitable for a future streaming autoregressive model, but #526 should be read as establishing the model-neutral chunk/timeline contract and synchronized file path. WebRTC audio and measured one-chunk-at-a-time latency are follow-up validation, not claims of this PR.