feat(runtime-v2): integrate LTX 2.5 audio-video generation

[jmccaffrey-nv]

Summary\n\n- add a concise LTX 2.5 synchronized text-to-audio-video application through the V2 runtime interface\n- emit finite TCHW uint8 video plus stereo 48 kHz AudioOutput in one StepResult\n- use one application-shared lazy backend with fresh per-session state and model/sequential/no-offload policies\n- pin the gated Lightricks/LTX-2.5-Diffusers checkpoint and the exact unreleased Diffusers commit required by the official model card\n- add CPU contracts, opt-in real-model smoke coverage, a six-case GPU benchmark, FFmpeg/FFprobe validation, JSON metrics, and a portable HTML gallery\n\n## Dependency\n\nBuilds on #526 for synchronized runtime audio output. This branch includes its two commits while #526 remains open; after #526 merges, those shared commits will fall out of this PR diff.\n\n## Validation\n\n- 363 V2 and adapter CPU tests passed; 1 real-GPU test deselected\n- real gated 9-frame 384x256 smoke passed with stereo 48 kHz audio\n- 6/6 real GPU matrix cases passed across 25, 121, and 241 frames and 768x512, 960x544, and 1280x736\n- all 54 codec, geometry, exact-frame, signal, motion, and A/V drift checks passed\n- A/V drift was 0.000667 seconds for every case\n- peak CUDA allocation ranged from 41.91 to 43.39 GiB on RTX PRO 6000 Blackwell\n- Ruff, formatting, type, lockfile, package version, entry-point, wheel, and sdist checks passed\n- standalone wheel metadata preserves the exact Diffusers Git SHA\n\n## Measured matrix\n\n| Case | Frames and resolution | Generation | Peak CUDA |\n| --- | ---: | ---: | ---: |\n| sync percussion | 25 at 768x512 | 48.81 s | 41.91 GiB |\n| dialogue portrait | 25 at 960x544 | 72.73 s | 41.98 GiB |\n| ocean wide | 25 at 1280x736 | 35.00 s | 42.22 GiB |\n| train medium | 121 at 768x512 | 43.14 s | 42.57 GiB |\n| market medium | 121 at 960x544 | 50.23 s | 42.85 GiB |\n| multishot maximum | 241 at 768x512 | 66.21 s | 43.39 GiB |\n\nThe integration evidence note records immutable source/model revisions, the exact benchmark invocation, complete media measurements, and artifact hashes.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

This PR extends runtime-v2 with synchronized normalized PCM output, transactional MP4 audio/video publication, and a standalone LTX 2.5 text-to-audio-video integration.

• Adds AudioOutput and session-level audio format contracts.
• Carries model audio through presentation and stages it for FFmpeg AAC muxing.
• Makes MP4 output transactional across runtime, session, application, encoder, and mux cleanup.
• Adds the LTX 2.5 application, pinned model dependency, validation, benchmarks, and packaging.

Confidence Score: 4/5

The LTX application’s enforced presentation policy is safe, but the generic runtime audio contract should prevent continuous UI presentation from creating desynchronized MP4 output before merging.

Audio sessions can retain the default 60 Hz continuous UI redraw while MP4 output timestamps every written frame at the 30 Hz model-step rate, extending video beyond the one-shot audio timeline and padding the remainder with silence.

Files Needing Attention: flashdreams/flashdreams/runtime_v2/session_desc.py, flashdreams/flashdreams/runtime_v2/mp4_output_sink.py

Important Files Changed

Filename	Overview
flashdreams/flashdreams/runtime_v2/session_desc.py	Adds paired audio format fields but leaves audio compatible with continuous frame re-presentation, enabling divergent MP4 video and audio timelines.
flashdreams/flashdreams/runtime_v2/session_runner.py	Adds transactional sink termination and orders application cleanup before output commit; continuous UI writes expose the audio timing issue for generic sessions.
flashdreams/flashdreams/runtime_v2/mp4_output_sink.py	Stages video and PCM, muxes AAC, and atomically publishes output, but derives media duration from UI frame writes at the model-step rate.
flashdreams/flashdreams/runtime_v2/mp4_audio.py	Implements bounded FFmpeg preflight/muxing and offset-aware interleaved PCM staging with explicit padding, truncation, and cleanup.
flashdreams/flashdreams/runtime_v2/presentation_manager.py	Delivers each chunk's audio once while retaining existing model-frame presentation semantics.
integrations_v2/t2av_ltx25/t2av_ltx25/app.py	Implements a finite LTX session with validated TCHW video, stereo 48 kHz audio, blocking backpressure, and new-frame-only presentation.
integrations_v2/t2av_ltx25/t2av_ltx25/backend.py	Loads the pinned Diffusers pipeline, applies offload policy, validates generated media, and converts output into runtime-compatible CPU tensors.
uv.lock	Locks the new integration and exact Diffusers Git revision alongside the regenerated workspace dependency graph.

Sequence Diagram

sequenceDiagram
  participant M as LTX Model Loop
  participant P as Presentation Manager
  participant U as UI Loop
  participant S as MP4 Sink
  participant F as FFmpeg
  M->>P: StepResult(video chunk, audio)
  loop Each UI tick
    P->>U: Presented model frame
    P-->>U: Audio once on first chunk frame
    U->>S: Presented StepResult
    S->>F: Encode video frame
    S->>S: Stage audio at sample_offset
  end
  S->>F: Finish video and encode/mux AAC
  F-->>S: Staged synchronized MP4
  S->>S: Atomic replace of target

Loading

_{Reviews (1): Last reviewed commit: "Merge origin/main into LTX 2.5 integrati..." | Re-trigger Greptile}

[greptile-apps]

Audio timeline desynchronizes

When an audio session retains ONLY_PRESENT_NEWEST with the default 60 Hz UI rate and 30 Hz step rate, the runtime writes the retained video frame on every UI tick but forwards each chunk's audio only once. Mp4OutputSink then timestamps all written frames at the step rate, doubling the video duration and padding the remainder with silence.

Knowledge Base Used: Runtime-v2 application execution