feat(LTX25-T2A-ONE-STAGE): text-to-audio, the one-stream forward, and the guided denoiser nothing had (#1005, #1013)

CMakeLists.txt

@@ -873,6 +873,12 @@ add_library(vllm STATIC
   # truncate-or-pad conform, and the frame-directory source. Its own TU for the
   # same reason the two above are: ltx2_video.cpp is a concurrent-edit lock.
   src/vllm/model_executor/models/ltx2_retake.cpp
+  # LTX-2.5 (ROW LTX25-T2A-ONE-STAGE, issue #1005): text-to-audio — the
+  # audio-only pipeline and its guided denoiser. Its own TU for the same reason
+  # the two above are, plus one this row adds: it is the only LTX pipeline that
+  # runs the DiT with `video = nullptr`, and keeping that call in one file is
+  # what makes "does anything else pass a video stream" a readable question.
+  src/vllm/model_executor/models/ltx2_t2a.cpp
   # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L6): the quantized loaders — the
   # FP8 and torchao-NVFP4 DiT arms, the torchao-NVFP4 text encoder, and the
   # load-time device staging GB10's ATS penalty makes the default.

docs/FEATURES.md

@@ -171,6 +171,8 @@ in `ltx2_text_encoder.cpp` is the call that would have to change.
 | LTX-2.5 Conv VAE decode arithmetic width | LTX-2.5 video VAE | `test_ltx2_vae` "the decode's convolution accumulates in f32", entering through `Ltx2VideoDecodeStreaming`; widening the accumulator to `double`, or deleting the production call site, each turns it RED | **f32**, the width `F.conv3d` uses at f32 AND bf16 (MEASURED). Was f64 at 8 sites ([#1008](https://github.com/mudler/vllm.cpp/issues/1008)). Conv sums BLOCKED per input channel, as torch's. STORAGE stays f32; bf16 owed |
 | LTX-2.5 Conv VAE decode threading | LTX-2.5 video VAE | `test_ltx2_vae` "the decode DISPATCHES its convolutions to the CPU threadpool" and "...BIT-IDENTICAL across thread counts", through `Ltx2VideoDecodeStreaming`; 34 golden margins UNCHANGED; TSan clean | **Parallel** over CONV output lines via `vt::cpu::ParallelForRows` ([#1009](https://github.com/mudler/vllm.cpp/issues/1009)). ~9x at 16-20 workers, contended box, 21-23% spread. Bit-identical at any count |
 | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated |
+| LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 18 mutations, 17 DETECTED (four by review of a conditional-only #1039 gate) and the 18th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only GUIDED arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name |
+| LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 ([#1039](https://github.com/mudler/vllm.cpp/issues/1039)) |
 | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) |
 | MTP speculation DEPTH (`num_speculative_tokens` > 1) | Qwen3.5/3.6 `mtp.*` heads | k=1..4 through the loader, greedy tokens unmoved, two witnesses per arm: the draft decode forwards the propose RAN, and whether the DELIVERED draft row varied with depth. `test_mtp_depth` 5/5, 63 assertions | Default stays k=1. NO speed claim at k>1. Drafts are proposed and verified, never ACCEPTED, and neither witness proves per-column provenance. Both await the owed DGX gate (#81) |
 | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on |

docs/USAGE.md

@@ -842,6 +842,74 @@ The two knobs beside it are per-GENERATION and therefore CLI and ABI only, becau
 `num_generated_keyframes` on the other pipelines, and `temporal_upsample_rounds`
 below.
 
+### LTX-2.5 text-to-audio: a render with no picture
+
+`--pipeline-kind t2a_one_stage` runs upstream's `T2AOneStagePipeline`, which
+generates a soundtrack and no video at all. The result carries an `audio.wav`,
+`frame_count = 0`, an empty frame directory and **no ffmpeg argv**, because there
+is nothing to mux.
+
+```sh
+ltx2-gen --dit ltx-2.5-dit.safetensors \
+         --audio-vae ltx-2.5-audio-vae-bf16.safetensors \
+         --encoder gemma4-12b-with-proj.safetensors --encoder-config gemma4.json \
+         --pipeline-kind t2a_one_stage --device cpu \
+         --frames 121 --steps 30 --prompt "rain on a tin roof, distant thunder" \
+         --workdir /tmp/t2a
+```
+
+**These file names are not a checkpoint pin, and no LTX-2.5 recipe in this
+document is.** None of them names a HuggingFace repo, a revision or a sha256,
+which AGENTS.md § *Say which weights, and from where* requires; MiniMax-H3 and
+MiniMax-Music3 below each carry a full table and LTX-2.5 carries none. That is
+campaign-wide and pre-existing rather than particular to this recipe, and it is
+recorded rather than invented, because no LTX-2.5 arm here has been rendered on
+real weights yet. Tracked by
+[#1048](https://github.com/mudler/vllm.cpp/issues/1048); read `--dit` above as
+"the LTX-2.5 transformer", which the other recipes on this page spell as
+`ltx-2.5-22b-distilled-fp8.safetensors` together with the `--dit-config` its
+missing `__metadata__` requires.
+
+**No `--video-vae` is needed**, and none is loaded: upstream's pipeline never
+constructs a video VAE. `--width` and `--height` are **refused** rather than
+ignored — upstream passes a 512x512 placeholder whose height and width it
+documents as unused, and only the frame count and the recipe's frame rate are
+read, to derive the duration.
+
+**It is the only GUIDED arm, and that changes what it costs and what it needs.**
+The distilled video recipes run one DiT forward per step. This one runs
+**three** by default — conditional, unconditional, and one with the audio
+self-attention perturbed (STG) — so it is roughly 3x the work per step, and it
+**requires a text tower**, because the unconditional pass conditions on the
+negative prompt. Loading with `prompt_embeds_path` alone gets a refusal naming
+`--audio-cfg-guidance-scale 1.0` as the way to turn the unconditional pass off.
+
+Six per-generation knobs mirror upstream's own CLI, and each takes the
+checkpoint generation's value when absent: `--negative-prompt`,
+`--audio-cfg-guidance-scale` (7.0), `--audio-stg-guidance-scale` (1.0),
+`--audio-rescale-scale` (0.7), `--audio-skip-step` (0) and `--audio-stg-blocks`
+(28 on the 2.3-and-later lineage), which is comma separated. A block index
+outside the DiT's own layer count is refused rather than clamped. There is no
+`modality_scale` knob: upstream pins it to 1.0 for this pipeline, because
+audio-only generation has no video modality to isolate.
+
+`--audio-rescale-scale` acts on the **denoised (x0) prediction**, not on the
+DiT's velocity, because upstream's guider sits behind an `X0Model` and combines
+already-converted tensors. The distinction is invisible at `0.0`, where the two
+readings agree exactly, and it changes the render at every other value — so a
+recipe or a script that was tuned against the velocity reading will not
+reproduce here at the default `0.7` (issue #1039).
+
+Being per-generation, those six reach the CLI and the C ABI and **not**
+`/v1/videos`, which forwards no per-generation extra to any engine (issue #928).
+`pipeline_kind` is a LOAD knob and does reach the server, so a server started
+with `--video-extra pipeline_kind=t2a_one_stage` renders every request as audio
+at the recipe's own guider values.
+
+**The accelerator is refused by name.** `device = 1` gets a refusal on this
+pipeline: the device forward takes both streams by reference and this pipeline
+has no video stream to give it. Use `--device cpu`.
+
 **What is not served.** `temporal_upsample_rounds` is defined and refused above
 `0`: the rounds loop that temporally doubles the latent, re-tiles the canvas and
 stitches it back is not ported. The refusal names it, and it names three things
@@ -2374,7 +2442,8 @@ or without the ComfyUI `model.diffusion_model.` prefix. Each family reads its ow
 knobs from `extras`. H3 takes `partition`. LTX-2.5 takes
 `audio_prompt_embeds_path` (the audio stream's conditioning, the twin of the
 seam's `prompt_embeds_path`, which carries the video stream), `pipeline_kind`
-(default `distilled_two_stage`), `model_version` (only for a checkpoint that
+(default `distilled_two_stage`; also `one_stage`, `dmd2`, `dfr`, `retake` and
+`t2a_one_stage`), `model_version` (only for a checkpoint that
 declares none), `dit_config_path`, `encoder_config_path`,
 `allow_unported_modules`, `max_phase`, `prompt_embeds_valid_rows`,
 `upsampler_path` and `duration_head_path`. An extra a family does not define is

examples/ltx2_gen/main.cpp

@@ -169,7 +169,21 @@ const char* Need(int argc, char** argv, int i, const char* flag) {
       "--width, --height and --frames are refused alongside it. --regenerate-video 0\n"
       "freezes the clip instead; --regenerate-audio has no effect while the source is\n"
       "a frame folder, because a folder carries no audio and both of upstream\'s audio\n"
-      "predicates test for one.\n");
+      "predicates test for one.\n\n"
+      "TEXT-TO-AUDIO renders a soundtrack and NO PICTURE. --pipeline-kind\n"
+      "t2a_one_stage selects it; the result carries an audio.wav, zero frames and no\n"
+      "ffmpeg argv, because there is nothing to mux. --video-vae is not needed and\n"
+      "--width/--height are refused: upstream passes a 512x512 placeholder whose\n"
+      "height and width it documents as unused, and only --frames and the recipe\'s\n"
+      "frame rate are read, to derive the DURATION. Unlike the distilled video\n"
+      "recipes this one is GUIDED: it runs three DiT forwards per step by default\n"
+      "(conditional, unconditional, and one with the audio self-attention perturbed),\n"
+      "so it needs a text tower for the negative prompt. --negative-prompt,\n"
+      "--audio-cfg-guidance-scale, --audio-stg-guidance-scale, --audio-rescale-scale,\n"
+      "--audio-skip-step and --audio-stg-blocks are upstream\'s own flags; absent, each\n"
+      "takes the checkpoint generation\'s own value. --audio-stg-blocks is comma\n"
+      "separated and a block index outside the DiT\'s layer count is refused rather\n"
+      "than clamped. The accelerator is REFUSED by name on this pipeline.\n");
   std::exit(code);
 }
 
@@ -186,6 +200,10 @@ int main(int argc, char** argv) {
   // RETAKE (row LTX25-RETAKE, #924): a source clip DIRECTORY and the window to
   // regenerate. `--ref-video` is a directory of frame_%06d.ppm, not a container.
   std::string ref_video, retake_start, retake_end, retake_fps, regen_video, regen_audio;
+  // TEXT-TO-AUDIO (row LTX25-T2A-ONE-STAGE, #1005): one flag per argument of
+  // upstream's `default_1_stage_t2a_arg_parser` (utils/args.py:1070-1120).
+  std::string negative_prompt, audio_cfg_scale, audio_stg_scale, audio_rescale;
+  std::string audio_skip_step, audio_stg_blocks;
 
   // The extras are BORROWED by the load call, so the strings must outlive it.
   // Kept as two parallel vectors of owned strings plus the char* views the ABI
@@ -262,6 +280,21 @@ int main(int argc, char** argv) {
       retake_end = Need(argc, argv, ++i, "--retake-end-time");
     else if (f == "--retake-frame-rate")
       retake_fps = Need(argc, argv, ++i, "--retake-frame-rate");
+    // TEXT-TO-AUDIO (#1005). Selected by `--pipeline-kind t2a_one_stage`, which
+    // is a LOAD extra; these six are per-generation and are refused by name on
+    // any other pipeline rather than accepted and ignored.
+    else if (f == "--negative-prompt")
+      negative_prompt = Need(argc, argv, ++i, "--negative-prompt");
+    else if (f == "--audio-cfg-guidance-scale")
+      audio_cfg_scale = Need(argc, argv, ++i, "--audio-cfg-guidance-scale");
+    else if (f == "--audio-stg-guidance-scale")
+      audio_stg_scale = Need(argc, argv, ++i, "--audio-stg-guidance-scale");
+    else if (f == "--audio-rescale-scale")
+      audio_rescale = Need(argc, argv, ++i, "--audio-rescale-scale");
+    else if (f == "--audio-skip-step")
+      audio_skip_step = Need(argc, argv, ++i, "--audio-skip-step");
+    else if (f == "--audio-stg-blocks")
+      audio_stg_blocks = Need(argc, argv, ++i, "--audio-stg-blocks");
     else if (f == "--regenerate-video")
       regen_video = Need(argc, argv, ++i, "--regenerate-video");
     else if (f == "--regenerate-audio")
@@ -330,7 +363,20 @@ int main(int argc, char** argv) {
                          std::make_pair("retake_end_time", &retake_end),
                          std::make_pair("retake_frame_rate", &retake_fps),
                          std::make_pair("regenerate_video", &regen_video),
-                         std::make_pair("regenerate_audio", &regen_audio)}) {
+                         std::make_pair("regenerate_audio", &regen_audio),
+                         // TEXT-TO-AUDIO (#1005). One flag per upstream CLI
+                         // argument (`default_1_stage_t2a_arg_parser`,
+                         // ltx-pipelines utils/args.py:1070-1120). They are
+                         // per-generation, so they ride this array rather than
+                         // the load one; `--pipeline-kind t2a_one_stage` is the
+                         // LOAD knob that selects the pipeline, and supplying
+                         // these without it is refused by name.
+                         std::make_pair("negative_prompt", &negative_prompt),
+                         std::make_pair("audio_cfg_guidance_scale", &audio_cfg_scale),
+                         std::make_pair("audio_stg_guidance_scale", &audio_stg_scale),
+                         std::make_pair("audio_rescale_scale", &audio_rescale),
+                         std::make_pair("audio_skip_step", &audio_skip_step),
+                         std::make_pair("audio_stg_blocks", &audio_stg_blocks)}) {
     if (kv.second->empty()) continue;
     gen_keys.emplace_back(kv.first);
     gen_values.push_back(*kv.second);