diff --git a/.agents/issue-index.md b/.agents/issue-index.md index e56bd969b..acb79c9e5 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -293,3 +293,11 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1029](https://github.com/mudler/vllm.cpp/issues/1029) | `ENG-EXPERT-STREAM` | `IsCudaKeepQuantSupported` (`src/vt/cuda/cuda_quant_dot.cu`) returns true for `kIQ1_S` and `kIQ1_XXXS` since [#967](https://github.com/mudler/vllm.cpp/pull/967), and THREE dispatch switches consume that predicate while #967 extended only the dense one. `MatmulBTQuantGroupedKernelCuda` uses the predicate to SKIP its CPU fallback and then dispatches through a `switch (w)` with no case for either dtype and no `default:`: it quantizes the activation, launches NOTHING, returns, and `CheckCuda(cudaGetLastError())` reports success because there was no launch to fail, so the output tensor is never written. Measured on GB10 by an independent review through a poisoned output buffer: both IQ1 encodings leave `-12345` in place at NMSE `4.58e6` / `9.96e6` against the CPU oracle, with `iq2_s` passing as the control. Reachable by DEFAULT (`qwen3_5_gguf_weights.cpp` -> `qwen3_5.cpp` `KqGrouped` -> `vt::MatmulBTQuantGrouped`, `VT_QWEN35_GROUPED_MOE` on) on the 96.92 % of `Qwen3.8-2.4T` those two encodings make up, so #967 converted correct-but-slow into silently wrong. Same omission in the fused `MoeGateUpSwiGLUGroupedCuda` seam, where it turned a NAMED refusal into silence. It landed green because the CUDA grouped dispatch had NO test: `grep -rl MatmulBTQuantGrouped tests/` found two files and neither mentioned `kCUDA`. Repaired by adding both arms to both grouped switches, a `default:` that THROWS and names the dtype on all three (the general fix: past that predicate there is no fallback left), and a grouped + fused CUDA gate over the dense gate's case table driven through a POISONED output buffer. Also seals the device codebooks against the CPU tables, which `cuda_quant_iq_tables.cuh` claimed from the day it landed and no test did: 266 of 2048 `d_iq1s_grid` entries (13.0 %) are never addressed by the gate's own `mt19937(0x5EED)` stream. Device evidence is OWED and the issue stays open for it: `dgx.casa` was unreachable throughout and this box has no CUDA device, so only the CPU arm ran. Spec [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1055](https://github.com/mudler/vllm.cpp/issues/1055) | `ENG-RECORD-CONFLICT-SURFACES` | `main` went RED on `check-public-doc-tables.py` at `e34d71379`: two prose paragraphs added beside a two-character source fix took docs/BENCHMARKS.md to 36 paragraphs of 35 and docs/FEATURES.md to 22 of 21, and the checker runs both in the `pre-push` hook and at `.github/workflows/ci.yml:160`, so every branch in the repository inherited a red it did not cause. Fixed in flow by folding each paragraph into the keyed row its content belongs to, which is what the checker's own message prescribes: the Apple Clang build disposition into the docs/BENCHMARKS.md `Open gaps` table, the Apple Clang platform fact into the docs/FEATURES.md backend table. No paragraph was deleted and no budget was raised. The deeper defect is the budget itself, a whole-page count on a shared file, which AGENTS.md Records names as the anti-pattern (`Limit an entry, not a shared file`); redesigning it is `ENG-RECORD-CONFLICT-SURFACES` scope, whose spec already carries the obligation | bug | | [#1044](https://github.com/mudler/vllm.cpp/issues/1044) | `LTX25-DECODE-THREADS` | The three parallel dispatch sites [#1009](https://github.com/mudler/vllm.cpp/issues/1009) added to the LTX-2.5 conv video VAE share ONE work-stealing cursor, so reverting any SINGLE one of them is detected by nothing: `CausalConv3d`'s padding gather (`src/vllm/model_executor/models/ltx2_video_vae.cpp:170 @ 249418305`), its output nest (`:218`) and `Linear3d` (`:276`). The instrument, the case "the decode DISPATCHES its convolutions to the CPU threadpool", reads `Threadpool::ChunkAdd(0)` and that cursor is seeded once per pool (`src/vt/cpu/cpu_threadpool.cpp:438 @ 249418305`, advanced at `:455`), so it gates "at least ONE of the three dispatches", never each site. MEASURED as T1/T2/T3 in [`ltx25-decode-threads.md`](specs/ltx25-decode-threads.md) §8.6 and reproduced independently by that row's reviewer: each single-site revert BUILT with 0 errors and left ctest at exit 0 and 42/42 + 10/10 green; only reverting all three (T0) goes red. NOT a correctness hole -- 34 golden margins were byte-identical, the bit-identity case `memcmp`s five worker counts, and ThreadSanitizer is clean against an 84-race positive control -- but a site can silently go serial again and only a wall-clock nobody runs in CI would notice. Closing it needs a per-dispatch `Threadpool::RunCount()` bumped in `Run()` (`src/vt/cpu/cpu_threadpool.h:112 @ 249418305`; no such counter exists) and an EXACT expected count rather than `> 0`, plus a fixture carrying a `res_x_y` block, because `MakeLtx2ThreadFixture` sets `decoder_blocks = {}` and `Linear3d` is unreachable without one. A new gate needs its own red-before evidence and its own fresh review, so it is a row rather than an in-flow repair. Listed under `## 7. Owed` in [`ltx25-decode-threads.md`](specs/ltx25-decode-threads.md) | verification | +| [#1005](https://github.com/mudler/vllm.cpp/issues/1005) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 text-to-audio (`T2AOneStagePipeline`, `t2a_one_stage.py:43`, `__call__` at `:109` @ `fd4ded7f`) is absent, and the three blockers are not the ones a reader would guess. (a) `Ltx2DitForward` refuses a one-stream call at `src/vllm/model_executor/models/ltx2_dit.cpp:765 @ 332aed738`, citing a weight contract that describes a checkpoint T2A never loads — upstream reads the ordinary AudioVideo FILE through `LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP` (`model_configurator.py:228-239`). (b) The same message advises `enabled=false` as the substitute, and it is NOT: `run_v2a = run_ax and (video is not None and vx.numel() > 0)` (`transformer.py:269`) tests PRESENCE, not `enabled`, so a disabled-but-present video stream still feeds v2a cross attention and still returns a finished waveform; our port mirrors that polarity at `src/vllm/model_executor/models/ltx2_dit.cpp:251 @ 332aed738`. (c) The engine has NO guided denoiser at all — `git grep -n 'guid\\|cfg_scale' src/vllm/multimodal/ltx2_video.cpp` is 0 against 66 for `ltx2` in the same file as the control — while T2A defaults to `cfg_scale=7.0` and `stg_scale=1.0`, i.e. THREE forwards per step. Spec [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | feature | +| [#1013](https://github.com/mudler/vllm.cpp/issues/1013) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 `one_stage` denoised from ZEROS. `OneStagePhase` (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1066 @ 332aed738`) left `Ltx2PhaseRecipe::noise_scale` at the struct default of 0.0, and 0.0 is not "no extra noise": `Ltx2GaussianNoise` is `latent + noise_scale * (noise - latent)` (`:218 @ 332aed738`), so the state stayed exactly as `create_initial_state` wrote it, which with no initial latent is all zeros. Upstream `ModalitySpec.noise_scale` defaults to 1.0 (`ltx-pipelines/utils/types.py:110 @ fd4ded7f`) and `TI2VidOneStagePipeline.__call__` constructs BOTH specs without it (`ti2vid_one_stage.py:233-239`); the two neighbouring recipes set it explicitly, which is what made the omission legible. No gate saw it because every end-to-end test loads `distilled_two_stage`, and a zero-initialized denoise still returns a finite clip of the right size, frame count and sample rate. FOUND and FIXED in flow by row `LTX25-T2A-ONE-STAGE`, whose `t2a_one_stage` rows are built FROM `OneStageRecipe` and would have inherited it. `dmd2` leaves the same field at 0.0 and is deliberately NOT corrected by analogy (its source is vLLM-Omni, not checked out here); that half is listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | +| [#1031](https://github.com/mudler/vllm.cpp/issues/1031) | `ENG-EXPERT-STREAM` | CLOSED as a duplicate of [#1022](https://github.com/mudler/vllm.cpp/issues/1022), and this row is the corrected one rather than the filed one. As filed it said `check-agent-record` and `test_check_agent_record` are RED on `origin/main` because `.agents/issue-index.md` lists issue #995 twice, and that the repair needed a contract decision plus a checker-semantics spec. It did not: [#1022](https://github.com/mudler/vllm.cpp/issues/1022) had already read the two #995 rows and found neither WELL-FORMED, and `ff264cb82` (PR [#1025](https://github.com/mudler/vllm.cpp/pull/1025)) landed the repair on `main` before this branch merged it. Measured at this branch's head rather than inferred from the merge: `python3 scripts/check-agent-record.py` prints `agent record OK: ENGINE=156 MODEL=377 QUANT=82 KERNEL=51 BACKEND=83` and exits 0. CORRECTED IN PLACE, and that is a narrow exception argued here rather than a licence: the row had not landed, this branch added it, and the net diff against `origin/main` is still additions only — which `scripts/check-issue-index-append-only.py --base origin/main` is what checks. Once it lands `merge=union` makes it permanent and un-correctable, so leaving a filed-and-refuted claim in the record was the more expensive option | bug | +| [#1039](https://github.com/mudler/vllm.cpp/issues/1039) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 T2A combined its guidance passes in VELOCITY space. Upstream hands the denoiser an `X0Model` (`ltx-pipelines/utils/blocks.py:480-482 @ fd4ded7f`) whose `forward` returns `to_denoised(latent, v, timesteps)` (`ltx-core/model/transformer/model.py:601-604`, `to_denoised` at `ltx-core/utils.py:39-52`), so `_guided_denoise` combines DENOISED tensors — `all_v, all_a = transformer(...)` at `utils/denoisers.py:188` and `audio_guider.calculate(...)` at `:203`. `Ltx2T2aGenerate` combined raw DiT velocities and applied `ToDenoised` once afterwards. `MultiModalGuider.calculate`'s LINEAR terms are invariant under `x0 = latent - sigma*v`, so the two forms agree exactly while `rescale_scale == 0`; the RESCALE branch (`guiders.py:268-271`) is not, because upstream's `std(x0_cond)/std(x0_pred)` scales the whole x0 to `f*(latent - sigma*v)` where scaling the velocity gives `latent - sigma*f*v` — a difference of `(f - 1) * latent`, non-zero wherever the latent is, which on this path is everywhere. `rescale_scale = 0.7` is the SHIPPED T2A default (`utils/constants.py:63`, `utils/args.py:1101-1106`), so every default render took the divergent branch. No gate saw it: the forward counts, perturbed blocks, latent absmax, waveform length, channel count and sample rate are identical between the two forms. FOUND by review of PR [#1032](https://github.com/mudler/vllm.cpp/pull/1032) and FIXED in flow on the same branch, before it landed. The VIDEO arm is unaffected and the reason is recorded rather than assumed: `Ltx2MultiModalGuidance` has exactly ONE production caller and the joint driver runs a single unguided forward per step. Spec [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | +| [#1048](https://github.com/mudler/vllm.cpp/issues/1048) | — | LTX-2.5 ships with NO checkpoint pin. `docs/USAGE.md` names six LTX-2.5 artifacts by bare file name and gives no HuggingFace repo, no revision and no sha256 for any of them: `:663-670` and `:2183-2188` on `origin/main` at `d1b0ea3a8`, plus the text-to-audio recipe [#1005](https://github.com/mudler/vllm.cpp/issues/1005) added at `:853-857`. AGENTS.md § *Say which weights, and from where* requires file name, size, exact repo AND revision, grouped by arm, with a sha256 for a quantized artifact. Campaign-wide and PRE-EXISTING rather than introduced by that row, measured rather than asserted: `grep -n sha256 docs/USAGE.md` returns two checkpoint hashes and BOTH belong to MiniMax-Music3 (`:3127`, `:3269`), while MiniMax-H3 (`:1950-1993`) and MiniMax-Music3 (`:3123-3149`) each carry a full repo + revision + sha256 table and LTX-2.5 carries none anywhere. RECORDED AND DELIBERATELY NOT FABRICATED: no LTX-2.5 row in the campaign claims a render on real weights — `dgx.casa` is down and every LTX-2.5 gate runs on a reduced fixture — so there is no checkpoint any of them was gated against to pin, and inventing a repo id would be worse than the gap. The real-checkpoint render owed by [#644](https://github.com/mudler/vllm.cpp/issues/644) is what closes it. Two smaller things fall out of the same gap and belong to the same change: the t2a recipe names `ltx-2.5-dit.safetensors`, a file name appearing nowhere else in the tree, where every other LTX-2.5 recipe names `ltx-2.5-22b-distilled-fp8.safetensors` plus the `--dit-config` its missing `__metadata__` requires; and whether that recipe runs at all without `--dit-config` is unverified. Found repairing the fresh review of [#1039](https://github.com/mudler/vllm.cpp/issues/1039) on PR [#1032](https://github.com/mudler/vllm.cpp/pull/1032). Listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | +| [#1049](https://github.com/mudler/vllm.cpp/issues/1049) | — | `Ltx2Guidance` (`src/vllm/model_executor/models/ltx2_pipeline.cpp:526 @ c1fe35592`) is dead in production: `git grep -n 'Ltx2Guidance(' -- src include examples tests` returns the declaration (`ltx2_pipeline.h:330`), the definition, and ONE call, at `tests/vllm/models/test_ltx2_pipeline.cpp:710`. It is also the only path to two ported guiders — `Ltx2CfgDelta` (`ltx2_pipeline.cpp:532`, plus `test_ltx2_pipeline.cpp:615`) and `Ltx2StgDelta` (`:534`, plus `:630`) — so both are reachable from no product entry point either. `Ltx2BatchedPerturbationConfig` (`ltx2_pipeline.h:380`) is the same shape from a different direction: constructed only at `test_ltx2_pipeline.cpp:832-859`, while the LTX-2.5 text-to-audio path perturbs through `Ltx2DitPerturbation`, a different type, and no other path perturbs at all. This is the test-only-driver shape [`reachability.md`](reachability.md) names. All four landed with #641. `Ltx2MultiModalGuidance` was the fourth member of the set and is no longer one: [#1005](https://github.com/mudler/vllm.cpp/issues/1005) gave it a production call site in `ltx2_t2a.cpp`. Closing this means either routing a product path through `Ltx2Guidance` with a configured `Ltx2GuiderKind` — `Ltx2GuiderSigmaBin` and `Ltx2GuiderParamsForSigma` are already ported beside it — or retiring the unreached arms into `.agents/completed/` with their provenance. Found repairing the fresh review of [#1039](https://github.com/mudler/vllm.cpp/issues/1039) on PR [#1032](https://github.com/mudler/vllm.cpp/pull/1032), where an earlier draft of that row's spec §6b claimed the row ended all four test-only drivers; §6b now carries the measured table. Listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | +| [#1050](https://github.com/mudler/vllm.cpp/issues/1050) | — | The guider rescale's `std` comment states a consequence that cannot exist. `src/vllm/model_executor/models/ltx2_pipeline.cpp:505-506 @ c1fe35592`, repeated at `include/vllm/model_executor/models/ltx2_pipeline.h:319-322`, says torch's `std` is the UNBIASED (N-1) estimator by default and "the biased one would be a small, everywhere, resolution-dependent gain error that no shape or finiteness check can see". `factor_raw` is `unbiased_std(cond) / unbiased_std(pred)`, two `std`s over the SAME `count`, so the divisor cancels exactly: `sqrt(ss_c/(n-1))/sqrt(ss_p/(n-1)) == sqrt(ss_c/ss_p) == sqrt(ss_c/n)/sqrt(ss_p/n)`. There is no gain error, small or otherwise, and nothing about it is resolution-dependent; the two forms differ only by f32 rounding in the divide. Worth a record rather than a silent correction because the comment tells the next reader a gate is needed there and it is not: the fresh review of [#1039](https://github.com/mudler/vllm.cpp/issues/1039) mutated the estimator to the biased form and it survived — correctly, because it is an IDENTITY — and a survivor at that site otherwise reads as a blind instrument and costs another investigation. The CODE is right as written and should stay `unbiased_std`, because the name is what mirrors torch even where the ratio does not care; the COMMENT is the defect. Pre-existing from `cefacd2d0` (#641). Found repairing that review on PR [#1032](https://github.com/mudler/vllm.cpp/pull/1032) and out of scope there under its explicit exclusions. Listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | +| [#1052](https://github.com/mudler/vllm.cpp/issues/1052) | — | `tests/vllm/v1/test_engine_core_proc.cpp:481` ("EngineCoreProc: immediate shutdown aborts in-flight requests") searches for the abort frame over a FIXED budget of 1000 dequeues while a `max_tokens=100000` request keeps the busy loop producing token deltas, so nothing bounds how many frames precede the abort and the budget is a bet on scheduling. MEASURED at `37e680cab`, same binary throughout, CPU-only Release on 20 cores: **2 failures in 3 `ctest -j4` runs** of the full 492-test suite (`CHECK( abort_seen ) is NOT correct!`), **0 in 25 solo runs** on an idle box at load 3.34, **0 in 25 solo runs against 20 spinning processes**, and 0 in two `ctest -R '^test_engine_core_proc$'` runs (`Passed 0.03 sec`). So CPU pressure alone does not reproduce it; it needs the `-j4` harness. The third `-j4` run failed `test_cpu_threadpool` INSTEAD, which is on the same load-dependent list, so the IDENTITY of the failing test rotates between runs of an unchanged binary and both pass alone with exit 0. NO ISSUE NAMED THIS TEST: PR [#1032](https://github.com/mudler/vllm.cpp/pull/1032)'s body attributed its flake to [#294](https://github.com/mudler/vllm.cpp/issues/294), which is a different defect in a different test (`test_async_llm` reusing an aborted request id), and a misattributed flake is worse than an untracked one because the next reader checks the citation, finds an open issue about something else, and stops looking. The assertion guards a real guarantee (an in-flight request gets a `kAbort` finish on immediate shutdown); the 1000-frame budget is the part that is a guess. Found repairing the fresh review of [#1039](https://github.com/mudler/vllm.cpp/issues/1039), on a branch that touches no file under `tests/vllm/v1/` or `src/vllm/v1/`. Listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | diff --git a/.agents/specs/ltx25-t2a-one-stage.md b/.agents/specs/ltx25-t2a-one-stage.md new file mode 100644 index 000000000..d3d4a5628 --- /dev/null +++ b/.agents/specs/ltx25-t2a-one-stage.md @@ -0,0 +1,885 @@ +# LTX-2.5 — text-to-audio (`T2AOneStagePipeline`) + +Row: `LTX25-T2A-ONE-STAGE`. Campaign: [`ltx-2-5.md`](ltx-2-5.md) (operator-owned; +**not edited by this row**). Issue: +[#1005](https://github.com/mudler/vllm.cpp/issues/1005). Parent campaign issues: +[#644](https://github.com/mudler/vllm.cpp/issues/644), +[#435](https://github.com/mudler/vllm.cpp/issues/435). + +Upstream pin: + +| Reference | Revision | +|---|---| +| Lightricks/LTX-2 (`packages/ltx-core`, `packages/ltx-pipelines`) | `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca` | + +Read from a local checkout at that revision. `git rev-parse HEAD` was run on a +clean tree (`git status --short` empty) before any anchor below was taken. + +--- + +## 0. Honesty statement — what this row does and does not claim + +This row makes LTX-2.5 render **audio with no video**, from a prompt, through +the shared video seam. It is the first path in this tree that returns a +`VideoResult` carrying zero frames. + +**Upstream ships no tests at this pin.** `find /home/mudler/_git/LTX-2 -name +'test_*.py'` returns **0**; the broader `find . -iname '*test*' -not -path +'./.git/*'` that row `LTX25-A2V-AUDIO-INPUT` ran returns empty as well. So "port +the upstream tests in the same change" has nothing to port, and the obligation +becomes what §5 does instead: pin upstream's *behaviours* — its constants, its +predicates, its refusals — each against a `file:line` anchor, and tie at least +one assertion to a **local** fact so the suite can see a claim about this tree go +stale. This is stated rather than quietly skipped. + +**No render on real weights is claimed.** The row is gated on the reduced +fixture. The GPU is out of bounds for this row (a render ladder holds +`dgx.casa`, and the box OOM-reboots when its 119 GiB unified pool is exhausted), +so the real-checkpoint T2A render is owed and named under `## Owed`. + +**The device arm is refused by name, not served.** See §7. + +--- + +## 1. Does the audio-only shape fit `Ltx2VideoEngine`? YES — and here is the +## derivation, because the opposite answer was the expected one + +The dispatch that opened this row expected a possible `NEEDS_DECISION` on the +entry point. It is not needed, and the reason is upstream's own shape rather +than a convenience here. + +**Upstream expresses T2A's duration through a VIDEO pixel shape.** +`t2a_one_stage.py:37-40` declares `_AUDIO_ONLY_PLACEHOLDER_RES = 512` and says in +as many words that "Audio-only generation reads ``frames`` and ``fps`` from the +pixel shape via ``AudioLatentShape.from_video_pixel_shape`` (height/width are +unused)". `__call__` then passes `width=512, height=512, frames=num_frames, +fps=frame_rate` into the SAME `DiffusionStage.__call__` every video pipeline +calls (`:163-167`, against `utils/blocks.py:501-512`). So the request shape T2A +needs is exactly the request shape `VideoGenParams` already carries: +`num_frames`, `frame_rate` (through `duration_seconds` or the recipe), `steps`, +`seed`, `prompt`, `output_dir`. + +**And `VideoResult` can already say "no picture".** It carries `frame_dir`, +`audio_path`, `frame_count`, `sample_rate` and `mux_argv` as independent fields +(`include/vllm/multimodal/video_engine.h`, `struct VideoResult`). An audio-only +result is `frame_count = 0`, an empty `frame_dir`, an `audio_path`, and an EMPTY +`mux_argv` — there is nothing to mux, and composing an ffmpeg argv over a frame +pattern that matches no file would hand the caller a command that fails. + +So the seam represents the behaviour, and AGENTS.md §"Shared seams" then binds +the other way round: a distinct entry point would be the parallel path that +section forbids. What this row does NOT do is bend the video path: the audio-only +render lives in its own translation unit (§3), mirroring upstream's own file +structure, and `Generate` branches to it before it resolves any video geometry. + +--- + +## 2. The gap, derived at `332aed738` + +Three blockers, each named by the symbol that has to change. None of them is +"the pipeline file is missing". + +### 2.1 `Ltx2DitForward` refuses a one-stream call, and its stated reason does not describe T2A + +`src/vllm/model_executor/models/ltx2_dit.cpp:765 @ 332aed738` is + +``` +VT_CHECK(video != nullptr && audio != nullptr, "... LTXModelType.VideoOnly and + LTXModelType.AudioOnly carry a different weight contract and are not ported"); +``` + +Re-derived at this tree rather than inherited. The reason is a claim about the +**checkpoint**, and it is true of a checkpoint that was SAVED as an `AudioOnly` +model. T2A does not load one. `t2a_one_stage.py:81-93` calls +`DiffusionStage.from_checkpoint(model_paths.transformer(), ..., +model_configurator=LTXAudioOnlyModelConfigurator, +model_sd_ops=LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP)`: the FILE is the ordinary +AudioVideo transformer, and the SDOps map (`model_configurator.py:228-239`) +merely restricts which of its keys are read — `audio_attn1`, `audio_attn2`, +`audio_ff`, `audio_patchify`, `audio_proj_out`, `audio_adaln_single`, +`audio_prompt`, `audio_scale_shift_table`. The comment at `:78-82` of the +pipeline says why: so "the video weights are never even read from disk". The +weight contract `EnumerateLtx2DitTensors` describes is therefore the one T2A's +file satisfies, and the refusal's cause is not this row's blocker. + +**What is left of that refusal after the lift is stated rather than dropped.** A +checkpoint containing ONLY the audio subset still cannot be loaded here, because +`Ltx2LoadDitFromSafetensors` enumerates the AV contract. That half stays, and it +moves to where it is true — the loader — instead of guarding a forward that +already handles the case. + +**The forward already handles it.** Every line of `Ltx2DitForward` below the +check is written against `video != nullptr` / `have_both` +(`ltx2_dit.cpp:786-869 @ 332aed738`), and `Ltx2TransformerBlockForward` computes +`run_vx` / `run_a2v` / `run_v2a` from `video_x != nullptr && tv > 0` +(`ltx2_dit.cpp:248-251 @ 332aed738`). Lifting the check reaches a path that was already +written; it does not add one. + +### 2.2 `enabled = false` is NOT the same shape, and the difference renders + +The refusal's own message says to "use `enabled` to run one stream of an AV +model", and `include/vllm/model_executor/models/ltx2.h:506-508 @ 332aed738` repeats it. That +advice is correct for the A2V and joint pipelines. **It is wrong for T2A**, and +this is the detail that fails silently if guessed. + +Upstream's predicate is + +``` +run_v2a = run_ax and (video is not None and vx.numel() > 0) # transformer.py:269 +``` + +It tests `video is not None`, **not** `video.enabled`. So an AV forward handed a +present-but-disabled video stream still runs video->audio cross attention, taking +its context from a video latent that T2A never intended to exist — and returns a +finished waveform. Our port mirrors that polarity exactly at +`ltx2_dit.cpp:251 @ 332aed738`, so the same trap is live here. + +`video = nullptr` is the only shape that reproduces `LTXModel.forward(video=None, +...)` (`model.py:492-538`, the `video_args = ... if video is not None else None` +at `:505`). It is also what makes `_init_preprocessors`' AudioOnly arm +(`model.py:351-365`, a plain `TransformerArgsPreprocessor` with no +`cross_scale_shift_adaln` and no `cross_gate_adaln`) equivalent to our +`have_both == false` path, which passes `other = nullptr` into `PrepareStream` +and therefore builds neither. + +### 2.3 The engine has NO guided denoiser, and T2A's defaults turn one on + +This is the finding the dispatch did not anticipate and it is the largest part of +the row. + +`Ltx2VideoEngine::Generate` runs exactly ONE DiT forward per step +(`ltx2_video.cpp`, the `Ltx2DitForward` / `Ltx2DitForwardDevice` call in the +denoise loop) and never reads a guider parameter: `git grep -n +'guid\|Guider\|cfg_scale' src/vllm/multimodal/ltx2_video.cpp` returns zero hits +against a positive control of 66 for `ltx2` in the same file. That is correct for +`distilled_two_stage`, which builds a `SimpleDenoiser` upstream too. It is not +correct for T2A. + +T2A builds a `FactoryGuidedDenoiser` (`t2a_one_stage.py:154-161`) over +`MultiModalGuiderParams` whose CLI defaults are the params table's +`audio_guider_params` — `cfg_scale=7.0`, `stg_scale=1.0`, `rescale_scale=0.7`, +`stg_blocks=[28]` on the 2.3/2.4/2.5 lineage (`utils/constants.py:58-66`, +`:82-87`, `:118`) — with `modality_scale` pinned to **1.0** by the CLI itself +(`t2a_one_stage.py:200-202`: "Audio-only generation has no video modality, so the +video->audio (v2a) cross-modal guidance is meaningless here. 1.0 disables it"). + +Read against `guiders.py:275-287`, those defaults mean +`do_unconditional_generation()` is TRUE (`cfg_scale != 1.0`), +`do_perturbed_generation()` is TRUE (`stg_scale != 0.0`) and +`do_isolated_modality_generation()` is FALSE. **Three forwards per step: cond, +uncond-text, uncond-perturbed.** Shipping T2A on the engine's single-forward path +would be a different trajectory from upstream's default, and no shape, token +count or file length could see it. + +The bricks exist and are gated; the WIRING does not. `Ltx2MultiModalGuidance` +(`ltx2_pipeline.h:323 @ 332aed738`), `Ltx2CfgDelta`, `Ltx2StgDelta`, +`Ltx2GuiderParamsForSigma`, `Ltx2PerturbationConfig` and +`Ltx2BatchedPerturbationConfig` are all ported with goldens and have **no product +caller** — the "test-only driver" shape `.agents/reachability.md` enumerates. What +is genuinely absent is one thing: the DiT forward runs upstream's +`perturbations=None` path only, and says so at +`include/vllm/model_executor/models/ltx2.h:42-46 @ 332aed738`. + +--- + +## 3. Design + +### 3.1 A new translation unit, mirroring upstream's file + +`include/vllm/model_executor/models/ltx2_t2a.h` + +`src/vllm/model_executor/models/ltx2_t2a.cpp`, holding `Ltx2T2aGenerate` — the +port of `T2AOneStagePipeline.__call__` (`t2a_one_stage.py:109-172`). +`Ltx2VideoEngine::Generate` branches to it at the top, before it resolves any +video geometry. This mirrors upstream's file structure, which AGENTS.md §"Shared +seams" requires, and it follows the precedent +`.agents/specs/ltx25-a2v-audio-input.md` §2 set ("A new translation unit per +concern, mirroring upstream's file structure rather than growing +`ltx2_video.cpp`"). + +It is NOT a parallel path: it is reached only through +`VideoEngine::Generate`, it consumes the same `Impl`-owned weights, and every +numeric it uses is an already-gated brick — `Ltx2SigmaSchedule`, +`Ltx2AudioPatchify` / `Ltx2AudioUnpatchify` / `Ltx2AudioPatchTimings`, +`Ltx2DitForward`, `Ltx2MultiModalGuidance`, `Ltx2EulerStep`, +`Ltx2AudioDecoderForward`, `Ltx2VocoderWithBweForward`, `MiniMaxH3WriteWav`. + +The duplication it does accept is the audio-only denoise loop. That is deliberate: +the joint loop in `ltx2_video.cpp` is ~200 lines of video-stream construction the +audio-only path has no counterpart for, and threading an `is_t2a` flag through it +would put nine new branches inside a function that already runs 1935 lines. + +### 3.2 Call order, mirroring `t2a_one_stage.py:123-172` + +1. `require_num_frames_source` (`:123`, `utils/blocks.py:894-905`). Auto duration + with no `DurationHead` is upstream's own refusal. Mirrored: this engine + constructs no duration head at all (`duration_head_path` is refused at load by + `CheckUnservedExtras`, #611), so an absent `num_frames` is refused with the + message naming the head. +2. Encode the prompt AND the negative prompt (`:127-135`). Upstream encodes both + in one `PromptEncoder` call and takes `.audio_encoding` from each + (`:134-135`). Here both go through `Ltx2EncodePromptToConditioning` and the + connector, and only the AUDIO half of each is kept — the video half is + computed and discarded, exactly as upstream computes both encodings and uses + only the audio one. +3. `resolve_num_frames` (`:137-139`) — an explicit count is returned verbatim + (`utils/blocks.py:920-921`). +4. `sigmas = self._scheduler.execute(steps=num_inference_steps)` (`:141-143`). + `LTX2Scheduler()` is HARD-CODED at `:67`, so this is `Ltx2SigmaSchedule` and + the recipe carries no distilled sigma table. The token count the shift is + derived from is the AUDIO latent's own (`schedulers.py:32` reads + `math.prod(latent.shape[2:])` of the unpatchified target). +5. `create_multimodal_guider_factory(params, negative_context)` (`:149-152`). + A plain `MultiModalGuiderParams` becomes a sigma-independent guider; the + sigma-binned factory arm is not reachable from any surface here and is owed. +6. The stage call with `video=None` and `audio=ModalitySpec(context=a_context_p)` + (`:154-170`). No initial latent, no freeze: the audio starts as pure noise. +7. `return self.audio_decoder(audio_state.latent)` (`:172`) — the audio VAE + decode and the BWE vocoder, which is what the ordinary render already does. + +### 3.3 The guided step + +Per step, mirroring `FactoryGuidedDenoiser` over `MultiModalGuider.calculate` +(`guiders.py:244-273`): + +| Pass | Run when | Inputs | +|---|---|---| +| `cond` | always | positive audio context, no perturbation | +| `uncond_text` | `DoUnconditionalGeneration()` | NEGATIVE audio context, no perturbation | +| `uncond_perturbed` | `DoPerturbedGeneration()` | positive context, audio self-attn perturbed on `stg_blocks` | +| `uncond_modality` | `DoIsolatedModalityGeneration()` | — **refused by name**, see §7 | + +`ShouldSkipStep(step)` (`guiders.py:287-291`) runs NO forward at all and reuses +the previous step's denoised prediction (`utils/denoisers.py:85-91`), which is +not the same as "skip the guidance and keep the conditional pass" — see M9 in §5. +`skip_step` defaults to 0, so it never fires on the default path and is reachable +only from the extra. + +The combination is `Ltx2MultiModalGuidance`, which already carries the two +details a re-derivation gets wrong: the `(scale - 1)` polarity on CFG and modality +against a bare `scale` on STG, and torch's UNBIASED `std` in the rescale. + +### 3.4 STG — the one genuinely new numeric + +`SKIP_AUDIO_SELF_ATTN` on a perturbed block replaces the attention output with the +raw value projection BEFORE `to_out`: `attention.py:558-577` computes `v = +self.to_v(context)` and, when `all_perturbed`, sets `out = v` without ever +projecting `q` or `k`; the blended form `out * mask + v * (1 - mask)` at `:571-572` +is the partial-batch case and reduces to the same thing at batch 1. + +Mirrored as an `all_perturbed` flag on `Ltx2AttentionArgs`, set per block from a +`Ltx2BatchedPerturbationConfig` handed to `Ltx2DitForward` — which is upstream's +own `perturbations` parameter on `LTXModel.forward` (`model.py:492`), so this is +mirroring a signature rather than inventing a seam. `nullptr` is upstream's +`perturbations=None` and every existing caller keeps its current behaviour +byte-for-byte. + +### 3.5 Where the request enters + +`pipeline_kind = t2a_one_stage` is a LOAD extra +(`kLtx2PipelineKindExtra`, already defined and already reachable from +`ltx2-gen --pipeline-kind` and from `vllm_video_model_params::extras`). This +matters for the reach claim: #928 records that `/v1/videos` forwards no +PER-GENERATION extra, and this row's selector is not one. The reach claim in §6b +is worded against what was checked rather than against the file the knob lives in. + +Two per-generation extras are added, both upstream CLI arguments: + +- `negative_prompt` (`--negative-prompt`, `utils/args.py:1083-1088`, defaulting + to `DEFAULT_NEGATIVE_PROMPT`, `utils/constants.py:186`). Absent means the + recipe's own default, which is `LightricksNegativePrompt()` — already in the + tree and already what `one_stage` on 2.4/2.5 resolves. +- `audio_cfg_guidance_scale` / `audio_stg_guidance_scale` / + `audio_rescale_scale` / `audio_skip_step` / `audio_stg_blocks` + (`utils/args.py:1089-1119`). Absent means the params table's own value. + +`modality_scale` gets NO extra, and that is not an omission: the CLI pins it to +1.0 at `t2a_one_stage.py:200-202` and there is no upstream surface that varies it +on this pipeline. + +### 3.6 The recipe + +`t2a_one_stage` rows for `2`, `2.3`, `2.4` and `2.5`, mirroring the `one_stage` +rows one for one, because the schedule is the same object: `LTX2Scheduler()` at +`t2a_one_stage.py:67` against `ti2vid_one_stage.py:81`, and the same +`detect_params` step count. What differs is what the recipe DECLARES about video: +`video_output_phase = -1` and a new `audio_only` flag, so a T2A recipe cannot be +run down the video path by accident. + +The 2.0 and 2.3 rows take `kOmniNegativePrompt` and the 2.4/2.5 rows take +`LightricksNegativePrompt()`, which is what `ResolveLtx2PipelineRecipe`'s +`one_stage` arm already does (`ltx2_pipeline.cpp:1239-1247 @ 332aed738`) — the negative prompt +travels with the generation, not with the pipeline. + +### 3.7 Load + +`video_vae_path` becomes optional **only** on a `t2a_one_stage` engine. +Upstream's `T2AOneStagePipeline.__init__` never calls `model_paths.video_vae()` +(`:53-107` constructs a `PromptEncoder`, a `DiffusionStage`, an `AudioDecoder` +and a `DurationPredictor`, and nothing else), so requiring one here would demand +a checkpoint the pipeline cannot use. `audio_vae_path` stays required, because +`AudioDecoder` is constructed unconditionally at `:94-100`. + +--- + +## 4. Risks + +**The `READER ANCHORS` gate.** `ltx2_video.cpp` carries a derived line-number +list re-derived and string-compared by `test_ltx2_video.cpp`. Any line inserted +above the last anchored line shifts it, and a clean `git merge` will not warn. +Mitigation: the list is re-derived at the final tree after the last merge of +`origin/main` with the test's own walk rather than by arithmetic, and it is named +as a merge hazard in the PR body. + +**AND THE ANCHORS DID MOVE.** The mitigation above once claimed the engine edit +is "a branch at the TOP of `Generate`, which is below every anchor", and the +list is `781 791 792 854 950 966 968 1046 1071 1176 1217` on `origin/main` +against `782 792 793 855 951 967 969 1060 1085 1190 1231` here. Two hunks sit +ABOVE the last anchor: the `ltx2_t2a.h` include at `@@ -36,6 +36,7`, which shifts +every anchor by one, and the audio-only video-VAE exception at +`@@ -974,8 +975,21`, which adds thirteen more and moves the last four by +fourteen. `@@ -1018,7 +1032,7` is above 1231 as well and is net zero. The +anchors were correctly RE-DERIVED with the test's own walk and the gate passes +23/23, so the outcome is right; only the stated reason was false, and a false +reason is what makes the next reader skip the re-derivation. + +**A silently wrong render.** Every failure mode here produces a playable WAV of +the right length: a guider that never runs its uncond pass, an STG pass that +perturbs the wrong block, a `video = nullptr` path that silently reintroduces v2a +cross attention, an audio latent that decoded from zeros. None is visible in the +output. Mitigation: §5's trace fields observe the EFFECT rather than the +bookkeeping, plus a lower bound so a zeroed tensor cannot pass. + +**A recorded value is not a reached one.** A sibling row's suite recorded a +latent digest and inferred the latent reached the phase; a build that started from +zeros passed everything. So every assertion below is written against the state the +loop actually ran over, read after the operation it claims, and at least one is a +CONTROL that must move in the opposite direction. + +**Concurrent edits.** `ltx2_video.cpp`, `ltx2.h`, `ltx2_pipeline.{h,cpp}`, +`docs/FEATURES.md` and `.agents/issue-index.md` are being edited by sibling rows +(`LTX25-RETAKE`, `LTX25-DFR-PIPELINE` both landed within the hour before this +row started). Mitigation: new behaviour lives in new TUs; keyed records are +reapplied by key with unrelated keys proven byte-identical; the index is appended +to and never edited. + +--- + +## 5. Tests and evidence + +Focused gate: `test_ltx2_video`, `test_ltx2_pipeline`, `test_ltx2`, `test_capi`. + +1. **RED first, through the production entry point.** The smallest failing test + loads the reduced fixture with `pipeline_kind = t2a_one_stage` and calls + `Ltx2VideoEngine::Generate` (reached from `vllm_video_generate`). It fails + first because `ResolveLtx2PipelineRecipe` refuses the kind by name. + Reachability is then proven by deleting the production call site and showing + the test goes RED (`.agents/reachability.md`). +2. **The render has NO picture and DOES have sound.** `frame_count == 0`, an + empty `mux_argv`, an `audio.wav` on disk whose sample rate is the vocoder's + own, and NO `frame_000000.ppm` in the output directory. The last one is the + half a field check cannot make: a build that wrote frames and reported zero + passes every other assertion here. +3. **The video stream never reaches the DiT.** A trace field records whether the + forward was handed a video stream at all, and it must be false on every step + of a T2A render and true on an ordinary one. This is the §2.2 trap, and a + count- or shape-shaped check cannot see it: an `enabled=false` video stream + produces a waveform of exactly the right length. +4. **The guider actually ran, and each arm is separable.** Counters for the + cond / uncond-text / uncond-perturbed forwards. At the default params the + counts must be `steps`, `steps`, `steps`; with `audio_cfg_guidance_scale=1` + the second must be 0 and the render must still complete; with + `audio_stg_guidance_scale=0` the third must be 0. A single "guidance ran" + boolean would be satisfied by a build that ran the uncond pass and then + ignored it, so the guided output is ALSO required to differ from the + unguided one on the same seed. +5. **STG perturbs the block it was told to.** Asking for `audio_stg_blocks` that + name a block index out of range is upstream's own failure and is refused; and + perturbing block N produces a different velocity from perturbing block M. A + test asserting only that "a perturbed pass ran" cannot tell a config that + perturbs everything from one that perturbs nothing. +6. **A LOCAL fact is tied to the lifted refusal**, per this campaign's standing + rule. A test asserting only upstream symbol names cannot see staleness: a + sibling proved this by replacing a refusal's local claim with a self-declared + falsehood and watching the suite stay green at 44/44. So the test re-derives, + from THIS tree, that `Ltx2DitForward` still guards its remaining half and that + the AudioVideo weight contract is still what the loader enumerates. +7. **A lower bound** on the decoded waveform's magnitude, so a silently zeroed or + constant buffer fails. A correlation or count-based check cannot see a scale + error and is not used alone. + +Every mutation records three facts: `git diff --stat` after applying, whether it +**BUILT** with the compile-error count beside it, and the **exit code** captured +directly rather than through a pipe. A mutation that fails to compile establishes +nothing and is recorded as such. `-tc` filters are comma-free and the case count +is asserted non-zero, because doctest splits on a comma and runs UNRELATED cases +to a green SUCCESS. + +### What the mutation pass actually found + +Focused gate `./build/tests/test_ltx2_video "--test-case=*t2a*"`. Each mutation +applied to ONE file, rebuilt, run, restored in a `finally` and the restore +verified by **sha256** rather than assumed. The harness rebuilds the restored +tree before anything else measures it. + +**Measured at 8 cases / 505 assertions / exit 0 unmutated**, on the tree this +pull request ships. The table was re-run at that tree rather than carried forward +from the earlier 6-case measurement, because a count carried forward while the +suite grows is exactly how this campaign's previous mutation records went stale. + +| Mutation | `git diff --stat` | BUILT | exit | verdict | +|---|---|---|---|---| +| M1 delete the production call site (the reachability mutation) | `ltx2_video.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED, 4 of 8 cases red | +| M2 hand the forward a present-but-DISABLED video stream instead of `nullptr` | `ltx2_t2a.cpp` (see note) | YES (0 errors) | 1 | DETECTED, 3 red | +| M3 never run the unconditional forward | `ltx2_t2a.cpp` (see note) | YES (0 errors) | 1 | DETECTED, 2 red | +| M4 ignore `stg_blocks` and perturb EVERY block | `ltx2_t2a.cpp` (see note) | YES (0 errors) | 1 | DETECTED, 3 red | +| M5 `all_perturbed` falls through to ordinary attention | `ltx2.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED, 2 red | +| M6 revert the `one_stage` `noise_scale` to the struct default (#1013) | `ltx2_pipeline.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED, 1 red | +| M7 scale the initial latent by `sigmas[0]` | `ltx2_t2a.cpp` (see note) | YES (0 errors) | 0 | **SURVIVED** — see below | +| M8 write a frame on the audio-only path | `ltx2_video.cpp \| 1 +` | YES (0 errors) | 1 | DETECTED, 1 red | +| M9 a skipped step RECOMPUTES the conditional forward instead of reusing | `ltx2_t2a.cpp \| 39 +++---` | YES (0 errors) | 1 | DETECTED, 1 red | + +**A note on the first fact for the `ltx2_t2a.cpp` rows, because it reported +something misleading and that is worth writing down rather than tidying away.** +`git diff --stat` is measured against `HEAD`, not against the pre-mutation +working tree, so on a run where that file also carried an uncommitted change the +stat reports 45-47 lines rather than the mutation's own 1-3. The number is +therefore not a measurement of the mutation on those rows. It is kept, with this +note, instead of being replaced by a prettier one: the fact the protocol asks for +is what the command printed. The M1, M5, M6, M8 and M9 rows were measured against +a clean file and their stats are the mutations'. + +**M9 is the mutation for a defect this port ACTUALLY SHIPPED in its first +draft**, rather than an invented one. `should_skip_step` (`guiders.py:287-291`) +does not mean "skip the guidance and keep the conditional prediction". Upstream +returns `DenoisedLatentResult.result_or_none(denoised=last_denoised_audio)` +(`utils/denoisers.py:85-91`) BEFORE it assembles any pass, so a skipped step runs +**no DiT forward at all** and reuses the previous step's denoised prediction. The +first draft ran the conditional forward and used it: a whole extra forward per +skipped step, on a different trajectory, producing a finished waveform of exactly +the right length. Nothing about the output separates the two, and the FORWARD +COUNT is the only thing that does. + +**M7 SURVIVED, and the resolution is the useful part.** It is the mutation a +reader coming from another flow-matching sampler expects to be REQUIRED — scaling +the initial noise by the first sigma — and it changed nothing. The reason is not +a blind gate: it is an identity, and it STAYS survived after the pin below, +because a pin on an identity cannot turn one arm red. `LTX2Scheduler` starts at `linspace(1, 0, steps ++ 1)[0] == 1`; the shift map sends 1 to `exp(s)/(exp(s) + (1/1 - 1))`, exactly 1 +(`schedulers.py:41-45`); the stretch sends it to `1 - (1 - 1)/scale_factor`, +again exactly 1 (`:47-55`). So `sigmas[0]` is 1.0 for every step count. + +Rather than record "a mutation survived", the identity is now GATED — the case +"the schedule starts at exactly 1.0" pins `sigmas.front() == 1.0F` across four +step counts. If upstream ever moves the first sigma off 1, that fires and the two +forms stop agreeing. + +**And that case found a second thing.** `steps = 1` returns `-nan`, on both +sides: the non-zero sigma list is `[1.0]`, so `one_minus_z` is `[0.0]`, +`scale_factor = 0 / (1 - terminal)` is 0, and the stretch computes `1 - 0/0` +(`schedulers.py:49-54`). It is upstream's own arithmetic, not a defect here, and +it is excluded from the pin with the reason written beside it rather than +silently skipped. A one-step schedule is recorded under `## Owed`. + +**Two harness notes, because both would otherwise read as verdicts about the +code.** A `.pyc` for `scripts/agent-start.py` was truncated to exactly 4096 bytes +on this shared box, and `scripts/agent-preflight.sh` reported +`FAIL test_agent_start` with an `EOFError: marshal data too short` — a corrupt +byte-cache presenting as a failing gate. Removing the file made it pass 20/20. +And M4's first form asserted the STG perturbation on a latent filled with a +constant: self-attention over identical rows returns a weighted average of +identical values, which IS the value projection, so the perturbation was a +numeric no-op and the case reported "the perturbation changed nothing" about a +correct build. The fixture latent now varies per element, and the reason is in +the test. + +--- + +## 6. Gates + +``` +cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF +cmake --build build -j6 +ctest --test-dir build -j4 --output-on-failure +``` + +Reported with `CONFIGURE_EXIT`, `BUILD_EXIT`, the `: error:` count, `ctest -N`, +`CTEST_EXIT`, the full pass/fail line, positive controls for `No space left` and +`BFD assertion`, the box load and free disk. Known-red on `main` proven +pre-existing rather than asserted: `windows-msvc-*` (#584), `check-env-doc` +(#995). A load-dependent failure is re-run alone with the load recorded before it +is charged to this row. + +## 6b. Reachability — the sentence the records must carry + +**A production entry point reaches this, and the test enters through it.** + +``` +include/vllm.h vllm_video_generate + -> src/capi/vllm_c.cpp engine->Generate(gen) + -> vllm::multimodal::VideoEngine::Generate + -> Ltx2VideoEngine::Generate (the t2a branch at the top) + -> Ltx2T2aGenerate (ltx2_t2a.cpp) + -> Ltx2DitForward(..., /*video=*/nullptr, &ain, ...) + -> Ltx2MultiModalGuidance -> Ltx2EulerStep + -> Ltx2AudioDecoderForward -> Ltx2VocoderWithBweForward -> audio.wav +``` + +The command-line arm is the same call: `ltx2-gen --pipeline-kind t2a_one_stage` +sets the load extra and calls `vllm_video_generate`, as a thin ABI client that +includes no internal header. + +**What is NOT reachable, stated rather than left to be found.** `/v1/videos` +carries LOAD parameters, so `pipeline_kind` reaches it in principle; the +per-generation extras this row adds do NOT, because +`VideoGenParamsFromRequest` never forwards `VideoRequest::metadata` to +`VideoGenParams::extras` (#928). A T2A render over that route therefore takes the +recipe's own guider defaults and cannot override them. That is a pre-existing +defect of the route rather than of this row, and the reach claim above is worded +to exclude it. Whether the route's LOAD path was exercised end to end here is +recorded in the final report as measured or unverified — it is not asserted. + +**This row ends ONE test-only driver, and leaves three standing.** An earlier +draft of this section claimed all four, and the tree contradicts it: + +| Symbol | Production call site after this row | +|---|---| +| `Ltx2MultiModalGuidance` | **yes** — `src/vllm/model_executor/models/ltx2_t2a.cpp`, in the guided step. This is the one this row ends | +| `Ltx2CfgDelta` | no — reached only through `Ltx2Guidance` (`ltx2_pipeline.cpp:532`) and from `tests/vllm/models/test_ltx2_pipeline.cpp:615` | +| `Ltx2StgDelta` | no — the same, at `ltx2_pipeline.cpp:534` and `test_ltx2_pipeline.cpp:630` | +| `Ltx2BatchedPerturbationConfig` | no — constructed nowhere outside `test_ltx2_pipeline.cpp:832-859`. T2A uses `Ltx2DitPerturbation`, which is a different type | + +`Ltx2Guidance` itself is the reason the two deltas stay dead: its only caller in +the tree is `tests/vllm/models/test_ltx2_pipeline.cpp:710`, so the switch that +would route a configured guider kind to them is reached from no product path. + +**This row does not owe that wiring.** All four landed with #641, before this row +existed, and none of them is on the T2A path — T2A resolves a +`Ltx2MultiModalGuiderParams` and calls `Ltx2MultiModalGuidance` directly, exactly +as `_guided_denoise` does upstream. Recorded here because the spec must not +assert what `git grep` refutes, and listed under `## Owed` with the issue that +tracks it. + +## 7. Quantized arms + +| Arm | Disposition | +|---|---| +| bf16 / f32 safetensors | **ported** — the reduced fixture is this, and the gate runs on it | +| NVFP4, FP8 (the DiT tower) | **ported by inheritance, and that is a claim about the load rather than about a render.** The T2A path consumes `Impl::dit.weights`, which is whatever arm `Ltx2LoadDitFromSafetensors` materialized; this row adds no GEMM, changes no dtype and selects no arm. It is recorded as UNMEASURED on real quantized weights, because the GPU was out of bounds | +| GGUF k-quants | **not applicable**, and not merely undone. Upstream ships no GGUF arm for any LTX-2 component: `quantization_factory.py:23-26` enumerates the inference kinds exhaustively as fp8-cast, fp8-scaled-mm, nvfp4-cast and nvfp4-prequant, with `assert_never` at `:50`. There is no upstream behaviour to mirror and no quant-matched llama.cpp comparison to serve, because llama.cpp does not carry this architecture | +| int8-convrot | **out of scope**, unchanged by this row and already refused by name through `kInt8ConvRot` | + +Dtype polarity: the T2A path is f32 throughout on the host arm, which is what +`Ltx2DitForward` declares and refuses to widen, and the audio VAE and vocoder are +f32 by upstream's own choice (`vocoder.py:585-595`, mirrored in +`ltx2_audio_vae.cpp:1-12`). No buffer here is wider than the arm it feeds. + +--- + +## 7b. The bug this row found and fixed in flow + +[#1013](https://github.com/mudler/vllm.cpp/issues/1013). `OneStagePhase` +(`ltx2_pipeline.cpp`) left `Ltx2PhaseRecipe::noise_scale` at the struct default +of **0.0**, and 0.0 is not "no extra noise": `Ltx2GaussianNoise` is +`latent + noise_scale * (noise - latent)`, so at 0.0 the state stays exactly as +`create_initial_state` wrote it, which with no initial latent is **all zeros**. +A `one_stage` render therefore denoised a zero tensor on both streams. + +Upstream's `ModalitySpec.noise_scale` defaults to 1.0 +(`ltx-pipelines/utils/types.py:110`) and `TI2VidOneStagePipeline.__call__` +constructs both specs without it (`ti2vid_one_stage.py:233-239`). The two +neighbouring recipes set it explicitly, which is what made the omission legible. + +Fixed here rather than deferred, per AGENTS.md § *Every change starts from an +issue*, because the `t2a_one_stage` rows are built FROM `OneStageRecipe` and +would have inherited it. No gate saw it because every end-to-end test loads +`distilled_two_stage`, and a zero-initialized denoise still returns a finite clip +of the right size, frame count and sample rate. M6 above is the mutation that now +holds it. + +**NOT claimed for `dmd2`.** `PositiveOnlyRecipe` leaves the same field at 0.0 and +its source is vLLM-Omni's `LTX_POSITIVE_ONLY_RECIPE`, which is not checked out +here. Left as-is and named rather than corrected by analogy: a recipe whose +upstream nobody read is exactly where a plausible fix lands wrong. Recorded in +#1013. + +## 7c. The second bug this row found and fixed in flow + +[#1039](https://github.com/mudler/vllm.cpp/issues/1039). **The guidance passes +were combined in VELOCITY space, and upstream combines x0.** Found by review of +pull request #1032 at `3d9d9c9bb`, before the branch landed. + +Upstream never hands the denoiser the raw velocity model. `DiffusionStage` +builds `X0Model(self._prepared_builder().build(device=target, **kwargs))` +(`ltx-pipelines/utils/blocks.py:480-482`), and `X0Model.forward` returns +`to_denoised(audio.latent, ax, audio.timesteps)` (`ltx-core +model/transformer/model.py:590-604`), which is `sample - velocity * sigma` +(`ltx-core utils.py:39-52`). So `_guided_denoise`'s +`all_v, all_a = transformer(...)` (`ltx-pipelines/utils/denoisers.py:188`) +already carries DENOISED tensors, and `audio_guider.calculate(cond_a, uncond_a, +ptb_a, mod_a)` at `:203` combines those. + +`Ltx2T2aGenerate` took `Ltx2DitForward`'s velocities straight into +`Ltx2MultiModalGuidance` and applied `ToDenoised` once to the result. + +**Why no existing gate saw it, and why the row's own §5 could not.** +`MultiModalGuider.calculate`'s linear terms (`guiders.py:261-266`) are invariant +under `x0 = latent - sigma*v`, so the two forms agree EXACTLY while +`rescale_scale == 0`. The rescale at `:268-271` is not invariant: upstream's +`factor` is `std(x0_cond)/std(x0_pred)` and it scales the whole x0, giving +`factor*(latent - sigma*v)`, where scaling the velocity gives +`latent - sigma*factor*v`. The two differ by `(factor - 1) * latent` — zero +only where the latent is zero, which on this path it never is, because the state +IS the unit-variance noise (§3.2 item 6). `rescale_scale = 0.7` is the shipped +default (`utils/constants.py:63`, `utils/args.py:1101-1106`), so the DEFAULT arm +took the divergent branch. Everything §5 observes — the three forward counters, +`t2a_video_stream_present`, `t2a_perturbed_blocks`, the latent absmax, the +waveform's length, channel count and sample rate — is identical between the two +forms. + +**Fixed by moving the conversion, not by moving the rescale.** The mirror is +structural: the per-pass `x0_model` lambda in `ltx2_t2a.cpp` IS `X0Model`, it +applies `ToDenoised` on the way out of every forward, and +`Ltx2MultiModalGuidance` stays a faithful port of `calculate` over whatever the +model returned. Reaching the same numbers by moving the rescale inside the +guidance seam would put `to_denoised` inside `calculate`, where upstream does not +have it, and would make the seam correct only for this one composition. + +**The VIDEO arm is unaffected, and that was checked rather than assumed.** +`git grep -n Ltx2MultiModalGuidance -- src include` returns exactly one +production call site, `ltx2_t2a.cpp`. `Ltx2PipelineParams::video_guider` and +`Ltx2PhaseRecipe::video_guidance` are carried by the recipe and read by nothing: +the joint driver in `ltx2_video.cpp` runs ONE unguided `Ltx2DitForward` per step +and applies `ToDenoised` to that single velocity (`:3034-3036`), which is the +same tensor in both spaces because there is no combination to be invariant +under. There is therefore no second instance of this defect to fix, and there +will be one the moment a guided video denoiser is wired — noted here because +that wiring is a live campaign item. + +**What the gate is, and what it deliberately is not.** The reduced fixture +CANNOT resolve the rescale's numeric consequence, and that is measured rather +than assumed: its DiT responds to the conditioning at ~1e-5 of its own output, +so `std(cond)/std(pred)` is 1.0 to 1e-5 in BOTH spaces, both factors land within +1e-5 of 1.0, and the two candidate step-0 predictions sit 7.6e-07 apart against +a span of 3.41. The first draft of the test asserted exactly that difference and +its own separation guard refused it — a case that would have been green either +way. So the row gates the defect at two places instead: + +1. `test_ltx2_video` "the guider is handed x0 predictions and not raw + velocities" — end to end through `LoadVideoEngine` and + `VideoEngine::Generate`, on the recipe's own guider with no extra touched. It + pins the EQUATION `cond == latent - sigma*velocity` between three recorded + step-0 tensors. That is exact in x0 space and off by the whole sample in + velocity space, so no fixture scale satisfies it by accident; a zeroed + velocity or a zero sample fails the two `REQUIRE`s that precede it rather + than passing it. +2. `test_ltx2_video` "rescale_scale 0 is the control because both spaces agree + there" — the numeric consequence, on the real `Ltx2MultiModalGuidance` seam + with a latent that makes it visible. MEASURED: relative disagreement between + the two spaces is **1.50e-07 at `rescale_scale = 0.0`** and **0.352 at the + shipped 0.7**. This is what makes 0.0 the control rather than the assertion + site. + +The observability this needed is four step-0 fields on `Ltx2T2aResult` and the +trace: the sample, the conditional pass's RAW velocity, the tensor handed to the +guider, and the guider's result, plus step 0's sigma. `first_step_cond` is +upstream's own `DenoisedLatentResult.cond` (`utils/denoisers.py:206`). + +### Mutations for #1039 + +Focused gate: three comma-free `--test-case` filters, each asserting a non-zero +case count. Each mutation applied to ONE file, rebuilt, run, restored in a +`finally` and the restore verified by **sha256**. `git diff --stat` is scoped to +the mutated file and measured against the committed fix, so the numbers are the +mutation's own. + +| Mutation | `git diff --stat` | BUILT | exit | verdict | +|---|---|---|---|---| +| N1 revert to velocity-space guidance (the defect) | `ltx2_t2a.cpp \| 4 ++--` | YES (0 errors) | 1 | DETECTED by case 1 | +| N2 delete the production call site | `ltx2_video.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED by case 1 AND the render case | +| N3 take x0 against a ZERO sample instead of the latent | `ltx2_t2a.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED by case 1 | +| N4 drop the rescale branch entirely (`guiders.py:268-271`) | `ltx2_pipeline.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED by the control case | + +N4 is the row that proves the control case is not decorative: it is the only one +of the four that case 1 does not see, and the only one the control does. + +**N1 is the RED-before**, and this is what it printed: + +``` +tests/vllm/multimodal/test_ltx2_video.cpp:5371: ERROR: + CHECK( err_x0 <= 1e-5 * latent_span ) is NOT correct! + values: CHECK( 3.43642 <= 3.38677e-05 ) + logged: sigma = 1 max|latent| = 3.38677 max|velocity| = 0.415609 + |cond - (latent - sigma*velocity)| = 3.43642 |cond - velocity| = 0 + elements = 3328 +tests/vllm/multimodal/test_ltx2_video.cpp:5378: ERROR: + CHECK( err_v > 1e-2 * latent_span ) is NOT correct! + values: CHECK( 0 > 0.0338677 ) +[doctest] test cases: 1 | 0 passed | 1 failed | 66 skipped +[doctest] assertions: 16 | 14 passed | 2 failed | +[doctest] Status: FAILURE! exit 1 +``` + +`|cond - velocity| = 0` **exactly** is the whole finding: the tensor handed to +`Ltx2MultiModalGuidance` WAS the raw DiT velocity. Green after, on the same +filter: 1 case, 16 assertions, 0 failed, exit 0. + +### The #1039 gate covered ONE of the three guidance arms + +The fresh review of `c1fe35592` passed on the correctness of the fix and +returned one blocking finding: the case above recorded `first_step_velocity` and +`first_step_cond` for the CONDITIONAL pass, nothing observed the other two arms, +and nothing pinned what `Ltx2EulerStep` consumed. `ltx2_t2a.cpp:41-43` claims the +conversion is applied to EVERY PASS, and the gate held that claim for one third +of them. + +Reproduced before the repair, on the same filter as the green run +(`--test-case=ltx2 t2a*`, comma-free, 10 cases / 526 assertions at +`c1fe35592`). Each mutation applied to ONE file, `git diff --stat` taken against +the pre-mutation working tree rather than against `HEAD`, rebuilt with the +`: error:` count printed beside the verdict, exit code captured directly, and +restored from a content snapshot with `os.utime(now)` and a sha256 compare. + +| Mutation | before the repair | after | +|---|---|---| +| A1 the PERTURBED (STG) pass alone left in velocity space | SURVIVED, exit 0, 10/526 | DETECTED, exit 1 | +| A2 the UNCONDITIONAL pass alone left in velocity space | SURVIVED, exit 0, 10/526 | DETECTED, exit 1 | +| R1b `ToDenoised` applied a SECOND time to the guider's output, between the step-0 record and the Euler step | SURVIVED, exit 0, 10/526 | DETECTED, exit 1 | +| R1c the same double application ABOVE the step-0 record, so the recorded `t2a_first_denoised` is itself doubly converted | SURVIVED, exit 0, 10/526 | DETECTED, exit 1 | +| A4 the perturbed arm's recorded velocity ZEROED — the guard, not a defect | n/a (the field did not exist) | DETECTED, exit 1, by the `REQUIRE` | + +R1c is not from the review. It was found while closing R1b: the reviewer's R1b +sits between the record and the step, so the recovered-Euler-input check sees +it, and moving the same edit one statement earlier does not. That is why the +repair adds a second, independent check rather than one. + +**The repair is observability plus three checks, not a change to the fix.** +`Ltx2T2aResult` and the trace gain a (velocity, x0) pair for the unconditional +and perturbed arms and the latent the Euler step wrote. The case then applies the +SAME equation to every arm, replays `Ltx2MultiModalGuidance` over the three +recorded arms and requires bit equality with `t2a_first_denoised`, and recovers +`t2a_first_next_latent` from `t2a_first_denoised` through the Euler formula. + +**Non-vacuity, per arm rather than once.** `latent_span > 1e-3` stays shared — +a zero sample makes the two candidate tensors coincide on every arm. Its partner +`sigma * velocity_span > 1e-6` moves INSIDE the per-arm loop, because a zero +velocity makes `to_denoised` the identity for that arm alone, and "expected zero, +and a stub also produces zero" is the trap this campaign has hit twice. A4 is the +mutation that proves that guard is armed: zeroing one arm's recorded velocity +takes the case red through the `REQUIRE`, at 538 assertions rather than 548 +because the `REQUIRE` aborts the case. The replay check adds its own +(`t2a_first_denoised != t2a_first_cond`, the guider MOVED what it was handed) and +the Euler check adds two (`|dt| > 1e-3`, so the step is not the identity, and +`scale > 1e-3`, so the residual bounds something). + +**The rescale's numeric difference is still NOT asserted, and the reason was +re-measured.** `std(cond)/std(pred)` is 1 to printed precision on this fixture, +so `factor = 0.7*1 + 0.3 = 1`, the rescale is an exact no-op in BOTH spaces, and +the difference term `(factor - 1) * latent` is identically zero. Owed against the +real-checkpoint render, as before. + +Green after: `--test-case=ltx2 t2a*` at 10 cases / **548** assertions / 0 failed +/ exit 0, up from 526. + +## Owed + +- **The rescale's numeric consequence END TO END.** Gated at the seam (0.352 + relative at the shipped 0.7) and at the space (exactly, through the engine), + and NOT on a render, because the reduced fixture's guidance deltas are ~1e-5 + of the prediction and both rescale factors land within 1e-5 of 1.0. What would + close it is the real-checkpoint render already owed below, where the DiT's + velocity is comparable to the sample. Tracked by + [#1039](https://github.com/mudler/vllm.cpp/issues/1039). +- **The DEVICE arm.** `Ltx2DitForwardDevice` dereferences `*video` + unconditionally from its first `PrepareStreamDev` call onward + (`src/vllm/model_executor/models/ltx2_device.cpp`, the two `PrepareStreamDev` + calls and the per-block `a.batch = video->batch`), so a one-stream device + forward is a rewrite of that function rather than a lifted check. T2A on + `device != 0` is REFUSED BY NAME rather than served the host forward behind a + device handle, because that substitution is what would make every later timing + claim false. Tracked by #1005. +- **STG on the DEVICE forward**, for the same reason. +- **The sigma-BINNED guider factory** (`guiders.py:294-342`, + `MultiModalGuiderFactory.from_dict`). `Ltx2GuiderParamsForSigma` is ported and + gated; nothing here constructs bins, because no upstream surface on this + pipeline varies params by sigma — `t2a_one_stage.py:196-205` passes a plain + `MultiModalGuiderParams`. A caller may pass a factory (`:116`), and no CLI + does. +- **`uncond_modality` (isolated-modality guidance).** REFUSED BY NAME on this + pipeline, and the refusal is upstream's own reasoning rather than a local + limit: `modality_scale` is pinned to 1.0 for T2A because there is no video + modality to isolate (`t2a_one_stage.py:200-202`). A caller who reaches it + through another pipeline gets a message naming the missing fourth forward. +- **AUTO duration.** `resolve_num_frames`' predicting arm needs a constructed + `DurationPredictor`; `Ltx2DurationHeadForward` is ported and gated (including + the audio-only case, `kLtx2DurAudioOnlyGolden`) and nothing constructs one. + Inherited from the engine, not introduced here. +- **A one-step schedule.** `Ltx2SigmaSchedule(1, ...)` returns `-nan` as its + first sigma, mirroring upstream's own division by zero + (`schedulers.py:49-54`). Neither side is checked against it; a T2A request with + `steps = 1` therefore produces NaN. Named rather than defended against, + because the correct behaviour is upstream's to decide and this port does not + get to invent one. +- **The `dmd2` recipe's `noise_scale`**, see §7b. +- **A real-checkpoint T2A render.** Gated on the reduced fixture only; the GPU + was out of bounds for this row. +- **THE LTX-2.5 CHECKPOINT PIN.** `docs/USAGE.md` names six LTX-2.5 artifacts by + bare file name and gives no HuggingFace repo, no revision and no sha256 for + any of them — `:663-670` and `:2183-2188` on `origin/main` at `d1b0ea3a8`, + plus the text-to-audio recipe this row added at `:853-857`. AGENTS.md + § *Say which weights, and from where* requires all three, per arm. It is + campaign-wide and pre-existing rather than introduced here: `grep -n sha256 + docs/USAGE.md` returns two checkpoint hashes and both belong to + MiniMax-Music3 (`:3127`, `:3269`), while MiniMax-H3 (`:1950-1993`) and + MiniMax-Music3 (`:3123-3149`) each carry a full table and LTX-2.5 carries + none. RECORDED AND NOT FABRICATED: this row claims no render on real weights, + so there is no checkpoint it was gated against to pin, and inventing a repo id + would be worse than the gap. The real-checkpoint render owed above is what + closes it. Tracked by + [#1048](https://github.com/mudler/vllm.cpp/issues/1048). +- **`Ltx2Guidance` and the two deltas it gates are DEAD in production**, and so + is `Ltx2BatchedPerturbationConfig`. See §6b for the measured table. All four + landed with #641, none is on the T2A path, and this row ends only + `Ltx2MultiModalGuidance`'s test-only-driver state. Tracked by + [#1049](https://github.com/mudler/vllm.cpp/issues/1049). +- **`test_engine_core_proc`'s immediate-shutdown case is load-dependent**, and + no issue named it until now. Its abort frame is searched for over a FIXED 1000 + dequeues while a `max_tokens=100000` request keeps producing token deltas, so + the budget is a bet on scheduling. MEASURED at `37e680cab`, same binary + throughout: 2 failures in 3 `ctest -j4` runs of the full suite, 0 in 25 solo + runs on an idle box, 0 in 25 solo runs against 20 spinning processes, and 0 in + two `ctest -R` runs. The third `-j4` run failed `test_cpu_threadpool` INSTEAD, + so the identity of the failing test rotates between runs of an unchanged + binary. This branch touches no file under `tests/vllm/v1/` or `src/vllm/v1/`. + The earlier revision of the PR body attributed it to #294, which is a + DIFFERENT defect in `test_async_llm`. Tracked by + [#1052](https://github.com/mudler/vllm.cpp/issues/1052). +- **The guider rescale's `std` comment states an impossible consequence.** + `ltx2_pipeline.cpp:505-506` and `ltx2_pipeline.h:319-322` say torch's unbiased + (N-1) `std` matters and the biased one "would be a small, everywhere, + resolution-dependent gain error". `factor = std(cond)/std(pred)` divides two + `std`s over the same count, so the `(n-1)` cancels exactly and the two + estimators give the same ratio. A review mutation from biased to unbiased + survived because it is an IDENTITY, not because the gate is blind — which is + worth writing down, because a survivor at that site otherwise reads as a blind + instrument and costs another investigation. The CODE is right; the COMMENT is + the defect. Pre-existing from `cefacd2d0` (#641) and out of this row's scope. + Tracked by [#1050](https://github.com/mudler/vllm.cpp/issues/1050). +- **Value goldens from executed upstream for the T2A COMPOSITION.** The bricks + either side have them; the chain does not. What would close it is a section in + `scripts/gen-ltx2-pipeline-goldens.py` that instantiates + `LTXAudioOnlyModelConfigurator` at reduced dimensions and runs one guided step. +- **`max_batch_size`** (`t2a_one_stage.py:120`, `:169`), the prompt enhancer + (`:118-119`), LoRA/quantization/compilation/offload constructor arguments + (`:56-63`) — the engine's own surfaces, unchanged by this row. + +## 8. Stop conditions + +Stop and report rather than widening scope if: lifting the `video != nullptr` +half of the DiT refusal moves any existing LTX-2.5 test; the STG perturbation +cannot be expressed without changing an existing forward's numerics; or the +`READER ANCHORS` gate cannot be re-derived deterministically. + +## 9. Now + +`ACTIVE` — spec committed before implementation, per `AGENTS.md` § *Spec before +code*. diff --git a/CMakeLists.txt b/CMakeLists.txt index 01da26eb5..0d34c20b3 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -873,6 +873,12 @@ add_library(vllm STATIC # truncate-or-pad conform, and the frame-directory source. Its own TU for the # same reason the two above are: ltx2_video.cpp is a concurrent-edit lock. src/vllm/model_executor/models/ltx2_retake.cpp + # LTX-2.5 (ROW LTX25-T2A-ONE-STAGE, issue #1005): text-to-audio — the + # audio-only pipeline and its guided denoiser. Its own TU for the same reason + # the two above are, plus one this row adds: it is the only LTX pipeline that + # runs the DiT with `video = nullptr`, and keeping that call in one file is + # what makes "does anything else pass a video stream" a readable question. + src/vllm/model_executor/models/ltx2_t2a.cpp # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L6): the quantized loaders — the # FP8 and torchao-NVFP4 DiT arms, the torchao-NVFP4 text encoder, and the # load-time device staging GB10's ATS penalty makes the default. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 66021e875..7fb35d3c5 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -171,6 +171,8 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 Conv VAE decode arithmetic width | LTX-2.5 video VAE | `test_ltx2_vae` "the decode's convolution accumulates in f32", entering through `Ltx2VideoDecodeStreaming`; widening the accumulator to `double`, or deleting the production call site, each turns it RED | **f32**, the width `F.conv3d` uses at f32 AND bf16 (MEASURED). Was f64 at 8 sites ([#1008](https://github.com/mudler/vllm.cpp/issues/1008)). Conv sums BLOCKED per input channel, as torch's. STORAGE stays f32; bf16 owed | | LTX-2.5 Conv VAE decode threading | LTX-2.5 video VAE | `test_ltx2_vae` "the decode DISPATCHES its convolutions to the CPU threadpool" and "...BIT-IDENTICAL across thread counts", through `Ltx2VideoDecodeStreaming`; 34 golden margins UNCHANGED; TSan clean | **Parallel** over CONV output lines via `vt::cpu::ParallelForRows` ([#1009](https://github.com/mudler/vllm.cpp/issues/1009)). ~9x at 16-20 workers, contended box, 21-23% spread. Bit-identical at any count | | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated | +| LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 18 mutations, 17 DETECTED (four by review of a conditional-only #1039 gate) and the 18th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only GUIDED arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | +| LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 ([#1039](https://github.com/mudler/vllm.cpp/issues/1039)) | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | MTP speculation DEPTH (`num_speculative_tokens` > 1) | Qwen3.5/3.6 `mtp.*` heads | k=1..4 through the loader, greedy tokens unmoved, two witnesses per arm: the draft decode forwards the propose RAN, and whether the DELIVERED draft row varied with depth. `test_mtp_depth` 5/5, 63 assertions | Default stays k=1. NO speed claim at k>1. Drafts are proposed and verified, never ACCEPTED, and neither witness proves per-column provenance. Both await the owed DGX gate (#81) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | diff --git a/docs/USAGE.md b/docs/USAGE.md index 040f1c4b9..46df2b369 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -842,6 +842,74 @@ The two knobs beside it are per-GENERATION and therefore CLI and ABI only, becau `num_generated_keyframes` on the other pipelines, and `temporal_upsample_rounds` below. +### LTX-2.5 text-to-audio: a render with no picture + +`--pipeline-kind t2a_one_stage` runs upstream's `T2AOneStagePipeline`, which +generates a soundtrack and no video at all. The result carries an `audio.wav`, +`frame_count = 0`, an empty frame directory and **no ffmpeg argv**, because there +is nothing to mux. + +```sh +ltx2-gen --dit ltx-2.5-dit.safetensors \ + --audio-vae ltx-2.5-audio-vae-bf16.safetensors \ + --encoder gemma4-12b-with-proj.safetensors --encoder-config gemma4.json \ + --pipeline-kind t2a_one_stage --device cpu \ + --frames 121 --steps 30 --prompt "rain on a tin roof, distant thunder" \ + --workdir /tmp/t2a +``` + +**These file names are not a checkpoint pin, and no LTX-2.5 recipe in this +document is.** None of them names a HuggingFace repo, a revision or a sha256, +which AGENTS.md § *Say which weights, and from where* requires; MiniMax-H3 and +MiniMax-Music3 below each carry a full table and LTX-2.5 carries none. That is +campaign-wide and pre-existing rather than particular to this recipe, and it is +recorded rather than invented, because no LTX-2.5 arm here has been rendered on +real weights yet. Tracked by +[#1048](https://github.com/mudler/vllm.cpp/issues/1048); read `--dit` above as +"the LTX-2.5 transformer", which the other recipes on this page spell as +`ltx-2.5-22b-distilled-fp8.safetensors` together with the `--dit-config` its +missing `__metadata__` requires. + +**No `--video-vae` is needed**, and none is loaded: upstream's pipeline never +constructs a video VAE. `--width` and `--height` are **refused** rather than +ignored — upstream passes a 512x512 placeholder whose height and width it +documents as unused, and only the frame count and the recipe's frame rate are +read, to derive the duration. + +**It is the only GUIDED arm, and that changes what it costs and what it needs.** +The distilled video recipes run one DiT forward per step. This one runs +**three** by default — conditional, unconditional, and one with the audio +self-attention perturbed (STG) — so it is roughly 3x the work per step, and it +**requires a text tower**, because the unconditional pass conditions on the +negative prompt. Loading with `prompt_embeds_path` alone gets a refusal naming +`--audio-cfg-guidance-scale 1.0` as the way to turn the unconditional pass off. + +Six per-generation knobs mirror upstream's own CLI, and each takes the +checkpoint generation's value when absent: `--negative-prompt`, +`--audio-cfg-guidance-scale` (7.0), `--audio-stg-guidance-scale` (1.0), +`--audio-rescale-scale` (0.7), `--audio-skip-step` (0) and `--audio-stg-blocks` +(28 on the 2.3-and-later lineage), which is comma separated. A block index +outside the DiT's own layer count is refused rather than clamped. There is no +`modality_scale` knob: upstream pins it to 1.0 for this pipeline, because +audio-only generation has no video modality to isolate. + +`--audio-rescale-scale` acts on the **denoised (x0) prediction**, not on the +DiT's velocity, because upstream's guider sits behind an `X0Model` and combines +already-converted tensors. The distinction is invisible at `0.0`, where the two +readings agree exactly, and it changes the render at every other value — so a +recipe or a script that was tuned against the velocity reading will not +reproduce here at the default `0.7` (issue #1039). + +Being per-generation, those six reach the CLI and the C ABI and **not** +`/v1/videos`, which forwards no per-generation extra to any engine (issue #928). +`pipeline_kind` is a LOAD knob and does reach the server, so a server started +with `--video-extra pipeline_kind=t2a_one_stage` renders every request as audio +at the recipe's own guider values. + +**The accelerator is refused by name.** `device = 1` gets a refusal on this +pipeline: the device forward takes both streams by reference and this pipeline +has no video stream to give it. Use `--device cpu`. + **What is not served.** `temporal_upsample_rounds` is defined and refused above `0`: the rounds loop that temporally doubles the latent, re-tiles the canvas and stitches it back is not ported. The refusal names it, and it names three things @@ -2374,7 +2442,8 @@ or without the ComfyUI `model.diffusion_model.` prefix. Each family reads its ow knobs from `extras`. H3 takes `partition`. LTX-2.5 takes `audio_prompt_embeds_path` (the audio stream's conditioning, the twin of the seam's `prompt_embeds_path`, which carries the video stream), `pipeline_kind` -(default `distilled_two_stage`), `model_version` (only for a checkpoint that +(default `distilled_two_stage`; also `one_stage`, `dmd2`, `dfr`, `retake` and +`t2a_one_stage`), `model_version` (only for a checkpoint that declares none), `dit_config_path`, `encoder_config_path`, `allow_unported_modules`, `max_phase`, `prompt_embeds_valid_rows`, `upsampler_path` and `duration_head_path`. An extra a family does not define is diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index 2e16774f0..7f4106f49 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -169,7 +169,21 @@ const char* Need(int argc, char** argv, int i, const char* flag) { "--width, --height and --frames are refused alongside it. --regenerate-video 0\n" "freezes the clip instead; --regenerate-audio has no effect while the source is\n" "a frame folder, because a folder carries no audio and both of upstream\'s audio\n" - "predicates test for one.\n"); + "predicates test for one.\n\n" + "TEXT-TO-AUDIO renders a soundtrack and NO PICTURE. --pipeline-kind\n" + "t2a_one_stage selects it; the result carries an audio.wav, zero frames and no\n" + "ffmpeg argv, because there is nothing to mux. --video-vae is not needed and\n" + "--width/--height are refused: upstream passes a 512x512 placeholder whose\n" + "height and width it documents as unused, and only --frames and the recipe\'s\n" + "frame rate are read, to derive the DURATION. Unlike the distilled video\n" + "recipes this one is GUIDED: it runs three DiT forwards per step by default\n" + "(conditional, unconditional, and one with the audio self-attention perturbed),\n" + "so it needs a text tower for the negative prompt. --negative-prompt,\n" + "--audio-cfg-guidance-scale, --audio-stg-guidance-scale, --audio-rescale-scale,\n" + "--audio-skip-step and --audio-stg-blocks are upstream\'s own flags; absent, each\n" + "takes the checkpoint generation\'s own value. --audio-stg-blocks is comma\n" + "separated and a block index outside the DiT\'s layer count is refused rather\n" + "than clamped. The accelerator is REFUSED by name on this pipeline.\n"); std::exit(code); } @@ -186,6 +200,10 @@ int main(int argc, char** argv) { // RETAKE (row LTX25-RETAKE, #924): a source clip DIRECTORY and the window to // regenerate. `--ref-video` is a directory of frame_%06d.ppm, not a container. std::string ref_video, retake_start, retake_end, retake_fps, regen_video, regen_audio; + // TEXT-TO-AUDIO (row LTX25-T2A-ONE-STAGE, #1005): one flag per argument of + // upstream's `default_1_stage_t2a_arg_parser` (utils/args.py:1070-1120). + std::string negative_prompt, audio_cfg_scale, audio_stg_scale, audio_rescale; + std::string audio_skip_step, audio_stg_blocks; // The extras are BORROWED by the load call, so the strings must outlive it. // Kept as two parallel vectors of owned strings plus the char* views the ABI @@ -262,6 +280,21 @@ int main(int argc, char** argv) { retake_end = Need(argc, argv, ++i, "--retake-end-time"); else if (f == "--retake-frame-rate") retake_fps = Need(argc, argv, ++i, "--retake-frame-rate"); + // TEXT-TO-AUDIO (#1005). Selected by `--pipeline-kind t2a_one_stage`, which + // is a LOAD extra; these six are per-generation and are refused by name on + // any other pipeline rather than accepted and ignored. + else if (f == "--negative-prompt") + negative_prompt = Need(argc, argv, ++i, "--negative-prompt"); + else if (f == "--audio-cfg-guidance-scale") + audio_cfg_scale = Need(argc, argv, ++i, "--audio-cfg-guidance-scale"); + else if (f == "--audio-stg-guidance-scale") + audio_stg_scale = Need(argc, argv, ++i, "--audio-stg-guidance-scale"); + else if (f == "--audio-rescale-scale") + audio_rescale = Need(argc, argv, ++i, "--audio-rescale-scale"); + else if (f == "--audio-skip-step") + audio_skip_step = Need(argc, argv, ++i, "--audio-skip-step"); + else if (f == "--audio-stg-blocks") + audio_stg_blocks = Need(argc, argv, ++i, "--audio-stg-blocks"); else if (f == "--regenerate-video") regen_video = Need(argc, argv, ++i, "--regenerate-video"); else if (f == "--regenerate-audio") @@ -330,7 +363,20 @@ int main(int argc, char** argv) { std::make_pair("retake_end_time", &retake_end), std::make_pair("retake_frame_rate", &retake_fps), std::make_pair("regenerate_video", ®en_video), - std::make_pair("regenerate_audio", ®en_audio)}) { + std::make_pair("regenerate_audio", ®en_audio), + // TEXT-TO-AUDIO (#1005). One flag per upstream CLI + // argument (`default_1_stage_t2a_arg_parser`, + // ltx-pipelines utils/args.py:1070-1120). They are + // per-generation, so they ride this array rather than + // the load one; `--pipeline-kind t2a_one_stage` is the + // LOAD knob that selects the pipeline, and supplying + // these without it is refused by name. + std::make_pair("negative_prompt", &negative_prompt), + std::make_pair("audio_cfg_guidance_scale", &audio_cfg_scale), + std::make_pair("audio_stg_guidance_scale", &audio_stg_scale), + std::make_pair("audio_rescale_scale", &audio_rescale), + std::make_pair("audio_skip_step", &audio_skip_step), + std::make_pair("audio_stg_blocks", &audio_stg_blocks)}) { if (kv.second->empty()) continue; gen_keys.emplace_back(kv.first); gen_values.push_back(*kv.second); diff --git a/include/vllm/model_executor/models/ltx2.h b/include/vllm/model_executor/models/ltx2.h index 339908dab..998cda7c9 100644 --- a/include/vllm/model_executor/models/ltx2.h +++ b/include/vllm/model_executor/models/ltx2.h @@ -39,11 +39,28 @@ // message naming the missing phase rather than silently computing in f32. // // NOT PORTED IN L2, recorded here so it cannot be discovered later: -// - Guidance perturbations (guidance/perturbations.py BatchedPerturbationConfig, -// transformer.py:330-397 `*_perturbation_mask` / `cross_attn_skip_all` and -// attention.py:545-552 `perturbation_mask`). L2 runs the no-perturbation -// configuration, whose masks are all-ones and whose flags are all false — -// upstream's own `perturbations=None` path (model.py:509-511). +// - The CROSS-attention guidance perturbations only, as of row +// LTX25-T2A-ONE-STAGE (#1005): `SKIP_A2V_CROSS_ATTN` and +// `SKIP_V2A_CROSS_ATTN` (guidance/perturbations.py:8-16, +// transformer.py:330-397 `cross_attn_skip_all`). Nothing upstream that this +// port serves constructs them — STG is built from `stg_blocks` and reaches +// the SELF-attention types alone (utils/constants.py:49-68 through +// guiders.py:194-211) — and `Ltx2Attention` refuses the flag on a cross call +// by name rather than applying the self-attention rule to it. +// +// THE SELF-ATTENTION HALF IS NOW PORTED, and this entry said the whole +// mechanism was unported until #1005. `Ltx2DitPerturbation` is upstream's +// `perturbations` argument (model.py:492) at the one batch size this port +// serves, and `Ltx2AttentionArgs::all_perturbed` is +// `use_attention = not all_perturbed` (attention.py:557). `nullptr` remains +// upstream's `perturbations=None` path (model.py:509-511) and is what every +// caller but text-to-audio passes. +// +// The BATCHED form (`BatchedPerturbationConfig`, perturbations.py:53-143, +// indexed [type, block, SAMPLE]) is still unported, and so is the partial +// blend it exists for (`out * mask + v * (1 - mask)`, attention.py:571-572). +// Both are degenerate at `Ltx2ModalityInput::batch == 1`, which is the only +// batch any path here runs. // - The caption projections (text_projection.py:31-38). LTX-2.5 is a 22B-form // checkpoint: `caption_proj_before_connector=true` puts them in the TEXT // ENCODER, so the DiT has none (model_configurator.py:199-219). They are @@ -430,6 +447,24 @@ struct Ltx2AttentionArgs { // a path whose K/V really are timestep-independent — the caller owns that. const Ltx2CrossKv* kv_in = nullptr; Ltx2CrossKv* kv_out = nullptr; + + // `all_perturbed` (attention.py:552-553, `use_attention = not all_perturbed` + // at :557). The STG perturbation: the attention output is REPLACED by the raw + // value projection, `out = v`, and `to_q` / `to_k` / the q,k RMSNorms / RoPE / + // the attention itself never run. `to_out` still does (`:579`), and the gate + // still does (`:576-578`) — the substitution happens INSIDE the attention, not + // around it, so a port that returned `v` to the caller and skipped `to_out` + // would be a different operator. + // + // Upstream also carries a PARTIAL form, `out = out * mask + v * (1 - mask)` + // (`:571-572`), which blends per BATCH ROW. It is not ported here and it is not + // needed: it exists so one batch can mix perturbed and unperturbed samples, + // and this port runs `max_batch_size = 1` (`Ltx2ModalityInput::batch`), where + // the mask is all-ones or all-zeros and the blend degenerates to exactly the + // two cases this flag expresses. Recorded rather than assumed, because a + // batched arm that reached this field would need the blend and would get the + // all-or-nothing answer silently. + bool all_perturbed = false; }; std::vector Ltx2Attention(vt::Device device, const Ltx2AttentionWeights& w, const float* x, const float* context, const Ltx2AttentionArgs& args); @@ -498,14 +533,59 @@ struct Ltx2DitOutputs { std::vector audio; // [batch, audio tokens, audio_out_channels] }; -// LTXModel.forward (model.py:492-538) for model_type=AudioVideo, plus the -// preprocessors it drives (transformer_args.py:263-411). BOTH streams are -// required: LTX-2.5 is an AudioVideo checkpoint, and the VideoOnly / AudioOnly -// model types (model.py:31-33) build a different parameter set, so they are -// refused rather than served by an ungated path. To run one stream of an AV model -// — which is what upstream's own pipeline does — clear `enabled` on the other; -// the audio<->video cross attention still reads its state, exactly as -// transformer.py:265-269 does. +// `perturbations` on LTXModel.forward (model.py:492), reduced to what this port +// can express. One entry per BLOCK; `true` means that block's self-attention is +// replaced by its value projection, which is STG. +// +// A `BatchedPerturbationConfig` upstream (guidance/perturbations.py:53-143) is +// indexed [type, block, SAMPLE]; this is indexed [block] alone, because +// `Ltx2ModalityInput::batch` is 1 on every path here and the sample axis is a +// degenerate one. `Ltx2BatchedPerturbationConfig` (ltx2_pipeline.h) is the +// batched form and stays ungated by any product caller — recorded as owed in +// .agents/specs/ltx25-t2a-one-stage.md rather than silently bypassed. +// +// EMPTY IS NOT "NOTHING PERTURBED BY COINCIDENCE": a vector of the wrong length +// is REFUSED, so a config built for a different layer count cannot silently +// perturb the first N blocks and leave the rest alone. +// +// The two CROSS-attention perturbation types upstream defines +// (SKIP_A2V_CROSS_ATTN, SKIP_V2A_CROSS_ATTN, perturbations.py:8-16) have no +// field here and are not ported. Nothing upstream that this port serves +// constructs them: STG is built from `stg_blocks` and reaches the self-attention +// types only (utils/constants.py:49-68 through guiders.py:194-211). +struct Ltx2DitPerturbation { + std::vector video_self_attn; // [num_layers], empty = none + std::vector audio_self_attn; // [num_layers], empty = none +}; + +// LTXModel.forward (model.py:492-538), plus the preprocessors it drives +// (transformer_args.py:263-411). +// +// EXACTLY ONE OF THE TWO STREAMS MAY BE NULL, which is upstream's own +// `video_args = ... if video is not None else None` (model.py:505) and the +// shape `T2AOneStagePipeline` runs (t2a_one_stage.py:167 passes `video=None`). +// Both null is refused: upstream refuses it too (transformer.py:259-260, "At +// least one of video or audio must be provided"). +// +// `video = nullptr` IS NOT `video->enabled = false`, and the difference renders +// rather than failing. Upstream's predicate is `run_v2a = run_ax and (video is +// not None and vx.numel() > 0)` (transformer.py:269) — it tests PRESENCE, not +// `enabled` — so a disabled-but-present video stream still feeds video->audio +// cross attention from that stream's latent. This header used to advise +// `enabled` as the way to run one stream, and for the audio-only case that +// advice was wrong. `enabled` remains correct for running one stream of a JOINT +// render, where the cross attention reading the other stream's state is the +// intent. +// +// WHAT IS *NOT* THE REASON, re-derived at this tree rather than inherited: the +// AudioOnly / VideoOnly WEIGHT CONTRACT is not what blocks a one-stream call, and +// this check used to say it was. `T2AOneStagePipeline` loads the ordinary +// AudioVideo checkpoint FILE and merely restricts which keys are read +// (LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP, model_configurator.py:228-239), so +// the contract `EnumerateLtx2DitTensors` describes is the one it satisfies. What +// remains true is a statement about the LOADER: a checkpoint saved with only the +// audio subset still cannot be materialized here, and that refusal lives at +// `Ltx2LoadDitFromSafetensors`, where it is about the file. // // `compute_dtype` must be vt::DType::kF32 — see the DTYPE note at the top of this // file. Anything else is REFUSED with a message naming phase L6. @@ -521,7 +601,8 @@ struct Ltx2DitOutputs { Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, const Ltx2DitWeights& weights, const Ltx2ModalityInput* video, const Ltx2ModalityInput* audio, vt::DType compute_dtype, - Ltx2PromptKvCache* cache = nullptr); + Ltx2PromptKvCache* cache = nullptr, + const Ltx2DitPerturbation* perturbations = nullptr); // One BasicAVTransformerBlock (transformer.py:254-417), exposed so the block is // gateable on its own. `video_x` / `audio_x` are updated IN PLACE. @@ -541,6 +622,9 @@ struct Ltx2BlockArgs { // only the static per-block table applies (:441). const float* video_prompt_modulation = nullptr; const float* audio_prompt_modulation = nullptr; + // STG for THIS block (attention.py:552-577). See `Ltx2DitPerturbation`. + bool video_self_attn_perturbed = false; + bool audio_self_attn_perturbed = false; // Audio<->video cross-attention AdaLN inputs (transformer_args.py:388-411). const float* video_cross_scale_shift = nullptr; // [batch, video tokens, 4 * dim] const float* video_cross_gate = nullptr; // [batch, 1, dim] diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index 419d546ac..93c75cc22 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -562,6 +562,17 @@ struct Ltx2PipelineRecipe { bool allow_negative_prompt = true; bool fixed_num_inference_steps = false; + // `T2AOneStagePipeline` (t2a_one_stage.py:43). TRUE means the pipeline passes + // `video=None` to the stage (`:167`) and returns a waveform and nothing else + // (`:172`) — there is no video latent, no video VAE decode and no frame. + // + // A FLAG ON THE RECIPE RATHER THAN A STRING COMPARE AT THE CALL SITE, because + // the engine has to answer "is there a picture" in four places (geometry, + // resolution guard, decode, artifacts) and four independent `kind == + // "t2a_one_stage"` tests are four chances for one of them to be missed on the + // next audio-only recipe. The recipe table is the one place that knows. + bool audio_only = false; + int64_t max_spatial_downscale() const; }; @@ -627,6 +638,21 @@ void Ltx2AssertResolution(int64_t height, int64_t width, int64_t divisor); // `use_keyframes_abs_pos_embedding` // ("retake", "2") Lightricks retake.py:85,287,290-294,313-324 // ("retake", "2.5") same +// ("t2a_one_stage", "2") Lightricks t2a_one_stage.py:43,109 (row +// ("t2a_one_stage", "2.3") LTX25-T2A-ONE-STAGE, #1005). The one_stage +// ("t2a_one_stage", "2.4") rows' own schedule with `audio_only` set: +// ("t2a_one_stage", "2.5") T2A hard-codes the SAME `LTX2Scheduler()` +// (t2a_one_stage.py:67 against +// ti2vid_one_stage.py:81) and the same +// `detect_params` step count, and differs in +// carrying no video stream at all +// +// The four `t2a_one_stage` rows mirror the four `one_stage` rows one for one, and +// the negative prompt follows the same split for the same reason: it travels with +// the GENERATION, not with the pipeline. There is no "which versions support +// text-to-audio" question upstream — `T2AOneStagePipeline` takes whatever +// `resolve_cli_params` read off the checkpoint (t2a_one_stage.py:178-179), so +// restricting these rows to 2.5 would be a local invention. // // The `retake` rows are Lightricks' `RetakePipeline` and have no vLLM-Omni // counterpart at all. Every value on them is read off `retake.py` rather than diff --git a/include/vllm/model_executor/models/ltx2_t2a.h b/include/vllm/model_executor/models/ltx2_t2a.h new file mode 100644 index 000000000..0fc064663 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_t2a.h @@ -0,0 +1,249 @@ +// LTX-2.5 TEXT-TO-AUDIO — the port of `T2AOneStagePipeline`, and the first path +// in this tree that renders no picture at all. +// +// Row: LTX25-T2A-ONE-STAGE. Spec: .agents/specs/ltx25-t2a-one-stage.md. +// Issue #1005. Campaign #644 / #435. +// +// ─── WHAT THIS TU IS A PORT OF (file:line on BOTH sides) ───────────────────── +// Upstream: Lightricks/LTX-2 @ fd4ded7f +// OURS <- UPSTREAM +// Ltx2T2aGenerate <- ltx-pipelines t2a_one_stage.py:109-172 +// (T2AOneStagePipeline.__call__) +// the guided step <- ltx-core components/guiders.py:244-273 +// (MultiModalGuider.calculate), reached through +// ltx-pipelines utils/denoisers.py:188-203 +// (FactoryGuidedDenoiser) +// the x0 wrapper on each pass <- ltx-core model/transformer/model.py:590-604 +// (X0Model.forward), which is what +// ltx-pipelines utils/blocks.py:480-482 builds +// and hands the denoiser +// the STG pass <- ltx-core model/transformer/attention.py:552-577 +// the audio latent shape <- ltx-core types.py:164-200 +// (AudioLatentShape.from_video_pixel_shape) +// the schedule <- ltx-core components/schedulers.py:21-57, +// hard-coded as LTX2Scheduler() at +// t2a_one_stage.py:67 +// +// ─── WHY IT IS ITS OWN TRANSLATION UNIT ────────────────────────────────────── +// +// Upstream's T2A is its own FILE with its own `__call__`, and AGENTS.md +// §"Shared seams" requires mirroring that structure. It is reached only through +// `Ltx2VideoEngine::Generate`, it owns no weights, and every numeric it uses is +// an already-gated brick — so it is a composition, not a second path. +// +// What it deliberately does NOT do is thread an `is_t2a` flag through the joint +// phase driver in `ltx2_video.cpp`. That function is 1900 lines and roughly a +// third of it constructs a video stream that does not exist here; nine new +// branches inside it would be nine chances to leave one behind, and a missed one +// renders. +// +// ─── THE FOUR THINGS THAT FAIL SILENTLY IF GUESSED ─────────────────────────── +// +// 1. `video = nullptr`, NOT `video->enabled = false`. Upstream's predicate is +// `run_v2a = run_ax and (video is not None and vx.numel() > 0)` +// (transformer.py:269) — it tests PRESENCE. A disabled-but-present video +// stream still feeds video->audio cross attention from a latent T2A never +// meant to exist, and still returns a finished, playable waveform. +// +// 2. THE GUIDER IS NOT OPTIONAL HERE. `distilled_two_stage` builds a +// `SimpleDenoiser` upstream and this engine's joint loop mirrors that with one +// forward per step. T2A builds a `FactoryGuidedDenoiser` +// (t2a_one_stage.py:154-161) whose CLI defaults are `cfg_scale=7.0` and +// `stg_scale=1.0` (utils/constants.py:58-66 through :118), so +// `do_unconditional_generation` and `do_perturbed_generation` are both TRUE +// (guiders.py:275-281) and the default path is THREE forwards per step. A +// single-forward T2A produces audio of exactly the right length on a +// trajectory the model was not asked for. +// +// 3. `modality_scale` IS PINNED TO 1.0, and the pin is upstream's, at the CLI +// layer: "Audio-only generation has no video modality, so the video->audio +// (v2a) cross-modal guidance is meaningless here. 1.0 disables it" +// (t2a_one_stage.py:200-202). The params table's own value is 3.0, so +// inheriting it would turn on a fourth forward against a modality that is not +// there. +// +// 4. THE GUIDER COMBINES X0, NOT VELOCITY (#1039). Upstream builds the +// denoiser's transformer as `X0Model(...)` (ltx-pipelines +// utils/blocks.py:480-482), so every pass `_guided_denoise` hands +// `MultiModalGuider.calculate` has ALREADY been converted with +// `to_denoised(latent, v, timesteps)` (model.py:590-604, `to_denoised` at +// ltx-core utils.py:39-52) before it is combined +// (utils/denoisers.py:188-203). +// +// Combining raw velocities and converting once afterwards is the SAME +// FUNCTION only while `rescale_scale == 0`, because `calculate`'s linear +// terms are invariant under `x0 = latent - sigma*v`. The rescale branch is +// not invariant: upstream computes `factor` from `std(x0_cond)/std(x0_pred)` +// and scales the whole x0, giving `factor*(latent - sigma*v)`; scaling the +// velocity instead gives `latent - sigma*factor*v`. The two differ by +// `(factor - 1) * latent` — zero only where the latent is zero, which on +// this path it never is (the state IS the unit-variance noise, item above). +// `rescale_scale = 0.7` is the shipped T2A default (utils/constants.py:63, +// utils/args.py:1101-1106), so this is the DEFAULT arm rather than an +// exotic one, and nothing about the rendered waveform separates the two. +#pragma once + +#include +#include +#include + +#include "vllm/model_executor/models/ltx2.h" +#include "vllm/model_executor/models/ltx2_audio_vae.h" +#include "vllm/model_executor/models/ltx2_loader.h" +#include "vllm/model_executor/models/ltx2_pipeline.h" +#include "vllm/model_executor/models/ltx2_video_vae.h" +#include "vt/device.h" + +namespace vllm { + +// Everything `T2AOneStagePipeline.__call__` reads, as borrowed pointers. Nothing +// here is owned: the engine holds the weights for its whole lifetime and this +// call runs inside its mutex. +struct Ltx2T2aRequest { + vt::Device device; + vt::DType compute_dtype = vt::DType::kF32; + const Ltx2DitParams* dit_params = nullptr; + const Ltx2DitWeights* dit_weights = nullptr; + + // The AUDIO conditioning rows, `[context_tokens, audio_cross_attention_dim]`. + // `negative_context` is `ctx_n.audio_encoding` (t2a_one_stage.py:135) and is + // null when the caller has no negative conditioning — which is a REFUSAL when + // the guider asks for an unconditional pass, never a silent drop to + // `uncond = 0`. + const float* context = nullptr; + const float* negative_context = nullptr; + int64_t context_tokens = 0; + + int64_t num_frames = 0; + double frame_rate = 0.0; + int64_t steps = 0; // <= 0 => the recipe's own count + + // `GaussianNoiser(generator=torch.Generator(...).manual_seed(seed))` + // (t2a_one_stage.py:124-125). Supplied by the CALLER rather than constructed + // here: the engine already owns the one Gaussian source in this tree, and a + // second generator seeded the same way would be a parallel path whose + // agreement with the first nothing gates. + Ltx2NoiseStream* noise = nullptr; + + Ltx2MultiModalGuiderParams guidance; + + const Ltx2AudioDecoderConfig* audio_cfg = nullptr; + const Ltx2VaeWeights* audio_weights = nullptr; + const Ltx2VocoderBweConfig* vocoder_cfg = nullptr; + const Ltx2VaeWeights* vocoder_weights = nullptr; +}; + +// The rendered soundtrack, plus the observability the render itself cannot be +// inspected for. +// +// EVERY COUNTER BELOW IS INCREMENTED AT THE FORWARD, not derived from the params +// that were supposed to drive it. A field written from `guidance.cfg_scale` +// would report a healthy uncond count on a build that computed the params and +// then ran one forward — which is the exact instrument failure +// `Ltx2ConditioningTrace::audio_frozen` already paid for on this campaign. +struct Ltx2T2aResult { + // [channels, samples_per_channel], the vocoder's own layout. + std::vector waveform; + int64_t channels = 0; + int64_t samples_per_channel = 0; + int64_t sample_rate = 0; + + int64_t audio_tokens = 0; + int64_t latent_frames = 0; + + // Forwards actually issued, by arm. + int64_t cond_forwards = 0; + int64_t uncond_forwards = 0; + int64_t perturbed_forwards = 0; + + // TRUE if any forward was handed a video stream. It must be FALSE on every + // T2A render, and it is recorded rather than asserted in prose because + // §"THE THREE THINGS" item 1 is invisible in the output: a run that passed a + // present-but-disabled video stream produces a waveform of exactly the right + // length, the right channel count and the right sample rate. + bool video_stream_present = false; + + // The blocks the STG pass actually perturbed, read off the vector handed to + // the forward. A count alone cannot tell "perturbed block 1" from "perturbed + // block 0", and `stg_blocks` is what decides which. + std::vector perturbed_blocks; + + // FNV-1a over the final audio latent's raw f32 bytes, and its max|x|. The + // digest detects CHANGE; the absmax is the lower bound a digest cannot make, + // because a latent that collapsed to zeros has a perfectly stable digest. + uint64_t latent_digest = 0; + double latent_absmax = 0.0; + + // EVERYTHING STEP 0 PRODUCED, in the order it produced it: the sampler's + // input, EVERY GUIDANCE PASS as a (raw velocity, x0 prediction) pair, the + // guider's result, and the latent the Euler step wrote. They exist because + // #1039 is invisible in every other field here — combining the guidance passes + // in VELOCITY space and converting once afterwards produces a waveform of the + // right length, the right channel count, the right sample rate and a perfectly + // healthy forward count — and together they make the question decidable by + // arithmetic rather than by magnitude: + // + // first_step_cond == first_step_latent - sigma * first_step_velocity + // first_step_uncond == first_step_latent - sigma * first_step_uncond_velocity + // first_step_perturbed == first_step_latent - sigma * first_step_perturbed_velocity + // + // holds when the guider is handed X0 PREDICTIONS, as `X0Model.forward` does + // (model.py:590-604, over the `X0Model(...)` that utils/blocks.py:480-482 + // builds), and fails when it is handed the velocities. `first_step_cond` is + // also upstream's own `DenoisedLatentResult.cond` (utils/denoisers.py:206), + // rather than a field invented for a test. + // + // ONE PAIR PER ARM, AND NOT ONLY THE CONDITIONAL ONE. The default T2A arm runs + // THREE forwards per step (header item 2), and a build that converts the + // conditional pass and leaves either of the other two in velocity space + // renders a different waveform through a guider whose cond term is impeccable. + // A single recorded pair holds the claim "`to_denoised` on the way out of the + // forward" for one third of the passes it is made about; the review that found + // #1039 mutated exactly those other two arms and the gate stayed green. + // + // `first_step_next_latent` is what `Ltx2EulerStep` WROTE, and it is here so + // that what the sampler CONSUMED is checkable rather than assumed: + // + // first_step_next_latent + // == latent + (latent - first_step_denoised)/sigma * (sigma_next - sigma) + // + // A second `ToDenoised` applied to the guider's output on the way into the + // step — the residue a partial #1039 repair leaves behind — moves this and + // nothing else. + // + // STEP 0 SPECIFICALLY, because it is the one step whose inputs do not depend + // on any earlier step, so two renders that differ only in a guider parameter + // share a bit-identical step-0 latent and bit-identical DiT passes. + // + // The uncond and perturbed pairs stay EMPTY when the guider does not ask for + // that arm (`cfg_scale == 1.0`, `stg_scale == 0.0`), because the forward did + // not run. An empty vector is the honest record of a pass that never happened; + // a zero-filled one of the right length would be indistinguishable from a + // forward that returned zeros. + std::vector first_step_latent; + std::vector first_step_velocity; + std::vector first_step_cond; + std::vector first_step_uncond_velocity; + std::vector first_step_uncond; + std::vector first_step_perturbed_velocity; + std::vector first_step_perturbed; + std::vector first_step_denoised; + std::vector first_step_next_latent; + double first_step_sigma = 0.0; +}; + +// `T2AOneStagePipeline.__call__` (t2a_one_stage.py:109-172). Throws +// std::runtime_error naming the problem; never renders a fallback. +Ltx2T2aResult Ltx2T2aGenerate(const Ltx2T2aRequest& req); + +// `stg_blocks` -> the per-block vector `Ltx2DitForward` takes +// (guidance/perturbations.py:19-33, `blocks is None` meaning ALL). Exposed so a +// test can pin the mapping without reaching into the pipeline, and so the one +// place that turns a block LIST into a block MASK is named. +// +// A block index outside `[0, num_layers)` is REFUSED. Upstream indexes +// `self.transformer_blocks` by it and would raise; silently ignoring it would +// run an unperturbed pass and call it STG. +std::vector Ltx2StgBlockMask(const std::vector& stg_blocks, int64_t num_layers); + +} // namespace vllm diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index affefbd1d..fe3007e6c 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -420,6 +420,45 @@ inline constexpr char kLtx2RetakeFrameRateExtra[] = "retake_frame_rate"; inline constexpr char kLtx2RegenerateVideoExtra[] = "regenerate_video"; inline constexpr char kLtx2RegenerateAudioExtra[] = "regenerate_audio"; +// ── TEXT-TO-AUDIO. Row LTX25-T2A-ONE-STAGE (#1005) ───────────────────────── +// +// These are read ONLY on a `pipeline_kind = t2a_one_stage` engine — that is a +// LOAD extra, so which pipeline runs is fixed before a request arrives. Supplied +// on any other pipeline they are REFUSED, because upstream's other entry points +// have no counterpart for them and a knob that silently does nothing is the +// defect this whole surface refuses by name elsewhere. +// +// `--negative-prompt` (ltx-pipelines utils/args.py:1083-1088). ABSENT MEANS the +// recipe's own default, which is upstream's `DEFAULT_NEGATIVE_PROMPT` +// (utils/constants.py:186) on the 2.4/2.5 rows. +// +// IT IS NOT COSMETIC ON THIS PIPELINE. T2A's CFG scale defaults to 7.0, so the +// negative conditioning is one of the two tensors the guidance delta is computed +// from (`(cfg_scale - 1) * (cond - uncond_text)`, guiders.py:262). An empty one +// is refused rather than substituted with zeros: a zero `uncond_text` turns the +// delta into `cfg_scale * cond`, which is a DIFFERENT render and not a missing +// one. +inline constexpr char kLtx2NegativePromptExtra[] = "negative_prompt"; + +// The audio guider, one CLI flag each (utils/args.py:1089-1119). ABSENT MEANS the +// params table's own value for the checkpoint's generation — 7.0 / 1.0 / 0.7 and +// block 28 on the 2.3-and-later lineage (utils/constants.py:58-66, :82-87). +// +// `audio_stg_blocks` is a COMMA-SEPARATED list, mirroring `nargs="*"`. An EMPTY +// value is upstream's empty list and means "perturb nothing" — which is refused +// alongside a non-zero STG scale rather than silently running a perturbed pass +// identical to the conditional one. Upstream's `blocks is None` ("every block", +// guidance/perturbations.py:19-33) has no CLI spelling and none is invented here. +// +// There is deliberately NO `modality_scale` knob: the CLI pins it to 1.0 for this +// pipeline and states the reason (t2a_one_stage.py:200-202), so exposing it would +// offer a fourth forward over a modality that does not exist. +inline constexpr char kLtx2AudioCfgScaleExtra[] = "audio_cfg_guidance_scale"; +inline constexpr char kLtx2AudioStgScaleExtra[] = "audio_stg_guidance_scale"; +inline constexpr char kLtx2AudioRescaleScaleExtra[] = "audio_rescale_scale"; +inline constexpr char kLtx2AudioSkipStepExtra[] = "audio_skip_step"; +inline constexpr char kLtx2AudioStgBlocksExtra[] = "audio_stg_blocks"; + // WHAT THE LAST `Generate()` ACTUALLY HANDED THE DiT's CROSS-ATTENTION. // // Every field is read off the exact f32 buffers `Ltx2ModalityInput::context` @@ -660,6 +699,80 @@ struct Ltx2ConditioningTrace { uint64_t retake_latent_digest = 0; double retake_latent_absmax = 0.0; + // ── TEXT-TO-AUDIO: what the audio-only render actually ran (#1005) ──────── + // + // Zero and false everywhere on a pipeline that is not `t2a_one_stage`. + // + // `t2a_video_stream_present` is the one that cannot be inferred from anything + // else here, and it is the field this block exists for. Upstream's own + // predicate is `run_v2a = run_ax and (video is not None and vx.numel() > 0)` + // (transformer.py:269): it tests PRESENCE, not `enabled`. So a build that + // handed the forward a present-but-DISABLED video stream would still feed + // video->audio cross attention from a latent T2A never meant to exist — and + // would return a waveform of exactly the right length, the right channel count + // and the right sample rate. There is no sample to compare and no digest of + // the output that says which happened. + // + // The three forward counters are incremented AT THE FORWARD, not derived from + // the guider parameters that were supposed to drive them. A field written off + // `cfg_scale` would report a healthy uncond count on a build that resolved the + // params and then ran one forward, which is the instrument failure + // `audio_sigma_max` above already paid for on this campaign. + // + // `t2a_perturbed_blocks` is read off the mask handed to the DiT rather than + // off the request's `stg_blocks`: a count alone cannot tell "perturbed block + // 1" from "perturbed block 0", and which block is perturbed is the whole of + // STG. + // + // The `t2a_first_*` block is everything step 0 produced, and it is the only + // observable that separates upstream's x0-space guidance combination from a + // velocity-space one (#1039). Every other field here — the forward counts, + // the perturbed blocks, the latent absmax, the waveform's length, channel + // count and sample rate — is identical between the two forms, and on a + // reduced fixture so is the rendered audio, because the guidance deltas are + // ~1e-5 of the prediction and the rescale factor lands within 1e-5 of 1.0 in + // BOTH spaces. What is not identical, and is not a matter of degree, is which + // tensor the guider was handed: + // + // t2a_first_cond == t2a_first_latent - sigma * t2a_first_velocity + // + // holds in x0 space (`X0Model.forward`, ltx-core model/transformer/ + // model.py:590-604) and fails in velocity space. + // + // ONE (velocity, x0) PAIR PER ARM. The default T2A guider runs THREE forwards + // per step, and the equation above decides only the pass it names. A build + // that converts the conditional pass and leaves the UNCONDITIONAL or the + // PERTURBED one in velocity space renders a different waveform with a healthy + // forward count, a correct `t2a_first_cond`, and nothing else to see it by — + // which is #1039 again, one arm over. + // + // `t2a_first_next_latent` is `Ltx2EulerStep`'s output, and it makes what the + // sampler CONSUMED checkable: + // + // next == latent + (latent - denoised)/sigma * (sigma_next - sigma) + // + // A second `to_denoised` applied to the guider's result on the way into the + // step moves this field and no other. + // + // The uncond and perturbed vectors are EMPTY when the guider does not ask for + // that arm, because the forward did not run. + bool t2a_rendered = false; + bool t2a_video_stream_present = false; + int64_t t2a_cond_forwards = 0; + int64_t t2a_uncond_forwards = 0; + int64_t t2a_perturbed_forwards = 0; + std::vector t2a_perturbed_blocks; + std::vector t2a_first_latent; + std::vector t2a_first_velocity; + std::vector t2a_first_cond; + std::vector t2a_first_uncond_velocity; + std::vector t2a_first_uncond; + std::vector t2a_first_perturbed_velocity; + std::vector t2a_first_perturbed; + std::vector t2a_first_denoised; + std::vector t2a_first_next_latent; + double t2a_first_sigma = 0.0; + // True only once the `Generate` that produced this conditioning RETURNED. The // trace is filled immediately after the connector and BEFORE the denoise loop, // because that is the only point at which the exact buffers cross-attention @@ -723,6 +836,21 @@ class Ltx2VideoEngine : public VideoEngine { Ltx2VideoEngine(); struct Impl; std::unique_ptr impl_; + + // `T2AOneStagePipeline.__call__` (ltx2_t2a.h), reached from `Generate` on an + // `audio_only` recipe and from nowhere else. + // + // A PRIVATE STATIC rather than a free function in the .cpp, because it needs + // `Impl` — a private nested type no non-member can name. The alternatives were + // a 150-line branch inside a function that is already 1900 lines, or a + // template whose only purpose is to deduce a type it is not allowed to spell. + // + // `audio_context` is the conditioning `Generate` already resolved: the audio + // half of the prompt encoding, after the connector. Passing it in rather than + // re-encoding is what keeps the audio-only arm from owning a second copy of + // the connector composition. + static VideoResult GenerateAudioOnly(Impl& im, const VideoGenParams& gen, + const float* audio_context, int64_t context_tokens); }; // Does this checkpoint set hold an LTX-2.5 DiT? Exposed for the registry and for diff --git a/src/vllm/model_executor/models/ltx2.cpp b/src/vllm/model_executor/models/ltx2.cpp index 188ee3a55..8dee3d093 100644 --- a/src/vllm/model_executor/models/ltx2.cpp +++ b/src/vllm/model_executor/models/ltx2.cpp @@ -822,6 +822,41 @@ std::vector Ltx2FeedForward(vt::Device device, const Ltx2FeedForwardWeigh return out; } +// attention.py:575-579 — everything the ordinary path and the STG-perturbed path +// share, which is the gate and `to_out`. Factored out rather than duplicated +// because the two arms differing HERE is the defect that would be invisible: a +// perturbed pass that skipped `to_out` returns a tensor of the right shape at the +// wrong width-space, and the block would add it to the residual and render. +static std::vector Ltx2AttentionEpilogue(vt::Queue& q, const Ltx2AttentionWeights& w, + const float* x, std::vector attn, + const Ltx2AttentionArgs& args, + vt::Device /*device*/) { + const int64_t batch = args.batch; + const int64_t tq = args.tokens; + const int64_t heads = args.heads; + const int64_t dim_head = args.dim_head; + const int64_t inner = heads * dim_head; + + // PytorchGatedAttention (ops.py:94-106), applied to the attention output BEFORE + // `to_out` (attention.py:576-579) and driven by the RAW input `x`, not by the + // attention output. Gating after `to_out` would be a different model. + if (w.to_gate_logits.weight.data != nullptr) { + std::vector logits(static_cast(batch * tq * heads)); + Linear(q, x, batch * tq, args.query_dim, w.to_gate_logits, logits.data()); + for (int64_t r = 0; r < batch * tq; ++r) { + for (int64_t h = 0; h < heads; ++h) { + const float gate = 2.0f / (1.0f + std::exp(-logits[static_cast(r * heads + h)])); + float* dst = attn.data() + r * inner + h * dim_head; + for (int64_t e = 0; e < dim_head; ++e) dst[e] *= gate; + } + } + } + + std::vector out(static_cast(batch * tq * args.query_dim)); + Linear(q, attn.data(), batch * tq, inner, w.to_out, out.data()); + return out; +} + std::vector Ltx2Attention(vt::Device device, const Ltx2AttentionWeights& w, const float* x, const float* context, const Ltx2AttentionArgs& args) { vt::Queue q{device, nullptr}; @@ -835,6 +870,28 @@ std::vector Ltx2Attention(vt::Device device, const Ltx2AttentionWeights& const int64_t s = context != nullptr ? args.context_tokens : tq; const int64_t ctx_dim = context != nullptr ? args.context_dim : args.query_dim; + // attention.py:557 — `use_attention = not all_perturbed`. The STG arm computes + // `to_v` and NOTHING else of the attention: no `to_q`, no `to_k`, no q/k + // RMSNorm, no RoPE, no scores. Written as an early exit rather than as a chain + // of `if (!perturbed)` guards so the skipped work is visibly skipped; a guarded + // form that still projected q and threw the result away would be numerically + // identical and would hide the whole point of the perturbation, which is that + // the query/key path does not run. + if (args.all_perturbed) { + VT_CHECK(context == nullptr, + "ltx2 attention: `all_perturbed` is upstream's SELF-attention STG perturbation " + "(guidance/perturbations.py:8-16 names SKIP_VIDEO_SELF_ATTN and " + "SKIP_AUDIO_SELF_ATTN). The CROSS-attention perturbations exist upstream " + "(SKIP_A2V_CROSS_ATTN, SKIP_V2A_CROSS_ATTN) and are NOT ported, so a cross call " + "carrying this flag is refused rather than served the self-attention rule"); + VT_CHECK(args.kv_in == nullptr && args.kv_out == nullptr, + "ltx2 attention: a perturbed pass computes no K, so it can neither fill nor read a " + "prompt K/V cache"); + std::vector vp(static_cast(batch * tq * inner)); + Linear(q, ctx, batch * s, ctx_dim, w.to_v, vp.data()); + return Ltx2AttentionEpilogue(q, w, x, std::move(vp), args, device); + } + // attention.py:559-565: v first, then q and k. The K/V half is exactly what the // prompt cache holds, so `kv_in` skips all three of to_v / to_k / k_norm. const bool reuse_kv = args.kv_in != nullptr; @@ -912,24 +969,7 @@ std::vector Ltx2Attention(vt::Device device, const Ltx2AttentionWeights& } } - // PytorchGatedAttention (ops.py:94-106), applied to the attention output BEFORE - // `to_out` (attention.py:576-579) and driven by the RAW input `x`, not by the - // attention output. Gating after `to_out` would be a different model. - if (w.to_gate_logits.weight.data != nullptr) { - std::vector logits(static_cast(batch * tq * heads)); - Linear(q, x, batch * tq, args.query_dim, w.to_gate_logits, logits.data()); - for (int64_t r = 0; r < batch * tq; ++r) { - for (int64_t h = 0; h < heads; ++h) { - const float gate = 2.0f / (1.0f + std::exp(-logits[static_cast(r * heads + h)])); - float* dst = attn.data() + r * inner + h * dim_head; - for (int64_t e = 0; e < dim_head; ++e) dst[e] *= gate; - } - } - } - - std::vector out(static_cast(batch * tq * args.query_dim)); - Linear(q, attn.data(), batch * tq, inner, w.to_out, out.data()); - return out; + return Ltx2AttentionEpilogue(q, w, x, std::move(attn), args, device); } } // namespace vllm diff --git a/src/vllm/model_executor/models/ltx2_dit.cpp b/src/vllm/model_executor/models/ltx2_dit.cpp index eaef5bb67..bdf6b98cf 100644 --- a/src/vllm/model_executor/models/ltx2_dit.cpp +++ b/src/vllm/model_executor/models/ltx2_dit.cpp @@ -275,6 +275,7 @@ void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, a.pe = args.video_pe; a.bias = args.video_self_bias; a.bias_rows = args.video_self_bias_rows; + a.all_perturbed = args.video_self_attn_perturbed; const std::vector msa = Ltx2Attention(device, w.attn1, norm_vx.data(), nullptr, a); PostSelfAttention(video_x, msa.data(), gate, batch * tv, dim, eps, &vx_normed); @@ -312,6 +313,7 @@ void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, a.pe = args.audio_pe; a.bias = args.audio_self_bias; a.bias_rows = args.audio_self_bias_rows; + a.all_perturbed = args.audio_self_attn_perturbed; const std::vector msa = Ltx2Attention(device, w.audio_attn1, norm_ax.data(), nullptr, a); PostSelfAttention(audio_x, msa.data(), gate, batch * ta, adim, eps, &ax_normed); @@ -752,19 +754,27 @@ Ltx2PromptIdentity Ltx2PromptIdentityOf(const Ltx2DitParams& params, Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, const Ltx2DitWeights& weights, const Ltx2ModalityInput* video, const Ltx2ModalityInput* audio, vt::DType compute_dtype, - Ltx2PromptKvCache* cache) { + Ltx2PromptKvCache* cache, const Ltx2DitPerturbation* perturbations) { VT_CHECK(compute_dtype == vt::DType::kF32, "ltx2: phase L2 ships only the f32 parity forward; the bf16 / FP8 / NVFP4 stream " "dtypes are phase L6 and are refused rather than silently computed in f32"); - // LTX-2.5 is an LTXModelType.AudioVideo checkpoint (model_configurator.py:47), - // and that is the only weight contract EnumerateLtx2DitTensors describes. The - // VideoOnly / AudioOnly types (model.py:31-33) build a DIFFERENT parameter set — - // no audio stream, no av_ca AdaLN embedders — so they are refused by name rather - // than served by a path no golden covers. Use `enabled` to run one stream of an - // AV model, which is what the pipeline itself does. - VT_CHECK(video != nullptr && audio != nullptr, - "ltx2: phase L2 ships the AudioVideo model type only; LTXModelType.VideoOnly and " - "LTXModelType.AudioOnly carry a different weight contract and are not ported"); + // transformer.py:259-260 — upstream's own refusal, in its own words: "At least + // one of video or audio must be provided". + // + // ONE stream may be null, which is what `T2AOneStagePipeline` runs + // (t2a_one_stage.py:167, `video=None`) and what model.py:505 expresses. + // This check used to demand BOTH and blamed the AudioOnly / VideoOnly WEIGHT + // CONTRACT for it. That reason was re-derived at this tree and does not + // describe the case: T2A loads the ordinary AudioVideo FILE and restricts which + // keys it reads (LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP, + // model_configurator.py:228-239), so the contract EnumerateLtx2DitTensors + // describes is the one it satisfies. Every line below was ALREADY written + // against `video != nullptr` / `have_both`, so lifting the guard reaches a path + // this file already had. The weight-contract statement survives where it is + // true — at the loader, about the file. + VT_CHECK(video != nullptr || audio != nullptr, + "ltx2: at least one of the video and audio streams must be present " + "(transformer.py:259-260)"); const int64_t dim = params.inner_dim(); const int64_t adim = params.audio_inner_dim(); // transformer_args.py:197 views the projected context to the STREAM width, so @@ -812,8 +822,24 @@ Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, params.audio_num_attention_heads, have_both ? video : nullptr); } + // `perturbations` (model.py:492). A vector that is not exactly `num_layers` + // long is REFUSED rather than indexed defensively: a config built for another + // layer count would otherwise perturb a prefix of the blocks and leave the rest + // alone, which is a legal-looking STG pass over the wrong blocks and renders. + if (perturbations != nullptr) { + for (const std::vector* v : + {&perturbations->video_self_attn, &perturbations->audio_self_attn}) { + VT_CHECK(v->empty() || static_cast(v->size()) == params.num_layers, + "ltx2: a perturbation vector is neither empty nor one entry per block"); + } + } + const bool use_cache = cache != nullptr; if (use_cache) { + VT_CHECK(video != nullptr && audio != nullptr, + "ltx2: the prompt K/V cache keys on BOTH streams' context tensors " + "(Ltx2PromptIdentityOf), so it is refused on a one-stream call rather than keyed on " + "half an identity — two different renders would otherwise share a cache entry"); // The cached K/V are a function of the PROMPT (and of nothing else on this // path — that is what use_prompt_adaln_single=false buys). A filled cache is // therefore bound to one prompt, and a call carrying another one is refused @@ -837,6 +863,12 @@ Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, a.audio_context_tokens = audio != nullptr ? audio->context_tokens : 0; a.video_enabled = video != nullptr && video->enabled; a.audio_enabled = audio != nullptr && audio->enabled; + if (perturbations != nullptr) { + a.video_self_attn_perturbed = !perturbations->video_self_attn.empty() && + perturbations->video_self_attn[static_cast(i)] != 0; + a.audio_self_attn_perturbed = !perturbations->audio_self_attn.empty() && + perturbations->audio_self_attn[static_cast(i)] != 0; + } a.video_timestep_modulation = vs.modulation.empty() ? nullptr : vs.modulation.data(); a.audio_timestep_modulation = as.modulation.empty() ? nullptr : as.modulation.data(); a.video_prompt_modulation = diff --git a/src/vllm/model_executor/models/ltx2_pipeline.cpp b/src/vllm/model_executor/models/ltx2_pipeline.cpp index 026227118..7291ecbb5 100644 --- a/src/vllm/model_executor/models/ltx2_pipeline.cpp +++ b/src/vllm/model_executor/models/ltx2_pipeline.cpp @@ -1068,6 +1068,23 @@ Ltx2PhaseRecipe OneStagePhase(const Ltx2PipelineParams& params) { phase.name = "generate"; phase.video_guidance = params.video_guider; phase.audio_guidance = params.audio_guider; + // #1013. This was left at the struct's 0.0 default, and 0.0 is not "no extra + // noise": `Ltx2GaussianNoise` is `latent + noise_scale * (noise - latent)`, so + // at 0.0 the state stays exactly as `create_initial_state` wrote it, which + // with no initial latent is ALL ZEROS. A one_stage render therefore denoised a + // zero tensor. + // + // Upstream's `ModalitySpec.noise_scale` defaults to 1.0 + // (ltx-pipelines/utils/types.py:110) and `TI2VidOneStagePipeline.__call__` + // constructs both specs without it (ti2vid_one_stage.py:233-239), so 1.0 is + // what reaches `GaussianNoiser.__call__`'s `torch.lerp(latent, noise, + // noise_scale)` (components/noisers.py:31). The two neighbouring recipes + // already set it explicitly, which is what made the omission legible. + // + // No gate saw it because every end-to-end test loads `distilled_two_stage`, + // and a zero-initialized denoise still returns a finite clip of the right + // size, frame count and sample rate. + phase.noise_scale = 1.0; return phase; } @@ -1085,6 +1102,43 @@ Ltx2PipelineRecipe OneStageRecipe(const Ltx2PipelineParams& params, return recipe; } +// `T2AOneStagePipeline` (t2a_one_stage.py:43). Built FROM `OneStageRecipe` +// rather than beside it, because upstream's difference between the two is not in +// the schedule: both hard-code `LTX2Scheduler()` (`:67` against +// ti2vid_one_stage.py:81) and both take `num_inference_steps` from the same +// `PipelineParams`. What differs is that there is no video. +// +// The geometry fields are left at the params table's values and are DEAD on this +// recipe — upstream fills the same slots with a 512x512 placeholder whose height +// and width it documents as unused (t2a_one_stage.py:37-40). Only `num_frames` +// and `frame_rate` are read, and they are read to derive the audio DURATION +// (`AudioLatentShape.from_video_pixel_shape`, types.py:184-200). +// +// The VIDEO guider is deliberately left at its default and is never consumed: +// upstream's T2A CLI constructs ONE `MultiModalGuiderParams` and it is the audio +// one (`:196-205`). Zeroing it here would look tidier and would be a fabricated +// value; leaving the params table's own entry says "this recipe does not read +// it" without inventing a number. +Ltx2PipelineRecipe T2aOneStageRecipe(const Ltx2PipelineParams& params, + const std::string& negative_prompt) { + Ltx2PipelineRecipe recipe = OneStageRecipe(params, negative_prompt); + recipe.audio_only = true; + // `video_output_phase` is already -1 on a fresh recipe; restated because on + // THIS recipe it is a statement rather than a default, and a later edit that + // gave the field a real value would otherwise silently ask for a video output + // from a pipeline that produces none. + recipe.video_output_phase = -1; + // `modality_scale=1.0` — the CLI pins it, and says why: "Audio-only generation + // has no video modality, so the video->audio (v2a) cross-modal guidance is + // meaningless here. 1.0 disables it" (t2a_one_stage.py:200-202). It is the ONE + // guider field T2A overrides against the params table's 3.0, and 1.0 is exactly + // the value `do_isolated_modality_generation` reads as OFF + // (guiders.py:283-285). Applied at the recipe rather than at the call site so + // no caller can reach the isolated-modality forward this port does not have. + recipe.phases[0].audio_guidance.modality_scale = 1.0; + return recipe; +} + // LTX_POSITIVE_ONLY_RECIPE (ltx2_recipes.py:116-124): every guidance knob at its // no-op value, and the official sigma schedule turned OFF. Ltx2PipelineRecipe PositiveOnlyRecipe() { @@ -1261,6 +1315,15 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, if (model_version == "2" || model_version == "2.3") return PositiveOnlyRecipe(); } else if (pipeline_kind == "retake") { if (model_version == "2" || model_version == "2.5") return RetakeRecipe(model_version); + } else if (pipeline_kind == "t2a_one_stage") { + if (model_version == "2") return T2aOneStageRecipe(Ltx2Params20(), kOmniNegativePrompt); + if (model_version == "2.3") return T2aOneStageRecipe(Ltx2Params23(), kOmniNegativePrompt); + if (model_version == "2.4") { + return T2aOneStageRecipe(Ltx2DetectPipelineParams("2.4"), LightricksNegativePrompt()); + } + if (model_version == "2.5") { + return T2aOneStageRecipe(Ltx2DetectPipelineParams("2.5"), LightricksNegativePrompt()); + } } Refuse("Unsupported LTX pipeline kind/version: '" + pipeline_kind + "'/'" + model_version + "'. Recipes are resolved from an EXACT (kind, version) table " diff --git a/src/vllm/model_executor/models/ltx2_t2a.cpp b/src/vllm/model_executor/models/ltx2_t2a.cpp new file mode 100644 index 000000000..aea6d9e62 --- /dev/null +++ b/src/vllm/model_executor/models/ltx2_t2a.cpp @@ -0,0 +1,417 @@ +// `T2AOneStagePipeline` (ltx-pipelines t2a_one_stage.py:43, `__call__` at :109) +// at Lightricks/LTX-2 @ fd4ded7f. See ltx2_t2a.h for the port map and for the +// three details that fail silently if guessed. +// +// Row LTX25-T2A-ONE-STAGE, issue #1005. + +#include "vllm/model_executor/models/ltx2_t2a.h" + +#include +#include +#include +#include +#include + +#include "vt/dtype.h" // VT_CHECK + +namespace vllm { +namespace { + +[[noreturn]] void Fail(const std::string& why) { throw std::runtime_error("ltx2 t2a: " + why); } + +uint64_t DigestF32(const std::vector& values) { + uint64_t h = 1469598103934665603ULL; + const auto* bytes = reinterpret_cast(values.data()); + const size_t n = values.size() * sizeof(float); + for (size_t i = 0; i < n; ++i) { + h ^= bytes[i]; + h *= 1099511628211ULL; + } + return h; +} + +double AbsMax(const std::vector& values) { + double m = 0.0; + for (const float v : values) m = std::max(m, std::abs(static_cast(v))); + return m; +} + +// `to_denoised` (ltx-core utils.py:39-52) as `X0Model.forward` applies it +// (model.py:590-604): the DiT emits a VELOCITY and everything downstream — the +// guider AND the sampler — wants the x0 prediction. It is therefore applied to +// EVERY PASS, on the way out of the forward, and not once to the guider's +// output: see ltx2_t2a.h item 4 (#1039). +// +// Identical arithmetic to the joint driver's own `ToDenoised`; kept here rather +// than shared because the joint one is a static in `ltx2_video.cpp`'s anonymous +// namespace and hoisting it would move lines above that file's gated READER +// ANCHORS list for no behavioural reason. +std::vector ToDenoised(const std::vector& sample, const std::vector& velocity, + const std::vector& timesteps, int64_t tokens, int64_t width) { + VT_CHECK(velocity.size() == sample.size(), "ltx2 t2a: the velocity is the wrong size"); + std::vector out(sample.size()); + for (int64_t t = 0; t < tokens; ++t) { + const float sigma = timesteps[static_cast(t)]; + for (int64_t c = 0; c < width; ++c) { + const size_t i = static_cast(t * width + c); + out[i] = sample[i] - sigma * velocity[i]; + } + } + return out; +} + +} // namespace + +std::vector Ltx2StgBlockMask(const std::vector& stg_blocks, int64_t num_layers) { + VT_CHECK(num_layers > 0, "ltx2 t2a: num_layers must be positive"); + std::vector mask(static_cast(num_layers), 0); + // `blocks is None` upstream means EVERY block (perturbations.py:19-33). An + // EMPTY list is not that: the CLI's `--audio-stg-blocks` with no values yields + // an empty list (utils/args.py:1107-1113, `nargs="*"`), and an empty list + // perturbs nothing. Conflating the two would turn "STG off" into "STG + // everywhere" for a caller who typed the flag and no numbers. + for (const int64_t b : stg_blocks) { + if (b < 0 || b >= num_layers) { + Fail("STG block index " + std::to_string(b) + " is outside [0, " + + std::to_string(num_layers) + + "). Upstream indexes `self.transformer_blocks` by it and raises; ignoring it here " + "would run an UNPERTURBED pass and report it as STG, which is a different guidance " + "delta on a render that still finishes"); + } + mask[static_cast(b)] = 1; + } + return mask; +} + +Ltx2T2aResult Ltx2T2aGenerate(const Ltx2T2aRequest& req) { + VT_CHECK(req.dit_params != nullptr && req.dit_weights != nullptr && req.audio_cfg != nullptr && + req.audio_weights != nullptr && req.vocoder_cfg != nullptr && + req.vocoder_weights != nullptr && req.noise != nullptr, + "ltx2 t2a: the request is missing a required borrowed pointer"); + const Ltx2DitParams& params = *req.dit_params; + + if (req.num_frames < 1) { + // `require_num_frames_source` (utils/blocks.py:894-905) — upstream's own + // fast refusal, raised at the TOP of `__call__` before prompt encoding, so + // an unsatisfiable auto-duration costs no work. + // + // WHAT IS *NOT* THE REASON: not the duration head's ARITHMETIC. + // `Ltx2DurationHeadForward` is ported and gated, including the audio-only + // case this pipeline would use — `test_ltx2_pipeline` runs it against + // `kLtx2DurAudioOnlyGolden`, generated from executed upstream. What is + // missing is a CONSTRUCTED head: nothing in this engine builds one, and + // `duration_head_path` is refused by name at load (#611). So the arithmetic + // exists and the object does not. + Fail("this request carries no frame count, and audio-only generation derives its DURATION " + "from one: `AudioLatentShape.from_video_pixel_shape` reads `frames` and `fps` off the " + "pixel shape (ltx-core types.py:184-200), which is why upstream passes a 512x512 " + "PLACEHOLDER resolution and a real frame count (t2a_one_stage.py:37-40, :163-166). " + "Auto duration needs a DurationHead this engine does not construct. Pass num_frames or " + "duration_seconds."); + } + if (req.frame_rate <= 0.0) Fail("frame_rate must be positive"); + if (req.context == nullptr || req.context_tokens < 1) { + Fail("no audio conditioning was supplied; `ctx_p.audio_encoding` is what this pipeline " + "cross-attends over (t2a_one_stage.py:134)"); + } + + // ── the audio latent shape (types.py:164-200) ───────────────────────────── + // + // `AudioLatentShape.from_video_pixel_shape` takes `frames` and `fps` from the + // pixel shape and NOTHING else — height and width are unused, which is exactly + // why upstream can pass a placeholder for them. + const Ltx2AudioPatchifierParams ap; + const double latents_per_second = static_cast(ap.sample_rate) / + static_cast(ap.hop_length) / + static_cast(ap.audio_latent_downsample_factor); + Ltx2AudioLatentShape ashape; + ashape.batch = 1; + ashape.channels = 8; // types.py:184-200 defaults, asserted against the DiT below + ashape.mel_bins = 16; + ashape.frames = static_cast( + std::llround(static_cast(req.num_frames) / req.frame_rate * latents_per_second)); + if (ashape.frames < 1) { + Fail("the audio latent resolved to zero frames for " + std::to_string(req.num_frames) + + " frames at " + std::to_string(req.frame_rate) + " fps"); + } + // The latent's channels x mel_bins IS the DiT's audio stream width, and a + // mismatch reinterprets the spectrogram rather than failing. Checked against + // BOTH factors and not only their product: a (16, 8) latent is the same width + // as an (8, 16) one and unpatchifies into a different tensor. + if (ashape.channels * ashape.mel_bins != params.audio_in_channels) { + Fail("the audio latent is " + std::to_string(ashape.channels) + " x " + + std::to_string(ashape.mel_bins) + " = " + + std::to_string(ashape.channels * ashape.mel_bins) + + " wide (types.py:184-200) but this DiT's audio stream takes " + + std::to_string(params.audio_in_channels)); + } + if (params.audio_cross_attention_dim < 1) Fail("this DiT declares no audio context width"); + + Ltx2T2aResult result; + result.latent_frames = ashape.frames; + + const int64_t width = ashape.channels * ashape.mel_bins; + const int64_t tokens = ashape.frames; // AudioPatchifier(patch_size=1) + result.audio_tokens = tokens; + + // ── the schedule (t2a_one_stage.py:141-143) ─────────────────────────────── + // + // `LTX2Scheduler()` is HARD-CODED at `:67`, so there is no scheduler-kind + // question here and no distilled sigma table: the recipe carries none and this + // computes them. + // + // THE TOKEN COUNT IS THE SCHEDULER'S OWN DEFAULT, NOT THE AUDIO LATENT'S, and + // this is the detail a re-derivation gets wrong. `execute` takes an OPTIONAL + // `latent` and falls back to `default_number_of_tokens = MAX_SHIFT_ANCHOR` + // when it is absent (schedulers.py:29, :32) — and `t2a_one_stage.py:141` calls + // `self._scheduler.execute(steps=num_inference_steps)` with no latent at all. + // The joint video driver in `ltx2_video.cpp` passes its own `target_tokens` + // and is right to, because the pipelines it mirrors pass a shape; copying that + // here would move `sigma_shift` by `(tokens - MAX_SHIFT_ANCHOR) * mm` + // (schedulers.py:36-38) and bend every sigma in the schedule. Nothing about + // the render's length, its channel count or its finiteness could see it. + // + // Passing 0 is `Ltx2SigmaSchedule`'s own spelling for "take the default" + // (ltx2_pipeline.h), so this is the fallback rather than a substitute for it. + const int64_t steps = req.steps; + if (steps < 1) Fail("num_inference_steps resolved to " + std::to_string(steps)); + const std::vector sigmas = Ltx2SigmaSchedule(steps, /*tokens=*/0); + const int64_t sigma_count = static_cast(sigmas.size()); + VT_CHECK(sigma_count >= 2, "ltx2 t2a: the schedule needs at least one step"); + + // ── the guider (t2a_one_stage.py:149-152) ───────────────────────────────── + const Ltx2MultiModalGuiderParams& g = req.guidance; + const bool want_uncond = g.DoUnconditionalGeneration(); + const bool want_perturbed = g.DoPerturbedGeneration(); + if (g.DoIsolatedModalityGeneration()) { + Fail("isolated-modality guidance (`modality_scale` = " + std::to_string(g.modality_scale) + + ") asks for a fourth forward over the OTHER modality, and this pipeline has no other " + "modality to run it over. Upstream pins `modality_scale` to 1.0 for exactly this " + "reason and says so: \"Audio-only generation has no video modality, so the " + "video->audio (v2a) cross-modal guidance is meaningless here. 1.0 disables it\" " + "(t2a_one_stage.py:200-202). Use 1.0."); + } + if (want_uncond && req.negative_context == nullptr) { + Fail("the guider asks for an unconditional pass (`cfg_scale` = " + + std::to_string(g.cfg_scale) + + ") and no negative conditioning was supplied. Upstream's guider carries a " + "`negative_context` and the CLI always fills it (t2a_one_stage.py:151, :193). " + "Substituting a zero tensor would make the CFG delta `cfg_scale * cond`, which is a " + "different render and not a missing one. Supply a negative prompt, or set the scale " + "to 1.0."); + } + std::vector stg_mask; + if (want_perturbed) { + stg_mask = Ltx2StgBlockMask(g.stg_blocks, params.num_layers); + for (int64_t b = 0; b < params.num_layers; ++b) { + if (stg_mask[static_cast(b)] != 0) result.perturbed_blocks.push_back(b); + } + if (result.perturbed_blocks.empty()) { + Fail("the guider asks for a perturbed pass (`stg_scale` = " + std::to_string(g.stg_scale) + + ") and `stg_blocks` names no block, so the perturbed forward would be identical to " + "the conditional one and the STG delta would be exactly zero — a full extra forward " + "per step that changes nothing. Name the blocks, or set the scale to 0."); + } + } + Ltx2DitPerturbation perturbation; + perturbation.audio_self_attn = stg_mask; + + // ── the state (helpers.py:428-447; ModalitySpec(context=...) alone at :168) ─ + // + // No initial latent and no freeze: T2A's audio `ModalitySpec` carries a + // context and nothing else (t2a_one_stage.py:168), so the denoise mask is all + // ones and there is no `clean` to blend back. + // + // THE NOISE IS UNIT VARIANCE, NOT SCALED BY `sigmas[0]`, and this draft scaled + // it until the chain was read. `ModalitySpec.noise_scale` defaults to 1.0 + // (utils/types.py:110), `create_noised_state` forwards it (helpers.py:434, + // :443) and `GaussianNoiser.__call__` is `torch.lerp(latent, noise, + // noise_scale)` (noisers.py:31) — at 1.0 the state IS the noise. A reader who + // knows other flow-matching samplers will expect the scaling, so the absence + // is written down. + // + // AND THE TWO FORMS AGREE HERE, WHICH IS WHY NO TEST SEPARATES THEM. + // MEASURED: a mutation adding `for (float& v : latent) v *= sigmas[0];` left + // the focused gate at 6 cases / 484 assertions / exit 0. That is not a blind + // instrument — it is an identity. `LTX2Scheduler` starts at `linspace(1, 0, + // steps + 1)[0] == 1`; the shift map sends 1 to `exp(s)/(exp(s) + (1/1 - 1))` + // which is exactly 1 (schedulers.py:41-45); and the stretch sends it to + // `1 - (1 - 1)/scale_factor`, again exactly 1 (`:47-55`). So `sigmas[0]` is + // 1.0 for EVERY step count, and the multiply is a no-op. + // + // That identity is GATED rather than left as this comment's word, in + // `test_ltx2_video`'s "the schedule starts at exactly 1.0". If upstream ever + // moves the first sigma off 1, that gate fires and this line becomes a real + // difference — which is the point of pinning it rather than pinning the + // mutation's survival. + std::vector latent = req.noise->Draw(tokens * width); + const std::vector positions_f = Ltx2AudioPatchTimings(ashape, ap); + const std::vector positions(positions_f.begin(), positions_f.end()); + + // ── the denoise loop (samplers.py:39-79) ────────────────────────────────── + // + // `last_denoised_audio` (utils/denoisers.py:85-91): a step the guider SKIPS + // reuses the previous step's denoised prediction instead of running a forward. + std::vector last_denoised; + for (int64_t step = 0; step + 1 < sigma_count; ++step) { + const float sigma = sigmas[static_cast(step)]; + // Every token carries the schedule's own sigma: the mask is all ones, so + // `timesteps_from_mask` is a constant fill (helpers.py:466-503). + const std::vector timesteps(static_cast(tokens), sigma); + + Ltx2ModalityInput ain; + ain.batch = 1; + ain.tokens = tokens; + ain.context_tokens = req.context_tokens; + ain.latent = latent.data(); + ain.timesteps = timesteps.data(); + ain.sigma = σ + ain.positions = positions.data(); + ain.context = req.context; + + // `should_skip_step` (guiders.py:287-291). `skip_step` defaults to 0, which + // never skips, so this is reachable only from an explicit request. + // + // A SKIPPED STEP RUNS NO FORWARD AT ALL AND REUSES THE PREVIOUS STEP'S + // DENOISED PREDICTION. This draft ran the CONDITIONAL forward and used it, + // which is a plausible reading of "skip the guidance" and is not what + // upstream does: `_guided_denoise` returns + // `DenoisedLatentResult.result_or_none(denoised=last_denoised_audio)` when + // every guider skips (`utils/denoisers.py:85-91`), before it assembles a + // single pass. The difference is a whole DiT forward per skipped step and a + // different trajectory, on a render that finishes either way. + // + // `step == 0` can never skip — `0 % (skip_step + 1)` is 0 — so + // `last_denoised` is always populated by the time this branch is taken. The + // guard is kept anyway, because "the arithmetic makes it impossible" is + // exactly the reasoning that a later change to `ShouldSkipStep` would + // silently invalidate, and the failure would be a read of an empty vector. + const bool skip = g.ShouldSkipStep(step); + if (skip && last_denoised.empty()) { + Fail("step " + std::to_string(step) + + " is a skipped step and no earlier step produced a denoised prediction to reuse. " + "`should_skip_step` is `step % (skip_step + 1) != 0` (guiders.py:287-291), which is " + "false at step 0, so this is unreachable through the request surface and is a defect " + "rather than a bad request"); + } + if (skip) { + latent = Ltx2EulerStep(latent.data(), last_denoised.data(), sigmas.data(), sigma_count, step, + static_cast(latent.size())); + continue; + } + + // THE VIDEO STREAM IS `nullptr`, NOT A DISABLED ONE. See ltx2_t2a.h item 1: + // upstream's `run_v2a` tests PRESENCE (transformer.py:269), so a + // present-but-disabled stream still feeds video->audio cross attention. + // + // AND EVERY PASS IS CONVERTED TO X0 *HERE*, BEFORE THE GUIDER SEES IT. This + // lambda is `X0Model` (model.py:590-604): upstream never hands the denoiser + // the raw velocity model, it hands `X0Model(builder.build(...))` + // (utils/blocks.py:480-482), so `_guided_denoise`'s + // `all_v, all_a = transformer(...)` at utils/denoisers.py:188 already + // carries DENOISED tensors and `audio_guider.calculate(...)` at `:203` + // combines those. See ltx2_t2a.h item 4 for why converting once after the + // guider instead is a different function on the DEFAULT arm (#1039). + // + // EVERY FORWARD GOES THROUGH THIS ONE LAMBDA, and that is what makes + // `video_stream_present` an OBSERVATION rather than a restatement. Written + // as `result.video_stream_present = false` beside a `nullptr` literal it + // would be a comment that compiles: a build that started passing a stream + // would report `false` and stay green. Derived at the call, a mutation that + // hands any forward a video stream flips it. + const auto x0_model = [&](const Ltx2ModalityInput* video, const Ltx2ModalityInput* audio, + const Ltx2DitPerturbation* p, + std::vector* velocity_out = nullptr) { + if (video != nullptr) result.video_stream_present = true; + const Ltx2DitOutputs out = Ltx2DitForward(req.device, params, *req.dit_weights, video, audio, + req.compute_dtype, /*cache=*/nullptr, p); + // The RAW velocity, before the conversion, recorded only where a caller + // asked for it. It is the other half of the pair that makes "which space + // did the guider combine" an arithmetic question — see the header. + if (velocity_out != nullptr) *velocity_out = out.audio; + // `to_denoised(audio.latent, ax, audio.timesteps)` (model.py:603). + return ToDenoised(latent, out.audio, timesteps, tokens, width); + }; + + const std::vector cond = x0_model( + /*video=*/nullptr, &ain, /*p=*/nullptr, + step == 0 ? &result.first_step_velocity : nullptr); + ++result.cond_forwards; + + std::vector denoised = cond; + if (want_uncond || want_perturbed) { + std::vector uncond_text; + std::vector uncond_perturbed; + if (want_uncond) { + Ltx2ModalityInput nin = ain; + nin.context = req.negative_context; + // The velocity of THIS arm, recorded beside its x0 exactly as the + // conditional pass's is. Recorded per arm rather than once, because the + // conversion is per pass and a claim made about "every pass" from one + // recorded pass is a claim about a third of them (see the header). + uncond_text = x0_model(/*video=*/nullptr, &nin, /*p=*/nullptr, + step == 0 ? &result.first_step_uncond_velocity : nullptr); + if (step == 0) result.first_step_uncond = uncond_text; + ++result.uncond_forwards; + } + if (want_perturbed) { + // The POSITIVE context with the self-attention perturbed — upstream + // perturbs the MODEL, never the conditioning (guiders.py:244-273 takes + // `uncond_perturbed` from a forward whose `perturbations` differ and + // whose context does not). + uncond_perturbed = x0_model(/*video=*/nullptr, &ain, &perturbation, + step == 0 ? &result.first_step_perturbed_velocity : nullptr); + if (step == 0) result.first_step_perturbed = uncond_perturbed; + ++result.perturbed_forwards; + } + denoised = Ltx2MultiModalGuidance(g, cond.data(), + want_uncond ? uncond_text.data() : nullptr, + want_perturbed ? uncond_perturbed.data() : nullptr, + /*uncond_modality=*/nullptr, + static_cast(cond.size())); + } + + // Kept for the next step's `should_skip_step` branch, which reuses it rather + // than recomputing (utils/denoisers.py:85-91). + if (step == 0) { + result.first_step_latent = latent; + result.first_step_cond = cond; + result.first_step_denoised = denoised; + result.first_step_sigma = static_cast(sigma); + } + last_denoised = std::move(denoised); + // `EulerDiffusionStep()` — `DiffusionStage.__call__`'s own default + // (utils/blocks.py:524-527), which T2A does not override (it passes no + // `stepper`, t2a_one_stage.py:154-170). The ancestral sampler that + // `distilled.py` selects for generation 2.5 reaches this pipeline through + // nothing. + latent = Ltx2EulerStep(latent.data(), last_denoised.data(), sigmas.data(), sigma_count, step, + static_cast(latent.size())); + // What the sampler WROTE, recorded after the step rather than derived from + // what was recorded before it. It is the only observable that says which + // tensor `Ltx2EulerStep` was actually handed: a second `ToDenoised` applied + // to `denoised` on the way in leaves every other field here untouched. + if (step == 0) result.first_step_next_latent = latent; + } + + result.latent_digest = DigestF32(latent); + result.latent_absmax = AbsMax(latent); + + // `clear_conditioning` + `unpatchify` (blocks.py:575-580). There is no + // conditioning item on this path and nothing appended, so the clear is an + // identity here — stated rather than called, because calling a no-op would + // suggest an append this pipeline cannot make. + const std::vector volume = Ltx2AudioUnpatchify(latent.data(), ashape); + + // ── the decode (t2a_one_stage.py:172) ───────────────────────────────────── + const Ltx2AudioSpectrogram mel = Ltx2AudioDecoderForward( + *req.audio_cfg, *req.audio_weights, volume, ashape.channels, ashape.frames, ashape.mel_bins); + result.waveform = Ltx2VocoderWithBweForward(*req.vocoder_cfg, *req.vocoder_weights, mel.data, + mel.channels, mel.frames, mel.mel_bins, + &result.samples_per_channel); + result.channels = mel.channels; + result.sample_rate = req.vocoder_cfg->output_sampling_rate; + return result; +} + +} // namespace vllm diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index ab4b8a592..bf59f5f35 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -36,6 +36,7 @@ #include "vllm/model_executor/models/ltx2_loader.h" #include "vllm/model_executor/models/ltx2_pipeline.h" #include "vllm/model_executor/models/ltx2_retake.h" +#include "vllm/model_executor/models/ltx2_t2a.h" #include "vllm/model_executor/models/ltx2_text_encoder.h" #include "vllm/model_executor/models/ltx2_upsampler.h" #include "vllm/model_executor/models/ltx2_tiling.h" @@ -363,7 +364,7 @@ constexpr char kLtx2DurationHeadPathExtra[] = "duration_head_path"; // they are no longer trusted: the list below is derived from this file on every // run and compared, and the failure prints the replacement to paste in. // READER ANCHORS (derived and gated by test_ltx2_video): -// 781 791 792 854 950 966 968 1046 1071 1176 1217 +// 782 792 793 855 951 967 969 1060 1085 1190 1231 const char* const kKnownLoadExtras[] = { kLtx2AudioPromptEmbedsExtra, kLtx2PipelineKindExtra, kLtx2ModelVersionExtra, kLtx2AllowUnportedExtra, kLtx2MaxPhaseExtra, kLtx2DitConfigPathExtra, @@ -974,8 +975,21 @@ std::unique_ptr Ltx2VideoEngine::Load(const VideoModelParams& p } // ── the video VAE ───────────────────────────────────────────────────────── - if (params.video_vae_path.empty()) Fail("video_vae_path is required"); - { + // + // REQUIRED, EXCEPT ON AN AUDIO-ONLY RECIPE, and the exception is upstream's + // shape rather than a convenience: `T2AOneStagePipeline.__init__` constructs a + // `PromptEncoder`, a `DiffusionStage`, an `AudioDecoder` and a + // `DurationPredictor` (t2a_one_stage.py:68-107) and never calls + // `model_paths.video_vae()`. Demanding one would make a text-to-audio load ask + // for a checkpoint the pipeline cannot use. + // + // Keyed on `recipe.audio_only` rather than on the kind STRING, so the next + // audio-only recipe inherits it instead of silently failing here. Supplying a + // video VAE anyway is accepted and loaded — it costs the caller memory and + // nothing else, and refusing it would break a caller who reuses one params + // object across pipelines. + if (params.video_vae_path.empty() && !im.recipe.audio_only) Fail("video_vae_path is required"); + if (!params.video_vae_path.empty()) { const SafetensorsFile f = SafetensorsFile::Open(params.video_vae_path); const nlohmann::json vae_config = Ltx2ReadCheckpointConfig(f); im.video_cfg = Ltx2ParseConvVideoDecoderConfig(vae_config, &im.video_kind); @@ -1018,7 +1032,7 @@ std::unique_ptr Ltx2VideoEngine::Load(const VideoModelParams& p } } } - if (im.video_cfg.in_channels != im.dit.params.out_channels) { + if (!params.video_vae_path.empty() && im.video_cfg.in_channels != im.dit.params.out_channels) { Fail("the video VAE takes " + std::to_string(im.video_cfg.in_channels) + " latent channels but the DiT emits " + std::to_string(im.dit.params.out_channels)); } @@ -1438,7 +1452,13 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { kv.first == kLtx2RetakeEndTimeExtra || kv.first == kLtx2RetakeFrameRateExtra || kv.first == kLtx2RegenerateVideoExtra || - kv.first == kLtx2RegenerateAudioExtra; + kv.first == kLtx2RegenerateAudioExtra || + kv.first == kLtx2NegativePromptExtra || + kv.first == kLtx2AudioCfgScaleExtra || + kv.first == kLtx2AudioStgScaleExtra || + kv.first == kLtx2AudioRescaleScaleExtra || + kv.first == kLtx2AudioSkipStepExtra || + kv.first == kLtx2AudioStgBlocksExtra; if (!known) { Fail("unknown per-generation extra '" + kv.first + "'. This family defines: " + std::string(kLtx2ImageCrfExtra) + ", " + kLtx2AudioPathExtra + ", " + @@ -1446,7 +1466,52 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { kLtx2GeneratedKeyframesExtra + ", " + kLtx2TemporalRoundsExtra + ", " + kLtx2RetakeStartTimeExtra + ", " + kLtx2RetakeEndTimeExtra + ", " + kLtx2RetakeFrameRateExtra + ", " + kLtx2RegenerateVideoExtra + ", " + - kLtx2RegenerateAudioExtra); + kLtx2RegenerateAudioExtra + ", " + kLtx2NegativePromptExtra + ", " + + kLtx2AudioCfgScaleExtra + ", " + kLtx2AudioStgScaleExtra + ", " + + kLtx2AudioRescaleScaleExtra + ", " + kLtx2AudioSkipStepExtra + ", " + + kLtx2AudioStgBlocksExtra); + } + } + // ── the TEXT-TO-AUDIO knobs belong to ONE pipeline (#1005) ──────────────── + // + // `pipeline_kind` is a LOAD extra, so which pipeline runs is settled before a + // request arrives and this is a decidable question rather than a guess. The + // guard runs in BOTH directions: the six T2A knobs are refused off a + // `t2a_one_stage` engine, and every other per-generation knob is refused ON + // one. Neither is padding. Upstream's `T2AOneStagePipeline.__call__` takes no + // image, no reference, no keyframe and no window (t2a_one_stage.py:109-122), + // and its guider arguments have no counterpart in any other `__call__`, so a + // knob crossing either way would silently do nothing to a render that still + // finishes. + { + const char* const kT2aOnly[] = {kLtx2NegativePromptExtra, kLtx2AudioCfgScaleExtra, + kLtx2AudioStgScaleExtra, kLtx2AudioRescaleScaleExtra, + kLtx2AudioSkipStepExtra, kLtx2AudioStgBlocksExtra}; + const char* const kNotOnT2a[] = {kLtx2ImageCrfExtra, kLtx2AudioPathExtra, + kLtx2AudioStartTimeExtra, kLtx2AudioMaxDurationExtra, + kLtx2GeneratedKeyframesExtra, kLtx2TemporalRoundsExtra, + kLtx2RetakeStartTimeExtra, kLtx2RetakeEndTimeExtra, + kLtx2RetakeFrameRateExtra, kLtx2RegenerateVideoExtra, + kLtx2RegenerateAudioExtra}; + for (const char* key : kT2aOnly) { + if (!im.recipe.audio_only && !VideoExtra(gen.extras, key).empty()) { + Fail("the '" + std::string(key) + + "' extra is text-to-audio's alone (ltx-pipelines utils/args.py:1083-1119, the " + "`default_1_stage_t2a_arg_parser`), and this engine was loaded with pipeline_kind '" + + im.pipeline_kind + + "'. Refused rather than ignored: no other pipeline `__call__` upstream takes a " + "guider argument at all, so accepting it here would report a configured render " + "that ran the recipe's own values"); + } + } + for (const char* key : kNotOnT2a) { + if (im.recipe.audio_only && !VideoExtra(gen.extras, key).empty()) { + Fail("the '" + std::string(key) + + "' extra has no meaning on a text-to-audio render, which produces no picture at " + "all. `T2AOneStagePipeline.__call__` takes a prompt, a negative prompt, a seed, a " + "frame rate, a step count, the audio guider and a frame count " + "(t2a_one_stage.py:109-122) and nothing else. Refused rather than ignored"); + } } } // The two audio WINDOW knobs only mean something alongside a file. Accepting @@ -1752,6 +1817,23 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { im.trace.video_absmax = AbsMax(v); im.trace.audio_absmax = AbsMax(a); } + + // ── TEXT-TO-AUDIO: the render that has no picture (#1005) ───────────────── + // + // `T2AOneStagePipeline.__call__` (t2a_one_stage.py:109-172), in its own + // translation unit (ltx2_t2a.h) mirroring upstream's own file. + // + // THE BRANCH SITS HERE, after the conditioning and before ANY video geometry. + // After, because T2A encodes a prompt exactly as every other pipeline does + // (`:127-135`) and duplicating that chain would give the audio-only arm its own + // copy of the connector composition. Before, because everything below this + // point — the resolution guard, the canvas, the phase loop, the decode — is + // about a video stream this pipeline does not have, and a `t2a` request that + // fell through would be refused by a message about latent grids. + if (im.recipe.audio_only) { + return GenerateAudioOnly(im, gen, audio_context, context_tokens); + } + // ── conditioning on pixels (row LTX25-IMAGE-COND, issue #644) ───────────── // // Upstream this is `ImageConditioner` (ltx-pipelines/utils/blocks.py:936-993, @@ -3384,6 +3466,243 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { return result; } +// ── TEXT-TO-AUDIO (row LTX25-T2A-ONE-STAGE, #1005) ────────────────────────── +// +// `T2AOneStagePipeline.__call__` (ltx-pipelines t2a_one_stage.py:109-172). The +// numerics live in `ltx2_t2a.cpp`, mirroring upstream's own file; this resolves +// the request, encodes the negative prompt and writes the artifact. +// +// PLACED BELOW `Generate` ON PURPOSE. The READER ANCHORS comment near the top of +// this file carries derived LINE NUMBERS into it and is gated by +// `test_ltx2_video`, so a definition inserted above the last anchored line would +// move every anchor under it for a reason that has nothing to do with this row. +VideoResult Ltx2VideoEngine::GenerateAudioOnly(Impl& im, const VideoGenParams& gen, + const float* audio_context, + int64_t context_tokens) { + const Ltx2PipelineRecipe& recipe = im.recipe; + VT_CHECK(recipe.audio_only, "ltx2 t2a: reached the audio-only path on a video recipe"); + + // Upstream's T2A CLI has no --height/--width, and its pipeline substitutes a + // 512x512 PLACEHOLDER whose height and width it documents as unused + // (t2a_one_stage.py:37-40, :163-164). Accepting a resolution here would take a + // number from the caller, ignore it, and return successfully. + if (gen.height > 0 || gen.width > 0) { + Fail("a text-to-audio request cannot carry a width or a height: there is no picture. " + "Upstream passes a 512x512 PLACEHOLDER into the stage and says so in as many words — " + "\"Audio-only generation reads `frames` and `fps` from the pixel shape via " + "`AudioLatentShape.from_video_pixel_shape` (height/width are unused)\" " + "(t2a_one_stage.py:37-40). Accepting one would ignore it and still succeed"); + } + if (!gen.first_frame_path.empty() || !gen.last_frame_path.empty() || + !gen.first_frame_ppm.empty() || !gen.ref_image_paths.empty() || + !gen.ref_video_dir.empty() || !gen.ref_audio_path.empty() || !gen.ref_audio_wav.empty()) { + Fail("a text-to-audio request cannot carry a keyframe, a reference image, a reference clip " + "or a reference waveform. `T2AOneStagePipeline.__call__` takes none of them " + "(t2a_one_stage.py:109-122) and its `DiffusionStage` call passes `video=None` " + "(`:167`), so there is no stream for any of them to condition"); + } + + // `num_frames` / `frame_rate` — the only two fields of the placeholder pixel + // shape T2A reads, and they exist to derive the audio DURATION. + const double fps = recipe.frame_rate; + int64_t frames = gen.num_frames > 1 ? gen.num_frames : recipe.num_frames; + if (gen.duration_seconds > 0.0) { + frames = static_cast(std::llround(gen.duration_seconds * fps)); + } + + // ── the guider (t2a_one_stage.py:196-205) ───────────────────────────────── + // + // The recipe already carries the params table's audio guider with + // `modality_scale` pinned to 1.0; each extra overrides ONE field, exactly as + // one CLI flag does. + Ltx2MultiModalGuiderParams guidance = recipe.phases.front().audio_guidance; + guidance.cfg_scale = ExtraDouble(gen.extras, kLtx2AudioCfgScaleExtra, guidance.cfg_scale); + guidance.stg_scale = ExtraDouble(gen.extras, kLtx2AudioStgScaleExtra, guidance.stg_scale); + guidance.rescale_scale = + ExtraDouble(gen.extras, kLtx2AudioRescaleScaleExtra, guidance.rescale_scale); + guidance.skip_step = ExtraInt(gen.extras, kLtx2AudioSkipStepExtra, guidance.skip_step); + if (guidance.skip_step < 0) { + Fail("'" + std::string(kLtx2AudioSkipStepExtra) + "' is " + + std::to_string(guidance.skip_step) + + "; `should_skip_step` is `step % (skip_step + 1)` (guiders.py:287-291) and a negative " + "value would take the modulus of a non-positive divisor"); + } + { + // `--audio-stg-blocks`, `nargs="*"` (utils/args.py:1107-1113). An extra that + // is PRESENT and empty is upstream's empty list — "perturb nothing" — and is + // kept distinct from an ABSENT extra, which takes the params table's own + // [28]. Collapsing the two would make `audio_stg_blocks=` silently mean + // block 28. + const auto at = gen.extras.find(kLtx2AudioStgBlocksExtra); + if (at != gen.extras.end()) { + guidance.stg_blocks.clear(); + const std::string& raw = at->second; + for (size_t i = 0; i < raw.size();) { + const size_t comma = raw.find(',', i); + const std::string token = raw.substr(i, comma == std::string::npos ? comma : comma - i); + if (!token.empty()) { + try { + guidance.stg_blocks.push_back(std::stoll(token)); + } catch (const std::exception&) { + Fail("'" + std::string(kLtx2AudioStgBlocksExtra) + "' holds '" + token + + "', which is not an integer block index"); + } + } + if (comma == std::string::npos) break; + i = comma + 1; + } + } + } + + // ── the negative conditioning (t2a_one_stage.py:127-135) ────────────────── + // + // Upstream encodes `[prompt, negative_prompt]` in ONE `PromptEncoder` call and + // takes `.audio_encoding` from each. Here the positive half was already + // resolved by `Generate`; this is the second half, through the same + // `Ltx2EncodePromptToConditioning` and the same connector. + // + // ONLY WHEN THE GUIDER ASKS FOR IT. `do_unconditional_generation` is + // `not isclose(cfg_scale, 1.0)` (guiders.py:275-277), so at scale 1.0 there is + // no unconditional forward and encoding a negative prompt would be a wasted + // 12B host-side pass per request. + std::vector negative_audio; + const float* negative_context = nullptr; + if (!guidance.DoUnconditionalGeneration()) { + // Nothing to do: the guidance delta's `uncond_text` term is switched off. + } else if (!im.has_encoder) { + Fail("this text-to-audio request needs an unconditional forward (`cfg_scale` = " + + std::to_string(guidance.cfg_scale) + + "), which needs the NEGATIVE prompt encoded — and no text tower is loaded, so this " + "engine can encode neither prompt. The `" + + std::string(kLtx2AudioPromptEmbedsExtra) + + "' fallback carries ONE conditioning stream and there is no second file for the " + "negative one. Load with encoder_path, or set '" + + std::string(kLtx2AudioCfgScaleExtra) + + "' to 1.0, which turns the unconditional pass off (guiders.py:275-277)"); + } else { + const std::string negative = + VideoExtra(gen.extras, kLtx2NegativePromptExtra, recipe.negative_prompt); + if (negative.empty()) { + Fail("this text-to-audio request needs a negative prompt and neither the '" + + std::string(kLtx2NegativePromptExtra) + + "' extra nor the recipe carries one. An EMPTY negative prompt is not the same as no " + "CFG: it still encodes and still steers, and upstream's CLI always supplies " + "`DEFAULT_NEGATIVE_PROMPT` (utils/args.py:1083-1088)"); + } + vt::Queue text_queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + const Ltx2PromptConditioning encoded = Ltx2EncodePromptToConditioning( + *im.tower, *im.tokenizer, im.gemma_ids, im.caption_projections, im.feature_cfg, negative, + text_queue); + negative_audio = encoded.conditioning.audio; + if (encoded.seq != context_tokens) { + // Upstream's two encodings come from ONE tokenization of a two-element + // list (t2a_one_stage.py:127-133), so they share a padded width by + // construction. A mismatch here means the two ran different geometries and + // the guidance delta would subtract tensors that do not correspond. + Fail("the negative prompt encoded to " + std::to_string(encoded.seq) + + " context rows and the prompt to " + std::to_string(context_tokens) + + "; upstream encodes both in one call and they cannot differ"); + } + if (im.has_connector) { + const Ltx2ConnectorEmbeddings through = + RunConnector(SafetensorsFile::Open(im.params.dit_path), im.video_connector_cfg, + im.audio_connector_cfg, encoded.conditioning.video, + encoded.conditioning.audio, encoded.conditioning.additive_mask, + context_tokens); + negative_audio = through.audio; + } + negative_context = negative_audio.data(); + } + + // ── the render ──────────────────────────────────────────────────────────── + if (im.on_device) { + // REFUSED BY NAME rather than served the host forward behind a device + // handle, which is the substitution this engine's header names as the thing + // that would make every later timing claim false. + // + // WHAT IS *NOT* THE REASON: not the STG perturbation and not the guider. + // `Ltx2DitPerturbation` is a plain argument either forward could take, and + // `Ltx2MultiModalGuidance` runs on host buffers on both arms. What is + // missing is narrower and is a fact about THIS tree: + // `Ltx2DitForwardDevice` dereferences `*video` unconditionally — both + // `PrepareStreamDev` calls take it by reference, and the per-block + // `a.batch = video->batch` reads through it — so a one-stream device forward + // is a rewrite of that function rather than the lifted check the host + // forward needed. Owed by #1005. + Fail("text-to-audio is not served on the accelerator. `Ltx2DitForwardDevice` takes BOTH " + "streams by reference and this pipeline has no video stream to give it " + "(`video=None`, t2a_one_stage.py:167). Refusing rather than running the host forward " + "behind a device handle. Load with device 0."); + } + + EngineNoiseStream noise(gen.has_seed ? gen.seed : static_cast(recipe.num_frames)); + Ltx2T2aRequest req; + req.device = im.device; + req.compute_dtype = im.compute_dtype; + req.dit_params = &im.dit.params; + req.dit_weights = &im.dit.weights; + req.context = audio_context; + req.negative_context = negative_context; + req.context_tokens = context_tokens; + req.num_frames = frames; + req.frame_rate = fps; + req.steps = gen.steps > 0 ? gen.steps : recipe.num_inference_steps; + req.noise = &noise; + req.guidance = guidance; + req.audio_cfg = &im.audio_cfg; + req.audio_weights = &im.audio_weights; + req.vocoder_cfg = &im.vocoder_cfg; + req.vocoder_weights = &im.vocoder_weights; + + const Ltx2T2aResult rendered = Ltx2T2aGenerate(req); + + im.trace.t2a_rendered = true; + im.trace.t2a_video_stream_present = rendered.video_stream_present; + im.trace.t2a_cond_forwards = rendered.cond_forwards; + im.trace.t2a_uncond_forwards = rendered.uncond_forwards; + im.trace.t2a_perturbed_forwards = rendered.perturbed_forwards; + im.trace.t2a_perturbed_blocks = rendered.perturbed_blocks; + im.trace.t2a_first_latent = rendered.first_step_latent; + im.trace.t2a_first_velocity = rendered.first_step_velocity; + im.trace.t2a_first_cond = rendered.first_step_cond; + im.trace.t2a_first_uncond_velocity = rendered.first_step_uncond_velocity; + im.trace.t2a_first_uncond = rendered.first_step_uncond; + im.trace.t2a_first_perturbed_velocity = rendered.first_step_perturbed_velocity; + im.trace.t2a_first_perturbed = rendered.first_step_perturbed; + im.trace.t2a_first_denoised = rendered.first_step_denoised; + im.trace.t2a_first_next_latent = rendered.first_step_next_latent; + im.trace.t2a_first_sigma = rendered.first_step_sigma; + im.trace.audio_tokens = rendered.audio_tokens; + im.trace.audio_latent_digest = rendered.latent_digest; + im.trace.audio_latent_absmax = rendered.latent_absmax; + + // ── the artifact ────────────────────────────────────────────────────────── + std::error_code ec; + std::filesystem::create_directories(gen.output_dir, ec); + if (ec) Fail("cannot create " + gen.output_dir + ": " + ec.message()); + + VideoResult result; + // `frame_dir` STAYS EMPTY, and that is the contract rather than an omission: a + // directory naming a frame pattern that matches no file is what a caller + // iterates and finds nothing in. `frame_count == 0` says the same thing, and + // saying it twice is what lets a consumer notice the disagreement if one of + // them is ever filled by accident. + result.frame_count = 0; + result.width = 0; + result.height = 0; + result.fps = 0; + result.audio_path = JoinPath(gen.output_dir, "audio.wav"); + WriteFileBytes(result.audio_path, + MiniMaxH3WriteWav(rendered.waveform, rendered.channels, + rendered.samples_per_channel, rendered.sample_rate)); + result.sample_rate = rendered.sample_rate; + // NO `mux_argv`, and none is composed. The argv this seam builds muxes a frame + // pattern with a soundtrack; over an empty pattern ffmpeg fails, so composing + // one would hand the caller a command that cannot run and call it a result. + im.trace.completed = true; + return result; +} + namespace { std::unique_ptr LoadLtx2VideoFamily(const VideoModelParams& params) { diff --git a/tests/vllm/models/test_ltx2.cpp b/tests/vllm/models/test_ltx2.cpp index 5dd0fc1a9..009c5062f 100644 --- a/tests/vllm/models/test_ltx2.cpp +++ b/tests/vllm/models/test_ltx2.cpp @@ -1325,13 +1325,67 @@ TEST_CASE("ltx2 keyframes: a bf16 embedding view is REFUSED by the f32 forward") CHECK_THROWS(Ltx2DitForward(Cpu(), p, set.weights, &m.video, &m.audio, vt::DType::kF32)); } -TEST_CASE("ltx2 forward: a single-stream model type is REFUSED") { +// REWRITTEN by row LTX25-T2A-ONE-STAGE (#1005), and the rewrite is the point +// rather than an accommodation. +// +// This case used to assert that a single-stream call THROWS, pinning a refusal +// whose stated reason was "LTXModelType.VideoOnly and LTXModelType.AudioOnly +// carry a different weight contract". That reason is about the CHECKPOINT, and +// it does not describe `T2AOneStagePipeline`: upstream loads the ordinary +// AudioVideo file and restricts which keys it reads +// (LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP, model_configurator.py:228-239), +// then calls `LTXModel.forward(video=None, ...)` (t2a_one_stage.py:167). +// +// So the assertion is not widened, it is REPLACED with the contract upstream +// actually has (transformer.py:259-260, "At least one of video or audio must be +// provided") — and the new form is strictly stronger, because it also pins what +// a one-stream call RETURNS. The old one could not tell a served one-stream +// forward from a broken one; both threw. +TEST_CASE("ltx2 forward: ONE stream runs, and both-null is refused") { const Ltx2DitParams p = ReducedParams(Ltx2RopeType::kSplit, false); WeightSet set = BuildWeights(p); Modalities m; BuildModalities(&m, false); - CHECK_THROWS(Ltx2DitForward(Cpu(), p, set.weights, &m.video, nullptr, vt::DType::kF32)); - CHECK_THROWS(Ltx2DitForward(Cpu(), p, set.weights, nullptr, &m.audio, vt::DType::kF32)); + + // Upstream's own refusal, and the only one left. + CHECK_THROWS(Ltx2DitForward(Cpu(), p, set.weights, nullptr, nullptr, vt::DType::kF32)); + + // VIDEO ALONE. `run_a2v` is false because there is no audio state, so the + // video output must still be the full sequence and the audio one EMPTY — + // `Ltx2DitOutputs` carries two vectors and a build that filled both would be + // reporting a stream it never ran. + const vllm::Ltx2DitOutputs v_only = + Ltx2DitForward(Cpu(), p, set.weights, &m.video, nullptr, vt::DType::kF32); + CHECK(v_only.audio.empty()); + REQUIRE(v_only.video.size() == + static_cast(m.video.batch * m.video.tokens * p.out_channels)); + for (const float x : v_only.video) REQUIRE(std::isfinite(x)); + + // AUDIO ALONE — the shape `T2AOneStagePipeline` runs. + const vllm::Ltx2DitOutputs a_only = + Ltx2DitForward(Cpu(), p, set.weights, nullptr, &m.audio, vt::DType::kF32); + CHECK(a_only.video.empty()); + REQUIRE(a_only.audio.size() == + static_cast(m.audio.batch * m.audio.tokens * p.audio_out_channels)); + for (const float x : a_only.audio) REQUIRE(std::isfinite(x)); + + // AND ONE STREAM IS NOT THE JOINT FORWARD WITH THE OTHER IGNORED. This is the + // assertion the old case had no way to make, and it is the whole reason + // `video = nullptr` is not `video->enabled = false`: upstream's `run_v2a` is + // `run_ax and (video is not None and vx.numel() > 0)` (transformer.py:269), so + // a present video stream — enabled or not — still feeds video->audio cross + // attention. If these were equal, the cross-modal path would be dead on the + // joint arm instead. + const vllm::Ltx2DitOutputs joint = + Ltx2DitForward(Cpu(), p, set.weights, &m.video, &m.audio, vt::DType::kF32); + REQUIRE(joint.audio.size() == a_only.audio.size()); + bool audio_differs = false; + for (size_t i = 0; i < joint.audio.size(); ++i) { + if (joint.audio[i] != a_only.audio[i]) audio_differs = true; + } + CHECK_MESSAGE(audio_differs, + "the audio-only forward equals the joint one, so video->audio cross attention " + "contributed nothing on the joint arm"); } // --------------------------------------------------------------------------- diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index bf25112ce..9c11de380 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -21,6 +21,7 @@ #include #include +#include #include #include #include @@ -4794,3 +4795,804 @@ TEST_CASE("ltx2 retake: the wrong recipe refuses, and the reference arm still do } } } + +// ─── TEXT-TO-AUDIO (row LTX25-T2A-ONE-STAGE, issue #1005) ──────────────────── +// +// `T2AOneStagePipeline` (ltx-pipelines t2a_one_stage.py:43, `__call__` at :109) +// at Lightricks/LTX-2 @ fd4ded7f. UPSTREAM SHIPS NO TESTS at that pin — `find +// /home/mudler/_git/LTX-2 -name 'test_*.py'` returns 0 — so there is nothing to +// port, and what follows pins upstream's BEHAVIOURS against `file:line` anchors +// instead. At least one assertion in each refusal case is tied to a LOCAL fact, +// because a case that asserts only upstream symbol names cannot see a refusal +// whose claim about THIS tree has gone stale. +namespace { + +// A t2a engine on the shipped fixture. NO `video_vae_path`, which is the load +// half of the row: upstream's `T2AOneStagePipeline.__init__` never calls +// `model_paths.video_vae()` (t2a_one_stage.py:68-107). +vllm::multimodal::VideoModelParams T2aParams(const ltx2_fixture::Paths& paths) { + vllm::multimodal::VideoModelParams mp; + mp.dit_path = paths.dit; + mp.audio_vae_path = paths.audio_vae; + mp.encoder_path = paths.encoder; + mp.extras[vllm::multimodal::kLtx2EncoderConfigPathExtra] = paths.encoder_config; + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = "t2a_one_stage"; + mp.device = 0; + return mp; +} + +vllm::multimodal::VideoGenParams T2aGen(const std::string& out_dir, const std::string& prompt) { + vllm::multimodal::VideoGenParams gen; + gen.prompt = prompt; + gen.num_frames = 25; + gen.steps = 2; // two sigma intervals is enough to exercise the loop + gen.has_seed = true; + gen.seed = 11; + gen.output_dir = out_dir; + // The reduced DiT has TWO blocks (ltx2_video_fixture.h `ReducedDitParams`), so + // the params table's own `stg_blocks = [28]` is out of range here. Named + // explicitly rather than by turning STG off, because the default-block refusal + // is its own case below and this one is about the render. + gen.extras[vllm::multimodal::kLtx2AudioStgBlocksExtra] = "1"; + // The recipe's own default negative prompt is upstream's + // `DEFAULT_NEGATIVE_PROMPT` — an English sentence — and this fixture's + // tokenizer carries a three-token vocabulary. Overriding it here is what the + // `--negative-prompt` flag is for (utils/args.py:1083-1088), and the DEFAULT's + // reachability is asserted separately on the recipe rather than by pushing an + // out-of-vocabulary string through a reduced tokenizer. + gen.extras[vllm::multimodal::kLtx2NegativePromptExtra] = "c b a"; + return gen; +} + +} // namespace + +TEST_CASE("ltx2 t2a: an audio-only render returns a waveform and NO picture") { + Workspace ws; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(T2aParams(ws.paths)); + REQUIRE(engine != nullptr); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + CHECK(ltx->pipeline_kind() == "t2a_one_stage"); + + const std::string out = ws.root + "/t2a"; + const vllm::multimodal::VideoResult result = engine->Generate(T2aGen(out, "a b c")); + + // 1. NO PICTURE, said three ways, because each catches a different build. + // `frame_count` catches a render that produced frames; the empty + // `frame_dir` and `mux_argv` catch one that produced none and still handed + // the caller a directory and an ffmpeg command over a pattern matching no + // file. + CHECK(result.frame_count == 0); + CHECK(result.frame_dir.empty()); + CHECK(result.mux_argv.empty()); + CHECK(result.mux_output_path.empty()); + CHECK(result.width == 0); + CHECK(result.height == 0); + // ...AND NO FRAME ON DISK. This is the half the fields cannot make: a build + // that wrote `frame_000000.ppm` and reported zero passes every check above. + { + std::ifstream frame(out + "/frame_000000.ppm", std::ios::binary); + CHECK_MESSAGE(!frame.good(), "an audio-only render wrote a frame"); + } + + // 2. THERE IS SOUND, at the vocoder's own output rate, and it is not silence. + // The lower bound is the assertion a size or a rate cannot make: a + // zero-initialized decode produces a perfectly well-formed WAV of exactly + // the right length. + CHECK(result.sample_rate == 48000); + const std::string wav = ReadAll(result.audio_path); + REQUIRE(wav.size() > 44); + CHECK(wav.compare(0, 4, "RIFF") == 0); + { + int peak = 0; + for (size_t i = 44; i + 1 < wav.size(); i += 2) { + int16_t s = 0; + std::memcpy(&s, wav.data() + i, sizeof(s)); + peak = std::max(peak, s < 0 ? -static_cast(s) : static_cast(s)); + } + CHECK_MESSAGE(peak > 0, "the rendered waveform is digital silence"); + } + + const vllm::multimodal::Ltx2ConditioningTrace trace = ltx->last_conditioning(); + CHECK(trace.completed); + CHECK(trace.t2a_rendered); + + // 3. NO VIDEO STREAM EVER REACHED THE DiT. This is the whole reason the field + // exists: upstream's `run_v2a` tests PRESENCE, not `enabled` + // (transformer.py:269), so a build that handed the forward a + // present-but-disabled video stream would feed video->audio cross attention + // from a latent this pipeline never meant to exist — and would return a + // waveform of exactly the right length, channel count and sample rate. + CHECK_FALSE(trace.t2a_video_stream_present); + + // 4. THE GUIDER RAN, arm by arm. The counters are incremented at the forward, + // so this is a statement about the passes that were issued rather than + // about the parameters that were meant to drive them. + CHECK(trace.t2a_cond_forwards == 2); + CHECK(trace.t2a_uncond_forwards == 2); + CHECK(trace.t2a_perturbed_forwards == 2); + // ...and STG perturbed the block that was ASKED for. A count alone cannot + // tell block 1 from block 0, and which block is perturbed is the whole of + // STG. + REQUIRE(trace.t2a_perturbed_blocks.size() == 1); + CHECK(trace.t2a_perturbed_blocks[0] == 1); + + // 5. The latent is populated. A digest alone is stable across a collapse to + // zeros; the absmax is the bound that is not. + CHECK(trace.audio_tokens > 0); + CHECK(trace.audio_latent_absmax > 1e-6); +} + +TEST_CASE("ltx2 t2a: the guidance ARMS are separable, and each one moves the render") { + Workspace ws; + const vllm::multimodal::VideoModelParams mp = T2aParams(ws.paths); + + struct Out { + vllm::multimodal::Ltx2ConditioningTrace trace; + std::string wav; + }; + const auto render = [&](const std::string& tag, + const std::map& overrides) { + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + REQUIRE(engine != nullptr); + vllm::multimodal::VideoGenParams gen = T2aGen(ws.root + "/" + tag, "a b c"); + for (const auto& kv : overrides) gen.extras[kv.first] = kv.second; + const vllm::multimodal::VideoResult result = engine->Generate(gen); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + return Out{ltx->last_conditioning(), ReadAll(result.audio_path)}; + }; + + const Out full = render("g_full", {}); + // `cfg_scale = 1.0` is `math.isclose(cfg_scale, 1.0)` — upstream's OWN + // predicate for "no unconditional generation" (guiders.py:275-277), and NOT an + // exact `!= 1.0` comparison. + const Out no_cfg = render("g_nocfg", {{vllm::multimodal::kLtx2AudioCfgScaleExtra, "1.0"}}); + const Out no_stg = render("g_nostg", {{vllm::multimodal::kLtx2AudioStgScaleExtra, "0.0"}}); + + // Each arm turns off exactly its own forward, and leaves the others alone. + CHECK(no_cfg.trace.t2a_uncond_forwards == 0); + CHECK(no_cfg.trace.t2a_perturbed_forwards == full.trace.t2a_perturbed_forwards); + CHECK(no_stg.trace.t2a_perturbed_forwards == 0); + CHECK(no_stg.trace.t2a_uncond_forwards == full.trace.t2a_uncond_forwards); + CHECK(no_cfg.trace.t2a_cond_forwards == full.trace.t2a_cond_forwards); + + // AND EACH ONE CHANGES THE RENDER. Without this, a build that issued the extra + // forwards and then discarded them would pass every counter above — which is + // exactly the "recorded value is not a reached one" failure this campaign has + // already paid for once. + CHECK(full.wav.size() == no_cfg.wav.size()); + CHECK(full.wav != no_cfg.wav); + CHECK(full.wav != no_stg.wav); + CHECK(no_cfg.wav != no_stg.wav); + + // The STG DELTA depends on WHICH block is perturbed. Two builds that perturb + // different blocks issue the same three forwards and differ only here, so a + // port that ignored `stg_blocks` and perturbed everything (or nothing) would + // pass every assertion above. + const Out block0 = render("g_b0", {{vllm::multimodal::kLtx2AudioStgBlocksExtra, "0"}}); + REQUIRE(block0.trace.t2a_perturbed_blocks.size() == 1); + CHECK(block0.trace.t2a_perturbed_blocks[0] == 0); + CHECK(block0.wav != full.wav); +} + +TEST_CASE("ltx2 t2a: the refusals name what is missing, and each is checked HERE") { + Workspace ws; + const vllm::multimodal::VideoModelParams mp = T2aParams(ws.paths); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + REQUIRE(engine != nullptr); + + const auto refuses = [&](vllm::multimodal::VideoGenParams gen, + const char* needle) -> std::string { + // `needle` is a `const char*`, and doctest stringifies a bare `char*` as a + // BOOL — a failure would print `true` instead of the string that was looked + // for. Bound to a std::string before it reaches any doctest macro. + const std::string want(needle); + INFO("needle = " << want); + try { + (void)engine->Generate(gen); + FAIL_CHECK("expected a refusal naming: " << want); + return std::string(); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO("refusal = " << msg); + CHECK_MESSAGE(msg.find(want) != std::string::npos, "the refusal did not name the needle"); + return msg; + } + }; + + SUBCASE("a resolution is refused rather than accepted and ignored") { + vllm::multimodal::VideoGenParams gen = T2aGen(ws.root + "/r_res", "a b c"); + gen.width = 64; + gen.height = 64; + refuses(gen, "height/width are unused"); + } + + SUBCASE("the params table's own STG block is out of range on THIS DiT") { + // The LOCAL fact, and it is what makes this case able to see staleness. The + // fixture's DiT has two blocks; upstream's default `stg_blocks` is [28]. The + // refusal must name the range it checked against, so a build that silently + // clamped or ignored the index would not produce this message. + vllm::multimodal::VideoGenParams gen = T2aGen(ws.root + "/r_stg", "a b c"); + gen.extras.erase(vllm::multimodal::kLtx2AudioStgBlocksExtra); + const std::string msg = refuses(gen, "STG block index 28 is outside [0, "); + // Derived from the tree rather than restated: the range in the message is + // the DiT the engine actually loaded, not a literal this test also knows. + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + CHECK(msg.find("[0, " + std::to_string(ltx->dit_params().num_layers) + ")") != + std::string::npos); + } + + SUBCASE("a perturbed pass over NO block is refused, not run as a no-op") { + vllm::multimodal::VideoGenParams gen = T2aGen(ws.root + "/r_empty", "a b c"); + gen.extras[vllm::multimodal::kLtx2AudioStgBlocksExtra] = ""; + refuses(gen, "the STG delta would be exactly zero"); + } + + SUBCASE("isolated-modality guidance has no second modality to run over") { + // Not reachable through an extra by design — there is no `modality_scale` + // knob — so this asserts the RECIPE pinned it, which is the thing that keeps + // the refusal unreachable. `Ltx2DetectPipelineParams("2.5")` carries 3.0. + const vllm::Ltx2PipelineRecipe t2a = vllm::ResolveLtx2PipelineRecipe("t2a_one_stage", "2.5"); + REQUIRE(t2a.audio_only); + REQUIRE(t2a.phases.size() == 1); + CHECK(t2a.phases[0].audio_guidance.modality_scale == 1.0); + CHECK(vllm::Ltx2DetectPipelineParams("2.5").audio_guider.modality_scale == 3.0); + } + + SUBCASE("a video-only knob on a t2a engine is refused rather than ignored") { + vllm::multimodal::VideoGenParams gen = T2aGen(ws.root + "/r_crf", "a b c"); + gen.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "0"; + refuses(gen, "no meaning on a text-to-audio render"); + } + + SUBCASE("a keyframe has no stream to condition") { + vllm::multimodal::VideoGenParams gen = T2aGen(ws.root + "/r_kf", "a b c"); + gen.first_frame_ppm = "P6\n1 1\n255\n\x01\x02\x03"; + refuses(gen, "`video=None`"); + } +} + +TEST_CASE("ltx2 t2a: a t2a-only knob is refused on the video pipelines") { + Workspace ws; + const vllm::multimodal::VideoModelParams mp = EncoderParams(ws.paths); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + REQUIRE(engine != nullptr); + vllm::multimodal::VideoGenParams gen = PromptedGen(ws.root + "/x", "a b c"); + gen.extras[vllm::multimodal::kLtx2AudioCfgScaleExtra] = "5.0"; + try { + (void)engine->Generate(gen); + FAIL_CHECK("a t2a guider knob must be refused on a video pipeline"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("text-to-audio's alone") != std::string::npos); + // It names the pipeline the ENGINE resolved, so a message that guessed + // would say something else. + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + CHECK(msg.find("'" + ltx->pipeline_kind() + "'") != std::string::npos); + } +} + +TEST_CASE("ltx2 t2a: the DiT forward runs ONE stream, and the old guard's reason was wrong") { + // The lifted refusal, checked against a LOCAL fact rather than against + // upstream symbol names alone. `Ltx2DitForward` used to demand BOTH streams + // and blamed the AudioOnly weight contract; the contract is not what blocked + // it, and the way to see that is that the AV weights this fixture writes are + // enough to run the audio stream by itself. + Workspace ws; + const vllm::SafetensorsFile dit_file = vllm::SafetensorsFile::Open(ws.paths.dit); + vllm::Ltx2DitLoadOptions options; + options.widen_to_f32 = true; // `Ltx2DitForward` is f32 by declaration + const vllm::Ltx2DitCheckpoint ckpt = vllm::Ltx2LoadDitFromSafetensors(dit_file, options); + + const int64_t tokens = 3; + // VARYING PER TOKEN, and that is load bearing rather than tidy. A latent whose + // rows are all equal makes self-attention return a weighted average of + // identical values — which is exactly the value projection — so the STG + // perturbation below would be a numeric no-op and the case would report + // "the perturbation changed nothing" about a build that applies it correctly. + // Measured: with a constant 0.25 fill this assertion failed on the working + // implementation. + std::vector latent(static_cast(tokens * ckpt.params.audio_in_channels)); + for (size_t i = 0; i < latent.size(); ++i) { + latent[i] = 0.25F + 0.01F * static_cast(i % 7) - 0.02F * static_cast(i % 3); + } + std::vector timesteps(static_cast(tokens), 0.5F); + std::vector positions(static_cast(tokens * 2)); + for (int64_t t = 0; t < tokens; ++t) { + positions[static_cast(t * 2)] = static_cast(t) * 0.04; + positions[static_cast(t * 2 + 1)] = static_cast(t + 1) * 0.04; + } + const int64_t ctx = 4; + std::vector context( + static_cast(ctx * ckpt.params.audio_cross_attention_dim), 0.1F); + const float sigma = 0.5F; + + vllm::Ltx2ModalityInput ain; + ain.tokens = tokens; + ain.context_tokens = ctx; + ain.latent = latent.data(); + ain.timesteps = timesteps.data(); + ain.sigma = σ + ain.positions = positions.data(); + ain.context = context.data(); + + const vllm::Ltx2DitOutputs out = vllm::Ltx2DitForward( + vt::Device{}, ckpt.params, ckpt.weights, /*video=*/nullptr, &ain, vt::DType::kF32); + CHECK(out.video.empty()); + REQUIRE(out.audio.size() == static_cast(tokens * ckpt.params.audio_out_channels)); + for (const float v : out.audio) REQUIRE(std::isfinite(v)); + + // BOTH null is still refused, which is upstream's own refusal + // (transformer.py:259-260) rather than a leftover of the old one. + CHECK_THROWS(vllm::Ltx2DitForward(vt::Device{}, ckpt.params, ckpt.weights, nullptr, nullptr, + vt::DType::kF32)); + + // And the STG perturbation MOVES the forward. Without this the flag would be + // a field nothing reads: a build that plumbed it and never applied it returns + // the same finite tensor of the same shape. + vllm::Ltx2DitPerturbation p; + p.audio_self_attn.assign(static_cast(ckpt.params.num_layers), 0); + p.audio_self_attn[0] = 1; + const vllm::Ltx2DitOutputs perturbed = + vllm::Ltx2DitForward(vt::Device{}, ckpt.params, ckpt.weights, nullptr, &ain, + vt::DType::kF32, /*cache=*/nullptr, &p); + REQUIRE(perturbed.audio.size() == out.audio.size()); + bool moved = false; + for (size_t i = 0; i < out.audio.size(); ++i) { + if (perturbed.audio[i] != out.audio[i]) moved = true; + } + CHECK_MESSAGE(moved, "the STG perturbation changed nothing"); + + // A vector of the wrong length is refused rather than indexed defensively. + vllm::Ltx2DitPerturbation bad; + bad.audio_self_attn.assign(static_cast(ckpt.params.num_layers + 1), 0); + CHECK_THROWS(vllm::Ltx2DitForward(vt::Device{}, ckpt.params, ckpt.weights, nullptr, &ain, + vt::DType::kF32, /*cache=*/nullptr, &bad)); +} + +TEST_CASE("ltx2 t2a: a SKIPPED step runs no forward and reuses the last prediction") { + // `should_skip_step` is `step % (skip_step + 1) != 0` (guiders.py:287-291), so + // `skip_step = 1` skips every ODD step. Upstream then returns + // `DenoisedLatentResult.result_or_none(denoised=last_denoised_audio)` + // (utils/denoisers.py:85-91) BEFORE it assembles a pass, so a skipped step + // costs no DiT forward at all. + // + // WHY A COUNT AND NOT A DIGEST. A build that "skipped the guidance" by running + // the conditional forward and using it — which is the plausible misreading, + // and what this port did on its first draft — produces a finished waveform of + // exactly the right length on a different trajectory. Nothing about the output + // separates the two. The FORWARD COUNT does, and it is the only thing that + // does. + Workspace ws; + const vllm::multimodal::VideoModelParams mp = T2aParams(ws.paths); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + REQUIRE(engine != nullptr); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + + vllm::multimodal::VideoGenParams gen = T2aGen(ws.root + "/skip", "a b c"); + gen.steps = 4; // four sigma intervals: steps 0..3, so 1 and 3 skip + gen.extras[vllm::multimodal::kLtx2AudioSkipStepExtra] = "1"; + (void)engine->Generate(gen); + const vllm::multimodal::Ltx2ConditioningTrace skipped = ltx->last_conditioning(); + + // Two of the four steps ran, and each ran all three arms. + CHECK(skipped.t2a_cond_forwards == 2); + CHECK(skipped.t2a_uncond_forwards == 2); + CHECK(skipped.t2a_perturbed_forwards == 2); + + // The control: the same request with no skipping runs all four. + vllm::multimodal::VideoGenParams full = T2aGen(ws.root + "/noskip", "a b c"); + full.steps = 4; + (void)engine->Generate(full); + const vllm::multimodal::Ltx2ConditioningTrace every = ltx->last_conditioning(); + CHECK(every.t2a_cond_forwards == 4); + CHECK(every.t2a_uncond_forwards == 4); + CHECK(every.t2a_perturbed_forwards == 4); + + // A negative skip is refused rather than taken modulo a non-positive divisor. + vllm::multimodal::VideoGenParams bad = T2aGen(ws.root + "/badskip", "a b c"); + bad.extras[vllm::multimodal::kLtx2AudioSkipStepExtra] = "-1"; + CHECK_THROWS(engine->Generate(bad)); +} + +TEST_CASE("ltx2 t2a: the schedule starts at exactly 1.0") { + // WHY THIS EXISTS, and it is a mutation result rather than a tidiness rule. A + // mutation that scaled the initial latent by `sigmas[0]` — the thing a reader + // coming from another flow-matching sampler expects to see — SURVIVED the + // focused gate at 6 cases / 484 assertions / exit 0. It survived because it is + // an IDENTITY, not because the gate is blind: `LTX2Scheduler` starts at + // `linspace(1, 0, steps + 1)[0] == 1`, the shift map sends 1 to exactly 1 + // (schedulers.py:41-45) and the stretch sends it to `1 - (1 - 1)/scale`, again + // exactly 1 (`:47-55`). + // + // Pinning the identity is what turns "a mutation survived" into a checked + // fact. If upstream ever moves the first sigma off 1, this fires and the two + // forms stop agreeing. + // NOT `steps = 1`, and the exclusion is upstream's arithmetic rather than a + // convenience. At one step the non-zero sigma list is `[1.0]`, so + // `one_minus_z` is `[0.0]`, `scale_factor = 0 / (1 - terminal)` is 0, and the + // stretch computes `1 - 0/0` — NaN, on both sides (schedulers.py:49-54). + // Measured here: `Ltx2SigmaSchedule(1, 0).front()` is `-nan`. Pinning it would + // be pinning a division by zero as if it were a value; a one-step schedule is + // a separate question and is recorded in the row spec rather than asserted. + for (const int64_t steps : {2, 4, 30, 40}) { + INFO("steps = " << steps); + const std::vector sigmas = vllm::Ltx2SigmaSchedule(steps, /*tokens=*/0); + REQUIRE(sigmas.size() == static_cast(steps) + 1); + CHECK(sigmas.front() == 1.0F); + CHECK(sigmas.back() == 0.0F); + } +} + +TEST_CASE("ltx2 t2a: the one_stage recipes noise their initial latent (#1013)") { + // A one_stage render used to start from ZEROS: `OneStagePhase` left + // `noise_scale` at the struct default of 0.0, and `Ltx2GaussianNoise` is + // `latent + noise_scale * (noise - latent)`, so at 0.0 the state stays exactly + // as `create_initial_state` wrote it. Upstream's `ModalitySpec.noise_scale` + // defaults to 1.0 (ltx-pipelines/utils/types.py:110) and + // `TI2VidOneStagePipeline.__call__` constructs both specs without it + // (ti2vid_one_stage.py:233-239). + // + // Gated on the RECIPE rather than on a render, because the value is what the + // engine reads and a render's own noise is not separable from it by eye. + for (const char* version : {"2", "2.3", "2.4", "2.5"}) { + INFO("version = ", version); + const vllm::Ltx2PipelineRecipe one = vllm::ResolveLtx2PipelineRecipe("one_stage", version); + REQUIRE(one.phases.size() == 1); + CHECK(one.phases[0].noise_scale == 1.0); + // And the t2a rows inherit it, which is the reason they are built FROM the + // one_stage recipe rather than beside it. + const vllm::Ltx2PipelineRecipe t2a = + vllm::ResolveLtx2PipelineRecipe("t2a_one_stage", version); + REQUIRE(t2a.phases.size() == 1); + CHECK(t2a.phases[0].noise_scale == 1.0); + CHECK(t2a.audio_only); + CHECK_FALSE(one.audio_only); + } +} + +TEST_CASE("ltx2 t2a: the guider is handed x0 predictions and not raw velocities") { + // #1039. Upstream hands the denoiser an `X0Model` (ltx-pipelines + // utils/blocks.py:480-482), so `_guided_denoise` combines DENOISED tensors: + // `all_v, all_a = transformer(...)` at utils/denoisers.py:188 and + // `audio_guider.calculate(cond_a, uncond_a, ptb_a, mod_a)` at `:203`, over an + // `X0Model.forward` that already applied `to_denoised(latent, v, timesteps)` + // (ltx-core model/transformer/model.py:590-604, `to_denoised` at + // ltx-core utils.py:39-52 — `sample - velocity * sigma`). + // + // This port combined raw DiT VELOCITIES and converted once afterwards. That is + // the same function only while `rescale_scale == 0`, because `calculate`'s + // linear terms are invariant under `x0 = latent - sigma*v`. The rescale branch + // is not: scaling the x0 by `factor` gives `factor*(latent - sigma*v)`, + // scaling the velocity gives `latent - sigma*factor*v`, and the two differ by + // `(factor - 1) * latent`. + // + // WHAT THIS CASE ASSERTS, AND WHY IT IS NOT THE RESCALE ARITHMETIC ITSELF. + // The rescale's numeric consequence is NOT resolvable on the reduced fixture, + // and that was MEASURED rather than assumed. The first draft of this case + // computed both candidate step-0 predictions in full — `factor * x0_pred` and + // `latent - sigma*factor_v*v_pred` — and its own separation guard refused + // them: this fixture's DiT responds to the conditioning at ~1e-5 of its own + // output, so `std(cond)/std(pred)` is 1.0 to 1e-5 in BOTH spaces, both + // factors land within 1e-5 of 1.0, and the two candidates sit 7.6e-07 apart + // against a span of 3.41. An assertion on that difference would be an + // assertion about f32 noise, and it would have been GREEN either way. + // + // So the rescale's consequence is gated at the seam by the case below, which + // measures 0.35 relative disagreement at `rescale_scale = 0.7` against + // 1.5e-07 at 0.0. This case gates what the fixture CAN decide exactly, which + // is the same defect one step earlier: WHICH TENSOR THE GUIDER WAS HANDED. + // + // WHAT MAKES THAT UNREACHABLE BY ACCIDENT — the sibling trap on this campaign + // was a test whose expectation a zero-filled stub also met. + // `cond == latent - sigma*velocity` is an equation between three recorded + // tensors, not a magnitude. It is exact in x0 space; in velocity space `cond` + // IS the velocity and the residual is `|latent - 2*sigma*velocity|`, i.e. the + // whole sample. No fixture scale satisfies it by accident, a zeroed velocity + // collapses it to `cond == latent` and is refused by the lower bound below, + // and a zeroed `cond` fails it outright. + // + // ALL THREE ARMS, AND THE STEP THAT CONSUMES THEM. An earlier draft of this + // case asserted the equation for the CONDITIONAL pass alone. The default T2A + // arm runs three forwards per step, so that draft held the file's own + // "applied to EVERY PASS" claim for one third of the passes, and three + // mutations survived it at 10 cases / 526 assertions / exit 0: + // + // A1 the PERTURBED pass alone left in velocity space + // A2 the UNCONDITIONAL pass alone left in velocity space + // R1b `ToDenoised` applied a SECOND time to the guider's output, between the + // step-0 recording and the Euler step + // + // and a fourth found while closing them: + // + // R1c the same double application placed ABOVE the step-0 recording, so the + // recorded `t2a_first_denoised` is itself doubly converted + // + // Each renders a different waveform of exactly the right length, through a + // guider whose `cond` term is impeccable. So the equation is applied to every + // recorded arm; the guider's own output is reproduced from the three recorded + // arms through the shipped seam, which is what R1c moves; and the Euler step's + // input is recovered from the latent it wrote, which is what R1b moves. + Workspace ws; + const vllm::multimodal::VideoModelParams mp = T2aParams(ws.paths); + + // The arm this case sits on, pinned as a LOCAL fact before anything is read + // off a render: `rescale_scale = 0.7` on the 2.3/2.4/2.5 lineage + // (ltx-pipelines utils/constants.py:63, and the `--audio-rescale-scale` + // default at utils/args.py:1101-1106). + const vllm::Ltx2PipelineRecipe t2a_recipe = + vllm::ResolveLtx2PipelineRecipe("t2a_one_stage", "2.5"); + REQUIRE(t2a_recipe.phases.size() == 1); + CHECK(t2a_recipe.phases[0].audio_guidance.rescale_scale == 0.7); + + // Through the production entry point — `LoadVideoEngine` then + // `VideoEngine::Generate`, which is what `vllm_video_generate` calls. Nothing + // here constructs a guider, a DiT or a modality by hand. `rescale_scale` is + // the recipe's own 0.7, pinned just above and left untouched by `T2aGen`, + // which is the field this case turns on. (`T2aGen` does set + // `audio_stg_blocks`, and that IS a guider field — the two-block fixture + // cannot take the params table's `[28]` — but it selects WHICH block the + // perturbed forward skips, not how the arms are combined.) + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + REQUIRE(engine != nullptr); + engine->Generate(T2aGen(ws.root + "/x0_space", "a b c")); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + REQUIRE(t.t2a_rendered); + + const size_t n = t.t2a_first_latent.size(); + REQUIRE(n > 0); + REQUIRE(t.t2a_first_velocity.size() == n); + REQUIRE(t.t2a_first_cond.size() == n); + REQUIRE(t.t2a_first_denoised.size() == n); + REQUIRE(t.t2a_first_next_latent.size() == n); + const double sigma = t.t2a_first_sigma; + REQUIRE(sigma > 0.0); + + double latent_span = 0.0; + for (size_t i = 0; i < n; ++i) { + latent_span = std::max(latent_span, std::abs(static_cast(t.t2a_first_latent[i]))); + } + // THE FIXTURE CAN DECIDE THIS AT ALL. The two candidate tensors for every arm + // are `latent - sigma*velocity` and `velocity`, and they coincide when the + // sample is zero. A REQUIRE, because nothing below discriminates once it + // fails. (Its per-arm partner, "the DiT returned no velocity", is next to each + // arm's own check: a zero velocity makes `to_denoised` the identity for THAT + // arm alone.) + REQUIRE_MESSAGE(latent_span > 1e-3, + "the step-0 sample is zero, so the two candidate tensors coincide and nothing " + "below discriminates"); + + // ── the equation, once per guidance pass ────────────────────────────────── + // + // EVERY ARM THE RENDER RAN, not only the conditional one. The default T2A + // guider runs three forwards per step (ltx2_t2a.h item 2), `x0_model` claims + // to convert EVERY PASS, and a conditional-only assertion holds that claim for + // one of the three. `t2a_first_uncond` / `t2a_first_perturbed` are empty when + // the guider did not ask for that arm; this render asks for both, which is + // asserted rather than assumed — an arm silently skipped would otherwise + // vacate its own check. + REQUIRE(t.t2a_uncond_forwards > 0); + REQUIRE(t.t2a_perturbed_forwards > 0); + struct Arm { + const char* name; + const std::vector& velocity; + const std::vector& x0; + }; + const Arm arms[] = { + {"cond", t.t2a_first_velocity, t.t2a_first_cond}, + {"uncond", t.t2a_first_uncond_velocity, t.t2a_first_uncond}, + {"perturbed", t.t2a_first_perturbed_velocity, t.t2a_first_perturbed}, + }; + for (const Arm& arm : arms) { + INFO("arm = " << arm.name); + REQUIRE(arm.velocity.size() == n); + REQUIRE(arm.x0.size() == n); + + double velocity_span = 0.0; + double err_x0 = 0.0; // |x0 - (latent - sigma*velocity)| -> 0 in x0 space + double err_v = 0.0; // |x0 - velocity| -> 0 in velocity space + for (size_t i = 0; i < n; ++i) { + const double lat = static_cast(t.t2a_first_latent[i]); + const double vel = static_cast(arm.velocity[i]); + const double x0 = static_cast(arm.x0[i]); + velocity_span = std::max(velocity_span, std::abs(vel)); + err_x0 = std::max(err_x0, std::abs(x0 - (lat - sigma * vel))); + err_v = std::max(err_v, std::abs(x0 - vel)); + } + INFO("sigma = " << sigma << " max|latent| = " << latent_span + << " max|velocity| = " << velocity_span + << " |x0 - (latent - sigma*velocity)| = " << err_x0 + << " |x0 - velocity| = " << err_v << " elements = " << n); + + // 1. `to_denoised` IS NOT THE IDENTITY ON THIS ARM. The second half of the + // non-vacuity guard, per arm: a zeroed velocity collapses the equation to + // `x0 == latent` and would let a stub satisfy it. + REQUIRE_MESSAGE(sigma * velocity_span > 1e-6, + "the DiT returned no velocity on this arm, so `to_denoised` is the identity " + "here and the two candidate tensors coincide"); + // 2. THE GUIDER WAS HANDED THE X0 PREDICTION, exactly — `to_denoised` on the + // way out of the forward, which is `X0Model.forward` (model.py:602-603). + CHECK_MESSAGE(err_x0 <= 1e-5 * latent_span, + "the tensor handed to `Ltx2MultiModalGuidance` on this arm is not " + "`latent - sigma*velocity`, which is what `X0Model.forward` returns (#1039): " + "residual " + << err_x0 << " against a tolerance of " << (1e-5 * latent_span)); + // 3. AND IT WAS NOT THE RAW VELOCITY. Said separately from check 2, because a + // build handing the guider some THIRD tensor fails 2 and would pass a lone + // "not the velocity" check; the pair says which of the two happened. + CHECK_MESSAGE(err_v > 1e-2 * latent_span, + "the tensor handed to `Ltx2MultiModalGuidance` on this arm IS the raw DiT " + "velocity, so the guidance is combined in velocity space and converted once " + "afterwards (#1039)"); + } + + // ── the guider's output is the guider's output ──────────────────────────── + // + // The three recorded arms, through the SHIPPED `Ltx2MultiModalGuidance` on the + // recipe's own params, must reproduce `t2a_first_denoised` bit for bit. This + // does not gate the guider's arithmetic — `Ltx2Rescale`'s own cases and the + // seam case below do that — it gates that the pipeline handed the guider these + // tensors and passed its result on UNTOUCHED. A second `to_denoised` applied + // to the combination is invisible in every per-arm check above, because it + // moves nothing the guider was handed. + // + // `stg_blocks` is the one guider field `T2aGen` overrides and the one + // `Ltx2MultiModalGuidance` does not read (it selects the perturbed forward's + // blocks, not the combination), so the recipe's params are the render's params + // for this call. + { + const std::vector replayed = vllm::Ltx2MultiModalGuidance( + t2a_recipe.phases[0].audio_guidance, t.t2a_first_cond.data(), t.t2a_first_uncond.data(), + t.t2a_first_perturbed.data(), /*uncond_modality=*/nullptr, static_cast(n)); + REQUIRE(replayed.size() == n); + double worst = 0.0; + for (size_t i = 0; i < n; ++i) { + worst = std::max(worst, std::abs(static_cast(replayed[i]) - + static_cast(t.t2a_first_denoised[i]))); + } + INFO("max|replayed guidance - t2a_first_denoised| = " << worst); + // EXACT, not a tolerance: it is the same function over the same f32 inputs, + // so any non-zero residual is another operation this pipeline applied. + CHECK_MESSAGE(worst == 0.0, + "`t2a_first_denoised` is not `Ltx2MultiModalGuidance` over the three recorded " + "arms, so something else was applied to the guider's result (#1039)"); + // And the combination MOVED what it was handed, so the arms checked above are + // real inputs to it rather than recorded values beside one. + CHECK(t.t2a_first_denoised != t.t2a_first_cond); + } + + // ── and the sampler consumed exactly that ───────────────────────────────── + // + // `Ltx2EulerStep` is `x + (x - denoised)/sigma * (sigma_next - sigma)` + // (ltx2_pipeline.cpp, `EulerDiffusionStep` at ltx-pipelines + // utils/blocks.py:524-527). Recovering `t2a_first_next_latent` from + // `t2a_first_denoised` pins WHICH tensor the step was handed. `ToDenoised` + // applied a second time between the recording and the step leaves every field + // above untouched and moves only this one. + // + // The schedule is re-derived from the shared seam rather than read off the + // render, and tied to it by the sigma the render recorded. + { + const std::vector sigmas = vllm::Ltx2SigmaSchedule(/*steps=*/2, /*tokens=*/0); + REQUIRE(sigmas.size() == 3); // `T2aGen` renders two steps + REQUIRE(static_cast(sigmas[0]) == sigma); + const double dt = static_cast(sigmas[1]) - static_cast(sigmas[0]); + REQUIRE_MESSAGE(std::abs(dt) > 1e-3, + "the first two sigmas coincide, so the Euler step is the identity and this " + "check cannot see what it consumed"); + double worst = 0.0; + double scale = 0.0; + for (size_t i = 0; i < n; ++i) { + const double lat = static_cast(t.t2a_first_latent[i]); + const double den = static_cast(t.t2a_first_denoised[i]); + const double expected = lat + (lat - den) / sigma * dt; + worst = std::max(worst, std::abs(static_cast(t.t2a_first_next_latent[i]) - expected)); + scale = std::max(scale, std::abs(expected)); + } + INFO("sigma = " << sigma << " -> " << sigmas[1] << " max|next - Euler(latent, denoised)| = " + << worst << " scale = " << scale); + REQUIRE_MESSAGE(scale > 1e-3, + "the recomputed Euler output is zero, so the residual below bounds nothing"); + CHECK_MESSAGE(worst <= 1e-5 * scale, + "the latent `Ltx2EulerStep` wrote is not the step over `t2a_first_denoised`, so " + "the sampler was handed some other tensor (#1039): residual " + << worst << " against a tolerance of " << (1e-5 * scale)); + } +} + +TEST_CASE("ltx2 t2a: rescale_scale 0 is the control because both spaces agree there") { + // #1039's control, executable rather than asserted in prose. The case above + // would be testing something OTHER than the defect if it also fired at + // `rescale_scale = 0`, because `MultiModalGuider.calculate`'s linear terms + // (guiders.py:261-266) are invariant under `x0 = latent - sigma*v`: + // + // latent - sigma*(c + a(c-u) + b(c-p)) == x0c + a(x0c-x0u) + b(x0c-x0p) + // + // The rescale at `:268-271` is the only part that is not. This case measures + // both, on the real seam, with a latent that makes the difference visible. + const int64_t n = 512; + std::vector latent(static_cast(n)); + std::vector v_cond(static_cast(n)); + std::vector v_uncond(static_cast(n)); + std::vector v_ptb(static_cast(n)); + // Deterministic and NON-CONSTANT. A zero latent erases `(factor - 1) * latent` + // entirely and a constant one reduces it to a uniform offset; either would + // make the disagreement below unmeasurable and the control meaningless. + uint64_t s = 0x9E3779B97F4A7C15ULL; + const auto next = [&s]() { + s ^= s << 13; + s ^= s >> 7; + s ^= s << 17; + return static_cast(static_cast(s >> 11) / 9007199254740992.0 * 2.0 - 1.0); + }; + for (int64_t i = 0; i < n; ++i) { + const size_t j = static_cast(i); + latent[j] = 2.0F * next(); + v_cond[j] = next(); + v_uncond[j] = next(); + v_ptb[j] = next(); + } + const float sigma = 0.83F; + std::vector x_cond(static_cast(n)); + std::vector x_uncond(static_cast(n)); + std::vector x_ptb(static_cast(n)); + for (int64_t i = 0; i < n; ++i) { + const size_t j = static_cast(i); + x_cond[j] = latent[j] - sigma * v_cond[j]; + x_uncond[j] = latent[j] - sigma * v_uncond[j]; + x_ptb[j] = latent[j] - sigma * v_ptb[j]; + } + + vllm::Ltx2MultiModalGuiderParams params; + params.cfg_scale = 7.0; // the T2A defaults (utils/constants.py:58-66) + params.stg_scale = 1.0; + params.modality_scale = 1.0; + params.skip_step = 0; + + const auto compare = [&](double rescale) { + params.rescale_scale = rescale; + // Upstream's shape: combine the X0 predictions. + const std::vector x0_space = vllm::Ltx2MultiModalGuidance( + params, x_cond.data(), x_uncond.data(), x_ptb.data(), /*uncond_modality=*/nullptr, n); + // The shape this port shipped: combine the VELOCITIES and convert once after. + const std::vector v_space = vllm::Ltx2MultiModalGuidance( + params, v_cond.data(), v_uncond.data(), v_ptb.data(), /*uncond_modality=*/nullptr, n); + double worst = 0.0; + double scale = 0.0; + for (int64_t i = 0; i < n; ++i) { + const size_t j = static_cast(i); + const double converted = static_cast(latent[j]) - + static_cast(sigma) * static_cast(v_space[j]); + worst = std::max(worst, std::abs(static_cast(x0_space[j]) - converted)); + scale = std::max(scale, std::abs(static_cast(x0_space[j]))); + } + REQUIRE(scale > 1e-3); + return worst / scale; + }; + + const double at_zero = compare(0.0); + const double at_default = compare(0.7); + INFO("relative disagreement: at rescale 0.0 = " << at_zero + << " at rescale 0.7 = " << at_default); + // AT 0.0 THE TWO SPACES ARE THE SAME FUNCTION, to f32 rounding. An assertion + // that fires here is not about #1039. + CHECK(at_zero < 1e-4); + // AT THE SHIPPED 0.7 THEY ARE NOT, by orders of magnitude more. That is the + // whole of the defect, and it is why the case above can sit on the default. + CHECK(at_default > 1e-2); + CHECK(at_default > 100.0 * at_zero); +}