From 36510ec2d70bac1a58d357ddac959468c0f42a92 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 01:36:33 +0000 Subject: [PATCH 1/5] spec(LTX25-GUIDED-VIDEO): the video denoise loop is unguided, and four pipelines wait on the same seam (#1092) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Ltx2PhaseRecipe::video_guidance` is set by every recipe and read by nothing, so a `pipeline_kind = one_stage` render ignores the `cfg_scale = 3.0`, `stg_scale = 1.0`, `rescale_scale = 0.7` and `modality_scale = 3.0` its own recipe resolved. Upstream's `ti2vid_one_stage.py:221-226` @ `fd4ded7f` builds a `FactoryGuidedDenoiser` from exactly those. The spec commits before the implementation on the same pull request, which is this repository's default shape. It records three missing parts rather than one: the denoiser itself (`utils/denoisers.py:62-207`), the negative conditioning the video path never encodes, and the `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN` perturbations without which `modality_scale = 3.0` — the default on every video row — cannot run. `ltx2.h:41-49` records those last two as not ported because "nothing upstream that this port serves constructs them"; that sentence was true for text-to-audio, which pins `modality_scale = 1.0`, and is false for every video pipeline. It also states what the row cannot claim: there is no oracle run behind any number in it. vLLM-Omni is UNPINNED (#633) and carries no LTX-2.5 recipe, and no LTX-2.5 checkpoint here has a recorded sha256 (#1048), so the guidance arithmetic is gated against upstream SOURCE and not upstream OUTPUT. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 1 + .agents/specs/ltx25-guided-video.md | 441 ++++++++++++++++++++++++++++ 2 files changed, 442 insertions(+) create mode 100644 .agents/specs/ltx25-guided-video.md diff --git a/.agents/issue-index.md b/.agents/issue-index.md index d32f263f6..5e6fdfd63 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -306,3 +306,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#989](https://github.com/mudler/vllm.cpp/issues/989) | `VT-FP8-QUANT-ARCH-GATE` | `scripts/check-pr-size.py`'s `classify_path` has no entry for `.agents/reachability.md` (added by `POLICY-NOTHING-LANDS-DEAD`, [#888](https://github.com/mudler/vllm.cpp/issues/888) @ `8f49ac3be`), and it FAILS CLOSED, so `pr-size` — a REQUIRED check — refuses every pull request that touches that guide, and `tests/scripts/test_check_pr_size.py` has been red on `main` ever since. Red SILENTLY: that suite is wired into no CI job and is not in `agent-preflight.sh`'s `SUITES`, so the only thing that ever loads it is `check-pr-size`'s own executable-evidence contract, which fires only when a PR edits a checker — the red is reachable exclusively by the next person who must touch that file, and presents to them as their own breakage (the [#584](https://github.com/mudler/vllm.cpp/issues/584)/[#965](https://github.com/mudler/vllm.cpp/issues/965) shape). Third instance of the class after [#856](https://github.com/mudler/vllm.cpp/issues/856) (`issue-index.md` + the style guides) and [#668](https://github.com/mudler/vllm.cpp/issues/668) (`.agents/oracles/*`), both fixed in flow by the row that tripped over them. FIXED IN FLOW while landing [#960](https://github.com/mudler/vllm.cpp/issues/960), which could not register its new checker's creation mutation without touching `check-pr-size.py` at all. NOT fixed: wiring that suite into CI, which is its own change and would red `main` until this landed | bug | | [#1066](https://github.com/mudler/vllm.cpp/issues/1066) | `ENG-EXPERT-STREAM` | `Qwen35ExpertStream` (`src/vllm/model_executor/models/qwen3_5.cpp`) is a **process-lifetime singleton** and keyed its slot cache on `(TowerId(base), expert)`, where `base` is the expert tower's host buffer **ADDRESS**. Its own comment stated the premise and drew the wrong conclusion: "A tower's identity is its base pointer, which is stable for the model's life". The premise is true; the conclusion does not follow, because the CACHE is not scoped to one model's life. Free a model, load another, and the allocator hands the new towers addresses the old ones held, so the new model's expert resolves to an entry filled from a DIFFERENT checkpoint — returned as a HIT, which by contract moves no bytes, so no counter moves and nothing downstream has anything to observe. MEASURED on two synthetic 4-layer/4-expert MoE models in one process, instrumenting `KqExpertSlice` to `memcmp` each returned slot against the tower slice it claims to be: **24 towers occupied 21 distinct addresses, and 20 of 222 slices returned another tower's bytes**; end to end the two arms disagreed on all 160 logits while each arm was internally deterministic (0 differing values on a repeat), which rules out nondeterminism. Invisible to every existing test of this row by construction, because all of them build the cache, store and streamer by hand and none runs two models through the production seam. Reachable by any process that loads a model, releases it, and loads another. Fixed by `OwnedTensor::TowerUid()`, a lazily assigned process-unique counter stamped on the tensor and re-stamped when `bytes` moves (so a copy cannot inherit an identity along with a different buffer); a counter cannot collide because it never goes backwards. Found and fixed while repairing the F1-F11 wiring review for [#912](https://github.com/mudler/vllm.cpp/issues/912). Spec [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1088](https://github.com/mudler/vllm.cpp/issues/1088) | `LTX25-RESOLUTION-ENVELOPE` | `docs/USAGE.md` published "Measured to complete on one GB10 = 320x192 at 25 frames" beside "Measured NOT to complete = 448x256 at 25 frames, the denoise finishes, then the decode loses about 59 GB in 24 s", and `docs/BENCHMARKS.md` said the same. Two renders on `dgx.casa` on 16 to 17 August 2026 against `main` `0b0b8900f`, which carries [#1041](https://github.com/mudler/vllm.cpp/issues/1041) threaded decode, [#1032](https://github.com/mudler/vllm.cpp/issues/1032) T2A and [#1036](https://github.com/mudler/vllm.cpp/issues/1036) f32 decode accumulators, in container `vllmcpp-build:gb10`, `Release`, `VLLM_CPP_CUDA=ON`, arch `121a`, `TRITON=ON`, CUTLASS absent so FlashAttention-2 was not built, `VLLM_CPP_CPU_THREADS=20`, NVFP4 transformer, no `--allow-unported`, **COMPLETED 448x256/25f in 3085 s and 704x448/25f in 4231 s**. 1024x576/25f was not attempted to completion because another session claimed the box. That is scheduling and not an envelope, so 704x448 is NOT a ceiling. The ~59 GiB cliff did NOT recur under a 2 s memory guard that would have seen it. `MemAvailable` floors were 38.96 GiB over 1289 samples at 448x256 and 38.89 GiB over 1743 samples at 704x448, with ZERO samples under 34 GiB on either, peak use of 80 of 119 GiB, and no reboot. The 704x448 artifact was verified rather than inferred from an exit code: 25/25 distinct frame md5s, 0 near-uniform and 0 near-black frames, adjacent-frame mean absolute difference 4.381 against a uniform-noise reference of 85.3 on the same shape, 0/24 zero-motion pairs, and audio at 48 kHz stereo, 1.010 s, -37.29 dBFS, 20/20 windows above threshold. NOT claimed: ONE run per geometry on a contended shared box with no oracle on either side, so two points establish no scaling law. The 59 GB observation is KEPT and attributed to its own run, which is rung F1 in `benchmark-record.md`, a prompt-embeds render with no text tower that an ARMED WATCHDOG ended at 13.77 GiB against an 18 GiB floor rather than the engine failing. Attributing it stays open as [#1014](https://github.com/mudler/vllm.cpp/issues/1014). The claim that most of a 320x192/25f render is spent in the host VAE decode is superseded too: after #1041 a resolution-independent ~1731 s single-threaded phase, measured at 1731 s and 1732 s across two rungs whose voxel counts differ 2.75x, is 57 to 66% of wall, filed as [#1087](https://github.com/mudler/vllm.cpp/issues/1087). FIXED IN FLOW in `docs/USAGE.md`, `docs/BENCHMARKS.md` and [`ltx25-resolution-envelope.md`](specs/ltx25-resolution-envelope.md) section 4.1 | bug | +| [#1092](https://github.com/mudler/vllm.cpp/issues/1092) | `LTX25-GUIDED-VIDEO` | The LTX-2.5 **video** denoise loop runs one UNGUIDED forward per step: `Ltx2PhaseRecipe::video_guidance` is set by every recipe (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1069 @ b5756ea8c`) and read by nothing, so a `pipeline_kind = one_stage` render ignores `cfg_scale = 3.0`, `stg_scale = 1.0`, `rescale_scale = 0.7` and `modality_scale = 3.0` and denoises along a different trajectory than `ti2vid_one_stage.py:221-226 @ fd4ded7f`, which builds a `FactoryGuidedDenoiser` from exactly those. `allow_guidance_override` (`ltx2_pipeline.h:534`) is dead the same way. Positive control for the grep: the same command for `audio_guidance` returns the T2A consumer at `ltx2_video.cpp:3527`. Blocks four more pipelines on one missing seam (`a2vid_two_stage.py:230`, `ti2vid_two_stages.py:248`, `ti2vid_two_stages_hq.py:271`, `keyframe_interpolation.py:232`). Spec [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) | bug | diff --git a/.agents/specs/ltx25-guided-video.md b/.agents/specs/ltx25-guided-video.md new file mode 100644 index 000000000..88d9c265b --- /dev/null +++ b/.agents/specs/ltx25-guided-video.md @@ -0,0 +1,441 @@ +# LTX25-GUIDED-VIDEO — the guided video denoiser, as a shared seam + +Row `LTX25-GUIDED-VIDEO`, under the campaign [`ltx-2-5.md`](ltx-2-5.md). +Issue [#1092](https://github.com/mudler/vllm.cpp/issues/1092). +Base: `origin/main` @ `b5756ea8c`. +Upstream: Lightricks `LTX-2` @ `fd4ded7f` (the revision every anchor below is +read at), secondary oracle `vllm-omni` (UNPINNED, #633) for the recipe tables +this tree already mirrors. + +Paths are relative to `packages/ltx-pipelines/src/ltx_pipelines/` and +`packages/ltx-core/src/ltx_core/` in that checkout, as the rest of the LTX-2.5 +specs use them. + +## 0. Honesty statement — what this row does and does not claim + +It claims: the LTX-2.5 **video** denoise path now runs upstream's guided +denoiser, that the guidance is combined in **x0** space, that one production +pipeline (`pipeline_kind = one_stage`) reaches it on its **own default +configuration**, and that the gate can see the space error that #1039 was. + +It does not claim: a numerical comparison against a running oracle. There is +none — vLLM-Omni is UNPINNED (#633) and carries no LTX-2.5 recipe at all, and no +LTX-2.5 checkpoint on this host has a recorded sha256 (#1048). Every anchor +below is **source read** at `fd4ded7f`, and every number below is measured on +**this tree's own reduced fixture**. That is what an ungateable lane looks like +when it is stated instead of implied. + +It does not claim to retire [#1049](https://github.com/mudler/vllm.cpp/issues/1049). +See section 6c: one of that issue's four symbols is reached by this row and three +are not, and forcing the other three would mean inventing a dispatch upstream +does not have. + +## 1. Scope + +**In:** + +- `Ltx2GuidedDenoise` — `_guided_denoise` (`utils/denoisers.py:62-207`) in its + own translation unit mirroring upstream's own file. +- The four passes it assembles: `cond`, `uncond`, `ptb`, `mod`. +- The `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN` halves of + `Ltx2DitPerturbation`, without which the `mod` pass cannot run and every video + guider default is unreachable. +- The negative conditioning for the video path — the second half of the encode + `GenerateAudioOnly` already performs and discards. +- The video guidance request extras that `default_1_stage_arg_parser` + (`utils/args.py:930-1010`) exposes, gated by `allow_guidance_override`. +- The `one_stage` pipeline as the reachable consumer. + +**Out, and owed rather than silently absent:** + +- The other three video pipelines that need this seam + (`a2vid_two_stage`, `ti2vid_two_stages`, `ti2vid_two_stages_hq`, + `keyframe_interpolation`). Each is its own row; this row exists so that they + are ordinary porting work rather than blocked. +- The **device-resident** arm of the `ptb` and `mod` passes. + `Ltx2DitForwardDevice` (`ltx2_device.h:136`) takes no `perturbations` + argument. Refused by name on that arm rather than run unperturbed, which would + produce a legal-looking render whose STG term is identically zero. See §4.3. +- `BatchedPerturbationConfig`'s partial blend (`attention.py:571-572`) and + batch > 1, which stay degenerate at the one batch size this port runs — the + statement `ltx2.h` already carries, unchanged. + +## 2. Upstream chain + +The executing chain for one guided step, top to bottom: + +| Step | Upstream | What it decides | +|---|---|---| +| the stage builds the model | `utils/blocks.py:480-482` — `X0Model(self._prepared_builder().build(...))` | the transformer the loop is handed is **already** an x0 model | +| the loop calls the denoiser | `utils/samplers.py:73-74` | one denoiser call per step | +| the denoiser assembles passes | `utils/denoisers.py:100-137` | `cond`, `uncond`, `ptb`, `mod`, in that order | +| the forward converts | `model/transformer/model.py:590-604` — `to_denoised(video.latent, vx, video.timesteps)` | **every** pass is x0 before any combination | +| the guider combines | `components/guiders.py:244-273`, per modality at `denoisers.py:203-204` | `cond + (cfg-1)(cond-uncond) + stg(cond-ptb) + (mod-1)(cond-modpass)`, then the rescale at `:268-271` | +| the loop post-processes | `utils/samplers.py:35` — `post_process_latent(denoised, ...)` | the conditioned tokens are pinned back **after** the guider, not per arm | +| the stepper steps | `utils/blocks.py:524-527` / `samplers.py:488-558` | Euler or ancestral, unchanged by this row | + +The pass list is **shared between the two modalities and the guiders are not**. +`denoisers.py:103-137` takes the union — one `uncond` pass if *either* guider +wants one, one `ptb` pass carrying *both* modalities' perturbations, one `mod` +pass if *either* wants one — and then `:203-204` combines each modality with its +**own** guider over the same splits. A per-modality pass list would run up to six +forwards where upstream runs four, and would give the audio stream a different +video state to cross-attend to on the video-only passes. That is the single +structural fact this port has to get right, and it is why the seam takes both +guiders rather than being called twice. + +The perturbation types are per direction, not per modality +(`guidance/perturbations.py:8-16`, applied at `model.py:443-458`): + +| Pass | Perturbations | Reaches | +|---|---|---| +| `cond` | none | — | +| `uncond` | none | negative context on both streams | +| `ptb` | `SKIP_VIDEO_SELF_ATTN` on `video_guider.stg_blocks`, `SKIP_AUDIO_SELF_ATTN` on `audio_guider.stg_blocks` | `attention.py:557` `use_attention = not all_perturbed` | +| `mod` | `SKIP_A2V_CROSS_ATTN` and `SKIP_V2A_CROSS_ATTN`, **all blocks** | `transformer.py:335,366` `cross_attn_skip_all` | + +## 3. Our baseline, derived at `b5756ea8c` + +`src/vllm/multimodal/ltx2_video.cpp:3036-3045` runs **one** forward per step and +converts its velocity: + +``` +const Ltx2DitOutputs velocity = im.on_device ? Ltx2DitForwardDevice(...) : Ltx2DitForward(...); +const std::vector v_denoised = PostProcessLatent(ToDenoised(video.latent, velocity.video, ...), video); +const std::vector a_denoised = PostProcessLatent(ToDenoised(audio.latent, velocity.audio, ...), audio); +``` + +Everything the recipe resolved for that step is set and read by nothing: + +``` +$ git grep -n 'video_guidance' -- src include # @ b5756ea8c +include/vllm/model_executor/models/ltx2_pipeline.h:526: Ltx2MultiModalGuiderParams video_guidance; +src/vllm/model_executor/models/ltx2_pipeline.cpp:1069: phase.video_guidance = params.video_guider; +``` + +The positive control for that grep is the same command for `audio_guidance`, +which returns the T2A consumer at `ltx2_video.cpp:3527`. The term and the path +set are right; the video consumer is genuinely absent. `allow_guidance_override` +(`ltx2_pipeline.h:534`) is the same shape: three recipes set it `false` and +nothing reads it. + +**What is already correct and is reused unchanged:** + +- `Ltx2MultiModalGuidance` (`ltx2_pipeline.cpp:479-522`) — `calculate` including + the unbiased-`std` rescale. Reviewed under #1032/#1039. +- `Ltx2BatchedPerturbationConfig` (`ltx2_pipeline.h:380-405`) — the full + four-type keep-mask, ported under #641 and, per #1049, constructed only by its + own test until this row. +- `ToDenoised` (`ltx2_video.cpp:277`) and `PostProcessLatent` (`:234`). +- The T2A driver (`ltx2_t2a.cpp:322-368`), which is the **template**: it is the + one place in this tree that already converts to x0 inside the model wrapper. + +**What is missing and why nothing noticed:** a token gate cannot see it, and this +path has no token gate. An unguided render returns a finite clip of the right +size, frame count and sample rate. It is #1039's family of defect one level up: +not the wrong space, the wrong number of forwards. + +## 4. Design + +### 4.1 The seam — `ltx2_denoisers.{h,cpp}` + +A new translation unit mirroring `ltx-pipelines/utils/denoisers.py`, rather than +another block inside `ltx2_pipeline.cpp`. Two reasons, and only the first is +about this row: + +1. Upstream has that file. `AGENTS.md` §Shared seams: new capability is additive + files mirroring the upstream structure. +2. `ltx2_pipeline.{h,cpp}` is concurrently edited by #921. A seam that four + later rows will extend does not want to live in the file with the most + contention. + +The transformer is a **callable**, exactly as `_guided_denoise(transformer, ...)` +takes one: + +``` +using Ltx2X0Model = std::function; +``` + +That is the structural claim this row is graded on. The x0 conversion happens +**inside the caller's lambda**, which is upstream's `X0Model` wrapper +(`blocks.py:480-482` builds it; `model.py:590-604` is its forward), so the seam +combines already-denoised tensors and cannot be handed a velocity. Converting +once after the guider instead is a **different function on the default arm** — +`rescale_scale` is 0.7 on every video row — and that is #1039, on the audio arm, +in this tree, six days ago. + +It also means the host forward and the device forward are the same seam with two +lambdas, and that the four later pipelines supply their own conditioning without +the seam knowing anything about keyframes, reference clips or two-stage +schedules. + +### 4.2 The passes + +`Ltx2GuidedDenoise` mirrors `denoisers.py:84-207` line for line: + +- `v_skip`/`a_skip` from `ShouldSkipStep` (`:84-85`); both skipping returns the + previous step's denoised pair with **no forward at all** (`:87-90`). +- `cond` always (`:100`). +- `uncond` when either guider asks or `force_uncond_pass` (`:102-109`), with the + negative context substituted per modality and `v_neg = v_context` when a + modality has none (`:107-108`). +- `ptb` when either guider perturbs, carrying both modalities' `stg_blocks` + (`:111-119`). +- `mod` when either guider isolates, all blocks, both cross directions + (`:121-137`). +- `enabled = not skip` per modality (`:151,161`), which is + `Ltx2ModalityInput::enabled` here. +- the combination per modality with that modality's own guider (`:203-204`). + +Perturbations route through `Ltx2BatchedPerturbationConfig`: one config built +over all N passes (`denuoisers.py:172-176`), then `BatchSlice(i, i+1)` per pass, +then flattened into the `Ltx2DitPerturbation` the forward takes. At batch 1 the +slice is the pass's own mask, which is exactly the degeneracy `ltx2.h` already +records. + +### 4.3 The cross-attention perturbation + +`Ltx2DitPerturbation` grows two booleans, `video_cross_attn_skip_all` and +`audio_cross_attn_skip_all`, mirroring `TransformerArgs.cross_attn_skip_all` +(`transformer_args.py:118`). They gate the A2V and V2A branches at +`ltx2_dit.cpp`'s `if (run_a2v)` / `if (run_v2a)`, mirroring +`transformer.py:335` and `:366`. Note the polarity: `video.cross_attn_skip_all` +gates **A2V** (audio into video) and `audio.cross_attn_skip_all` gates **V2A**, +because the flag rides on the stream being *written*. + +The snapshot of `vx_pre`/`ax_pre` stays outside both guards, as upstream's +`vx_pre_av = vx` at `:333` does, so a build where only one direction is skipped +still reads the pre-cross state for the other. + +`ltx2.h:41-49`'s NOT-PORTED entry is corrected in the same change. Its stated +reason — "nothing upstream that this port serves constructs them" — was true for +text-to-audio, which pins `modality_scale = 1.0` (`t2a_one_stage.py:200-202`), +and is false for every video pipeline, all of which default it to 3.0. + +**The device arm is refused, not degraded.** `Ltx2DitForwardDevice` has no +`perturbations` parameter, so a `ptb` or `mod` pass on that arm would have to +run unperturbed. The result is a finite clip whose STG and modality terms are +identically zero — indistinguishable from a working render. The refusal names +the missing function and the owed issue. CFG alone (a different context, no +perturbation) is served on both arms. + +### 4.4 The negative conditioning + +`Generate` already encodes the positive prompt into both streams +(`ltx2_video.cpp:1771-1806`). The negative half is the same chain with +`recipe.negative_prompt` (or the `negative_prompt` extra), through the same +connector, and is encoded **only when a guider asks for it** +(`do_unconditional_generation`, `guiders.py:275-277`) — at `cfg_scale = 1.0` +there is no unconditional forward and encoding it would be a wasted host-side +12B pass per request. + +Two fallbacks exist for an engine with no text tower, matching the two that +already exist for the positive stream (`prompt_embeds_path`, +`audio_prompt_embeds_path`): `negative_prompt_embeds_path` and +`negative_audio_prompt_embeds_path`. This is a **local adaptation**, recorded as +one: upstream encodes `[prompt, negative_prompt]` in one `PromptEncoder` call +(`ti2vid_one_stage.py:170-178`) and has no embeds surface at all. The adaptation +is the existing one applied to the second of upstream's two encodings, not a new +concept. Without a tower and without those files, a guider that asks for the +unconditional pass is refused by name, exactly as T2A is at +`ltx2_video.cpp:3583-3593`. + +### 4.5 The request extras + +Mirroring `default_1_stage_arg_parser` (`utils/args.py:947-1010`), one extra per +flag, each overriding one field: + +| Extra | Upstream flag | Field | +|---|---|---| +| `video_cfg_guidance_scale` | `--video-cfg-guidance-scale` | `cfg_scale` | +| `video_stg_guidance_scale` | `--video-stg-guidance-scale` | `stg_scale` | +| `video_rescale_scale` | `--video-rescale-scale` | `rescale_scale` | +| `video_stg_blocks` | `--video-stg-blocks` | `stg_blocks` | +| `a2v_guidance_scale` | `--a2v-guidance-scale` | video `modality_scale` | +| `v2a_guidance_scale` | `--v2a-guidance-scale` | audio `modality_scale` | + +The audio row already exists for T2A (`ltx2_video.h:456-460`) and is reused for +the joint path. Every override is refused on a phase whose +`allow_guidance_override` is `false` — the distilled and retake recipes, whose +guidance is trained in — which is the first read that field has ever had. + +An extra that is PRESENT and empty is upstream's empty list for `nargs="*"`, and +stays distinct from an ABSENT extra, which takes the params table's own value. +That distinction is already made for `audio_stg_blocks` (`ltx2_video.cpp:3540`) +and is made the same way here. + +## 5. Port map + +| Upstream | Here | +|---|---| +| `utils/denoisers.py:62-207` `_guided_denoise` | `Ltx2GuidedDenoise`, `src/vllm/model_executor/models/ltx2_denoisers.cpp` | +| `utils/denoisers.py:25-28` `_POSITIVE_ONLY_GUIDER` | the default-constructed `Ltx2MultiModalGuiderParams`, whose defaults are already `cfg 1.0 / stg 0.0 / modality 1.0` | +| `model/transformer/model.py:590-604` `X0Model.forward` | the caller's `Ltx2X0Model` lambda, `ltx2_video.cpp` | +| `guidance/perturbations.py:8-16` cross types | `Ltx2DitPerturbation::{video,audio}_cross_attn_skip_all` | +| `model/transformer/transformer.py:335,366` `cross_attn_skip_all` | `ltx2_dit.cpp` A2V / V2A guards | +| `components/guiders.py:244-273` | `Ltx2MultiModalGuidance` (unchanged) | +| `utils/args.py:947-1010` | the six extras in §4.5 | +| `ti2vid_one_stage.py:210-226` | the `one_stage` consumer in `ltx2_video.cpp` | + +## 6. Gates + +``` +cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF +cmake --build build -j6 && ctest --test-dir build -j4 --output-on-failure +``` + +Focused: `ctest --test-dir build -R 'ltx2' --output-on-failure`. + +Known-red and cited by the issue that names the test, not the family: +`windows-msvc-*` (#584, no `main` baseline), `test_async_llm` (#294), +`test_engine_core_proc` (#1052), `test_serve_low_tools` (#428), +`test_cpu_x86_llamacpp_floor` exit 4 (#618). + +## 6b. Reachability — the sentence the records must carry + +Entry point: `vllm_video_generate` → `VideoEngine::Generate` +(`include/vllm.h`) on an engine loaded with `pipeline_kind = one_stage`, which +is a documented value of a documented load extra and needs no other flag. The +chain is `Generate` → the phase loop → `Ltx2GuidedDenoise`. No test constructs a +guider, a DiT or a modality by hand to reach it. + +The reachability mutation is the deletion of that call — the +`Ltx2GuidedDenoise(...)` line in the phase loop, replaced by the single unguided +forward this row removes — with the focused gate rerun. A green gate there would +mean the suite measures the seam and not the pipeline. + +## 6c. #1049 — partly retired, and the rest argued rather than deferred + +| Symbol | Before | After | +|---|---|---| +| `Ltx2BatchedPerturbationConfig` | test-only | **reached**, `ltx2_denoisers.cpp` | +| `Ltx2Guidance` | test-only | still test-only | +| `Ltx2CfgDelta` | reached only via `Ltx2Guidance` | unchanged | +| `Ltx2StgDelta` | reached only via `Ltx2Guidance` | unchanged | + +`Ltx2Guidance` is a **kind dispatch upstream does not have**. Every LTX-2 +pipeline builds a `MultiModalGuider` and calls `calculate`; there is no object +that holds a `GuiderKind` and selects between CFG-only, STG-only and multi-modal +arms. Routing the production combination through `Ltx2Guidance(kMultiModal, ...)` +to make the symbol live would add a switch statement between the caller and the +function upstream actually calls, and would still leave `Ltx2CfgDelta` and +`Ltx2StgDelta` — the two arms nothing can select — dead. #1049 stays open, its +scope narrows to those three symbols, and the honest disposition is that they +are ported-but-unreachable arms of `guiders.py:23-27,70-74`, not a wiring gap +this row can close. + +## 7. Tests to port + +Upstream's own tests for this path are `pytest` over `torch` and cannot be run +here; the harness adaptation is the whole of it. What is preserved is the +**structure of what they assert** plus the four defects this tree has already +had on the sibling arm. + +### 7.1 The per-arm invariant, on every arm + +`cond == latent - sigma*velocity`, per pass, from the trace the render records. +Exact in x0 space; in velocity space the residual is the whole sample. The RED +prints `|x0 - velocity| = 0` **exactly**, which is unambiguous. + +Non-vacuity is `REQUIRE`d, not assumed, twice: a zero latent makes the two +candidate tensors coincide, and a zero velocity on a given arm makes +`to_denoised` the identity for that arm alone. + +**Every arm**, because #1039's first gate covered only the conditional pass and +three mutations survived it. This path has four arms, so it needs four rows plus +the two double-application positions. + +### 7.2 The seam-level rescale control + +`rescale_scale = 0.0` against `0.7` on the shipped +`Ltx2MultiModalGuidance`, measuring the disagreement between combining in x0 +space and combining in velocity space. At 0.0 the linear terms are invariant and +the two are the same function; at 0.7 they are not. A gate that fires at 0.0 is +not about this defect. The existing T2A case measured 1.50e-07 against 0.352; +the video case adds the **modality** term, which the T2A control could not carry +because T2A pins `modality_scale = 1.0`. + +### 7.3 The pass count + +`std(cond)/std(pred)` is 1.0 to 1e-5 on this fixture in **both** spaces, so a +naive numeric assertion on the rescale difference passes whether or not the bug +exists (7.6e-07 against a span of 3.41, measured under #1039). The instrument +that works is the **count of forwards by kind**, recorded at the call and not +asserted in prose: an arm silently skipped changes a counter no output does. + +### 7.4 The mutations this gate must survive + +| # | Mutation | Must go RED at | +|---|---|---| +| M1 | `cond` pass left in velocity space | §7.1 cond row | +| M2 | `uncond` pass left in velocity space | §7.1 uncond row | +| M3 | `ptb` pass left in velocity space | §7.1 ptb row | +| M4 | `mod` pass left in velocity space | §7.1 mod row | +| M5 | second `ToDenoised` **below** the step-0 record | the Euler-recovery check | +| M6 | second `ToDenoised` **above** the step-0 record | the guider-replay check | +| M7 | `uncond` pass given the positive context | the replay check / a uncond≠cond check | +| M8 | `mod` pass given no cross-attn perturbation | a mod≠cond check | +| M9 | `ptb` pass given no self-attn perturbation | a ptb≠cond check | +| M10 | `PostProcessLatent` applied per arm instead of after the guider | the replay check | +| M11 | the production call site deleted (reachability) | the whole case | + +Each mutation reports three facts: `git diff --stat`, whether it **BUILT** with +the compile-error count, and the exit code captured directly. A non-building +mutation reads exactly like a passing test. + +## 8. Risks and decisions + +**R1 — the space error, on the default arm.** `rescale_scale` is 0.7 on the +2.4/2.5 video row and 0.45 on the HQ row; both are non-zero, so a space error +hits the default. Mitigated by §7.1 and §7.2, and by the conversion living in +the caller's lambda where the seam cannot receive a velocity. + +**R2 — the shared pass list.** Assembling per modality is the plausible wrong +design and it renders. Mitigated by the pass-count trace and by the seam taking +both guiders. + +**R3 — cost.** Four forwards per step where there was one. That is upstream's +own cost — `denoisers.py` batches them into one call, this port runs them +serially — and it is a **correctness** row, so the throughput axis is not traded +against it. Recorded, not hidden: a `one_stage` render is now up to 4x the DiT +work per step. `distilled_two_stage`, the default recipe and the one every +benchmark on this row's campaign used, denoises with `SimpleDenoiser` upstream +(`distilled.py:266,295`) and is **unchanged** by this row. + +**R4 — the device arm.** §4.3. Refused by name, owed by a new issue, rather than +run unperturbed. + +**R5 — concurrent edits.** #921 touches `ltx2_pipeline.{h,cpp}` and the stepper +enum. This row's new code is in a new file; its edits to `ltx2_pipeline.h` are +additive constants only. + +## 9. Stop conditions + +Stop and report `NEEDS_DECISION` rather than narrowing silently if: + +- the `mod` pass cannot be made to differ from the `cond` pass on the fixture, + because then §7.4 M8 cannot go red and the isolated-modality arm is gated by + nothing; +- the guided `one_stage` render cannot be reached without a text tower **and** + the negative-embeds adaptation is judged out of scope, because then the + consumer is unreachable in-tree and this becomes a seam-only row. + +## Owed + +- **The device-resident `ptb` and `mod` passes.** `Ltx2DitForwardDevice` takes + no `perturbations`. Owned by this row's follow-up issue; refused by name until + then. +- **The other four pipelines** — `a2vid_two_stage`, `ti2vid_two_stages`, + `ti2vid_two_stages_hq`, `keyframe_interpolation`. Each needs its own row; none + is blocked on this seam any more. +- **#1049's remaining three symbols** — see §6c. +- **An oracle-run comparison.** vLLM-Omni is UNPINNED (#633) and carries no + LTX-2.5 recipe; no LTX-2.5 checkpoint here has a recorded sha256 (#1048). The + guidance arithmetic is gated against upstream **source**, not against upstream + **output**, and that is the ceiling on this row's evidence. + +## Now + +Spec committed. Implementation and gate follow on the same pull request. From f4eacff4c5596e7f48a67a2f113540921c9eb9ee Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 02:13:35 +0000 Subject: [PATCH 2/5] feat(LTX25-GUIDED-VIDEO): the guided video denoiser, and the sentence in ltx2.h that was wrong (#1092) The LTX-2.5 video denoise loop ran ONE unguided forward per step. Every recipe resolved a `video_guidance` that nothing read, so a `pipeline_kind = one_stage` render ignored `cfg_scale = 3.0`, `stg_scale = 1.0`, `rescale_scale = 0.7` and `modality_scale = 3.0` and denoised along a different trajectory than `ti2vid_one_stage.py:221-226` @ `fd4ded7f`, which builds a `FactoryGuidedDenoiser` from exactly those. `_guided_denoise` (`ltx-pipelines utils/denoisers.py:62-207`) is now ported, in its own translation unit because upstream has its own file. It assembles the four passes -- cond, uncond, perturbed, isolated-modality -- from the UNION of what the two guiders want, runs each through the caller's `X0Model`, and combines each modality with its own guider over the same splits. Four unported pipelines (`a2vid_two_stage.py:230`, `ti2vid_two_stages.py:248`, `ti2vid_two_stages_hq.py:271`, `keyframe_interpolation.py:232`) were each blocked on this one piece. ## The conversion lives in the caller's lambda, not in the seam `DiffusionStage` never hands the loop a velocity model: it hands `X0Model(builder.build(...))` (`utils/blocks.py:480-482`, forward at `model.py:590-604`). So the seam takes a callable and can only ever receive denoised tensors. Combining velocities and converting once afterwards is a different function wherever `rescale_scale != 0`, which is 0.7 on every video row of the params table. That defect shipped on the audio arm of this tree and is #1039. `post_process_latent` therefore stays OUT of the denoiser. Upstream applies it in the LOOP, to the guider's result (`utils/samplers.py:35`, `:484`). Applying it per arm would make every arm agree on exactly the conditioned tokens, which zeroes the guidance delta precisely where a keyframe or a reference clip is conditioning. ## The NOT-PORTED note in ltx2.h was wrong, not stale It refused `SKIP_A2V_CROSS_ATTN` and `SKIP_V2A_CROSS_ATTN` because "nothing upstream that this port serves constructs them -- STG is built from `stg_blocks` and reaches the SELF-attention types alone". STG does. The isolated-modality pass does not: `_guided_denoise` builds BOTH cross types with `blocks=None` whenever either guider has `modality_scale != 1.0` (`denoisers.py:121-137`), and every VIDEO row of the params table sets it to 3.0. The sentence was true of text-to-audio, which pins the field to 1.0 (`t2a_one_stage.py:200-202`), and it was written while text-to-audio was the only guided path here. Both directions are now `cross_attn_skip_all` booleans on `Ltx2DitPerturbation`, gating the A2V and V2A branches exactly as `transformer.py:335` and `:366` do. The same shape appears once more, and is corrected the same way: `negative_prompt` and the five `audio_*` guider knobs were refused on every non-t2a engine, on the same reading. `default_1_stage_arg_parser` carries the whole audio guider row beside the video one (`utils/args.py:1011-1075`) and `TI2VidOneStagePipeline` consumes both. That refusal is gone; the direction that survives is the one that refuses a knob describing a PICTURE on a pipeline that renders none. ## What a caller gains, and what is refused Seven per-generation extras mirror `default_1_stage_arg_parser`, reaching `ltx2-gen` and the C ABI. Every one is refused whole on a phase whose recipe sets `allow_guidance_override = false` -- the distilled and retake recipes, whose guidance is distilled into the weights. That field had never been read before. The unconditional forward needs a negative conditioning. With a tower it is the second half of the encode `GenerateAudioOnly` already performed and discarded; without one, `negative_prompt_embeds_path` and `negative_audio_prompt_embeds_path` are the negative half of the existing embeds fallback, and are recorded as a local adaptation rather than an upstream surface. With neither, a `cfg_scale` other than 1.0 is refused by name rather than served the positive context twice, which would leave the whole classifier-free term at exactly zero. Two more refusals exist because the alternative renders: an `stg_blocks` list that names no block this checkpoint has (upstream's membership test would perturb nothing, leaving `stg_scale * (cond - perturbed)` at zero), and the perturbed or isolated-modality pass on the device arm, where `Ltx2DitForwardDevice` takes no `perturbations` argument. ## What is unchanged `distilled_two_stage`, `dfr`, `retake` and `dmd2` keep the guider they always had, which is `Ltx2MultiModalGuiderParams`'s own default construction and is upstream's `_POSITIVE_ONLY_GUIDER` (`denoisers.py:25-28`). They issue one forward per step through the new seam and their goldens did not move. ## Evidence Spec [`.agents/specs/ltx25-guided-video.md`](.agents/specs/ltx25-guided-video.md), committed at `36510ec2d`, before this change. The gate asserts `x0 == latent - sigma*velocity` per token on ALL FOUR arms -- exact in x0 space, off by the whole sample in velocity space -- with the non-vacuity guard stated twice, once for a zero sample and once per arm for a zero velocity. It then replays `Ltx2MultiModalGuidance` over the recorded arms, and again over arms REBUILT FROM THE RAW VELOCITIES, which is the only check that can see something applied to every arm alike. It recovers the Euler step's input from the latent the sampler wrote. A seam-level control puts the two spaces 1.5e-07 apart at `rescale_scale = 0` and far apart at 0.7, with the modality term present, which the T2A control could not carry. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-guided-video.md | 2 +- CMakeLists.txt | 6 + docs/FEATURES.md | 3 + docs/USAGE.md | 67 +- examples/ltx2_gen/main.cpp | 54 +- include/vllm/model_executor/models/ltx2.h | 81 ++- .../model_executor/models/ltx2_denoisers.h | 179 +++++ include/vllm/multimodal/ltx2_video.h | 106 +++ .../model_executor/models/ltx2_denoisers.cpp | 357 +++++++++ src/vllm/model_executor/models/ltx2_dit.cpp | 17 +- src/vllm/multimodal/ltx2_video.cpp | 513 ++++++++++++- tests/vllm/multimodal/ltx2_video_fixture.h | 14 + tests/vllm/multimodal/test_ltx2_video.cpp | 687 +++++++++++++++++- 13 files changed, 2001 insertions(+), 85 deletions(-) create mode 100644 include/vllm/model_executor/models/ltx2_denoisers.h create mode 100644 src/vllm/model_executor/models/ltx2_denoisers.cpp diff --git a/.agents/specs/ltx25-guided-video.md b/.agents/specs/ltx25-guided-video.md index 88d9c265b..f034c3abe 100644 --- a/.agents/specs/ltx25-guided-video.md +++ b/.agents/specs/ltx25-guided-video.md @@ -190,7 +190,7 @@ schedules. - the combination per modality with that modality's own guider (`:203-204`). Perturbations route through `Ltx2BatchedPerturbationConfig`: one config built -over all N passes (`denuoisers.py:172-176`), then `BatchSlice(i, i+1)` per pass, +over all N passes (`denoisers.py:172-176`), then `BatchSlice(i, i+1)` per pass, then flattened into the `Ltx2DitPerturbation` the forward takes. At batch 1 the slice is the pass's own mask, which is exactly the degeneracy `ltx2.h` already records. diff --git a/CMakeLists.txt b/CMakeLists.txt index d9b5686b2..898640405 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -879,6 +879,12 @@ add_library(vllm STATIC # runs the DiT with `video = nullptr`, and keeping that call in one file is # what makes "does anything else pass a video stream" a readable question. src/vllm/model_executor/models/ltx2_t2a.cpp + # LTX-2.5 (ROW LTX25-GUIDED-VIDEO, issue #1092): `_guided_denoise` — the pass + # assembly four unported pipelines are each blocked on, and the piece the + # VIDEO denoise loop never had. Its own TU because upstream has its own file + # (ltx-pipelines utils/denoisers.py) and because ltx2_pipeline.cpp, where the + # guiders live, is a concurrent-edit lock. + src/vllm/model_executor/models/ltx2_denoisers.cpp # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L6): the quantized loaders — the # FP8 and torchao-NVFP4 DiT arms, the torchao-NVFP4 text encoder, and the # load-time device staging GB10's ATS penalty makes the default. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index d4060676b..def199794 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -175,6 +175,9 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated | | LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 18 mutations, 17 DETECTED (four by review of a conditional-only #1039 gate) and the 18th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only GUIDED arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | | LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 ([#1039](https://github.com/mudler/vllm.cpp/issues/1039)) | +| LTX-2.5 VIDEO guidance | LTX-2.5 DiT, joint video+audio | `test_ltx2_video`'s `ltx2 one_stage:` cases through `Generate`; all FOUR arms carry the x0 invariant and the guider output replays EXACTLY | `--pipeline-kind one_stage` runs `_guided_denoise`: 4 forwards/step, combined per modality in **x0**. Was ONE unguided forward, every `video_guidance` field dead ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) | +| LTX-2.5 cross-attention perturbations | LTX-2.5 DiT | `test_ltx2_video` asserts the isolated-modality pass reached the DiT with both directions off and moved the tensor | `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN` ported, which `modality_scale = 3.0` selects on every video row. The DEVICE forward takes no perturbations, so that pass is refused there by name | +| LTX-2.5 guidance knobs | LTX-2.5 request surface | `test_ltx2_video` renders with an override and refuses one on a fixed recipe | Seven video/audio guider extras mirroring `default_1_stage_arg_parser`, plus a negative embeds pair for a tower-less engine. Refused whole on `distilled_two_stage` and `retake`, whose guidance is distilled in | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | MTP speculation DEPTH (`num_speculative_tokens` > 1) | Qwen3.5/3.6 `mtp.*` heads | k=1..4 through the loader, greedy tokens unmoved, two witnesses per arm: the draft decode forwards the propose RAN, and whether the DELIVERED draft row varied with depth. `test_mtp_depth` 5/5, 63 assertions | Default stays k=1. NO speed claim at k>1. Drafts are proposed and verified, never ACCEPTED, and neither witness proves per-column provenance. Both await the owed DGX gate (#81) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | diff --git a/docs/USAGE.md b/docs/USAGE.md index 2ff0dcc24..55e2e5307 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -914,14 +914,18 @@ ignored — upstream passes a 512x512 placeholder whose height and width it documents as unused, and only the frame count and the recipe's frame rate are read, to derive the duration. -**It is the only GUIDED arm, and that changes what it costs and what it needs.** -The distilled video recipes run one DiT forward per step. This one runs -**three** by default — conditional, unconditional, and one with the audio +**It is a GUIDED arm, and that changes what it costs and what it needs.** The +distilled video recipes run one DiT forward per step. This one runs **three** by +default — conditional, unconditional, and one with the audio self-attention perturbed (STG) — so it is roughly 3x the work per step, and it **requires a text tower**, because the unconditional pass conditions on the negative prompt. Loading with `prompt_embeds_path` alone gets a refusal naming `--audio-cfg-guidance-scale 1.0` as the way to turn the unconditional pass off. +It was the only guided arm here until row LTX25-GUIDED-VIDEO +([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) gave the joint video +path its own denoiser; see *LTX-2.5 video guidance* below. + Six per-generation knobs mirror upstream's own CLI, and each takes the checkpoint generation's value when absent: `--negative-prompt`, `--audio-cfg-guidance-scale` (7.0), `--audio-stg-guidance-scale` (1.0), @@ -948,6 +952,61 @@ at the recipe's own guider values. pipeline: the device forward takes both streams by reference and this pipeline has no video stream to give it. Use `--device cpu`. +### LTX-2.5 video guidance: `--pipeline-kind one_stage` + +`one_stage` mirrors upstream's `TI2VidOneStagePipeline`, which builds a +`FactoryGuidedDenoiser` from the params table's own video and audio guiders. On +the 2.4/2.5 lineage those resolve to `cfg_scale = 3.0`, `stg_scale = 1.0`, +`rescale_scale = 0.7` and `modality_scale = 3.0`. + +Until [#1092](https://github.com/mudler/vllm.cpp/issues/1092) this port read none +of it: the joint denoise loop ran one unguided forward per step. A `one_stage` +render therefore finished, at the right size and frame count, along a different +trajectory than upstream's. It now runs **four** forwards per step and combines +them per modality: + +| Pass | What differs | Selected by | +|---|---|---| +| conditional | nothing | always | +| unconditional | the negative conditioning | `cfg_scale != 1.0` | +| perturbed | video/audio self-attention skipped on `stg_blocks` | `stg_scale != 0.0` | +| isolated modality | the audio<->video cross attention off in every block | `modality_scale != 1.0` | + +Seven per-generation knobs mirror upstream's `default_1_stage_arg_parser` and +each takes the checkpoint generation's value when absent: +`--video-cfg-guidance-scale`, `--video-stg-guidance-scale`, +`--video-rescale-scale`, `--video-skip-step`, `--video-stg-blocks` (comma +separated), `--a2v-guidance-scale` and `--v2a-guidance-scale`. The audio row and +`--negative-prompt` are shared with text-to-audio and are no longer refused on a +video pipeline; upstream's parser carries both rows side by side, and the old +refusal rested on a reading of upstream that was wrong and harmless only while +nothing here read them. + +**The unconditional forward needs a negative conditioning, and there are two +ways to supply one.** With a text tower, `--negative-prompt` (or the recipe's +own default) is encoded through the same chain as the positive prompt. Without +one, `--negative-prompt-embeds` and `--negative-audio-prompt-embeds` are the +negative half of the `prompt_embeds_path` fallback: two files at the DiT's two +cross-attention widths, the same row count as the positive pair. With neither, a +`cfg_scale` other than 1.0 is **refused by name** rather than served the positive +context twice, which would leave the whole classifier-free term at exactly zero. + +**A block index the checkpoint does not have is refused.** `stg_blocks` is a +membership test upstream, so naming block 28 on a model with fewer blocks +perturbs nothing and leaves `stg_scale * (cond - perturbed)` at exactly zero. + +**The distilled and retake recipes refuse every one of these flags.** Their +guidance is distilled into the weights, so honouring an override would sample a +trajectory the weights were never trained for. Their guiders are upstream's +positive-only one, so they still issue one forward per step and their output is +unchanged by this row. + +**The accelerator is refused for the perturbed and isolated-modality passes.** +`Ltx2DitForwardDevice` takes no perturbation argument, so those two passes on +`device = 1` would run an unperturbed forward and leave both terms at zero. +Classifier-free guidance alone is a different context and no perturbation, and +runs on both arms. + **What is not served.** `temporal_upsample_rounds` is defined and refused above `0`: the rounds loop that temporally doubles the latent, re-tiles the canvas and stitches it back is not ported. The refusal names it, and it names three things @@ -2499,6 +2558,8 @@ seam's `prompt_embeds_path`, which carries the video stream), `pipeline_kind` (default `distilled_two_stage`; also `one_stage`, `dmd2`, `dfr`, `retake` and `t2a_one_stage`), `model_version` (only for a checkpoint that declares none), `dit_config_path`, `encoder_config_path`, +`negative_prompt_embeds_path` and `negative_audio_prompt_embeds_path` (the +negative half of the same fallback, for the unconditional forward), `allow_unported_modules`, `max_phase`, `prompt_embeds_valid_rows`, `upsampler_path` and `duration_head_path`. An extra a family does not define is refused, never ignored. One caveat inside that set: `duration_head_path` is diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index 7f4106f49..6d99243e2 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -183,7 +183,19 @@ const char* Need(int argc, char** argv, int i, const char* flag) { "--audio-skip-step and --audio-stg-blocks are upstream\'s own flags; absent, each\n" "takes the checkpoint generation\'s own value. --audio-stg-blocks is comma\n" "separated and a block index outside the DiT\'s layer count is refused rather\n" - "than clamped. The accelerator is REFUSED by name on this pipeline.\n"); + "than clamped. The accelerator is REFUSED by name on this pipeline.\n\n" + "GUIDANCE ON A VIDEO RENDER. --pipeline-kind one_stage runs upstream's guided\n" + "denoiser: FOUR DiT forwards per step (conditional, unconditional, perturbed,\n" + "and one with the audio<->video cross attention off), combined per modality in\n" + "x0 space. --video-cfg-guidance-scale, --video-stg-guidance-scale,\n" + "--video-rescale-scale, --video-skip-step, --video-stg-blocks,\n" + "--a2v-guidance-scale and --v2a-guidance-scale are upstream's own flags and\n" + "take the checkpoint generation's value when absent. The unconditional forward\n" + "needs a NEGATIVE conditioning: either a text tower plus --negative-prompt, or\n" + "--negative-prompt-embeds with --negative-audio-prompt-embeds. Absent both, a\n" + "cfg scale other than 1.0 is refused by name. --pipeline-kind\n" + "distilled_two_stage and retake distil their guidance INTO the weights and\n" + "refuse every one of these flags rather than applying it.\n"); std::exit(code); } @@ -205,6 +217,13 @@ int main(int argc, char** argv) { std::string negative_prompt, audio_cfg_scale, audio_stg_scale, audio_rescale; std::string audio_skip_step, audio_stg_blocks; + // THE VIDEO GUIDER (row LTX25-GUIDED-VIDEO, #1092): the other half of the same + // parser, `default_1_stage_arg_parser` (utils/args.py:947-1010). `--negative- + // prompt` above is shared by both, which is why it is not repeated here. + std::string video_cfg_scale, video_stg_scale, video_rescale, video_skip_step; + std::string video_stg_blocks, a2v_scale, v2a_scale; + std::string negative_embeds, negative_audio_embeds; + // The extras are BORROWED by the load call, so the strings must outlive it. // Kept as two parallel vectors of owned strings plus the char* views the ABI // takes, built once after parsing. @@ -238,6 +257,13 @@ int main(int argc, char** argv) { else if (f == "--model-version") SetExtra("model_version", Need(argc, argv, ++i, f.c_str())); else if (f == "--pipeline-kind") SetExtra("pipeline_kind", Need(argc, argv, ++i, f.c_str())); else if (f == "--upsampler") SetExtra("upsampler_path", Need(argc, argv, ++i, f.c_str())); + else if (f == "--negative-prompt-embeds") { + negative_embeds = Need(argc, argv, ++i, f.c_str()); + SetExtra("negative_prompt_embeds_path", negative_embeds); + } else if (f == "--negative-audio-prompt-embeds") { + negative_audio_embeds = Need(argc, argv, ++i, f.c_str()); + SetExtra("negative_audio_prompt_embeds_path", negative_audio_embeds); + } // Kept although the library REFUSES this extra by name (#611): the duration // head is unported, and forwarding the flag gets the caller that named // refusal instead of "unknown option", which says nothing about why. @@ -295,6 +321,20 @@ int main(int argc, char** argv) { audio_skip_step = Need(argc, argv, ++i, "--audio-skip-step"); else if (f == "--audio-stg-blocks") audio_stg_blocks = Need(argc, argv, ++i, "--audio-stg-blocks"); + else if (f == "--video-cfg-guidance-scale") + video_cfg_scale = Need(argc, argv, ++i, "--video-cfg-guidance-scale"); + else if (f == "--video-stg-guidance-scale") + video_stg_scale = Need(argc, argv, ++i, "--video-stg-guidance-scale"); + else if (f == "--video-rescale-scale") + video_rescale = Need(argc, argv, ++i, "--video-rescale-scale"); + else if (f == "--video-skip-step") + video_skip_step = Need(argc, argv, ++i, "--video-skip-step"); + else if (f == "--video-stg-blocks") + video_stg_blocks = Need(argc, argv, ++i, "--video-stg-blocks"); + else if (f == "--a2v-guidance-scale") + a2v_scale = Need(argc, argv, ++i, "--a2v-guidance-scale"); + else if (f == "--v2a-guidance-scale") + v2a_scale = Need(argc, argv, ++i, "--v2a-guidance-scale"); else if (f == "--regenerate-video") regen_video = Need(argc, argv, ++i, "--regenerate-video"); else if (f == "--regenerate-audio") @@ -376,7 +416,17 @@ int main(int argc, char** argv) { std::make_pair("audio_stg_guidance_scale", &audio_stg_scale), std::make_pair("audio_rescale_scale", &audio_rescale), std::make_pair("audio_skip_step", &audio_skip_step), - std::make_pair("audio_stg_blocks", &audio_stg_blocks)}) { + std::make_pair("audio_stg_blocks", &audio_stg_blocks), + // THE VIDEO GUIDER (#1092). Per-generation for the same + // reason the audio row is, and refused whole on a recipe + // whose guidance is distilled into the weights. + std::make_pair("video_cfg_guidance_scale", &video_cfg_scale), + std::make_pair("video_stg_guidance_scale", &video_stg_scale), + std::make_pair("video_rescale_scale", &video_rescale), + std::make_pair("video_skip_step", &video_skip_step), + std::make_pair("video_stg_blocks", &video_stg_blocks), + std::make_pair("a2v_guidance_scale", &a2v_scale), + std::make_pair("v2a_guidance_scale", &v2a_scale)}) { if (kv.second->empty()) continue; gen_keys.emplace_back(kv.first); gen_values.push_back(*kv.second); diff --git a/include/vllm/model_executor/models/ltx2.h b/include/vllm/model_executor/models/ltx2.h index 998cda7c9..962564daa 100644 --- a/include/vllm/model_executor/models/ltx2.h +++ b/include/vllm/model_executor/models/ltx2.h @@ -39,25 +39,34 @@ // message naming the missing phase rather than silently computing in f32. // // NOT PORTED IN L2, recorded here so it cannot be discovered later: -// - The CROSS-attention guidance perturbations only, as of row -// LTX25-T2A-ONE-STAGE (#1005): `SKIP_A2V_CROSS_ATTN` and -// `SKIP_V2A_CROSS_ATTN` (guidance/perturbations.py:8-16, -// transformer.py:330-397 `cross_attn_skip_all`). Nothing upstream that this -// port serves constructs them — STG is built from `stg_blocks` and reaches -// the SELF-attention types alone (utils/constants.py:49-68 through -// guiders.py:194-211) — and `Ltx2Attention` refuses the flag on a cross call -// by name rather than applying the self-attention rule to it. +// - PORTED 2026-08-17 by row LTX25-GUIDED-VIDEO (#1092): the CROSS-attention +// guidance perturbations, `SKIP_A2V_CROSS_ATTN` and `SKIP_V2A_CROSS_ATTN` +// (guidance/perturbations.py:8-16, transformer.py:335,366 +// `cross_attn_skip_all`). They are `Ltx2DitPerturbation`'s two booleans. // -// THE SELF-ATTENTION HALF IS NOW PORTED, and this entry said the whole -// mechanism was unported until #1005. `Ltx2DitPerturbation` is upstream's -// `perturbations` argument (model.py:492) at the one batch size this port -// serves, and `Ltx2AttentionArgs::all_perturbed` is +// WHY THE PREVIOUS ENTRY WAS WRONG RATHER THAN MERELY STALE. It refused them +// on the ground that "nothing upstream that this port serves constructs +// them — STG is built from `stg_blocks` and reaches the SELF-attention types +// alone". STG does. The isolated-modality pass does not: `_guided_denoise` +// builds BOTH cross types with `blocks=None` whenever either guider has +// `modality_scale != 1.0` (denoisers.py:121-137, guiders.py:283-285). Every +// VIDEO row of the params table sets it to 3.0 +// (utils/constants.py:40-80), so this was upstream's default on the video +// path the whole time. The sentence was true of text-to-audio, which pins +// the field to 1.0 (t2a_one_stage.py:200-202), and it was written while +// text-to-audio was the only guided path here. +// +// THE SELF-ATTENTION HALF was ported by #1005. `Ltx2DitPerturbation` is +// upstream's `perturbations` argument (model.py:492) at the one batch size +// this port serves, and `Ltx2AttentionArgs::all_perturbed` is // `use_attention = not all_perturbed` (attention.py:557). `nullptr` remains -// upstream's `perturbations=None` path (model.py:509-511) and is what every -// caller but text-to-audio passes. +// upstream's `perturbations=None` path (model.py:509-511) and is what an +// unguided phase still passes. // // The BATCHED form (`BatchedPerturbationConfig`, perturbations.py:53-143, -// indexed [type, block, SAMPLE]) is still unported, and so is the partial +// indexed [type, block, SAMPLE]) is ported in `ltx2_pipeline.h` and reached +// by `Ltx2GuidedDenoise`, which builds one config over the pass list and +// slices it per pass. What is still unported is batch > 1 and the partial // blend it exists for (`out * mask + v * (1 - mask)`, attention.py:571-572). // Both are degenerate at `Ltx2ModalityInput::batch == 1`, which is the only // batch any path here runs. @@ -541,21 +550,39 @@ struct Ltx2DitOutputs { // indexed [type, block, SAMPLE]; this is indexed [block] alone, because // `Ltx2ModalityInput::batch` is 1 on every path here and the sample axis is a // degenerate one. `Ltx2BatchedPerturbationConfig` (ltx2_pipeline.h) is the -// batched form and stays ungated by any product caller — recorded as owed in -// .agents/specs/ltx25-t2a-one-stage.md rather than silently bypassed. +// batched form, and row LTX25-GUIDED-VIDEO (#1092) is what gave it a product +// caller: `Ltx2GuidedDenoise` builds one config over all four passes and slices +// it per pass, which is `denoisers.py:172-176` and is where the ONE-sample +// flattening below happens. // // EMPTY IS NOT "NOTHING PERTURBED BY COINCIDENCE": a vector of the wrong length // is REFUSED, so a config built for a different layer count cannot silently // perturb the first N blocks and leave the rest alone. // -// The two CROSS-attention perturbation types upstream defines -// (SKIP_A2V_CROSS_ATTN, SKIP_V2A_CROSS_ATTN, perturbations.py:8-16) have no -// field here and are not ported. Nothing upstream that this port serves -// constructs them: STG is built from `stg_blocks` and reaches the self-attention -// types only (utils/constants.py:49-68 through guiders.py:194-211). +// ALL FOUR upstream perturbation types are represented here. The two CROSS +// directions arrived with #1092 and are booleans rather than per-block vectors, +// because the one thing that builds them asks for ALL blocks +// (`Perturbation(type=..., blocks=None)`, denoisers.py:132-135) and upstream's +// own reader is the per-block scalar `cross_attn_skip_all` (transformer.py:335, +// :366) rather than a mask multiply. A per-block cross vector would be a surface +// with no constructor. +// +// WHAT THIS ENTRY USED TO SAY, kept because the sentence was load-bearing and +// wrong: "Nothing upstream that this port serves constructs them: STG is built +// from `stg_blocks` and reaches the self-attention types only". That was true +// while text-to-audio was the only guided path here — it pins +// `modality_scale = 1.0` (t2a_one_stage.py:200-202), which is exactly the value +// `do_isolated_modality_generation` reads as OFF. Every VIDEO row defaults it to +// 3.0 (utils/constants.py:40-80), so the isolated-modality pass is upstream's +// DEFAULT there and these two types are on the reachable path. struct Ltx2DitPerturbation { std::vector video_self_attn; // [num_layers], empty = none std::vector audio_self_attn; // [num_layers], empty = none + // `cross_attn_skip_all` on the VIDEO args, i.e. SKIP_A2V_CROSS_ATTN: the + // audio-to-video direction, which WRITES the video stream. + bool video_cross_attn_skip_all = false; + // `cross_attn_skip_all` on the AUDIO args, i.e. SKIP_V2A_CROSS_ATTN. + bool audio_cross_attn_skip_all = false; }; // LTXModel.forward (model.py:492-538), plus the preprocessors it drives @@ -625,6 +652,16 @@ struct Ltx2BlockArgs { // STG for THIS block (attention.py:552-577). See `Ltx2DitPerturbation`. bool video_self_attn_perturbed = false; bool audio_self_attn_perturbed = false; + // `cross_attn_skip_all` (transformer_args.py:118, read at transformer.py:335 + // and :366). THE FLAG RIDES ON THE STREAM BEING WRITTEN, not on the stream + // being read: `video.cross_attn_skip_all` skips A2V, which writes the VIDEO + // stream from audio keys, and `audio.cross_attn_skip_all` skips V2A. Swapping + // them still renders — both directions are off on the one pass that sets + // either, because `_guided_denoise` builds the isolated-modality pass with + // BOTH (denoisers.py:130-136) — so this is one of the places a comment has to + // carry what a test on the shipped path cannot separate. + bool video_cross_attn_skip_all = false; + bool audio_cross_attn_skip_all = false; // Audio<->video cross-attention AdaLN inputs (transformer_args.py:388-411). const float* video_cross_scale_shift = nullptr; // [batch, video tokens, 4 * dim] const float* video_cross_gate = nullptr; // [batch, 1, dim] diff --git a/include/vllm/model_executor/models/ltx2_denoisers.h b/include/vllm/model_executor/models/ltx2_denoisers.h new file mode 100644 index 000000000..1bd50c3d6 --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_denoisers.h @@ -0,0 +1,179 @@ +// LTX-2.5 denoisers — `ltx-pipelines/utils/denoisers.py` @ Lightricks/LTX-2 +// fd4ded7f, in its own translation unit because upstream has its own file. +// +// Row LTX25-GUIDED-VIDEO, issue +// https://github.com/mudler/vllm.cpp/issues/1092. Spec +// .agents/specs/ltx25-guided-video.md. +// +// ── WHAT THIS FILE IS FOR ────────────────────────────────────────────────── +// +// Upstream has three denoisers (`SimpleDenoiser`, `GuidedDenoiser`, +// `FactoryGuidedDenoiser`) and they share ONE function: `_guided_denoise` +// (denoisers.py:62-207). That function is what this file ports. It is the piece +// four unported pipelines are each blocked on — `a2vid_two_stage.py:230`, +// `ti2vid_two_stages.py:248`, `ti2vid_two_stages_hq.py:271`, +// `keyframe_interpolation.py:232` — and it is what a `pipeline_kind = one_stage` +// render here was missing entirely: `ti2vid_one_stage.py:221-226` builds a +// `FactoryGuidedDenoiser` and this port ran one unguided forward per step. +// +// ── THE FOUR THINGS THAT ARE EASY TO GET WRONG AND STILL RENDER ──────────── +// +// 1. THE SPACE. The transformer this seam is handed is upstream's `X0Model` +// (built at blocks.py:480-482, forward at model.py:590-604), so every pass it +// returns is ALREADY `latent - sigma * velocity` and the guider combines +// denoised tensors. Combining velocities and converting once afterwards is a +// DIFFERENT function whenever `rescale_scale != 0` (guiders.py:268-271), and +// it is 0.7 on every video row of the params table. That defect shipped on +// the audio arm of this tree and is #1039. This seam cannot commit it, +// because it never sees a velocity: the conversion lives in the caller's +// `Ltx2X0Model`, which is where upstream puts it. +// +// 2. ONE PASS LIST, TWO GUIDERS. `_guided_denoise` takes the UNION of what the +// two guiders want — one `uncond` pass if either asks (`:102-109`), one `ptb` +// pass carrying both modalities' `stg_blocks` (`:111-119`), one `mod` pass if +// either asks (`:121-137`) — and then combines each modality with its OWN +// guider over the same splits (`:203-204`). Running a per-modality pass list +// instead would issue up to six forwards where upstream issues four, and +// would hand the audio stream a different video state to cross-attend to on +// the video-only passes. Both renders finish. +// +// 3. `post_process_latent` IS NOT PART OF THE DENOISER. It is applied by the +// LOOP, to the guider's OUTPUT (`samplers.py:35`, `:484`), not to each arm on +// the way out of the forward. Pinning the conditioned tokens per arm makes +// every arm agree on those tokens, which silently zeroes the guidance delta +// exactly where a keyframe or a reference clip is conditioning. This seam +// therefore returns the raw guided prediction and the caller post-processes. +// +// 4. A SKIPPED STEP RUNS NO FORWARD. When both guiders skip, upstream returns +// the PREVIOUS step's denoised pair (`:87-90`) rather than running the +// conditional pass and using it. `skip_step` is 0 in every params table, so +// this is reachable only through an explicit request; it is ported because +// the request surface exposes it. +// +// ── WHAT IS NOT HERE ─────────────────────────────────────────────────────── +// +// The BATCHING. Upstream concatenates the passes along the batch axis and issues +// ONE transformer call (`:141-186`); this issues one call per pass. That is a +// throughput difference and not a numerical one at `batch == 1`, and it is the +// same adaptation `Ltx2DitPerturbation` already records: every path in this port +// runs `Ltx2ModalityInput::batch == 1`. + +#pragma once + +#include +#include +#include + +#include "vllm/model_executor/models/ltx2.h" +#include "vllm/model_executor/models/ltx2_pipeline.h" + +namespace vllm { + +// What one pass of the `X0Model` returns. `Ltx2DitOutputs` carries the raw +// velocity; this carries the denoised prediction AND the velocity it came from, +// because "which space was this combined in" is an arithmetic question between +// three tensors and cannot be answered from the denoised one alone. The velocity +// is what the trace records per arm and what the gate's per-arm invariant +// `x0 == latent - sigma * velocity` is checked against. +struct Ltx2X0Outputs { + std::vector video; // `to_denoised(video.latent, vx, video.timesteps)` + std::vector audio; // `to_denoised(audio.latent, ax, audio.timesteps)` + std::vector video_velocity; // the DiT's own output, before the conversion + std::vector audio_velocity; +}; + +// `X0Model` (model.py:590-604), supplied by the caller. +// +// It is a callable and not a (params, weights) pair on purpose. Upstream's +// `_guided_denoise(transformer, ...)` takes the model the same way, the host and +// the device forward are two lambdas over one seam, and the four pipelines that +// will use this next supply their own conditioning without this file learning +// anything about keyframes, reference clips or two-stage schedules. +// +// EITHER STREAM MAY BE NULL, which is upstream's absent modality. `perturbations` +// is null on the passes that have none, which is upstream's +// `PerturbationConfig.empty()` reaching `model.py:509-511`. +using Ltx2X0Model = std::function; + +// The pass names of `_guided_denoise` (`:100-137`), in the order it appends +// them. The order is not cosmetic: it is the batch order upstream splits back +// out at `:188-190`, and it is the order the perturbation config is built in. +enum class Ltx2DenoisePass { + kCond = 0, + kUncond = 1, + kPerturbed = 2, + kModality = 3, +}; +inline constexpr int64_t kLtx2DenoisePassCount = 4; + +struct Ltx2GuidedDenoiseInputs { + // The two streams, exactly as they would be handed to the forward for the + // conditional pass. The seam copies them per pass and overrides `context` and + // `enabled`; it never touches the latent, the timesteps or the positions. + const Ltx2ModalityInput* video = nullptr; + const Ltx2ModalityInput* audio = nullptr; + + // `guider.negative_context` (guiders.py:236). A null is upstream's + // `negative_context is None`, whose branch at `:107-108` falls back to the + // POSITIVE context rather than refusing — and which `:104-106` refuses when + // that modality's guider is the one asking. Both are mirrored. + const float* video_negative_context = nullptr; + const float* audio_negative_context = nullptr; + + Ltx2MultiModalGuiderParams video_guider; + Ltx2MultiModalGuiderParams audio_guider; + + // `transformer.num_blocks` (denoisers.py:180). Needed to size the perturbation + // masks, and refused when it disagrees with the DiT the caller's lambda drives + // — a mask built for another block count perturbs a prefix and renders. + int64_t num_blocks = 0; + + int64_t step_index = 0; + // `force_uncond_pass` (`:76`, read at `:102-103`). `RetakePipeline` is the one + // upstream caller that sets it (retake.py:305-311). + bool force_uncond_pass = false; + + // `_last_denoised_video` / `_last_denoised_audio` (denoisers.py:274-275). Null + // on the first step. A skipped step with nothing to reuse is refused rather + // than reading an empty vector. + const std::vector* last_denoised_video = nullptr; + const std::vector* last_denoised_audio = nullptr; +}; + +// Everything one call produced. The per-pass tensors are kept because a gate +// that can only see the combination cannot tell which arm was converted in which +// space, and because #1039's first gate covered one arm out of three and three +// mutations survived it. +struct Ltx2GuidedDenoiseResult { + // `DenoisedLatentResult.denoised` (utils/types.py), per modality. Empty when + // that modality was absent. + std::vector video_denoised; + std::vector audio_denoised; + bool video_skipped = false; + bool audio_skipped = false; + + // Which passes ran, and what each returned. Indexed by `Ltx2DenoisePass`. + bool pass_ran[kLtx2DenoisePassCount] = {false, false, false, false}; + std::vector video_pass[kLtx2DenoisePassCount]; // x0 + std::vector audio_pass[kLtx2DenoisePassCount]; + std::vector video_pass_velocity[kLtx2DenoisePassCount]; // raw + std::vector audio_pass_velocity[kLtx2DenoisePassCount]; + + // The blocks the perturbed pass actually asked the DiT to skip, read off the + // mask that was handed over rather than copied from the guider params. A + // config that is BUILT and not HANDED OVER is invisible in the params. + std::vector perturbed_video_blocks; + std::vector perturbed_audio_blocks; + // Whether the isolated-modality pass reached the DiT with BOTH cross + // directions off, observed at the call for the same reason. + bool modality_pass_skipped_a2v = false; + bool modality_pass_skipped_v2a = false; +}; + +// `_guided_denoise` (denoisers.py:62-207). +Ltx2GuidedDenoiseResult Ltx2GuidedDenoise(const Ltx2X0Model& transformer, + const Ltx2GuidedDenoiseInputs& in); + +} // namespace vllm diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index fe3007e6c..c038a96a8 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -459,6 +459,53 @@ inline constexpr char kLtx2AudioRescaleScaleExtra[] = "audio_rescale_scale"; inline constexpr char kLtx2AudioSkipStepExtra[] = "audio_skip_step"; inline constexpr char kLtx2AudioStgBlocksExtra[] = "audio_stg_blocks"; +// THE VIDEO GUIDER, row LTX25-GUIDED-VIDEO (#1092). The same row of flags on the +// other stream, from the same parser (`default_1_stage_arg_parser`, +// utils/args.py:947-1010). ABSENT MEANS the params table's own value: 3.0 / 1.0 / +// 0.7 / 3.0 and block 28 on the 2.3-and-later lineage +// (utils/constants.py:40-88). +// +// THE MODALITY KNOBS EXIST HERE AND NOT ON THE T2A ROW ABOVE, and the asymmetry +// is upstream's rather than an oversight on either side. Text-to-audio has no +// video stream, so `t2a_one_stage.py:200-202` pins `modality_scale = 1.0` and +// exposes no flag. A joint render has both streams and the parser exposes +// `--a2v-guidance-scale` and `--v2a-guidance-scale`, which are the video and +// audio guiders' `modality_scale` respectively (utils/args.py:987-996 and its +// audio counterpart). Reaching either turns on a fourth DiT forward per step. +// +// EVERY ONE OF THESE IS REFUSED on a phase whose recipe sets +// `allow_guidance_override = false` — the distilled two-stage and retake +// recipes, whose guidance is distilled INTO the weights. Honouring an override +// there would sample a trajectory the weights were never trained for, which is +// the same argument `fixed_num_inference_steps` already makes for the schedule. +inline constexpr char kLtx2VideoCfgScaleExtra[] = "video_cfg_guidance_scale"; +inline constexpr char kLtx2VideoStgScaleExtra[] = "video_stg_guidance_scale"; +inline constexpr char kLtx2VideoRescaleScaleExtra[] = "video_rescale_scale"; +inline constexpr char kLtx2VideoSkipStepExtra[] = "video_skip_step"; +inline constexpr char kLtx2VideoStgBlocksExtra[] = "video_stg_blocks"; +inline constexpr char kLtx2A2vGuidanceScaleExtra[] = "a2v_guidance_scale"; +inline constexpr char kLtx2V2aGuidanceScaleExtra[] = "v2a_guidance_scale"; + +// THE NEGATIVE CONDITIONING FOR AN ENGINE WITH NO TEXT TOWER, and a LOCAL +// ADAPTATION recorded as one. +// +// Upstream has no embeds surface at all: every pipeline encodes +// `[prompt, negative_prompt]` in ONE `PromptEncoder` call +// (ti2vid_one_stage.py:170-178) and takes `.video_encoding` / `.audio_encoding` +// from each half. `prompt_embeds_path` and the `audio_prompt_embeds_path` extra +// are this port's own affordance for running the DiT without a 12B tower; these +// two are the SAME affordance applied to the second of upstream's two +// encodings, not a new concept. +// +// They are supplied together with each other, and only alongside the positive +// pair. Without them and without a tower, a guider that asks for the +// unconditional pass is REFUSED BY NAME rather than served the positive context +// twice — which would make `(cfg_scale - 1) * (cond - uncond)` identically zero +// and produce an unguided render wearing a guided render's configuration. +inline constexpr char kLtx2NegativePromptEmbedsExtra[] = "negative_prompt_embeds_path"; +inline constexpr char kLtx2NegativeAudioPromptEmbedsExtra[] = + "negative_audio_prompt_embeds_path"; + // WHAT THE LAST `Generate()` ACTUALLY HANDED THE DiT's CROSS-ATTENTION. // // Every field is read off the exact f32 buffers `Ltx2ModalityInput::context` @@ -773,6 +820,65 @@ struct Ltx2ConditioningTrace { std::vector t2a_first_next_latent; double t2a_first_sigma = 0.0; + // ── the GUIDED VIDEO denoise, row LTX25-GUIDED-VIDEO (#1092) ────────────── + // + // Everything the FIRST step of the FIRST phase produced, and nothing else. One + // step decides every question below, and recording every step would hold a + // whole real trajectory in memory. + // + // WHY THE VELOCITIES SIT BESIDE THE X0 TENSORS. "Which space was this combined + // in" is an equation between three tensors — `x0 == latent - sigma*velocity` — + // and cannot be answered from the x0 tensor alone. It is exact in x0 space and + // off by the whole sample in velocity space, which is what makes it a gate + // rather than a tolerance. Recorded PER ARM, because a claim about "every + // pass" made from one recorded pass is a claim about a quarter of them: + // #1039's first gate covered the conditional arm alone and three mutations + // survived it. + bool video_guided = false; + int64_t video_cond_forwards = 0; + int64_t video_uncond_forwards = 0; + int64_t video_perturbed_forwards = 0; + int64_t video_modality_forwards = 0; + // Read off the mask handed to the DiT, not copied from the guider params: a + // perturbation that is BUILT and not HANDED OVER leaves the params untouched + // and the render finite. + std::vector video_perturbed_blocks; + std::vector video_audio_perturbed_blocks; + bool video_modality_skipped_a2v = false; + bool video_modality_skipped_v2a = false; + // The guidance phase 0 resolved, after the request overrides. The gate replays + // `Ltx2MultiModalGuidance` over the recorded arms with these, so a build that + // resolved different params fails the replay instead of agreeing with itself. + double video_guidance_cfg_scale = 0.0; + double video_guidance_stg_scale = 0.0; + double video_guidance_rescale_scale = 0.0; + double video_guidance_modality_scale = 0.0; + std::vector video_first_latent; + std::vector video_first_cond_velocity; + std::vector video_first_cond; + std::vector video_first_uncond_velocity; + std::vector video_first_uncond; + std::vector video_first_perturbed_velocity; + std::vector video_first_perturbed; + std::vector video_first_modality_velocity; + std::vector video_first_modality; + // The guider's output BEFORE `post_process_latent`, which is what + // `Ltx2MultiModalGuidance` returned and what the replay must reproduce + // exactly. + std::vector video_first_denoised; + // And AFTER it, which is what the stepper was handed. Two fields rather than + // one, because `post_process_latent` is the identity whenever no token is + // conditioned and a single field could not say which of the two a build passed + // on. + std::vector video_first_stepper_input; + std::vector video_first_next_latent; + double video_first_sigma = 0.0; + // The PER-TOKEN timesteps step 0 ran at, so the invariant is checked with the + // same sigma `ToDenoised` used rather than with the schedule scalar — they + // differ exactly where a token is conditioned, which is where getting it wrong + // re-noises a keyframe. + std::vector video_first_timesteps; + // True only once the `Generate` that produced this conditioning RETURNED. The // trace is filled immediately after the connector and BEFORE the denoise loop, // because that is the only point at which the exact buffers cross-attention diff --git a/src/vllm/model_executor/models/ltx2_denoisers.cpp b/src/vllm/model_executor/models/ltx2_denoisers.cpp new file mode 100644 index 000000000..b73a9f94f --- /dev/null +++ b/src/vllm/model_executor/models/ltx2_denoisers.cpp @@ -0,0 +1,357 @@ +// `_guided_denoise` (ltx-pipelines utils/denoisers.py:62-207) at +// Lightricks/LTX-2 @ fd4ded7f. See ltx2_denoisers.h for the four things that +// fail silently if guessed. +// +// Row LTX25-GUIDED-VIDEO, issue #1092. + +#include "vllm/model_executor/models/ltx2_denoisers.h" + +#include +#include +#include +#include + +namespace vllm { +namespace { + +[[noreturn]] void Fail(const std::string& why) { + throw std::runtime_error("ltx2 guided denoise: " + why); +} + +// `_POSITIVE_ONLY_GUIDER` (denoisers.py:25-28) is +// `MultiModalGuiderParams(cfg_scale=1.0, stg_scale=0.0, modality_scale=1.0)`, +// which is `Ltx2MultiModalGuiderParams`'s own default construction. Stated as a +// function rather than inlined so the identity is checkable by eye against +// `_ensure_guider` (`:31-33`). +Ltx2MultiModalGuiderParams PositiveOnlyGuider() { return Ltx2MultiModalGuiderParams{}; } + +// `perturbations.mask(type, block)` collapsed to this port's one sample. The +// KEEP polarity is upstream's — 1 keeps, 0 perturbs (perturbations.py:53-56) — +// so the DiT flag, which is `all_perturbed`, is the negation. +bool PerturbedAt(const Ltx2BatchedPerturbationConfig& config, Ltx2PerturbationType type, + int64_t block, int64_t sample) { + const std::vector mask = config.Mask(type, block); + return mask[static_cast(sample)] == 0; +} + +} // namespace + +Ltx2GuidedDenoiseResult Ltx2GuidedDenoise(const Ltx2X0Model& transformer, + const Ltx2GuidedDenoiseInputs& in) { + if (in.video == nullptr && in.audio == nullptr) { + // `transformer.py:259-260` refuses it upstream, one level lower. + Fail("both modalities are null; at least one of `video` or `audio` must be provided"); + } + if (in.num_blocks < 1) { + Fail("`num_blocks` is " + std::to_string(in.num_blocks) + + "; the perturbation masks are sized by it (denoisers.py:180) and a wrong count " + "perturbs a prefix of the blocks and renders"); + } + + // `_ensure_guider` (denoisers.py:31-33): an ABSENT modality takes the + // positive-only guider, so its `calculate` returns `cond` unchanged and it asks + // for no extra pass. A modality that is PRESENT keeps its caller's guider even + // when every scale is at its no-op value. + const Ltx2MultiModalGuiderParams video_guider = + in.video != nullptr ? in.video_guider : PositiveOnlyGuider(); + const Ltx2MultiModalGuiderParams audio_guider = + in.audio != nullptr ? in.audio_guider : PositiveOnlyGuider(); + + Ltx2GuidedDenoiseResult result; + + // `should_skip_step` (denoisers.py:84-85). + const bool v_skip = video_guider.ShouldSkipStep(in.step_index); + const bool a_skip = audio_guider.ShouldSkipStep(in.step_index); + result.video_skipped = v_skip; + result.audio_skipped = a_skip; + + // `if v_skip and a_skip` (`:87-90`) — NO FORWARD AT ALL. Running the + // conditional pass and using it is the plausible reading of "skip the + // guidance" and is a whole DiT forward per skipped step and a different + // trajectory, on a render that finishes either way. + auto reuse = [&](const std::vector* last, const char* which) { + if (last == nullptr || last->empty()) { + Fail(std::string("step ") + std::to_string(in.step_index) + + " skips the " + which + + " guider and no earlier step produced a denoised prediction to reuse. " + "`should_skip_step` is `step % (skip_step + 1) != 0` (guiders.py:287-291), which is " + "false at step 0, so this is unreachable through the request surface and is a defect " + "rather than a bad request"); + } + return *last; + }; + if (v_skip && a_skip) { + if (in.video != nullptr) result.video_denoised = reuse(in.last_denoised_video, "video"); + if (in.audio != nullptr) result.audio_denoised = reuse(in.last_denoised_audio, "audio"); + return result; + } + + // ── the pass list (denoisers.py:97-137) ─────────────────────────────────── + // + // ONE list for BOTH modalities, and the union of what the two guiders want. + // See ltx2_denoisers.h item 2 for the render a per-modality list produces. + struct Pass { + Ltx2DenoisePass kind; + const float* video_context; + const float* audio_context; + Ltx2PerturbationConfig perturbation; + }; + std::vector passes; + + const float* v_context = in.video != nullptr ? in.video->context : nullptr; + const float* a_context = in.audio != nullptr ? in.audio->context : nullptr; + if (in.video != nullptr && v_context == nullptr) { + Fail("v_context is required when video_state is provided (denoisers.py:92-93)"); + } + if (in.audio != nullptr && a_context == nullptr) { + Fail("a_context is required when audio_state is provided (denoisers.py:94-95)"); + } + passes.push_back({Ltx2DenoisePass::kCond, v_context, a_context, Ltx2PerturbationConfig{}}); + + // `:102-109`. `force_uncond_pass` adds the pass for a modality that is PRESENT + // even when its own guider does not ask (retake.py:305-311 is the one upstream + // caller that sets it). + const bool v_needs_neg = video_guider.DoUnconditionalGeneration() || + (in.force_uncond_pass && in.video != nullptr); + const bool a_needs_neg = audio_guider.DoUnconditionalGeneration() || + (in.force_uncond_pass && in.audio != nullptr); + if (v_needs_neg || a_needs_neg) { + if (v_needs_neg && in.video_negative_context == nullptr) { + Fail("negative context is required for unconditioned denoising on the VIDEO stream " + "(denoisers.py:104-105). `do_unconditional_generation` is " + "`not isclose(cfg_scale, 1.0)` (guiders.py:275-277), so either supply the negative " + "conditioning or set the video cfg scale to 1.0"); + } + if (a_needs_neg && in.audio_negative_context == nullptr) { + Fail("negative context is required for unconditioned denoising on the AUDIO stream " + "(denoisers.py:106-107). `do_unconditional_generation` is " + "`not isclose(cfg_scale, 1.0)` (guiders.py:275-277), so either supply the negative " + "conditioning or set the audio cfg scale to 1.0"); + } + // `:108-109` — a modality with no negative context falls back to its POSITIVE + // one rather than being dropped from the pass. That is not a defensive + // default: it is how a pass forced for the OTHER modality still carries a + // legal context for this one. + passes.push_back({Ltx2DenoisePass::kUncond, + in.video_negative_context != nullptr ? in.video_negative_context : v_context, + in.audio_negative_context != nullptr ? in.audio_negative_context : a_context, + Ltx2PerturbationConfig{}}); + } + + // `:111-119`. ONE perturbed pass carrying BOTH modalities' blocks. + { + Ltx2PerturbationConfig stg; + if (video_guider.DoPerturbedGeneration()) { + Ltx2Perturbation p; + p.type = Ltx2PerturbationType::kSkipVideoSelfAttn; + p.blocks = video_guider.stg_blocks; + stg.perturbations.push_back(std::move(p)); + } + if (audio_guider.DoPerturbedGeneration()) { + Ltx2Perturbation p; + p.type = Ltx2PerturbationType::kSkipAudioSelfAttn; + p.blocks = audio_guider.stg_blocks; + stg.perturbations.push_back(std::move(p)); + } + // A BLOCK LIST THAT MISSES EVERY BLOCK IS A WASTED FORWARD AND A ZERO TERM. + // `Perturbation.is_perturbed` is `block in self.blocks` + // (perturbations.py:26-33), so `stg_blocks = [28]` on a model with fewer + // blocks perturbs nothing: the perturbed pass returns the conditional pass's + // own tensor and `stg_scale * (cond - perturbed)` is exactly zero. The render + // is finite, the right size, and carries no spatio-temporal guidance at all. + // Upstream never meets this because it only ever runs 48-block checkpoints; + // this port runs reduced ones, and a smaller checkpoint is a legal thing to + // hand it. + const auto check_reaches_a_block = [&](const Ltx2MultiModalGuiderParams& guider, + const char* which) { + if (!guider.DoPerturbedGeneration()) return; + for (const int64_t block : guider.stg_blocks) { + if (block >= 0 && block < in.num_blocks) return; + } + Fail(std::string("the ") + which + " STG scale is " + std::to_string(guider.stg_scale) + + " and none of its " + std::to_string(guider.stg_blocks.size()) + + " stg_blocks is in range for this DiT's " + std::to_string(in.num_blocks) + + " blocks, so the perturbed forward would be identical to the conditional one and " + "`stg_scale * (cond - perturbed)` would be exactly zero (guiders.py:264). Name blocks " + "this checkpoint has, or set the STG scale to 0.0"); + }; + if (!stg.perturbations.empty()) { + check_reaches_a_block(video_guider, "video"); + check_reaches_a_block(audio_guider, "audio"); + passes.push_back({Ltx2DenoisePass::kPerturbed, v_context, a_context, std::move(stg)}); + } + } + + // `:121-137`. The isolated-modality pass: BOTH cross directions, ALL blocks + // (`blocks=None`), when EITHER guider isolates. `modality_scale` is 3.0 on + // every video row of the params table (utils/constants.py:40-80), so this is + // the default arm rather than a corner. + if (video_guider.DoIsolatedModalityGeneration() || + audio_guider.DoIsolatedModalityGeneration()) { + Ltx2PerturbationConfig mod; + Ltx2Perturbation a2v; + a2v.type = Ltx2PerturbationType::kSkipA2vCrossAttn; + a2v.all_blocks = true; + Ltx2Perturbation v2a; + v2a.type = Ltx2PerturbationType::kSkipV2aCrossAttn; + v2a.all_blocks = true; + mod.perturbations.push_back(std::move(a2v)); + mod.perturbations.push_back(std::move(v2a)); + passes.push_back({Ltx2DenoisePass::kModality, v_context, a_context, std::move(mod)}); + } + + // ── the perturbation config (denoisers.py:172-186) ─────────────────────── + // + // ONE batched config over the whole pass list, then one sample slice per pass, + // which is upstream's `batched_ptb_configs` followed by the per-sample mask the + // block reads. Building a fresh single-sample config per pass would be + // arithmetically identical and would leave `Ltx2BatchedPerturbationConfig` — + // the shared seam that mirrors `BatchedPerturbationConfig` — with no product + // caller, which is the defect #1049 records. + const int64_t pass_count = static_cast(passes.size()); + std::vector configs; + configs.reserve(passes.size()); + for (const Pass& p : passes) configs.push_back(p.perturbation); + const Ltx2BatchedPerturbationConfig batched(configs, in.num_blocks); + + // ── run the passes (`:186`, one call there, `pass_count` calls here) ─────── + for (int64_t index = 0; index < pass_count; ++index) { + const Pass& pass = passes[static_cast(index)]; + + const Ltx2BatchedPerturbationConfig slice = batched.BatchSlice(index, index + 1); + Ltx2DitPerturbation perturbation; + bool any = false; + for (int64_t block = 0; block < in.num_blocks; ++block) { + const bool v = + PerturbedAt(slice, Ltx2PerturbationType::kSkipVideoSelfAttn, block, /*sample=*/0); + const bool a = + PerturbedAt(slice, Ltx2PerturbationType::kSkipAudioSelfAttn, block, /*sample=*/0); + if (v || a) any = true; + // Both vectors are sized whenever either is, because `Ltx2DitForward` + // refuses a vector that is neither empty nor one entry per block and an + // empty one means "nothing perturbed" for that stream. + perturbation.video_self_attn.push_back(v ? 1 : 0); + perturbation.audio_self_attn.push_back(a ? 1 : 0); + } + // The cross flags are not per block, because `Ltx2DitPerturbation` has no + // per-block cross vector and upstream's reader is the per-block scalar + // `cross_attn_skip_all` (transformer.py:335,366) rather than a mask + // multiply. That flattening is only sound while the config says the same + // thing on every block, which is what `blocks=None` produces + // (denoisers.py:132-135) — so it is CHECKED here rather than assumed. A + // block-list cross perturbation would otherwise be silently widened to all + // blocks, which renders. + const auto flatten_cross = [&](Ltx2PerturbationType type, const char* name) { + const bool first = PerturbedAt(slice, type, /*block=*/0, /*sample=*/0); + for (int64_t block = 1; block < in.num_blocks; ++block) { + if (PerturbedAt(slice, type, block, /*sample=*/0) == first) continue; + Fail(std::string("the ") + name + + " cross-attention perturbation differs between block 0 and block " + + std::to_string(block) + ". `Ltx2DitPerturbation` carries one boolean per direction " + "because the only thing upstream builds these with is `blocks=None` " + "(denoisers.py:132-135); a per-block cross perturbation cannot be represented and " + "would be widened to every block rather than refused"); + } + return first; + }; + perturbation.video_cross_attn_skip_all = + flatten_cross(Ltx2PerturbationType::kSkipA2vCrossAttn, "audio-to-video"); + perturbation.audio_cross_attn_skip_all = + flatten_cross(Ltx2PerturbationType::kSkipV2aCrossAttn, "video-to-audio"); + if (perturbation.video_cross_attn_skip_all || perturbation.audio_cross_attn_skip_all) { + any = true; + } + if (!any) { + // `PerturbationConfig.empty()` reaches the forward as upstream's + // `perturbations=None` (model.py:509-511), not as an all-ones mask, so the + // conditional and unconditional passes take the same path an unguided + // render takes. + perturbation = Ltx2DitPerturbation{}; + } + + Ltx2ModalityInput video_in; + Ltx2ModalityInput audio_in; + if (in.video != nullptr) { + video_in = *in.video; + video_in.context = pass.video_context; + // `enabled=not v_skip` (`:151`). A skipped modality stays PRESENT, so the + // other stream's cross attention still reads its latent + // (transformer.py:269 tests presence, not `enabled`). + video_in.enabled = !v_skip; + } + if (in.audio != nullptr) { + audio_in = *in.audio; + audio_in.context = pass.audio_context; + audio_in.enabled = !a_skip; + } + + const bool perturbed = any; + Ltx2X0Outputs out = transformer(in.video != nullptr ? &video_in : nullptr, + in.audio != nullptr ? &audio_in : nullptr, + perturbed ? &perturbation : nullptr); + + const size_t slot = static_cast(pass.kind); + result.pass_ran[slot] = true; + result.video_pass[slot] = std::move(out.video); + result.audio_pass[slot] = std::move(out.audio); + result.video_pass_velocity[slot] = std::move(out.video_velocity); + result.audio_pass_velocity[slot] = std::move(out.audio_velocity); + + // Observed at the call rather than restated from the guider params: a + // perturbation that is BUILT and not HANDED OVER leaves the params untouched + // and the render finite. Derived here, so a mutation that drops the argument + // moves this record. + if (pass.kind == Ltx2DenoisePass::kPerturbed && perturbed) { + for (int64_t block = 0; block < in.num_blocks; ++block) { + if (perturbation.video_self_attn[static_cast(block)] != 0) { + result.perturbed_video_blocks.push_back(block); + } + if (perturbation.audio_self_attn[static_cast(block)] != 0) { + result.perturbed_audio_blocks.push_back(block); + } + } + } + if (pass.kind == Ltx2DenoisePass::kModality && perturbed) { + result.modality_pass_skipped_a2v = perturbation.video_cross_attn_skip_all; + result.modality_pass_skipped_v2a = perturbation.audio_cross_attn_skip_all; + } + } + + // ── the combination (`:192-204`) ────────────────────────────────────────── + // + // EACH MODALITY WITH ITS OWN GUIDER, over the SAME splits. `r.get("uncond", + // (0.0, 0.0))` is upstream's absent pass and is the float 0.0 its + // `calculate` signature admits; a null here is the same thing, and + // `Ltx2MultiModalGuidance` reads it as 0.0 (guiders.py:247-249). + const auto at = [&result](const std::vector* passes_array, Ltx2DenoisePass kind) { + const size_t slot = static_cast(kind); + return result.pass_ran[slot] ? passes_array[slot].data() : nullptr; + }; + const size_t cond_slot = static_cast(Ltx2DenoisePass::kCond); + + if (in.video != nullptr) { + if (v_skip) { + result.video_denoised = reuse(in.last_denoised_video, "video"); + } else { + const std::vector& cond = result.video_pass[cond_slot]; + result.video_denoised = Ltx2MultiModalGuidance( + video_guider, cond.data(), at(result.video_pass, Ltx2DenoisePass::kUncond), + at(result.video_pass, Ltx2DenoisePass::kPerturbed), + at(result.video_pass, Ltx2DenoisePass::kModality), static_cast(cond.size())); + } + } + if (in.audio != nullptr) { + if (a_skip) { + result.audio_denoised = reuse(in.last_denoised_audio, "audio"); + } else { + const std::vector& cond = result.audio_pass[cond_slot]; + result.audio_denoised = Ltx2MultiModalGuidance( + audio_guider, cond.data(), at(result.audio_pass, Ltx2DenoisePass::kUncond), + at(result.audio_pass, Ltx2DenoisePass::kPerturbed), + at(result.audio_pass, Ltx2DenoisePass::kModality), static_cast(cond.size())); + } + } + return result; +} + +} // namespace vllm diff --git a/src/vllm/model_executor/models/ltx2_dit.cpp b/src/vllm/model_executor/models/ltx2_dit.cpp index bdf6b98cf..82227471e 100644 --- a/src/vllm/model_executor/models/ltx2_dit.cpp +++ b/src/vllm/model_executor/models/ltx2_dit.cpp @@ -349,7 +349,14 @@ void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, /*mod_index=*/0); }; - if (run_a2v) { + // `if run_a2v and not video.cross_attn_skip_all` (transformer.py:335). The + // guard is INSIDE the `run_a2v || run_v2a` block, exactly as upstream's is, + // so a pass that skips one direction still took the `vx_pre` / `ax_pre` + // snapshot above and the surviving direction reads the pre-cross state. + // Hoisting it into the outer condition would be equivalent only while both + // directions are always skipped together, which is true of the one caller + // today and is not a property of the flag. + if (run_a2v && !args.video_cross_attn_skip_all) { std::vector scale_v, shift_v, scale_a, shift_a; av_scale_shift(w.scale_shift_table_a2v_ca_video, args.video_cross_scale_shift, tv, dim, 0, &scale_v, &shift_v); @@ -377,7 +384,8 @@ void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, AddGatedBroadcast(video_x, out, gate, batch, tv, dim); } - if (run_v2a) { + // `if run_v2a and not audio.cross_attn_skip_all` (transformer.py:366). + if (run_v2a && !args.audio_cross_attn_skip_all) { std::vector scale_a, shift_a, scale_v, shift_v; av_scale_shift(w.scale_shift_table_a2v_ca_audio, args.audio_cross_scale_shift, ta, adim, 2, &scale_a, &shift_a); @@ -868,6 +876,11 @@ Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, perturbations->video_self_attn[static_cast(i)] != 0; a.audio_self_attn_perturbed = !perturbations->audio_self_attn.empty() && perturbations->audio_self_attn[static_cast(i)] != 0; + // Not indexed by block: the only thing that builds these asks for ALL + // blocks (`blocks=None`, denoisers.py:132-135), and upstream's reader is a + // per-block scalar rather than a mask multiply (transformer.py:335,366). + a.video_cross_attn_skip_all = perturbations->video_cross_attn_skip_all; + a.audio_cross_attn_skip_all = perturbations->audio_cross_attn_skip_all; } a.video_timestep_modulation = vs.modulation.empty() ? nullptr : vs.modulation.data(); a.audio_timestep_modulation = as.modulation.empty() ? nullptr : as.modulation.data(); diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index 6201dd748..ccd137760 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -30,6 +30,7 @@ #include "vllm/model_executor/models/ltx2_audio_vae.h" #include "vllm/model_executor/models/ltx2_conditioning.h" #include "vllm/model_executor/models/ltx2_connector.h" +#include "vllm/model_executor/models/ltx2_denoisers.h" #include "vllm/model_executor/models/ltx2_device.h" #include "vllm/model_executor/models/ltx2_dfr.h" #include "vllm/model_executor/models/ltx2_image_preprocess.h" @@ -373,13 +374,14 @@ constexpr char kLtx2DurationHeadPathExtra[] = "duration_head_path"; // they are no longer trusted: the list below is derived from this file on every // run and compared, and the failure prints the replacement to paste in. // READER ANCHORS (derived and gated by test_ltx2_video): -// 791 801 802 864 960 976 978 1069 1094 1199 1240 +// 798 808 809 871 967 983 985 1076 1101 1206 1247 1289 1291 const char* const kKnownLoadExtras[] = { kLtx2AudioPromptEmbedsExtra, kLtx2PipelineKindExtra, kLtx2ModelVersionExtra, kLtx2AllowUnportedExtra, kLtx2MaxPhaseExtra, kLtx2DitConfigPathExtra, kLtx2PromptValidRowsExtra, kLtx2EncoderConfigPathExtra, "upsampler_path", kLtx2DurationHeadPathExtra, kLtx2LoraPathExtra, kLtx2LoraStrengthExtra, + kLtx2NegativePromptEmbedsExtra, kLtx2NegativeAudioPromptEmbedsExtra, }; // FNV-1a over the raw bytes of a float buffer — the `Ltx2ConditioningTrace` @@ -624,6 +626,11 @@ struct Ltx2VideoEngine::Impl { // did on every checkpoint. std::vector video_prompt_embeds, audio_prompt_embeds; int64_t prompt_tokens = 0; + // The NEGATIVE half of the same fallback (row LTX25-GUIDED-VIDEO, #1092). + // Empty when the load supplied none, which is what makes a guider that asks + // for the unconditional forward a refusal rather than a silent reuse of the + // positive context. + std::vector negative_video_prompt_embeds, negative_audio_prompt_embeds; // The connector's CONFIGURATION is kept; its WEIGHTS are not. They are ~8 GB // of f32 at the shipped widths (ltx2_loader.h), the conditioning they process @@ -1269,6 +1276,60 @@ std::unique_ptr Ltx2VideoEngine::Load(const VideoModelParams& p im.video_prompt_embeds = encoded.video; im.audio_prompt_embeds = encoded.audio; } + + // ── the NEGATIVE half (row LTX25-GUIDED-VIDEO, #1092) ────────────────── + // + // The same two files for upstream's second encoding. Loaded here, INSIDE the + // positive block, because a negative pair without a positive one conditions + // nothing: `prompt_embeds_path` is what a tower-less engine renders from. + // The two negative files follow the positive pair's own rule — supplied + // together or not at all — and must agree with it on row count, because the + // guidance delta subtracts them elementwise. + const std::string negative_video_path = + VideoExtra(params.extras, kLtx2NegativePromptEmbedsExtra); + const std::string negative_audio_path = + VideoExtra(params.extras, kLtx2NegativeAudioPromptEmbedsExtra); + if (negative_video_path.empty() != negative_audio_path.empty()) { + Fail("the '" + std::string(kLtx2NegativePromptEmbedsExtra) + "' and '" + + std::string(kLtx2NegativeAudioPromptEmbedsExtra) + + "' extras are supplied together or not at all, for the same reason the positive pair " + "is: LTX-2.5 conditions TWO streams at two widths and one of them alone would leave a " + "stream unconditioned on the unconditional forward, which renders."); + } + if (!negative_video_path.empty()) { + im.negative_video_prompt_embeds = + ReadF32File(kLtx2NegativePromptEmbedsExtra, negative_video_path); + im.negative_audio_prompt_embeds = + ReadF32File(kLtx2NegativeAudioPromptEmbedsExtra, negative_audio_path); + if (static_cast(im.negative_video_prompt_embeds.size()) != v_rows * vw || + static_cast(im.negative_audio_prompt_embeds.size()) != a_rows * aw) { + Fail("the negative prompt embeds hold " + + std::to_string(im.negative_video_prompt_embeds.size()) + " / " + + std::to_string(im.negative_audio_prompt_embeds.size()) + + " floats and the positive pair holds " + + std::to_string(im.video_prompt_embeds.size()) + " / " + + std::to_string(im.audio_prompt_embeds.size()) + + " at widths " + std::to_string(vw) + " / " + std::to_string(aw) + + ". Upstream encodes `[prompt, negative_prompt]` in ONE call, so the two halves " + "share a padded width by construction and `(cfg_scale - 1) * (cond - uncond)` " + "subtracts them elementwise"); + } + if (im.has_connector) { + // Through the SAME connector, with the SAME mask. A negative stream that + // skipped it would be compared against a positive stream that did not, + // and the delta would be dominated by the connector rather than by the + // prompt. + std::vector additive(static_cast(v_rows), 0.0f); + for (int64_t s = im.prompt_valid_rows; s < v_rows; ++s) { + additive[static_cast(s)] = -std::numeric_limits::max(); + } + const Ltx2ConnectorEmbeddings encoded = RunConnector( + dit_file, im.video_connector_cfg, im.audio_connector_cfg, + im.negative_video_prompt_embeds, im.negative_audio_prompt_embeds, additive, v_rows); + im.negative_video_prompt_embeds = encoded.video; + im.negative_audio_prompt_embeds = encoded.audio; + } + } } return engine; } @@ -1433,6 +1494,142 @@ void AssertGeneratedKeyframesSupported(bool has_embedding, const std::string& di "admitted upstream (#902). Supply a generated-keyframe checkpoint, or drop the request."); } +// ── the guiders (row LTX25-GUIDED-VIDEO, #1092) ──────────────────────────── + +// `--*-stg-blocks`, `nargs="*"` (utils/args.py:980-985, :1040-1045). An extra +// that is PRESENT and empty is upstream's empty list — "perturb nothing" — and +// stays distinct from an ABSENT extra, which takes the params table's own value. +// Collapsing the two would make `video_stg_blocks=` silently mean block 28. +void ApplyStgBlocksExtra(const std::map& extras, const char* key, + std::vector* blocks) { + const auto at = extras.find(key); + if (at == extras.end()) return; + blocks->clear(); + const std::string& raw = at->second; + for (size_t i = 0; i < raw.size();) { + const size_t comma = raw.find(',', i); + const std::string token = raw.substr(i, comma == std::string::npos ? comma : comma - i); + if (!token.empty()) { + try { + blocks->push_back(std::stoll(token)); + } catch (const std::exception&) { + Fail("'" + std::string(key) + "' holds '" + token + + "', which is not an integer block index"); + } + } + if (comma == std::string::npos) break; + i = comma + 1; + } +} + +// One CLI flag each, from `default_1_stage_arg_parser` (utils/args.py:947-1010 +// for the video row, :1011-1075 for the audio one). Each extra overrides ONE +// field of the phase's own resolved guider, which is what one flag does. +// +// REFUSED WHOLESALE on a phase that fixes its guidance. `allow_guidance_override +// = false` is set by the distilled two-stage and retake recipes +// (ltx2_recipes.py:125-158, retake.py:53) whose guidance is distilled INTO the +// weights, and until this row nothing read it. Honouring an override there would +// sample a trajectory the weights were never trained for — the same argument +// `fixed_num_inference_steps` already makes about the schedule, and the same +// reason it is a refusal rather than a silent clamp. +void ApplyGuidanceOverrides(const std::map& extras, + const Ltx2PhaseRecipe& phase, Ltx2MultiModalGuiderParams* video, + Ltx2MultiModalGuiderParams* audio) { + static const char* const kVideoKeys[] = { + kLtx2VideoCfgScaleExtra, kLtx2VideoStgScaleExtra, kLtx2VideoRescaleScaleExtra, + kLtx2VideoSkipStepExtra, kLtx2VideoStgBlocksExtra, kLtx2A2vGuidanceScaleExtra, + kLtx2AudioCfgScaleExtra, kLtx2AudioStgScaleExtra, kLtx2AudioRescaleScaleExtra, + kLtx2AudioSkipStepExtra, kLtx2AudioStgBlocksExtra, kLtx2V2aGuidanceScaleExtra}; + if (!phase.allow_guidance_override) { + for (const char* key : kVideoKeys) { + if (extras.find(key) == extras.end()) continue; + Fail("phase '" + phase.name + + "' fixes its own guidance, so the '" + std::string(key) + + "' extra is refused rather than applied. This recipe's scales are distilled INTO the " + "weights (ltx2_recipes.py:125-158), and a render that honoured the override would " + "sample a trajectory they were never trained for."); + } + return; + } + video->cfg_scale = ExtraDouble(extras, kLtx2VideoCfgScaleExtra, video->cfg_scale); + video->stg_scale = ExtraDouble(extras, kLtx2VideoStgScaleExtra, video->stg_scale); + video->rescale_scale = ExtraDouble(extras, kLtx2VideoRescaleScaleExtra, video->rescale_scale); + video->modality_scale = ExtraDouble(extras, kLtx2A2vGuidanceScaleExtra, video->modality_scale); + video->skip_step = ExtraInt(extras, kLtx2VideoSkipStepExtra, video->skip_step); + ApplyStgBlocksExtra(extras, kLtx2VideoStgBlocksExtra, &video->stg_blocks); + + audio->cfg_scale = ExtraDouble(extras, kLtx2AudioCfgScaleExtra, audio->cfg_scale); + audio->stg_scale = ExtraDouble(extras, kLtx2AudioStgScaleExtra, audio->stg_scale); + audio->rescale_scale = ExtraDouble(extras, kLtx2AudioRescaleScaleExtra, audio->rescale_scale); + audio->modality_scale = ExtraDouble(extras, kLtx2V2aGuidanceScaleExtra, audio->modality_scale); + audio->skip_step = ExtraInt(extras, kLtx2AudioSkipStepExtra, audio->skip_step); + ApplyStgBlocksExtra(extras, kLtx2AudioStgBlocksExtra, &audio->stg_blocks); + + const auto check_skip = [](const char* key, int64_t value) { + if (value >= 0) return; + Fail("'" + std::string(key) + "' is " + std::to_string(value) + + "; `should_skip_step` is `step % (skip_step + 1)` (guiders.py:287-291) and a negative " + "value would take the modulus of a non-positive divisor"); + }; + check_skip(kLtx2VideoSkipStepExtra, video->skip_step); + check_skip(kLtx2AudioSkipStepExtra, audio->skip_step); + // `stg_blocks` is only read when the perturbed pass runs, so an out-of-range + // block is refused HERE rather than inside the DiT, where the message would be + // about a mask length. `blocks is None` — upstream's "every block" — has no CLI + // spelling and none is invented, so an EMPTY list beside a non-zero STG scale + // is a perturbed pass identical to the conditional one, which is a wasted + // forward and a guidance term of exactly zero. + const auto check_blocks = [&](const Ltx2MultiModalGuiderParams& g, const char* which, + const char* key) { + if (!g.DoPerturbedGeneration()) return; + if (g.stg_blocks.empty()) { + Fail(std::string("the ") + which + " STG scale is " + std::to_string(g.stg_scale) + + " and its block list is EMPTY, so the perturbed forward would be identical to the " + "conditional one and `stg_scale * (cond - perturbed)` would be exactly zero " + "(guiders.py:264). Set '" + std::string(key) + + "' to the blocks to perturb, or set the STG scale to 0.0"); + } + }; + check_blocks(*video, "video", kLtx2VideoStgBlocksExtra); + check_blocks(*audio, "audio", kLtx2AudioStgBlocksExtra); +} + +// Everything step 0 of phase 0 produced, for the gate that decides WHICH SPACE +// each arm was combined in. Derived at the call from what the seam returned, so +// a mutation to any arm moves a recorded field rather than leaving a comment +// that compiles. +void RecordFirstGuidedStep(Ltx2ConditioningTrace* trace, const Ltx2GuidedDenoiseResult& guided, + const std::vector& latent, + const std::vector& timesteps, double sigma, + const std::vector& stepper_input) { + const auto slot = [](Ltx2DenoisePass pass) { return static_cast(pass); }; + trace->video_guided = true; + trace->video_cond_forwards = guided.pass_ran[slot(Ltx2DenoisePass::kCond)] ? 1 : 0; + trace->video_uncond_forwards = guided.pass_ran[slot(Ltx2DenoisePass::kUncond)] ? 1 : 0; + trace->video_perturbed_forwards = guided.pass_ran[slot(Ltx2DenoisePass::kPerturbed)] ? 1 : 0; + trace->video_modality_forwards = guided.pass_ran[slot(Ltx2DenoisePass::kModality)] ? 1 : 0; + trace->video_perturbed_blocks = guided.perturbed_video_blocks; + trace->video_audio_perturbed_blocks = guided.perturbed_audio_blocks; + trace->video_modality_skipped_a2v = guided.modality_pass_skipped_a2v; + trace->video_modality_skipped_v2a = guided.modality_pass_skipped_v2a; + trace->video_first_latent = latent; + trace->video_first_timesteps = timesteps; + trace->video_first_cond = guided.video_pass[slot(Ltx2DenoisePass::kCond)]; + trace->video_first_cond_velocity = guided.video_pass_velocity[slot(Ltx2DenoisePass::kCond)]; + trace->video_first_uncond = guided.video_pass[slot(Ltx2DenoisePass::kUncond)]; + trace->video_first_uncond_velocity = guided.video_pass_velocity[slot(Ltx2DenoisePass::kUncond)]; + trace->video_first_perturbed = guided.video_pass[slot(Ltx2DenoisePass::kPerturbed)]; + trace->video_first_perturbed_velocity = + guided.video_pass_velocity[slot(Ltx2DenoisePass::kPerturbed)]; + trace->video_first_modality = guided.video_pass[slot(Ltx2DenoisePass::kModality)]; + trace->video_first_modality_velocity = + guided.video_pass_velocity[slot(Ltx2DenoisePass::kModality)]; + trace->video_first_denoised = guided.video_denoised; + trace->video_first_stepper_input = stepper_input; + trace->video_first_sigma = sigma; +} + } // namespace VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { @@ -1467,7 +1664,17 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { kv.first == kLtx2AudioStgScaleExtra || kv.first == kLtx2AudioRescaleScaleExtra || kv.first == kLtx2AudioSkipStepExtra || - kv.first == kLtx2AudioStgBlocksExtra; + kv.first == kLtx2AudioStgBlocksExtra || + // The VIDEO guider's row (row LTX25-GUIDED-VIDEO, #1092), + // from the same parser as the audio row above + // (utils/args.py:947-1010). + kv.first == kLtx2VideoCfgScaleExtra || + kv.first == kLtx2VideoStgScaleExtra || + kv.first == kLtx2VideoRescaleScaleExtra || + kv.first == kLtx2VideoSkipStepExtra || + kv.first == kLtx2VideoStgBlocksExtra || + kv.first == kLtx2A2vGuidanceScaleExtra || + kv.first == kLtx2V2aGuidanceScaleExtra; if (!known) { Fail("unknown per-generation extra '" + kv.first + "'. This family defines: " + std::string(kLtx2ImageCrfExtra) + ", " + kLtx2AudioPathExtra + ", " + @@ -1478,41 +1685,50 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { kLtx2RegenerateAudioExtra + ", " + kLtx2NegativePromptExtra + ", " + kLtx2AudioCfgScaleExtra + ", " + kLtx2AudioStgScaleExtra + ", " + kLtx2AudioRescaleScaleExtra + ", " + kLtx2AudioSkipStepExtra + ", " + - kLtx2AudioStgBlocksExtra); + kLtx2AudioStgBlocksExtra + ", " + kLtx2VideoCfgScaleExtra + ", " + + kLtx2VideoStgScaleExtra + ", " + kLtx2VideoRescaleScaleExtra + ", " + + kLtx2VideoSkipStepExtra + ", " + kLtx2VideoStgBlocksExtra + ", " + + kLtx2A2vGuidanceScaleExtra + ", " + kLtx2V2aGuidanceScaleExtra); } } - // ── the TEXT-TO-AUDIO knobs belong to ONE pipeline (#1005) ──────────────── + // ── the knobs that belong to ONE pipeline (#1005, corrected by #1092) ───── // // `pipeline_kind` is a LOAD extra, so which pipeline runs is settled before a - // request arrives and this is a decidable question rather than a guess. The - // guard runs in BOTH directions: the six T2A knobs are refused off a - // `t2a_one_stage` engine, and every other per-generation knob is refused ON - // one. Neither is padding. Upstream's `T2AOneStagePipeline.__call__` takes no - // image, no reference, no keyframe and no window (t2a_one_stage.py:109-122), - // and its guider arguments have no counterpart in any other `__call__`, so a - // knob crossing either way would silently do nothing to a render that still - // finishes. + // request arrives and this is a decidable question rather than a guess. + // + // WHAT #1092 CORRECTED, and why the old list was defensible until it was not. + // Row LTX25-T2A-ONE-STAGE refused `negative_prompt` and the five `audio_*` + // guider knobs on ANY non-t2a engine, reasoning that "no other pipeline + // `__call__` upstream takes a guider argument at all". That sentence was + // FALSE about upstream and TRUE about this port. Upstream's + // `default_1_stage_arg_parser` carries `--negative-prompt` + // (utils/args.py:937-946) and the whole audio guider row + // (`:1011-1075`) alongside the video one, and `TI2VidOneStagePipeline` + // consumes both through `audio_guider_params` (ti2vid_one_stage.py:215-218). + // What made the refusal harmless was that NOTHING HERE READ THEM on a joint + // render — the video denoise loop was unguided. Row LTX25-GUIDED-VIDEO makes + // them live, so the refusal would now reject a flag upstream serves. + // + // The guard therefore keeps one direction and drops the other: the knobs that + // describe a PICTURE are refused on a text-to-audio engine, which produces + // none. `T2AOneStagePipeline.__call__` takes a prompt, a negative prompt, a + // seed, a frame rate, a step count, the audio guider and a frame count + // (t2a_one_stage.py:109-122) and nothing else. { - const char* const kT2aOnly[] = {kLtx2NegativePromptExtra, kLtx2AudioCfgScaleExtra, - kLtx2AudioStgScaleExtra, kLtx2AudioRescaleScaleExtra, - kLtx2AudioSkipStepExtra, kLtx2AudioStgBlocksExtra}; const char* const kNotOnT2a[] = {kLtx2ImageCrfExtra, kLtx2AudioPathExtra, kLtx2AudioStartTimeExtra, kLtx2AudioMaxDurationExtra, kLtx2GeneratedKeyframesExtra, kLtx2TemporalRoundsExtra, kLtx2RetakeStartTimeExtra, kLtx2RetakeEndTimeExtra, kLtx2RetakeFrameRateExtra, kLtx2RegenerateVideoExtra, - kLtx2RegenerateAudioExtra}; - for (const char* key : kT2aOnly) { - if (!im.recipe.audio_only && !VideoExtra(gen.extras, key).empty()) { - Fail("the '" + std::string(key) + - "' extra is text-to-audio's alone (ltx-pipelines utils/args.py:1083-1119, the " - "`default_1_stage_t2a_arg_parser`), and this engine was loaded with pipeline_kind '" + - im.pipeline_kind + - "'. Refused rather than ignored: no other pipeline `__call__` upstream takes a " - "guider argument at all, so accepting it here would report a configured render " - "that ran the recipe's own values"); - } - } + kLtx2RegenerateAudioExtra, + // The VIDEO guider's own row: there is no + // video stream to guide, and upstream's t2a + // parser exposes none of them + // (utils/args.py:1083-1119). + kLtx2VideoCfgScaleExtra, kLtx2VideoStgScaleExtra, + kLtx2VideoRescaleScaleExtra, kLtx2VideoSkipStepExtra, + kLtx2VideoStgBlocksExtra, kLtx2A2vGuidanceScaleExtra, + kLtx2V2aGuidanceScaleExtra}; for (const char* key : kNotOnT2a) { if (im.recipe.audio_only && !VideoExtra(gen.extras, key).empty()) { Fail("the '" + std::string(key) + @@ -2323,6 +2539,145 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { im.trace.retake_latent_absmax = AbsMax(retake_video_volume); } + // ── THE GUIDERS, and the negative conditioning they ask for (#1092) ─────── + // + // `create_multimodal_guider_factory(params=..., negative_context=...)` once per + // stream, before the stage runs (ti2vid_one_stage.py:210-218). Resolved for + // EVERY phase up front rather than inside the loop, because the negative + // encode below is a host-side pass over the text tower and must happen once + // for the whole render if ANY phase asks for it. + // + // A phase whose recipe sets no guidance keeps `Ltx2MultiModalGuiderParams`'s + // own defaults — `cfg 1.0 / stg 0.0 / modality 1.0 / rescale 0.0` — which is + // exactly `_POSITIVE_ONLY_GUIDER` (denoisers.py:25-28). Only `OneStagePhase` + // sets real scales, so `distilled_two_stage`, `dfr`, `retake` and `dmd2` run + // ONE forward per step through the guided seam and combine it with a guider + // whose every term is zero, which is `SimpleDenoiser`'s output. Upstream + // selects `SimpleDenoiser` by PIPELINE (distilled.py:266,295) rather than by + // params; the two agree here because the recipes that select it are exactly + // the recipes whose guidance is the no-op one. + struct PhaseGuidance { + Ltx2MultiModalGuiderParams video; + Ltx2MultiModalGuiderParams audio; + }; + std::vector phase_guidance(recipe.phases.size()); + bool wants_negative = false; + bool wants_perturbation = false; + for (size_t p = 0; p < recipe.phases.size(); ++p) { + phase_guidance[p].video = recipe.phases[p].video_guidance; + phase_guidance[p].audio = recipe.phases[p].audio_guidance; + ApplyGuidanceOverrides(gen.extras, recipe.phases[p], &phase_guidance[p].video, + &phase_guidance[p].audio); + if (phase_guidance[p].video.DoUnconditionalGeneration() || + phase_guidance[p].audio.DoUnconditionalGeneration()) { + wants_negative = true; + } + if (phase_guidance[p].video.DoPerturbedGeneration() || + phase_guidance[p].audio.DoPerturbedGeneration() || + phase_guidance[p].video.DoIsolatedModalityGeneration() || + phase_guidance[p].audio.DoIsolatedModalityGeneration()) { + wants_perturbation = true; + } + } + + // REFUSED BY NAME, not degraded. `Ltx2DitForwardDevice` (ltx2_device.h:136) + // takes no `perturbations` argument, so the perturbed and isolated-modality + // passes on the device arm would run an UNPERTURBED forward — a finite clip + // whose `stg_scale * (cond - perturbed)` and `(modality_scale - 1) * (cond - + // mod)` terms are identically zero, and which is indistinguishable from a + // working render at every output this engine has. Classifier-free guidance + // alone is a different CONTEXT and no perturbation, so it is served on both + // arms. + if (im.on_device && wants_perturbation) { + Fail("this render's guidance needs a PERTURBED forward (STG, or the isolated-modality pass " + "that `modality_scale != 1.0` selects) and `Ltx2DitForwardDevice` takes no " + "`perturbations` argument, so the device-resident arm cannot run one. Refusing rather " + "than running an unperturbed forward, which would leave the STG and modality terms " + "exactly zero and render. Set '" + + std::string(kLtx2VideoStgScaleExtra) + "' and '" + + std::string(kLtx2AudioStgScaleExtra) + "' to 0.0 and '" + + std::string(kLtx2A2vGuidanceScaleExtra) + "' and '" + + std::string(kLtx2V2aGuidanceScaleExtra) + + "' to 1.0 to run classifier-free guidance alone on this arm, or load with device 0. " + "Owed by row LTX25-GUIDED-VIDEO (#1092)."); + } + + // The second half of upstream's ONE `PromptEncoder` call over + // `[prompt, negative_prompt]` (ti2vid_one_stage.py:170-178). Encoded ONLY when + // a guider asks: `do_unconditional_generation` is `not isclose(cfg_scale, 1.0)` + // (guiders.py:275-277), and at 1.0 there is no unconditional forward, so + // encoding it would be a wasted host-side 12B pass per request. + std::vector negative_video, negative_audio; + const float* negative_video_context = nullptr; + const float* negative_audio_context = nullptr; + if (wants_negative) { + if (!im.negative_video_prompt_embeds.empty() && gen.prompt.empty()) { + // The embeds fallback's own second half. Taken only when the request + // carries no prompt, which is the same polarity the POSITIVE fallback has + // above: a typed prompt encodes both halves through the tower. + if (im.prompt_tokens != context_tokens) { + Fail("the negative prompt embeds hold " + std::to_string(im.prompt_tokens) + + " rows and this request's conditioning holds " + std::to_string(context_tokens) + + "; the guidance delta would subtract tensors that do not correspond"); + } + negative_video_context = im.negative_video_prompt_embeds.data(); + negative_audio_context = im.negative_audio_prompt_embeds.data(); + } else if (!im.has_encoder) { + Fail("this render needs an unconditional forward (the video cfg scale is " + + std::to_string(phase_guidance[0].video.cfg_scale) + " and the audio one is " + + std::to_string(phase_guidance[0].audio.cfg_scale) + + "), which needs the NEGATIVE prompt encoded — and no text tower is loaded. The " + "positive `prompt_embeds_path` fallback carries ONE conditioning pair; supply the " + "second through '" + + std::string(kLtx2NegativePromptEmbedsExtra) + "' and '" + + std::string(kLtx2NegativeAudioPromptEmbedsExtra) + + "', load with encoder_path, or set '" + std::string(kLtx2VideoCfgScaleExtra) + + "' and '" + std::string(kLtx2AudioCfgScaleExtra) + + "' to 1.0, which turns the unconditional pass off (guiders.py:275-277)"); + } else { + const std::string negative = + VideoExtra(gen.extras, kLtx2NegativePromptExtra, recipe.negative_prompt); + if (negative.empty()) { + Fail("this render needs a negative prompt and neither the '" + + std::string(kLtx2NegativePromptExtra) + + "' extra nor the recipe carries one. An EMPTY negative prompt is not the same as no " + "CFG: it still encodes and still steers, and upstream's CLI always supplies " + "`DEFAULT_NEGATIVE_PROMPT` (utils/args.py:937-946)"); + } + if (!recipe.allow_negative_prompt) { + Fail("this recipe takes no negative prompt (`prompts_to_encode` is `[prompt]` alone), so " + "a guider asking for the unconditional forward is a contradiction rather than a " + "request this engine can serve"); + } + vt::Queue text_queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + const Ltx2PromptConditioning encoded = Ltx2EncodePromptToConditioning( + *im.tower, *im.tokenizer, im.gemma_ids, im.caption_projections, im.feature_cfg, + negative, text_queue); + negative_video = encoded.conditioning.video; + negative_audio = encoded.conditioning.audio; + if (encoded.seq != context_tokens) { + // Upstream's two encodings come from ONE tokenization of a two-element + // list, so they share a padded width by construction. A mismatch means + // the two ran different geometries and the guidance delta would subtract + // tensors that do not correspond. + Fail("the negative prompt encoded to " + std::to_string(encoded.seq) + + " context rows and the prompt to " + std::to_string(context_tokens) + + "; upstream encodes both in one call and they cannot differ"); + } + if (im.has_connector) { + const Ltx2ConnectorEmbeddings through = + RunConnector(SafetensorsFile::Open(im.params.dit_path), im.video_connector_cfg, + im.audio_connector_cfg, encoded.conditioning.video, + encoded.conditioning.audio, encoded.conditioning.additive_mask, + context_tokens); + negative_video = through.video; + negative_audio = through.audio; + } + negative_video_context = negative_video.data(); + negative_audio_context = negative_audio.data(); + } + } + for (int64_t phase_index = 0; phase_index <= last_phase; ++phase_index) { const Ltx2PhaseRecipe& phase = recipe.phases[static_cast(phase_index)]; const int64_t phase_h = height / phase.spatial_downscale; @@ -2933,6 +3288,27 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // from the state noise, so its first draw is not the initial latent's. SplitMixGaussian loop_noise(seed + static_cast(phase.noise_seed_offset)); const int64_t sigma_count = static_cast(sigmas.size()); + + // This phase's two guiders, resolved once. `GuidedDenoiser` is constructed + // per stage upstream and holds its guiders for the whole loop + // (ti2vid_one_stage.py:221-226), so resolving them per step would let a + // request override change meaning halfway down a schedule. + const Ltx2MultiModalGuiderParams& video_guidance = + phase_guidance[static_cast(phase_index)].video; + const Ltx2MultiModalGuiderParams& audio_guidance = + phase_guidance[static_cast(phase_index)].audio; + if (phase_index == 0) { + im.trace.video_guidance_cfg_scale = video_guidance.cfg_scale; + im.trace.video_guidance_stg_scale = video_guidance.stg_scale; + im.trace.video_guidance_rescale_scale = video_guidance.rescale_scale; + im.trace.video_guidance_modality_scale = video_guidance.modality_scale; + } + + // `_last_denoised_video` / `_last_denoised_audio` (denoisers.py:274-275): + // per DENOISER, so per phase, and empty until the first step fills them. A + // skipped step reuses them instead of running a forward. + std::vector last_denoised_video; + std::vector last_denoised_audio; for (int64_t step = 0; step + 1 < sigma_count; ++step) { const float sigma = sigmas[static_cast(step)]; const std::vector v_timesteps = TimestepsFromMask(video, sigma); @@ -3029,20 +3405,75 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { ain.positions = audio.positions.data(); ain.context = audio_context; + // ── the X0 MODEL (model.py:590-604), and the guided denoiser ────────── + // + // `DiffusionStage` never hands the loop the raw velocity model: it hands + // `X0Model(builder.build(...))` (utils/blocks.py:480-482). So `to_denoised` + // belongs HERE, inside the wrapper, applied to EVERY pass on its way out of + // the forward — and the guider downstream combines already-denoised + // tensors. Converting once after the guider instead is a DIFFERENT function + // wherever `rescale_scale != 0` (guiders.py:268-271), which is 0.7 on every + // video row of the params table. That defect shipped on the audio arm of + // this tree and is #1039. + // // One graph, two residencies. On the CPU this is the L2 parity forward in // its declared f32; on an accelerator it is the phase-L8 device-resident // forward over the bf16 the DiT was STAGED at, and the two agree on - // everything but where the bytes live and how wide they are. - const Ltx2DitOutputs velocity = - im.on_device ? Ltx2DitForwardDevice(*im.queue, im.dit.params, im.dit.weights, &vin, - &ain, im.compute_dtype) - : Ltx2DitForward(im.device, im.dit.params, im.dit.weights, &vin, &ain, - im.compute_dtype); - - const std::vector v_denoised = PostProcessLatent( - ToDenoised(video.latent, velocity.video, v_timesteps, video.tokens, video.width), video); - const std::vector a_denoised = PostProcessLatent( - ToDenoised(audio.latent, velocity.audio, a_timesteps, audio.tokens, audio.width), audio); + // everything but where the bytes live and how wide they are. The device + // forward takes no `perturbations`, which is why a guider that asks for the + // perturbed or isolated-modality pass on that arm is refused before the + // loop rather than served an unperturbed forward. + const Ltx2X0Model x0_model = [&](const Ltx2ModalityInput* v, const Ltx2ModalityInput* a, + const Ltx2DitPerturbation* p) { + const Ltx2DitOutputs velocity = + im.on_device ? Ltx2DitForwardDevice(*im.queue, im.dit.params, im.dit.weights, v, a, + im.compute_dtype) + : Ltx2DitForward(im.device, im.dit.params, im.dit.weights, v, a, + im.compute_dtype, /*cache=*/nullptr, p); + Ltx2X0Outputs out; + out.video_velocity = velocity.video; + out.audio_velocity = velocity.audio; + // The PER-TOKEN timesteps, not the schedule scalar: a conditioned token + // sits at timestep 0 and using the scalar there re-noises it. + out.video = + ToDenoised(video.latent, velocity.video, v_timesteps, video.tokens, video.width); + out.audio = + ToDenoised(audio.latent, velocity.audio, a_timesteps, audio.tokens, audio.width); + return out; + }; + + Ltx2GuidedDenoiseInputs denoise_in; + denoise_in.video = &vin; + denoise_in.audio = &ain; + denoise_in.video_negative_context = negative_video_context; + denoise_in.audio_negative_context = negative_audio_context; + denoise_in.video_guider = video_guidance; + denoise_in.audio_guider = audio_guidance; + denoise_in.num_blocks = im.dit.params.num_layers; + denoise_in.step_index = step; + denoise_in.last_denoised_video = &last_denoised_video; + denoise_in.last_denoised_audio = &last_denoised_audio; + const Ltx2GuidedDenoiseResult guided = Ltx2GuidedDenoise(x0_model, denoise_in); + + // `post_process_latent` is applied by the LOOP to the guider's OUTPUT + // (samplers.py:35, :484), never per arm inside the denoiser. Pinning the + // conditioned tokens per arm would make every arm agree on exactly those + // tokens, which zeroes the guidance delta precisely where a keyframe or a + // reference clip is conditioning — a render that is correct everywhere the + // conditioning is absent. + // + // `last_denoised_*` keeps what the GUIDER returned, before the + // post-process, because that is what `_last_denoised_video` holds + // (denoisers.py:299-300) and what a skipped step reuses. + last_denoised_video = guided.video_denoised; + last_denoised_audio = guided.audio_denoised; + const std::vector v_denoised = PostProcessLatent(guided.video_denoised, video); + const std::vector a_denoised = PostProcessLatent(guided.audio_denoised, audio); + + if (phase_index == 0 && step == 0) { + RecordFirstGuidedStep(&im.trace, guided, video.latent, v_timesteps, + static_cast(sigma), v_denoised); + } const bool terminal = sigmas[static_cast(step + 1)] == 0.0F; if (phase.stepper == Ltx2StepperKind::kEulerAncestral) { @@ -3051,6 +3482,7 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // taking an ancestral step there would re-noise the finished latent. video.latent = v_denoised; audio.latent = a_denoised; + if (phase_index == 0 && step == 0) im.trace.video_first_next_latent = video.latent; continue; } const std::vector v_noise = @@ -3075,6 +3507,11 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { sigma_count, step, static_cast(audio.latent.size())); } + // What the sampler WROTE, recorded after the step rather than derived from + // what was recorded before it. It is the only observable that says which + // tensor the stepper was actually handed: a second `ToDenoised` on the way + // in leaves every other recorded field untouched. + if (phase_index == 0 && step == 0) im.trace.video_first_next_latent = video.latent; } // `clear_conditioning` + `unpatchify` (blocks.py:575-580, in that order). diff --git a/tests/vllm/multimodal/ltx2_video_fixture.h b/tests/vllm/multimodal/ltx2_video_fixture.h index 37fc7e93f..7306dc501 100644 --- a/tests/vllm/multimodal/ltx2_video_fixture.h +++ b/tests/vllm/multimodal/ltx2_video_fixture.h @@ -1425,6 +1425,12 @@ inline void WritePromptEmbeds(const std::string& path, const std::string& tag, i // The whole set, as an engine would be pointed at it. struct Paths { std::string dit, video_vae, audio_vae, upsampler, video_embeds, audio_embeds; + // The NEGATIVE half of the embeds fallback (row LTX25-GUIDED-VIDEO, #1092). + // Written from DIFFERENT tags than the positive pair, deliberately: a negative + // conditioning equal to the positive one makes `cond - uncond` identically + // zero, so the classifier-free term would vanish and every assertion about it + // would pass for the wrong reason. + std::string negative_video_embeds, negative_audio_embeds; // Phase L13: the text tower, and the Gemma config the shipped encoder does not // carry. Written by every fixture; POINTING the engine at them is opt-in, // because a load that materializes a tower is not what most cases here gate. @@ -1446,6 +1452,8 @@ inline Paths WriteFixture(const std::string& dir, int64_t prompt_tokens = 4) { p.upsampler = dir + "/upsampler.safetensors"; p.video_embeds = dir + "/video_prompt_embeds.f32"; p.audio_embeds = dir + "/audio_prompt_embeds.f32"; + p.negative_video_embeds = dir + "/negative_video_prompt_embeds.f32"; + p.negative_audio_embeds = dir + "/negative_audio_prompt_embeds.f32"; p.encoder = dir + "/text_encoder.safetensors"; p.encoder_config = dir + "/gemma_config.json"; WriteReducedTextEncoder(dit, p.encoder); @@ -1457,6 +1465,12 @@ inline Paths WriteFixture(const std::string& dir, int64_t prompt_tokens = 4) { WritePromptEmbeds(p.video_embeds, "ltx2.embeds.video", prompt_tokens, dit.cross_attention_dim); WritePromptEmbeds(p.audio_embeds, "ltx2.embeds.audio", prompt_tokens, dit.audio_cross_attention_dim); + // `.negative` tags, so the two halves differ. `Param` seeds from the NAME, so + // these are as deterministic as the positive pair and independent of it. + WritePromptEmbeds(p.negative_video_embeds, "ltx2.embeds.video.negative", prompt_tokens, + dit.cross_attention_dim); + WritePromptEmbeds(p.negative_audio_embeds, "ltx2.embeds.audio.negative", prompt_tokens, + dit.audio_cross_attention_dim); return p; } diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 9c11de380..7b1924958 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -84,10 +84,38 @@ vllm::multimodal::VideoModelParams FixtureParams(const ltx2_fixture::Paths& path mp.audio_vae_path = paths.audio_vae; mp.prompt_embeds_path = paths.video_embeds; mp.extras[vllm::multimodal::kLtx2AudioPromptEmbedsExtra] = paths.audio_embeds; + // The NEGATIVE half (row LTX25-GUIDED-VIDEO, #1092), on the SHARED helper + // rather than only on the cases that gate guidance. A `one_stage` engine's own + // recipe resolves `cfg_scale = 3.0`, so upstream's unconditional forward is + // this fixture's DEFAULT configuration, and an engine that could not run it + // would make every one_stage case here a refusal. `distilled_two_stage` — the + // default kind, and what most cases below load — resolves `cfg_scale = 1.0` + // and never reads these. + mp.extras[vllm::multimodal::kLtx2NegativePromptEmbedsExtra] = paths.negative_video_embeds; + mp.extras[vllm::multimodal::kLtx2NegativeAudioPromptEmbedsExtra] = paths.negative_audio_embeds; mp.device = 0; return mp; } +// The ONE guider field a `one_stage` render on this fixture has to override, and +// the reason is the fixture rather than the row: the reduced DiT has TWO blocks +// (ltx2_video_fixture.h `ReducedDitParams`), so the params table's own +// `stg_blocks = [28]` (utils/constants.py:83-88) names a block this checkpoint +// does not have. Left alone, the perturbed forward would perturb nothing and +// `stg_scale * (cond - perturbed)` would be exactly zero — which the engine now +// refuses by name rather than rendering, so this is what turns that refusal into +// a render. Named explicitly rather than by setting the STG scale to 0, because +// turning the perturbed pass OFF is a different configuration and would vacate +// every assertion about it. +// +// Nothing else is overridden: `cfg_scale`, `rescale_scale` and `modality_scale` +// stay at the recipe's own 3.0 / 0.7 / 3.0, which is what makes the guided cases +// below sit on the DEFAULT arm. +void OneStageFixtureGuidance(vllm::multimodal::VideoGenParams* gen) { + gen->extras[vllm::multimodal::kLtx2VideoStgBlocksExtra] = "1"; + gen->extras[vllm::multimodal::kLtx2AudioStgBlocksExtra] = "1"; +} + // The smallest request the fixture's own scale factors admit: (8, 32, 32) means // 64x64 pixels is a 2x2 latent and 9 frames is 2 latent frames. vllm::multimodal::VideoGenParams FixtureGen(const std::string& out_dir) { @@ -755,6 +783,7 @@ TEST_CASE("ltx2 video: a size that does not divide the latent grid is REFUSED, p vllm::multimodal::LoadVideoEngine(mp); vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/one_stage_96"); gen.width = 96; + OneStageFixtureGuidance(&gen); const vllm::multimodal::VideoResult result = engine->Generate(gen); CHECK(result.width == 96); CHECK(result.height == 64); @@ -954,6 +983,12 @@ TEST_CASE("ltx2 video: every accepted load extra is READ by something") { // have readers -- `lora_path` builds an `Ltx2LoraSpec` and `lora_strength` // is parsed into it -- so they belong here and not in `refused`. vllm::multimodal::kLtx2LoraPathExtra, vllm::multimodal::kLtx2LoraStrengthExtra, + // Row LTX25-GUIDED-VIDEO (#1092): the NEGATIVE half of the embeds + // fallback. Both are read where the positive pair is, and both are read + // again by the guided denoise loop when a guider asks for the + // unconditional forward. + vllm::multimodal::kLtx2NegativePromptEmbedsExtra, + vllm::multimodal::kLtx2NegativeAudioPromptEmbedsExtra, }; // The keys the family defines and does NOT serve. Growing this list is a // deliberate act; growing it silently is the defect #611 records. @@ -981,8 +1016,8 @@ TEST_CASE("ltx2 video: every accepted load extra is READ by something") { // Every name this row inventoried is still accepted... for (const std::string& key : served) CHECK(listing.find(key) != std::string::npos); for (const std::string& key : refused) CHECK(listing.find(key) != std::string::npos); - // ...and there is no THIRTEENTH name that this inventory has never seen. The - // separator is ", ", so the count is one more than the separators. + // ...and there is no name past the end of this inventory that it has never + // seen. The separator is ", ", so the count is one more than the separators. size_t names = 1; for (size_t at = listing.find(", "); at != std::string::npos; at = listing.find(", ", at + 2)) { ++names; @@ -1084,14 +1119,16 @@ TEST_CASE("ltx2 video: the recorded reader anchors are the ones in the source") } REQUIRE(array_end > array_line); - // The eleven SERVED keys, by the token each is spelled with in the source. Order - // is irrelevant — the comparison is on the sorted set — so this list is not a - // second place the anchors live. + // The thirteen SERVED keys, by the token each is spelled with in the source. + // Order is irrelevant — the comparison is on the sorted set — so this list is + // not a second place the anchors live. The last two arrived with row + // LTX25-GUIDED-VIDEO (#1092). const std::vector served_tokens = { "kLtx2AudioPromptEmbedsExtra", "kLtx2PipelineKindExtra", "kLtx2ModelVersionExtra", "kLtx2AllowUnportedExtra", "kLtx2MaxPhaseExtra", "kLtx2DitConfigPathExtra", "kLtx2PromptValidRowsExtra", "kLtx2EncoderConfigPathExtra", "\"upsampler_path\"", "kLtx2LoraPathExtra", "kLtx2LoraStrengthExtra", + "kLtx2NegativePromptEmbedsExtra", "kLtx2NegativeAudioPromptEmbedsExtra", }; std::vector derived; for (const std::string& token : served_tokens) { @@ -3180,6 +3217,7 @@ TEST_CASE("ltx2 video: a LAST-frame keyframe is APPENDED, and the sequence is tr vllm::multimodal::VideoGenParams gen = request("one_stage_kf", kf_a_path); gen.steps = 2; // one_stage admits a step override; 50 would gate nothing extra + OneStageFixtureGuidance(&gen); (void)engine->Generate(gen); const vllm::multimodal::Ltx2ConditioningTrace trace = ltx2->last_conditioning(); @@ -5057,26 +5095,82 @@ TEST_CASE("ltx2 t2a: the refusals name what is missing, and each is checked HERE } } -TEST_CASE("ltx2 t2a: a t2a-only knob is refused on the video pipelines") { +TEST_CASE("ltx2: the AUDIO guider knobs are NOT t2a-only, and this case used to say they were") { + // WHAT THIS CASE ASSERTED UNTIL ROW LTX25-GUIDED-VIDEO (#1092): that + // `audio_cfg_guidance_scale` on a video pipeline is refused "text-to-audio's + // alone". The premise behind that refusal was that "no other pipeline + // `__call__` upstream takes a guider argument at all", and it is FALSE about + // upstream: `default_1_stage_arg_parser` carries the whole audio guider row + // beside the video one (ltx-pipelines utils/args.py:1011-1075 @ fd4ded7f) and + // `TI2VidOneStagePipeline` builds `audio_guider_params` from it + // (ti2vid_one_stage.py:215-218). It was harmless only because the joint + // render here was unguided, so nothing could have read the knob. + // + // The correction is kept as an executable statement rather than a deletion, + // because "this used to be refused" is exactly what a later reader needs. Workspace ws; - const vllm::multimodal::VideoModelParams mp = EncoderParams(ws.paths); + // A `one_stage` engine with the fixture's own text tower, which is the + // configuration these knobs describe: `distilled_two_stage` fixes its guidance + // and refuses every override, so asking it would test the other guard. + vllm::multimodal::VideoModelParams mp = EncoderParams(ws.paths); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = "one_stage"; const std::unique_ptr engine = vllm::multimodal::LoadVideoEngine(mp); REQUIRE(engine != nullptr); - vllm::multimodal::VideoGenParams gen = PromptedGen(ws.root + "/x", "a b c"); - gen.extras[vllm::multimodal::kLtx2AudioCfgScaleExtra] = "5.0"; - try { + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + + // FIRST, the negative prompt through the TOWER, which is the path the embeds + // fallback exists to stand in for. The recipe's own default negative prompt is + // upstream's `DEFAULT_NEGATIVE_PROMPT` -- an English sentence -- and this + // fixture's tokenizer carries a three-token vocabulary, so it is overridden + // here exactly as `--negative-prompt` is for (utils/args.py:937-946). + { + vllm::multimodal::VideoGenParams gen = PromptedGen(ws.root + "/tower_negative", "a b c"); + gen.steps = 2; + OneStageFixtureGuidance(&gen); + gen.extras[vllm::multimodal::kLtx2NegativePromptExtra] = "c b a"; + (void)engine->Generate(gen); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + CHECK(t.video_uncond_forwards == 1); + // And it was a DIFFERENT forward, so the tower's negative encoding reached + // the DiT rather than the positive one being handed over twice. + REQUIRE(!t.video_first_uncond.empty()); + CHECK(t.video_first_uncond != t.video_first_cond); + } + + // SECOND, the knob itself. At 1.0 on both streams there is no unconditional + // pass at all (guiders.py:275-277) and nothing to encode. Without this the + // case would pass on a build that accepted the knob and ignored it, which is + // the defect the extras surface exists to refuse. + { + vllm::multimodal::VideoGenParams gen = PromptedGen(ws.root + "/audio_knob", "a b c"); + gen.steps = 2; + OneStageFixtureGuidance(&gen); + gen.extras[vllm::multimodal::kLtx2AudioCfgScaleExtra] = "1.0"; + gen.extras[vllm::multimodal::kLtx2VideoCfgScaleExtra] = "1.0"; (void)engine->Generate(gen); - FAIL_CHECK("a t2a guider knob must be refused on a video pipeline"); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + CHECK(t.completed); + CHECK(t.video_uncond_forwards == 0); + } + + // The direction that SURVIVES: a knob that describes a picture, on an engine + // that renders none. + const std::unique_ptr t2a = + vllm::multimodal::LoadVideoEngine(T2aParams(ws.paths)); + REQUIRE(t2a != nullptr); + vllm::multimodal::VideoGenParams t2a_gen = T2aGen(ws.root + "/video_knob", "a b c"); + t2a_gen.extras[vllm::multimodal::kLtx2VideoCfgScaleExtra] = "5.0"; + try { + (void)t2a->Generate(t2a_gen); + FAIL_CHECK("a VIDEO guider knob must be refused on a text-to-audio engine"); } catch (const std::exception& e) { const std::string msg = e.what(); INFO(msg); - CHECK(msg.find("text-to-audio's alone") != std::string::npos); - // It names the pipeline the ENGINE resolved, so a message that guessed - // would say something else. - const auto* ltx = dynamic_cast(engine.get()); - REQUIRE(ltx != nullptr); - CHECK(msg.find("'" + ltx->pipeline_kind() + "'") != std::string::npos); + CHECK(msg.find("no meaning on a text-to-audio render") != std::string::npos); + CHECK(msg.find(vllm::multimodal::kLtx2VideoCfgScaleExtra) != std::string::npos); } } @@ -5596,3 +5690,562 @@ TEST_CASE("ltx2 t2a: rescale_scale 0 is the control because both spaces agree th CHECK(at_default > 1e-2); CHECK(at_default > 100.0 * at_zero); } + +// ─── row LTX25-GUIDED-VIDEO (#1092): the guided VIDEO denoiser ────────────── +// +// The video denoise loop ran ONE unguided forward per step and applied +// `ToDenoised` to it, while every recipe resolved a video guider that nothing +// read. These cases gate the four passes upstream's `_guided_denoise` assembles +// (ltx-pipelines utils/denoisers.py:97-137 @ fd4ded7f) and, for each of them, +// WHICH SPACE it was combined in. +// +// They enter through the production entry point — `LoadVideoEngine` then +// `VideoEngine::Generate`, which is what `vllm_video_generate` calls — on +// `pipeline_kind = one_stage`, whose OWN recipe resolves `cfg_scale = 3.0`, +// `stg_scale = 1.0`, `rescale_scale = 0.7` and `modality_scale = 3.0`. Nothing +// below constructs a guider, a DiT, a modality or a perturbation by hand. + +namespace { + +// `one_stage` on the shipped fixture, guided by its own recipe. The only guider +// field overridden is the STG block list, and `OneStageFixtureGuidance` says why. +vllm::multimodal::VideoModelParams OneStageParams(const ltx2_fixture::Paths& paths) { + vllm::multimodal::VideoModelParams mp = FixtureParams(paths); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = "one_stage"; + return mp; +} + +vllm::multimodal::VideoGenParams OneStageGen(const std::string& out_dir) { + vllm::multimodal::VideoGenParams gen = FixtureGen(out_dir); + gen.steps = 2; // two sigma intervals is enough to exercise the loop + OneStageFixtureGuidance(&gen); + return gen; +} + +double MaxAbsOf(const std::vector& v) { + double m = 0.0; + for (const float x : v) m = std::max(m, std::abs(static_cast(x))); + return m; +} + +double MaxAbsDiffOf(const std::vector& a, const std::vector& b) { + REQUIRE(a.size() == b.size()); + double m = 0.0; + for (size_t i = 0; i < a.size(); ++i) { + m = std::max(m, std::abs(static_cast(a[i]) - static_cast(b[i]))); + } + return m; +} + +} // namespace + +TEST_CASE("ltx2 one_stage: all four guidance arms are combined in X0 space (#1092)") { + // THE DEFECT THIS CASE EXISTS FOR, in two layers. + // + // The outer one is that the video path ran no guidance at all. That is caught + // by the pass counts below, which are read off the seam's own record of which + // forwards it issued rather than inferred from an output. + // + // The inner one is #1039's, on a path that never had it: `MultiModalGuider` + // combines DENOISED predictions, because `DiffusionStage` hands the loop an + // `X0Model` (utils/blocks.py:480-482) and not the raw velocity model. The + // guider's LINEAR terms are invariant under `x0 = latent - sigma*v`, so the + // difference is entirely in the rescale at guiders.py:268-271 — and + // `rescale_scale` is 0.7 on this recipe, which is the DEFAULT arm. + // + // WHAT MAKES THAT UNREACHABLE BY ACCIDENT. `cond == latent - sigma*velocity` is + // an EQUATION between three recorded tensors, not a magnitude. It is exact in + // x0 space; in velocity space `cond` IS the velocity and the residual is the + // whole sample. No fixture scale satisfies it by accident, a zeroed velocity + // collapses it to `cond == latent` and is refused by the lower bound below, + // and a zeroed `cond` fails it outright. + // + // ALL FOUR ARMS. #1039's first gate asserted the equation for the conditional + // pass alone; the T2A arm runs three forwards and three mutations survived + // that draft. This arm runs FOUR. + Workspace ws; + + // The arm this case sits on, pinned as a LOCAL fact before anything is read off + // a render. `rescale_scale = 0.7` on the 2.4/2.5 lineage (ltx-pipelines + // utils/constants.py:40-80, reached through `_PARAMS_SINCE_VERSION`). + const vllm::Ltx2PipelineRecipe recipe = vllm::ResolveLtx2PipelineRecipe("one_stage", "2.5"); + REQUIRE(recipe.phases.size() == 1); + const vllm::Ltx2MultiModalGuiderParams row = recipe.phases[0].video_guidance; + CHECK(row.cfg_scale == 3.0); + CHECK(row.stg_scale == 1.0); + CHECK(row.rescale_scale == 0.7); + CHECK(row.modality_scale == 3.0); + + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(OneStageParams(ws.paths)); + REQUIRE(engine != nullptr); + (void)engine->Generate(OneStageGen(ws.root + "/guided_x0")); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + + // ── the render ran the guidance its recipe asked for ────────────────────── + // + // Counts, not tensors. An arm silently skipped changes a counter and changes no + // output shape, no frame count and no sample rate. + REQUIRE_MESSAGE(t.video_guided, "the video denoise did not go through the guided seam at all"); + CHECK(t.video_cond_forwards == 1); + CHECK_MESSAGE(t.video_uncond_forwards == 1, + "no unconditional forward ran, so `(cfg_scale - 1) * (cond - uncond)` is " + "identically zero on a recipe whose cfg_scale is 3.0"); + CHECK_MESSAGE(t.video_perturbed_forwards == 1, + "no perturbed forward ran, so `stg_scale * (cond - perturbed)` is identically " + "zero on a recipe whose stg_scale is 1.0"); + CHECK_MESSAGE(t.video_modality_forwards == 1, + "no isolated-modality forward ran, so `(modality_scale - 1) * (cond - mod)` is " + "identically zero on a recipe whose modality_scale is 3.0"); + // The guidance the ENGINE resolved, which is what the replay below uses. A + // build that resolved different scales fails the replay rather than agreeing + // with itself. + CHECK(t.video_guidance_cfg_scale == row.cfg_scale); + CHECK(t.video_guidance_stg_scale == row.stg_scale); + CHECK(t.video_guidance_rescale_scale == row.rescale_scale); + CHECK(t.video_guidance_modality_scale == row.modality_scale); + + // The perturbations REACHED the DiT, read off the mask that was handed over + // rather than off the guider params. A config that is BUILT and not HANDED + // OVER leaves the params untouched and renders. + CHECK(t.video_perturbed_blocks == std::vector{1}); + CHECK(t.video_audio_perturbed_blocks == std::vector{1}); + CHECK_MESSAGE(t.video_modality_skipped_a2v, + "the isolated-modality pass reached the DiT without SKIP_A2V_CROSS_ATTN, so it " + "is the conditional pass again (denoisers.py:130-136)"); + CHECK_MESSAGE(t.video_modality_skipped_v2a, + "the isolated-modality pass reached the DiT without SKIP_V2A_CROSS_ATTN"); + + const size_t n = t.video_first_latent.size(); + REQUIRE(n > 0); + const size_t tokens = t.video_first_timesteps.size(); + REQUIRE(tokens > 0); + const size_t width = n / tokens; + REQUIRE(width * tokens == n); + + // THE FIXTURE CAN DECIDE THIS AT ALL. The two candidate tensors for every arm + // are `latent - sigma*velocity` and `velocity`, and they coincide when the + // sample is zero. A REQUIRE, because nothing below discriminates once it fails. + const double latent_span = MaxAbsOf(t.video_first_latent); + REQUIRE_MESSAGE(latent_span > 1e-3, + "the step-0 sample is zero, so the two candidate tensors coincide and nothing " + "below discriminates"); + + // ── the equation, once per guidance pass ────────────────────────────────── + struct Arm { + const char* name; + const std::vector& velocity; + const std::vector& x0; + }; + const Arm arms[] = { + {"cond", t.video_first_cond_velocity, t.video_first_cond}, + {"uncond", t.video_first_uncond_velocity, t.video_first_uncond}, + {"perturbed", t.video_first_perturbed_velocity, t.video_first_perturbed}, + {"modality", t.video_first_modality_velocity, t.video_first_modality}, + }; + for (const Arm& arm : arms) { + INFO("arm = " << arm.name); + REQUIRE(arm.velocity.size() == n); + REQUIRE(arm.x0.size() == n); + + double velocity_span = 0.0; + double sigma_velocity_span = 0.0; + double err_x0 = 0.0; // |x0 - (latent - sigma*velocity)| -> 0 in x0 space + double err_v = 0.0; // |x0 - velocity| -> 0 in velocity space + for (size_t token = 0; token < tokens; ++token) { + // The PER-TOKEN sigma, which is what `X0Model.forward` uses + // (model.py:601-604 passes `video.timesteps`). Using the schedule scalar + // here would pass on a build that used it too, and that build re-noises + // every conditioned token. + const double sigma = static_cast(t.video_first_timesteps[token]); + for (size_t c = 0; c < width; ++c) { + const size_t i = token * width + c; + const double lat = static_cast(t.video_first_latent[i]); + const double vel = static_cast(arm.velocity[i]); + const double x0 = static_cast(arm.x0[i]); + velocity_span = std::max(velocity_span, std::abs(vel)); + sigma_velocity_span = std::max(sigma_velocity_span, std::abs(sigma * vel)); + err_x0 = std::max(err_x0, std::abs(x0 - (lat - sigma * vel))); + err_v = std::max(err_v, std::abs(x0 - vel)); + } + } + INFO("max|latent| = " << latent_span << " max|velocity| = " << velocity_span + << " max|sigma*velocity| = " << sigma_velocity_span + << " |x0 - (latent - sigma*velocity)| = " << err_x0 + << " |x0 - velocity| = " << err_v << " elements = " << n); + + // 1. `to_denoised` IS NOT THE IDENTITY ON THIS ARM. The second half of the + // non-vacuity guard, per arm: a zeroed velocity collapses the equation to + // `x0 == latent` and would let a stub satisfy it. + REQUIRE_MESSAGE(sigma_velocity_span > 1e-6, + "the DiT returned no velocity on this arm, so `to_denoised` is the identity " + "here and the two candidate tensors coincide"); + // 2. THE GUIDER WAS HANDED THE X0 PREDICTION, exactly. + CHECK_MESSAGE(err_x0 <= 1e-5 * latent_span, + "the tensor handed to `Ltx2MultiModalGuidance` on this arm is not " + "`latent - sigma*velocity`, which is what `X0Model.forward` returns " + "(model.py:590-604, #1039): residual " + << err_x0 << " against a tolerance of " << (1e-5 * latent_span)); + // 3. AND IT WAS NOT THE RAW VELOCITY. Said separately from check 2, because a + // build handing the guider some THIRD tensor fails 2 and would pass a lone + // "not the velocity" check; the pair says which of the two happened. + CHECK_MESSAGE(err_v > 1e-2 * latent_span, + "the tensor handed to `Ltx2MultiModalGuidance` on this arm IS the raw DiT " + "velocity, so the guidance is combined in velocity space and converted once " + "afterwards (#1039)"); + } + + // ── each arm is a DIFFERENT forward ─────────────────────────────────────── + // + // Without these, an arm whose CONTEXT or PERTURBATION never reached the DiT + // satisfies every check above: it is a perfectly converted x0 prediction of the + // conditional pass, and its guidance term is exactly zero. + CHECK_MESSAGE(MaxAbsDiffOf(t.video_first_uncond, t.video_first_cond) > 1e-6 * latent_span, + "the unconditional pass returned the conditional pass's own tensor, so the " + "negative context did not reach the forward"); + CHECK_MESSAGE(MaxAbsDiffOf(t.video_first_perturbed, t.video_first_cond) > 1e-6 * latent_span, + "the perturbed pass returned the conditional pass's own tensor, so the " + "self-attention perturbation did not reach the forward"); + CHECK_MESSAGE(MaxAbsDiffOf(t.video_first_modality, t.video_first_cond) > 1e-6 * latent_span, + "the isolated-modality pass returned the conditional pass's own tensor, so the " + "cross-attention perturbation did not reach the forward (transformer.py:335,366)"); + + // ── the guider's output is the guider's output ──────────────────────────── + // + // The four recorded arms, through the SHIPPED `Ltx2MultiModalGuidance` on the + // recipe's own params, must reproduce `video_first_denoised` bit for bit. This + // does not gate the guider's arithmetic — the seam case below does that — it + // gates that the pipeline handed the guider these tensors and passed its result + // on UNTOUCHED. A second `to_denoised` applied to the combination is invisible + // in every per-arm check above, because it moves nothing the guider was handed. + { + const std::vector replayed = vllm::Ltx2MultiModalGuidance( + row, t.video_first_cond.data(), t.video_first_uncond.data(), + t.video_first_perturbed.data(), t.video_first_modality.data(), static_cast(n)); + REQUIRE(replayed.size() == n); + const double worst = MaxAbsDiffOf(replayed, t.video_first_denoised); + INFO("max|replayed guidance - video_first_denoised| = " << worst); + // EXACT, not a tolerance: it is the same function over the same f32 inputs, + // so any non-zero residual is another operation this pipeline applied. + CHECK_MESSAGE(worst == 0.0, + "`video_first_denoised` is not `Ltx2MultiModalGuidance` over the four recorded " + "arms, so something else was applied to the guider's result (#1039)"); + // And the combination MOVED what it was handed, so the arms checked above are + // real inputs to it rather than recorded values beside one. + CHECK(t.video_first_denoised != t.video_first_cond); + } + + // ── the same combination, over arms REBUILT FROM THE RAW VELOCITIES ─────── + // + // WHY THIS IS NOT THE PREVIOUS CHECK AGAIN. The replay above is fed the arms + // the seam recorded, so anything applied to EVERY arm on the way out of the + // forward is invisible to it: the replay and the pipeline agree because they + // agree about the same altered inputs. `post_process_latent` applied per arm + // instead of once to the guider's result is exactly that shape, and it is not + // hypothetical -- upstream applies it in the LOOP (utils/samplers.py:35), one + // level above the denoiser, and applying it a level lower is the obvious + // simplification. + // + // It is also invisible to the per-arm invariant, and that took working out. + // `post_process_latent` is `x*mask + clean*(1-mask)`, so it only moves tokens + // whose denoise mask is 0 -- and on such a token the schedule sigma is 0 too + // (`timesteps_from_mask`, utils/helpers.py:494-503), so the invariant reads + // `x0 == latent`, and a conditioned token's `latent` IS its clean value. The + // two placements therefore agree token by token and differ only through + // `cond.std()` and `pred.std()`, which the rescale computes over the WHOLE + // tensor and which change for every element at once. + // + // Rebuilding the arms from `latent` and the raw velocities is independent of + // anything applied to the arms, so it sees that. It is exact rather than + // approximate because it repeats `ToDenoised`'s own arithmetic: the subtraction + // in double, the store in f32. + { + const auto rebuild = [&](const std::vector& velocity) { + std::vector out(n); + for (size_t token = 0; token < tokens; ++token) { + const double sigma = static_cast(t.video_first_timesteps[token]); + for (size_t c = 0; c < width; ++c) { + const size_t i = token * width + c; + out[i] = static_cast(static_cast(t.video_first_latent[i]) - + sigma * static_cast(velocity[i])); + } + } + return out; + }; + const std::vector c = rebuild(t.video_first_cond_velocity); + const std::vector u = rebuild(t.video_first_uncond_velocity); + const std::vector p = rebuild(t.video_first_perturbed_velocity); + const std::vector m = rebuild(t.video_first_modality_velocity); + const std::vector replayed = vllm::Ltx2MultiModalGuidance( + row, c.data(), u.data(), p.data(), m.data(), static_cast(n)); + const double worst = MaxAbsDiffOf(replayed, t.video_first_denoised); + INFO("max|guidance over rebuilt arms - video_first_denoised| = " << worst); + CHECK_MESSAGE(worst == 0.0, + "the guider's result is not `Ltx2MultiModalGuidance` over `latent - sigma*v` " + "for the four RAW velocities, so something was applied to the arms between the " + "forward and the combination"); + } + + // ── `post_process_latent` came AFTER the guider, and the sampler consumed + // exactly what it produced ───────────────────────────────────────────── + // + // `_step_state` applies `post_process_latent(denoised, ...)` to the DENOISER's + // result (utils/samplers.py:35), never per arm inside it. On this render no + // token is conditioned, so the two tensors coincide — asserted rather than + // assumed, because it is what makes the Euler recovery below a statement about + // `video_first_denoised`. + REQUIRE(t.video_first_stepper_input.size() == n); + CHECK(t.video_first_stepper_input == t.video_first_denoised); + + { + // `Ltx2EulerStep` is `x + (x - denoised)/sigma * (sigma_next - sigma)` + // (`EulerDiffusionStep`, ltx-pipelines utils/blocks.py:524-527). Recovering + // `video_first_next_latent` from `video_first_stepper_input` pins WHICH + // tensor the step was handed: `ToDenoised` applied a second time between the + // recording and the step leaves every field above untouched and moves only + // this one. + // + // The schedule is re-derived from the shared seam rather than read off the + // render, and tied to it by the sigma the render recorded. + REQUIRE(t.schedule_tokens > 0); + const std::vector sigmas = vllm::Ltx2SigmaSchedule(/*steps=*/2, t.schedule_tokens); + REQUIRE(sigmas.size() == 3); + const double sigma = t.video_first_sigma; + REQUIRE(sigma > 0.0); + REQUIRE(static_cast(sigmas[0]) == sigma); + const double dt = static_cast(sigmas[1]) - static_cast(sigmas[0]); + REQUIRE_MESSAGE(std::abs(dt) > 1e-3, + "the first two sigmas coincide, so the Euler step is the identity and this " + "check cannot see what it consumed"); + REQUIRE(t.video_first_next_latent.size() == n); + double worst = 0.0; + double scale = 0.0; + for (size_t i = 0; i < n; ++i) { + const double lat = static_cast(t.video_first_latent[i]); + const double den = static_cast(t.video_first_stepper_input[i]); + const double expected = lat + (lat - den) / sigma * dt; + worst = + std::max(worst, std::abs(static_cast(t.video_first_next_latent[i]) - expected)); + scale = std::max(scale, std::abs(expected)); + } + INFO("sigma = " << sigma << " -> " << sigmas[1] + << " max|next - Euler(latent, denoised)| = " << worst + << " scale = " << scale); + REQUIRE_MESSAGE(scale > 1e-3, + "the recomputed Euler output is zero, so the residual below bounds nothing"); + CHECK_MESSAGE(worst <= 1e-5 * scale, + "the latent `Ltx2EulerStep` wrote is not the step over the recorded denoised " + "prediction, so the sampler was handed some other tensor (#1039): residual " + << worst << " against a tolerance of " << (1e-5 * scale)); + } +} + +TEST_CASE("ltx2 one_stage: rescale_scale 0 is the control, and the MODALITY term is in it") { + // #1039's control on the VIDEO row, which carries a term the T2A control could + // not: `modality_scale` is 3.0 here and pinned to 1.0 there + // (t2a_one_stage.py:200-202), so the isolated-modality arm has never been in a + // space control before. + // + // The case above would be testing something OTHER than the defect if it also + // fired at `rescale_scale = 0`, because `MultiModalGuider.calculate`'s linear + // terms (guiders.py:261-266) are invariant under `x0 = latent - sigma*v`: + // + // latent - sigma*(c + a(c-u) + b(c-p) + d(c-m)) + // == x0c + a(x0c-x0u) + b(x0c-x0p) + d(x0c-x0m) + // + // The rescale at `:268-271` is the only part that is not. + const int64_t n = 512; + std::vector latent(static_cast(n)); + std::vector v_cond(static_cast(n)); + std::vector v_uncond(static_cast(n)); + std::vector v_ptb(static_cast(n)); + std::vector v_mod(static_cast(n)); + // Deterministic and NON-CONSTANT. A zero latent erases `(factor - 1) * latent` + // entirely and a constant one reduces it to a uniform offset; either would make + // the disagreement below unmeasurable and the control meaningless. + uint64_t s = 0x9E3779B97F4A7C15ULL; + const auto next = [&s]() { + s ^= s << 13; + s ^= s >> 7; + s ^= s << 17; + return static_cast(static_cast(s >> 11) / 9007199254740992.0 * 2.0 - 1.0); + }; + for (int64_t i = 0; i < n; ++i) { + const size_t j = static_cast(i); + latent[j] = 2.0F * next(); + v_cond[j] = next(); + v_uncond[j] = next(); + v_ptb[j] = next(); + v_mod[j] = next(); + } + const float sigma = 0.83F; + std::vector x_cond(static_cast(n)); + std::vector x_uncond(static_cast(n)); + std::vector x_ptb(static_cast(n)); + std::vector x_mod(static_cast(n)); + for (int64_t i = 0; i < n; ++i) { + const size_t j = static_cast(i); + x_cond[j] = latent[j] - sigma * v_cond[j]; + x_uncond[j] = latent[j] - sigma * v_uncond[j]; + x_ptb[j] = latent[j] - sigma * v_ptb[j]; + x_mod[j] = latent[j] - sigma * v_mod[j]; + } + + // The 2.4/2.5 VIDEO row, read from the shared recipe table rather than typed. + const vllm::Ltx2PipelineRecipe recipe = vllm::ResolveLtx2PipelineRecipe("one_stage", "2.5"); + REQUIRE(recipe.phases.size() == 1); + vllm::Ltx2MultiModalGuiderParams params = recipe.phases[0].video_guidance; + REQUIRE(params.rescale_scale == 0.7); + REQUIRE(params.modality_scale == 3.0); + + const auto compare = [&](double rescale) { + params.rescale_scale = rescale; + // Upstream's shape: combine the X0 predictions. + const std::vector x0_space = vllm::Ltx2MultiModalGuidance( + params, x_cond.data(), x_uncond.data(), x_ptb.data(), x_mod.data(), n); + // The shape a port reaches for by accident: combine the VELOCITIES and + // convert once after. + const std::vector v_space = vllm::Ltx2MultiModalGuidance( + params, v_cond.data(), v_uncond.data(), v_ptb.data(), v_mod.data(), n); + double worst = 0.0; + double scale = 0.0; + for (int64_t i = 0; i < n; ++i) { + const size_t j = static_cast(i); + const double converted = static_cast(latent[j]) - + static_cast(sigma) * static_cast(v_space[j]); + worst = std::max(worst, std::abs(static_cast(x0_space[j]) - converted)); + scale = std::max(scale, std::abs(static_cast(x0_space[j]))); + } + REQUIRE(scale > 1e-3); + return worst / scale; + }; + + const double at_zero = compare(0.0); + const double at_default = compare(0.7); + INFO("relative disagreement: at rescale 0.0 = " << at_zero + << " at rescale 0.7 = " << at_default); + // AT 0.0 THE TWO SPACES ARE THE SAME FUNCTION, to f32 rounding — with the + // modality term present, which is the half this control adds. + CHECK(at_zero < 1e-4); + // AT THE SHIPPED 0.7 THEY ARE NOT, by orders of magnitude more. + CHECK(at_default > 1e-2); + CHECK(at_default > 100.0 * at_zero); +} + +TEST_CASE("ltx2 guided video: the refusals that would otherwise RENDER (#1092)") { + Workspace ws; + + SUBCASE("an unconditional forward with no negative conditioning is refused BY NAME") { + // The positive embeds alone, which is what every engine here loaded before + // this row. `cfg_scale = 3.0` asks for a forward whose context does not + // exist; serving the POSITIVE context twice would make the whole CFG term + // exactly zero and render an unguided clip wearing a guided configuration. + vllm::multimodal::VideoModelParams mp = OneStageParams(ws.paths); + mp.extras.erase(vllm::multimodal::kLtx2NegativePromptEmbedsExtra); + mp.extras.erase(vllm::multimodal::kLtx2NegativeAudioPromptEmbedsExtra); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + try { + (void)engine->Generate(OneStageGen(ws.root + "/no_negative")); + FAIL("a cfg_scale of 3.0 with no negative conditioning must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("unconditional forward") != std::string::npos); + CHECK(msg.find(vllm::multimodal::kLtx2NegativePromptEmbedsExtra) != std::string::npos); + } + } + + SUBCASE("cfg_scale 1.0 turns the unconditional pass off instead of needing one") { + // The other half of the branch above, and what makes it a statement about the + // GUIDER rather than a blanket requirement: `do_unconditional_generation` is + // `not isclose(cfg_scale, 1.0)` (guiders.py:275-277), so at 1.0 there is no + // pass and nothing to encode. + vllm::multimodal::VideoModelParams mp = OneStageParams(ws.paths); + mp.extras.erase(vllm::multimodal::kLtx2NegativePromptEmbedsExtra); + mp.extras.erase(vllm::multimodal::kLtx2NegativeAudioPromptEmbedsExtra); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + vllm::multimodal::VideoGenParams gen = OneStageGen(ws.root + "/cfg_one"); + gen.extras[vllm::multimodal::kLtx2VideoCfgScaleExtra] = "1.0"; + gen.extras[vllm::multimodal::kLtx2AudioCfgScaleExtra] = "1.0"; + const vllm::multimodal::VideoResult result = engine->Generate(gen); + CHECK(result.width == 64); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + CHECK(t.video_uncond_forwards == 0); + // And the OTHER two passes still ran, so this subcase turned off exactly one + // arm rather than the guidance. + CHECK(t.video_perturbed_forwards == 1); + CHECK(t.video_modality_forwards == 1); + } + + SUBCASE("an STG block this checkpoint does not have is refused, not silently ignored") { + // `Perturbation.is_perturbed` is `block in self.blocks` + // (guidance/perturbations.py:26-33), so a block index past the end perturbs + // NOTHING: the perturbed forward returns the conditional pass's own tensor + // and `stg_scale * (cond - perturbed)` is exactly zero. The render is finite, + // the right size, and carries no spatio-temporal guidance whatever. + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(OneStageParams(ws.paths)); + vllm::multimodal::VideoGenParams gen = OneStageGen(ws.root + "/stg_oob"); + gen.extras[vllm::multimodal::kLtx2VideoStgBlocksExtra] = "28"; + try { + (void)engine->Generate(gen); + FAIL("block 28 on a two-block DiT perturbs nothing and must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("stg_blocks") != std::string::npos); + CHECK(msg.find("exactly zero") != std::string::npos); + } + } + + SUBCASE("a recipe that fixes its guidance refuses the override rather than applying it") { + // `allow_guidance_override = false` on the distilled two-stage recipe + // (ltx2_recipes.py:125-158), whose scales are distilled INTO the weights. + // Until this row nothing read that field at all. + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras["upsampler_path"] = ws.paths.upsampler; // the two-stage recipe's phase 1 + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/fixed_guidance"); + gen.extras[vllm::multimodal::kLtx2VideoCfgScaleExtra] = "5.0"; + try { + (void)engine->Generate(gen); + FAIL("the distilled recipe fixes its guidance and must refuse the override"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("fixes its own guidance") != std::string::npos); + CHECK(msg.find(vllm::multimodal::kLtx2VideoCfgScaleExtra) != std::string::npos); + } + } + + SUBCASE("the DISTILLED recipe runs ONE forward, which is what SimpleDenoiser is") { + // The guided seam is on every video render now, so the recipes upstream + // denoises with `SimpleDenoiser` (distilled.py:266,295) must still issue one + // forward per step. Their guiders are `Ltx2MultiModalGuiderParams`'s own + // defaults, which is `_POSITIVE_ONLY_GUIDER` (denoisers.py:25-28). + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras["upsampler_path"] = ws.paths.upsampler; // the two-stage recipe's phase 1 + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + (void)engine->Generate(FixtureGen(ws.root + "/distilled_simple")); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.video_guided); + CHECK(t.video_cond_forwards == 1); + CHECK(t.video_uncond_forwards == 0); + CHECK(t.video_perturbed_forwards == 0); + CHECK(t.video_modality_forwards == 0); + // And the guider was the identity over that one pass, so this recipe's + // trajectory is unchanged by the seam. + CHECK(t.video_first_denoised == t.video_first_cond); + } +} From 420f6b4741ef8b5faae0874ccfb2d622d7c4a7d6 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 02:56:10 +0000 Subject: [PATCH 3/5] test(LTX25-GUIDED-VIDEO): the mutation that stayed green was an identity, and proving it took a conditioned render (#1092) The mutation pass on this row ran twelve mutations. Eleven went RED. The twelfth -- `post_process_latent` ADDED to each guidance arm, with the after-guider application left in place -- stayed green at 71 cases / 2145 assertions / exit 0, and this commit is what came of chasing it. The first reading was that the gate could not see it, so the gate grew a check that replays `Ltx2MultiModalGuidance` over arms REBUILT from `latent` and the four RAW velocities. That replay is independent of anything applied to every arm alike, which is exactly the shape of the mutation. It did not move the green either, and that was the clue. The second reading is that the mutation changes nothing. `post_process_latent` is `x*mask + clean*(1-mask)` (`utils/helpers.py:462-464`), so it can only touch a token whose denoise mask is 0. Such a token's per-token sigma is 0 (`timesteps_from_mask`, `:494-503`), so `X0Model` returns `latent - 0*v`, which is `latent`; and a conditioned token's `latent` IS its clean value, which the conditioner writes and the Euler step preserves. Every arm already equals what post-processing would write. ## The tempting next sentence is false, and that is the test this adds "So it does not matter where it is applied" does not follow. The guider's rescale (`guiders.py:268-271`) is a scalar over the WHOLE tensor, so it multiplies the conditioned tokens too: every guidance term is zero on a token where all four arms agree, leaving `pred = latent * factor` there. The after-guider `post_process_latent` is what pins those tokens back to `clean`. Remove it, or move it down into the denoiser, and the conditioned tokens leave the step scaled by a number nobody asked for, on a render that finishes. The unconditioned case could not see any of this: with every mask entry at 1, `post_process_latent` is a literal no-op and every placement of it passes. So this adds a CONDITIONED `one_stage` case, on the same guided configuration with an image conditioning, which asserts both halves -- that no arm was touched, and that the guider's result was, on exactly the mask-0 tokens -- behind a positive control that the post-process moved something at all. MEASURED: the MOVE mutation is RED against it (71 cases / 2133 assertions / exit 1, 0 compile errors, `1 file changed, 4 insertions(+), 3 deletions(-)`). The ADD mutation stays green, and the arm assertion now states why in an executable form rather than leaving a green unexplained. ## And one guard at the call rather than at the recipe `Ltx2DitForwardDevice` has no `perturbations` parameter, so the argument list silently drops one. The pre-loop refusal is a statement about the RECIPE; this adds the statement about the CALL, in the `X0Model` lambda itself, because the two are not the same check and a pass that reached the device arm with a perturbation would leave the STG and modality terms at exactly zero with nothing in the frames, the shapes or the counts to show for it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-guided-video.md | 70 +++++++++++- src/vllm/multimodal/ltx2_video.cpp | 12 ++ tests/vllm/multimodal/test_ltx2_video.cpp | 133 ++++++++++++++++++++++ 3 files changed, 214 insertions(+), 1 deletion(-) diff --git a/.agents/specs/ltx25-guided-video.md b/.agents/specs/ltx25-guided-video.md index f034c3abe..bfd8c2fd0 100644 --- a/.agents/specs/ltx25-guided-video.md +++ b/.agents/specs/ltx25-guided-video.md @@ -436,6 +436,74 @@ Stop and report `NEEDS_DECISION` rather than narrowing silently if: guidance arithmetic is gated against upstream **source**, not against upstream **output**, and that is the ceiling on this row's evidence. +## 10. What the implementation actually did, against §4 + +Three things moved from the design, and each is recorded here rather than left +for a reader to diff. + +**The seam runs on EVERY video render, not only the guided ones.** §4 left this +open. The implementation routes the phase loop through `Ltx2GuidedDenoise` +unconditionally, because a phase whose recipe sets no guidance keeps +`Ltx2MultiModalGuiderParams`'s own defaults and those ARE +`_POSITIVE_ONLY_GUIDER` (`denoisers.py:25-28`): one pass, and a `calculate` whose +every term is zero. Upstream selects `SimpleDenoiser` by PIPELINE +(`distilled.py:266,295`) rather than by params, and the two agree here because +the recipes that select it are exactly the recipes whose guidance is the no-op +one. **Measured, not argued:** every existing golden in `test_ltx2_video` held +unchanged across the change, including the `distilled_two_stage` renders and the +DFR and retake ones. + +**Two refusals that upstream does not have.** Both are cases where upstream's +behaviour is a silent zero rather than an error, and where the render finishes: + +- an `stg_blocks` list naming no block this checkpoint has. Upstream's + `Perturbation.is_perturbed` is a membership test (`perturbations.py:26-33`), so + `[28]` on a two-block DiT perturbs nothing and leaves + `stg_scale * (cond - perturbed)` at exactly zero. Upstream never meets it + because it only runs 48-block checkpoints; this port runs reduced ones. +- an EMPTY `stg_blocks` beside a non-zero STG scale, for the same reason. + +**A cross perturbation that differs between blocks is refused rather than +widened.** `Ltx2DitPerturbation` carries one boolean per direction, which is +sound only while the config says the same thing on every block — which +`blocks=None` guarantees. That is checked in the seam rather than assumed. + +## 11. Outcome + +Landed on `row/LTX25-GUIDED-VIDEO`. Twelve mutations, eleven RED and one proven +an IDENTITY; the table with all three facts per mutation is in the pull request +body. + +**The one green, and what chasing it changed.** `post_process_latent` ADDED per +arm, with the after-guider application left in place, runs the whole suite to +71 cases / 2145 assertions / exit 0. Two readings of that green were tried and +the first one was wrong. + +The first reading was that the gate could not see it, so a check was added that +replays the guider over arms REBUILT from `latent` and the four raw velocities -- +independent of anything applied to the arms alike. It did not move the green +either, which was the clue. + +The second reading, measured rather than argued, is that the mutation is a NO-OP. +`post_process_latent` is `x*mask + clean*(1-mask)`, so it can only touch a token +whose denoise mask is 0; such a token's per-token sigma is 0 +(`timesteps_from_mask`, `utils/helpers.py:494-503`), so `X0Model` returns +`latent - 0*v`, which is `latent`; and a conditioned token's `latent` IS its +clean value. Every arm already equals what post-processing would write. + +**What that produced is a better test, not a waived mutation.** The tempting +proof of the above -- "so applying it anywhere is the same" -- is false for the +guider's RESULT, because the rescale (`guiders.py:268-271`) is a scalar over the +whole tensor and multiplies the conditioned tokens too. Take the after-guider +application away, or move it down into the denoiser, and those tokens leave the +step scaled by a number nobody asked for. The row therefore carries a CONDITIONED +`one_stage` case which asserts both halves: that no arm was touched, and that the +guider's result was, on exactly the mask-0 tokens. The MOVE mutation is RED +against it (71 cases / 2133 assertions / exit 1); the ADD mutation stays green +and the case now says why in an assertion instead of leaving it unexplained. + +**A residual, stated.** Nothing here compares against a running oracle; see §0. + ## Now -Spec committed. Implementation and gate follow on the same pull request. +Implementation and gate landed on the branch; the row awaits a fresh review. diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index ccd137760..c7b7d15f8 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -3425,6 +3425,18 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // loop rather than served an unperturbed forward. const Ltx2X0Model x0_model = [&](const Ltx2ModalityInput* v, const Ltx2ModalityInput* a, const Ltx2DitPerturbation* p) { + // The refusal above is a statement about the RECIPE; this is a statement + // about the CALL, and the two are not the same check. A pass that reached + // here with a perturbation on the device arm would have it silently + // dropped by the argument list below, which is the shape of defect this + // file keeps finding: correct output for the wrong reason, with the STG + // and modality terms at exactly zero and nothing in the frames, the + // shapes or the counts to show for it. + VT_CHECK(!im.on_device || p == nullptr, + "ltx2 video: a perturbed forward reached the device-resident arm, where " + "`Ltx2DitForwardDevice` has no `perturbations` argument to take it. The guidance " + "resolution refuses this before the loop, so reaching it is a defect rather than " + "a bad request. Owed by row LTX25-GUIDED-VIDEO (#1092)."); const Ltx2DitOutputs velocity = im.on_device ? Ltx2DitForwardDevice(*im.queue, im.dit.params, im.dit.weights, v, a, im.compute_dtype) diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 7b1924958..88a9e59ad 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -6135,6 +6135,139 @@ TEST_CASE("ltx2 one_stage: rescale_scale 0 is the control, and the MODALITY term CHECK(at_default > 100.0 * at_zero); } +TEST_CASE("ltx2 one_stage: post_process_latent runs AFTER the guider, not per arm (#1092)") { + // WHERE `post_process_latent` IS APPLIED, gated on a render that has something + // for it to move. The unconditioned case above cannot see this at all: every + // denoise mask entry is 1 there, so `x*mask + clean*(1-mask)` + // (utils/helpers.py:462-464) is a literal no-op and any placement of it passes. + // + // TWO THINGS ARE TRUE HERE AND THEY ARE EASY TO CONFUSE, so both are asserted. + // + // (1) Applying it to each ARM is an IDENTITY, and that is not a gap in this + // case -- it is arithmetic. A conditioned token arrives with its per-token + // sigma at 0 (`timesteps_from_mask`, utils/helpers.py:494-503), so + // `X0Model` returns `latent - 0*v`, which is `latent`; and a conditioned + // token's `latent` IS its clean value, which is what the conditioner wrote + // and what the Euler step preserves. So every arm already equals what + // post-processing would write. MEASURED: adding it per arm runs the whole + // suite to 71 cases / 2145 assertions / exit 0, and the arm assertion below + // is what says WHY rather than leaving the green unexplained. + // + // (2) Applying it after the GUIDER is emphatically not an identity, and that is + // the thing worth gating. The guider's rescale (guiders.py:268-271) is a + // scalar over the WHOLE tensor, so it multiplies the conditioned tokens too + // -- `pred = latent * factor` there, because every guidance term is zero on + // a token where all four arms agree. `post_process_latent` is what pins + // them back to `clean`. Take it out, or move it a level down into the + // denoiser, and the conditioned tokens leave the step scaled by a number + // nobody asked for, on a render that finishes. + // + // So this case asserts that the arms were NOT touched and that the guider's + // result WAS, on exactly the mask-0 tokens. + Workspace ws; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(OneStageParams(ws.paths)); + REQUIRE(engine != nullptr); + vllm::multimodal::VideoGenParams gen = OneStageGen(ws.root + "/conditioned"); + gen.first_frame_ppm = ConditioningPpm(20, 28, 1); + gen.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "0"; + (void)engine->Generate(gen); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + REQUIRE(t.video_guided); + // The same four arms as the case above, not a degenerate set. + CHECK(t.video_uncond_forwards == 1); + CHECK(t.video_perturbed_forwards == 1); + CHECK(t.video_modality_forwards == 1); + + const size_t tokens = t.video_first_timesteps.size(); + REQUIRE(tokens > 0); + const size_t n = t.video_first_latent.size(); + REQUIRE(n > 0); + const size_t width = n / tokens; + REQUIRE(width * tokens == n); + + // NON-VACUITY, both ends. With no conditioned token this case is the one above + // again; with every token conditioned there is nothing left to denoise. + size_t conditioned = 0; + for (size_t token = 0; token < tokens; ++token) { + if (t.video_first_timesteps[token] == 0.0F) ++conditioned; + } + INFO("conditioned tokens = " << conditioned << " of " << tokens); + REQUIRE_MESSAGE(conditioned > 0, + "no token arrived at the denoiser with a zero timestep, so the image " + "conditioning did not reach the denoise mask and this case tests nothing"); + REQUIRE_MESSAGE(conditioned < tokens, + "EVERY token is conditioned, so there is nothing left to denoise"); + + // (2), AND THE POSITIVE CONTROL FOR THE WHOLE CASE. `post_process_latent` MOVES + // something on this render: the tensor the stepper was handed is not the + // guider's own output. Without this, every assertion here would be satisfied by + // a render where post-processing happened to be a no-op, which is exactly what + // the unconditioned case above is. + REQUIRE(t.video_first_stepper_input.size() == n); + REQUIRE_MESSAGE(t.video_first_stepper_input != t.video_first_denoised, + "`post_process_latent` changed nothing on this render, so it cannot matter " + "WHERE it was applied and this case discriminates nothing"); + + // AND IT MOVED ONLY THE CONDITIONED TOKENS, which is what makes the next + // assertion a statement about placement rather than about some third operation. + for (size_t token = 0; token < tokens; ++token) { + const bool is_conditioned = t.video_first_timesteps[token] == 0.0F; + for (size_t c = 0; c < width; ++c) { + const size_t i = token * width + c; + const bool moved = t.video_first_stepper_input[i] != t.video_first_denoised[i]; + if (moved == is_conditioned) continue; + INFO("token = " << token << " channel = " << c); + FAIL_CHECK("`post_process_latent` moved a token whose denoise mask does not match: it is " + "`x*mask + clean*(1-mask)` and must move exactly the mask-0 tokens"); + break; + } + } + + // (1). Every arm the forward returned is `latent - sigma*velocity`, INCLUDING + // on the conditioned tokens, where that is `latent` itself. This is what makes + // the per-arm placement an identity rather than an undetected defect, and it is + // asserted rather than argued because the argument depends on a conditioned + // token's `latent` being its clean value -- a property of the CONDITIONER, one + // file away, that nothing here would otherwise hold. + const std::vector* arms[] = {&t.video_first_cond, &t.video_first_uncond, + &t.video_first_perturbed, &t.video_first_modality}; + const std::vector* velocities[] = { + &t.video_first_cond_velocity, &t.video_first_uncond_velocity, + &t.video_first_perturbed_velocity, &t.video_first_modality_velocity}; + const char* names[] = {"cond", "uncond", "perturbed", "modality"}; + for (size_t k = 0; k < 4; ++k) { + INFO("arm = " << names[k]); + REQUIRE(arms[k]->size() == n); + REQUIRE(velocities[k]->size() == n); + double worst = 0.0; + for (size_t token = 0; token < tokens; ++token) { + const double sigma = static_cast(t.video_first_timesteps[token]); + for (size_t c = 0; c < width; ++c) { + const size_t i = token * width + c; + // `ToDenoised` subtracts in double and STORES f32, so the expectation is + // rounded the same way. Comparing against the unrounded double leaves one + // ULP of disagreement -- measured at 5.96e-08, which is 2^-24 -- and a + // tolerance wide enough to absorb it would also absorb a real defect an + // order of magnitude away. + const float expected = static_cast(static_cast(t.video_first_latent[i]) - + sigma * static_cast((*velocities[k])[i])); + worst = std::max(worst, std::abs(static_cast((*arms[k])[i]) - + static_cast(expected))); + } + } + INFO("max|arm - (latent - sigma*velocity)| = " << worst); + CHECK_MESSAGE(worst == 0.0, + "this arm is not `latent - sigma*velocity` on every token, so something was " + "applied to it between the forward and the guider -- and if that something is " + "`post_process_latent`, it has stopped being an identity on the arms and the " + "per-arm placement is now a real divergence rather than a harmless one"); + } +} + TEST_CASE("ltx2 guided video: the refusals that would otherwise RENDER (#1092)") { Workspace ws; From d26eda44ec2fbc316ab71f049ee2ea64716490c4 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 08:30:36 +0000 Subject: [PATCH 4/5] fix(LTX25-GUIDED-VIDEO): the two cross flags were gated together, and three mutations proved it (#1092) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fresh review returned CHANGES REQUESTED on `420f6b474`: one blocking finding and ten non-blocking ones. It reproduced the core of the row exactly — the seam, the x0 space, all four arms, the reachability, the gate — and independently confirmed the live defect the row fixes, so none of that is revisited here. B1, THE BLOCKING ONE. `Ltx2DitPerturbation`'s two cross booleans were gated together and not per direction. Three mutations of `ltx2_dit.cpp`, each built clean with zero compile errors and each exit status captured directly, were GREEN over the whole `test_ltx2_video` binary: the DiT ignoring `video_cross_attn_skip_all` (M12), ignoring `audio_cross_attn_skip_all` (M13), and SWAPPING which flag gates which direction (M15). Only ignoring BOTH (M14) was caught. A build that applies exactly one direction, or applies both to the wrong ones, renders — on the DEFAULT video arm, whose `modality_scale` is 3.0 — with the isolated-modality term half wrong. Two things made the shipped-path case blind to it. The end-to-end `MaxAbsDiffOf(video_first_modality, video_first_cond)` still fires with one direction applied, because the modality pass still differs from `cond`. And `Ltx2ConditioningTrace::video_modality_skipped_{a2v,v2a}` is assigned from the perturbation struct THE SEAM BUILT (`ltx2_denoisers.cpp:315-316`), so it records what was handed over and nothing about what the DiT did with it — while its message claimed the latter. That message is corrected rather than left to overstate. The repair is test-only. Separation comes from upstream's own predicates (`transformer.py:265-269`): `run_a2v` needs the VIDEO stream enabled and the audio stream merely PRESENT, and `run_v2a` the reverse, so a forward with `audio->enabled = false` runs A2V alone and one with `video->enabled = false` runs V2A alone. Each row asserts BOTH halves — the flag for that direction moves the written stream, and the flag for the other direction leaves it bit-identical — which is what makes the swap detectable and not only the omission. The `ltx2.h` disclosure that called M15 untestable is retired: its premise (both directions are off together on the shipped path) was true and its conclusion did not follow, because nothing obliges the separating test to use the shipped combination. M12, M13, M14 and M15 are now all RED. B3, the other finding with product code behind it. The EMPTY `stg_blocks` refusal is a real divergence and its recorded reason was wrong. Measured at Lightricks/LTX-2 `fd4ded7f`: `ltx-pipelines/docs/multimodal-guidance.md:13` documents "Set to `[]` to disable STG" in the same table and idiom as `stg_scale` -> 0.0; the field DEFAULTS to `[]` (`guiders.py:204`); the flags are `nargs="*"` (`args.py:979-985`, `:1039-1045`, `:1107-1113`) so the empty list has a CLI spelling, and `nargs="+"` was the one-character way to forbid it; `LTX_2_3_HQ_PARAMS` SHIPS it on both modalities (`constants.py:105`, `:113`); and a whole-tree search found no validation of `stg_blocks` at all, with the null results recorded. `blocks=None` means EVERY block and `blocks=[]` means NO block (`perturbations.py:26-33`), and `ApplyStgBlocksExtra` exists to keep PRESENT-and-empty distinct from ABSENT — which the refusal then made unreachable. Dropped in `ApplyGuidanceOverrides` and exempted in `check_reaches_a_block`; the out-of-range refusal stays, because that is a request disagreeing with the CHECKPOINT rather than a caller asking for nothing. Upstream does not skip the pass either (`do_perturbed_generation` reads `stg_scale` alone, `guiders.py:279-281`), so the new case asserts the pass RAN, perturbed no block, and returned `cond` bit for bit. One sub-claim of B3 is REJECTED on evidence. It argued that `audio_stg_blocks=""` is still accepted on `t2a_one_stage`. That path does return before `ApplyGuidanceOverrides`, and the request is still refused — by `ltx2_t2a.cpp:203-214`, which `git log -S` puts on `main` at `0b0b8900f` with #1032, not on this branch. So there is no asymmetry today; both arms refuse and both diverge from upstream. Fixing the video half creates one, which is why #1111 is filed, indexed, and listed under `## Owed` rather than left implied. It is not fixed in flow because it changes a landed row's gated behaviour and one of its cases. B6, the anchors, re-derived against `fd4ded7f` from the sentence making each claim rather than by reading text out of the cited span. `_guided_denoise` is 61-211 and not 62-207; `enabled = not skip` is at 158 and 168, where 151 and 161 are the `= None` initializers; the V2A guard is 367 and 366 is blank; the batched config is built at 182-187, where 172-176 is a comment plus the per-sample replication; the partial blend is 572-573; the one `PromptEncoder` call is 166-174; `default_1_stage_arg_parser` is 930-1067 with its guider flags at 947-1066; `cross_attn_skip_all` is DECLARED at `transformer_args.py:70` and 118 is a call site; `modality_scale = 3.0` is at `constants.py:54, :64` with `_PARAMS_SINCE_VERSION` at 130-133, and the cited 40-80 covers neither; `CFGGuider` and `STGGuider` are 11-27 and 56-74; the `perturbations` ARGUMENT is `model.py:493` and 492 is the `def`. No gate protects a spec anchor (#632), so the 43 replacements were applied by a script that asserts the expected hit count per edit and refuses the whole run on a mismatch; two were caught that way and re-derived. The rest. B2: `docs/FEATURES.md` still called T2A "the only GUIDED arm", which this row's own new row two lines below made false — corrected inside the existing cell at 202 of 220 chars, with the page's prose-paragraph count unchanged at 21 of 21, because adding a paragraph there re-reds `main` for the whole repo (#1055). B4: `INFO("arm = " << arm.name)` printed `arm = 1`, doctest stringifying a `const char*` through its bool overload, so a single-arm regression could not be attributed from the output — wrapped in `std::string` at all three sites in the file. B5: the rescale control's modality claim is structurally true and numerically inert, and the case now MEASURES that (`4.054e-01` at `modality_scale` 3.0 against `4.118e-01` at 1.0) instead of implying coverage it does not provide; the modality arm's gate is the per-arm invariant, which M4 turns red. B7: "the seam cannot be handed a velocity" is caller discipline and not a type guarantee, since `Ltx2X0Outputs` carries the velocity beside the prediction — the claim is restated and the code left alone, because dropping the velocity would delete what the invariant is checked against. B10: the new `docs/USAGE.md` section gains the `/v1/videos` caveat its two siblings carry, a flag-to-extra table with the raw key spellings an ABI caller needs, and the empty-list behaviour B3 decided. B8: `origin/main` is merged in and the gate rerun on the merged tree. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 3 +- .agents/specs/ltx25-guided-video.md | 223 ++++++++++-- docs/FEATURES.md | 2 +- docs/USAGE.md | 58 +++- examples/ltx2_gen/main.cpp | 2 +- include/vllm/model_executor/models/ltx2.h | 45 ++- .../model_executor/models/ltx2_denoisers.h | 20 +- include/vllm/multimodal/ltx2_video.h | 4 +- .../model_executor/models/ltx2_denoisers.cpp | 21 +- src/vllm/model_executor/models/ltx2_dit.cpp | 6 +- src/vllm/multimodal/ltx2_video.cpp | 63 ++-- tests/vllm/multimodal/test_ltx2_video.cpp | 327 +++++++++++++++++- 12 files changed, 655 insertions(+), 119 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 4e65e2cc8..104f05f09 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -307,7 +307,6 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1066](https://github.com/mudler/vllm.cpp/issues/1066) | `ENG-EXPERT-STREAM` | `Qwen35ExpertStream` (`src/vllm/model_executor/models/qwen3_5.cpp`) is a **process-lifetime singleton** and keyed its slot cache on `(TowerId(base), expert)`, where `base` is the expert tower's host buffer **ADDRESS**. Its own comment stated the premise and drew the wrong conclusion: "A tower's identity is its base pointer, which is stable for the model's life". The premise is true; the conclusion does not follow, because the CACHE is not scoped to one model's life. Free a model, load another, and the allocator hands the new towers addresses the old ones held, so the new model's expert resolves to an entry filled from a DIFFERENT checkpoint — returned as a HIT, which by contract moves no bytes, so no counter moves and nothing downstream has anything to observe. MEASURED on two synthetic 4-layer/4-expert MoE models in one process, instrumenting `KqExpertSlice` to `memcmp` each returned slot against the tower slice it claims to be: **24 towers occupied 21 distinct addresses, and 20 of 222 slices returned another tower's bytes**; end to end the two arms disagreed on all 160 logits while each arm was internally deterministic (0 differing values on a repeat), which rules out nondeterminism. Invisible to every existing test of this row by construction, because all of them build the cache, store and streamer by hand and none runs two models through the production seam. Reachable by any process that loads a model, releases it, and loads another. Fixed by `OwnedTensor::TowerUid()`, a lazily assigned process-unique counter stamped on the tensor and re-stamped when `bytes` moves (so a copy cannot inherit an identity along with a different buffer); a counter cannot collide because it never goes backwards. Found and fixed while repairing the F1-F11 wiring review for [#912](https://github.com/mudler/vllm.cpp/issues/912). Spec [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1083](https://github.com/mudler/vllm.cpp/issues/1083) | `MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation` | `CleanCaption`'s italic unwrap emulated upstream's `(?!\*)` with a CAPTURED `($|[^*])` (`src/vllm/model_executor/models/minimax_music3_ar.cpp:85,114` @ `a332fb98d`), and a captured group is not a zero-width assertion: consuming the trailing neighbour advanced `regex_replace` past it, so an emphasis span opening within ONE character of the previous close was never examined and the surviving asterisks re-paired ACROSS the intended spans. `*a* *b* *c*` -> `a *b c*` where `_clean_caption` (`encoders.py:72` @ diffusers `c6da9936`) gives `a b c`, and `Warm *lo-fi* *jazzy* keys with a *soft* *brushed* snare` -> `Warm lo-fi *jazzy keys with a soft brushed* snare` — a re-association, not a leftover marker, so the caption handed to the tokenizer is a string upstream would never emit. `encoders.py`'s own header states that whitespace-level prompt changes change the generated audio, so this is a checkpoint-contract break. Invisible to the gate because `markdown_and_tags`, the only golden with italics, carries ONE span per line and the defect needs adjacency. FIXED by porting the trailing side LITERALLY as `(?!\*)` — std::regex's ECMAScript grammar has negative lookahead though not lookbehind, so only the leading `(? 1` branch (`ltx2_lora.cpp:243-248`) is reached only by `test_ltx2_lora.cpp:384,:492`. So it is correct code guarding a state nothing can construct yet, and the state it guards is what N-adapter fusion ([#932](https://github.com/mudler/vllm.cpp/issues/932)) introduces - which is where the reachability half belongs. Two fix shapes, neither chosen here: refuse the second `--lora` in the CLI, or accumulate and let the library refusal fire. Second defect in the same area and from the same landing: `ltx2_video.cpp:362-363` says "nine of these ten reach a reader" about `kKnownLoadExtras`, which now holds TWELVE entries (`:377-383`) after `lora_path` and `lora_strength` landed with #923; eleven of twelve reach a reader and `duration_head_path` is still the one that does not | bug | | [#1098](https://github.com/mudler/vllm.cpp/issues/1098) | `ROAD-V1-LTX25` | `README.md` cannot be corrected, and TWO gates each refuse the fix independently. The claims that are wrong right now: **"37 registered architectures"** four times (`README.md:11,:80,:253,:296`) where `docs/FEATURES.md` says **40** in three places, corrected two commits earlier in `9143196c7`; and **ZERO `LTX` occurrences** against a `minimax` control of 7, so the video-generation announcement names one of the two shipped video families. Blocker 1: `README.md` measures **29,989 chars against `MAX_README_CHARS = 30000`** (`scripts/check-readme-structure.py:47`), so the `LTX-2.5` matrix row (~130 chars) could only land by DELETING another architecture's row - the shared-file lock AGENTS.md forbids in its own words ("Limit an entry, not a shared file"), and the third instance after the two whole-file budgets [#364](https://github.com/mudler/vllm.cpp/issues/364) retired on that argument, `MAX_CHARS` in `check-now-current.py` and the `chars` key in `check-public-doc-tables.py`. The per-ENTRY caps beside it (`MAX_CELL_CHARS = 220`, `MAX_PARAGRAPH_CHARS = 900`) are what actually stop a landing page decaying into a status log. Blocker 2, which is the decisive one: `check-doc-checkpoint.py:346-354` refuses ANY README change that does not also touch a LANDING SOURCE (`:104-113` - `.agents/mission.md`, `CMakeLists.txt`, three `benchmarks/demo/*.json`, `examples/{cli,server}/main.cpp`), evaluated PER COMMIT ([#573](https://github.com/mudler/vllm.cpp/issues/573)) so splitting the edit out does not help, and its own comment calls the rule deliberate and directly tested. It has no arm for a README CORRECTION as against README CHURN, and a correction has a natural witness: the value disagrees with the projection that owns it. So a two-family paragraph was written, MEASURED to fit at 445 chars against the old 438 with 4 to spare, and then REVERTED unlanded; it is preserved verbatim in the issue thread rather than lost. Asked: whether the checkpoint gate should distinguish correction from churn, and whether the whole-file cap should exist at all. NOT asked: raise the constant, which is widening an assertion to keep a gate green | bug | +| [#1092](https://github.com/mudler/vllm.cpp/issues/1092) | `LTX25-GUIDED-VIDEO` | The LTX-2.5 **video** denoise loop runs one UNGUIDED forward per step: `Ltx2PhaseRecipe::video_guidance` is set by every recipe (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1069 @ b5756ea8c`) and read by nothing, so a `pipeline_kind = one_stage` render ignores `cfg_scale = 3.0`, `stg_scale = 1.0`, `rescale_scale = 0.7` and `modality_scale = 3.0` and denoises along a different trajectory than `ti2vid_one_stage.py:221-226 @ fd4ded7f`, which builds a `FactoryGuidedDenoiser` from exactly those. `allow_guidance_override` (`ltx2_pipeline.h:534`) is dead the same way. Positive control for the grep: the same command for `audio_guidance` returns the T2A consumer at `ltx2_video.cpp:3527`. Blocks four more pipelines on one missing seam (`a2vid_two_stage.py:230`, `ti2vid_two_stages.py:248`, `ti2vid_two_stages_hq.py:271`, `keyframe_interpolation.py:232`). Spec [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) | bug | +| [#1111](https://github.com/mudler/vllm.cpp/issues/1111) | `LTX25-GUIDED-VIDEO` | `Ltx2T2aGuidedStep` (`src/vllm/model_executor/models/ltx2_t2a.cpp:203-214` @ `420f6b474`) builds the STG block mask and refuses when NO bit is set, and that one predicate catches two different things. One is a request that disagrees with the CHECKPOINT - `[28]` on a two-block DiT - which is a LOCAL condition, because upstream only ever runs 48-block checkpoints and this port runs reduced ones. The other is an EMPTY list, which is upstream's OWN documented spelling for "STG off" and is refused here. Measured at Lightricks/LTX-2 `fd4ded7f`: `ltx-pipelines/docs/multimodal-guidance.md:13` says "Set to `[]` to disable STG", in the same table and the same idiom as `stg_scale` -> 0.0 and `cfg_scale` -> 1.0; `MultiModalGuiderParams.stg_blocks` DEFAULTS to `[]` (`guiders.py:204`, `field(default_factory=list)`); `--audio-stg-blocks` is `nargs="*"` (`args.py:1039-1045`, `:1107-1113`) so the empty list has a CLI spelling, and `nargs="+"` was the one-character way to forbid it; `LTX_2_3_HQ_PARAMS` SHIPS `stg_blocks=[]` on both modalities (`constants.py:105,:113`); and a whole-tree search found NO validation of `stg_blocks` anywhere - no emptiness, length or range check, no assert, no raise - with the null results recorded. `blocks=None` means EVERY block and `blocks=[]` means NO block (`perturbations.py:26-33`), so the empty list is how a caller says the second thing, and both this tree's parsers go out of their way to keep PRESENT-and-empty distinct from ABSENT - then refuse the one configuration that distinction exists to express. Upstream does not skip the pass either: `do_perturbed_generation` reads `stg_scale` alone (`guiders.py:279-281`), so an empty list disables the STG SIGNAL and not the STG COST. FOUND while repairing the fresh review of [#1092](https://github.com/mudler/vllm.cpp/issues/1092), whose B3 finding is the same divergence on the VIDEO path; that half IS fixed in flow (`ltx2_video.cpp` `ApplyGuidanceOverrides`, `ltx2_denoisers.cpp` `check_reaches_a_block`). NOT fixed here because this half landed with [#1032](https://github.com/mudler/vllm.cpp/issues/1032) and moving it changes a landed row's gated behaviour and its case "a perturbed pass over NO block is refused", which is the normal row-spec-review path rather than an in-flow repair. Until then the two arms disagree about the same request. Listed under `## Owed` in [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) | bug | diff --git a/.agents/specs/ltx25-guided-video.md b/.agents/specs/ltx25-guided-video.md index bfd8c2fd0..d43824805 100644 --- a/.agents/specs/ltx25-guided-video.md +++ b/.agents/specs/ltx25-guided-video.md @@ -34,7 +34,7 @@ does not have. **In:** -- `Ltx2GuidedDenoise` — `_guided_denoise` (`utils/denoisers.py:62-207`) in its +- `Ltx2GuidedDenoise` — `_guided_denoise` (`utils/denoisers.py:61-211`) in its own translation unit mirroring upstream's own file. - The four passes it assembles: `cond`, `uncond`, `ptb`, `mod`. - The `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN` halves of @@ -43,20 +43,27 @@ does not have. - The negative conditioning for the video path — the second half of the encode `GenerateAudioOnly` already performs and discards. - The video guidance request extras that `default_1_stage_arg_parser` - (`utils/args.py:930-1010`) exposes, gated by `allow_guidance_override`. + (`utils/args.py:930-1067`) exposes, gated by `allow_guidance_override`. - The `one_stage` pipeline as the reachable consumer. **Out, and owed rather than silently absent:** -- The other three video pipelines that need this seam - (`a2vid_two_stage`, `ti2vid_two_stages`, `ti2vid_two_stages_hq`, - `keyframe_interpolation`). Each is its own row; this row exists so that they - are ordinary porting work rather than blocked. +- The other video pipelines that need this seam — `a2vid_two_stage`, + `ti2vid_two_stages` ([#1093](https://github.com/mudler/vllm.cpp/issues/1093)), + `ti2vid_two_stages_hq` (owned by #921), + `keyframe_interpolation` ([#1096](https://github.com/mudler/vllm.cpp/issues/1096)), + and beside them `hdr_ic_lora` ([#1094](https://github.com/mudler/vllm.cpp/issues/1094)) + and `dubit` ([#1095](https://github.com/mudler/vllm.cpp/issues/1095)), which are + blocked on other things. This list named four pipelines and no issue numbers + when the spec was written; `281e6a120` (#1099) filed them while this branch was + open, and #1093 names this row's seam as one of the two things it waits on. + Each is its own row; this row exists so that they are ordinary porting work + rather than blocked. - The **device-resident** arm of the `ptb` and `mod` passes. `Ltx2DitForwardDevice` (`ltx2_device.h:136`) takes no `perturbations` argument. Refused by name on that arm rather than run unperturbed, which would produce a legal-looking render whose STG term is identically zero. See §4.3. -- `BatchedPerturbationConfig`'s partial blend (`attention.py:571-572`) and +- `BatchedPerturbationConfig`'s partial blend (`attention.py:572-573`) and batch > 1, which stay degenerate at the one batch size this port runs — the statement `ltx2.h` already carries, unchanged. @@ -92,7 +99,7 @@ The perturbation types are per direction, not per modality | `cond` | none | — | | `uncond` | none | negative context on both streams | | `ptb` | `SKIP_VIDEO_SELF_ATTN` on `video_guider.stg_blocks`, `SKIP_AUDIO_SELF_ATTN` on `audio_guider.stg_blocks` | `attention.py:557` `use_attention = not all_perturbed` | -| `mod` | `SKIP_A2V_CROSS_ATTN` and `SKIP_V2A_CROSS_ATTN`, **all blocks** | `transformer.py:335,366` `cross_attn_skip_all` | +| `mod` | `SKIP_A2V_CROSS_ATTN` and `SKIP_V2A_CROSS_ATTN`, **all blocks** | `transformer.py:335,367` `cross_attn_skip_all` | ## 3. Our baseline, derived at `b5756ea8c` @@ -161,10 +168,20 @@ using Ltx2X0Model = std::functionenabled = false` runs A2V alone, and one with `video->enabled = false` +runs V2A alone. Each row asserts both halves — the flag for that direction MOVES +the written stream, and the flag for the other direction leaves it BIT-IDENTICAL +— which is what makes the swap detectable rather than only the omission. The +per-block-pair coupling that defeats a both-enabled forward (block 1's V2A reads +what block 0's A2V wrote) never arises, because only one direction runs at all. + ## 8. Risks and decisions **R1 — the space error, on the default arm.** `rescale_scale` is 0.7 on the @@ -427,10 +487,18 @@ Stop and report `NEEDS_DECISION` rather than narrowing silently if: - **The device-resident `ptb` and `mod` passes.** `Ltx2DitForwardDevice` takes no `perturbations`. Owned by this row's follow-up issue; refused by name until then. -- **The other four pipelines** — `a2vid_two_stage`, `ti2vid_two_stages`, - `ti2vid_two_stages_hq`, `keyframe_interpolation`. Each needs its own row; none - is blocked on this seam any more. +- **The other four pipelines** — `a2vid_two_stage`, `ti2vid_two_stages` + ([#1093](https://github.com/mudler/vllm.cpp/issues/1093)), + `ti2vid_two_stages_hq` (#921), + `keyframe_interpolation` ([#1096](https://github.com/mudler/vllm.cpp/issues/1096)). + Each needs its own row; none is blocked on this seam any more. - **#1049's remaining three symbols** — see §6c. +- **[#1111](https://github.com/mudler/vllm.cpp/issues/1111) — the T2A arm still + refuses an EMPTY `audio_stg_blocks`.** The same divergence this row's review + found on the video path, in `ltx2_t2a.cpp:203-214`, which landed with #1032 and + is not this row's code. Fixing it moves a landed row's gated behaviour and one + of its cases, so it takes the normal row-spec-review path. Until then the two + arms disagree about the same request. - **An oracle-run comparison.** vLLM-Omni is UNPINNED (#633) and carries no LTX-2.5 recipe; no LTX-2.5 checkpoint here has a recorded sha256 (#1048). The guidance arithmetic is gated against upstream **source**, not against upstream @@ -453,15 +521,47 @@ one. **Measured, not argued:** every existing golden in `test_ltx2_video` held unchanged across the change, including the `distilled_two_stage` renders and the DFR and retake ones. -**Two refusals that upstream does not have.** Both are cases where upstream's +**One refusal that upstream does not have.** It is a case where upstream's behaviour is a silent zero rather than an error, and where the render finishes: - an `stg_blocks` list naming no block this checkpoint has. Upstream's `Perturbation.is_perturbed` is a membership test (`perturbations.py:26-33`), so `[28]` on a two-block DiT perturbs nothing and leaves `stg_scale * (cond - perturbed)` at exactly zero. Upstream never meets it - because it only runs 48-block checkpoints; this port runs reduced ones. -- an EMPTY `stg_blocks` beside a non-zero STG scale, for the same reason. + because it only runs 48-block checkpoints; this port runs reduced ones. The + refusal is about a request that disagrees with the **checkpoint**, which is a + local condition rather than an upstream one. + +**A second refusal was landed on this branch and is retired in the review +repair.** An EMPTY `stg_blocks` beside a non-zero STG scale was refused for the +same "the term would be exactly zero" reason. Every clause of that reason is +true, and none of it makes the configuration illegal upstream — which is the only +question a mirror gets to ask. Measured at `fd4ded7f`: + +| Evidence | Where | +|---|---| +| "Set to `[]` to disable STG", in the same table and idiom as `stg_scale` → 0.0 | `ltx-pipelines/docs/multimodal-guidance.md:13` | +| `MultiModalGuiderParams.stg_blocks` DEFAULTS to `[]` | `guiders.py:204` | +| `--video-stg-blocks` / `--audio-stg-blocks` are `nargs="*"`, so `[]` has a CLI spelling; `nargs="+"` was the one-character way to forbid it | `args.py:979-985`, `:1039-1045`, `:1107-1113` | +| `LTX_2_3_HQ_PARAMS` SHIPS `stg_blocks=[]` on both modalities | `constants.py:105`, `:113` | +| no validation of `stg_blocks` anywhere in that tree — no emptiness, length or range check | measured by a whole-tree search with the null results recorded | + +`blocks=None` means EVERY block and `blocks=[]` means NO block +(`perturbations.py:26-33`), so the empty list is how a caller says the second +thing — and `ApplyStgBlocksExtra` exists precisely to keep PRESENT-and-empty +distinct from ABSENT. Refusing it made that distinction unreachable, which is +the shape of the defect rather than a matter of taste. Upstream does not skip +the pass either: `do_perturbed_generation` reads `stg_scale` alone +(`guiders.py:279-281`), so the `ptb` entry is appended and its result equals +`cond`. This port now does the same, and the case asserts the exact equality +rather than a tolerance. + +**The same refusal exists on the T2A arm and is NOT changed here.** +`ltx2_t2a.cpp:203-214` computes the block mask and refuses when no bit is set, +which catches the empty list as well as the out-of-range one. It landed with +[#1032](https://github.com/mudler/vllm.cpp/issues/1032) and is not this row's +code; changing it would alter a landed row's gated behaviour and one of its +cases. Filed and listed under `## Owed`. **A cross perturbation that differs between blocks is refused rather than widened.** `Ltx2DitPerturbation` carries one boolean per direction, which is @@ -504,6 +604,61 @@ and the case now says why in an assertion instead of leaving it unexplained. **A residual, stated.** Nothing here compares against a running oracle; see §0. +### 11.1 The fresh review, and what it moved + +One BLOCKING finding and ten non-blocking ones. The core was reproduced and is +not revisited: the seam, the x0 space, all four arms, the reachability, and the +gate. The reviewer also confirmed the live defect independently — `video_guidance` +had exactly two hits at the merge base, a declaration and a write with no reader, +against a positive control where `audio_guidance` finds its T2A consumer. + +| Finding | Disposition | +|---|---| +| B1 (blocking) — the two cross booleans were not gated per direction: M12, M13 and M15 were GREEN | REPAIRED. §7.5, a direct DiT-level case per direction. All four mutations now RED | +| B2 — `docs/FEATURES.md` said T2A was "the only GUIDED arm", which this row's own new row made false | REPAIRED in the existing cell, 202 of 220 chars, prose paragraphs unchanged at 21 of 21 | +| B3 — the EMPTY `stg_blocks` refusal diverges from upstream | REPAIRED on this row's paths; the T2A one is #1111. See §10 | +| B4 — `INFO("arm = " << arm.name)` printed `arm = 1` | REPAIRED at all three sites in the file | +| B5 — the rescale control's modality claim was numerically inert | RESTATED, and now asserted rather than argued. See §7.2 | +| B6 — upstream anchor drift | REPAIRED, 41 scripted replacements with the hit count asserted per edit. See §11.2 | +| B7 — "the seam cannot be handed a velocity" is caller discipline, not a type guarantee | RESTATED in §4.1 and in `ltx2_denoisers.h`; the code is unchanged, which is the right outcome | +| B8 — the branch was behind `origin/main` | MERGED and re-gated; §1 now carries #1093 to #1096 from `281e6a120` | +| B9, B11 | recorded by the reviewer as not this repair's | +| B10 — the new `docs/USAGE.md` section documented flags and not extra keys | REPAIRED: the `/v1/videos` caveat, a flag-to-extra table, and the empty-list behaviour | + +**One sub-claim was REJECTED on evidence.** B3 argued an asymmetry: that +`audio_stg_blocks=""` is still accepted on `t2a_one_stage`, because that path +returns before `ApplyGuidanceOverrides`. It does return there, and the request is +still refused — by `ltx2_t2a.cpp:203-214`, which builds the block mask and fails +when no bit is set, and which `git log -S` puts on `main` at `0b0b8900f` with +#1032 rather than on this branch. So there is no asymmetry today; the two arms +agree, and both diverge from upstream. Fixing the video half creates the +asymmetry, which is why #1111 exists and is listed under `## Owed` rather than +left implied. + +### 11.2 The anchors + +Re-derived against `fd4ded7f` from the sentence making each claim, never by +reading text out of the cited span. Corrected: `_guided_denoise` **61-211** (was +62-207); `enabled = not skip` at **158, 168** (was 151, 161 — the `= None` +initializers); the V2A `cross_attn_skip_all` guard at **367** (366 is blank); the +batched config built at **182-187** (172-176 is the comment plus the per-sample +replication at :175); the partial blend at **572-573**; the one `PromptEncoder` +call at **166-174**; `default_1_stage_arg_parser` **930-1067** and its guider +flags **947-1066**; the two `--*-stg-blocks` flags at **979-985** and +**1039-1045**; `cross_attn_skip_all` DECLARED at `transformer_args.py:70` (118 is +a call site); `modality_scale = 3.0` at `constants.py:54, :64` with +`_PARAMS_SINCE_VERSION` at **130-133** (the cited 40-80 covers neither); +`CFGGuider` / `STGGuider` at **11-27** and **56-74**; the `perturbations` +ARGUMENT at `model.py:493` (492 is the `def`); `t2a_one_stage.py:202` (200-201 +are its comment); `ti2vid_one_stage.py:211-226` (210 is blank). + +No gate protects a spec anchor ([#632](https://github.com/mudler/vllm.cpp/issues/632)), +so the edits were applied by a script that asserts the expected hit count per +replacement and refuses the whole run on a mismatch — two of the 43 planned edits +were caught that way and re-derived. + ## Now -Implementation and gate landed on the branch; the row awaits a fresh review. +The fresh review returned CHANGES REQUESTED; the blocking finding and the nine +non-blocking ones this repair owns are answered on the branch (§11.1), the branch +is merged up to `origin/main`, and the row awaits a second fresh review. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index dd6ce4ebc..1287984db 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -173,7 +173,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 Conv VAE decode arithmetic width | LTX-2.5 video VAE | `test_ltx2_vae` "the decode's convolution accumulates in f32", entering through `Ltx2VideoDecodeStreaming`; widening the accumulator to `double`, or deleting the production call site, each turns it RED | **f32**, the width `F.conv3d` uses at f32 AND bf16 (MEASURED). Was f64 at 8 sites ([#1008](https://github.com/mudler/vllm.cpp/issues/1008)). Conv sums BLOCKED per input channel, as torch's. STORAGE stays f32; bf16 owed | | LTX-2.5 Conv VAE decode threading | LTX-2.5 video VAE | `test_ltx2_vae` "the decode DISPATCHES its convolutions to the CPU threadpool" and "...BIT-IDENTICAL across thread counts", through `Ltx2VideoDecodeStreaming`; 34 golden margins UNCHANGED; TSan clean | **Parallel** over CONV output lines via `vt::cpu::ParallelForRows` ([#1009](https://github.com/mudler/vllm.cpp/issues/1009)). ~9x at 16-20 workers, contended box, 21-23% spread. Bit-identical at any count | | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated | -| LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 18 mutations, 17 DETECTED (four by review of a conditional-only #1039 gate) and the 18th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only GUIDED arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | +| LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 18 mutations, 17 DETECTED (four by review of a conditional-only #1039 gate) and the 18th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only AUDIO-ONLY guided arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | | LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 ([#1039](https://github.com/mudler/vllm.cpp/issues/1039)) | | LTX-2.5 VIDEO guidance | LTX-2.5 DiT, joint video+audio | `test_ltx2_video`'s `ltx2 one_stage:` cases through `Generate`; all FOUR arms carry the x0 invariant and the guider output replays EXACTLY | `--pipeline-kind one_stage` runs `_guided_denoise`: 4 forwards/step, combined per modality in **x0**. Was ONE unguided forward, every `video_guidance` field dead ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) | | LTX-2.5 cross-attention perturbations | LTX-2.5 DiT | `test_ltx2_video` asserts the isolated-modality pass reached the DiT with both directions off and moved the tensor | `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN` ported, which `modality_scale = 3.0` selects on every video row. The DEVICE forward takes no perturbations, so that pass is refused there by name | diff --git a/docs/USAGE.md b/docs/USAGE.md index 77fa942e8..3579e30b0 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1000,27 +1000,65 @@ them per modality: | isolated modality | the audio<->video cross attention off in every block | `modality_scale != 1.0` | Seven per-generation knobs mirror upstream's `default_1_stage_arg_parser` and -each takes the checkpoint generation's value when absent: -`--video-cfg-guidance-scale`, `--video-stg-guidance-scale`, -`--video-rescale-scale`, `--video-skip-step`, `--video-stg-blocks` (comma -separated), `--a2v-guidance-scale` and `--v2a-guidance-scale`. The audio row and +each takes the checkpoint generation's value when absent. The audio row and `--negative-prompt` are shared with text-to-audio and are no longer refused on a video pipeline; upstream's parser carries both rows side by side, and the old refusal rested on a reading of upstream that was wrong and harmless only while nothing here read them. +| `ltx2-gen` flag | per-generation extra | meaning | +|---|---|---| +| `--video-cfg-guidance-scale` | `video_cfg_guidance_scale` | video `cfg_scale`; `1.0` turns the unconditional forward off | +| `--video-stg-guidance-scale` | `video_stg_guidance_scale` | video `stg_scale`; `0.0` turns the perturbed forward off | +| `--video-rescale-scale` | `video_rescale_scale` | video `rescale_scale`, applied to the DENOISED prediction | +| `--video-skip-step` | `video_skip_step` | `0` never skips; `n` runs every `n+1`-th step | +| `--video-stg-blocks` | `video_stg_blocks` | comma separated block indices; EMPTY disables STG, see below | +| `--a2v-guidance-scale` | `a2v_guidance_scale` | video `modality_scale`; `1.0` turns the isolated-modality forward off | +| `--v2a-guidance-scale` | `v2a_guidance_scale` | audio `modality_scale` | +| `--negative-prompt` | `negative_prompt` | the unconditional forward's conditioning | + +The audio row is the same six spellings with `audio_` in place of `video_`: +`audio_cfg_guidance_scale`, `audio_stg_guidance_scale`, `audio_rescale_scale`, +`audio_skip_step`, `audio_stg_blocks`, and `v2a_guidance_scale` for its +`modality_scale`. + +Those extras ride the per-generation `extra_keys` / `extra_values` array on +`vllm_video_params`, so the C ABI reaches the same path with no new field. They +are per-GENERATION and therefore reach the CLI and the C ABI and **not** +`/v1/videos`, which forwards no per-generation extra to any engine +([#928](https://github.com/mudler/vllm.cpp/issues/928)). `pipeline_kind` is a +LOAD knob and does reach the server, so a server started with +`--video-extra pipeline_kind=one_stage` renders every request through the guided +denoiser at the recipe's own guider values and no request can change them. + +**An EMPTY `--video-stg-blocks` is accepted and means "perturb no block".** That +is upstream's own idiom — `docs/multimodal-guidance.md:13` says "Set to `[]` to +disable STG", the field defaults to `[]`, the flags are `nargs="*"`, and the +shipped HQ params row uses it — and it stays distinct from OMITTING the flag, +which takes the params table's value. It disables the STG signal and not the STG +cost: upstream selects the perturbed pass from `stg_scale` alone, so the forward +still runs and contributes exactly zero. Set the scale to `0.0` to skip the +forward as well. This page and this port refused the empty list until +2026-08-17. + **The unconditional forward needs a negative conditioning, and there are two ways to supply one.** With a text tower, `--negative-prompt` (or the recipe's own default) is encoded through the same chain as the positive prompt. Without -one, `--negative-prompt-embeds` and `--negative-audio-prompt-embeds` are the -negative half of the `prompt_embeds_path` fallback: two files at the DiT's two -cross-attention widths, the same row count as the positive pair. With neither, a +one, `--negative-prompt-embeds` and `--negative-audio-prompt-embeds` — the LOAD +extras `negative_prompt_embeds_path` and `negative_audio_prompt_embeds_path` — +are the negative half of the `prompt_embeds_path` fallback: two files at the +DiT's two cross-attention widths, the same row count as the positive pair. Being +LOAD extras they DO reach the server, through `--video-extra`. With neither, a `cfg_scale` other than 1.0 is **refused by name** rather than served the positive context twice, which would leave the whole classifier-free term at exactly zero. -**A block index the checkpoint does not have is refused.** `stg_blocks` is a -membership test upstream, so naming block 28 on a model with fewer blocks -perturbs nothing and leaves `stg_scale * (cond - perturbed)` at exactly zero. +**A block index the checkpoint does not have is refused**, which is the case the +empty list above is NOT. `stg_blocks` is a membership test upstream, so naming +block 28 on a model with fewer blocks perturbs nothing and leaves +`stg_scale * (cond - perturbed)` at exactly zero — the same zero, reached by a +request that disagrees with the checkpoint rather than by a caller who asked for +no perturbation. Upstream never meets it because it only ships 48-block +checkpoints, so this refusal is local to this port and is named as such. **The distilled and retake recipes refuse every one of these flags.** Their guidance is distilled into the weights, so honouring an override would sample a diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index 6d99243e2..f94908d04 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -218,7 +218,7 @@ int main(int argc, char** argv) { std::string audio_skip_step, audio_stg_blocks; // THE VIDEO GUIDER (row LTX25-GUIDED-VIDEO, #1092): the other half of the same - // parser, `default_1_stage_arg_parser` (utils/args.py:947-1010). `--negative- + // parser, `default_1_stage_arg_parser` (utils/args.py:947-1066). `--negative- // prompt` above is shared by both, which is why it is not repeated here. std::string video_cfg_scale, video_stg_scale, video_rescale, video_skip_step; std::string video_stg_blocks, a2v_scale, v2a_scale; diff --git a/include/vllm/model_executor/models/ltx2.h b/include/vllm/model_executor/models/ltx2.h index 962564daa..472e1c080 100644 --- a/include/vllm/model_executor/models/ltx2.h +++ b/include/vllm/model_executor/models/ltx2.h @@ -41,7 +41,7 @@ // NOT PORTED IN L2, recorded here so it cannot be discovered later: // - PORTED 2026-08-17 by row LTX25-GUIDED-VIDEO (#1092): the CROSS-attention // guidance perturbations, `SKIP_A2V_CROSS_ATTN` and `SKIP_V2A_CROSS_ATTN` -// (guidance/perturbations.py:8-16, transformer.py:335,366 +// (guidance/perturbations.py:8-16, transformer.py:335,367 // `cross_attn_skip_all`). They are `Ltx2DitPerturbation`'s two booleans. // // WHY THE PREVIOUS ENTRY WAS WRONG RATHER THAN MERELY STALE. It refused them @@ -51,13 +51,13 @@ // builds BOTH cross types with `blocks=None` whenever either guider has // `modality_scale != 1.0` (denoisers.py:121-137, guiders.py:283-285). Every // VIDEO row of the params table sets it to 3.0 -// (utils/constants.py:40-80), so this was upstream's default on the video +// (utils/constants.py:54, :64), so this was upstream's default on the video // path the whole time. The sentence was true of text-to-audio, which pins -// the field to 1.0 (t2a_one_stage.py:200-202), and it was written while +// the field to 1.0 (t2a_one_stage.py:202), and it was written while // text-to-audio was the only guided path here. // // THE SELF-ATTENTION HALF was ported by #1005. `Ltx2DitPerturbation` is -// upstream's `perturbations` argument (model.py:492) at the one batch size +// upstream's `perturbations` argument (model.py:493) at the one batch size // this port serves, and `Ltx2AttentionArgs::all_perturbed` is // `use_attention = not all_perturbed` (attention.py:557). `nullptr` remains // upstream's `perturbations=None` path (model.py:509-511) and is what an @@ -67,7 +67,7 @@ // indexed [type, block, SAMPLE]) is ported in `ltx2_pipeline.h` and reached // by `Ltx2GuidedDenoise`, which builds one config over the pass list and // slices it per pass. What is still unported is batch > 1 and the partial -// blend it exists for (`out * mask + v * (1 - mask)`, attention.py:571-572). +// blend it exists for (`out * mask + v * (1 - mask)`, attention.py:572-573). // Both are degenerate at `Ltx2ModalityInput::batch == 1`, which is the only // batch any path here runs. // - The caption projections (text_projection.py:31-38). LTX-2.5 is a 22B-form @@ -542,7 +542,7 @@ struct Ltx2DitOutputs { std::vector audio; // [batch, audio tokens, audio_out_channels] }; -// `perturbations` on LTXModel.forward (model.py:492), reduced to what this port +// `perturbations` on LTXModel.forward (model.py:493), reduced to what this port // can express. One entry per BLOCK; `true` means that block's self-attention is // replaced by its value projection, which is STG. // @@ -552,7 +552,7 @@ struct Ltx2DitOutputs { // degenerate one. `Ltx2BatchedPerturbationConfig` (ltx2_pipeline.h) is the // batched form, and row LTX25-GUIDED-VIDEO (#1092) is what gave it a product // caller: `Ltx2GuidedDenoise` builds one config over all four passes and slices -// it per pass, which is `denoisers.py:172-176` and is where the ONE-sample +// it per pass, which is `denoisers.py:182-187` and is where the ONE-sample // flattening below happens. // // EMPTY IS NOT "NOTHING PERTURBED BY COINCIDENCE": a vector of the wrong length @@ -564,16 +564,16 @@ struct Ltx2DitOutputs { // because the one thing that builds them asks for ALL blocks // (`Perturbation(type=..., blocks=None)`, denoisers.py:132-135) and upstream's // own reader is the per-block scalar `cross_attn_skip_all` (transformer.py:335, -// :366) rather than a mask multiply. A per-block cross vector would be a surface +// :367) rather than a mask multiply. A per-block cross vector would be a surface // with no constructor. // // WHAT THIS ENTRY USED TO SAY, kept because the sentence was load-bearing and // wrong: "Nothing upstream that this port serves constructs them: STG is built // from `stg_blocks` and reaches the self-attention types only". That was true // while text-to-audio was the only guided path here — it pins -// `modality_scale = 1.0` (t2a_one_stage.py:200-202), which is exactly the value +// `modality_scale = 1.0` (t2a_one_stage.py:202), which is exactly the value // `do_isolated_modality_generation` reads as OFF. Every VIDEO row defaults it to -// 3.0 (utils/constants.py:40-80), so the isolated-modality pass is upstream's +// 3.0 (utils/constants.py:54, :64), so the isolated-modality pass is upstream's // DEFAULT there and these two types are on the reachable path. struct Ltx2DitPerturbation { std::vector video_self_attn; // [num_layers], empty = none @@ -652,14 +652,25 @@ struct Ltx2BlockArgs { // STG for THIS block (attention.py:552-577). See `Ltx2DitPerturbation`. bool video_self_attn_perturbed = false; bool audio_self_attn_perturbed = false; - // `cross_attn_skip_all` (transformer_args.py:118, read at transformer.py:335 - // and :366). THE FLAG RIDES ON THE STREAM BEING WRITTEN, not on the stream + // `cross_attn_skip_all` (transformer_args.py:70, read at transformer.py:335 + // and :367). THE FLAG RIDES ON THE STREAM BEING WRITTEN, not on the stream // being read: `video.cross_attn_skip_all` skips A2V, which writes the VIDEO - // stream from audio keys, and `audio.cross_attn_skip_all` skips V2A. Swapping - // them still renders — both directions are off on the one pass that sets - // either, because `_guided_denoise` builds the isolated-modality pass with - // BOTH (denoisers.py:130-136) — so this is one of the places a comment has to - // carry what a test on the shipped path cannot separate. + // stream from audio keys, and `audio.cross_attn_skip_all` skips V2A. + // + // THIS COMMENT USED TO SAY A TEST COULD NOT SEPARATE THEM, and it was the + // wrong conclusion from a true premise. The premise: on the SHIPPED path both + // directions are off together, because `_guided_denoise` builds the + // isolated-modality pass with BOTH (denoisers.py:125-138), so swapping the two + // flags renders identically there. The conclusion does not follow, because + // nothing obliges the separating test to use the shipped combination. + // `run_a2v` and `run_v2a` read the two streams' `enabled` flags + // asymmetrically (transformer.py:265-269), so a forward with one stream + // PRESENT but DISABLED runs exactly one cross direction and each flag becomes + // observable alone. That is what + // "ltx2 dit: each CROSS perturbation gates ITS OWN direction and no other" + // does; it is red against a build that applies only one direction (M12, M13) + // and against a build that swaps them (M15), all three of which were GREEN + // over the shipped-path case alone. bool video_cross_attn_skip_all = false; bool audio_cross_attn_skip_all = false; // Audio<->video cross-attention AdaLN inputs (transformer_args.py:388-411). diff --git a/include/vllm/model_executor/models/ltx2_denoisers.h b/include/vllm/model_executor/models/ltx2_denoisers.h index 1bd50c3d6..f9d52e971 100644 --- a/include/vllm/model_executor/models/ltx2_denoisers.h +++ b/include/vllm/model_executor/models/ltx2_denoisers.h @@ -9,7 +9,7 @@ // // Upstream has three denoisers (`SimpleDenoiser`, `GuidedDenoiser`, // `FactoryGuidedDenoiser`) and they share ONE function: `_guided_denoise` -// (denoisers.py:62-207). That function is what this file ports. It is the piece +// (denoisers.py:61-211). That function is what this file ports. It is the piece // four unported pipelines are each blocked on — `a2vid_two_stage.py:230`, // `ti2vid_two_stages.py:248`, `ti2vid_two_stages_hq.py:271`, // `keyframe_interpolation.py:232` — and it is what a `pipeline_kind = one_stage` @@ -24,9 +24,19 @@ // denoised tensors. Combining velocities and converting once afterwards is a // DIFFERENT function whenever `rescale_scale != 0` (guiders.py:268-271), and // it is 0.7 on every video row of the params table. That defect shipped on -// the audio arm of this tree and is #1039. This seam cannot commit it, -// because it never sees a velocity: the conversion lives in the caller's -// `Ltx2X0Model`, which is where upstream puts it. +// the audio arm of this tree and is #1039. The conversion therefore lives in +// the caller's `Ltx2X0Model`, which is where upstream puts it, and the seam +// combines `Ltx2X0Outputs::video` / `::audio`. +// +// THAT IS CALLER DISCIPLINE AND NOT A TYPE GUARANTEE, and this comment +// claimed the stronger thing until 2026-08-17. `Ltx2X0Outputs` carries the +// raw velocity beside the denoised prediction (below), so a lambda that fills +// `video` with what belongs in `video_velocity` type-checks and renders. +// Nothing in the signature can stop it; the four per-arm invariants in +// `test_ltx2_video` do, and mutations M1-M4 — one per arm, each handing the +// seam a velocity — are red against them. A structural claim a type does not +// enforce is worth less than a gate that catches the substitution, so the +// gate is where this is argued. // // 2. ONE PASS LIST, TWO GUIDERS. `_guided_denoise` takes the UNION of what the // two guiders want — one `uncond` pass if either asks (`:102-109`), one `ptb` @@ -172,7 +182,7 @@ struct Ltx2GuidedDenoiseResult { bool modality_pass_skipped_v2a = false; }; -// `_guided_denoise` (denoisers.py:62-207). +// `_guided_denoise` (denoisers.py:61-211). Ltx2GuidedDenoiseResult Ltx2GuidedDenoise(const Ltx2X0Model& transformer, const Ltx2GuidedDenoiseInputs& in); diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index c038a96a8..2ae214989 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -461,7 +461,7 @@ inline constexpr char kLtx2AudioStgBlocksExtra[] = "audio_stg_blocks"; // THE VIDEO GUIDER, row LTX25-GUIDED-VIDEO (#1092). The same row of flags on the // other stream, from the same parser (`default_1_stage_arg_parser`, -// utils/args.py:947-1010). ABSENT MEANS the params table's own value: 3.0 / 1.0 / +// utils/args.py:947-1066). ABSENT MEANS the params table's own value: 3.0 / 1.0 / // 0.7 / 3.0 and block 28 on the 2.3-and-later lineage // (utils/constants.py:40-88). // @@ -491,7 +491,7 @@ inline constexpr char kLtx2V2aGuidanceScaleExtra[] = "v2a_guidance_scale"; // // Upstream has no embeds surface at all: every pipeline encodes // `[prompt, negative_prompt]` in ONE `PromptEncoder` call -// (ti2vid_one_stage.py:170-178) and takes `.video_encoding` / `.audio_encoding` +// (ti2vid_one_stage.py:166-174) and takes `.video_encoding` / `.audio_encoding` // from each half. `prompt_embeds_path` and the `audio_prompt_embeds_path` extra // are this port's own affordance for running the DiT without a 12B tower; these // two are the SAME affordance applied to the second of upstream's two diff --git a/src/vllm/model_executor/models/ltx2_denoisers.cpp b/src/vllm/model_executor/models/ltx2_denoisers.cpp index b73a9f94f..5c12acfd3 100644 --- a/src/vllm/model_executor/models/ltx2_denoisers.cpp +++ b/src/vllm/model_executor/models/ltx2_denoisers.cpp @@ -1,4 +1,4 @@ -// `_guided_denoise` (ltx-pipelines utils/denoisers.py:62-207) at +// `_guided_denoise` (ltx-pipelines utils/denoisers.py:61-211) at // Lightricks/LTX-2 @ fd4ded7f. See ltx2_denoisers.h for the four things that // fail silently if guessed. // @@ -162,9 +162,20 @@ Ltx2GuidedDenoiseResult Ltx2GuidedDenoise(const Ltx2X0Model& transformer, // Upstream never meets this because it only ever runs 48-block checkpoints; // this port runs reduced ones, and a smaller checkpoint is a legal thing to // hand it. + // + // AN EMPTY LIST IS EXEMPT, and it was not until 2026-08-17. `blocks=[]` is + // upstream's documented spelling for "perturb no block", distinct from + // `blocks=None`'s "perturb every block" (perturbations.py:26-33), named as + // the way to disable STG at `ltx-pipelines/docs/multimodal-guidance.md:13`, + // shipped in `LTX_2_3_HQ_PARAMS` (constants.py:105, :113), and reachable + // through `nargs="*"` (args.py:979-985). Upstream runs the pass and takes + // the zero term; so does this. What is refused is a list that NAMES blocks + // and reaches none of them, which is a request that disagrees with the + // CHECKPOINT rather than a caller who asked for nothing. const auto check_reaches_a_block = [&](const Ltx2MultiModalGuiderParams& guider, const char* which) { if (!guider.DoPerturbedGeneration()) return; + if (guider.stg_blocks.empty()) return; for (const int64_t block : guider.stg_blocks) { if (block >= 0 && block < in.num_blocks) return; } @@ -184,7 +195,7 @@ Ltx2GuidedDenoiseResult Ltx2GuidedDenoise(const Ltx2X0Model& transformer, // `:121-137`. The isolated-modality pass: BOTH cross directions, ALL blocks // (`blocks=None`), when EITHER guider isolates. `modality_scale` is 3.0 on - // every video row of the params table (utils/constants.py:40-80), so this is + // every video row of the params table (utils/constants.py:54, :64), so this is // the default arm rather than a corner. if (video_guider.DoIsolatedModalityGeneration() || audio_guider.DoIsolatedModalityGeneration()) { @@ -200,7 +211,7 @@ Ltx2GuidedDenoiseResult Ltx2GuidedDenoise(const Ltx2X0Model& transformer, passes.push_back({Ltx2DenoisePass::kModality, v_context, a_context, std::move(mod)}); } - // ── the perturbation config (denoisers.py:172-186) ─────────────────────── + // ── the perturbation config (denoisers.py:182-187) ─────────────────────── // // ONE batched config over the whole pass list, then one sample slice per pass, // which is upstream's `batched_ptb_configs` followed by the per-sample mask the @@ -235,7 +246,7 @@ Ltx2GuidedDenoiseResult Ltx2GuidedDenoise(const Ltx2X0Model& transformer, } // The cross flags are not per block, because `Ltx2DitPerturbation` has no // per-block cross vector and upstream's reader is the per-block scalar - // `cross_attn_skip_all` (transformer.py:335,366) rather than a mask + // `cross_attn_skip_all` (transformer.py:335,367) rather than a mask // multiply. That flattening is only sound while the config says the same // thing on every block, which is what `blocks=None` produces // (denoisers.py:132-135) — so it is CHECKED here rather than assumed. A @@ -274,7 +285,7 @@ Ltx2GuidedDenoiseResult Ltx2GuidedDenoise(const Ltx2X0Model& transformer, if (in.video != nullptr) { video_in = *in.video; video_in.context = pass.video_context; - // `enabled=not v_skip` (`:151`). A skipped modality stays PRESENT, so the + // `enabled=not v_skip` (`:158`). A skipped modality stays PRESENT, so the // other stream's cross attention still reads its latent // (transformer.py:269 tests presence, not `enabled`). video_in.enabled = !v_skip; diff --git a/src/vllm/model_executor/models/ltx2_dit.cpp b/src/vllm/model_executor/models/ltx2_dit.cpp index 82227471e..0dd7583eb 100644 --- a/src/vllm/model_executor/models/ltx2_dit.cpp +++ b/src/vllm/model_executor/models/ltx2_dit.cpp @@ -384,7 +384,7 @@ void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, AddGatedBroadcast(video_x, out, gate, batch, tv, dim); } - // `if run_v2a and not audio.cross_attn_skip_all` (transformer.py:366). + // `if run_v2a and not audio.cross_attn_skip_all` (transformer.py:367). if (run_v2a && !args.audio_cross_attn_skip_all) { std::vector scale_a, shift_a, scale_v, shift_v; av_scale_shift(w.scale_shift_table_a2v_ca_audio, args.audio_cross_scale_shift, ta, adim, 2, @@ -830,7 +830,7 @@ Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, params.audio_num_attention_heads, have_both ? video : nullptr); } - // `perturbations` (model.py:492). A vector that is not exactly `num_layers` + // `perturbations` (model.py:493). A vector that is not exactly `num_layers` // long is REFUSED rather than indexed defensively: a config built for another // layer count would otherwise perturb a prefix of the blocks and leave the rest // alone, which is a legal-looking STG pass over the wrong blocks and renders. @@ -878,7 +878,7 @@ Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, perturbations->audio_self_attn[static_cast(i)] != 0; // Not indexed by block: the only thing that builds these asks for ALL // blocks (`blocks=None`, denoisers.py:132-135), and upstream's reader is a - // per-block scalar rather than a mask multiply (transformer.py:335,366). + // per-block scalar rather than a mask multiply (transformer.py:335,367). a.video_cross_attn_skip_all = perturbations->video_cross_attn_skip_all; a.audio_cross_attn_skip_all = perturbations->audio_cross_attn_skip_all; } diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index c7b7d15f8..2cd0be1af 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -1496,7 +1496,7 @@ void AssertGeneratedKeyframesSupported(bool has_embedding, const std::string& di // ── the guiders (row LTX25-GUIDED-VIDEO, #1092) ──────────────────────────── -// `--*-stg-blocks`, `nargs="*"` (utils/args.py:980-985, :1040-1045). An extra +// `--*-stg-blocks`, `nargs="*"` (utils/args.py:979-985, :1039-1045). An extra // that is PRESENT and empty is upstream's empty list — "perturb nothing" — and // stays distinct from an ABSENT extra, which takes the params table's own value. // Collapsing the two would make `video_stg_blocks=` silently mean block 28. @@ -1522,8 +1522,8 @@ void ApplyStgBlocksExtra(const std::map& extras, const } } -// One CLI flag each, from `default_1_stage_arg_parser` (utils/args.py:947-1010 -// for the video row, :1011-1075 for the audio one). Each extra overrides ONE +// One CLI flag each, from `default_1_stage_arg_parser` (utils/args.py:947-1066: +// the video row's six flags open at :948 and the audio row's at :1008). Each extra overrides ONE // field of the phase's own resolved guider, which is what one flag does. // // REFUSED WHOLESALE on a phase that fixes its guidance. `allow_guidance_override @@ -1574,25 +1574,40 @@ void ApplyGuidanceOverrides(const std::map& extras, }; check_skip(kLtx2VideoSkipStepExtra, video->skip_step); check_skip(kLtx2AudioSkipStepExtra, audio->skip_step); - // `stg_blocks` is only read when the perturbed pass runs, so an out-of-range - // block is refused HERE rather than inside the DiT, where the message would be - // about a mask length. `blocks is None` — upstream's "every block" — has no CLI - // spelling and none is invented, so an EMPTY list beside a non-zero STG scale - // is a perturbed pass identical to the conditional one, which is a wasted - // forward and a guidance term of exactly zero. - const auto check_blocks = [&](const Ltx2MultiModalGuiderParams& g, const char* which, - const char* key) { - if (!g.DoPerturbedGeneration()) return; - if (g.stg_blocks.empty()) { - Fail(std::string("the ") + which + " STG scale is " + std::to_string(g.stg_scale) + - " and its block list is EMPTY, so the perturbed forward would be identical to the " - "conditional one and `stg_scale * (cond - perturbed)` would be exactly zero " - "(guiders.py:264). Set '" + std::string(key) + - "' to the blocks to perturb, or set the STG scale to 0.0"); - } - }; - check_blocks(*video, "video", kLtx2VideoStgBlocksExtra); - check_blocks(*audio, "audio", kLtx2AudioStgBlocksExtra); + // AN EMPTY LIST IS NOT REFUSED, and this function refused it until 2026-08-17. + // + // The refusal read: an empty `stg_blocks` beside a non-zero STG scale is a + // perturbed pass identical to the conditional one, so it is a wasted forward + // and a guidance term of exactly zero. Every clause of that is true and none + // of it makes the configuration illegal upstream, which is the only question + // a mirror gets to ask. Measured at Lightricks/LTX-2 `fd4ded7f`: + // + // - `packages/ltx-pipelines/docs/multimodal-guidance.md:13` documents it as + // THE way to turn STG off: "Set to `[]` to disable STG", in the same table + // and the same idiom as `stg_scale` -> 0.0 and `cfg_scale` -> 1.0. + // - `MultiModalGuiderParams.stg_blocks` DEFAULTS to `[]` + // (guiders.py:204, `field(default_factory=list)`). + // - `--video-stg-blocks` / `--audio-stg-blocks` are `nargs="*"` + // (args.py:979-985, :1039-1045, :1107-1113), so the flag with zero values + // parses to `[]`. `nargs="+"` was the one-character way to forbid it. + // - `LTX_2_3_HQ_PARAMS` SHIPS `stg_blocks=[]` on both modalities + // (constants.py:105, :113). + // - There is no validation of `stg_blocks` anywhere in that tree: no + // emptiness check, no length check, no range check against the block + // count. + // + // Upstream's semantics are unambiguous and are the reason `[]` is meaningful: + // `blocks=None` means EVERY block and `blocks=[]` means NO block + // (perturbations.py:26-33). The empty list is how a caller says the second + // thing, and `ApplyStgBlocksExtra` above exists to keep PRESENT-and-empty + // distinct from ABSENT for exactly that reason. Refusing it here made that + // distinction unreachable. + // + // WHAT IS STILL REFUSED, one layer down in `Ltx2GuidedDenoise`: a list that + // NAMES blocks and reaches none of them, e.g. `[28]` on a two-block DiT. That + // is a local condition rather than an upstream one — upstream only ever runs + // 48-block checkpoints and this port runs reduced ones — and it is a mismatch + // between a request and a checkpoint rather than an expressed intent. } // Everything step 0 of phase 0 produced, for the gate that decides WHICH SPACE @@ -1667,7 +1682,7 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { kv.first == kLtx2AudioStgBlocksExtra || // The VIDEO guider's row (row LTX25-GUIDED-VIDEO, #1092), // from the same parser as the audio row above - // (utils/args.py:947-1010). + // (utils/args.py:947-1066). kv.first == kLtx2VideoCfgScaleExtra || kv.first == kLtx2VideoStgScaleExtra || kv.first == kLtx2VideoRescaleScaleExtra || @@ -2603,7 +2618,7 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { } // The second half of upstream's ONE `PromptEncoder` call over - // `[prompt, negative_prompt]` (ti2vid_one_stage.py:170-178). Encoded ONLY when + // `[prompt, negative_prompt]` (ti2vid_one_stage.py:166-174). Encoded ONLY when // a guider asks: `do_unconditional_generation` is `not isclose(cfg_scale, 1.0)` // (guiders.py:275-277), and at 1.0 there is no unconditional forward, so // encoding it would be a wasted host-side 12B pass per request. diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 88a9e59ad..09fb59f9c 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -5252,6 +5252,202 @@ TEST_CASE("ltx2 t2a: the DiT forward runs ONE stream, and the old guard's reason vt::DType::kF32, /*cache=*/nullptr, &bad)); } +TEST_CASE("ltx2 dit: each CROSS perturbation gates ITS OWN direction and no other (#1092)") { + // WHY THIS CASE EXISTS, and it is a review mutation result rather than a + // symmetry a reader would ask for. + // + // Three mutations were run against the case above's sibling — the end-to-end + // `one_stage` guidance case — each built clean and each with its exit status + // captured directly: + // + // M12 the DiT ignores `video_cross_attn_skip_all` GREEN, exit 0 + // M13 the DiT ignores `audio_cross_attn_skip_all` GREEN, exit 0 + // M14 the DiT ignores BOTH RED, exit 1 + // M15 the DiT SWAPS which flag gates which direction GREEN, exit 0 + // + // A build that plumbs the flags and applies NEITHER was caught. A build that + // applies exactly one, or applies both to the wrong directions, was not — and + // the half-wrong build renders, on the DEFAULT video arm, whose + // `modality_scale` is 3.0. The end-to-end assertions cannot separate them: + // `MaxAbsDiffOf(video_first_modality, video_first_cond)` still fires with one + // direction applied, because the modality pass still differs from `cond`; and + // `Ltx2ConditioningTrace::video_modality_skipped_{a2v,v2a}` is assigned from + // the perturbation struct THE SEAM BUILT (ltx2_denoisers.cpp:315-316), which + // says what was handed over and nothing about what the DiT did with it. + // + // HOW ONE DIRECTION IS ISOLATED AT ALL, on a DiT with more than one block. + // Within a block the two directions are independent: both read the pre-cross + // snapshots `vx_pre` / `ax_pre` (transformer.py:333). ACROSS blocks they are + // not — block 1's V2A reads the video state block 0's A2V wrote — so a + // both-streams-enabled forward cannot attribute a change to a direction, and + // this fixture's DiT has two blocks. The separation therefore comes from + // upstream's own predicates (transformer.py:265-269): + // + // run_a2v = run_vx and audio is present run_vx = video.ENABLED and ... + // run_v2a = run_ax and video is present run_ax = audio.ENABLED and ... + // + // so `audio->enabled = false` with the audio stream still PRESENT runs A2V and + // not V2A, and `video->enabled = false` runs V2A and not A2V. That is the + // configuration `ltx2.h` already documents as rendering rather than failing, + // and it makes each direction observable alone. + Workspace ws; + const vllm::SafetensorsFile dit_file = vllm::SafetensorsFile::Open(ws.paths.dit); + vllm::Ltx2DitLoadOptions options; + options.widen_to_f32 = true; // `Ltx2DitForward` is f32 by declaration + const vllm::Ltx2DitCheckpoint ckpt = vllm::Ltx2LoadDitFromSafetensors(dit_file, options); + const vllm::Ltx2DitParams& params = ckpt.params; + + const int64_t video_tokens = 2, audio_tokens = 4, context_tokens = 3; + // VARYING PER TOKEN. A latent whose rows are all equal makes attention return + // a weighted average of identical values, which is the value projection again, + // and a perturbation that removes the whole branch would still be measurable — + // but the SELF-attention case above measured a constant fill turning its own + // assertion into a false negative, so the same discipline is applied here. + const auto fill = [](std::vector* v, float base) { + for (size_t i = 0; i < v->size(); ++i) { + (*v)[i] = base + 0.01F * static_cast(i % 7) - 0.02F * static_cast(i % 3); + } + }; + std::vector video_latent(static_cast(video_tokens * params.in_channels)); + std::vector audio_latent(static_cast(audio_tokens * params.audio_in_channels)); + fill(&video_latent, 0.25F); + fill(&audio_latent, 0.30F); + std::vector video_timesteps(static_cast(video_tokens), 0.5F); + std::vector audio_timesteps(static_cast(audio_tokens), 0.5F); + const float sigma = 0.5F; + std::vector video_positions(static_cast(3 * video_tokens * 2)); + std::vector audio_positions(static_cast(audio_tokens * 2)); + for (int64_t d = 0; d < 3; ++d) { + for (int64_t t = 0; t < video_tokens; ++t) { + video_positions[static_cast((d * video_tokens + t) * 2)] = + static_cast(t) * 0.04; + video_positions[static_cast((d * video_tokens + t) * 2 + 1)] = + static_cast(t + 1) * 0.04; + } + } + for (int64_t t = 0; t < audio_tokens; ++t) { + audio_positions[static_cast(t * 2)] = static_cast(t) * 0.04; + audio_positions[static_cast(t * 2 + 1)] = static_cast(t + 1) * 0.04; + } + std::vector video_context(static_cast(context_tokens * params.cross_attention_dim), + 0.05F); + std::vector audio_context( + static_cast(context_tokens * params.audio_cross_attention_dim), 0.07F); + + struct Streams { + vllm::Ltx2ModalityInput video; + vllm::Ltx2ModalityInput audio; + }; + const auto make = [&](bool video_enabled, bool audio_enabled) { + Streams s; + s.video.tokens = video_tokens; + s.video.context_tokens = context_tokens; + s.video.enabled = video_enabled; + s.video.latent = video_latent.data(); + s.video.timesteps = video_timesteps.data(); + s.video.sigma = σ + s.video.positions = video_positions.data(); + s.video.context = video_context.data(); + s.audio.tokens = audio_tokens; + s.audio.context_tokens = context_tokens; + s.audio.enabled = audio_enabled; + s.audio.latent = audio_latent.data(); + s.audio.timesteps = audio_timesteps.data(); + s.audio.sigma = σ + s.audio.positions = audio_positions.data(); + s.audio.context = audio_context.data(); + return s; + }; + const auto run = [&](Streams& io, const vllm::Ltx2DitPerturbation* p) { + return vllm::Ltx2DitForward(vt::Device{}, params, ckpt.weights, &io.video, &io.audio, + vt::DType::kF32, /*cache=*/nullptr, p); + }; + // `MaxAbsOf` / `MaxAbsDiffOf` are defined further down this file, after this + // case, so the two measurements are local rather than moved — moving them + // would churn a block three other cases read. + const auto max_abs = [](const std::vector& v) { + double m = 0.0; + for (const float x : v) m = std::max(m, std::abs(static_cast(x))); + return m; + }; + const auto moved = [](const std::vector& a, const std::vector& b) { + REQUIRE(a.size() == b.size()); + double m = 0.0; + for (size_t i = 0; i < a.size(); ++i) { + m = std::max(m, std::abs(static_cast(a[i]) - static_cast(b[i]))); + } + return m > 0.0; + }; + + vllm::Ltx2DitPerturbation skip_a2v; + skip_a2v.video_cross_attn_skip_all = true; // SKIP_A2V_CROSS_ATTN + vllm::Ltx2DitPerturbation skip_v2a; + skip_v2a.audio_cross_attn_skip_all = true; // SKIP_V2A_CROSS_ATTN + + SUBCASE("SKIP_A2V_CROSS_ATTN moves the VIDEO stream and SKIP_V2A does not") { + // `audio->enabled = false`, audio still present: A2V runs, V2A does not. + auto io = make(/*video_enabled=*/true, /*audio_enabled=*/false); + const vllm::Ltx2DitOutputs base = run(io, nullptr); + REQUIRE(base.video.size() == static_cast(video_tokens * params.out_channels)); + // NON-VACUITY. A zero output would make both comparisons below trivially + // equal, so the "did not move" half would pass on a forward that computed + // nothing at all. + REQUIRE(max_abs(base.video) > 1e-6); + + const vllm::Ltx2DitOutputs a2v_off = run(io, &skip_a2v); + REQUIRE(a2v_off.video.size() == base.video.size()); + CHECK_MESSAGE(moved(a2v_off.video, base.video), + "`video_cross_attn_skip_all` changed nothing on a forward where A2V is the only " + "cross direction running, so the flag reaches no guard " + "(transformer.py:335). The isolated-modality pass is then the conditional pass " + "again in the audio->video direction, on a recipe whose modality_scale is 3.0"); + + const vllm::Ltx2DitOutputs v2a_off = run(io, &skip_v2a); + REQUIRE(v2a_off.video.size() == base.video.size()); + CHECK_MESSAGE(v2a_off.video == base.video, + "`audio_cross_attn_skip_all` moved the VIDEO stream on a forward that runs no " + "V2A at all, so the two flags are wired to each other's directions. The flag " + "rides on the stream being WRITTEN (transformer.py:335, :367)"); + } + + SUBCASE("SKIP_V2A_CROSS_ATTN moves the AUDIO stream and SKIP_A2V does not") { + // `video->enabled = false`, video still present: V2A runs, A2V does not. + auto io = make(/*video_enabled=*/false, /*audio_enabled=*/true); + const vllm::Ltx2DitOutputs base = run(io, nullptr); + REQUIRE(base.audio.size() == static_cast(audio_tokens * params.audio_out_channels)); + REQUIRE(max_abs(base.audio) > 1e-6); + + const vllm::Ltx2DitOutputs v2a_off = run(io, &skip_v2a); + REQUIRE(v2a_off.audio.size() == base.audio.size()); + CHECK_MESSAGE(moved(v2a_off.audio, base.audio), + "`audio_cross_attn_skip_all` changed nothing on a forward where V2A is the only " + "cross direction running, so the flag reaches no guard (transformer.py:367)"); + + const vllm::Ltx2DitOutputs a2v_off = run(io, &skip_a2v); + REQUIRE(a2v_off.audio.size() == base.audio.size()); + CHECK_MESSAGE(a2v_off.audio == base.audio, + "`video_cross_attn_skip_all` moved the AUDIO stream on a forward that runs no " + "A2V at all, so the two flags are wired to each other's directions"); + } + + SUBCASE("the isolated-modality pass's OWN configuration moves both streams") { + // Both directions off with both streams enabled, which is what + // `_guided_denoise` builds for the `mod` pass (denoisers.py:125-138, + // `blocks=None` on both types). This is the shipped combination; the two + // subcases above are what separates its halves. + auto io = make(/*video_enabled=*/true, /*audio_enabled=*/true); + const vllm::Ltx2DitOutputs base = run(io, nullptr); + REQUIRE(max_abs(base.video) > 1e-6); + REQUIRE(max_abs(base.audio) > 1e-6); + vllm::Ltx2DitPerturbation both; + both.video_cross_attn_skip_all = true; + both.audio_cross_attn_skip_all = true; + const vllm::Ltx2DitOutputs off = run(io, &both); + CHECK(moved(off.video, base.video)); + CHECK(moved(off.audio, base.audio)); + } +} + TEST_CASE("ltx2 t2a: a SKIPPED step runs no forward and reuses the last prediction") { // `should_skip_step` is `step % (skip_step + 1) != 0` (guiders.py:287-291), so // `skip_step = 1` skips every ODD step. Upstream then returns @@ -5492,7 +5688,7 @@ TEST_CASE("ltx2 t2a: the guider is handed x0 predictions and not raw velocities" {"perturbed", t.t2a_first_perturbed_velocity, t.t2a_first_perturbed}, }; for (const Arm& arm : arms) { - INFO("arm = " << arm.name); + INFO("arm = " << std::string(arm.name)); REQUIRE(arm.velocity.size() == n); REQUIRE(arm.x0.size() == n); @@ -5767,7 +5963,8 @@ TEST_CASE("ltx2 one_stage: all four guidance arms are combined in X0 space (#109 // The arm this case sits on, pinned as a LOCAL fact before anything is read off // a render. `rescale_scale = 0.7` on the 2.4/2.5 lineage (ltx-pipelines - // utils/constants.py:40-80, reached through `_PARAMS_SINCE_VERSION`). + // utils/constants.py:53 video / :63 audio, reached through `_PARAMS_SINCE_VERSION` at + // :130-133). const vllm::Ltx2PipelineRecipe recipe = vllm::ResolveLtx2PipelineRecipe("one_stage", "2.5"); REQUIRE(recipe.phases.size() == 1); const vllm::Ltx2MultiModalGuiderParams row = recipe.phases[0].video_guidance; @@ -5813,11 +6010,23 @@ TEST_CASE("ltx2 one_stage: all four guidance arms are combined in X0 space (#109 // OVER leaves the params untouched and renders. CHECK(t.video_perturbed_blocks == std::vector{1}); CHECK(t.video_audio_perturbed_blocks == std::vector{1}); + // WHAT THESE TWO MEASURE, said exactly, because the message they used to carry + // claimed more. `video_modality_skipped_{a2v,v2a}` is assigned from the + // `Ltx2DitPerturbation` THE SEAM BUILT and handed over + // (ltx2_denoisers.cpp:315-316), so it says the seam asked for both directions + // — which is `blocks=None` on both types (denoisers.py:125-138) — and says + // NOTHING about what the DiT did with the request. What the DiT does with each + // flag is gated separately and per direction by + // "ltx2 dit: each CROSS perturbation gates ITS OWN direction and no other", + // which exists because mutations that applied exactly one direction, or + // swapped the two, survived this case. CHECK_MESSAGE(t.video_modality_skipped_a2v, - "the isolated-modality pass reached the DiT without SKIP_A2V_CROSS_ATTN, so it " - "is the conditional pass again (denoisers.py:130-136)"); + "the seam built the isolated-modality pass WITHOUT asking for SKIP_A2V_CROSS_ATTN, " + "so that pass is the conditional pass again in the audio->video direction " + "(denoisers.py:125-138)"); CHECK_MESSAGE(t.video_modality_skipped_v2a, - "the isolated-modality pass reached the DiT without SKIP_V2A_CROSS_ATTN"); + "the seam built the isolated-modality pass WITHOUT asking for " + "SKIP_V2A_CROSS_ATTN"); const size_t n = t.video_first_latent.size(); REQUIRE(n > 0); @@ -5847,7 +6056,7 @@ TEST_CASE("ltx2 one_stage: all four guidance arms are combined in X0 space (#109 {"modality", t.video_first_modality_velocity, t.video_first_modality}, }; for (const Arm& arm : arms) { - INFO("arm = " << arm.name); + INFO("arm = " << std::string(arm.name)); REQUIRE(arm.velocity.size() == n); REQUIRE(arm.x0.size() == n); @@ -5911,7 +6120,7 @@ TEST_CASE("ltx2 one_stage: all four guidance arms are combined in X0 space (#109 "self-attention perturbation did not reach the forward"); CHECK_MESSAGE(MaxAbsDiffOf(t.video_first_modality, t.video_first_cond) > 1e-6 * latent_span, "the isolated-modality pass returned the conditional pass's own tensor, so the " - "cross-attention perturbation did not reach the forward (transformer.py:335,366)"); + "cross-attention perturbation did not reach the forward (transformer.py:335,367)"); // ── the guider's output is the guider's output ──────────────────────────── // @@ -6043,11 +6252,20 @@ TEST_CASE("ltx2 one_stage: all four guidance arms are combined in X0 space (#109 } } -TEST_CASE("ltx2 one_stage: rescale_scale 0 is the control, and the MODALITY term is in it") { - // #1039's control on the VIDEO row, which carries a term the T2A control could - // not: `modality_scale` is 3.0 here and pinned to 1.0 there - // (t2a_one_stage.py:200-202), so the isolated-modality arm has never been in a - // space control before. +TEST_CASE("ltx2 one_stage: rescale_scale 0 is the control and the modality term is INERT in it") { + // #1039's control on the VIDEO row. It runs with `modality_scale = 3.0`, which + // the T2A control could not carry because that pipeline pins it to 1.0 + // (t2a_one_stage.py:202) — so the isolated-modality arm is inside a space + // control here for the first time. + // + // WHAT THAT IS WORTH, measured rather than implied, and the case's own title + // said more than the number supports until 2026-08-17. Presence is coverage, + // not discriminating power: the third measurement below pins `modality_scale` + // to 1.0 and the disagreement at the shipped rescale barely moves. A reader + // must not lean on this control for modality coverage. THE MODALITY ARM'S GATE + // IS THE PER-ARM INVARIANT in the case above, whose `modality` row is the one + // mutation M4 (the `mod` pass left in velocity space) turns red; this control + // gates the RESCALE, on a guider that happens to have four terms. // // The case above would be testing something OTHER than the defect if it also // fired at `rescale_scale = 0`, because `MultiModalGuider.calculate`'s linear @@ -6125,14 +6343,39 @@ TEST_CASE("ltx2 one_stage: rescale_scale 0 is the control, and the MODALITY term const double at_zero = compare(0.0); const double at_default = compare(0.7); - INFO("relative disagreement: at rescale 0.0 = " << at_zero - << " at rescale 0.7 = " << at_default); + // The same pair with the modality term switched OFF, which is what T2A's + // control already measured. Restored afterwards so nothing below reads a + // mutated params object. + const double shipped_modality = params.modality_scale; + params.modality_scale = 1.0; + const double at_zero_no_modality = compare(0.0); + const double at_default_no_modality = compare(0.7); + params.modality_scale = shipped_modality; + + INFO("relative disagreement: at rescale 0.0 = " + << at_zero << " at rescale 0.7 = " << at_default + << " | modality_scale pinned to 1.0: at 0.0 = " << at_zero_no_modality + << " at 0.7 = " << at_default_no_modality); // AT 0.0 THE TWO SPACES ARE THE SAME FUNCTION, to f32 rounding — with the - // modality term present, which is the half this control adds. + // modality term present, which is the arm this control adds over T2A's. CHECK(at_zero < 1e-4); // AT THE SHIPPED 0.7 THEY ARE NOT, by orders of magnitude more. CHECK(at_default > 1e-2); CHECK(at_default > 100.0 * at_zero); + + // AND THE MODALITY TERM IS NOT WHAT SEPARATES THEM. Asserted rather than left + // in prose, because the case's own comment implied the opposite and a later + // reader would otherwise treat this control as modality coverage. The two + // `0.7` numbers agree to well inside a factor of two: adding a fourth linear + // term changes what the rescale is computed over and does not change whether + // the rescale is the term that breaks the equivalence. + CHECK(at_zero_no_modality < 1e-4); + CHECK(at_default_no_modality > 1e-2); + CHECK_MESSAGE(at_default_no_modality > 0.5 * at_default, + "the modality term turned out to carry the disagreement after all, which would " + "make this control modality coverage rather than rescale coverage"); + CHECK_MESSAGE(at_default_no_modality < 2.0 * at_default, + "the modality term turned out to carry the disagreement after all"); } TEST_CASE("ltx2 one_stage: post_process_latent runs AFTER the guider, not per arm (#1092)") { @@ -6240,7 +6483,7 @@ TEST_CASE("ltx2 one_stage: post_process_latent runs AFTER the guider, not per ar &t.video_first_perturbed_velocity, &t.video_first_modality_velocity}; const char* names[] = {"cond", "uncond", "perturbed", "modality"}; for (size_t k = 0; k < 4; ++k) { - INFO("arm = " << names[k]); + INFO("arm = " << std::string(names[k])); REQUIRE(arms[k]->size() == n); REQUIRE(velocities[k]->size() == n); double worst = 0.0; @@ -6338,6 +6581,58 @@ TEST_CASE("ltx2 guided video: the refusals that would otherwise RENDER (#1092)") } } + SUBCASE("an EMPTY stg_blocks is SERVED - it is upstream's own way to disable STG") { + // THIS SUBCASE ASSERTED A REFUSAL UNTIL 2026-08-17. Measured at + // Lightricks/LTX-2 `fd4ded7f`: `docs/multimodal-guidance.md:13` documents + // "Set to `[]` to disable STG"; `MultiModalGuiderParams.stg_blocks` defaults + // to `[]` (guiders.py:204); the flags are `nargs="*"` (args.py:979-985) so + // the empty list has a CLI spelling; `LTX_2_3_HQ_PARAMS` ships it on both + // modalities (constants.py:105, :113); and nothing in that tree validates + // the list at all. Refusing it made this port reject a configuration its + // reference documents, ships and cannot express any other way. + // + // Upstream does NOT skip the pass either: `do_perturbed_generation` reads + // `stg_scale` alone (guiders.py:279-281), so the "ptb" entry is appended and + // the batch carries a sample whose result equals `cond`. The forward count + // below is that fact, and it is why an empty list disables the STG SIGNAL + // and not the STG COST. + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(OneStageParams(ws.paths)); + REQUIRE(engine != nullptr); + vllm::multimodal::VideoGenParams gen = OneStageGen(ws.root + "/stg_empty"); + gen.extras[vllm::multimodal::kLtx2VideoStgBlocksExtra] = ""; + gen.extras[vllm::multimodal::kLtx2AudioStgBlocksExtra] = ""; + (void)engine->Generate(gen); // it RENDERS; a throw fails the case + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + REQUIRE(t.video_guided); + // The pass still ran, because the scale still asks for it. + CHECK(t.video_perturbed_forwards == 1); + // And it perturbed nothing, read off the mask handed to the DiT. + CHECK(t.video_perturbed_blocks.empty()); + CHECK(t.video_audio_perturbed_blocks.empty()); + // So the STG term is not merely small, it is EXACTLY zero: the perturbed arm + // is the conditional arm bit for bit. An exact comparison, because a + // tolerance here would also pass on a build that perturbed a block and + // happened to move little. + REQUIRE(!t.video_first_cond.empty()); + CHECK_MESSAGE(t.video_first_perturbed == t.video_first_cond, + "an empty stg_blocks perturbed something, so PRESENT-and-empty was collapsed " + "onto some other value (`blocks=None` is EVERY block upstream, " + "perturbations.py:26-33)"); + // The control that this is about EMPTINESS and not about the extra being + // read at all: the same render with a real block moves the arm. + vllm::multimodal::VideoGenParams named = OneStageGen(ws.root + "/stg_named"); + named.extras[vllm::multimodal::kLtx2VideoStgBlocksExtra] = "1"; + named.extras[vllm::multimodal::kLtx2AudioStgBlocksExtra] = "1"; + (void)engine->Generate(named); + const vllm::multimodal::Ltx2ConditioningTrace n = ltx->last_conditioning(); + CHECK(n.video_perturbed_blocks == std::vector{1}); + CHECK(n.video_first_perturbed != n.video_first_cond); + } + SUBCASE("a recipe that fixes its guidance refuses the override rather than applying it") { // `allow_guidance_override = false` on the distilled two-stage recipe // (ltx2_recipes.py:125-158), whose scales are distilled INTO the weights. From 09aee8cc25e99c480d0842db1e0da220e294fc53 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 08:51:51 +0000 Subject: [PATCH 5/5] docs(LTX25-GUIDED-VIDEO): the cross-perturbation row says what the gate measures (#1092) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `docs/FEATURES.md`'s evidence cell for the cross-attention perturbations read "asserts the isolated-modality pass reached the DiT with both directions off and moved the tensor", which is exactly the claim the review found overstated: that assertion is satisfied by a build applying one direction, or the wrong two. The cell now names what the gate actually does — each direction alone, on a forward where one stream is PRESENT but DISABLED so only that direction runs, and with the other flag required to leave the stream bit-identical, which is the half that detects a swap. Cell measured at 218 of 220 chars and the page's prose-paragraph count is unchanged at 21 of 21, so nothing else on the page has to move. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- docs/FEATURES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 1287984db..d8d566ec9 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -176,7 +176,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 18 mutations, 17 DETECTED (four by review of a conditional-only #1039 gate) and the 18th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only AUDIO-ONLY guided arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | | LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 ([#1039](https://github.com/mudler/vllm.cpp/issues/1039)) | | LTX-2.5 VIDEO guidance | LTX-2.5 DiT, joint video+audio | `test_ltx2_video`'s `ltx2 one_stage:` cases through `Generate`; all FOUR arms carry the x0 invariant and the guider output replays EXACTLY | `--pipeline-kind one_stage` runs `_guided_denoise`: 4 forwards/step, combined per modality in **x0**. Was ONE unguided forward, every `video_guidance` field dead ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) | -| LTX-2.5 cross-attention perturbations | LTX-2.5 DiT | `test_ltx2_video` asserts the isolated-modality pass reached the DiT with both directions off and moved the tensor | `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN` ported, which `modality_scale = 3.0` selects on every video row. The DEVICE forward takes no perturbations, so that pass is refused there by name | +| LTX-2.5 cross-attention perturbations | LTX-2.5 DiT | `test_ltx2_video` gates each direction ALONE, on a forward where one stream is PRESENT but DISABLED so only that one runs: the flag moves the stream it writes, the other leaves it bit-identical. Swapping the two is RED | `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN` ported, which `modality_scale = 3.0` selects on every video row. The DEVICE forward takes no perturbations, so that pass is refused there by name | | LTX-2.5 guidance knobs | LTX-2.5 request surface | `test_ltx2_video` renders with an override and refuses one on a fixed recipe | Seven video/audio guider extras mirroring `default_1_stage_arg_parser`, plus a negative embeds pair for a tower-less engine. Refused whole on `distilled_two_stage` and `retake`, whose guidance is distilled in | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | MTP speculation DEPTH (`num_speculative_tokens` > 1) | Qwen3.5/3.6 `mtp.*` heads | k=1..4 through the loader, greedy tokens unmoved, two witnesses per arm: the draft decode forwards the propose RAN, and whether the DELIVERED draft row varied with depth. `test_mtp_depth` 5/5, 63 assertions | Default stays k=1. NO speed claim at k>1. Drafts are proposed and verified, never ACCEPTED, and neither witness proves per-column provenance. Both await the owed DGX gate (#81) |