From 360295e76170a7aa24d7f835f9c1bade693f9262 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 16 Aug 2026 10:39:15 +0000 Subject: [PATCH 1/9] spec(LTX25-T2A-ONE-STAGE): text-to-audio, and the two refusals whose stated reasons do not describe it (#1005) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `T2AOneStagePipeline` (`t2a_one_stage.py:43`, `__call__` at `:109` @ `fd4ded7f`) is LTX-2.5's audio-only arm and the first path here that would return a render with no picture. This commit is the spec, ahead of the implementation. Three findings shaped the design, and two of them are stale refusals rather than missing code. FIRST, `Ltx2DitForward` refuses a one-stream call at `ltx2_dit.cpp:765` because "LTXModelType.VideoOnly and LTXModelType.AudioOnly carry a different weight contract". That is a claim about the CHECKPOINT, and T2A never loads one: it reads the ordinary AudioVideo file through `LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP` (`model_configurator.py:228-239`) and builds an AudioOnly MODULE out of the subset. Every line of the forward below that check is already written against `video != nullptr`, so the guard covers a path the code already handles. SECOND, the same message advises `enabled = false` as the substitute, and it is not one. Upstream's predicate is `run_v2a = run_ax and (video is not None and vx.numel() > 0)` (`transformer.py:269`) — it tests PRESENCE, not `enabled` — so a disabled-but-present video stream still feeds video-to-audio cross attention from a latent T2A never meant to exist, and still returns a finished waveform. Our port mirrors that polarity at `ltx2_dit.cpp:251`, so the trap is live here too. THIRD, and this is the part the dispatch did not anticipate: the engine has no guided denoiser at all. `git grep -n 'guid\|cfg_scale' src/vllm/multimodal/ltx2_video.cpp` returns 0 against 66 for `ltx2` in the same file as the control. That is correct for `distilled_two_stage`, which builds a `SimpleDenoiser` upstream too, and wrong for T2A, whose CLI defaults are `cfg_scale=7.0` and `stg_scale=1.0` — three forwards per step. The guidance bricks are all ported and gated and none has a product caller. The entry-point question the dispatch raised resolves the other way from what it expected: the audio-only shape FITS `Ltx2VideoEngine`, because upstream itself expresses T2A's duration through a placeholder `VideoPixelShape` (`t2a_one_stage.py:37-40`, `:163-167`) and `VideoResult` already carries `frame_count` and `audio_path` as independent fields. Section 1 derives it. Issue #1005 is linked from this spec and from `.agents/issue-index.md`, appended rather than edited. The index row and the spec agree, and the pull request body will carry the same number. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 1 + .agents/specs/ltx25-t2a-one-stage.md | 518 +++++++++++++++++++++++++++ 2 files changed, 519 insertions(+) create mode 100644 .agents/specs/ltx25-t2a-one-stage.md diff --git a/.agents/issue-index.md b/.agents/issue-index.md index fa9c1f7ab..fbe67d42c 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -269,3 +269,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#986](https://github.com/mudler/vllm.cpp/issues/986) | `LTX25-DFR-PIPELINE` | `DFRPipeline` (`dfr_pipeline.py` + `dfr_layout.py` @ `fd4ded7f`) has no representation in this tree — `git grep -i dfr` over `src include tests examples docs` returns ZERO product hits, against 87 `ltx2` hits in `ltx2.cpp` alone as the positive control ([#604](https://github.com/mudler/vllm.cpp/issues/604)). It matters beyond its own feature because it is the **ONLY** upstream consumer of the temporal x2 latent upsampler this project already ships: [`ltx25-temporal-upsampler.md`](specs/ltx25-temporal-upsampler.md) section 7 records the operator as ported, loader-parsed and gated but *"not reachable from any shipped pipeline"*, and `docs/FEATURES.md` carries that as `Temporal x2 ups gated, UNDRIVEN`. No issue tracked that state — `undriven` returned zero hits across open and closed issues. DFR also needs the generated-keyframe-slot READBACK that [#920](https://github.com/mudler/vllm.cpp/issues/920) refused by name and left owed after it CLOSED, so the debt had a spec bullet and no open issue; it needs the LAYOUT and the EXTRACTION but **not** the standalone single-frame decode, because DFR never decodes its slots — it hands them to the spatial upsampler (`dfr_pipeline.py:348`) and feeds them back as `initial_keyframes` (`:364`). Spec [`ltx25-dfr-pipeline.md`](specs/ltx25-dfr-pipeline.md). Campaign [#644](https://github.com/mudler/vllm.cpp/issues/644) | feature | | [#995](https://github.com/mudler/vllm.cpp/issues/995) | `ENG-EXPERT-STREAM` | `check-env-doc` and `test_check_env_doc` RED in `scripts/agent-preflight.sh` on **every** branch based on `3005447f8`: `VT_MOE_EXPERT_STREAM`, `VT_MOE_EXPERT_STREAM_SLOTS` and `VT_MOE_EXPERT_STREAM_SLOT_BYTES` are read from `src/vllm/model_executor/models/qwen3_5.cpp` (`:5145`, `:5164` @ `3005447f8`) and appear in neither `docs/ENVIRONMENT.md` (`grep -c` returns **0**) nor `scripts/env-doc-allowlist.txt`. Arrived with [#993](https://github.com/mudler/vllm.cpp/pull/993). Found while gating [#986](https://github.com/mudler/vllm.cpp/issues/986) and proved pre-existing with a matched-arm check rather than asserted: the three sites are in a file that branch does not touch, and `git diff origin/main...HEAD | grep '^+.*VT_MOE_EXPERT'` returns nothing. NOT fixed in flow: documenting a knob means stating its default and when to touch it, and the expert-streamer's slot accounting belongs to the row that added it, so a plausible-sounding entry written by a passer-by is how `docs/ENVIRONMENT.md` stops being trustworthy. It is a PRE-FLIGHT gate, so it fails before every edit and presents to each author as a red their own diff caused, which is the shape [#965](https://github.com/mudler/vllm.cpp/issues/965) and [#968](https://github.com/mudler/vllm.cpp/issues/968) both took | bug | | [#987](https://github.com/mudler/vllm.cpp/issues/987) | `LTX25-RETAKE` | Two `ltx-2.5` refusal messages state reasons that are no longer true. (a) `src/vllm/multimodal/ltx2_video.cpp:1608 @ 0e1bee42f` says "nothing reads `ref_video_dir` at all", and MiniMax-H3 has always consumed the directory in full (`ReadReferenceClipChw`, `src/vllm/multimodal/minimax_h3_video.cpp:135 @ 0e1bee42f`, called at `:650`); [#975](https://github.com/mudler/vllm.cpp/issues/975) inherited the wider claim from this message. The claim that holds is narrower: the LTX-2.5 engine never reads the directory's CONTENTS. (b) `ltx2_video.cpp:1636-1638 @ 0e1bee42f` says "there is no AUDIO_VAE_ENCODER key filter", and `c2019b0e3` landed `Ltx2AudioVaeEncoderKeyRules()` (`include/vllm/model_executor/models/ltx2_audio_input.h:73 @ 0e1bee42f`) with a live call through `Ltx2EncodeAudioToLatent`. Both rewritten in the `WHAT IS *NOT* THE REASON` shape in the same flow, with one assertion tied to the LOCAL fact that the LTX side now reads the directory | bug | +| [#1005](https://github.com/mudler/vllm.cpp/issues/1005) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 text-to-audio (`T2AOneStagePipeline`, `t2a_one_stage.py:43`, `__call__` at `:109` @ `fd4ded7f`) is absent, and the three blockers are not the ones a reader would guess. (a) `Ltx2DitForward` refuses a one-stream call at `src/vllm/model_executor/models/ltx2_dit.cpp:765 @ 332aed738`, citing a weight contract that describes a checkpoint T2A never loads — upstream reads the ordinary AudioVideo FILE through `LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP` (`model_configurator.py:228-239`). (b) The same message advises `enabled=false` as the substitute, and it is NOT: `run_v2a = run_ax and (video is not None and vx.numel() > 0)` (`transformer.py:269`) tests PRESENCE, not `enabled`, so a disabled-but-present video stream still feeds v2a cross attention and still returns a finished waveform; our port mirrors that polarity at `ltx2_dit.cpp:251`. (c) The engine has NO guided denoiser at all — `git grep -n 'guid\\|cfg_scale' src/vllm/multimodal/ltx2_video.cpp` is 0 against 66 for `ltx2` in the same file as the control — while T2A defaults to `cfg_scale=7.0` and `stg_scale=1.0`, i.e. THREE forwards per step. Spec [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | feature | diff --git a/.agents/specs/ltx25-t2a-one-stage.md b/.agents/specs/ltx25-t2a-one-stage.md new file mode 100644 index 000000000..cea503b93 --- /dev/null +++ b/.agents/specs/ltx25-t2a-one-stage.md @@ -0,0 +1,518 @@ +# LTX-2.5 — text-to-audio (`T2AOneStagePipeline`) + +Row: `LTX25-T2A-ONE-STAGE`. Campaign: [`ltx-2-5.md`](ltx-2-5.md) (operator-owned; +**not edited by this row**). Issue: +[#1005](https://github.com/mudler/vllm.cpp/issues/1005). Parent campaign issues: +[#644](https://github.com/mudler/vllm.cpp/issues/644), +[#435](https://github.com/mudler/vllm.cpp/issues/435). + +Upstream pin: + +| Reference | Revision | +|---|---| +| Lightricks/LTX-2 (`packages/ltx-core`, `packages/ltx-pipelines`) | `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca` | + +Read from a local checkout at that revision. `git rev-parse HEAD` was run on a +clean tree (`git status --short` empty) before any anchor below was taken. + +--- + +## 0. Honesty statement — what this row does and does not claim + +This row makes LTX-2.5 render **audio with no video**, from a prompt, through +the shared video seam. It is the first path in this tree that returns a +`VideoResult` carrying zero frames. + +**Upstream ships no tests at this pin.** `find /home/mudler/_git/LTX-2 -name +'test_*.py'` returns **0**; the broader `find . -iname '*test*' -not -path +'./.git/*'` that row `LTX25-A2V-AUDIO-INPUT` ran returns empty as well. So "port +the upstream tests in the same change" has nothing to port, and the obligation +becomes what §5 does instead: pin upstream's *behaviours* — its constants, its +predicates, its refusals — each against a `file:line` anchor, and tie at least +one assertion to a **local** fact so the suite can see a claim about this tree go +stale. This is stated rather than quietly skipped. + +**No render on real weights is claimed.** The row is gated on the reduced +fixture. The GPU is out of bounds for this row (a render ladder holds +`dgx.casa`, and the box OOM-reboots when its 119 GiB unified pool is exhausted), +so the real-checkpoint T2A render is owed and named under `## Owed`. + +**The device arm is refused by name, not served.** See §7. + +--- + +## 1. Does the audio-only shape fit `Ltx2VideoEngine`? YES — and here is the +## derivation, because the opposite answer was the expected one + +The dispatch that opened this row expected a possible `NEEDS_DECISION` on the +entry point. It is not needed, and the reason is upstream's own shape rather +than a convenience here. + +**Upstream expresses T2A's duration through a VIDEO pixel shape.** +`t2a_one_stage.py:37-40` declares `_AUDIO_ONLY_PLACEHOLDER_RES = 512` and says in +as many words that "Audio-only generation reads ``frames`` and ``fps`` from the +pixel shape via ``AudioLatentShape.from_video_pixel_shape`` (height/width are +unused)". `__call__` then passes `width=512, height=512, frames=num_frames, +fps=frame_rate` into the SAME `DiffusionStage.__call__` every video pipeline +calls (`:163-167`, against `utils/blocks.py:501-512`). So the request shape T2A +needs is exactly the request shape `VideoGenParams` already carries: +`num_frames`, `frame_rate` (through `duration_seconds` or the recipe), `steps`, +`seed`, `prompt`, `output_dir`. + +**And `VideoResult` can already say "no picture".** It carries `frame_dir`, +`audio_path`, `frame_count`, `sample_rate` and `mux_argv` as independent fields +(`include/vllm/multimodal/video_engine.h`, `struct VideoResult`). An audio-only +result is `frame_count = 0`, an empty `frame_dir`, an `audio_path`, and an EMPTY +`mux_argv` — there is nothing to mux, and composing an ffmpeg argv over a frame +pattern that matches no file would hand the caller a command that fails. + +So the seam represents the behaviour, and AGENTS.md §"Shared seams" then binds +the other way round: a distinct entry point would be the parallel path that +section forbids. What this row does NOT do is bend the video path: the audio-only +render lives in its own translation unit (§3), mirroring upstream's own file +structure, and `Generate` branches to it before it resolves any video geometry. + +--- + +## 2. The gap, derived at `332aed738` + +Three blockers, each named by the symbol that has to change. None of them is +"the pipeline file is missing". + +### 2.1 `Ltx2DitForward` refuses a one-stream call, and its stated reason does not describe T2A + +`src/vllm/model_executor/models/ltx2_dit.cpp:765` is + +``` +VT_CHECK(video != nullptr && audio != nullptr, "... LTXModelType.VideoOnly and + LTXModelType.AudioOnly carry a different weight contract and are not ported"); +``` + +Re-derived at this tree rather than inherited. The reason is a claim about the +**checkpoint**, and it is true of a checkpoint that was SAVED as an `AudioOnly` +model. T2A does not load one. `t2a_one_stage.py:81-93` calls +`DiffusionStage.from_checkpoint(model_paths.transformer(), ..., +model_configurator=LTXAudioOnlyModelConfigurator, +model_sd_ops=LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP)`: the FILE is the ordinary +AudioVideo transformer, and the SDOps map (`model_configurator.py:228-239`) +merely restricts which of its keys are read — `audio_attn1`, `audio_attn2`, +`audio_ff`, `audio_patchify`, `audio_proj_out`, `audio_adaln_single`, +`audio_prompt`, `audio_scale_shift_table`. The comment at `:78-82` of the +pipeline says why: so "the video weights are never even read from disk". The +weight contract `EnumerateLtx2DitTensors` describes is therefore the one T2A's +file satisfies, and the refusal's cause is not this row's blocker. + +**What is left of that refusal after the lift is stated rather than dropped.** A +checkpoint containing ONLY the audio subset still cannot be loaded here, because +`Ltx2LoadDitFromSafetensors` enumerates the AV contract. That half stays, and it +moves to where it is true — the loader — instead of guarding a forward that +already handles the case. + +**The forward already handles it.** Every line of `Ltx2DitForward` below the +check is written against `video != nullptr` / `have_both` +(`ltx2_dit.cpp:786-869`), and `Ltx2TransformerBlockForward` computes +`run_vx` / `run_a2v` / `run_v2a` from `video_x != nullptr && tv > 0` +(`ltx2_dit.cpp:248-251`). Lifting the check reaches a path that was already +written; it does not add one. + +### 2.2 `enabled = false` is NOT the same shape, and the difference renders + +The refusal's own message says to "use `enabled` to run one stream of an AV +model", and `include/vllm/model_executor/models/ltx2.h:506-508` repeats it. That +advice is correct for the A2V and joint pipelines. **It is wrong for T2A**, and +this is the detail that fails silently if guessed. + +Upstream's predicate is + +``` +run_v2a = run_ax and (video is not None and vx.numel() > 0) # transformer.py:269 +``` + +It tests `video is not None`, **not** `video.enabled`. So an AV forward handed a +present-but-disabled video stream still runs video->audio cross attention, taking +its context from a video latent that T2A never intended to exist — and returns a +finished waveform. Our port mirrors that polarity exactly at +`ltx2_dit.cpp:251`, so the same trap is live here. + +`video = nullptr` is the only shape that reproduces `LTXModel.forward(video=None, +...)` (`model.py:492-538`, the `video_args = ... if video is not None else None` +at `:501`). It is also what makes `_init_preprocessors`' AudioOnly arm +(`model.py:350-365`, a plain `TransformerArgsPreprocessor` with no +`cross_scale_shift_adaln` and no `cross_gate_adaln`) equivalent to our +`have_both == false` path, which passes `other = nullptr` into `PrepareStream` +and therefore builds neither. + +### 2.3 The engine has NO guided denoiser, and T2A's defaults turn one on + +This is the finding the dispatch did not anticipate and it is the largest part of +the row. + +`Ltx2VideoEngine::Generate` runs exactly ONE DiT forward per step +(`ltx2_video.cpp`, the `Ltx2DitForward` / `Ltx2DitForwardDevice` call in the +denoise loop) and never reads a guider parameter: `git grep -n +'guid\|Guider\|cfg_scale' src/vllm/multimodal/ltx2_video.cpp` returns zero hits +against a positive control of 66 for `ltx2` in the same file. That is correct for +`distilled_two_stage`, which builds a `SimpleDenoiser` upstream too. It is not +correct for T2A. + +T2A builds a `FactoryGuidedDenoiser` (`t2a_one_stage.py:154-161`) over +`MultiModalGuiderParams` whose CLI defaults are the params table's +`audio_guider_params` — `cfg_scale=7.0`, `stg_scale=1.0`, `rescale_scale=0.7`, +`stg_blocks=[28]` on the 2.3/2.4/2.5 lineage (`utils/constants.py:58-66`, +`:82-87`, `:118`) — with `modality_scale` pinned to **1.0** by the CLI itself +(`t2a_one_stage.py:200-203`: "Audio-only generation has no video modality, so the +video->audio (v2a) cross-modal guidance is meaningless here. 1.0 disables it"). + +Read against `guiders.py:275-287`, those defaults mean +`do_unconditional_generation()` is TRUE (`cfg_scale != 1.0`), +`do_perturbed_generation()` is TRUE (`stg_scale != 0.0`) and +`do_isolated_modality_generation()` is FALSE. **Three forwards per step: cond, +uncond-text, uncond-perturbed.** Shipping T2A on the engine's single-forward path +would be a different trajectory from upstream's default, and no shape, token +count or file length could see it. + +The bricks exist and are gated; the WIRING does not. `Ltx2MultiModalGuidance` +(`ltx2_pipeline.h:323`), `Ltx2CfgDelta`, `Ltx2StgDelta`, +`Ltx2GuiderParamsForSigma`, `Ltx2PerturbationConfig` and +`Ltx2BatchedPerturbationConfig` are all ported with goldens and have **no product +caller** — the "test-only driver" shape `.agents/reachability.md` enumerates. What +is genuinely absent is one thing: the DiT forward runs upstream's +`perturbations=None` path only, and says so at +`include/vllm/model_executor/models/ltx2.h:42-46`. + +--- + +## 3. Design + +### 3.1 A new translation unit, mirroring upstream's file + +`include/vllm/model_executor/models/ltx2_t2a.h` + +`src/vllm/model_executor/models/ltx2_t2a.cpp`, holding `Ltx2T2aGenerate` — the +port of `T2AOneStagePipeline.__call__` (`t2a_one_stage.py:109-172`). +`Ltx2VideoEngine::Generate` branches to it at the top, before it resolves any +video geometry. This mirrors upstream's file structure, which AGENTS.md §"Shared +seams" requires, and it follows the precedent +`.agents/specs/ltx25-a2v-audio-input.md` §2 set ("A new translation unit per +concern, mirroring upstream's file structure rather than growing +`ltx2_video.cpp`"). + +It is NOT a parallel path: it is reached only through +`VideoEngine::Generate`, it consumes the same `Impl`-owned weights, and every +numeric it uses is an already-gated brick — `Ltx2SigmaSchedule`, +`Ltx2AudioPatchify` / `Ltx2AudioUnpatchify` / `Ltx2AudioPatchTimings`, +`Ltx2DitForward`, `Ltx2MultiModalGuidance`, `Ltx2EulerStep`, +`Ltx2AudioDecoderForward`, `Ltx2VocoderWithBweForward`, `MiniMaxH3WriteWav`. + +The duplication it does accept is the audio-only denoise loop. That is deliberate: +the joint loop in `ltx2_video.cpp` is ~200 lines of video-stream construction the +audio-only path has no counterpart for, and threading an `is_t2a` flag through it +would put nine new branches inside a function that already runs 1935 lines. + +### 3.2 Call order, mirroring `t2a_one_stage.py:123-172` + +1. `require_num_frames_source` (`:123`, `utils/blocks.py:894-905`). Auto duration + with no `DurationHead` is upstream's own refusal. Mirrored: this engine + constructs no duration head at all (`duration_head_path` is refused at load by + `CheckUnservedExtras`, #611), so an absent `num_frames` is refused with the + message naming the head. +2. Encode the prompt AND the negative prompt (`:127-135`). Upstream encodes both + in one `PromptEncoder` call and takes `.audio_encoding` from each + (`:134-135`). Here both go through `Ltx2EncodePromptToConditioning` and the + connector, and only the AUDIO half of each is kept — the video half is + computed and discarded, exactly as upstream computes both encodings and uses + only the audio one. +3. `resolve_num_frames` (`:137-139`) — an explicit count is returned verbatim + (`utils/blocks.py:920-921`). +4. `sigmas = self._scheduler.execute(steps=num_inference_steps)` (`:141-143`). + `LTX2Scheduler()` is HARD-CODED at `:67`, so this is `Ltx2SigmaSchedule` and + the recipe carries no distilled sigma table. The token count the shift is + derived from is the AUDIO latent's own (`schedulers.py:32` reads + `math.prod(latent.shape[2:])` of the unpatchified target). +5. `create_multimodal_guider_factory(params, negative_context)` (`:149-152`). + A plain `MultiModalGuiderParams` becomes a sigma-independent guider; the + sigma-binned factory arm is not reachable from any surface here and is owed. +6. The stage call with `video=None` and `audio=ModalitySpec(context=a_context_p)` + (`:154-170`). No initial latent, no freeze: the audio starts as pure noise. +7. `return self.audio_decoder(audio_state.latent)` (`:172`) — the audio VAE + decode and the BWE vocoder, which is what the ordinary render already does. + +### 3.3 The guided step + +Per step, mirroring `FactoryGuidedDenoiser` over `MultiModalGuider.calculate` +(`guiders.py:244-273`): + +| Pass | Run when | Inputs | +|---|---|---| +| `cond` | always | positive audio context, no perturbation | +| `uncond_text` | `DoUnconditionalGeneration()` | NEGATIVE audio context, no perturbation | +| `uncond_perturbed` | `DoPerturbedGeneration()` | positive context, audio self-attn perturbed on `stg_blocks` | +| `uncond_modality` | `DoIsolatedModalityGeneration()` | — **refused by name**, see §7 | + +`ShouldSkipStep(step)` (`guiders.py:288-291`) short-circuits to the plain +conditional prediction, and `skip_step` defaults to 0 so it never fires on the +default path — which is why the branch is written to be reachable from the extra +rather than left implicit. + +The combination is `Ltx2MultiModalGuidance`, which already carries the two +details a re-derivation gets wrong: the `(scale - 1)` polarity on CFG and modality +against a bare `scale` on STG, and torch's UNBIASED `std` in the rescale. + +### 3.4 STG — the one genuinely new numeric + +`SKIP_AUDIO_SELF_ATTN` on a perturbed block replaces the attention output with the +raw value projection BEFORE `to_out`: `attention.py:558-577` computes `v = +self.to_v(context)` and, when `all_perturbed`, sets `out = v` without ever +projecting `q` or `k`; the blended form `out * mask + v * (1 - mask)` at `:571-572` +is the partial-batch case and reduces to the same thing at batch 1. + +Mirrored as an `all_perturbed` flag on `Ltx2AttentionArgs`, set per block from a +`Ltx2BatchedPerturbationConfig` handed to `Ltx2DitForward` — which is upstream's +own `perturbations` parameter on `LTXModel.forward` (`model.py:492`), so this is +mirroring a signature rather than inventing a seam. `nullptr` is upstream's +`perturbations=None` and every existing caller keeps its current behaviour +byte-for-byte. + +### 3.5 Where the request enters + +`pipeline_kind = t2a_one_stage` is a LOAD extra +(`kLtx2PipelineKindExtra`, already defined and already reachable from +`ltx2-gen --pipeline-kind` and from `vllm_video_model_params::extras`). This +matters for the reach claim: #928 records that `/v1/videos` forwards no +PER-GENERATION extra, and this row's selector is not one. The reach claim in §6b +is worded against what was checked rather than against the file the knob lives in. + +Two per-generation extras are added, both upstream CLI arguments: + +- `negative_prompt` (`--negative-prompt`, `utils/args.py:1083-1088`, defaulting + to `DEFAULT_NEGATIVE_PROMPT`, `utils/constants.py:186`). Absent means the + recipe's own default, which is `LightricksNegativePrompt()` — already in the + tree and already what `one_stage` on 2.4/2.5 resolves. +- `audio_cfg_guidance_scale` / `audio_stg_guidance_scale` / + `audio_rescale_scale` / `audio_skip_step` / `audio_stg_blocks` + (`utils/args.py:1089-1119`). Absent means the params table's own value. + +`modality_scale` gets NO extra, and that is not an omission: the CLI pins it to +1.0 at `t2a_one_stage.py:200-203` and there is no upstream surface that varies it +on this pipeline. + +### 3.6 The recipe + +`t2a_one_stage` rows for `2`, `2.3`, `2.4` and `2.5`, mirroring the `one_stage` +rows one for one, because the schedule is the same object: `LTX2Scheduler()` at +`t2a_one_stage.py:67` against `ti2vid_one_stage.py:81`, and the same +`detect_params` step count. What differs is what the recipe DECLARES about video: +`video_output_phase = -1` and a new `audio_only` flag, so a T2A recipe cannot be +run down the video path by accident. + +The 2.0 and 2.3 rows take `kOmniNegativePrompt` and the 2.4/2.5 rows take +`LightricksNegativePrompt()`, which is what `ResolveLtx2PipelineRecipe`'s +`one_stage` arm already does (`ltx2_pipeline.cpp:1239-1247`) — the negative prompt +travels with the generation, not with the pipeline. + +### 3.7 Load + +`video_vae_path` becomes optional **only** on a `t2a_one_stage` engine. +Upstream's `T2AOneStagePipeline.__init__` never calls `model_paths.video_vae()` +(`:53-107` constructs a `PromptEncoder`, a `DiffusionStage`, an `AudioDecoder` +and a `DurationPredictor`, and nothing else), so requiring one here would demand +a checkpoint the pipeline cannot use. `audio_vae_path` stays required, because +`AudioDecoder` is constructed unconditionally at `:94-100`. + +--- + +## 4. Risks + +**The `READER ANCHORS` gate.** `ltx2_video.cpp` carries a derived line-number +list re-derived and string-compared by `test_ltx2_video.cpp`. Any line inserted +above the last anchored line shifts it, and a clean `git merge` will not warn. +Mitigation: the engine edit is a branch at the TOP of `Generate`, which is below +every anchor; the list is re-derived at the final tree after the last merge of +`origin/main` with the test's own walk rather than by arithmetic, and it is named +as a merge hazard in the PR body. + +**A silently wrong render.** Every failure mode here produces a playable WAV of +the right length: a guider that never runs its uncond pass, an STG pass that +perturbs the wrong block, a `video = nullptr` path that silently reintroduces v2a +cross attention, an audio latent that decoded from zeros. None is visible in the +output. Mitigation: §5's trace fields observe the EFFECT rather than the +bookkeeping, plus a lower bound so a zeroed tensor cannot pass. + +**A recorded value is not a reached one.** A sibling row's suite recorded a +latent digest and inferred the latent reached the phase; a build that started from +zeros passed everything. So every assertion below is written against the state the +loop actually ran over, read after the operation it claims, and at least one is a +CONTROL that must move in the opposite direction. + +**Concurrent edits.** `ltx2_video.cpp`, `ltx2.h`, `ltx2_pipeline.{h,cpp}`, +`docs/FEATURES.md` and `.agents/issue-index.md` are being edited by sibling rows +(`LTX25-RETAKE`, `LTX25-DFR-PIPELINE` both landed within the hour before this +row started). Mitigation: new behaviour lives in new TUs; keyed records are +reapplied by key with unrelated keys proven byte-identical; the index is appended +to and never edited. + +--- + +## 5. Tests and evidence + +Focused gate: `test_ltx2_video`, `test_ltx2_pipeline`, `test_ltx2`, `test_capi`. + +1. **RED first, through the production entry point.** The smallest failing test + loads the reduced fixture with `pipeline_kind = t2a_one_stage` and calls + `Ltx2VideoEngine::Generate` (reached from `vllm_video_generate`). It fails + first because `ResolveLtx2PipelineRecipe` refuses the kind by name. + Reachability is then proven by deleting the production call site and showing + the test goes RED (`.agents/reachability.md`). +2. **The render has NO picture and DOES have sound.** `frame_count == 0`, an + empty `mux_argv`, an `audio.wav` on disk whose sample rate is the vocoder's + own, and NO `frame_000000.ppm` in the output directory. The last one is the + half a field check cannot make: a build that wrote frames and reported zero + passes every other assertion here. +3. **The video stream never reaches the DiT.** A trace field records whether the + forward was handed a video stream at all, and it must be false on every step + of a T2A render and true on an ordinary one. This is the §2.2 trap, and a + count- or shape-shaped check cannot see it: an `enabled=false` video stream + produces a waveform of exactly the right length. +4. **The guider actually ran, and each arm is separable.** Counters for the + cond / uncond-text / uncond-perturbed forwards. At the default params the + counts must be `steps`, `steps`, `steps`; with `audio_cfg_guidance_scale=1` + the second must be 0 and the render must still complete; with + `audio_stg_guidance_scale=0` the third must be 0. A single "guidance ran" + boolean would be satisfied by a build that ran the uncond pass and then + ignored it, so the guided output is ALSO required to differ from the + unguided one on the same seed. +5. **STG perturbs the block it was told to.** Asking for `audio_stg_blocks` that + name a block index out of range is upstream's own failure and is refused; and + perturbing block N produces a different velocity from perturbing block M. A + test asserting only that "a perturbed pass ran" cannot tell a config that + perturbs everything from one that perturbs nothing. +6. **A LOCAL fact is tied to the lifted refusal**, per this campaign's standing + rule. A test asserting only upstream symbol names cannot see staleness: a + sibling proved this by replacing a refusal's local claim with a self-declared + falsehood and watching the suite stay green at 44/44. So the test re-derives, + from THIS tree, that `Ltx2DitForward` still guards its remaining half and that + the AudioVideo weight contract is still what the loader enumerates. +7. **A lower bound** on the decoded waveform's magnitude, so a silently zeroed or + constant buffer fails. A correlation or count-based check cannot see a scale + error and is not used alone. + +Every mutation records three facts: `git diff --stat` after applying, whether it +**BUILT** with the compile-error count beside it, and the **exit code** captured +directly rather than through a pipe. A mutation that fails to compile establishes +nothing and is recorded as such. `-tc` filters are comma-free and the case count +is asserted non-zero, because doctest splits on a comma and runs UNRELATED cases +to a green SUCCESS. + +--- + +## 6. Gates + +``` +cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF +cmake --build build -j6 +ctest --test-dir build -j4 --output-on-failure +``` + +Reported with `CONFIGURE_EXIT`, `BUILD_EXIT`, the `: error:` count, `ctest -N`, +`CTEST_EXIT`, the full pass/fail line, positive controls for `No space left` and +`BFD assertion`, the box load and free disk. Known-red on `main` proven +pre-existing rather than asserted: `windows-msvc-*` (#584), `check-env-doc` +(#995). A load-dependent failure is re-run alone with the load recorded before it +is charged to this row. + +## 6b. Reachability — the sentence the records must carry + +**A production entry point reaches this, and the test enters through it.** + +``` +include/vllm.h vllm_video_generate + -> src/capi/vllm_c.cpp engine->Generate(gen) + -> vllm::multimodal::VideoEngine::Generate + -> Ltx2VideoEngine::Generate (the t2a branch at the top) + -> Ltx2T2aGenerate (ltx2_t2a.cpp) + -> Ltx2DitForward(..., /*video=*/nullptr, &ain, ...) + -> Ltx2MultiModalGuidance -> Ltx2EulerStep + -> Ltx2AudioDecoderForward -> Ltx2VocoderWithBweForward -> audio.wav +``` + +The command-line arm is the same call: `ltx2-gen --pipeline-kind t2a_one_stage` +sets the load extra and calls `vllm_video_generate`, as a thin ABI client that +includes no internal header. + +**What is NOT reachable, stated rather than left to be found.** `/v1/videos` +carries LOAD parameters, so `pipeline_kind` reaches it in principle; the +per-generation extras this row adds do NOT, because +`VideoGenParamsFromRequest` never forwards `VideoRequest::metadata` to +`VideoGenParams::extras` (#928). A T2A render over that route therefore takes the +recipe's own guider defaults and cannot override them. That is a pre-existing +defect of the route rather than of this row, and the reach claim above is worded +to exclude it. Whether the route's LOAD path was exercised end to end here is +recorded in the final report as measured or unverified — it is not asserted. + +**This row ends a test-only-driver.** `Ltx2MultiModalGuidance`, `Ltx2CfgDelta`, +`Ltx2StgDelta` and `Ltx2BatchedPerturbationConfig` have had test call sites and +no product one since they landed, which is exactly the shape +`.agents/reachability.md` names. + +## 7. Quantized arms + +| Arm | Disposition | +|---|---| +| bf16 / f32 safetensors | **ported** — the reduced fixture is this, and the gate runs on it | +| NVFP4, FP8 (the DiT tower) | **ported by inheritance, and that is a claim about the load rather than about a render.** The T2A path consumes `Impl::dit.weights`, which is whatever arm `Ltx2LoadDitFromSafetensors` materialized; this row adds no GEMM, changes no dtype and selects no arm. It is recorded as UNMEASURED on real quantized weights, because the GPU was out of bounds | +| GGUF k-quants | **not applicable**, and not merely undone. Upstream ships no GGUF arm for any LTX-2 component: `quantization_factory.py:23-26` enumerates the inference kinds exhaustively as fp8-cast, fp8-scaled-mm, nvfp4-cast and nvfp4-prequant, with `assert_never` at `:50`. There is no upstream behaviour to mirror and no quant-matched llama.cpp comparison to serve, because llama.cpp does not carry this architecture | +| int8-convrot | **out of scope**, unchanged by this row and already refused by name through `kInt8ConvRot` | + +Dtype polarity: the T2A path is f32 throughout on the host arm, which is what +`Ltx2DitForward` declares and refuses to widen, and the audio VAE and vocoder are +f32 by upstream's own choice (`vocoder.py:585-595`, mirrored in +`ltx2_audio_vae.cpp:1-12`). No buffer here is wider than the arm it feeds. + +--- + +## Owed + +- **The DEVICE arm.** `Ltx2DitForwardDevice` dereferences `*video` + unconditionally from its first `PrepareStreamDev` call onward + (`src/vllm/model_executor/models/ltx2_device.cpp`, the two `PrepareStreamDev` + calls and the per-block `a.batch = video->batch`), so a one-stream device + forward is a rewrite of that function rather than a lifted check. T2A on + `device != 0` is REFUSED BY NAME rather than served the host forward behind a + device handle, because that substitution is what would make every later timing + claim false. Tracked by #1005. +- **STG on the DEVICE forward**, for the same reason. +- **The sigma-BINNED guider factory** (`guiders.py:294-342`, + `MultiModalGuiderFactory.from_dict`). `Ltx2GuiderParamsForSigma` is ported and + gated; nothing here constructs bins, because no upstream surface on this + pipeline varies params by sigma — `t2a_one_stage.py:196-205` passes a plain + `MultiModalGuiderParams`. A caller may pass a factory (`:116`), and no CLI + does. +- **`uncond_modality` (isolated-modality guidance).** REFUSED BY NAME on this + pipeline, and the refusal is upstream's own reasoning rather than a local + limit: `modality_scale` is pinned to 1.0 for T2A because there is no video + modality to isolate (`t2a_one_stage.py:200-203`). A caller who reaches it + through another pipeline gets a message naming the missing fourth forward. +- **AUTO duration.** `resolve_num_frames`' predicting arm needs a constructed + `DurationPredictor`; `Ltx2DurationHeadForward` is ported and gated (including + the audio-only case, `kLtx2DurAudioOnlyGolden`) and nothing constructs one. + Inherited from the engine, not introduced here. +- **A real-checkpoint T2A render.** Gated on the reduced fixture only; the GPU + was out of bounds for this row. +- **Value goldens from executed upstream for the T2A COMPOSITION.** The bricks + either side have them; the chain does not. What would close it is a section in + `scripts/gen-ltx2-pipeline-goldens.py` that instantiates + `LTXAudioOnlyModelConfigurator` at reduced dimensions and runs one guided step. +- **`max_batch_size`** (`t2a_one_stage.py:120`, `:169`), the prompt enhancer + (`:118-119`), LoRA/quantization/compilation/offload constructor arguments + (`:56-63`) — the engine's own surfaces, unchanged by this row. + +## 8. Stop conditions + +Stop and report rather than widening scope if: lifting the `video != nullptr` +half of the DiT refusal moves any existing LTX-2.5 test; the STG perturbation +cannot be expressed without changing an existing forward's numerics; or the +`READER ANCHORS` gate cannot be re-derived deterministically. + +## 9. Now + +`ACTIVE` — spec committed before implementation, per `AGENTS.md` § *Spec before +code*. From 23c31ad6b815391b50edd02ecdb357e7d4daced2 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 16 Aug 2026 11:58:25 +0000 Subject: [PATCH 2/9] feat(LTX25-T2A-ONE-STAGE): text-to-audio, the one-stream forward, and the guided denoiser nothing had (#1005, #1013) `T2AOneStagePipeline` (`t2a_one_stage.py:43`, `__call__` at `:109` @ `fd4ded7f`) renders a soundtrack and no picture. This is the first path here that returns a `VideoResult` with zero frames, and the first that runs the DiT with `video = nullptr`. The audio-only shape FITS `Ltx2VideoEngine` rather than needing its own entry point, and the reason is upstream's own: T2A expresses its duration through a placeholder `VideoPixelShape` at 512x512 whose height and width it documents as unused (`t2a_one_stage.py:37-40`), then calls the SAME `DiffusionStage.__call__` every video pipeline calls. `VideoResult` already carries `frame_count` and `audio_path` as independent fields, so an audio-only result is `frame_count = 0`, an empty `frame_dir` and an EMPTY `mux_argv` -- composing an ffmpeg argv over a frame pattern matching no file would hand the caller a command that cannot run. The numerics live in a new TU mirroring upstream's own file, reached from `Generate` before any video geometry is resolved. THREE THINGS FAIL SILENTLY IF GUESSED, and two of them were refusals whose stated reasons did not describe this case. FIRST, `Ltx2DitForward` demanded BOTH streams and blamed the AudioOnly weight contract. That is a claim about the CHECKPOINT and T2A never loads one: upstream reads the ordinary AudioVideo FILE through `LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP` (`model_configurator.py:228-239`) and builds an AudioOnly MODULE from the subset. Every line below that guard was already written against `video != nullptr`, so lifting it reaches a path the file already had. What remains true moves to where it is true, at the loader. SECOND, the same message advised `enabled = false` as the substitute, and for this case it is wrong. Upstream's predicate is `run_v2a = run_ax and (video is not None and vx.numel() > 0)` (`transformer.py:269`): it tests PRESENCE. A disabled-but-present video stream still feeds video-to-audio cross attention from a latent T2A never meant to exist, and still returns a playable waveform of the right length. `Ltx2ConditioningTrace::t2a_video_stream_present` observes it at the forward rather than restating the argument, and a mutation passing a disabled stream turns the gate red. THIRD, the engine had no guided denoiser at all -- one forward per step, no guider parameter read anywhere. That is correct for `distilled_two_stage`, which builds a `SimpleDenoiser` upstream too, and wrong for T2A, whose defaults are `cfg_scale = 7.0` and `stg_scale = 1.0`, i.e. THREE forwards per step. So this adds the unconditional pass, the STG pass, and `all_perturbed` on `Ltx2Attention` -- upstream's `use_attention = not all_perturbed` (`attention.py:557`), which replaces the attention output with the raw value projection before `to_out`. `Ltx2MultiModalGuidance`, `Ltx2CfgDelta`, `Ltx2StgDelta` and `Ltx2PerturbationConfig` were all ported, gated, and reached by nothing but their own tests until now. AND ONE BUG, FILED AND FIXED IN THE SAME FLOW (#1013). `OneStagePhase` left `noise_scale` at the struct default of 0.0, and 0.0 is not "no extra noise": `Ltx2GaussianNoise` is `latent + noise_scale * (noise - latent)`, so the state stayed exactly as `create_initial_state` wrote it, which with no initial latent is ALL ZEROS. A `one_stage` render denoised a zero tensor. Upstream's `ModalitySpec.noise_scale` defaults to 1.0 (`utils/types.py:110`) and `TI2VidOneStagePipeline` constructs both specs without it (`ti2vid_one_stage.py:233-239`). No gate saw it because every end-to-end test loads `distilled_two_stage`. Fixed here because the `t2a_one_stage` rows are built FROM `OneStageRecipe` and would have inherited it. `dmd2` leaves the same field at 0.0 and is NOT corrected by analogy: its source is vLLM-Omni's recipe, which is not checked out here. `tests/vllm/models/test_ltx2.cpp`'s single-stream case is REPLACED rather than widened. It pinned the old refusal's message; the new form pins upstream's actual contract (`transformer.py:259-260`) and is strictly stronger, because it also asserts what a one-stream call RETURNS and that it is not the joint forward with the other stream ignored. The old assertion could not tell a served one-stream forward from a broken one, since both threw. Eight mutations, each recording three facts. Seven DETECTED, including the reachability mutation: deleting the production call site turns the focused gate red. The eighth -- scaling the initial latent by `sigmas[0]` -- SURVIVED, and the resolution is in the spec: it is an identity, because `LTX2Scheduler` pins the first sigma to exactly 1.0 for every step count. That identity is now gated rather than assumed, and the case that gates it found a second thing: `steps = 1` returns `-nan` on both sides, from upstream's own `1 - 0/0`. The device arm is REFUSED BY NAME rather than served the host forward behind a device handle. `Ltx2DitForwardDevice` takes both streams by reference throughout, so a one-stream device forward is a rewrite of that function rather than the lifted check the host forward needed. Owed under `## Owed`, tracked by #1005. The `READER ANCHORS` list in `ltx2_video.cpp` is re-derived at this tree with the test's own walk, because the load path grew lines above it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 2 +- .agents/specs/ltx25-t2a-one-stage.md | 108 ++++- CMakeLists.txt | 6 + docs/FEATURES.md | 1 + docs/USAGE.md | 52 ++- examples/ltx2_gen/main.cpp | 50 ++- include/vllm/model_executor/models/ltx2.h | 112 ++++- .../model_executor/models/ltx2_pipeline.h | 26 ++ include/vllm/model_executor/models/ltx2_t2a.h | 168 +++++++ include/vllm/multimodal/ltx2_video.h | 85 ++++ src/vllm/model_executor/models/ltx2.cpp | 76 +++- src/vllm/model_executor/models/ltx2_dit.cpp | 52 ++- .../model_executor/models/ltx2_pipeline.cpp | 63 +++ src/vllm/model_executor/models/ltx2_t2a.cpp | 347 +++++++++++++++ src/vllm/multimodal/ltx2_video.cpp | 321 +++++++++++++- tests/vllm/models/test_ltx2.cpp | 60 ++- tests/vllm/multimodal/test_ltx2_video.cpp | 418 ++++++++++++++++++ 17 files changed, 1880 insertions(+), 67 deletions(-) create mode 100644 include/vllm/model_executor/models/ltx2_t2a.h create mode 100644 src/vllm/model_executor/models/ltx2_t2a.cpp diff --git a/.agents/issue-index.md b/.agents/issue-index.md index fbe67d42c..de70aed78 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -269,4 +269,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#986](https://github.com/mudler/vllm.cpp/issues/986) | `LTX25-DFR-PIPELINE` | `DFRPipeline` (`dfr_pipeline.py` + `dfr_layout.py` @ `fd4ded7f`) has no representation in this tree — `git grep -i dfr` over `src include tests examples docs` returns ZERO product hits, against 87 `ltx2` hits in `ltx2.cpp` alone as the positive control ([#604](https://github.com/mudler/vllm.cpp/issues/604)). It matters beyond its own feature because it is the **ONLY** upstream consumer of the temporal x2 latent upsampler this project already ships: [`ltx25-temporal-upsampler.md`](specs/ltx25-temporal-upsampler.md) section 7 records the operator as ported, loader-parsed and gated but *"not reachable from any shipped pipeline"*, and `docs/FEATURES.md` carries that as `Temporal x2 ups gated, UNDRIVEN`. No issue tracked that state — `undriven` returned zero hits across open and closed issues. DFR also needs the generated-keyframe-slot READBACK that [#920](https://github.com/mudler/vllm.cpp/issues/920) refused by name and left owed after it CLOSED, so the debt had a spec bullet and no open issue; it needs the LAYOUT and the EXTRACTION but **not** the standalone single-frame decode, because DFR never decodes its slots — it hands them to the spatial upsampler (`dfr_pipeline.py:348`) and feeds them back as `initial_keyframes` (`:364`). Spec [`ltx25-dfr-pipeline.md`](specs/ltx25-dfr-pipeline.md). Campaign [#644](https://github.com/mudler/vllm.cpp/issues/644) | feature | | [#995](https://github.com/mudler/vllm.cpp/issues/995) | `ENG-EXPERT-STREAM` | `check-env-doc` and `test_check_env_doc` RED in `scripts/agent-preflight.sh` on **every** branch based on `3005447f8`: `VT_MOE_EXPERT_STREAM`, `VT_MOE_EXPERT_STREAM_SLOTS` and `VT_MOE_EXPERT_STREAM_SLOT_BYTES` are read from `src/vllm/model_executor/models/qwen3_5.cpp` (`:5145`, `:5164` @ `3005447f8`) and appear in neither `docs/ENVIRONMENT.md` (`grep -c` returns **0**) nor `scripts/env-doc-allowlist.txt`. Arrived with [#993](https://github.com/mudler/vllm.cpp/pull/993). Found while gating [#986](https://github.com/mudler/vllm.cpp/issues/986) and proved pre-existing with a matched-arm check rather than asserted: the three sites are in a file that branch does not touch, and `git diff origin/main...HEAD | grep '^+.*VT_MOE_EXPERT'` returns nothing. NOT fixed in flow: documenting a knob means stating its default and when to touch it, and the expert-streamer's slot accounting belongs to the row that added it, so a plausible-sounding entry written by a passer-by is how `docs/ENVIRONMENT.md` stops being trustworthy. It is a PRE-FLIGHT gate, so it fails before every edit and presents to each author as a red their own diff caused, which is the shape [#965](https://github.com/mudler/vllm.cpp/issues/965) and [#968](https://github.com/mudler/vllm.cpp/issues/968) both took | bug | | [#987](https://github.com/mudler/vllm.cpp/issues/987) | `LTX25-RETAKE` | Two `ltx-2.5` refusal messages state reasons that are no longer true. (a) `src/vllm/multimodal/ltx2_video.cpp:1608 @ 0e1bee42f` says "nothing reads `ref_video_dir` at all", and MiniMax-H3 has always consumed the directory in full (`ReadReferenceClipChw`, `src/vllm/multimodal/minimax_h3_video.cpp:135 @ 0e1bee42f`, called at `:650`); [#975](https://github.com/mudler/vllm.cpp/issues/975) inherited the wider claim from this message. The claim that holds is narrower: the LTX-2.5 engine never reads the directory's CONTENTS. (b) `ltx2_video.cpp:1636-1638 @ 0e1bee42f` says "there is no AUDIO_VAE_ENCODER key filter", and `c2019b0e3` landed `Ltx2AudioVaeEncoderKeyRules()` (`include/vllm/model_executor/models/ltx2_audio_input.h:73 @ 0e1bee42f`) with a live call through `Ltx2EncodeAudioToLatent`. Both rewritten in the `WHAT IS *NOT* THE REASON` shape in the same flow, with one assertion tied to the LOCAL fact that the LTX side now reads the directory | bug | -| [#1005](https://github.com/mudler/vllm.cpp/issues/1005) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 text-to-audio (`T2AOneStagePipeline`, `t2a_one_stage.py:43`, `__call__` at `:109` @ `fd4ded7f`) is absent, and the three blockers are not the ones a reader would guess. (a) `Ltx2DitForward` refuses a one-stream call at `src/vllm/model_executor/models/ltx2_dit.cpp:765 @ 332aed738`, citing a weight contract that describes a checkpoint T2A never loads — upstream reads the ordinary AudioVideo FILE through `LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP` (`model_configurator.py:228-239`). (b) The same message advises `enabled=false` as the substitute, and it is NOT: `run_v2a = run_ax and (video is not None and vx.numel() > 0)` (`transformer.py:269`) tests PRESENCE, not `enabled`, so a disabled-but-present video stream still feeds v2a cross attention and still returns a finished waveform; our port mirrors that polarity at `ltx2_dit.cpp:251`. (c) The engine has NO guided denoiser at all — `git grep -n 'guid\\|cfg_scale' src/vllm/multimodal/ltx2_video.cpp` is 0 against 66 for `ltx2` in the same file as the control — while T2A defaults to `cfg_scale=7.0` and `stg_scale=1.0`, i.e. THREE forwards per step. Spec [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | feature | +| [#1005](https://github.com/mudler/vllm.cpp/issues/1005) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 text-to-audio (`T2AOneStagePipeline`, `t2a_one_stage.py:43`, `__call__` at `:109` @ `fd4ded7f`) is absent, and the three blockers are not the ones a reader would guess. (a) `Ltx2DitForward` refuses a one-stream call at `src/vllm/model_executor/models/ltx2_dit.cpp:765 @ 332aed738`, citing a weight contract that describes a checkpoint T2A never loads — upstream reads the ordinary AudioVideo FILE through `LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP` (`model_configurator.py:228-239`). (b) The same message advises `enabled=false` as the substitute, and it is NOT: `run_v2a = run_ax and (video is not None and vx.numel() > 0)` (`transformer.py:269`) tests PRESENCE, not `enabled`, so a disabled-but-present video stream still feeds v2a cross attention and still returns a finished waveform; our port mirrors that polarity at `src/vllm/model_executor/models/ltx2_dit.cpp:251 @ 332aed738`. (c) The engine has NO guided denoiser at all — `git grep -n 'guid\\|cfg_scale' src/vllm/multimodal/ltx2_video.cpp` is 0 against 66 for `ltx2` in the same file as the control — while T2A defaults to `cfg_scale=7.0` and `stg_scale=1.0`, i.e. THREE forwards per step. Spec [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | feature | diff --git a/.agents/specs/ltx25-t2a-one-stage.md b/.agents/specs/ltx25-t2a-one-stage.md index cea503b93..a4d1af061 100644 --- a/.agents/specs/ltx25-t2a-one-stage.md +++ b/.agents/specs/ltx25-t2a-one-stage.md @@ -81,7 +81,7 @@ Three blockers, each named by the symbol that has to change. None of them is ### 2.1 `Ltx2DitForward` refuses a one-stream call, and its stated reason does not describe T2A -`src/vllm/model_executor/models/ltx2_dit.cpp:765` is +`src/vllm/model_executor/models/ltx2_dit.cpp:765 @ 332aed738` is ``` VT_CHECK(video != nullptr && audio != nullptr, "... LTXModelType.VideoOnly and @@ -110,15 +110,15 @@ already handles the case. **The forward already handles it.** Every line of `Ltx2DitForward` below the check is written against `video != nullptr` / `have_both` -(`ltx2_dit.cpp:786-869`), and `Ltx2TransformerBlockForward` computes +(`ltx2_dit.cpp:786-869 @ 332aed738`), and `Ltx2TransformerBlockForward` computes `run_vx` / `run_a2v` / `run_v2a` from `video_x != nullptr && tv > 0` -(`ltx2_dit.cpp:248-251`). Lifting the check reaches a path that was already +(`ltx2_dit.cpp:248-251 @ 332aed738`). Lifting the check reaches a path that was already written; it does not add one. ### 2.2 `enabled = false` is NOT the same shape, and the difference renders The refusal's own message says to "use `enabled` to run one stream of an AV -model", and `include/vllm/model_executor/models/ltx2.h:506-508` repeats it. That +model", and `include/vllm/model_executor/models/ltx2.h:506-508 @ 332aed738` repeats it. That advice is correct for the A2V and joint pipelines. **It is wrong for T2A**, and this is the detail that fails silently if guessed. @@ -132,7 +132,7 @@ It tests `video is not None`, **not** `video.enabled`. So an AV forward handed a present-but-disabled video stream still runs video->audio cross attention, taking its context from a video latent that T2A never intended to exist — and returns a finished waveform. Our port mirrors that polarity exactly at -`ltx2_dit.cpp:251`, so the same trap is live here. +`ltx2_dit.cpp:251 @ 332aed738`, so the same trap is live here. `video = nullptr` is the only shape that reproduces `LTXModel.forward(video=None, ...)` (`model.py:492-538`, the `video_args = ... if video is not None else None` @@ -160,7 +160,7 @@ T2A builds a `FactoryGuidedDenoiser` (`t2a_one_stage.py:154-161`) over `audio_guider_params` — `cfg_scale=7.0`, `stg_scale=1.0`, `rescale_scale=0.7`, `stg_blocks=[28]` on the 2.3/2.4/2.5 lineage (`utils/constants.py:58-66`, `:82-87`, `:118`) — with `modality_scale` pinned to **1.0** by the CLI itself -(`t2a_one_stage.py:200-203`: "Audio-only generation has no video modality, so the +(`t2a_one_stage.py:200-202`: "Audio-only generation has no video modality, so the video->audio (v2a) cross-modal guidance is meaningless here. 1.0 disables it"). Read against `guiders.py:275-287`, those defaults mean @@ -172,13 +172,13 @@ would be a different trajectory from upstream's default, and no shape, token count or file length could see it. The bricks exist and are gated; the WIRING does not. `Ltx2MultiModalGuidance` -(`ltx2_pipeline.h:323`), `Ltx2CfgDelta`, `Ltx2StgDelta`, +(`ltx2_pipeline.h:323 @ 332aed738`), `Ltx2CfgDelta`, `Ltx2StgDelta`, `Ltx2GuiderParamsForSigma`, `Ltx2PerturbationConfig` and `Ltx2BatchedPerturbationConfig` are all ported with goldens and have **no product caller** — the "test-only driver" shape `.agents/reachability.md` enumerates. What is genuinely absent is one thing: the DiT forward runs upstream's `perturbations=None` path only, and says so at -`include/vllm/model_executor/models/ltx2.h:42-46`. +`include/vllm/model_executor/models/ltx2.h:42-46 @ 332aed738`. --- @@ -248,7 +248,7 @@ Per step, mirroring `FactoryGuidedDenoiser` over `MultiModalGuider.calculate` | `uncond_perturbed` | `DoPerturbedGeneration()` | positive context, audio self-attn perturbed on `stg_blocks` | | `uncond_modality` | `DoIsolatedModalityGeneration()` | — **refused by name**, see §7 | -`ShouldSkipStep(step)` (`guiders.py:288-291`) short-circuits to the plain +`ShouldSkipStep(step)` (`guiders.py:287-291`) short-circuits to the plain conditional prediction, and `skip_step` defaults to 0 so it never fires on the default path — which is why the branch is written to be reachable from the extra rather than left implicit. @@ -292,7 +292,7 @@ Two per-generation extras are added, both upstream CLI arguments: (`utils/args.py:1089-1119`). Absent means the params table's own value. `modality_scale` gets NO extra, and that is not an omission: the CLI pins it to -1.0 at `t2a_one_stage.py:200-203` and there is no upstream surface that varies it +1.0 at `t2a_one_stage.py:200-202` and there is no upstream surface that varies it on this pipeline. ### 3.6 The recipe @@ -306,7 +306,7 @@ run down the video path by accident. The 2.0 and 2.3 rows take `kOmniNegativePrompt` and the 2.4/2.5 rows take `LightricksNegativePrompt()`, which is what `ResolveLtx2PipelineRecipe`'s -`one_stage` arm already does (`ltx2_pipeline.cpp:1239-1247`) — the negative prompt +`one_stage` arm already does (`ltx2_pipeline.cpp:1239-1247 @ 332aed738`) — the negative prompt travels with the generation, not with the pipeline. ### 3.7 Load @@ -402,6 +402,56 @@ nothing and is recorded as such. `-tc` filters are comma-free and the case count is asserted non-zero, because doctest splits on a comma and runs UNRELATED cases to a green SUCCESS. +### What the mutation pass actually found + +Focused gate `./build/tests/test_ltx2_video "--test-case=*t2a*"`, unmutated at +7 cases / 496 assertions / exit 0. Each mutation applied to ONE file, rebuilt, +run, restored in a `finally` and the restore verified by **sha256** rather than +assumed. The harness rebuilds the restored tree before anything else measures it. + +| Mutation | `git diff --stat` | BUILT | exit | verdict | +|---|---|---|---|---| +| M1 delete the production call site (the reachability mutation) | `ltx2_video.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED, 3 of 6 cases failed | +| M2 hand the forward a present-but-DISABLED video stream instead of `nullptr` | `ltx2_t2a.cpp \| 3 ++-` | YES (0 errors) | 1 | DETECTED | +| M3 never run the unconditional forward | `ltx2_t2a.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED | +| M4 ignore `stg_blocks` and perturb EVERY block | `ltx2_t2a.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED | +| M5 `all_perturbed` falls through to ordinary attention | `ltx2.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED | +| M6 revert the `one_stage` `noise_scale` to the struct default (#1013) | `ltx2_pipeline.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED | +| M7 scale the initial latent by `sigmas[0]` | `ltx2_t2a.cpp \| 1 +` | YES (0 errors) | 0 | **SURVIVED** — see below | +| M8 write a frame on the audio-only path | `ltx2_video.cpp \| 1 +` | YES (0 errors) | 1 | DETECTED | + +**M7 SURVIVED, and the resolution is the useful part.** It is the mutation a +reader coming from another flow-matching sampler expects to be REQUIRED — scaling +the initial noise by the first sigma — and it changed nothing. The reason is not +a blind gate: it is an identity. `LTX2Scheduler` starts at `linspace(1, 0, steps ++ 1)[0] == 1`; the shift map sends 1 to `exp(s)/(exp(s) + (1/1 - 1))`, exactly 1 +(`schedulers.py:41-45`); the stretch sends it to `1 - (1 - 1)/scale_factor`, +again exactly 1 (`:47-55`). So `sigmas[0]` is 1.0 for every step count. + +Rather than record "a mutation survived", the identity is now GATED — the case +"the schedule starts at exactly 1.0" pins `sigmas.front() == 1.0F` across four +step counts. If upstream ever moves the first sigma off 1, that fires and the two +forms stop agreeing. + +**And that case found a second thing.** `steps = 1` returns `-nan`, on both +sides: the non-zero sigma list is `[1.0]`, so `one_minus_z` is `[0.0]`, +`scale_factor = 0 / (1 - terminal)` is 0, and the stretch computes `1 - 0/0` +(`schedulers.py:49-54`). It is upstream's own arithmetic, not a defect here, and +it is excluded from the pin with the reason written beside it rather than +silently skipped. A one-step schedule is recorded under `## Owed`. + +**Two harness notes, because both would otherwise read as verdicts about the +code.** A `.pyc` for `scripts/agent-start.py` was truncated to exactly 4096 bytes +on this shared box, and `scripts/agent-preflight.sh` reported +`FAIL test_agent_start` with an `EOFError: marshal data too short` — a corrupt +byte-cache presenting as a failing gate. Removing the file made it pass 20/20. +And M4's first form asserted the STG perturbation on a latent filled with a +constant: self-attention over identical rows returns a weighted average of +identical values, which IS the value projection, so the perturbation was a +numeric no-op and the case reported "the perturbation changed nothing" about a +correct build. The fixture latent now varies per element, and the reason is in +the test. + --- ## 6. Gates @@ -469,6 +519,33 @@ f32 by upstream's own choice (`vocoder.py:585-595`, mirrored in --- +## 7b. The bug this row found and fixed in flow + +[#1013](https://github.com/mudler/vllm.cpp/issues/1013). `OneStagePhase` +(`ltx2_pipeline.cpp`) left `Ltx2PhaseRecipe::noise_scale` at the struct default +of **0.0**, and 0.0 is not "no extra noise": `Ltx2GaussianNoise` is +`latent + noise_scale * (noise - latent)`, so at 0.0 the state stays exactly as +`create_initial_state` wrote it, which with no initial latent is **all zeros**. +A `one_stage` render therefore denoised a zero tensor on both streams. + +Upstream's `ModalitySpec.noise_scale` defaults to 1.0 +(`ltx-pipelines/utils/types.py:110`) and `TI2VidOneStagePipeline.__call__` +constructs both specs without it (`ti2vid_one_stage.py:233-239`). The two +neighbouring recipes set it explicitly, which is what made the omission legible. + +Fixed here rather than deferred, per AGENTS.md § *Every change starts from an +issue*, because the `t2a_one_stage` rows are built FROM `OneStageRecipe` and +would have inherited it. No gate saw it because every end-to-end test loads +`distilled_two_stage`, and a zero-initialized denoise still returns a finite clip +of the right size, frame count and sample rate. M6 above is the mutation that now +holds it. + +**NOT claimed for `dmd2`.** `PositiveOnlyRecipe` leaves the same field at 0.0 and +its source is vLLM-Omni's `LTX_POSITIVE_ONLY_RECIPE`, which is not checked out +here. Left as-is and named rather than corrected by analogy: a recipe whose +upstream nobody read is exactly where a plausible fix lands wrong. Recorded in +#1013. + ## Owed - **The DEVICE arm.** `Ltx2DitForwardDevice` dereferences `*video` @@ -489,12 +566,19 @@ f32 by upstream's own choice (`vocoder.py:585-595`, mirrored in - **`uncond_modality` (isolated-modality guidance).** REFUSED BY NAME on this pipeline, and the refusal is upstream's own reasoning rather than a local limit: `modality_scale` is pinned to 1.0 for T2A because there is no video - modality to isolate (`t2a_one_stage.py:200-203`). A caller who reaches it + modality to isolate (`t2a_one_stage.py:200-202`). A caller who reaches it through another pipeline gets a message naming the missing fourth forward. - **AUTO duration.** `resolve_num_frames`' predicting arm needs a constructed `DurationPredictor`; `Ltx2DurationHeadForward` is ported and gated (including the audio-only case, `kLtx2DurAudioOnlyGolden`) and nothing constructs one. Inherited from the engine, not introduced here. +- **A one-step schedule.** `Ltx2SigmaSchedule(1, ...)` returns `-nan` as its + first sigma, mirroring upstream's own division by zero + (`schedulers.py:49-54`). Neither side is checked against it; a T2A request with + `steps = 1` therefore produces NaN. Named rather than defended against, + because the correct behaviour is upstream's to decide and this port does not + get to invent one. +- **The `dmd2` recipe's `noise_scale`**, see §7b. - **A real-checkpoint T2A render.** Gated on the reduced fixture only; the GPU was out of bounds for this row. - **Value goldens from executed upstream for the T2A COMPOSITION.** The bricks diff --git a/CMakeLists.txt b/CMakeLists.txt index 1457c0f79..0d058a2b0 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -873,6 +873,12 @@ add_library(vllm STATIC # truncate-or-pad conform, and the frame-directory source. Its own TU for the # same reason the two above are: ltx2_video.cpp is a concurrent-edit lock. src/vllm/model_executor/models/ltx2_retake.cpp + # LTX-2.5 (ROW LTX25-T2A-ONE-STAGE, issue #1005): text-to-audio — the + # audio-only pipeline and its guided denoiser. Its own TU for the same reason + # the two above are, plus one this row adds: it is the only LTX pipeline that + # runs the DiT with `video = nullptr`, and keeping that call in one file is + # what makes "does anything else pass a video stream" a readable question. + src/vllm/model_executor/models/ltx2_t2a.cpp # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L6): the quantized loaders — the # FP8 and torchao-NVFP4 DiT arms, the torchao-NVFP4 text encoder, and the # load-time device staging GB10's ATS penalty makes the default. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index d5784a9dc..6e60752fd 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -169,6 +169,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 DFR base + generated keyframe slots | LTX-2.5 (21.00B video+audio) | gated vs EXECUTED upstream `dfr_layout` + 3 `dfr_pipeline` helpers @ `fd4ded7f` (`test_ltx2_dfr` 11/11, 652 assertions); canvas, tiles, stitch, carry-forward as EXACT index vectors, since each defect is plausible| `--pipeline-kind dfr`. Canvas PADS 9 to 25 then trims back; slots on the x8 grid, MARKED, read back BEFORE the trim. `num_generated_keyframes` SERVED elsewhere. Temporal ROUNDS refused (#986); detail LoRA refused (#975)| | LTX-2.5 tiled + streaming Conv VAE decode | LTX-2.5 video VAE | gated vs executed upstream `ltx_core` @ `fd4ded7f` (`test_ltx2_tiling` 10/10, 915 assertions); one-tile and untiled-spatial controls BIT-EXACT vs untiled on both causality arms; an untiled frames axis is REFUSED | Streams temporal chunks through upstream's AUTO layout (768/64 px, 80/24 frames); above one tile the pixel volume is never materialized. NO-OP below 768px and 81 frames; 81-120 IS tiled, differing 6.70% of range | | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated | +| LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 8 mutations, 7 DETECTED and the 8th proven an identity rather than a blind spot (no count is restated here, because `ctest` is the authority) | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only GUIDED arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending | diff --git a/docs/USAGE.md b/docs/USAGE.md index 85a2273cf..26ce60278 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -842,6 +842,55 @@ The two knobs beside it are per-GENERATION and therefore CLI and ABI only, becau `num_generated_keyframes` on the other pipelines, and `temporal_upsample_rounds` below. +### LTX-2.5 text-to-audio: a render with no picture + +`--pipeline-kind t2a_one_stage` runs upstream's `T2AOneStagePipeline`, which +generates a soundtrack and no video at all. The result carries an `audio.wav`, +`frame_count = 0`, an empty frame directory and **no ffmpeg argv**, because there +is nothing to mux. + +```sh +ltx2-gen --dit ltx-2.5-dit.safetensors \ + --audio-vae ltx-2.5-audio-vae.safetensors \ + --encoder gemma4-12b-with-proj.safetensors --encoder-config gemma4.json \ + --pipeline-kind t2a_one_stage --device cpu \ + --frames 121 --steps 30 --prompt "rain on a tin roof, distant thunder" \ + --workdir /tmp/t2a +``` + +**No `--video-vae` is needed**, and none is loaded: upstream's pipeline never +constructs a video VAE. `--width` and `--height` are **refused** rather than +ignored — upstream passes a 512x512 placeholder whose height and width it +documents as unused, and only the frame count and the recipe's frame rate are +read, to derive the duration. + +**It is the only GUIDED arm, and that changes what it costs and what it needs.** +The distilled video recipes run one DiT forward per step. This one runs +**three** by default — conditional, unconditional, and one with the audio +self-attention perturbed (STG) — so it is roughly 3x the work per step, and it +**requires a text tower**, because the unconditional pass conditions on the +negative prompt. Loading with `prompt_embeds_path` alone gets a refusal naming +`--audio-cfg-guidance-scale 1.0` as the way to turn the unconditional pass off. + +Six per-generation knobs mirror upstream's own CLI, and each takes the +checkpoint generation's value when absent: `--negative-prompt`, +`--audio-cfg-guidance-scale` (7.0), `--audio-stg-guidance-scale` (1.0), +`--audio-rescale-scale` (0.7), `--audio-skip-step` (0) and `--audio-stg-blocks` +(28 on the 2.3-and-later lineage), which is comma separated. A block index +outside the DiT's own layer count is refused rather than clamped. There is no +`modality_scale` knob: upstream pins it to 1.0 for this pipeline, because +audio-only generation has no video modality to isolate. + +Being per-generation, those six reach the CLI and the C ABI and **not** +`/v1/videos`, which forwards no per-generation extra to any engine (issue #928). +`pipeline_kind` is a LOAD knob and does reach the server, so a server started +with `--video-extra pipeline_kind=t2a_one_stage` renders every request as audio +at the recipe's own guider values. + +**The accelerator is refused by name.** `device = 1` gets a refusal on this +pipeline: the device forward takes both streams by reference and this pipeline +has no video stream to give it. Use `--device cpu`. + **What is not served.** `temporal_upsample_rounds` is defined and refused above `0`: the rounds loop that temporally doubles the latent, re-tiles the canvas and stitches it back is not ported. The refusal names it, and it names three things @@ -2286,7 +2335,8 @@ or without the ComfyUI `model.diffusion_model.` prefix. Each family reads its ow knobs from `extras`. H3 takes `partition`. LTX-2.5 takes `audio_prompt_embeds_path` (the audio stream's conditioning, the twin of the seam's `prompt_embeds_path`, which carries the video stream), `pipeline_kind` -(default `distilled_two_stage`), `model_version` (only for a checkpoint that +(default `distilled_two_stage`; also `one_stage`, `dmd2`, `dfr`, `retake` and +`t2a_one_stage`), `model_version` (only for a checkpoint that declares none), `dit_config_path`, `encoder_config_path`, `allow_unported_modules`, `max_phase`, `prompt_embeds_valid_rows`, `upsampler_path` and `duration_head_path`. An extra a family does not define is diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index 2e16774f0..7f4106f49 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -169,7 +169,21 @@ const char* Need(int argc, char** argv, int i, const char* flag) { "--width, --height and --frames are refused alongside it. --regenerate-video 0\n" "freezes the clip instead; --regenerate-audio has no effect while the source is\n" "a frame folder, because a folder carries no audio and both of upstream\'s audio\n" - "predicates test for one.\n"); + "predicates test for one.\n\n" + "TEXT-TO-AUDIO renders a soundtrack and NO PICTURE. --pipeline-kind\n" + "t2a_one_stage selects it; the result carries an audio.wav, zero frames and no\n" + "ffmpeg argv, because there is nothing to mux. --video-vae is not needed and\n" + "--width/--height are refused: upstream passes a 512x512 placeholder whose\n" + "height and width it documents as unused, and only --frames and the recipe\'s\n" + "frame rate are read, to derive the DURATION. Unlike the distilled video\n" + "recipes this one is GUIDED: it runs three DiT forwards per step by default\n" + "(conditional, unconditional, and one with the audio self-attention perturbed),\n" + "so it needs a text tower for the negative prompt. --negative-prompt,\n" + "--audio-cfg-guidance-scale, --audio-stg-guidance-scale, --audio-rescale-scale,\n" + "--audio-skip-step and --audio-stg-blocks are upstream\'s own flags; absent, each\n" + "takes the checkpoint generation\'s own value. --audio-stg-blocks is comma\n" + "separated and a block index outside the DiT\'s layer count is refused rather\n" + "than clamped. The accelerator is REFUSED by name on this pipeline.\n"); std::exit(code); } @@ -186,6 +200,10 @@ int main(int argc, char** argv) { // RETAKE (row LTX25-RETAKE, #924): a source clip DIRECTORY and the window to // regenerate. `--ref-video` is a directory of frame_%06d.ppm, not a container. std::string ref_video, retake_start, retake_end, retake_fps, regen_video, regen_audio; + // TEXT-TO-AUDIO (row LTX25-T2A-ONE-STAGE, #1005): one flag per argument of + // upstream's `default_1_stage_t2a_arg_parser` (utils/args.py:1070-1120). + std::string negative_prompt, audio_cfg_scale, audio_stg_scale, audio_rescale; + std::string audio_skip_step, audio_stg_blocks; // The extras are BORROWED by the load call, so the strings must outlive it. // Kept as two parallel vectors of owned strings plus the char* views the ABI @@ -262,6 +280,21 @@ int main(int argc, char** argv) { retake_end = Need(argc, argv, ++i, "--retake-end-time"); else if (f == "--retake-frame-rate") retake_fps = Need(argc, argv, ++i, "--retake-frame-rate"); + // TEXT-TO-AUDIO (#1005). Selected by `--pipeline-kind t2a_one_stage`, which + // is a LOAD extra; these six are per-generation and are refused by name on + // any other pipeline rather than accepted and ignored. + else if (f == "--negative-prompt") + negative_prompt = Need(argc, argv, ++i, "--negative-prompt"); + else if (f == "--audio-cfg-guidance-scale") + audio_cfg_scale = Need(argc, argv, ++i, "--audio-cfg-guidance-scale"); + else if (f == "--audio-stg-guidance-scale") + audio_stg_scale = Need(argc, argv, ++i, "--audio-stg-guidance-scale"); + else if (f == "--audio-rescale-scale") + audio_rescale = Need(argc, argv, ++i, "--audio-rescale-scale"); + else if (f == "--audio-skip-step") + audio_skip_step = Need(argc, argv, ++i, "--audio-skip-step"); + else if (f == "--audio-stg-blocks") + audio_stg_blocks = Need(argc, argv, ++i, "--audio-stg-blocks"); else if (f == "--regenerate-video") regen_video = Need(argc, argv, ++i, "--regenerate-video"); else if (f == "--regenerate-audio") @@ -330,7 +363,20 @@ int main(int argc, char** argv) { std::make_pair("retake_end_time", &retake_end), std::make_pair("retake_frame_rate", &retake_fps), std::make_pair("regenerate_video", ®en_video), - std::make_pair("regenerate_audio", ®en_audio)}) { + std::make_pair("regenerate_audio", ®en_audio), + // TEXT-TO-AUDIO (#1005). One flag per upstream CLI + // argument (`default_1_stage_t2a_arg_parser`, + // ltx-pipelines utils/args.py:1070-1120). They are + // per-generation, so they ride this array rather than + // the load one; `--pipeline-kind t2a_one_stage` is the + // LOAD knob that selects the pipeline, and supplying + // these without it is refused by name. + std::make_pair("negative_prompt", &negative_prompt), + std::make_pair("audio_cfg_guidance_scale", &audio_cfg_scale), + std::make_pair("audio_stg_guidance_scale", &audio_stg_scale), + std::make_pair("audio_rescale_scale", &audio_rescale), + std::make_pair("audio_skip_step", &audio_skip_step), + std::make_pair("audio_stg_blocks", &audio_stg_blocks)}) { if (kv.second->empty()) continue; gen_keys.emplace_back(kv.first); gen_values.push_back(*kv.second); diff --git a/include/vllm/model_executor/models/ltx2.h b/include/vllm/model_executor/models/ltx2.h index 339908dab..998cda7c9 100644 --- a/include/vllm/model_executor/models/ltx2.h +++ b/include/vllm/model_executor/models/ltx2.h @@ -39,11 +39,28 @@ // message naming the missing phase rather than silently computing in f32. // // NOT PORTED IN L2, recorded here so it cannot be discovered later: -// - Guidance perturbations (guidance/perturbations.py BatchedPerturbationConfig, -// transformer.py:330-397 `*_perturbation_mask` / `cross_attn_skip_all` and -// attention.py:545-552 `perturbation_mask`). L2 runs the no-perturbation -// configuration, whose masks are all-ones and whose flags are all false — -// upstream's own `perturbations=None` path (model.py:509-511). +// - The CROSS-attention guidance perturbations only, as of row +// LTX25-T2A-ONE-STAGE (#1005): `SKIP_A2V_CROSS_ATTN` and +// `SKIP_V2A_CROSS_ATTN` (guidance/perturbations.py:8-16, +// transformer.py:330-397 `cross_attn_skip_all`). Nothing upstream that this +// port serves constructs them — STG is built from `stg_blocks` and reaches +// the SELF-attention types alone (utils/constants.py:49-68 through +// guiders.py:194-211) — and `Ltx2Attention` refuses the flag on a cross call +// by name rather than applying the self-attention rule to it. +// +// THE SELF-ATTENTION HALF IS NOW PORTED, and this entry said the whole +// mechanism was unported until #1005. `Ltx2DitPerturbation` is upstream's +// `perturbations` argument (model.py:492) at the one batch size this port +// serves, and `Ltx2AttentionArgs::all_perturbed` is +// `use_attention = not all_perturbed` (attention.py:557). `nullptr` remains +// upstream's `perturbations=None` path (model.py:509-511) and is what every +// caller but text-to-audio passes. +// +// The BATCHED form (`BatchedPerturbationConfig`, perturbations.py:53-143, +// indexed [type, block, SAMPLE]) is still unported, and so is the partial +// blend it exists for (`out * mask + v * (1 - mask)`, attention.py:571-572). +// Both are degenerate at `Ltx2ModalityInput::batch == 1`, which is the only +// batch any path here runs. // - The caption projections (text_projection.py:31-38). LTX-2.5 is a 22B-form // checkpoint: `caption_proj_before_connector=true` puts them in the TEXT // ENCODER, so the DiT has none (model_configurator.py:199-219). They are @@ -430,6 +447,24 @@ struct Ltx2AttentionArgs { // a path whose K/V really are timestep-independent — the caller owns that. const Ltx2CrossKv* kv_in = nullptr; Ltx2CrossKv* kv_out = nullptr; + + // `all_perturbed` (attention.py:552-553, `use_attention = not all_perturbed` + // at :557). The STG perturbation: the attention output is REPLACED by the raw + // value projection, `out = v`, and `to_q` / `to_k` / the q,k RMSNorms / RoPE / + // the attention itself never run. `to_out` still does (`:579`), and the gate + // still does (`:576-578`) — the substitution happens INSIDE the attention, not + // around it, so a port that returned `v` to the caller and skipped `to_out` + // would be a different operator. + // + // Upstream also carries a PARTIAL form, `out = out * mask + v * (1 - mask)` + // (`:571-572`), which blends per BATCH ROW. It is not ported here and it is not + // needed: it exists so one batch can mix perturbed and unperturbed samples, + // and this port runs `max_batch_size = 1` (`Ltx2ModalityInput::batch`), where + // the mask is all-ones or all-zeros and the blend degenerates to exactly the + // two cases this flag expresses. Recorded rather than assumed, because a + // batched arm that reached this field would need the blend and would get the + // all-or-nothing answer silently. + bool all_perturbed = false; }; std::vector Ltx2Attention(vt::Device device, const Ltx2AttentionWeights& w, const float* x, const float* context, const Ltx2AttentionArgs& args); @@ -498,14 +533,59 @@ struct Ltx2DitOutputs { std::vector audio; // [batch, audio tokens, audio_out_channels] }; -// LTXModel.forward (model.py:492-538) for model_type=AudioVideo, plus the -// preprocessors it drives (transformer_args.py:263-411). BOTH streams are -// required: LTX-2.5 is an AudioVideo checkpoint, and the VideoOnly / AudioOnly -// model types (model.py:31-33) build a different parameter set, so they are -// refused rather than served by an ungated path. To run one stream of an AV model -// — which is what upstream's own pipeline does — clear `enabled` on the other; -// the audio<->video cross attention still reads its state, exactly as -// transformer.py:265-269 does. +// `perturbations` on LTXModel.forward (model.py:492), reduced to what this port +// can express. One entry per BLOCK; `true` means that block's self-attention is +// replaced by its value projection, which is STG. +// +// A `BatchedPerturbationConfig` upstream (guidance/perturbations.py:53-143) is +// indexed [type, block, SAMPLE]; this is indexed [block] alone, because +// `Ltx2ModalityInput::batch` is 1 on every path here and the sample axis is a +// degenerate one. `Ltx2BatchedPerturbationConfig` (ltx2_pipeline.h) is the +// batched form and stays ungated by any product caller — recorded as owed in +// .agents/specs/ltx25-t2a-one-stage.md rather than silently bypassed. +// +// EMPTY IS NOT "NOTHING PERTURBED BY COINCIDENCE": a vector of the wrong length +// is REFUSED, so a config built for a different layer count cannot silently +// perturb the first N blocks and leave the rest alone. +// +// The two CROSS-attention perturbation types upstream defines +// (SKIP_A2V_CROSS_ATTN, SKIP_V2A_CROSS_ATTN, perturbations.py:8-16) have no +// field here and are not ported. Nothing upstream that this port serves +// constructs them: STG is built from `stg_blocks` and reaches the self-attention +// types only (utils/constants.py:49-68 through guiders.py:194-211). +struct Ltx2DitPerturbation { + std::vector video_self_attn; // [num_layers], empty = none + std::vector audio_self_attn; // [num_layers], empty = none +}; + +// LTXModel.forward (model.py:492-538), plus the preprocessors it drives +// (transformer_args.py:263-411). +// +// EXACTLY ONE OF THE TWO STREAMS MAY BE NULL, which is upstream's own +// `video_args = ... if video is not None else None` (model.py:505) and the +// shape `T2AOneStagePipeline` runs (t2a_one_stage.py:167 passes `video=None`). +// Both null is refused: upstream refuses it too (transformer.py:259-260, "At +// least one of video or audio must be provided"). +// +// `video = nullptr` IS NOT `video->enabled = false`, and the difference renders +// rather than failing. Upstream's predicate is `run_v2a = run_ax and (video is +// not None and vx.numel() > 0)` (transformer.py:269) — it tests PRESENCE, not +// `enabled` — so a disabled-but-present video stream still feeds video->audio +// cross attention from that stream's latent. This header used to advise +// `enabled` as the way to run one stream, and for the audio-only case that +// advice was wrong. `enabled` remains correct for running one stream of a JOINT +// render, where the cross attention reading the other stream's state is the +// intent. +// +// WHAT IS *NOT* THE REASON, re-derived at this tree rather than inherited: the +// AudioOnly / VideoOnly WEIGHT CONTRACT is not what blocks a one-stream call, and +// this check used to say it was. `T2AOneStagePipeline` loads the ordinary +// AudioVideo checkpoint FILE and merely restricts which keys are read +// (LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP, model_configurator.py:228-239), so +// the contract `EnumerateLtx2DitTensors` describes is the one it satisfies. What +// remains true is a statement about the LOADER: a checkpoint saved with only the +// audio subset still cannot be materialized here, and that refusal lives at +// `Ltx2LoadDitFromSafetensors`, where it is about the file. // // `compute_dtype` must be vt::DType::kF32 — see the DTYPE note at the top of this // file. Anything else is REFUSED with a message naming phase L6. @@ -521,7 +601,8 @@ struct Ltx2DitOutputs { Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, const Ltx2DitWeights& weights, const Ltx2ModalityInput* video, const Ltx2ModalityInput* audio, vt::DType compute_dtype, - Ltx2PromptKvCache* cache = nullptr); + Ltx2PromptKvCache* cache = nullptr, + const Ltx2DitPerturbation* perturbations = nullptr); // One BasicAVTransformerBlock (transformer.py:254-417), exposed so the block is // gateable on its own. `video_x` / `audio_x` are updated IN PLACE. @@ -541,6 +622,9 @@ struct Ltx2BlockArgs { // only the static per-block table applies (:441). const float* video_prompt_modulation = nullptr; const float* audio_prompt_modulation = nullptr; + // STG for THIS block (attention.py:552-577). See `Ltx2DitPerturbation`. + bool video_self_attn_perturbed = false; + bool audio_self_attn_perturbed = false; // Audio<->video cross-attention AdaLN inputs (transformer_args.py:388-411). const float* video_cross_scale_shift = nullptr; // [batch, video tokens, 4 * dim] const float* video_cross_gate = nullptr; // [batch, 1, dim] diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index 419d546ac..93c75cc22 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -562,6 +562,17 @@ struct Ltx2PipelineRecipe { bool allow_negative_prompt = true; bool fixed_num_inference_steps = false; + // `T2AOneStagePipeline` (t2a_one_stage.py:43). TRUE means the pipeline passes + // `video=None` to the stage (`:167`) and returns a waveform and nothing else + // (`:172`) — there is no video latent, no video VAE decode and no frame. + // + // A FLAG ON THE RECIPE RATHER THAN A STRING COMPARE AT THE CALL SITE, because + // the engine has to answer "is there a picture" in four places (geometry, + // resolution guard, decode, artifacts) and four independent `kind == + // "t2a_one_stage"` tests are four chances for one of them to be missed on the + // next audio-only recipe. The recipe table is the one place that knows. + bool audio_only = false; + int64_t max_spatial_downscale() const; }; @@ -627,6 +638,21 @@ void Ltx2AssertResolution(int64_t height, int64_t width, int64_t divisor); // `use_keyframes_abs_pos_embedding` // ("retake", "2") Lightricks retake.py:85,287,290-294,313-324 // ("retake", "2.5") same +// ("t2a_one_stage", "2") Lightricks t2a_one_stage.py:43,109 (row +// ("t2a_one_stage", "2.3") LTX25-T2A-ONE-STAGE, #1005). The one_stage +// ("t2a_one_stage", "2.4") rows' own schedule with `audio_only` set: +// ("t2a_one_stage", "2.5") T2A hard-codes the SAME `LTX2Scheduler()` +// (t2a_one_stage.py:67 against +// ti2vid_one_stage.py:81) and the same +// `detect_params` step count, and differs in +// carrying no video stream at all +// +// The four `t2a_one_stage` rows mirror the four `one_stage` rows one for one, and +// the negative prompt follows the same split for the same reason: it travels with +// the GENERATION, not with the pipeline. There is no "which versions support +// text-to-audio" question upstream — `T2AOneStagePipeline` takes whatever +// `resolve_cli_params` read off the checkpoint (t2a_one_stage.py:178-179), so +// restricting these rows to 2.5 would be a local invention. // // The `retake` rows are Lightricks' `RetakePipeline` and have no vLLM-Omni // counterpart at all. Every value on them is read off `retake.py` rather than diff --git a/include/vllm/model_executor/models/ltx2_t2a.h b/include/vllm/model_executor/models/ltx2_t2a.h new file mode 100644 index 000000000..80af49c4d --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_t2a.h @@ -0,0 +1,168 @@ +// LTX-2.5 TEXT-TO-AUDIO — the port of `T2AOneStagePipeline`, and the first path +// in this tree that renders no picture at all. +// +// Row: LTX25-T2A-ONE-STAGE. Spec: .agents/specs/ltx25-t2a-one-stage.md. +// Issue #1005. Campaign #644 / #435. +// +// ─── WHAT THIS TU IS A PORT OF (file:line on BOTH sides) ───────────────────── +// Upstream: Lightricks/LTX-2 @ fd4ded7f +// OURS <- UPSTREAM +// Ltx2T2aGenerate <- ltx-pipelines t2a_one_stage.py:109-172 +// (T2AOneStagePipeline.__call__) +// the guided step <- ltx-core components/guiders.py:244-273 +// (MultiModalGuider.calculate), reached through +// ltx-pipelines utils/denoisers.py +// (FactoryGuidedDenoiser) +// the STG pass <- ltx-core model/transformer/attention.py:552-577 +// the audio latent shape <- ltx-core types.py:164-200 +// (AudioLatentShape.from_video_pixel_shape) +// the schedule <- ltx-core components/schedulers.py:21-57, +// hard-coded as LTX2Scheduler() at +// t2a_one_stage.py:67 +// +// ─── WHY IT IS ITS OWN TRANSLATION UNIT ────────────────────────────────────── +// +// Upstream's T2A is its own FILE with its own `__call__`, and AGENTS.md +// §"Shared seams" requires mirroring that structure. It is reached only through +// `Ltx2VideoEngine::Generate`, it owns no weights, and every numeric it uses is +// an already-gated brick — so it is a composition, not a second path. +// +// What it deliberately does NOT do is thread an `is_t2a` flag through the joint +// phase driver in `ltx2_video.cpp`. That function is 1900 lines and roughly a +// third of it constructs a video stream that does not exist here; nine new +// branches inside it would be nine chances to leave one behind, and a missed one +// renders. +// +// ─── THE THREE THINGS THAT FAIL SILENTLY IF GUESSED ────────────────────────── +// +// 1. `video = nullptr`, NOT `video->enabled = false`. Upstream's predicate is +// `run_v2a = run_ax and (video is not None and vx.numel() > 0)` +// (transformer.py:269) — it tests PRESENCE. A disabled-but-present video +// stream still feeds video->audio cross attention from a latent T2A never +// meant to exist, and still returns a finished, playable waveform. +// +// 2. THE GUIDER IS NOT OPTIONAL HERE. `distilled_two_stage` builds a +// `SimpleDenoiser` upstream and this engine's joint loop mirrors that with one +// forward per step. T2A builds a `FactoryGuidedDenoiser` +// (t2a_one_stage.py:154-161) whose CLI defaults are `cfg_scale=7.0` and +// `stg_scale=1.0` (utils/constants.py:58-66 through :118), so +// `do_unconditional_generation` and `do_perturbed_generation` are both TRUE +// (guiders.py:275-281) and the default path is THREE forwards per step. A +// single-forward T2A produces audio of exactly the right length on a +// trajectory the model was not asked for. +// +// 3. `modality_scale` IS PINNED TO 1.0, and the pin is upstream's, at the CLI +// layer: "Audio-only generation has no video modality, so the video->audio +// (v2a) cross-modal guidance is meaningless here. 1.0 disables it" +// (t2a_one_stage.py:200-202). The params table's own value is 3.0, so +// inheriting it would turn on a fourth forward against a modality that is not +// there. +#pragma once + +#include +#include +#include + +#include "vllm/model_executor/models/ltx2.h" +#include "vllm/model_executor/models/ltx2_audio_vae.h" +#include "vllm/model_executor/models/ltx2_loader.h" +#include "vllm/model_executor/models/ltx2_pipeline.h" +#include "vllm/model_executor/models/ltx2_video_vae.h" +#include "vt/device.h" + +namespace vllm { + +// Everything `T2AOneStagePipeline.__call__` reads, as borrowed pointers. Nothing +// here is owned: the engine holds the weights for its whole lifetime and this +// call runs inside its mutex. +struct Ltx2T2aRequest { + vt::Device device; + vt::DType compute_dtype = vt::DType::kF32; + const Ltx2DitParams* dit_params = nullptr; + const Ltx2DitWeights* dit_weights = nullptr; + + // The AUDIO conditioning rows, `[context_tokens, audio_cross_attention_dim]`. + // `negative_context` is `ctx_n.audio_encoding` (t2a_one_stage.py:135) and is + // null when the caller has no negative conditioning — which is a REFUSAL when + // the guider asks for an unconditional pass, never a silent drop to + // `uncond = 0`. + const float* context = nullptr; + const float* negative_context = nullptr; + int64_t context_tokens = 0; + + int64_t num_frames = 0; + double frame_rate = 0.0; + int64_t steps = 0; // <= 0 => the recipe's own count + + // `GaussianNoiser(generator=torch.Generator(...).manual_seed(seed))` + // (t2a_one_stage.py:124-125). Supplied by the CALLER rather than constructed + // here: the engine already owns the one Gaussian source in this tree, and a + // second generator seeded the same way would be a parallel path whose + // agreement with the first nothing gates. + Ltx2NoiseStream* noise = nullptr; + + Ltx2MultiModalGuiderParams guidance; + + const Ltx2AudioDecoderConfig* audio_cfg = nullptr; + const Ltx2VaeWeights* audio_weights = nullptr; + const Ltx2VocoderBweConfig* vocoder_cfg = nullptr; + const Ltx2VaeWeights* vocoder_weights = nullptr; +}; + +// The rendered soundtrack, plus the observability the render itself cannot be +// inspected for. +// +// EVERY COUNTER BELOW IS INCREMENTED AT THE FORWARD, not derived from the params +// that were supposed to drive it. A field written from `guidance.cfg_scale` +// would report a healthy uncond count on a build that computed the params and +// then ran one forward — which is the exact instrument failure +// `Ltx2ConditioningTrace::audio_frozen` already paid for on this campaign. +struct Ltx2T2aResult { + // [channels, samples_per_channel], the vocoder's own layout. + std::vector waveform; + int64_t channels = 0; + int64_t samples_per_channel = 0; + int64_t sample_rate = 0; + + int64_t audio_tokens = 0; + int64_t latent_frames = 0; + + // Forwards actually issued, by arm. + int64_t cond_forwards = 0; + int64_t uncond_forwards = 0; + int64_t perturbed_forwards = 0; + + // TRUE if any forward was handed a video stream. It must be FALSE on every + // T2A render, and it is recorded rather than asserted in prose because + // §"THE THREE THINGS" item 1 is invisible in the output: a run that passed a + // present-but-disabled video stream produces a waveform of exactly the right + // length, the right channel count and the right sample rate. + bool video_stream_present = false; + + // The blocks the STG pass actually perturbed, read off the vector handed to + // the forward. A count alone cannot tell "perturbed block 1" from "perturbed + // block 0", and `stg_blocks` is what decides which. + std::vector perturbed_blocks; + + // FNV-1a over the final audio latent's raw f32 bytes, and its max|x|. The + // digest detects CHANGE; the absmax is the lower bound a digest cannot make, + // because a latent that collapsed to zeros has a perfectly stable digest. + uint64_t latent_digest = 0; + double latent_absmax = 0.0; +}; + +// `T2AOneStagePipeline.__call__` (t2a_one_stage.py:109-172). Throws +// std::runtime_error naming the problem; never renders a fallback. +Ltx2T2aResult Ltx2T2aGenerate(const Ltx2T2aRequest& req); + +// `stg_blocks` -> the per-block vector `Ltx2DitForward` takes +// (guidance/perturbations.py:19-33, `blocks is None` meaning ALL). Exposed so a +// test can pin the mapping without reaching into the pipeline, and so the one +// place that turns a block LIST into a block MASK is named. +// +// A block index outside `[0, num_layers)` is REFUSED. Upstream indexes +// `self.transformer_blocks` by it and would raise; silently ignoring it would +// run an unperturbed pass and call it STG. +std::vector Ltx2StgBlockMask(const std::vector& stg_blocks, int64_t num_layers); + +} // namespace vllm diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index affefbd1d..eedd6f0c0 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -420,6 +420,45 @@ inline constexpr char kLtx2RetakeFrameRateExtra[] = "retake_frame_rate"; inline constexpr char kLtx2RegenerateVideoExtra[] = "regenerate_video"; inline constexpr char kLtx2RegenerateAudioExtra[] = "regenerate_audio"; +// ── TEXT-TO-AUDIO. Row LTX25-T2A-ONE-STAGE (#1005) ───────────────────────── +// +// These are read ONLY on a `pipeline_kind = t2a_one_stage` engine — that is a +// LOAD extra, so which pipeline runs is fixed before a request arrives. Supplied +// on any other pipeline they are REFUSED, because upstream's other entry points +// have no counterpart for them and a knob that silently does nothing is the +// defect this whole surface refuses by name elsewhere. +// +// `--negative-prompt` (ltx-pipelines utils/args.py:1083-1088). ABSENT MEANS the +// recipe's own default, which is upstream's `DEFAULT_NEGATIVE_PROMPT` +// (utils/constants.py:186) on the 2.4/2.5 rows. +// +// IT IS NOT COSMETIC ON THIS PIPELINE. T2A's CFG scale defaults to 7.0, so the +// negative conditioning is one of the two tensors the guidance delta is computed +// from (`(cfg_scale - 1) * (cond - uncond_text)`, guiders.py:262). An empty one +// is refused rather than substituted with zeros: a zero `uncond_text` turns the +// delta into `cfg_scale * cond`, which is a DIFFERENT render and not a missing +// one. +inline constexpr char kLtx2NegativePromptExtra[] = "negative_prompt"; + +// The audio guider, one CLI flag each (utils/args.py:1089-1119). ABSENT MEANS the +// params table's own value for the checkpoint's generation — 7.0 / 1.0 / 0.7 and +// block 28 on the 2.3-and-later lineage (utils/constants.py:58-66, :82-87). +// +// `audio_stg_blocks` is a COMMA-SEPARATED list, mirroring `nargs="*"`. An EMPTY +// value is upstream's empty list and means "perturb nothing" — which is refused +// alongside a non-zero STG scale rather than silently running a perturbed pass +// identical to the conditional one. Upstream's `blocks is None` ("every block", +// guidance/perturbations.py:19-33) has no CLI spelling and none is invented here. +// +// There is deliberately NO `modality_scale` knob: the CLI pins it to 1.0 for this +// pipeline and states the reason (t2a_one_stage.py:200-202), so exposing it would +// offer a fourth forward over a modality that does not exist. +inline constexpr char kLtx2AudioCfgScaleExtra[] = "audio_cfg_guidance_scale"; +inline constexpr char kLtx2AudioStgScaleExtra[] = "audio_stg_guidance_scale"; +inline constexpr char kLtx2AudioRescaleScaleExtra[] = "audio_rescale_scale"; +inline constexpr char kLtx2AudioSkipStepExtra[] = "audio_skip_step"; +inline constexpr char kLtx2AudioStgBlocksExtra[] = "audio_stg_blocks"; + // WHAT THE LAST `Generate()` ACTUALLY HANDED THE DiT's CROSS-ATTENTION. // // Every field is read off the exact f32 buffers `Ltx2ModalityInput::context` @@ -660,6 +699,37 @@ struct Ltx2ConditioningTrace { uint64_t retake_latent_digest = 0; double retake_latent_absmax = 0.0; + // ── TEXT-TO-AUDIO: what the audio-only render actually ran (#1005) ──────── + // + // Zero and false everywhere on a pipeline that is not `t2a_one_stage`. + // + // `t2a_video_stream_present` is the one that cannot be inferred from anything + // else here, and it is the field this block exists for. Upstream's own + // predicate is `run_v2a = run_ax and (video is not None and vx.numel() > 0)` + // (transformer.py:269): it tests PRESENCE, not `enabled`. So a build that + // handed the forward a present-but-DISABLED video stream would still feed + // video->audio cross attention from a latent T2A never meant to exist — and + // would return a waveform of exactly the right length, the right channel count + // and the right sample rate. There is no sample to compare and no digest of + // the output that says which happened. + // + // The three forward counters are incremented AT THE FORWARD, not derived from + // the guider parameters that were supposed to drive them. A field written off + // `cfg_scale` would report a healthy uncond count on a build that resolved the + // params and then ran one forward, which is the instrument failure + // `audio_sigma_max` above already paid for on this campaign. + // + // `t2a_perturbed_blocks` is read off the mask handed to the DiT rather than + // off the request's `stg_blocks`: a count alone cannot tell "perturbed block + // 1" from "perturbed block 0", and which block is perturbed is the whole of + // STG. + bool t2a_rendered = false; + bool t2a_video_stream_present = false; + int64_t t2a_cond_forwards = 0; + int64_t t2a_uncond_forwards = 0; + int64_t t2a_perturbed_forwards = 0; + std::vector t2a_perturbed_blocks; + // True only once the `Generate` that produced this conditioning RETURNED. The // trace is filled immediately after the connector and BEFORE the denoise loop, // because that is the only point at which the exact buffers cross-attention @@ -723,6 +793,21 @@ class Ltx2VideoEngine : public VideoEngine { Ltx2VideoEngine(); struct Impl; std::unique_ptr impl_; + + // `T2AOneStagePipeline.__call__` (ltx2_t2a.h), reached from `Generate` on an + // `audio_only` recipe and from nowhere else. + // + // A PRIVATE STATIC rather than a free function in the .cpp, because it needs + // `Impl` — a private nested type no non-member can name. The alternatives were + // a 150-line branch inside a function that is already 1900 lines, or a + // template whose only purpose is to deduce a type it is not allowed to spell. + // + // `audio_context` is the conditioning `Generate` already resolved: the audio + // half of the prompt encoding, after the connector. Passing it in rather than + // re-encoding is what keeps the audio-only arm from owning a second copy of + // the connector composition. + static VideoResult GenerateAudioOnly(Impl& im, const VideoGenParams& gen, + const float* audio_context, int64_t context_tokens); }; // Does this checkpoint set hold an LTX-2.5 DiT? Exposed for the registry and for diff --git a/src/vllm/model_executor/models/ltx2.cpp b/src/vllm/model_executor/models/ltx2.cpp index 188ee3a55..8dee3d093 100644 --- a/src/vllm/model_executor/models/ltx2.cpp +++ b/src/vllm/model_executor/models/ltx2.cpp @@ -822,6 +822,41 @@ std::vector Ltx2FeedForward(vt::Device device, const Ltx2FeedForwardWeigh return out; } +// attention.py:575-579 — everything the ordinary path and the STG-perturbed path +// share, which is the gate and `to_out`. Factored out rather than duplicated +// because the two arms differing HERE is the defect that would be invisible: a +// perturbed pass that skipped `to_out` returns a tensor of the right shape at the +// wrong width-space, and the block would add it to the residual and render. +static std::vector Ltx2AttentionEpilogue(vt::Queue& q, const Ltx2AttentionWeights& w, + const float* x, std::vector attn, + const Ltx2AttentionArgs& args, + vt::Device /*device*/) { + const int64_t batch = args.batch; + const int64_t tq = args.tokens; + const int64_t heads = args.heads; + const int64_t dim_head = args.dim_head; + const int64_t inner = heads * dim_head; + + // PytorchGatedAttention (ops.py:94-106), applied to the attention output BEFORE + // `to_out` (attention.py:576-579) and driven by the RAW input `x`, not by the + // attention output. Gating after `to_out` would be a different model. + if (w.to_gate_logits.weight.data != nullptr) { + std::vector logits(static_cast(batch * tq * heads)); + Linear(q, x, batch * tq, args.query_dim, w.to_gate_logits, logits.data()); + for (int64_t r = 0; r < batch * tq; ++r) { + for (int64_t h = 0; h < heads; ++h) { + const float gate = 2.0f / (1.0f + std::exp(-logits[static_cast(r * heads + h)])); + float* dst = attn.data() + r * inner + h * dim_head; + for (int64_t e = 0; e < dim_head; ++e) dst[e] *= gate; + } + } + } + + std::vector out(static_cast(batch * tq * args.query_dim)); + Linear(q, attn.data(), batch * tq, inner, w.to_out, out.data()); + return out; +} + std::vector Ltx2Attention(vt::Device device, const Ltx2AttentionWeights& w, const float* x, const float* context, const Ltx2AttentionArgs& args) { vt::Queue q{device, nullptr}; @@ -835,6 +870,28 @@ std::vector Ltx2Attention(vt::Device device, const Ltx2AttentionWeights& const int64_t s = context != nullptr ? args.context_tokens : tq; const int64_t ctx_dim = context != nullptr ? args.context_dim : args.query_dim; + // attention.py:557 — `use_attention = not all_perturbed`. The STG arm computes + // `to_v` and NOTHING else of the attention: no `to_q`, no `to_k`, no q/k + // RMSNorm, no RoPE, no scores. Written as an early exit rather than as a chain + // of `if (!perturbed)` guards so the skipped work is visibly skipped; a guarded + // form that still projected q and threw the result away would be numerically + // identical and would hide the whole point of the perturbation, which is that + // the query/key path does not run. + if (args.all_perturbed) { + VT_CHECK(context == nullptr, + "ltx2 attention: `all_perturbed` is upstream's SELF-attention STG perturbation " + "(guidance/perturbations.py:8-16 names SKIP_VIDEO_SELF_ATTN and " + "SKIP_AUDIO_SELF_ATTN). The CROSS-attention perturbations exist upstream " + "(SKIP_A2V_CROSS_ATTN, SKIP_V2A_CROSS_ATTN) and are NOT ported, so a cross call " + "carrying this flag is refused rather than served the self-attention rule"); + VT_CHECK(args.kv_in == nullptr && args.kv_out == nullptr, + "ltx2 attention: a perturbed pass computes no K, so it can neither fill nor read a " + "prompt K/V cache"); + std::vector vp(static_cast(batch * tq * inner)); + Linear(q, ctx, batch * s, ctx_dim, w.to_v, vp.data()); + return Ltx2AttentionEpilogue(q, w, x, std::move(vp), args, device); + } + // attention.py:559-565: v first, then q and k. The K/V half is exactly what the // prompt cache holds, so `kv_in` skips all three of to_v / to_k / k_norm. const bool reuse_kv = args.kv_in != nullptr; @@ -912,24 +969,7 @@ std::vector Ltx2Attention(vt::Device device, const Ltx2AttentionWeights& } } - // PytorchGatedAttention (ops.py:94-106), applied to the attention output BEFORE - // `to_out` (attention.py:576-579) and driven by the RAW input `x`, not by the - // attention output. Gating after `to_out` would be a different model. - if (w.to_gate_logits.weight.data != nullptr) { - std::vector logits(static_cast(batch * tq * heads)); - Linear(q, x, batch * tq, args.query_dim, w.to_gate_logits, logits.data()); - for (int64_t r = 0; r < batch * tq; ++r) { - for (int64_t h = 0; h < heads; ++h) { - const float gate = 2.0f / (1.0f + std::exp(-logits[static_cast(r * heads + h)])); - float* dst = attn.data() + r * inner + h * dim_head; - for (int64_t e = 0; e < dim_head; ++e) dst[e] *= gate; - } - } - } - - std::vector out(static_cast(batch * tq * args.query_dim)); - Linear(q, attn.data(), batch * tq, inner, w.to_out, out.data()); - return out; + return Ltx2AttentionEpilogue(q, w, x, std::move(attn), args, device); } } // namespace vllm diff --git a/src/vllm/model_executor/models/ltx2_dit.cpp b/src/vllm/model_executor/models/ltx2_dit.cpp index eaef5bb67..bdf6b98cf 100644 --- a/src/vllm/model_executor/models/ltx2_dit.cpp +++ b/src/vllm/model_executor/models/ltx2_dit.cpp @@ -275,6 +275,7 @@ void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, a.pe = args.video_pe; a.bias = args.video_self_bias; a.bias_rows = args.video_self_bias_rows; + a.all_perturbed = args.video_self_attn_perturbed; const std::vector msa = Ltx2Attention(device, w.attn1, norm_vx.data(), nullptr, a); PostSelfAttention(video_x, msa.data(), gate, batch * tv, dim, eps, &vx_normed); @@ -312,6 +313,7 @@ void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, a.pe = args.audio_pe; a.bias = args.audio_self_bias; a.bias_rows = args.audio_self_bias_rows; + a.all_perturbed = args.audio_self_attn_perturbed; const std::vector msa = Ltx2Attention(device, w.audio_attn1, norm_ax.data(), nullptr, a); PostSelfAttention(audio_x, msa.data(), gate, batch * ta, adim, eps, &ax_normed); @@ -752,19 +754,27 @@ Ltx2PromptIdentity Ltx2PromptIdentityOf(const Ltx2DitParams& params, Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, const Ltx2DitWeights& weights, const Ltx2ModalityInput* video, const Ltx2ModalityInput* audio, vt::DType compute_dtype, - Ltx2PromptKvCache* cache) { + Ltx2PromptKvCache* cache, const Ltx2DitPerturbation* perturbations) { VT_CHECK(compute_dtype == vt::DType::kF32, "ltx2: phase L2 ships only the f32 parity forward; the bf16 / FP8 / NVFP4 stream " "dtypes are phase L6 and are refused rather than silently computed in f32"); - // LTX-2.5 is an LTXModelType.AudioVideo checkpoint (model_configurator.py:47), - // and that is the only weight contract EnumerateLtx2DitTensors describes. The - // VideoOnly / AudioOnly types (model.py:31-33) build a DIFFERENT parameter set — - // no audio stream, no av_ca AdaLN embedders — so they are refused by name rather - // than served by a path no golden covers. Use `enabled` to run one stream of an - // AV model, which is what the pipeline itself does. - VT_CHECK(video != nullptr && audio != nullptr, - "ltx2: phase L2 ships the AudioVideo model type only; LTXModelType.VideoOnly and " - "LTXModelType.AudioOnly carry a different weight contract and are not ported"); + // transformer.py:259-260 — upstream's own refusal, in its own words: "At least + // one of video or audio must be provided". + // + // ONE stream may be null, which is what `T2AOneStagePipeline` runs + // (t2a_one_stage.py:167, `video=None`) and what model.py:505 expresses. + // This check used to demand BOTH and blamed the AudioOnly / VideoOnly WEIGHT + // CONTRACT for it. That reason was re-derived at this tree and does not + // describe the case: T2A loads the ordinary AudioVideo FILE and restricts which + // keys it reads (LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP, + // model_configurator.py:228-239), so the contract EnumerateLtx2DitTensors + // describes is the one it satisfies. Every line below was ALREADY written + // against `video != nullptr` / `have_both`, so lifting the guard reaches a path + // this file already had. The weight-contract statement survives where it is + // true — at the loader, about the file. + VT_CHECK(video != nullptr || audio != nullptr, + "ltx2: at least one of the video and audio streams must be present " + "(transformer.py:259-260)"); const int64_t dim = params.inner_dim(); const int64_t adim = params.audio_inner_dim(); // transformer_args.py:197 views the projected context to the STREAM width, so @@ -812,8 +822,24 @@ Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, params.audio_num_attention_heads, have_both ? video : nullptr); } + // `perturbations` (model.py:492). A vector that is not exactly `num_layers` + // long is REFUSED rather than indexed defensively: a config built for another + // layer count would otherwise perturb a prefix of the blocks and leave the rest + // alone, which is a legal-looking STG pass over the wrong blocks and renders. + if (perturbations != nullptr) { + for (const std::vector* v : + {&perturbations->video_self_attn, &perturbations->audio_self_attn}) { + VT_CHECK(v->empty() || static_cast(v->size()) == params.num_layers, + "ltx2: a perturbation vector is neither empty nor one entry per block"); + } + } + const bool use_cache = cache != nullptr; if (use_cache) { + VT_CHECK(video != nullptr && audio != nullptr, + "ltx2: the prompt K/V cache keys on BOTH streams' context tensors " + "(Ltx2PromptIdentityOf), so it is refused on a one-stream call rather than keyed on " + "half an identity — two different renders would otherwise share a cache entry"); // The cached K/V are a function of the PROMPT (and of nothing else on this // path — that is what use_prompt_adaln_single=false buys). A filled cache is // therefore bound to one prompt, and a call carrying another one is refused @@ -837,6 +863,12 @@ Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, a.audio_context_tokens = audio != nullptr ? audio->context_tokens : 0; a.video_enabled = video != nullptr && video->enabled; a.audio_enabled = audio != nullptr && audio->enabled; + if (perturbations != nullptr) { + a.video_self_attn_perturbed = !perturbations->video_self_attn.empty() && + perturbations->video_self_attn[static_cast(i)] != 0; + a.audio_self_attn_perturbed = !perturbations->audio_self_attn.empty() && + perturbations->audio_self_attn[static_cast(i)] != 0; + } a.video_timestep_modulation = vs.modulation.empty() ? nullptr : vs.modulation.data(); a.audio_timestep_modulation = as.modulation.empty() ? nullptr : as.modulation.data(); a.video_prompt_modulation = diff --git a/src/vllm/model_executor/models/ltx2_pipeline.cpp b/src/vllm/model_executor/models/ltx2_pipeline.cpp index 026227118..7291ecbb5 100644 --- a/src/vllm/model_executor/models/ltx2_pipeline.cpp +++ b/src/vllm/model_executor/models/ltx2_pipeline.cpp @@ -1068,6 +1068,23 @@ Ltx2PhaseRecipe OneStagePhase(const Ltx2PipelineParams& params) { phase.name = "generate"; phase.video_guidance = params.video_guider; phase.audio_guidance = params.audio_guider; + // #1013. This was left at the struct's 0.0 default, and 0.0 is not "no extra + // noise": `Ltx2GaussianNoise` is `latent + noise_scale * (noise - latent)`, so + // at 0.0 the state stays exactly as `create_initial_state` wrote it, which + // with no initial latent is ALL ZEROS. A one_stage render therefore denoised a + // zero tensor. + // + // Upstream's `ModalitySpec.noise_scale` defaults to 1.0 + // (ltx-pipelines/utils/types.py:110) and `TI2VidOneStagePipeline.__call__` + // constructs both specs without it (ti2vid_one_stage.py:233-239), so 1.0 is + // what reaches `GaussianNoiser.__call__`'s `torch.lerp(latent, noise, + // noise_scale)` (components/noisers.py:31). The two neighbouring recipes + // already set it explicitly, which is what made the omission legible. + // + // No gate saw it because every end-to-end test loads `distilled_two_stage`, + // and a zero-initialized denoise still returns a finite clip of the right + // size, frame count and sample rate. + phase.noise_scale = 1.0; return phase; } @@ -1085,6 +1102,43 @@ Ltx2PipelineRecipe OneStageRecipe(const Ltx2PipelineParams& params, return recipe; } +// `T2AOneStagePipeline` (t2a_one_stage.py:43). Built FROM `OneStageRecipe` +// rather than beside it, because upstream's difference between the two is not in +// the schedule: both hard-code `LTX2Scheduler()` (`:67` against +// ti2vid_one_stage.py:81) and both take `num_inference_steps` from the same +// `PipelineParams`. What differs is that there is no video. +// +// The geometry fields are left at the params table's values and are DEAD on this +// recipe — upstream fills the same slots with a 512x512 placeholder whose height +// and width it documents as unused (t2a_one_stage.py:37-40). Only `num_frames` +// and `frame_rate` are read, and they are read to derive the audio DURATION +// (`AudioLatentShape.from_video_pixel_shape`, types.py:184-200). +// +// The VIDEO guider is deliberately left at its default and is never consumed: +// upstream's T2A CLI constructs ONE `MultiModalGuiderParams` and it is the audio +// one (`:196-205`). Zeroing it here would look tidier and would be a fabricated +// value; leaving the params table's own entry says "this recipe does not read +// it" without inventing a number. +Ltx2PipelineRecipe T2aOneStageRecipe(const Ltx2PipelineParams& params, + const std::string& negative_prompt) { + Ltx2PipelineRecipe recipe = OneStageRecipe(params, negative_prompt); + recipe.audio_only = true; + // `video_output_phase` is already -1 on a fresh recipe; restated because on + // THIS recipe it is a statement rather than a default, and a later edit that + // gave the field a real value would otherwise silently ask for a video output + // from a pipeline that produces none. + recipe.video_output_phase = -1; + // `modality_scale=1.0` — the CLI pins it, and says why: "Audio-only generation + // has no video modality, so the video->audio (v2a) cross-modal guidance is + // meaningless here. 1.0 disables it" (t2a_one_stage.py:200-202). It is the ONE + // guider field T2A overrides against the params table's 3.0, and 1.0 is exactly + // the value `do_isolated_modality_generation` reads as OFF + // (guiders.py:283-285). Applied at the recipe rather than at the call site so + // no caller can reach the isolated-modality forward this port does not have. + recipe.phases[0].audio_guidance.modality_scale = 1.0; + return recipe; +} + // LTX_POSITIVE_ONLY_RECIPE (ltx2_recipes.py:116-124): every guidance knob at its // no-op value, and the official sigma schedule turned OFF. Ltx2PipelineRecipe PositiveOnlyRecipe() { @@ -1261,6 +1315,15 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, if (model_version == "2" || model_version == "2.3") return PositiveOnlyRecipe(); } else if (pipeline_kind == "retake") { if (model_version == "2" || model_version == "2.5") return RetakeRecipe(model_version); + } else if (pipeline_kind == "t2a_one_stage") { + if (model_version == "2") return T2aOneStageRecipe(Ltx2Params20(), kOmniNegativePrompt); + if (model_version == "2.3") return T2aOneStageRecipe(Ltx2Params23(), kOmniNegativePrompt); + if (model_version == "2.4") { + return T2aOneStageRecipe(Ltx2DetectPipelineParams("2.4"), LightricksNegativePrompt()); + } + if (model_version == "2.5") { + return T2aOneStageRecipe(Ltx2DetectPipelineParams("2.5"), LightricksNegativePrompt()); + } } Refuse("Unsupported LTX pipeline kind/version: '" + pipeline_kind + "'/'" + model_version + "'. Recipes are resolved from an EXACT (kind, version) table " diff --git a/src/vllm/model_executor/models/ltx2_t2a.cpp b/src/vllm/model_executor/models/ltx2_t2a.cpp new file mode 100644 index 000000000..2cce2a646 --- /dev/null +++ b/src/vllm/model_executor/models/ltx2_t2a.cpp @@ -0,0 +1,347 @@ +// `T2AOneStagePipeline` (ltx-pipelines t2a_one_stage.py:43, `__call__` at :109) +// at Lightricks/LTX-2 @ fd4ded7f. See ltx2_t2a.h for the port map and for the +// three details that fail silently if guessed. +// +// Row LTX25-T2A-ONE-STAGE, issue #1005. + +#include "vllm/model_executor/models/ltx2_t2a.h" + +#include +#include +#include +#include +#include + +#include "vt/dtype.h" // VT_CHECK + +namespace vllm { +namespace { + +[[noreturn]] void Fail(const std::string& why) { throw std::runtime_error("ltx2 t2a: " + why); } + +uint64_t DigestF32(const std::vector& values) { + uint64_t h = 1469598103934665603ULL; + const auto* bytes = reinterpret_cast(values.data()); + const size_t n = values.size() * sizeof(float); + for (size_t i = 0; i < n; ++i) { + h ^= bytes[i]; + h *= 1099511628211ULL; + } + return h; +} + +double AbsMax(const std::vector& values) { + double m = 0.0; + for (const float v : values) m = std::max(m, std::abs(static_cast(v))); + return m; +} + +// `to_denoised` (ltx-core model/transformer/utils.py:38-50) through +// `X0Model.forward` (model.py:590-604): the DiT emits a VELOCITY and the sampler +// wants the x0 prediction. Identical arithmetic to the joint driver's own +// `ToDenoised`; kept here rather than shared because the joint one is a static in +// `ltx2_video.cpp`'s anonymous namespace and hoisting it would move lines above +// that file's gated READER ANCHORS list for no behavioural reason. +std::vector ToDenoised(const std::vector& sample, const std::vector& velocity, + const std::vector& timesteps, int64_t tokens, int64_t width) { + VT_CHECK(velocity.size() == sample.size(), "ltx2 t2a: the velocity is the wrong size"); + std::vector out(sample.size()); + for (int64_t t = 0; t < tokens; ++t) { + const float sigma = timesteps[static_cast(t)]; + for (int64_t c = 0; c < width; ++c) { + const size_t i = static_cast(t * width + c); + out[i] = sample[i] - sigma * velocity[i]; + } + } + return out; +} + +} // namespace + +std::vector Ltx2StgBlockMask(const std::vector& stg_blocks, int64_t num_layers) { + VT_CHECK(num_layers > 0, "ltx2 t2a: num_layers must be positive"); + std::vector mask(static_cast(num_layers), 0); + // `blocks is None` upstream means EVERY block (perturbations.py:19-33). An + // EMPTY list is not that: the CLI's `--audio-stg-blocks` with no values yields + // an empty list (utils/args.py:1107-1113, `nargs="*"`), and an empty list + // perturbs nothing. Conflating the two would turn "STG off" into "STG + // everywhere" for a caller who typed the flag and no numbers. + for (const int64_t b : stg_blocks) { + if (b < 0 || b >= num_layers) { + Fail("STG block index " + std::to_string(b) + " is outside [0, " + + std::to_string(num_layers) + + "). Upstream indexes `self.transformer_blocks` by it and raises; ignoring it here " + "would run an UNPERTURBED pass and report it as STG, which is a different guidance " + "delta on a render that still finishes"); + } + mask[static_cast(b)] = 1; + } + return mask; +} + +Ltx2T2aResult Ltx2T2aGenerate(const Ltx2T2aRequest& req) { + VT_CHECK(req.dit_params != nullptr && req.dit_weights != nullptr && req.audio_cfg != nullptr && + req.audio_weights != nullptr && req.vocoder_cfg != nullptr && + req.vocoder_weights != nullptr && req.noise != nullptr, + "ltx2 t2a: the request is missing a required borrowed pointer"); + const Ltx2DitParams& params = *req.dit_params; + + if (req.num_frames < 1) { + // `require_num_frames_source` (utils/blocks.py:894-905) — upstream's own + // fast refusal, raised at the TOP of `__call__` before prompt encoding, so + // an unsatisfiable auto-duration costs no work. + // + // WHAT IS *NOT* THE REASON: not the duration head's ARITHMETIC. + // `Ltx2DurationHeadForward` is ported and gated, including the audio-only + // case this pipeline would use — `test_ltx2_pipeline` runs it against + // `kLtx2DurAudioOnlyGolden`, generated from executed upstream. What is + // missing is a CONSTRUCTED head: nothing in this engine builds one, and + // `duration_head_path` is refused by name at load (#611). So the arithmetic + // exists and the object does not. + Fail("this request carries no frame count, and audio-only generation derives its DURATION " + "from one: `AudioLatentShape.from_video_pixel_shape` reads `frames` and `fps` off the " + "pixel shape (ltx-core types.py:184-200), which is why upstream passes a 512x512 " + "PLACEHOLDER resolution and a real frame count (t2a_one_stage.py:37-40, :163-166). " + "Auto duration needs a DurationHead this engine does not construct. Pass num_frames or " + "duration_seconds."); + } + if (req.frame_rate <= 0.0) Fail("frame_rate must be positive"); + if (req.context == nullptr || req.context_tokens < 1) { + Fail("no audio conditioning was supplied; `ctx_p.audio_encoding` is what this pipeline " + "cross-attends over (t2a_one_stage.py:134)"); + } + + // ── the audio latent shape (types.py:164-200) ───────────────────────────── + // + // `AudioLatentShape.from_video_pixel_shape` takes `frames` and `fps` from the + // pixel shape and NOTHING else — height and width are unused, which is exactly + // why upstream can pass a placeholder for them. + const Ltx2AudioPatchifierParams ap; + const double latents_per_second = static_cast(ap.sample_rate) / + static_cast(ap.hop_length) / + static_cast(ap.audio_latent_downsample_factor); + Ltx2AudioLatentShape ashape; + ashape.batch = 1; + ashape.channels = 8; // types.py:184-200 defaults, asserted against the DiT below + ashape.mel_bins = 16; + ashape.frames = static_cast( + std::llround(static_cast(req.num_frames) / req.frame_rate * latents_per_second)); + if (ashape.frames < 1) { + Fail("the audio latent resolved to zero frames for " + std::to_string(req.num_frames) + + " frames at " + std::to_string(req.frame_rate) + " fps"); + } + // The latent's channels x mel_bins IS the DiT's audio stream width, and a + // mismatch reinterprets the spectrogram rather than failing. Checked against + // BOTH factors and not only their product: a (16, 8) latent is the same width + // as an (8, 16) one and unpatchifies into a different tensor. + if (ashape.channels * ashape.mel_bins != params.audio_in_channels) { + Fail("the audio latent is " + std::to_string(ashape.channels) + " x " + + std::to_string(ashape.mel_bins) + " = " + + std::to_string(ashape.channels * ashape.mel_bins) + + " wide (types.py:184-200) but this DiT's audio stream takes " + + std::to_string(params.audio_in_channels)); + } + if (params.audio_cross_attention_dim < 1) Fail("this DiT declares no audio context width"); + + Ltx2T2aResult result; + result.latent_frames = ashape.frames; + + const int64_t width = ashape.channels * ashape.mel_bins; + const int64_t tokens = ashape.frames; // AudioPatchifier(patch_size=1) + result.audio_tokens = tokens; + + // ── the schedule (t2a_one_stage.py:141-143) ─────────────────────────────── + // + // `LTX2Scheduler()` is HARD-CODED at `:67`, so there is no scheduler-kind + // question here and no distilled sigma table: the recipe carries none and this + // computes them. + // + // THE TOKEN COUNT IS THE SCHEDULER'S OWN DEFAULT, NOT THE AUDIO LATENT'S, and + // this is the detail a re-derivation gets wrong. `execute` takes an OPTIONAL + // `latent` and falls back to `default_number_of_tokens = MAX_SHIFT_ANCHOR` + // when it is absent (schedulers.py:29, :32) — and `t2a_one_stage.py:141` calls + // `self._scheduler.execute(steps=num_inference_steps)` with no latent at all. + // The joint video driver in `ltx2_video.cpp` passes its own `target_tokens` + // and is right to, because the pipelines it mirrors pass a shape; copying that + // here would move `sigma_shift` by `(tokens - MAX_SHIFT_ANCHOR) * mm` + // (schedulers.py:36-38) and bend every sigma in the schedule. Nothing about + // the render's length, its channel count or its finiteness could see it. + // + // Passing 0 is `Ltx2SigmaSchedule`'s own spelling for "take the default" + // (ltx2_pipeline.h), so this is the fallback rather than a substitute for it. + const int64_t steps = req.steps; + if (steps < 1) Fail("num_inference_steps resolved to " + std::to_string(steps)); + const std::vector sigmas = Ltx2SigmaSchedule(steps, /*tokens=*/0); + const int64_t sigma_count = static_cast(sigmas.size()); + VT_CHECK(sigma_count >= 2, "ltx2 t2a: the schedule needs at least one step"); + + // ── the guider (t2a_one_stage.py:149-152) ───────────────────────────────── + const Ltx2MultiModalGuiderParams& g = req.guidance; + const bool want_uncond = g.DoUnconditionalGeneration(); + const bool want_perturbed = g.DoPerturbedGeneration(); + if (g.DoIsolatedModalityGeneration()) { + Fail("isolated-modality guidance (`modality_scale` = " + std::to_string(g.modality_scale) + + ") asks for a fourth forward over the OTHER modality, and this pipeline has no other " + "modality to run it over. Upstream pins `modality_scale` to 1.0 for exactly this " + "reason and says so: \"Audio-only generation has no video modality, so the " + "video->audio (v2a) cross-modal guidance is meaningless here. 1.0 disables it\" " + "(t2a_one_stage.py:200-202). Use 1.0."); + } + if (want_uncond && req.negative_context == nullptr) { + Fail("the guider asks for an unconditional pass (`cfg_scale` = " + + std::to_string(g.cfg_scale) + + ") and no negative conditioning was supplied. Upstream's guider carries a " + "`negative_context` and the CLI always fills it (t2a_one_stage.py:151, :193). " + "Substituting a zero tensor would make the CFG delta `cfg_scale * cond`, which is a " + "different render and not a missing one. Supply a negative prompt, or set the scale " + "to 1.0."); + } + std::vector stg_mask; + if (want_perturbed) { + stg_mask = Ltx2StgBlockMask(g.stg_blocks, params.num_layers); + for (int64_t b = 0; b < params.num_layers; ++b) { + if (stg_mask[static_cast(b)] != 0) result.perturbed_blocks.push_back(b); + } + if (result.perturbed_blocks.empty()) { + Fail("the guider asks for a perturbed pass (`stg_scale` = " + std::to_string(g.stg_scale) + + ") and `stg_blocks` names no block, so the perturbed forward would be identical to " + "the conditional one and the STG delta would be exactly zero — a full extra forward " + "per step that changes nothing. Name the blocks, or set the scale to 0."); + } + } + Ltx2DitPerturbation perturbation; + perturbation.audio_self_attn = stg_mask; + + // ── the state (helpers.py:428-447; ModalitySpec(context=...) alone at :168) ─ + // + // No initial latent and no freeze: T2A's audio `ModalitySpec` carries a + // context and nothing else (t2a_one_stage.py:168), so the denoise mask is all + // ones and there is no `clean` to blend back. + // + // THE NOISE IS UNIT VARIANCE, NOT SCALED BY `sigmas[0]`, and this draft scaled + // it until the chain was read. `ModalitySpec.noise_scale` defaults to 1.0 + // (utils/types.py:110), `create_noised_state` forwards it (helpers.py:434, + // :443) and `GaussianNoiser.__call__` is `torch.lerp(latent, noise, + // noise_scale)` (noisers.py:31) — at 1.0 the state IS the noise. A reader who + // knows other flow-matching samplers will expect the scaling, so the absence + // is written down. + // + // AND THE TWO FORMS AGREE HERE, WHICH IS WHY NO TEST SEPARATES THEM. + // MEASURED: a mutation adding `for (float& v : latent) v *= sigmas[0];` left + // the focused gate at 6 cases / 484 assertions / exit 0. That is not a blind + // instrument — it is an identity. `LTX2Scheduler` starts at `linspace(1, 0, + // steps + 1)[0] == 1`; the shift map sends 1 to `exp(s)/(exp(s) + (1/1 - 1))` + // which is exactly 1 (schedulers.py:41-45); and the stretch sends it to + // `1 - (1 - 1)/scale_factor`, again exactly 1 (`:47-55`). So `sigmas[0]` is + // 1.0 for EVERY step count, and the multiply is a no-op. + // + // That identity is GATED rather than left as this comment's word, in + // `test_ltx2_video`'s "the schedule starts at exactly 1.0". If upstream ever + // moves the first sigma off 1, that gate fires and this line becomes a real + // difference — which is the point of pinning it rather than pinning the + // mutation's survival. + std::vector latent = req.noise->Draw(tokens * width); + const std::vector positions_f = Ltx2AudioPatchTimings(ashape, ap); + const std::vector positions(positions_f.begin(), positions_f.end()); + + // ── the denoise loop (samplers.py:39-79) ────────────────────────────────── + for (int64_t step = 0; step + 1 < sigma_count; ++step) { + const float sigma = sigmas[static_cast(step)]; + // Every token carries the schedule's own sigma: the mask is all ones, so + // `timesteps_from_mask` is a constant fill (helpers.py:466-503). + const std::vector timesteps(static_cast(tokens), sigma); + + Ltx2ModalityInput ain; + ain.batch = 1; + ain.tokens = tokens; + ain.context_tokens = req.context_tokens; + ain.latent = latent.data(); + ain.timesteps = timesteps.data(); + ain.sigma = σ + ain.positions = positions.data(); + ain.context = req.context; + + // `should_skip_step` (guiders.py:287-291). `skip_step` defaults to 0, which + // never skips, so this is reachable only from an explicit request — and it + // is written as a real branch rather than left implicit because the arm it + // selects is "the conditional prediction alone", which is what a build with + // NO guider also produces. Conflating them would make the default path and + // the skipped path indistinguishable in the counters below. + const bool skip = g.ShouldSkipStep(step); + + // THE VIDEO STREAM IS `nullptr`, NOT A DISABLED ONE. See ltx2_t2a.h item 1: + // upstream's `run_v2a` tests PRESENCE (transformer.py:269), so a + // present-but-disabled stream still feeds video->audio cross attention. + // + // EVERY FORWARD GOES THROUGH THIS ONE LAMBDA, and that is what makes + // `video_stream_present` an OBSERVATION rather than a restatement. Written + // as `result.video_stream_present = false` beside a `nullptr` literal it + // would be a comment that compiles: a build that started passing a stream + // would report `false` and stay green. Derived at the call, a mutation that + // hands any forward a video stream flips it. + const auto forward = [&](const Ltx2ModalityInput* video, const Ltx2ModalityInput* audio, + const Ltx2DitPerturbation* p) { + if (video != nullptr) result.video_stream_present = true; + return Ltx2DitForward(req.device, params, *req.dit_weights, video, audio, req.compute_dtype, + /*cache=*/nullptr, p); + }; + + const Ltx2DitOutputs cond = forward(/*video=*/nullptr, &ain, /*p=*/nullptr); + ++result.cond_forwards; + + std::vector velocity = cond.audio; + if (!skip && (want_uncond || want_perturbed)) { + std::vector uncond_text; + std::vector uncond_perturbed; + if (want_uncond) { + Ltx2ModalityInput nin = ain; + nin.context = req.negative_context; + uncond_text = forward(/*video=*/nullptr, &nin, /*p=*/nullptr).audio; + ++result.uncond_forwards; + } + if (want_perturbed) { + // The POSITIVE context with the self-attention perturbed — upstream + // perturbs the MODEL, never the conditioning (guiders.py:244-273 takes + // `uncond_perturbed` from a forward whose `perturbations` differ and + // whose context does not). + uncond_perturbed = forward(/*video=*/nullptr, &ain, &perturbation).audio; + ++result.perturbed_forwards; + } + velocity = Ltx2MultiModalGuidance(g, cond.audio.data(), + want_uncond ? uncond_text.data() : nullptr, + want_perturbed ? uncond_perturbed.data() : nullptr, + /*uncond_modality=*/nullptr, + static_cast(cond.audio.size())); + } + + const std::vector denoised = ToDenoised(latent, velocity, timesteps, tokens, width); + // `EulerDiffusionStep()` — `DiffusionStage.__call__`'s own default + // (utils/blocks.py:524-527), which T2A does not override (it passes no + // `stepper`, t2a_one_stage.py:154-170). The ancestral sampler that + // `distilled.py` selects for generation 2.5 reaches this pipeline through + // nothing. + latent = Ltx2EulerStep(latent.data(), denoised.data(), sigmas.data(), sigma_count, step, + static_cast(latent.size())); + } + + result.latent_digest = DigestF32(latent); + result.latent_absmax = AbsMax(latent); + + // `clear_conditioning` + `unpatchify` (blocks.py:575-580). There is no + // conditioning item on this path and nothing appended, so the clear is an + // identity here — stated rather than called, because calling a no-op would + // suggest an append this pipeline cannot make. + const std::vector volume = Ltx2AudioUnpatchify(latent.data(), ashape); + + // ── the decode (t2a_one_stage.py:172) ───────────────────────────────────── + const Ltx2AudioSpectrogram mel = Ltx2AudioDecoderForward( + *req.audio_cfg, *req.audio_weights, volume, ashape.channels, ashape.frames, ashape.mel_bins); + result.waveform = Ltx2VocoderWithBweForward(*req.vocoder_cfg, *req.vocoder_weights, mel.data, + mel.channels, mel.frames, mel.mel_bins, + &result.samples_per_channel); + result.channels = mel.channels; + result.sample_rate = req.vocoder_cfg->output_sampling_rate; + return result; +} + +} // namespace vllm diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index ab4b8a592..30f939d7d 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -36,6 +36,7 @@ #include "vllm/model_executor/models/ltx2_loader.h" #include "vllm/model_executor/models/ltx2_pipeline.h" #include "vllm/model_executor/models/ltx2_retake.h" +#include "vllm/model_executor/models/ltx2_t2a.h" #include "vllm/model_executor/models/ltx2_text_encoder.h" #include "vllm/model_executor/models/ltx2_upsampler.h" #include "vllm/model_executor/models/ltx2_tiling.h" @@ -363,7 +364,7 @@ constexpr char kLtx2DurationHeadPathExtra[] = "duration_head_path"; // they are no longer trusted: the list below is derived from this file on every // run and compared, and the failure prints the replacement to paste in. // READER ANCHORS (derived and gated by test_ltx2_video): -// 781 791 792 854 950 966 968 1046 1071 1176 1217 +// 782 792 793 855 951 967 969 1060 1085 1190 1231 const char* const kKnownLoadExtras[] = { kLtx2AudioPromptEmbedsExtra, kLtx2PipelineKindExtra, kLtx2ModelVersionExtra, kLtx2AllowUnportedExtra, kLtx2MaxPhaseExtra, kLtx2DitConfigPathExtra, @@ -974,8 +975,21 @@ std::unique_ptr Ltx2VideoEngine::Load(const VideoModelParams& p } // ── the video VAE ───────────────────────────────────────────────────────── - if (params.video_vae_path.empty()) Fail("video_vae_path is required"); - { + // + // REQUIRED, EXCEPT ON AN AUDIO-ONLY RECIPE, and the exception is upstream's + // shape rather than a convenience: `T2AOneStagePipeline.__init__` constructs a + // `PromptEncoder`, a `DiffusionStage`, an `AudioDecoder` and a + // `DurationPredictor` (t2a_one_stage.py:68-107) and never calls + // `model_paths.video_vae()`. Demanding one would make a text-to-audio load ask + // for a checkpoint the pipeline cannot use. + // + // Keyed on `recipe.audio_only` rather than on the kind STRING, so the next + // audio-only recipe inherits it instead of silently failing here. Supplying a + // video VAE anyway is accepted and loaded — it costs the caller memory and + // nothing else, and refusing it would break a caller who reuses one params + // object across pipelines. + if (params.video_vae_path.empty() && !im.recipe.audio_only) Fail("video_vae_path is required"); + if (!params.video_vae_path.empty()) { const SafetensorsFile f = SafetensorsFile::Open(params.video_vae_path); const nlohmann::json vae_config = Ltx2ReadCheckpointConfig(f); im.video_cfg = Ltx2ParseConvVideoDecoderConfig(vae_config, &im.video_kind); @@ -1018,7 +1032,7 @@ std::unique_ptr Ltx2VideoEngine::Load(const VideoModelParams& p } } } - if (im.video_cfg.in_channels != im.dit.params.out_channels) { + if (!params.video_vae_path.empty() && im.video_cfg.in_channels != im.dit.params.out_channels) { Fail("the video VAE takes " + std::to_string(im.video_cfg.in_channels) + " latent channels but the DiT emits " + std::to_string(im.dit.params.out_channels)); } @@ -1438,7 +1452,13 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { kv.first == kLtx2RetakeEndTimeExtra || kv.first == kLtx2RetakeFrameRateExtra || kv.first == kLtx2RegenerateVideoExtra || - kv.first == kLtx2RegenerateAudioExtra; + kv.first == kLtx2RegenerateAudioExtra || + kv.first == kLtx2NegativePromptExtra || + kv.first == kLtx2AudioCfgScaleExtra || + kv.first == kLtx2AudioStgScaleExtra || + kv.first == kLtx2AudioRescaleScaleExtra || + kv.first == kLtx2AudioSkipStepExtra || + kv.first == kLtx2AudioStgBlocksExtra; if (!known) { Fail("unknown per-generation extra '" + kv.first + "'. This family defines: " + std::string(kLtx2ImageCrfExtra) + ", " + kLtx2AudioPathExtra + ", " + @@ -1446,7 +1466,52 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { kLtx2GeneratedKeyframesExtra + ", " + kLtx2TemporalRoundsExtra + ", " + kLtx2RetakeStartTimeExtra + ", " + kLtx2RetakeEndTimeExtra + ", " + kLtx2RetakeFrameRateExtra + ", " + kLtx2RegenerateVideoExtra + ", " + - kLtx2RegenerateAudioExtra); + kLtx2RegenerateAudioExtra + ", " + kLtx2NegativePromptExtra + ", " + + kLtx2AudioCfgScaleExtra + ", " + kLtx2AudioStgScaleExtra + ", " + + kLtx2AudioRescaleScaleExtra + ", " + kLtx2AudioSkipStepExtra + ", " + + kLtx2AudioStgBlocksExtra); + } + } + // ── the TEXT-TO-AUDIO knobs belong to ONE pipeline (#1005) ──────────────── + // + // `pipeline_kind` is a LOAD extra, so which pipeline runs is settled before a + // request arrives and this is a decidable question rather than a guess. The + // guard runs in BOTH directions: the six T2A knobs are refused off a + // `t2a_one_stage` engine, and every other per-generation knob is refused ON + // one. Neither is padding. Upstream's `T2AOneStagePipeline.__call__` takes no + // image, no reference, no keyframe and no window (t2a_one_stage.py:109-122), + // and its guider arguments have no counterpart in any other `__call__`, so a + // knob crossing either way would silently do nothing to a render that still + // finishes. + { + const char* const kT2aOnly[] = {kLtx2NegativePromptExtra, kLtx2AudioCfgScaleExtra, + kLtx2AudioStgScaleExtra, kLtx2AudioRescaleScaleExtra, + kLtx2AudioSkipStepExtra, kLtx2AudioStgBlocksExtra}; + const char* const kNotOnT2a[] = {kLtx2ImageCrfExtra, kLtx2AudioPathExtra, + kLtx2AudioStartTimeExtra, kLtx2AudioMaxDurationExtra, + kLtx2GeneratedKeyframesExtra, kLtx2TemporalRoundsExtra, + kLtx2RetakeStartTimeExtra, kLtx2RetakeEndTimeExtra, + kLtx2RetakeFrameRateExtra, kLtx2RegenerateVideoExtra, + kLtx2RegenerateAudioExtra}; + for (const char* key : kT2aOnly) { + if (!im.recipe.audio_only && !VideoExtra(gen.extras, key).empty()) { + Fail("the '" + std::string(key) + + "' extra is text-to-audio's alone (ltx-pipelines utils/args.py:1083-1119, the " + "`default_1_stage_t2a_arg_parser`), and this engine was loaded with pipeline_kind '" + + im.pipeline_kind + + "'. Refused rather than ignored: no other pipeline `__call__` upstream takes a " + "guider argument at all, so accepting it here would report a configured render " + "that ran the recipe's own values"); + } + } + for (const char* key : kNotOnT2a) { + if (im.recipe.audio_only && !VideoExtra(gen.extras, key).empty()) { + Fail("the '" + std::string(key) + + "' extra has no meaning on a text-to-audio render, which produces no picture at " + "all. `T2AOneStagePipeline.__call__` takes a prompt, a negative prompt, a seed, a " + "frame rate, a step count, the audio guider and a frame count " + "(t2a_one_stage.py:109-122) and nothing else. Refused rather than ignored"); + } } } // The two audio WINDOW knobs only mean something alongside a file. Accepting @@ -1752,6 +1817,23 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { im.trace.video_absmax = AbsMax(v); im.trace.audio_absmax = AbsMax(a); } + + // ── TEXT-TO-AUDIO: the render that has no picture (#1005) ───────────────── + // + // `T2AOneStagePipeline.__call__` (t2a_one_stage.py:109-172), in its own + // translation unit (ltx2_t2a.h) mirroring upstream's own file. + // + // THE BRANCH SITS HERE, after the conditioning and before ANY video geometry. + // After, because T2A encodes a prompt exactly as every other pipeline does + // (`:127-135`) and duplicating that chain would give the audio-only arm its own + // copy of the connector composition. Before, because everything below this + // point — the resolution guard, the canvas, the phase loop, the decode — is + // about a video stream this pipeline does not have, and a `t2a` request that + // fell through would be refused by a message about latent grids. + if (im.recipe.audio_only) { + return GenerateAudioOnly(im, gen, audio_context, context_tokens); + } + // ── conditioning on pixels (row LTX25-IMAGE-COND, issue #644) ───────────── // // Upstream this is `ImageConditioner` (ltx-pipelines/utils/blocks.py:936-993, @@ -3384,6 +3466,233 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { return result; } +// ── TEXT-TO-AUDIO (row LTX25-T2A-ONE-STAGE, #1005) ────────────────────────── +// +// `T2AOneStagePipeline.__call__` (ltx-pipelines t2a_one_stage.py:109-172). The +// numerics live in `ltx2_t2a.cpp`, mirroring upstream's own file; this resolves +// the request, encodes the negative prompt and writes the artifact. +// +// PLACED BELOW `Generate` ON PURPOSE. The READER ANCHORS comment near the top of +// this file carries derived LINE NUMBERS into it and is gated by +// `test_ltx2_video`, so a definition inserted above the last anchored line would +// move every anchor under it for a reason that has nothing to do with this row. +VideoResult Ltx2VideoEngine::GenerateAudioOnly(Impl& im, const VideoGenParams& gen, + const float* audio_context, + int64_t context_tokens) { + const Ltx2PipelineRecipe& recipe = im.recipe; + VT_CHECK(recipe.audio_only, "ltx2 t2a: reached the audio-only path on a video recipe"); + + // Upstream's T2A CLI has no --height/--width, and its pipeline substitutes a + // 512x512 PLACEHOLDER whose height and width it documents as unused + // (t2a_one_stage.py:37-40, :163-164). Accepting a resolution here would take a + // number from the caller, ignore it, and return successfully. + if (gen.height > 0 || gen.width > 0) { + Fail("a text-to-audio request cannot carry a width or a height: there is no picture. " + "Upstream passes a 512x512 PLACEHOLDER into the stage and says so in as many words — " + "\"Audio-only generation reads `frames` and `fps` from the pixel shape via " + "`AudioLatentShape.from_video_pixel_shape` (height/width are unused)\" " + "(t2a_one_stage.py:37-40). Accepting one would ignore it and still succeed"); + } + if (!gen.first_frame_path.empty() || !gen.last_frame_path.empty() || + !gen.first_frame_ppm.empty() || !gen.ref_image_paths.empty() || + !gen.ref_video_dir.empty() || !gen.ref_audio_path.empty() || !gen.ref_audio_wav.empty()) { + Fail("a text-to-audio request cannot carry a keyframe, a reference image, a reference clip " + "or a reference waveform. `T2AOneStagePipeline.__call__` takes none of them " + "(t2a_one_stage.py:109-122) and its `DiffusionStage` call passes `video=None` " + "(`:167`), so there is no stream for any of them to condition"); + } + + // `num_frames` / `frame_rate` — the only two fields of the placeholder pixel + // shape T2A reads, and they exist to derive the audio DURATION. + const double fps = recipe.frame_rate; + int64_t frames = gen.num_frames > 1 ? gen.num_frames : recipe.num_frames; + if (gen.duration_seconds > 0.0) { + frames = static_cast(std::llround(gen.duration_seconds * fps)); + } + + // ── the guider (t2a_one_stage.py:196-205) ───────────────────────────────── + // + // The recipe already carries the params table's audio guider with + // `modality_scale` pinned to 1.0; each extra overrides ONE field, exactly as + // one CLI flag does. + Ltx2MultiModalGuiderParams guidance = recipe.phases.front().audio_guidance; + guidance.cfg_scale = ExtraDouble(gen.extras, kLtx2AudioCfgScaleExtra, guidance.cfg_scale); + guidance.stg_scale = ExtraDouble(gen.extras, kLtx2AudioStgScaleExtra, guidance.stg_scale); + guidance.rescale_scale = + ExtraDouble(gen.extras, kLtx2AudioRescaleScaleExtra, guidance.rescale_scale); + guidance.skip_step = ExtraInt(gen.extras, kLtx2AudioSkipStepExtra, guidance.skip_step); + if (guidance.skip_step < 0) { + Fail("'" + std::string(kLtx2AudioSkipStepExtra) + "' is " + + std::to_string(guidance.skip_step) + + "; `should_skip_step` is `step % (skip_step + 1)` (guiders.py:287-291) and a negative " + "value would take the modulus of a non-positive divisor"); + } + { + // `--audio-stg-blocks`, `nargs="*"` (utils/args.py:1107-1113). An extra that + // is PRESENT and empty is upstream's empty list — "perturb nothing" — and is + // kept distinct from an ABSENT extra, which takes the params table's own + // [28]. Collapsing the two would make `audio_stg_blocks=` silently mean + // block 28. + const auto at = gen.extras.find(kLtx2AudioStgBlocksExtra); + if (at != gen.extras.end()) { + guidance.stg_blocks.clear(); + const std::string& raw = at->second; + for (size_t i = 0; i < raw.size();) { + const size_t comma = raw.find(',', i); + const std::string token = raw.substr(i, comma == std::string::npos ? comma : comma - i); + if (!token.empty()) { + try { + guidance.stg_blocks.push_back(std::stoll(token)); + } catch (const std::exception&) { + Fail("'" + std::string(kLtx2AudioStgBlocksExtra) + "' holds '" + token + + "', which is not an integer block index"); + } + } + if (comma == std::string::npos) break; + i = comma + 1; + } + } + } + + // ── the negative conditioning (t2a_one_stage.py:127-135) ────────────────── + // + // Upstream encodes `[prompt, negative_prompt]` in ONE `PromptEncoder` call and + // takes `.audio_encoding` from each. Here the positive half was already + // resolved by `Generate`; this is the second half, through the same + // `Ltx2EncodePromptToConditioning` and the same connector. + // + // ONLY WHEN THE GUIDER ASKS FOR IT. `do_unconditional_generation` is + // `not isclose(cfg_scale, 1.0)` (guiders.py:275-277), so at scale 1.0 there is + // no unconditional forward and encoding a negative prompt would be a wasted + // 12B host-side pass per request. + std::vector negative_audio; + const float* negative_context = nullptr; + if (!guidance.DoUnconditionalGeneration()) { + // Nothing to do: the guidance delta's `uncond_text` term is switched off. + } else if (!im.has_encoder) { + Fail("this text-to-audio request needs an unconditional forward (`cfg_scale` = " + + std::to_string(guidance.cfg_scale) + + "), which needs the NEGATIVE prompt encoded — and no text tower is loaded, so this " + "engine can encode neither prompt. The `" + + std::string(kLtx2AudioPromptEmbedsExtra) + + "' fallback carries ONE conditioning stream and there is no second file for the " + "negative one. Load with encoder_path, or set '" + + std::string(kLtx2AudioCfgScaleExtra) + + "' to 1.0, which turns the unconditional pass off (guiders.py:275-277)"); + } else { + const std::string negative = + VideoExtra(gen.extras, kLtx2NegativePromptExtra, recipe.negative_prompt); + if (negative.empty()) { + Fail("this text-to-audio request needs a negative prompt and neither the '" + + std::string(kLtx2NegativePromptExtra) + + "' extra nor the recipe carries one. An EMPTY negative prompt is not the same as no " + "CFG: it still encodes and still steers, and upstream's CLI always supplies " + "`DEFAULT_NEGATIVE_PROMPT` (utils/args.py:1083-1088)"); + } + vt::Queue text_queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + const Ltx2PromptConditioning encoded = Ltx2EncodePromptToConditioning( + *im.tower, *im.tokenizer, im.gemma_ids, im.caption_projections, im.feature_cfg, negative, + text_queue); + negative_audio = encoded.conditioning.audio; + if (encoded.seq != context_tokens) { + // Upstream's two encodings come from ONE tokenization of a two-element + // list (t2a_one_stage.py:127-133), so they share a padded width by + // construction. A mismatch here means the two ran different geometries and + // the guidance delta would subtract tensors that do not correspond. + Fail("the negative prompt encoded to " + std::to_string(encoded.seq) + + " context rows and the prompt to " + std::to_string(context_tokens) + + "; upstream encodes both in one call and they cannot differ"); + } + if (im.has_connector) { + const Ltx2ConnectorEmbeddings through = + RunConnector(SafetensorsFile::Open(im.params.dit_path), im.video_connector_cfg, + im.audio_connector_cfg, encoded.conditioning.video, + encoded.conditioning.audio, encoded.conditioning.additive_mask, + context_tokens); + negative_audio = through.audio; + } + negative_context = negative_audio.data(); + } + + // ── the render ──────────────────────────────────────────────────────────── + if (im.on_device) { + // REFUSED BY NAME rather than served the host forward behind a device + // handle, which is the substitution this engine's header names as the thing + // that would make every later timing claim false. + // + // WHAT IS *NOT* THE REASON: not the STG perturbation and not the guider. + // `Ltx2DitPerturbation` is a plain argument either forward could take, and + // `Ltx2MultiModalGuidance` runs on host buffers on both arms. What is + // missing is narrower and is a fact about THIS tree: + // `Ltx2DitForwardDevice` dereferences `*video` unconditionally — both + // `PrepareStreamDev` calls take it by reference, and the per-block + // `a.batch = video->batch` reads through it — so a one-stream device forward + // is a rewrite of that function rather than the lifted check the host + // forward needed. Owed by #1005. + Fail("text-to-audio is not served on the accelerator. `Ltx2DitForwardDevice` takes BOTH " + "streams by reference and this pipeline has no video stream to give it " + "(`video=None`, t2a_one_stage.py:167). Refusing rather than running the host forward " + "behind a device handle. Load with device 0."); + } + + EngineNoiseStream noise(gen.has_seed ? gen.seed : static_cast(recipe.num_frames)); + Ltx2T2aRequest req; + req.device = im.device; + req.compute_dtype = im.compute_dtype; + req.dit_params = &im.dit.params; + req.dit_weights = &im.dit.weights; + req.context = audio_context; + req.negative_context = negative_context; + req.context_tokens = context_tokens; + req.num_frames = frames; + req.frame_rate = fps; + req.steps = gen.steps > 0 ? gen.steps : recipe.num_inference_steps; + req.noise = &noise; + req.guidance = guidance; + req.audio_cfg = &im.audio_cfg; + req.audio_weights = &im.audio_weights; + req.vocoder_cfg = &im.vocoder_cfg; + req.vocoder_weights = &im.vocoder_weights; + + const Ltx2T2aResult rendered = Ltx2T2aGenerate(req); + + im.trace.t2a_rendered = true; + im.trace.t2a_video_stream_present = rendered.video_stream_present; + im.trace.t2a_cond_forwards = rendered.cond_forwards; + im.trace.t2a_uncond_forwards = rendered.uncond_forwards; + im.trace.t2a_perturbed_forwards = rendered.perturbed_forwards; + im.trace.t2a_perturbed_blocks = rendered.perturbed_blocks; + im.trace.audio_tokens = rendered.audio_tokens; + im.trace.audio_latent_digest = rendered.latent_digest; + im.trace.audio_latent_absmax = rendered.latent_absmax; + + // ── the artifact ────────────────────────────────────────────────────────── + std::error_code ec; + std::filesystem::create_directories(gen.output_dir, ec); + if (ec) Fail("cannot create " + gen.output_dir + ": " + ec.message()); + + VideoResult result; + // `frame_dir` STAYS EMPTY, and that is the contract rather than an omission: a + // directory naming a frame pattern that matches no file is what a caller + // iterates and finds nothing in. `frame_count == 0` says the same thing, and + // saying it twice is what lets a consumer notice the disagreement if one of + // them is ever filled by accident. + result.frame_count = 0; + result.width = 0; + result.height = 0; + result.fps = 0; + result.audio_path = JoinPath(gen.output_dir, "audio.wav"); + WriteFileBytes(result.audio_path, + MiniMaxH3WriteWav(rendered.waveform, rendered.channels, + rendered.samples_per_channel, rendered.sample_rate)); + result.sample_rate = rendered.sample_rate; + // NO `mux_argv`, and none is composed. The argv this seam builds muxes a frame + // pattern with a soundtrack; over an empty pattern ffmpeg fails, so composing + // one would hand the caller a command that cannot run and call it a result. + im.trace.completed = true; + return result; +} + namespace { std::unique_ptr LoadLtx2VideoFamily(const VideoModelParams& params) { diff --git a/tests/vllm/models/test_ltx2.cpp b/tests/vllm/models/test_ltx2.cpp index 5dd0fc1a9..009c5062f 100644 --- a/tests/vllm/models/test_ltx2.cpp +++ b/tests/vllm/models/test_ltx2.cpp @@ -1325,13 +1325,67 @@ TEST_CASE("ltx2 keyframes: a bf16 embedding view is REFUSED by the f32 forward") CHECK_THROWS(Ltx2DitForward(Cpu(), p, set.weights, &m.video, &m.audio, vt::DType::kF32)); } -TEST_CASE("ltx2 forward: a single-stream model type is REFUSED") { +// REWRITTEN by row LTX25-T2A-ONE-STAGE (#1005), and the rewrite is the point +// rather than an accommodation. +// +// This case used to assert that a single-stream call THROWS, pinning a refusal +// whose stated reason was "LTXModelType.VideoOnly and LTXModelType.AudioOnly +// carry a different weight contract". That reason is about the CHECKPOINT, and +// it does not describe `T2AOneStagePipeline`: upstream loads the ordinary +// AudioVideo file and restricts which keys it reads +// (LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP, model_configurator.py:228-239), +// then calls `LTXModel.forward(video=None, ...)` (t2a_one_stage.py:167). +// +// So the assertion is not widened, it is REPLACED with the contract upstream +// actually has (transformer.py:259-260, "At least one of video or audio must be +// provided") — and the new form is strictly stronger, because it also pins what +// a one-stream call RETURNS. The old one could not tell a served one-stream +// forward from a broken one; both threw. +TEST_CASE("ltx2 forward: ONE stream runs, and both-null is refused") { const Ltx2DitParams p = ReducedParams(Ltx2RopeType::kSplit, false); WeightSet set = BuildWeights(p); Modalities m; BuildModalities(&m, false); - CHECK_THROWS(Ltx2DitForward(Cpu(), p, set.weights, &m.video, nullptr, vt::DType::kF32)); - CHECK_THROWS(Ltx2DitForward(Cpu(), p, set.weights, nullptr, &m.audio, vt::DType::kF32)); + + // Upstream's own refusal, and the only one left. + CHECK_THROWS(Ltx2DitForward(Cpu(), p, set.weights, nullptr, nullptr, vt::DType::kF32)); + + // VIDEO ALONE. `run_a2v` is false because there is no audio state, so the + // video output must still be the full sequence and the audio one EMPTY — + // `Ltx2DitOutputs` carries two vectors and a build that filled both would be + // reporting a stream it never ran. + const vllm::Ltx2DitOutputs v_only = + Ltx2DitForward(Cpu(), p, set.weights, &m.video, nullptr, vt::DType::kF32); + CHECK(v_only.audio.empty()); + REQUIRE(v_only.video.size() == + static_cast(m.video.batch * m.video.tokens * p.out_channels)); + for (const float x : v_only.video) REQUIRE(std::isfinite(x)); + + // AUDIO ALONE — the shape `T2AOneStagePipeline` runs. + const vllm::Ltx2DitOutputs a_only = + Ltx2DitForward(Cpu(), p, set.weights, nullptr, &m.audio, vt::DType::kF32); + CHECK(a_only.video.empty()); + REQUIRE(a_only.audio.size() == + static_cast(m.audio.batch * m.audio.tokens * p.audio_out_channels)); + for (const float x : a_only.audio) REQUIRE(std::isfinite(x)); + + // AND ONE STREAM IS NOT THE JOINT FORWARD WITH THE OTHER IGNORED. This is the + // assertion the old case had no way to make, and it is the whole reason + // `video = nullptr` is not `video->enabled = false`: upstream's `run_v2a` is + // `run_ax and (video is not None and vx.numel() > 0)` (transformer.py:269), so + // a present video stream — enabled or not — still feeds video->audio cross + // attention. If these were equal, the cross-modal path would be dead on the + // joint arm instead. + const vllm::Ltx2DitOutputs joint = + Ltx2DitForward(Cpu(), p, set.weights, &m.video, &m.audio, vt::DType::kF32); + REQUIRE(joint.audio.size() == a_only.audio.size()); + bool audio_differs = false; + for (size_t i = 0; i < joint.audio.size(); ++i) { + if (joint.audio[i] != a_only.audio[i]) audio_differs = true; + } + CHECK_MESSAGE(audio_differs, + "the audio-only forward equals the joint one, so video->audio cross attention " + "contributed nothing on the joint arm"); } // --------------------------------------------------------------------------- diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index bf25112ce..c0cdee17e 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -4794,3 +4794,421 @@ TEST_CASE("ltx2 retake: the wrong recipe refuses, and the reference arm still do } } } + +// ─── TEXT-TO-AUDIO (row LTX25-T2A-ONE-STAGE, issue #1005) ──────────────────── +// +// `T2AOneStagePipeline` (ltx-pipelines t2a_one_stage.py:43, `__call__` at :109) +// at Lightricks/LTX-2 @ fd4ded7f. UPSTREAM SHIPS NO TESTS at that pin — `find +// /home/mudler/_git/LTX-2 -name 'test_*.py'` returns 0 — so there is nothing to +// port, and what follows pins upstream's BEHAVIOURS against `file:line` anchors +// instead. At least one assertion in each refusal case is tied to a LOCAL fact, +// because a case that asserts only upstream symbol names cannot see a refusal +// whose claim about THIS tree has gone stale. +namespace { + +// A t2a engine on the shipped fixture. NO `video_vae_path`, which is the load +// half of the row: upstream's `T2AOneStagePipeline.__init__` never calls +// `model_paths.video_vae()` (t2a_one_stage.py:68-107). +vllm::multimodal::VideoModelParams T2aParams(const ltx2_fixture::Paths& paths) { + vllm::multimodal::VideoModelParams mp; + mp.dit_path = paths.dit; + mp.audio_vae_path = paths.audio_vae; + mp.encoder_path = paths.encoder; + mp.extras[vllm::multimodal::kLtx2EncoderConfigPathExtra] = paths.encoder_config; + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = "t2a_one_stage"; + mp.device = 0; + return mp; +} + +vllm::multimodal::VideoGenParams T2aGen(const std::string& out_dir, const std::string& prompt) { + vllm::multimodal::VideoGenParams gen; + gen.prompt = prompt; + gen.num_frames = 25; + gen.steps = 2; // two sigma intervals is enough to exercise the loop + gen.has_seed = true; + gen.seed = 11; + gen.output_dir = out_dir; + // The reduced DiT has TWO blocks (ltx2_video_fixture.h `ReducedDitParams`), so + // the params table's own `stg_blocks = [28]` is out of range here. Named + // explicitly rather than by turning STG off, because the default-block refusal + // is its own case below and this one is about the render. + gen.extras[vllm::multimodal::kLtx2AudioStgBlocksExtra] = "1"; + // The recipe's own default negative prompt is upstream's + // `DEFAULT_NEGATIVE_PROMPT` — an English sentence — and this fixture's + // tokenizer carries a three-token vocabulary. Overriding it here is what the + // `--negative-prompt` flag is for (utils/args.py:1083-1088), and the DEFAULT's + // reachability is asserted separately on the recipe rather than by pushing an + // out-of-vocabulary string through a reduced tokenizer. + gen.extras[vllm::multimodal::kLtx2NegativePromptExtra] = "c b a"; + return gen; +} + +} // namespace + +TEST_CASE("ltx2 t2a: an audio-only render returns a waveform and NO picture") { + Workspace ws; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(T2aParams(ws.paths)); + REQUIRE(engine != nullptr); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + CHECK(ltx->pipeline_kind() == "t2a_one_stage"); + + const std::string out = ws.root + "/t2a"; + const vllm::multimodal::VideoResult result = engine->Generate(T2aGen(out, "a b c")); + + // 1. NO PICTURE, said three ways, because each catches a different build. + // `frame_count` catches a render that produced frames; the empty + // `frame_dir` and `mux_argv` catch one that produced none and still handed + // the caller a directory and an ffmpeg command over a pattern matching no + // file. + CHECK(result.frame_count == 0); + CHECK(result.frame_dir.empty()); + CHECK(result.mux_argv.empty()); + CHECK(result.mux_output_path.empty()); + CHECK(result.width == 0); + CHECK(result.height == 0); + // ...AND NO FRAME ON DISK. This is the half the fields cannot make: a build + // that wrote `frame_000000.ppm` and reported zero passes every check above. + { + std::ifstream frame(out + "/frame_000000.ppm", std::ios::binary); + CHECK_MESSAGE(!frame.good(), "an audio-only render wrote a frame"); + } + + // 2. THERE IS SOUND, at the vocoder's own output rate, and it is not silence. + // The lower bound is the assertion a size or a rate cannot make: a + // zero-initialized decode produces a perfectly well-formed WAV of exactly + // the right length. + CHECK(result.sample_rate == 48000); + const std::string wav = ReadAll(result.audio_path); + REQUIRE(wav.size() > 44); + CHECK(wav.compare(0, 4, "RIFF") == 0); + { + int peak = 0; + for (size_t i = 44; i + 1 < wav.size(); i += 2) { + int16_t s = 0; + std::memcpy(&s, wav.data() + i, sizeof(s)); + peak = std::max(peak, s < 0 ? -static_cast(s) : static_cast(s)); + } + CHECK_MESSAGE(peak > 0, "the rendered waveform is digital silence"); + } + + const vllm::multimodal::Ltx2ConditioningTrace trace = ltx->last_conditioning(); + CHECK(trace.completed); + CHECK(trace.t2a_rendered); + + // 3. NO VIDEO STREAM EVER REACHED THE DiT. This is the whole reason the field + // exists: upstream's `run_v2a` tests PRESENCE, not `enabled` + // (transformer.py:269), so a build that handed the forward a + // present-but-disabled video stream would feed video->audio cross attention + // from a latent this pipeline never meant to exist — and would return a + // waveform of exactly the right length, channel count and sample rate. + CHECK_FALSE(trace.t2a_video_stream_present); + + // 4. THE GUIDER RAN, arm by arm. The counters are incremented at the forward, + // so this is a statement about the passes that were issued rather than + // about the parameters that were meant to drive them. + CHECK(trace.t2a_cond_forwards == 2); + CHECK(trace.t2a_uncond_forwards == 2); + CHECK(trace.t2a_perturbed_forwards == 2); + // ...and STG perturbed the block that was ASKED for. A count alone cannot + // tell block 1 from block 0, and which block is perturbed is the whole of + // STG. + REQUIRE(trace.t2a_perturbed_blocks.size() == 1); + CHECK(trace.t2a_perturbed_blocks[0] == 1); + + // 5. The latent is populated. A digest alone is stable across a collapse to + // zeros; the absmax is the bound that is not. + CHECK(trace.audio_tokens > 0); + CHECK(trace.audio_latent_absmax > 1e-6); +} + +TEST_CASE("ltx2 t2a: the guidance ARMS are separable, and each one moves the render") { + Workspace ws; + const vllm::multimodal::VideoModelParams mp = T2aParams(ws.paths); + + struct Out { + vllm::multimodal::Ltx2ConditioningTrace trace; + std::string wav; + }; + const auto render = [&](const std::string& tag, + const std::map& overrides) { + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + REQUIRE(engine != nullptr); + vllm::multimodal::VideoGenParams gen = T2aGen(ws.root + "/" + tag, "a b c"); + for (const auto& kv : overrides) gen.extras[kv.first] = kv.second; + const vllm::multimodal::VideoResult result = engine->Generate(gen); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + return Out{ltx->last_conditioning(), ReadAll(result.audio_path)}; + }; + + const Out full = render("g_full", {}); + // `cfg_scale = 1.0` is `math.isclose(cfg_scale, 1.0)` — upstream's OWN + // predicate for "no unconditional generation" (guiders.py:275-277), and NOT an + // exact `!= 1.0` comparison. + const Out no_cfg = render("g_nocfg", {{vllm::multimodal::kLtx2AudioCfgScaleExtra, "1.0"}}); + const Out no_stg = render("g_nostg", {{vllm::multimodal::kLtx2AudioStgScaleExtra, "0.0"}}); + + // Each arm turns off exactly its own forward, and leaves the others alone. + CHECK(no_cfg.trace.t2a_uncond_forwards == 0); + CHECK(no_cfg.trace.t2a_perturbed_forwards == full.trace.t2a_perturbed_forwards); + CHECK(no_stg.trace.t2a_perturbed_forwards == 0); + CHECK(no_stg.trace.t2a_uncond_forwards == full.trace.t2a_uncond_forwards); + CHECK(no_cfg.trace.t2a_cond_forwards == full.trace.t2a_cond_forwards); + + // AND EACH ONE CHANGES THE RENDER. Without this, a build that issued the extra + // forwards and then discarded them would pass every counter above — which is + // exactly the "recorded value is not a reached one" failure this campaign has + // already paid for once. + CHECK(full.wav.size() == no_cfg.wav.size()); + CHECK(full.wav != no_cfg.wav); + CHECK(full.wav != no_stg.wav); + CHECK(no_cfg.wav != no_stg.wav); + + // The STG DELTA depends on WHICH block is perturbed. Two builds that perturb + // different blocks issue the same three forwards and differ only here, so a + // port that ignored `stg_blocks` and perturbed everything (or nothing) would + // pass every assertion above. + const Out block0 = render("g_b0", {{vllm::multimodal::kLtx2AudioStgBlocksExtra, "0"}}); + REQUIRE(block0.trace.t2a_perturbed_blocks.size() == 1); + CHECK(block0.trace.t2a_perturbed_blocks[0] == 0); + CHECK(block0.wav != full.wav); +} + +TEST_CASE("ltx2 t2a: the refusals name what is missing, and each is checked HERE") { + Workspace ws; + const vllm::multimodal::VideoModelParams mp = T2aParams(ws.paths); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + REQUIRE(engine != nullptr); + + const auto refuses = [&](vllm::multimodal::VideoGenParams gen, + const char* needle) -> std::string { + // `needle` is a `const char*`, and doctest stringifies a bare `char*` as a + // BOOL — a failure would print `true` instead of the string that was looked + // for. Bound to a std::string before it reaches any doctest macro. + const std::string want(needle); + INFO("needle = " << want); + try { + (void)engine->Generate(gen); + FAIL_CHECK("expected a refusal naming: " << want); + return std::string(); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO("refusal = " << msg); + CHECK_MESSAGE(msg.find(want) != std::string::npos, "the refusal did not name the needle"); + return msg; + } + }; + + SUBCASE("a resolution is refused rather than accepted and ignored") { + vllm::multimodal::VideoGenParams gen = T2aGen(ws.root + "/r_res", "a b c"); + gen.width = 64; + gen.height = 64; + refuses(gen, "height/width are unused"); + } + + SUBCASE("the params table's own STG block is out of range on THIS DiT") { + // The LOCAL fact, and it is what makes this case able to see staleness. The + // fixture's DiT has two blocks; upstream's default `stg_blocks` is [28]. The + // refusal must name the range it checked against, so a build that silently + // clamped or ignored the index would not produce this message. + vllm::multimodal::VideoGenParams gen = T2aGen(ws.root + "/r_stg", "a b c"); + gen.extras.erase(vllm::multimodal::kLtx2AudioStgBlocksExtra); + const std::string msg = refuses(gen, "STG block index 28 is outside [0, "); + // Derived from the tree rather than restated: the range in the message is + // the DiT the engine actually loaded, not a literal this test also knows. + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + CHECK(msg.find("[0, " + std::to_string(ltx->dit_params().num_layers) + ")") != + std::string::npos); + } + + SUBCASE("a perturbed pass over NO block is refused, not run as a no-op") { + vllm::multimodal::VideoGenParams gen = T2aGen(ws.root + "/r_empty", "a b c"); + gen.extras[vllm::multimodal::kLtx2AudioStgBlocksExtra] = ""; + refuses(gen, "the STG delta would be exactly zero"); + } + + SUBCASE("isolated-modality guidance has no second modality to run over") { + // Not reachable through an extra by design — there is no `modality_scale` + // knob — so this asserts the RECIPE pinned it, which is the thing that keeps + // the refusal unreachable. `Ltx2DetectPipelineParams("2.5")` carries 3.0. + const vllm::Ltx2PipelineRecipe t2a = vllm::ResolveLtx2PipelineRecipe("t2a_one_stage", "2.5"); + REQUIRE(t2a.audio_only); + REQUIRE(t2a.phases.size() == 1); + CHECK(t2a.phases[0].audio_guidance.modality_scale == 1.0); + CHECK(vllm::Ltx2DetectPipelineParams("2.5").audio_guider.modality_scale == 3.0); + } + + SUBCASE("a video-only knob on a t2a engine is refused rather than ignored") { + vllm::multimodal::VideoGenParams gen = T2aGen(ws.root + "/r_crf", "a b c"); + gen.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "0"; + refuses(gen, "no meaning on a text-to-audio render"); + } + + SUBCASE("a keyframe has no stream to condition") { + vllm::multimodal::VideoGenParams gen = T2aGen(ws.root + "/r_kf", "a b c"); + gen.first_frame_ppm = "P6\n1 1\n255\n\x01\x02\x03"; + refuses(gen, "`video=None`"); + } +} + +TEST_CASE("ltx2 t2a: a t2a-only knob is refused on the video pipelines") { + Workspace ws; + const vllm::multimodal::VideoModelParams mp = EncoderParams(ws.paths); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + REQUIRE(engine != nullptr); + vllm::multimodal::VideoGenParams gen = PromptedGen(ws.root + "/x", "a b c"); + gen.extras[vllm::multimodal::kLtx2AudioCfgScaleExtra] = "5.0"; + try { + (void)engine->Generate(gen); + FAIL_CHECK("a t2a guider knob must be refused on a video pipeline"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("text-to-audio's alone") != std::string::npos); + // It names the pipeline the ENGINE resolved, so a message that guessed + // would say something else. + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + CHECK(msg.find("'" + ltx->pipeline_kind() + "'") != std::string::npos); + } +} + +TEST_CASE("ltx2 t2a: the DiT forward runs ONE stream, and the old guard's reason was wrong") { + // The lifted refusal, checked against a LOCAL fact rather than against + // upstream symbol names alone. `Ltx2DitForward` used to demand BOTH streams + // and blamed the AudioOnly weight contract; the contract is not what blocked + // it, and the way to see that is that the AV weights this fixture writes are + // enough to run the audio stream by itself. + Workspace ws; + const vllm::SafetensorsFile dit_file = vllm::SafetensorsFile::Open(ws.paths.dit); + vllm::Ltx2DitLoadOptions options; + options.widen_to_f32 = true; // `Ltx2DitForward` is f32 by declaration + const vllm::Ltx2DitCheckpoint ckpt = vllm::Ltx2LoadDitFromSafetensors(dit_file, options); + + const int64_t tokens = 3; + // VARYING PER TOKEN, and that is load bearing rather than tidy. A latent whose + // rows are all equal makes self-attention return a weighted average of + // identical values — which is exactly the value projection — so the STG + // perturbation below would be a numeric no-op and the case would report + // "the perturbation changed nothing" about a build that applies it correctly. + // Measured: with a constant 0.25 fill this assertion failed on the working + // implementation. + std::vector latent(static_cast(tokens * ckpt.params.audio_in_channels)); + for (size_t i = 0; i < latent.size(); ++i) { + latent[i] = 0.25F + 0.01F * static_cast(i % 7) - 0.02F * static_cast(i % 3); + } + std::vector timesteps(static_cast(tokens), 0.5F); + std::vector positions(static_cast(tokens * 2)); + for (int64_t t = 0; t < tokens; ++t) { + positions[static_cast(t * 2)] = static_cast(t) * 0.04; + positions[static_cast(t * 2 + 1)] = static_cast(t + 1) * 0.04; + } + const int64_t ctx = 4; + std::vector context( + static_cast(ctx * ckpt.params.audio_cross_attention_dim), 0.1F); + const float sigma = 0.5F; + + vllm::Ltx2ModalityInput ain; + ain.tokens = tokens; + ain.context_tokens = ctx; + ain.latent = latent.data(); + ain.timesteps = timesteps.data(); + ain.sigma = σ + ain.positions = positions.data(); + ain.context = context.data(); + + const vllm::Ltx2DitOutputs out = vllm::Ltx2DitForward( + vt::Device{}, ckpt.params, ckpt.weights, /*video=*/nullptr, &ain, vt::DType::kF32); + CHECK(out.video.empty()); + REQUIRE(out.audio.size() == static_cast(tokens * ckpt.params.audio_out_channels)); + for (const float v : out.audio) REQUIRE(std::isfinite(v)); + + // BOTH null is still refused, which is upstream's own refusal + // (transformer.py:259-260) rather than a leftover of the old one. + CHECK_THROWS(vllm::Ltx2DitForward(vt::Device{}, ckpt.params, ckpt.weights, nullptr, nullptr, + vt::DType::kF32)); + + // And the STG perturbation MOVES the forward. Without this the flag would be + // a field nothing reads: a build that plumbed it and never applied it returns + // the same finite tensor of the same shape. + vllm::Ltx2DitPerturbation p; + p.audio_self_attn.assign(static_cast(ckpt.params.num_layers), 0); + p.audio_self_attn[0] = 1; + const vllm::Ltx2DitOutputs perturbed = + vllm::Ltx2DitForward(vt::Device{}, ckpt.params, ckpt.weights, nullptr, &ain, + vt::DType::kF32, /*cache=*/nullptr, &p); + REQUIRE(perturbed.audio.size() == out.audio.size()); + bool moved = false; + for (size_t i = 0; i < out.audio.size(); ++i) { + if (perturbed.audio[i] != out.audio[i]) moved = true; + } + CHECK_MESSAGE(moved, "the STG perturbation changed nothing"); + + // A vector of the wrong length is refused rather than indexed defensively. + vllm::Ltx2DitPerturbation bad; + bad.audio_self_attn.assign(static_cast(ckpt.params.num_layers + 1), 0); + CHECK_THROWS(vllm::Ltx2DitForward(vt::Device{}, ckpt.params, ckpt.weights, nullptr, &ain, + vt::DType::kF32, /*cache=*/nullptr, &bad)); +} + +TEST_CASE("ltx2 t2a: the schedule starts at exactly 1.0") { + // WHY THIS EXISTS, and it is a mutation result rather than a tidiness rule. A + // mutation that scaled the initial latent by `sigmas[0]` — the thing a reader + // coming from another flow-matching sampler expects to see — SURVIVED the + // focused gate at 6 cases / 484 assertions / exit 0. It survived because it is + // an IDENTITY, not because the gate is blind: `LTX2Scheduler` starts at + // `linspace(1, 0, steps + 1)[0] == 1`, the shift map sends 1 to exactly 1 + // (schedulers.py:41-45) and the stretch sends it to `1 - (1 - 1)/scale`, again + // exactly 1 (`:47-55`). + // + // Pinning the identity is what turns "a mutation survived" into a checked + // fact. If upstream ever moves the first sigma off 1, this fires and the two + // forms stop agreeing. + // NOT `steps = 1`, and the exclusion is upstream's arithmetic rather than a + // convenience. At one step the non-zero sigma list is `[1.0]`, so + // `one_minus_z` is `[0.0]`, `scale_factor = 0 / (1 - terminal)` is 0, and the + // stretch computes `1 - 0/0` — NaN, on both sides (schedulers.py:49-54). + // Measured here: `Ltx2SigmaSchedule(1, 0).front()` is `-nan`. Pinning it would + // be pinning a division by zero as if it were a value; a one-step schedule is + // a separate question and is recorded in the row spec rather than asserted. + for (const int64_t steps : {2, 4, 30, 40}) { + INFO("steps = " << steps); + const std::vector sigmas = vllm::Ltx2SigmaSchedule(steps, /*tokens=*/0); + REQUIRE(sigmas.size() == static_cast(steps) + 1); + CHECK(sigmas.front() == 1.0F); + CHECK(sigmas.back() == 0.0F); + } +} + +TEST_CASE("ltx2 t2a: the one_stage recipes noise their initial latent (#1013)") { + // A one_stage render used to start from ZEROS: `OneStagePhase` left + // `noise_scale` at the struct default of 0.0, and `Ltx2GaussianNoise` is + // `latent + noise_scale * (noise - latent)`, so at 0.0 the state stays exactly + // as `create_initial_state` wrote it. Upstream's `ModalitySpec.noise_scale` + // defaults to 1.0 (ltx-pipelines/utils/types.py:110) and + // `TI2VidOneStagePipeline.__call__` constructs both specs without it + // (ti2vid_one_stage.py:233-239). + // + // Gated on the RECIPE rather than on a render, because the value is what the + // engine reads and a render's own noise is not separable from it by eye. + for (const char* version : {"2", "2.3", "2.4", "2.5"}) { + INFO("version = ", version); + const vllm::Ltx2PipelineRecipe one = vllm::ResolveLtx2PipelineRecipe("one_stage", version); + REQUIRE(one.phases.size() == 1); + CHECK(one.phases[0].noise_scale == 1.0); + // And the t2a rows inherit it, which is the reason they are built FROM the + // one_stage recipe rather than beside it. + const vllm::Ltx2PipelineRecipe t2a = + vllm::ResolveLtx2PipelineRecipe("t2a_one_stage", version); + REQUIRE(t2a.phases.size() == 1); + CHECK(t2a.phases[0].noise_scale == 1.0); + CHECK(t2a.audio_only); + CHECK_FALSE(one.audio_only); + } +} From 668fcfbddc4a27d9485779dc5b64fffd0d1dd77b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 16 Aug 2026 12:10:00 +0000 Subject: [PATCH 3/9] record(#1013): the one_stage zero-latent bug gets an index row, because it was fixed in flow and not by its own row AGENTS.md requires an issue to be linked from three places that agree: the index, the row's spec and the pull request body. #1013 was FILED and FIXED inside row LTX25-T2A-ONE-STAGE, so the spec carries it in section 7b and the pull request body names it; this is the third place. The row it names is the row that fixed it rather than a dash, because `scripts/check-agent-record.py` counts a row that names neither an owner nor a spec `## Owed` entry, and this one has an owner. The `dmd2` half is the part that is NOT fixed, and that half is listed under `## Owed` in the same spec rather than being implied by silence. Both repo-local anchors in the row are SHA-anchored to `332aed738`, because the row describes the tree BEFORE the fix and this branch edits that file. An index row cannot be corrected after it lands. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 270a3be12..219e5270d 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -271,3 +271,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#987](https://github.com/mudler/vllm.cpp/issues/987) | `LTX25-RETAKE` | Two `ltx-2.5` refusal messages state reasons that are no longer true. (a) `src/vllm/multimodal/ltx2_video.cpp:1608 @ 0e1bee42f` says "nothing reads `ref_video_dir` at all", and MiniMax-H3 has always consumed the directory in full (`ReadReferenceClipChw`, `src/vllm/multimodal/minimax_h3_video.cpp:135 @ 0e1bee42f`, called at `:650`); [#975](https://github.com/mudler/vllm.cpp/issues/975) inherited the wider claim from this message. The claim that holds is narrower: the LTX-2.5 engine never reads the directory's CONTENTS. (b) `ltx2_video.cpp:1636-1638 @ 0e1bee42f` says "there is no AUDIO_VAE_ENCODER key filter", and `c2019b0e3` landed `Ltx2AudioVaeEncoderKeyRules()` (`include/vllm/model_executor/models/ltx2_audio_input.h:73 @ 0e1bee42f`) with a live call through `Ltx2EncodeAudioToLatent`. Both rewritten in the `WHAT IS *NOT* THE REASON` shape in the same flow, with one assertion tied to the LOCAL fact that the LTX side now reads the directory | bug | | [#1005](https://github.com/mudler/vllm.cpp/issues/1005) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 text-to-audio (`T2AOneStagePipeline`, `t2a_one_stage.py:43`, `__call__` at `:109` @ `fd4ded7f`) is absent, and the three blockers are not the ones a reader would guess. (a) `Ltx2DitForward` refuses a one-stream call at `src/vllm/model_executor/models/ltx2_dit.cpp:765 @ 332aed738`, citing a weight contract that describes a checkpoint T2A never loads — upstream reads the ordinary AudioVideo FILE through `LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP` (`model_configurator.py:228-239`). (b) The same message advises `enabled=false` as the substitute, and it is NOT: `run_v2a = run_ax and (video is not None and vx.numel() > 0)` (`transformer.py:269`) tests PRESENCE, not `enabled`, so a disabled-but-present video stream still feeds v2a cross attention and still returns a finished waveform; our port mirrors that polarity at `src/vllm/model_executor/models/ltx2_dit.cpp:251 @ 332aed738`. (c) The engine has NO guided denoiser at all — `git grep -n 'guid\\|cfg_scale' src/vllm/multimodal/ltx2_video.cpp` is 0 against 66 for `ltx2` in the same file as the control — while T2A defaults to `cfg_scale=7.0` and `stg_scale=1.0`, i.e. THREE forwards per step. Spec [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | feature | | [#995](https://github.com/mudler/vllm.cpp/issues/995) | `ENG-EXPERT-STREAM` | `check-env-doc` and `test_check_env_doc` are RED on `origin/main`, so every branch cut from it inherits a preflight failure its own diff did not cause: `VT_MOE_EXPERT_STREAM`, `VT_MOE_EXPERT_STREAM_SLOTS` and `VT_MOE_EXPERT_STREAM_SLOT_BYTES` are read from `src/vllm/model_executor/models/qwen3_5.cpp` (`:5145`, `:5195`, `:5189` @ `4496ef196`) and appear in neither `docs/ENVIRONMENT.md` nor `scripts/env-doc-allowlist.txt`. They arrived with the `ENG-EXPERT-STREAM` W4 wiring commit `3005447f8` ([#993](https://github.com/mudler/vllm.cpp/pull/993)). Same shape as [#965](https://github.com/mudler/vllm.cpp/issues/965)/[#968](https://github.com/mudler/vllm.cpp/issues/968): a pre-existing red presenting to each author in turn as their own. Fixed by DOCUMENTING all three in `docs/ENVIRONMENT.md`, not by allowlisting them — a knob that changes the host memory a MoE decode reserves and disables the default-on grouped-MoE path is a deployment surface, not a kernel-internal micro-tuning switch | bug +| [#1013](https://github.com/mudler/vllm.cpp/issues/1013) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 `one_stage` denoised from ZEROS. `OneStagePhase` (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1066 @ 332aed738`) left `Ltx2PhaseRecipe::noise_scale` at the struct default of 0.0, and 0.0 is not "no extra noise": `Ltx2GaussianNoise` is `latent + noise_scale * (noise - latent)` (`:218 @ 332aed738`), so the state stayed exactly as `create_initial_state` wrote it, which with no initial latent is all zeros. Upstream `ModalitySpec.noise_scale` defaults to 1.0 (`ltx-pipelines/utils/types.py:110 @ fd4ded7f`) and `TI2VidOneStagePipeline.__call__` constructs BOTH specs without it (`ti2vid_one_stage.py:233-239`); the two neighbouring recipes set it explicitly, which is what made the omission legible. No gate saw it because every end-to-end test loads `distilled_two_stage`, and a zero-initialized denoise still returns a finite clip of the right size, frame count and sample rate. FOUND and FIXED in flow by row `LTX25-T2A-ONE-STAGE`, whose `t2a_one_stage` rows are built FROM `OneStageRecipe` and would have inherited it. `dmd2` leaves the same field at 0.0 and is deliberately NOT corrected by analogy (its source is vLLM-Omni, not checked out here); that half is listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | From 1d4a507c16dec09dba062b94bdf1129126bb7b01 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 16 Aug 2026 12:24:55 +0000 Subject: [PATCH 4/9] fix(LTX25-T2A-ONE-STAGE): a SKIPPED guidance step runs no forward at all, and this port ran one (#1005) `should_skip_step` (`guiders.py:287-291`) does not mean "skip the guidance and keep the conditional prediction", which is how the first draft of `Ltx2T2aGenerate` read it. Upstream returns `DenoisedLatentResult.result_or_none(denoised=last_denoised_audio)` (`utils/denoisers.py:85-91`) BEFORE it assembles a single pass, so a skipped step runs **no DiT forward at all** and reuses the previous step's denoised prediction. The difference is a whole extra forward per skipped step and a different trajectory, on a render that finishes either way. Nothing about the output can see it: the waveform is the right length, the right channel count and the right sample rate on both. The FORWARD COUNT is the only thing that separates them, and the new case asserts it with a control that runs the same request unskipped. `step == 0` can never skip, because `0 % (skip_step + 1)` is 0, so `last_denoised` is always populated when the branch is taken. The guard is kept anyway and refuses rather than reading an empty vector: "the arithmetic makes it impossible" is exactly the reasoning a later change to `ShouldSkipStep` would invalidate silently. `audio_skip_step` defaults to 0 and never skips, so this was reachable only from an explicit request. It is fixed rather than refused because the reuse is four lines and refusing a knob upstream serves is a worse answer than serving it. The mutation is recorded as M9 in the row's spec, and it is a mutation for a defect this port ACTUALLY SHIPPED rather than an invented one. The whole table is re-measured at this tree (8 cases / 505 assertions unmutated) rather than carried forward from the earlier six-case run, and the spec now records that `git diff --stat` reported 45-47 lines on four rows because it measures against `HEAD` rather than against the pre-mutation working tree. That number is kept with the explanation instead of being replaced by a prettier one. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-t2a-one-stage.md | 67 +++++++++++++++------ docs/FEATURES.md | 2 +- src/vllm/model_executor/models/ltx2_t2a.cpp | 44 +++++++++++--- tests/vllm/multimodal/test_ltx2_video.cpp | 47 +++++++++++++++ 4 files changed, 133 insertions(+), 27 deletions(-) diff --git a/.agents/specs/ltx25-t2a-one-stage.md b/.agents/specs/ltx25-t2a-one-stage.md index a4d1af061..96bdfc90b 100644 --- a/.agents/specs/ltx25-t2a-one-stage.md +++ b/.agents/specs/ltx25-t2a-one-stage.md @@ -136,8 +136,8 @@ finished waveform. Our port mirrors that polarity exactly at `video = nullptr` is the only shape that reproduces `LTXModel.forward(video=None, ...)` (`model.py:492-538`, the `video_args = ... if video is not None else None` -at `:501`). It is also what makes `_init_preprocessors`' AudioOnly arm -(`model.py:350-365`, a plain `TransformerArgsPreprocessor` with no +at `:505`). It is also what makes `_init_preprocessors`' AudioOnly arm +(`model.py:351-365`, a plain `TransformerArgsPreprocessor` with no `cross_scale_shift_adaln` and no `cross_gate_adaln`) equivalent to our `have_both == false` path, which passes `other = nullptr` into `PrepareStream` and therefore builds neither. @@ -248,10 +248,11 @@ Per step, mirroring `FactoryGuidedDenoiser` over `MultiModalGuider.calculate` | `uncond_perturbed` | `DoPerturbedGeneration()` | positive context, audio self-attn perturbed on `stg_blocks` | | `uncond_modality` | `DoIsolatedModalityGeneration()` | — **refused by name**, see §7 | -`ShouldSkipStep(step)` (`guiders.py:287-291`) short-circuits to the plain -conditional prediction, and `skip_step` defaults to 0 so it never fires on the -default path — which is why the branch is written to be reachable from the extra -rather than left implicit. +`ShouldSkipStep(step)` (`guiders.py:287-291`) runs NO forward at all and reuses +the previous step's denoised prediction (`utils/denoisers.py:85-91`), which is +not the same as "skip the guidance and keep the conditional pass" — see M9 in §5. +`skip_step` defaults to 0, so it never fires on the default path and is reachable +only from the extra. The combination is `Ltx2MultiModalGuidance`, which already carries the two details a re-derivation gets wrong: the `(scale - 1)` polarity on CFG and modality @@ -404,26 +405,54 @@ to a green SUCCESS. ### What the mutation pass actually found -Focused gate `./build/tests/test_ltx2_video "--test-case=*t2a*"`, unmutated at -7 cases / 496 assertions / exit 0. Each mutation applied to ONE file, rebuilt, -run, restored in a `finally` and the restore verified by **sha256** rather than -assumed. The harness rebuilds the restored tree before anything else measures it. +Focused gate `./build/tests/test_ltx2_video "--test-case=*t2a*"`. Each mutation +applied to ONE file, rebuilt, run, restored in a `finally` and the restore +verified by **sha256** rather than assumed. The harness rebuilds the restored +tree before anything else measures it. + +**Measured at 8 cases / 505 assertions / exit 0 unmutated**, on the tree this +pull request ships. The table was re-run at that tree rather than carried forward +from the earlier 6-case measurement, because a count carried forward while the +suite grows is exactly how this campaign's previous mutation records went stale. | Mutation | `git diff --stat` | BUILT | exit | verdict | |---|---|---|---|---| -| M1 delete the production call site (the reachability mutation) | `ltx2_video.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED, 3 of 6 cases failed | -| M2 hand the forward a present-but-DISABLED video stream instead of `nullptr` | `ltx2_t2a.cpp \| 3 ++-` | YES (0 errors) | 1 | DETECTED | -| M3 never run the unconditional forward | `ltx2_t2a.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED | -| M4 ignore `stg_blocks` and perturb EVERY block | `ltx2_t2a.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED | -| M5 `all_perturbed` falls through to ordinary attention | `ltx2.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED | -| M6 revert the `one_stage` `noise_scale` to the struct default (#1013) | `ltx2_pipeline.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED | -| M7 scale the initial latent by `sigmas[0]` | `ltx2_t2a.cpp \| 1 +` | YES (0 errors) | 0 | **SURVIVED** — see below | -| M8 write a frame on the audio-only path | `ltx2_video.cpp \| 1 +` | YES (0 errors) | 1 | DETECTED | +| M1 delete the production call site (the reachability mutation) | `ltx2_video.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED, 4 of 8 cases red | +| M2 hand the forward a present-but-DISABLED video stream instead of `nullptr` | `ltx2_t2a.cpp` (see note) | YES (0 errors) | 1 | DETECTED, 3 red | +| M3 never run the unconditional forward | `ltx2_t2a.cpp` (see note) | YES (0 errors) | 1 | DETECTED, 2 red | +| M4 ignore `stg_blocks` and perturb EVERY block | `ltx2_t2a.cpp` (see note) | YES (0 errors) | 1 | DETECTED, 3 red | +| M5 `all_perturbed` falls through to ordinary attention | `ltx2.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED, 2 red | +| M6 revert the `one_stage` `noise_scale` to the struct default (#1013) | `ltx2_pipeline.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED, 1 red | +| M7 scale the initial latent by `sigmas[0]` | `ltx2_t2a.cpp` (see note) | YES (0 errors) | 0 | **SURVIVED** — see below | +| M8 write a frame on the audio-only path | `ltx2_video.cpp \| 1 +` | YES (0 errors) | 1 | DETECTED, 1 red | +| M9 a skipped step RECOMPUTES the conditional forward instead of reusing | `ltx2_t2a.cpp \| 39 +++---` | YES (0 errors) | 1 | DETECTED, 1 red | + +**A note on the first fact for the `ltx2_t2a.cpp` rows, because it reported +something misleading and that is worth writing down rather than tidying away.** +`git diff --stat` is measured against `HEAD`, not against the pre-mutation +working tree, so on a run where that file also carried an uncommitted change the +stat reports 45-47 lines rather than the mutation's own 1-3. The number is +therefore not a measurement of the mutation on those rows. It is kept, with this +note, instead of being replaced by a prettier one: the fact the protocol asks for +is what the command printed. The M1, M5, M6, M8 and M9 rows were measured against +a clean file and their stats are the mutations'. + +**M9 is the mutation for a defect this port ACTUALLY SHIPPED in its first +draft**, rather than an invented one. `should_skip_step` (`guiders.py:287-291`) +does not mean "skip the guidance and keep the conditional prediction". Upstream +returns `DenoisedLatentResult.result_or_none(denoised=last_denoised_audio)` +(`utils/denoisers.py:85-91`) BEFORE it assembles any pass, so a skipped step runs +**no DiT forward at all** and reuses the previous step's denoised prediction. The +first draft ran the conditional forward and used it: a whole extra forward per +skipped step, on a different trajectory, producing a finished waveform of exactly +the right length. Nothing about the output separates the two, and the FORWARD +COUNT is the only thing that does. **M7 SURVIVED, and the resolution is the useful part.** It is the mutation a reader coming from another flow-matching sampler expects to be REQUIRED — scaling the initial noise by the first sigma — and it changed nothing. The reason is not -a blind gate: it is an identity. `LTX2Scheduler` starts at `linspace(1, 0, steps +a blind gate: it is an identity, and it STAYS survived after the pin below, +because a pin on an identity cannot turn one arm red. `LTX2Scheduler` starts at `linspace(1, 0, steps + 1)[0] == 1`; the shift map sends 1 to `exp(s)/(exp(s) + (1/1 - 1))`, exactly 1 (`schedulers.py:41-45`); the stretch sends it to `1 - (1 - 1)/scale_factor`, again exactly 1 (`:47-55`). So `sigmas[0]` is 1.0 for every step count. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 6e60752fd..618780c8c 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -169,7 +169,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 DFR base + generated keyframe slots | LTX-2.5 (21.00B video+audio) | gated vs EXECUTED upstream `dfr_layout` + 3 `dfr_pipeline` helpers @ `fd4ded7f` (`test_ltx2_dfr` 11/11, 652 assertions); canvas, tiles, stitch, carry-forward as EXACT index vectors, since each defect is plausible| `--pipeline-kind dfr`. Canvas PADS 9 to 25 then trims back; slots on the x8 grid, MARKED, read back BEFORE the trim. `num_generated_keyframes` SERVED elsewhere. Temporal ROUNDS refused (#986); detail LoRA refused (#975)| | LTX-2.5 tiled + streaming Conv VAE decode | LTX-2.5 video VAE | gated vs executed upstream `ltx_core` @ `fd4ded7f` (`test_ltx2_tiling` 10/10, 915 assertions); one-tile and untiled-spatial controls BIT-EXACT vs untiled on both causality arms; an untiled frames axis is REFUSED | Streams temporal chunks through upstream's AUTO layout (768/64 px, 80/24 frames); above one tile the pixel volume is never materialized. NO-OP below 768px and 81 frames; 81-120 IS tiled, differing 6.70% of range | | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated | -| LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 8 mutations, 7 DETECTED and the 8th proven an identity rather than a blind spot (no count is restated here, because `ctest` is the authority) | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only GUIDED arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | +| LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 9 mutations, 8 DETECTED (one of them a skipped guidance step that recomputes instead of reusing) and the 9th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only GUIDED arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending | diff --git a/src/vllm/model_executor/models/ltx2_t2a.cpp b/src/vllm/model_executor/models/ltx2_t2a.cpp index 2cce2a646..ddfc0f34d 100644 --- a/src/vllm/model_executor/models/ltx2_t2a.cpp +++ b/src/vllm/model_executor/models/ltx2_t2a.cpp @@ -245,6 +245,10 @@ Ltx2T2aResult Ltx2T2aGenerate(const Ltx2T2aRequest& req) { const std::vector positions(positions_f.begin(), positions_f.end()); // ── the denoise loop (samplers.py:39-79) ────────────────────────────────── + // + // `last_denoised_audio` (utils/denoisers.py:85-91): a step the guider SKIPS + // reuses the previous step's denoised prediction instead of running a forward. + std::vector last_denoised; for (int64_t step = 0; step + 1 < sigma_count; ++step) { const float sigma = sigmas[static_cast(step)]; // Every token carries the schedule's own sigma: the mask is all ones, so @@ -262,12 +266,35 @@ Ltx2T2aResult Ltx2T2aGenerate(const Ltx2T2aRequest& req) { ain.context = req.context; // `should_skip_step` (guiders.py:287-291). `skip_step` defaults to 0, which - // never skips, so this is reachable only from an explicit request — and it - // is written as a real branch rather than left implicit because the arm it - // selects is "the conditional prediction alone", which is what a build with - // NO guider also produces. Conflating them would make the default path and - // the skipped path indistinguishable in the counters below. + // never skips, so this is reachable only from an explicit request. + // + // A SKIPPED STEP RUNS NO FORWARD AT ALL AND REUSES THE PREVIOUS STEP'S + // DENOISED PREDICTION. This draft ran the CONDITIONAL forward and used it, + // which is a plausible reading of "skip the guidance" and is not what + // upstream does: `_guided_denoise` returns + // `DenoisedLatentResult.result_or_none(denoised=last_denoised_audio)` when + // every guider skips (`utils/denoisers.py:85-91`), before it assembles a + // single pass. The difference is a whole DiT forward per skipped step and a + // different trajectory, on a render that finishes either way. + // + // `step == 0` can never skip — `0 % (skip_step + 1)` is 0 — so + // `last_denoised` is always populated by the time this branch is taken. The + // guard is kept anyway, because "the arithmetic makes it impossible" is + // exactly the reasoning that a later change to `ShouldSkipStep` would + // silently invalidate, and the failure would be a read of an empty vector. const bool skip = g.ShouldSkipStep(step); + if (skip && last_denoised.empty()) { + Fail("step " + std::to_string(step) + + " is a skipped step and no earlier step produced a denoised prediction to reuse. " + "`should_skip_step` is `step % (skip_step + 1) != 0` (guiders.py:287-291), which is " + "false at step 0, so this is unreachable through the request surface and is a defect " + "rather than a bad request"); + } + if (skip) { + latent = Ltx2EulerStep(latent.data(), last_denoised.data(), sigmas.data(), sigma_count, step, + static_cast(latent.size())); + continue; + } // THE VIDEO STREAM IS `nullptr`, NOT A DISABLED ONE. See ltx2_t2a.h item 1: // upstream's `run_v2a` tests PRESENCE (transformer.py:269), so a @@ -290,7 +317,7 @@ Ltx2T2aResult Ltx2T2aGenerate(const Ltx2T2aRequest& req) { ++result.cond_forwards; std::vector velocity = cond.audio; - if (!skip && (want_uncond || want_perturbed)) { + if (want_uncond || want_perturbed) { std::vector uncond_text; std::vector uncond_perturbed; if (want_uncond) { @@ -314,7 +341,10 @@ Ltx2T2aResult Ltx2T2aGenerate(const Ltx2T2aRequest& req) { static_cast(cond.audio.size())); } - const std::vector denoised = ToDenoised(latent, velocity, timesteps, tokens, width); + // Kept for the next step's `should_skip_step` branch, which reuses it rather + // than recomputing (utils/denoisers.py:85-91). + last_denoised = ToDenoised(latent, velocity, timesteps, tokens, width); + const std::vector& denoised = last_denoised; // `EulerDiffusionStep()` — `DiffusionStage.__call__`'s own default // (utils/blocks.py:524-527), which T2A does not override (it passes no // `stepper`, t2a_one_stage.py:154-170). The ancestral sampler that diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index c0cdee17e..a7e40fa59 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -5157,6 +5157,53 @@ TEST_CASE("ltx2 t2a: the DiT forward runs ONE stream, and the old guard's reason vt::DType::kF32, /*cache=*/nullptr, &bad)); } +TEST_CASE("ltx2 t2a: a SKIPPED step runs no forward and reuses the last prediction") { + // `should_skip_step` is `step % (skip_step + 1) != 0` (guiders.py:287-291), so + // `skip_step = 1` skips every ODD step. Upstream then returns + // `DenoisedLatentResult.result_or_none(denoised=last_denoised_audio)` + // (utils/denoisers.py:85-91) BEFORE it assembles a pass, so a skipped step + // costs no DiT forward at all. + // + // WHY A COUNT AND NOT A DIGEST. A build that "skipped the guidance" by running + // the conditional forward and using it — which is the plausible misreading, + // and what this port did on its first draft — produces a finished waveform of + // exactly the right length on a different trajectory. Nothing about the output + // separates the two. The FORWARD COUNT does, and it is the only thing that + // does. + Workspace ws; + const vllm::multimodal::VideoModelParams mp = T2aParams(ws.paths); + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + REQUIRE(engine != nullptr); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + + vllm::multimodal::VideoGenParams gen = T2aGen(ws.root + "/skip", "a b c"); + gen.steps = 4; // four sigma intervals: steps 0..3, so 1 and 3 skip + gen.extras[vllm::multimodal::kLtx2AudioSkipStepExtra] = "1"; + (void)engine->Generate(gen); + const vllm::multimodal::Ltx2ConditioningTrace skipped = ltx->last_conditioning(); + + // Two of the four steps ran, and each ran all three arms. + CHECK(skipped.t2a_cond_forwards == 2); + CHECK(skipped.t2a_uncond_forwards == 2); + CHECK(skipped.t2a_perturbed_forwards == 2); + + // The control: the same request with no skipping runs all four. + vllm::multimodal::VideoGenParams full = T2aGen(ws.root + "/noskip", "a b c"); + full.steps = 4; + (void)engine->Generate(full); + const vllm::multimodal::Ltx2ConditioningTrace every = ltx->last_conditioning(); + CHECK(every.t2a_cond_forwards == 4); + CHECK(every.t2a_uncond_forwards == 4); + CHECK(every.t2a_perturbed_forwards == 4); + + // A negative skip is refused rather than taken modulo a non-positive divisor. + vllm::multimodal::VideoGenParams bad = T2aGen(ws.root + "/badskip", "a b c"); + bad.extras[vllm::multimodal::kLtx2AudioSkipStepExtra] = "-1"; + CHECK_THROWS(engine->Generate(bad)); +} + TEST_CASE("ltx2 t2a: the schedule starts at exactly 1.0") { // WHY THIS EXISTS, and it is a mutation result rather than a tidiness rule. A // mutation that scaled the initial latent by `sigmas[0]` — the thing a reader From 1d14a7440390e7ac5c77aa0bcead0a6a105de955 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 16 Aug 2026 12:35:06 +0000 Subject: [PATCH 5/9] record(#1031): check-agent-record is RED on main, and the obvious repair is the one the append-only rule forbids MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `scripts/agent-preflight.sh` reports `FAIL check-agent-record` and `FAIL test_check_agent_record` on this branch with `.agents/issue-index.md: issue #995 listed twice`. It is not this branch's doing: `git show origin/main:.agents/issue-index.md` carries both rows, and `git diff origin/main -- .agents/issue-index.md` on this branch shows only the #1005 and #1013 rows it appended. Derived rather than inferred. The `uniq -d` over every row key returns exactly one line, so #995 is the ONLY duplicate and the count of 2 is not an artifact of a partial match. Filed rather than fixed, and the reason is a rule and not a budget. Both rows are legitimate: one filed #995 and the other was appended by the change that closed it. AGENTS.md says "Never edit a row and never delete one", so the obvious repair is forbidden, and it is the "make a red gate green by deleting an assertion" shape that section warns about. The checker's premise is also narrower than reality — it reads a duplicate as two branches union-merging, and this is one branch appending twice on purpose, months apart. Reconciling the two needs a contract decision and a checker-semantics change, which needs its own spec, a red-before test and green-after evidence. The row names `ENG-EXPERT-STREAM` as the owner, which is the row whose two commits appended both #995 rows, so `check-agent-record.py`'s unowned-row count does not grow. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 3f3875e73..f0779c547 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -273,3 +273,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#995](https://github.com/mudler/vllm.cpp/issues/995) | `ENG-EXPERT-STREAM` | `check-env-doc` and `test_check_env_doc` are RED on `origin/main`, so every branch cut from it inherits a preflight failure its own diff did not cause: `VT_MOE_EXPERT_STREAM`, `VT_MOE_EXPERT_STREAM_SLOTS` and `VT_MOE_EXPERT_STREAM_SLOT_BYTES` are read from `src/vllm/model_executor/models/qwen3_5.cpp` (`:5145`, `:5195`, `:5189` @ `4496ef196`) and appear in neither `docs/ENVIRONMENT.md` nor `scripts/env-doc-allowlist.txt`. They arrived with the `ENG-EXPERT-STREAM` W4 wiring commit `3005447f8` ([#993](https://github.com/mudler/vllm.cpp/pull/993)). Same shape as [#965](https://github.com/mudler/vllm.cpp/issues/965)/[#968](https://github.com/mudler/vllm.cpp/issues/968): a pre-existing red presenting to each author in turn as their own. Fixed by DOCUMENTING all three in `docs/ENVIRONMENT.md`, not by allowlisting them — a knob that changes the host memory a MoE decode reserves and disables the default-on grouped-MoE path is a deployment surface, not a kernel-internal micro-tuning switch | bug | [#1013](https://github.com/mudler/vllm.cpp/issues/1013) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 `one_stage` denoised from ZEROS. `OneStagePhase` (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1066 @ 332aed738`) left `Ltx2PhaseRecipe::noise_scale` at the struct default of 0.0, and 0.0 is not "no extra noise": `Ltx2GaussianNoise` is `latent + noise_scale * (noise - latent)` (`:218 @ 332aed738`), so the state stayed exactly as `create_initial_state` wrote it, which with no initial latent is all zeros. Upstream `ModalitySpec.noise_scale` defaults to 1.0 (`ltx-pipelines/utils/types.py:110 @ fd4ded7f`) and `TI2VidOneStagePipeline.__call__` constructs BOTH specs without it (`ti2vid_one_stage.py:233-239`); the two neighbouring recipes set it explicitly, which is what made the omission legible. No gate saw it because every end-to-end test loads `distilled_two_stage`, and a zero-initialized denoise still returns a finite clip of the right size, frame count and sample rate. FOUND and FIXED in flow by row `LTX25-T2A-ONE-STAGE`, whose `t2a_one_stage` rows are built FROM `OneStageRecipe` and would have inherited it. `dmd2` leaves the same field at 0.0 and is deliberately NOT corrected by analogy (its source is vLLM-Omni, not checked out here); that half is listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | | [#1023](https://github.com/mudler/vllm.cpp/issues/1023) | `ENG-EXPERT-STREAM` | The IQ1_S (ggml 19) / IQ1_XXXS (ggml 66) decode landed in [#946](https://github.com/mudler/vllm.cpp/pull/946) with every parameter EXCEPT the grid pinned only by self-consistency. The grid seal (FNV-1a digest + lane census) works and stops one table short: `ReferenceDotF64` (`tests/vt/test_ops_quant_dot.cpp:231-236`) and the G3 NMSE reference (`:915`) both decode the weight with `vt::cpu::BlockToFloat`, the function under test, so they are independent only in the SUMMATION. Three injected defects, each applied and compiled, left the suite green with an UNCHANGED assertion count: `kIq1sDelta` `0.125F`->`0.25F` (`cpu_quant_iq_tables.h:422`, affects BOTH encodings), the IQ1_S delta sign inverted in dequant AND vec_dot, and the IQ1_S scale read from `qh` bits 13-15 instead of 12-14 in both paths. Second defect, same PR: `gguf_dequant.cpp:107-116` lists no `case 19` and no `case 66`, so the expansion path throws `unsupported ggml type` for the two encodings the target checkpoints are 96.92 % made of, and `RouteGgufTensor` (`gguf_keep_quant.cpp:122`) sends a tensor there whenever `VT_CPU_REF` is on, keep-quant is off, K is ragged, or the role is not verbatim — a refusal to load on the reference lane. **ggml 18 (IQ3_XXS) carried the same omission**, pre-existing since the DeepSeek-V4 UD-IQ2_XXS port, and all three are one shared `switch` branch. Repaired by golden vectors whose EXPECTED values come from the ORACLES themselves (`ggml_get_type_traits(type)->to_float` built from `ggml-org/llama.cpp @ 237ad9b96` for 18/19 and `unslothai/llama.cpp @ 36fe8e1cc` for 66) over REAL checkpoint bytes, which is the first reference for this encoding family that is not this tree's own decoder. Also: the `2e-3` NMSE ceiling passed the doubled-delta defect that `6e-4` fails (measured 5.240e-4 unmutated, 6.967e-4 mutated on `iq1_s`), and on `iq1_xxxs` that defect moves the statistic the WRONG WAY (3.109e-4 -> 1.420e-4), so no ceiling can catch it and only the goldens can; and both new census cases claimed TOTAL coverage of 1702 records while summing to 864, F32's 838 tensors omitted. Spec [`expert-streaming.md`](specs/expert-streaming.md) | bug | +| [#1031](https://github.com/mudler/vllm.cpp/issues/1031) | `ENG-EXPERT-STREAM` | `check-agent-record` and `test_check_agent_record` are RED on `origin/main` itself, so every branch cut from it inherits the failure: `.agents/issue-index.md: issue #995 listed twice`. Derived rather than inferred — `git show origin/main:.agents/issue-index.md \| awk -F"\|" "/^\\\| \\\[#/{print \$2}" \| sort \| uniq -d` returns exactly ONE line and the count for it is 2, so it is the only duplicate and not a partial match. BOTH rows are legitimate under the append-only rule: one filed #995, the other was appended by the change that closed it. So the checker contradicts AGENTS.md § *Every change starts from an issue* ("Never edit a row and never delete one") rather than catching a mistake, and its premise is narrower than reality — it reads a duplicate as two branches union-merging, where this is one branch appending twice on purpose. NOT fixed by the row that found it: deleting a row is forbidden and is the "make a red gate green by deleting an assertion" shape, and the repair needs a contract decision plus a checker-semantics spec. Owned by `ENG-EXPERT-STREAM`, whose two commits appended both rows | bug | From 31c2c6d15739b28d299c11229f32b0de91a71767 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 16 Aug 2026 17:10:18 +0000 Subject: [PATCH 6/9] record(LTX25-T2A-ONE-STAGE): the #1031 row describes a red this branch already merged the fix for MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This branch appends an index row for #1031 saying `check-agent-record` and `test_check_agent_record` are RED on `origin/main` because `.agents/issue-index.md` lists issue #995 twice, and that the repair needs a contract decision plus a checker-semantics spec. It does not, and it did not by the time this branch was rebased. #1022 had already read both #995 rows and found neither well-formed, and `ff264cb82` (PR #1025) landed that repair on `main` before this branch merged it at `3d9d9c9bb`. Measured here rather than inferred from the merge: `python3 scripts/check-agent-record.py` prints `agent record OK: ENGINE=156 MODEL=377 QUANT=82 KERNEL=51 BACKEND=83` and exits 0. #1031 is closed as a duplicate of #1022. Corrected in place rather than left, and that is a narrow exception argued here rather than a licence to edit rows. `.agents/issue-index.md` carries `merge=union`: once this row lands it can never be corrected, because an edit to a landed row is duplicated rather than merged. It has not landed. This branch added it, so the net diff against `origin/main` is still additions only, which is exactly what `scripts/check-issue-index-append-only.py --base origin/main` checks — and it exits 0 on this commit. No row that is already on `main` is touched. One note on the instrument, because it reads as a verdict about the tree and is not. That checker diffs `merge-base..HEAD`, so it inspects COMMITTED state and is blind to the working tree: deleting a row of `main`'s in the working tree leaves it printing `OK: issue index append-only` and exiting 0. It has to be run after the commit, and it was. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 2389f498d..6527c88fa 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -275,4 +275,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1023](https://github.com/mudler/vllm.cpp/issues/1023) | `ENG-EXPERT-STREAM` | The IQ1_S (ggml 19) / IQ1_XXXS (ggml 66) decode landed in [#946](https://github.com/mudler/vllm.cpp/pull/946) with every parameter EXCEPT the grid pinned only by self-consistency. The grid seal (FNV-1a digest + lane census) works and stops one table short: `ReferenceDotF64` (`tests/vt/test_ops_quant_dot.cpp:231-236`) and the G3 NMSE reference (`:915`) both decode the weight with `vt::cpu::BlockToFloat`, the function under test, so they are independent only in the SUMMATION. Three injected defects, each applied and compiled, left the suite green with an UNCHANGED assertion count: `kIq1sDelta` `0.125F`->`0.25F` (`cpu_quant_iq_tables.h:422`, affects BOTH encodings), the IQ1_S delta sign inverted in dequant AND vec_dot, and the IQ1_S scale read from `qh` bits 13-15 instead of 12-14 in both paths. Second defect, same PR: `gguf_dequant.cpp:107-116` lists no `case 19` and no `case 66`, so the expansion path throws `unsupported ggml type` for the two encodings the target checkpoints are 96.92 % made of, and `RouteGgufTensor` (`gguf_keep_quant.cpp:122`) sends a tensor there whenever `VT_CPU_REF` is on, keep-quant is off, K is ragged, or the role is not verbatim — a refusal to load on the reference lane. **ggml 18 (IQ3_XXS) carried the same omission**, pre-existing since the DeepSeek-V4 UD-IQ2_XXS port, and all three are one shared `switch` branch. Repaired by golden vectors whose EXPECTED values come from the ORACLES themselves (`ggml_get_type_traits(type)->to_float` built from `ggml-org/llama.cpp @ 237ad9b96` for 18/19 and `unslothai/llama.cpp @ 36fe8e1cc` for 66) over REAL checkpoint bytes, which is the first reference for this encoding family that is not this tree's own decoder. Also: the `2e-3` NMSE ceiling passed the doubled-delta defect that `6e-4` fails (measured 5.240e-4 unmutated, 6.967e-4 mutated on `iq1_s`), and on `iq1_xxxs` that defect moves the statistic the WRONG WAY (3.109e-4 -> 1.420e-4), so no ceiling can catch it and only the goldens can; and both new census cases claimed TOTAL coverage of 1702 records while summing to 864, F32's 838 tensors omitted. Spec [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1005](https://github.com/mudler/vllm.cpp/issues/1005) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 text-to-audio (`T2AOneStagePipeline`, `t2a_one_stage.py:43`, `__call__` at `:109` @ `fd4ded7f`) is absent, and the three blockers are not the ones a reader would guess. (a) `Ltx2DitForward` refuses a one-stream call at `src/vllm/model_executor/models/ltx2_dit.cpp:765 @ 332aed738`, citing a weight contract that describes a checkpoint T2A never loads — upstream reads the ordinary AudioVideo FILE through `LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP` (`model_configurator.py:228-239`). (b) The same message advises `enabled=false` as the substitute, and it is NOT: `run_v2a = run_ax and (video is not None and vx.numel() > 0)` (`transformer.py:269`) tests PRESENCE, not `enabled`, so a disabled-but-present video stream still feeds v2a cross attention and still returns a finished waveform; our port mirrors that polarity at `src/vllm/model_executor/models/ltx2_dit.cpp:251 @ 332aed738`. (c) The engine has NO guided denoiser at all — `git grep -n 'guid\\|cfg_scale' src/vllm/multimodal/ltx2_video.cpp` is 0 against 66 for `ltx2` in the same file as the control — while T2A defaults to `cfg_scale=7.0` and `stg_scale=1.0`, i.e. THREE forwards per step. Spec [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | feature | | [#1013](https://github.com/mudler/vllm.cpp/issues/1013) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 `one_stage` denoised from ZEROS. `OneStagePhase` (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1066 @ 332aed738`) left `Ltx2PhaseRecipe::noise_scale` at the struct default of 0.0, and 0.0 is not "no extra noise": `Ltx2GaussianNoise` is `latent + noise_scale * (noise - latent)` (`:218 @ 332aed738`), so the state stayed exactly as `create_initial_state` wrote it, which with no initial latent is all zeros. Upstream `ModalitySpec.noise_scale` defaults to 1.0 (`ltx-pipelines/utils/types.py:110 @ fd4ded7f`) and `TI2VidOneStagePipeline.__call__` constructs BOTH specs without it (`ti2vid_one_stage.py:233-239`); the two neighbouring recipes set it explicitly, which is what made the omission legible. No gate saw it because every end-to-end test loads `distilled_two_stage`, and a zero-initialized denoise still returns a finite clip of the right size, frame count and sample rate. FOUND and FIXED in flow by row `LTX25-T2A-ONE-STAGE`, whose `t2a_one_stage` rows are built FROM `OneStageRecipe` and would have inherited it. `dmd2` leaves the same field at 0.0 and is deliberately NOT corrected by analogy (its source is vLLM-Omni, not checked out here); that half is listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | -| [#1031](https://github.com/mudler/vllm.cpp/issues/1031) | `ENG-EXPERT-STREAM` | `check-agent-record` and `test_check_agent_record` are RED on `origin/main` itself, so every branch cut from it inherits the failure: `.agents/issue-index.md: issue #995 listed twice`. Derived rather than inferred — `git show origin/main:.agents/issue-index.md \| awk -F"\|" "/^\\\| \\\[#/{print \$2}" \| sort \| uniq -d` returns exactly ONE line and the count for it is 2, so it is the only duplicate and not a partial match. BOTH rows are legitimate under the append-only rule: one filed #995, the other was appended by the change that closed it. So the checker contradicts AGENTS.md § *Every change starts from an issue* ("Never edit a row and never delete one") rather than catching a mistake, and its premise is narrower than reality — it reads a duplicate as two branches union-merging, where this is one branch appending twice on purpose. NOT fixed by the row that found it: deleting a row is forbidden and is the "make a red gate green by deleting an assertion" shape, and the repair needs a contract decision plus a checker-semantics spec. Owned by `ENG-EXPERT-STREAM`, whose two commits appended both rows | bug | +| [#1031](https://github.com/mudler/vllm.cpp/issues/1031) | `ENG-EXPERT-STREAM` | CLOSED as a duplicate of [#1022](https://github.com/mudler/vllm.cpp/issues/1022), and this row is the corrected one rather than the filed one. As filed it said `check-agent-record` and `test_check_agent_record` are RED on `origin/main` because `.agents/issue-index.md` lists issue #995 twice, and that the repair needed a contract decision plus a checker-semantics spec. It did not: [#1022](https://github.com/mudler/vllm.cpp/issues/1022) had already read the two #995 rows and found neither WELL-FORMED, and `ff264cb82` (PR [#1025](https://github.com/mudler/vllm.cpp/pull/1025)) landed the repair on `main` before this branch merged it. Measured at this branch's head rather than inferred from the merge: `python3 scripts/check-agent-record.py` prints `agent record OK: ENGINE=156 MODEL=377 QUANT=82 KERNEL=51 BACKEND=83` and exits 0. CORRECTED IN PLACE, and that is a narrow exception argued here rather than a licence: the row had not landed, this branch added it, and the net diff against `origin/main` is still additions only — which `scripts/check-issue-index-append-only.py --base origin/main` is what checks. Once it lands `merge=union` makes it permanent and un-correctable, so leaving a filed-and-refuted claim in the record was the more expensive option | bug | From c1fe355925b141fe4149d8710b38aac4c9dd6a17 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 16 Aug 2026 17:19:46 +0000 Subject: [PATCH 7/9] fix(LTX25-T2A-ONE-STAGE): T2A guidance combined velocities, and upstream combines x0 (#1039) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Upstream never hands the denoiser the raw velocity model. `DiffusionStage` builds `X0Model(self._prepared_builder().build(device=target, **kwargs))` (ltx-pipelines `utils/blocks.py:480-482` @ `fd4ded7f`), and `X0Model.forward` returns `to_denoised(audio.latent, ax, audio.timesteps)` (ltx-core `model/transformer/model.py:590-604`), which is `sample - velocity * sigma` (ltx-core `utils.py:39-52`). So `_guided_denoise`'s `all_v, all_a = transformer(...)` (`utils/denoisers.py:188`) already carries DENOISED tensors, and `audio_guider.calculate(cond_a, uncond_a, ptb_a, mod_a)` at `:203` combines those. `Ltx2T2aGenerate` took `Ltx2DitForward`'s velocities straight into `Ltx2MultiModalGuidance` and applied `ToDenoised` once to the result. That is the same function only while `rescale_scale == 0`. `calculate`'s linear terms (`guiders.py:261-266`) are invariant under `x0 = latent - sigma*v`; the rescale at `:268-271` is not. Upstream's `factor` is `std(x0_cond)/std(x0_pred)` and it scales the whole x0, giving `factor*(latent - sigma*v)`, where scaling the velocity gives `latent - sigma*factor*v`. The two differ by `(factor - 1) * latent`, non-zero wherever the latent is — and on this path the state IS the unit-variance noise, so everywhere. `rescale_scale = 0.7` is the shipped T2A default (`utils/constants.py:63`, `utils/args.py:1101-1106`), so every default render took the divergent branch. Nothing already gated could see it. The three forward counters, `t2a_video_stream_present`, `t2a_perturbed_blocks`, the latent absmax and the waveform's length, channel count and sample rate are identical between the two forms. Fixed by moving the conversion, not by moving the rescale. The per-pass `x0_model` lambda IS `X0Model`: it applies `ToDenoised` on the way out of every forward, so the guider combines x0 and `Ltx2MultiModalGuidance` stays a faithful port of `calculate` over whatever the model returned. Reaching the same numbers by moving the rescale into the guidance seam would put `to_denoised` inside `calculate`, where upstream does not have it, and would leave the seam correct only for this one composition. THE VIDEO ARM IS UNAFFECTED, checked rather than assumed. `git grep -n Ltx2MultiModalGuidance -- src include` returns exactly one production call site, `ltx2_t2a.cpp`. `video_guider` and `video_guidance` are recipe fields nothing reads: the joint driver runs ONE unguided forward per step and applies `ToDenoised` to that single velocity (`ltx2_video.cpp:3034-3036`), which is the same tensor in either space because there is no combination to be invariant under. THE REDUCED FIXTURE CANNOT RESOLVE THE RESCALE'S NUMERIC CONSEQUENCE, measured rather than assumed. Its DiT responds to the conditioning at ~1e-5 of its own output, so `std(cond)/std(pred)` is 1.0 to 1e-5 in BOTH spaces, both factors land within 1e-5 of 1.0, and the two candidate step-0 predictions sit 7.6e-07 apart against a span of 3.41. The first draft of the test asserted exactly that difference and its own separation guard refused it — a case that would have been green either way. So the defect is gated at two places instead: - END TO END, through `LoadVideoEngine` and `VideoEngine::Generate` on the recipe's own guider with no extra touched, by pinning the EQUATION `cond == latent - sigma*velocity` between three recorded step-0 tensors. Exact in x0 space, off by the whole sample in velocity space; no fixture scale meets it by accident, and a zero sample or a zero velocity fails the two preceding REQUIREs rather than passing it. - AT THE SEAM, on the real `Ltx2MultiModalGuidance` with a latent that makes it visible. MEASURED: the two spaces disagree by 1.50e-07 relative at `rescale_scale = 0.0` and by 0.352 at the shipped 0.7. That is what makes 0.0 the control rather than the assertion site. Observability added: four step-0 tensors and step 0's sigma on `Ltx2T2aResult` and the trace — the sample, the conditional pass's raw velocity, the tensor handed to the guider, and the guider's result. `first_step_cond` is upstream's own `DenoisedLatentResult.cond` (`utils/denoisers.py:206`). Mutations, each applied to one file, rebuilt, run, and restored with the restore verified by sha256. Three comma-free `--test-case` filters, each asserting a non-zero case count, because doctest splits `-tc` on commas and runs unrelated cases to a green SUCCESS: | Mutation | `git diff --stat` | BUILT | exit | verdict | |---|---|---|---|---| | N1 revert to velocity-space guidance | `ltx2_t2a.cpp \| 4 +-` | YES (0 errors) | 1 | DETECTED | | N2 delete the production call site | `ltx2_video.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED | | N3 take x0 against a ZERO sample | `ltx2_t2a.cpp \| 3 +-` | YES (0 errors) | 1 | DETECTED | | N4 drop the rescale branch entirely | `ltx2_pipeline.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED | N4 is why the seam case is not decorative: it is the only one of the four the end-to-end case does not see. No GPU result is claimed. `dgx.casa` is down, so there is no render on real weights here, and the 18.17 % figure in #1039 is synthetic-tensor algebra rather than a measurement. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 1 + .agents/specs/ltx25-t2a-one-stage.md | 128 +++++++++++ docs/FEATURES.md | 3 +- docs/USAGE.md | 7 + include/vllm/model_executor/models/ltx2_t2a.h | 54 ++++- include/vllm/multimodal/ltx2_video.h | 20 ++ src/vllm/model_executor/models/ltx2_t2a.cpp | 65 ++++-- src/vllm/multimodal/ltx2_video.cpp | 5 + tests/vllm/multimodal/test_ltx2_video.cpp | 206 ++++++++++++++++++ 9 files changed, 467 insertions(+), 22 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 6527c88fa..6cf34d5ac 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -276,3 +276,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1005](https://github.com/mudler/vllm.cpp/issues/1005) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 text-to-audio (`T2AOneStagePipeline`, `t2a_one_stage.py:43`, `__call__` at `:109` @ `fd4ded7f`) is absent, and the three blockers are not the ones a reader would guess. (a) `Ltx2DitForward` refuses a one-stream call at `src/vllm/model_executor/models/ltx2_dit.cpp:765 @ 332aed738`, citing a weight contract that describes a checkpoint T2A never loads — upstream reads the ordinary AudioVideo FILE through `LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP` (`model_configurator.py:228-239`). (b) The same message advises `enabled=false` as the substitute, and it is NOT: `run_v2a = run_ax and (video is not None and vx.numel() > 0)` (`transformer.py:269`) tests PRESENCE, not `enabled`, so a disabled-but-present video stream still feeds v2a cross attention and still returns a finished waveform; our port mirrors that polarity at `src/vllm/model_executor/models/ltx2_dit.cpp:251 @ 332aed738`. (c) The engine has NO guided denoiser at all — `git grep -n 'guid\\|cfg_scale' src/vllm/multimodal/ltx2_video.cpp` is 0 against 66 for `ltx2` in the same file as the control — while T2A defaults to `cfg_scale=7.0` and `stg_scale=1.0`, i.e. THREE forwards per step. Spec [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | feature | | [#1013](https://github.com/mudler/vllm.cpp/issues/1013) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 `one_stage` denoised from ZEROS. `OneStagePhase` (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1066 @ 332aed738`) left `Ltx2PhaseRecipe::noise_scale` at the struct default of 0.0, and 0.0 is not "no extra noise": `Ltx2GaussianNoise` is `latent + noise_scale * (noise - latent)` (`:218 @ 332aed738`), so the state stayed exactly as `create_initial_state` wrote it, which with no initial latent is all zeros. Upstream `ModalitySpec.noise_scale` defaults to 1.0 (`ltx-pipelines/utils/types.py:110 @ fd4ded7f`) and `TI2VidOneStagePipeline.__call__` constructs BOTH specs without it (`ti2vid_one_stage.py:233-239`); the two neighbouring recipes set it explicitly, which is what made the omission legible. No gate saw it because every end-to-end test loads `distilled_two_stage`, and a zero-initialized denoise still returns a finite clip of the right size, frame count and sample rate. FOUND and FIXED in flow by row `LTX25-T2A-ONE-STAGE`, whose `t2a_one_stage` rows are built FROM `OneStageRecipe` and would have inherited it. `dmd2` leaves the same field at 0.0 and is deliberately NOT corrected by analogy (its source is vLLM-Omni, not checked out here); that half is listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | | [#1031](https://github.com/mudler/vllm.cpp/issues/1031) | `ENG-EXPERT-STREAM` | CLOSED as a duplicate of [#1022](https://github.com/mudler/vllm.cpp/issues/1022), and this row is the corrected one rather than the filed one. As filed it said `check-agent-record` and `test_check_agent_record` are RED on `origin/main` because `.agents/issue-index.md` lists issue #995 twice, and that the repair needed a contract decision plus a checker-semantics spec. It did not: [#1022](https://github.com/mudler/vllm.cpp/issues/1022) had already read the two #995 rows and found neither WELL-FORMED, and `ff264cb82` (PR [#1025](https://github.com/mudler/vllm.cpp/pull/1025)) landed the repair on `main` before this branch merged it. Measured at this branch's head rather than inferred from the merge: `python3 scripts/check-agent-record.py` prints `agent record OK: ENGINE=156 MODEL=377 QUANT=82 KERNEL=51 BACKEND=83` and exits 0. CORRECTED IN PLACE, and that is a narrow exception argued here rather than a licence: the row had not landed, this branch added it, and the net diff against `origin/main` is still additions only — which `scripts/check-issue-index-append-only.py --base origin/main` is what checks. Once it lands `merge=union` makes it permanent and un-correctable, so leaving a filed-and-refuted claim in the record was the more expensive option | bug | +| [#1039](https://github.com/mudler/vllm.cpp/issues/1039) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 T2A combined its guidance passes in VELOCITY space. Upstream hands the denoiser an `X0Model` (`ltx-pipelines/utils/blocks.py:480-482 @ fd4ded7f`) whose `forward` returns `to_denoised(latent, v, timesteps)` (`ltx-core/model/transformer/model.py:601-604`, `to_denoised` at `ltx-core/utils.py:39-52`), so `_guided_denoise` combines DENOISED tensors — `all_v, all_a = transformer(...)` at `utils/denoisers.py:188` and `audio_guider.calculate(...)` at `:203`. `Ltx2T2aGenerate` combined raw DiT velocities and applied `ToDenoised` once afterwards. `MultiModalGuider.calculate`'s LINEAR terms are invariant under `x0 = latent - sigma*v`, so the two forms agree exactly while `rescale_scale == 0`; the RESCALE branch (`guiders.py:268-271`) is not, because upstream's `std(x0_cond)/std(x0_pred)` scales the whole x0 to `f*(latent - sigma*v)` where scaling the velocity gives `latent - sigma*f*v` — a difference of `(f - 1) * latent`, non-zero wherever the latent is, which on this path is everywhere. `rescale_scale = 0.7` is the SHIPPED T2A default (`utils/constants.py:63`, `utils/args.py:1101-1106`), so every default render took the divergent branch. No gate saw it: the forward counts, perturbed blocks, latent absmax, waveform length, channel count and sample rate are identical between the two forms. FOUND by review of PR [#1032](https://github.com/mudler/vllm.cpp/pull/1032) and FIXED in flow on the same branch, before it landed. The VIDEO arm is unaffected and the reason is recorded rather than assumed: `Ltx2MultiModalGuidance` has exactly ONE production caller and the joint driver runs a single unguided forward per step. Spec [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | diff --git a/.agents/specs/ltx25-t2a-one-stage.md b/.agents/specs/ltx25-t2a-one-stage.md index 96bdfc90b..4d4f1d6a0 100644 --- a/.agents/specs/ltx25-t2a-one-stage.md +++ b/.agents/specs/ltx25-t2a-one-stage.md @@ -575,8 +575,136 @@ here. Left as-is and named rather than corrected by analogy: a recipe whose upstream nobody read is exactly where a plausible fix lands wrong. Recorded in #1013. +## 7c. The second bug this row found and fixed in flow + +[#1039](https://github.com/mudler/vllm.cpp/issues/1039). **The guidance passes +were combined in VELOCITY space, and upstream combines x0.** Found by review of +pull request #1032 at `3d9d9c9bb`, before the branch landed. + +Upstream never hands the denoiser the raw velocity model. `DiffusionStage` +builds `X0Model(self._prepared_builder().build(device=target, **kwargs))` +(`ltx-pipelines/utils/blocks.py:480-482`), and `X0Model.forward` returns +`to_denoised(audio.latent, ax, audio.timesteps)` (`ltx-core +model/transformer/model.py:590-604`), which is `sample - velocity * sigma` +(`ltx-core utils.py:39-52`). So `_guided_denoise`'s +`all_v, all_a = transformer(...)` (`ltx-pipelines/utils/denoisers.py:188`) +already carries DENOISED tensors, and `audio_guider.calculate(cond_a, uncond_a, +ptb_a, mod_a)` at `:203` combines those. + +`Ltx2T2aGenerate` took `Ltx2DitForward`'s velocities straight into +`Ltx2MultiModalGuidance` and applied `ToDenoised` once to the result. + +**Why no existing gate saw it, and why the row's own §5 could not.** +`MultiModalGuider.calculate`'s linear terms (`guiders.py:261-266`) are invariant +under `x0 = latent - sigma*v`, so the two forms agree EXACTLY while +`rescale_scale == 0`. The rescale at `:268-271` is not invariant: upstream's +`factor` is `std(x0_cond)/std(x0_pred)` and it scales the whole x0, giving +`factor*(latent - sigma*v)`, where scaling the velocity gives +`latent - sigma*factor*v`. The two differ by `(factor - 1) * latent` — zero +only where the latent is zero, which on this path it never is, because the state +IS the unit-variance noise (§3.2 item 6). `rescale_scale = 0.7` is the shipped +default (`utils/constants.py:63`, `utils/args.py:1101-1106`), so the DEFAULT arm +took the divergent branch. Everything §5 observes — the three forward counters, +`t2a_video_stream_present`, `t2a_perturbed_blocks`, the latent absmax, the +waveform's length, channel count and sample rate — is identical between the two +forms. + +**Fixed by moving the conversion, not by moving the rescale.** The mirror is +structural: the per-pass `x0_model` lambda in `ltx2_t2a.cpp` IS `X0Model`, it +applies `ToDenoised` on the way out of every forward, and +`Ltx2MultiModalGuidance` stays a faithful port of `calculate` over whatever the +model returned. Reaching the same numbers by moving the rescale inside the +guidance seam would put `to_denoised` inside `calculate`, where upstream does not +have it, and would make the seam correct only for this one composition. + +**The VIDEO arm is unaffected, and that was checked rather than assumed.** +`git grep -n Ltx2MultiModalGuidance -- src include` returns exactly one +production call site, `ltx2_t2a.cpp`. `Ltx2PipelineParams::video_guider` and +`Ltx2PhaseRecipe::video_guidance` are carried by the recipe and read by nothing: +the joint driver in `ltx2_video.cpp` runs ONE unguided `Ltx2DitForward` per step +and applies `ToDenoised` to that single velocity (`:3034-3036`), which is the +same tensor in both spaces because there is no combination to be invariant +under. There is therefore no second instance of this defect to fix, and there +will be one the moment a guided video denoiser is wired — noted here because +that wiring is a live campaign item. + +**What the gate is, and what it deliberately is not.** The reduced fixture +CANNOT resolve the rescale's numeric consequence, and that is measured rather +than assumed: its DiT responds to the conditioning at ~1e-5 of its own output, +so `std(cond)/std(pred)` is 1.0 to 1e-5 in BOTH spaces, both factors land within +1e-5 of 1.0, and the two candidate step-0 predictions sit 7.6e-07 apart against +a span of 3.41. The first draft of the test asserted exactly that difference and +its own separation guard refused it — a case that would have been green either +way. So the row gates the defect at two places instead: + +1. `test_ltx2_video` "the guider is handed x0 predictions and not raw + velocities" — end to end through `LoadVideoEngine` and + `VideoEngine::Generate`, on the recipe's own guider with no extra touched. It + pins the EQUATION `cond == latent - sigma*velocity` between three recorded + step-0 tensors. That is exact in x0 space and off by the whole sample in + velocity space, so no fixture scale satisfies it by accident; a zeroed + velocity or a zero sample fails the two `REQUIRE`s that precede it rather + than passing it. +2. `test_ltx2_video` "rescale_scale 0 is the control because both spaces agree + there" — the numeric consequence, on the real `Ltx2MultiModalGuidance` seam + with a latent that makes it visible. MEASURED: relative disagreement between + the two spaces is **1.50e-07 at `rescale_scale = 0.0`** and **0.352 at the + shipped 0.7**. This is what makes 0.0 the control rather than the assertion + site. + +The observability this needed is four step-0 fields on `Ltx2T2aResult` and the +trace: the sample, the conditional pass's RAW velocity, the tensor handed to the +guider, and the guider's result, plus step 0's sigma. `first_step_cond` is +upstream's own `DenoisedLatentResult.cond` (`utils/denoisers.py:206`). + +### Mutations for #1039 + +Focused gate: three comma-free `--test-case` filters, each asserting a non-zero +case count. Each mutation applied to ONE file, rebuilt, run, restored in a +`finally` and the restore verified by **sha256**. `git diff --stat` is scoped to +the mutated file and measured against the committed fix, so the numbers are the +mutation's own. + +| Mutation | `git diff --stat` | BUILT | exit | verdict | +|---|---|---|---|---| +| N1 revert to velocity-space guidance (the defect) | `ltx2_t2a.cpp \| 4 ++--` | YES (0 errors) | 1 | DETECTED by case 1 | +| N2 delete the production call site | `ltx2_video.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED by case 1 AND the render case | +| N3 take x0 against a ZERO sample instead of the latent | `ltx2_t2a.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED by case 1 | +| N4 drop the rescale branch entirely (`guiders.py:268-271`) | `ltx2_pipeline.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED by the control case | + +N4 is the row that proves the control case is not decorative: it is the only one +of the four that case 1 does not see, and the only one the control does. + +**N1 is the RED-before**, and this is what it printed: + +``` +tests/vllm/multimodal/test_ltx2_video.cpp:5371: ERROR: + CHECK( err_x0 <= 1e-5 * latent_span ) is NOT correct! + values: CHECK( 3.43642 <= 3.38677e-05 ) + logged: sigma = 1 max|latent| = 3.38677 max|velocity| = 0.415609 + |cond - (latent - sigma*velocity)| = 3.43642 |cond - velocity| = 0 + elements = 3328 +tests/vllm/multimodal/test_ltx2_video.cpp:5378: ERROR: + CHECK( err_v > 1e-2 * latent_span ) is NOT correct! + values: CHECK( 0 > 0.0338677 ) +[doctest] test cases: 1 | 0 passed | 1 failed | 66 skipped +[doctest] assertions: 16 | 14 passed | 2 failed | +[doctest] Status: FAILURE! exit 1 +``` + +`|cond - velocity| = 0` **exactly** is the whole finding: the tensor handed to +`Ltx2MultiModalGuidance` WAS the raw DiT velocity. Green after, on the same +filter: 1 case, 16 assertions, 0 failed, exit 0. + ## Owed +- **The rescale's numeric consequence END TO END.** Gated at the seam (0.352 + relative at the shipped 0.7) and at the space (exactly, through the engine), + and NOT on a render, because the reduced fixture's guidance deltas are ~1e-5 + of the prediction and both rescale factors land within 1e-5 of 1.0. What would + close it is the real-checkpoint render already owed below, where the DiT's + velocity is comparable to the sample. Tracked by + [#1039](https://github.com/mudler/vllm.cpp/issues/1039). - **The DEVICE arm.** `Ltx2DitForwardDevice` dereferences `*video` unconditionally from its first `PrepareStreamDev` call onward (`src/vllm/model_executor/models/ltx2_device.cpp`, the two `PrepareStreamDev` diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 1a16950f3..66e1dac85 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -170,7 +170,8 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 tiled + streaming Conv VAE decode | LTX-2.5 video VAE | gated vs executed upstream `ltx_core` @ `fd4ded7f` (`test_ltx2_tiling` 10/10, 915 assertions); one-tile and untiled-spatial controls BIT-EXACT vs untiled on both causality arms; an untiled frames axis is REFUSED | Streams temporal chunks through upstream's AUTO layout (768/64 px, 80/24 frames); above one tile the pixel volume is never materialized. NO-OP below 768px and 81 frames; 81-120 IS tiled, differing 6.70% of range | | LTX-2.5 Conv VAE decode arithmetic width | LTX-2.5 video VAE | `test_ltx2_vae` "the decode's convolution accumulates in f32", entering through `Ltx2VideoDecodeStreaming`; widening the accumulator to `double`, or deleting the production call site, each turns it RED | **f32**, the width `F.conv3d` uses at f32 AND bf16 (MEASURED). Was f64 at 8 sites ([#1008](https://github.com/mudler/vllm.cpp/issues/1008)). Conv sums BLOCKED per input channel, as torch's. STORAGE stays f32; bf16 owed | | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated | -| LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 9 mutations, 8 DETECTED (one of them a skipped guidance step that recomputes instead of reusing) and the 9th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only GUIDED arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | +| LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 13 mutations, 12 DETECTED (one of them a skipped guidance step that recomputes instead of reusing) and the 13th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only GUIDED arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | +| LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions and not raw velocities" through `Generate`, plus a seam case measuring the two spaces 1.50e-07 apart at `rescale_scale = 0.0` and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 ([#1039](https://github.com/mudler/vllm.cpp/issues/1039)) | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending | diff --git a/docs/USAGE.md b/docs/USAGE.md index 26ce60278..17e3bd7ee 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -881,6 +881,13 @@ outside the DiT's own layer count is refused rather than clamped. There is no `modality_scale` knob: upstream pins it to 1.0 for this pipeline, because audio-only generation has no video modality to isolate. +`--audio-rescale-scale` acts on the **denoised (x0) prediction**, not on the +DiT's velocity, because upstream's guider sits behind an `X0Model` and combines +already-converted tensors. The distinction is invisible at `0.0`, where the two +readings agree exactly, and it changes the render at every other value — so a +recipe or a script that was tuned against the velocity reading will not +reproduce here at the default `0.7` (issue #1039). + Being per-generation, those six reach the CLI and the C ABI and **not** `/v1/videos`, which forwards no per-generation extra to any engine (issue #928). `pipeline_kind` is a LOAD knob and does reach the server, so a server started diff --git a/include/vllm/model_executor/models/ltx2_t2a.h b/include/vllm/model_executor/models/ltx2_t2a.h index 80af49c4d..12b13ae23 100644 --- a/include/vllm/model_executor/models/ltx2_t2a.h +++ b/include/vllm/model_executor/models/ltx2_t2a.h @@ -11,8 +11,12 @@ // (T2AOneStagePipeline.__call__) // the guided step <- ltx-core components/guiders.py:244-273 // (MultiModalGuider.calculate), reached through -// ltx-pipelines utils/denoisers.py +// ltx-pipelines utils/denoisers.py:188-203 // (FactoryGuidedDenoiser) +// the x0 wrapper on each pass <- ltx-core model/transformer/model.py:590-604 +// (X0Model.forward), which is what +// ltx-pipelines utils/blocks.py:480-482 builds +// and hands the denoiser // the STG pass <- ltx-core model/transformer/attention.py:552-577 // the audio latent shape <- ltx-core types.py:164-200 // (AudioLatentShape.from_video_pixel_shape) @@ -33,7 +37,7 @@ // branches inside it would be nine chances to leave one behind, and a missed one // renders. // -// ─── THE THREE THINGS THAT FAIL SILENTLY IF GUESSED ────────────────────────── +// ─── THE FOUR THINGS THAT FAIL SILENTLY IF GUESSED ─────────────────────────── // // 1. `video = nullptr`, NOT `video->enabled = false`. Upstream's predicate is // `run_v2a = run_ax and (video is not None and vx.numel() > 0)` @@ -57,6 +61,26 @@ // (t2a_one_stage.py:200-202). The params table's own value is 3.0, so // inheriting it would turn on a fourth forward against a modality that is not // there. +// +// 4. THE GUIDER COMBINES X0, NOT VELOCITY (#1039). Upstream builds the +// denoiser's transformer as `X0Model(...)` (ltx-pipelines +// utils/blocks.py:480-482), so every pass `_guided_denoise` hands +// `MultiModalGuider.calculate` has ALREADY been converted with +// `to_denoised(latent, v, timesteps)` (model.py:590-604, `to_denoised` at +// ltx-core utils.py:39-52) before it is combined +// (utils/denoisers.py:188-203). +// +// Combining raw velocities and converting once afterwards is the SAME +// FUNCTION only while `rescale_scale == 0`, because `calculate`'s linear +// terms are invariant under `x0 = latent - sigma*v`. The rescale branch is +// not invariant: upstream computes `factor` from `std(x0_cond)/std(x0_pred)` +// and scales the whole x0, giving `factor*(latent - sigma*v)`; scaling the +// velocity instead gives `latent - sigma*factor*v`. The two differ by +// `(factor - 1) * latent` — zero only where the latent is zero, which on +// this path it never is (the state IS the unit-variance noise, item above). +// `rescale_scale = 0.7` is the shipped T2A default (utils/constants.py:63, +// utils/args.py:1101-1106), so this is the DEFAULT arm rather than an +// exotic one, and nothing about the rendered waveform separates the two. #pragma once #include @@ -149,6 +173,32 @@ struct Ltx2T2aResult { // because a latent that collapsed to zeros has a perfectly stable digest. uint64_t latent_digest = 0; double latent_absmax = 0.0; + + // EVERYTHING STEP 0 PRODUCED, in the order it produced it: the sampler's + // input, the conditional pass's RAW DiT output, the tensor that pass handed + // the guider, and the guider's result. They exist because #1039 is invisible + // in every other field here — combining the guidance passes in VELOCITY space + // and converting once afterwards produces a waveform of the right length, the + // right channel count, the right sample rate and a perfectly healthy forward + // count — and together they make the question decidable by arithmetic rather + // than by magnitude: + // + // first_step_cond == first_step_latent - sigma * first_step_velocity + // + // holds when the guider is handed X0 PREDICTIONS, as `X0Model.forward` does + // (model.py:590-604, over the `X0Model(...)` that utils/blocks.py:480-482 + // builds), and fails when it is handed the velocities. `first_step_cond` is + // also upstream's own `DenoisedLatentResult.cond` (utils/denoisers.py:206), + // rather than a field invented for a test. + // + // STEP 0 SPECIFICALLY, because it is the one step whose inputs do not depend + // on any earlier step, so two renders that differ only in a guider parameter + // share a bit-identical step-0 latent and bit-identical DiT passes. + std::vector first_step_latent; + std::vector first_step_velocity; + std::vector first_step_cond; + std::vector first_step_denoised; + double first_step_sigma = 0.0; }; // `T2AOneStagePipeline.__call__` (t2a_one_stage.py:109-172). Throws diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index eedd6f0c0..7c8d84b58 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -723,12 +723,32 @@ struct Ltx2ConditioningTrace { // off the request's `stg_blocks`: a count alone cannot tell "perturbed block // 1" from "perturbed block 0", and which block is perturbed is the whole of // STG. + // + // The `t2a_first_*` block is everything step 0 produced, and it is the only + // observable that separates upstream's x0-space guidance combination from a + // velocity-space one (#1039). Every other field here — the forward counts, + // the perturbed blocks, the latent absmax, the waveform's length, channel + // count and sample rate — is identical between the two forms, and on a + // reduced fixture so is the rendered audio, because the guidance deltas are + // ~1e-5 of the prediction and the rescale factor lands within 1e-5 of 1.0 in + // BOTH spaces. What is not identical, and is not a matter of degree, is which + // tensor the guider was handed: + // + // t2a_first_cond == t2a_first_latent - sigma * t2a_first_velocity + // + // holds in x0 space (`X0Model.forward`, ltx-core model/transformer/ + // model.py:590-604) and fails in velocity space. bool t2a_rendered = false; bool t2a_video_stream_present = false; int64_t t2a_cond_forwards = 0; int64_t t2a_uncond_forwards = 0; int64_t t2a_perturbed_forwards = 0; std::vector t2a_perturbed_blocks; + std::vector t2a_first_latent; + std::vector t2a_first_velocity; + std::vector t2a_first_cond; + std::vector t2a_first_denoised; + double t2a_first_sigma = 0.0; // True only once the `Generate` that produced this conditioning RETURNED. The // trace is filled immediately after the connector and BEFORE the denoise loop, diff --git a/src/vllm/model_executor/models/ltx2_t2a.cpp b/src/vllm/model_executor/models/ltx2_t2a.cpp index ddfc0f34d..44a85d4ce 100644 --- a/src/vllm/model_executor/models/ltx2_t2a.cpp +++ b/src/vllm/model_executor/models/ltx2_t2a.cpp @@ -36,12 +36,16 @@ double AbsMax(const std::vector& values) { return m; } -// `to_denoised` (ltx-core model/transformer/utils.py:38-50) through -// `X0Model.forward` (model.py:590-604): the DiT emits a VELOCITY and the sampler -// wants the x0 prediction. Identical arithmetic to the joint driver's own -// `ToDenoised`; kept here rather than shared because the joint one is a static in -// `ltx2_video.cpp`'s anonymous namespace and hoisting it would move lines above -// that file's gated READER ANCHORS list for no behavioural reason. +// `to_denoised` (ltx-core utils.py:39-52) as `X0Model.forward` applies it +// (model.py:590-604): the DiT emits a VELOCITY and everything downstream — the +// guider AND the sampler — wants the x0 prediction. It is therefore applied to +// EVERY PASS, on the way out of the forward, and not once to the guider's +// output: see ltx2_t2a.h item 4 (#1039). +// +// Identical arithmetic to the joint driver's own `ToDenoised`; kept here rather +// than shared because the joint one is a static in `ltx2_video.cpp`'s anonymous +// namespace and hoisting it would move lines above that file's gated READER +// ANCHORS list for no behavioural reason. std::vector ToDenoised(const std::vector& sample, const std::vector& velocity, const std::vector& timesteps, int64_t tokens, int64_t width) { VT_CHECK(velocity.size() == sample.size(), "ltx2 t2a: the velocity is the wrong size"); @@ -300,30 +304,48 @@ Ltx2T2aResult Ltx2T2aGenerate(const Ltx2T2aRequest& req) { // upstream's `run_v2a` tests PRESENCE (transformer.py:269), so a // present-but-disabled stream still feeds video->audio cross attention. // + // AND EVERY PASS IS CONVERTED TO X0 *HERE*, BEFORE THE GUIDER SEES IT. This + // lambda is `X0Model` (model.py:590-604): upstream never hands the denoiser + // the raw velocity model, it hands `X0Model(builder.build(...))` + // (utils/blocks.py:480-482), so `_guided_denoise`'s + // `all_v, all_a = transformer(...)` at utils/denoisers.py:188 already + // carries DENOISED tensors and `audio_guider.calculate(...)` at `:203` + // combines those. See ltx2_t2a.h item 4 for why converting once after the + // guider instead is a different function on the DEFAULT arm (#1039). + // // EVERY FORWARD GOES THROUGH THIS ONE LAMBDA, and that is what makes // `video_stream_present` an OBSERVATION rather than a restatement. Written // as `result.video_stream_present = false` beside a `nullptr` literal it // would be a comment that compiles: a build that started passing a stream // would report `false` and stay green. Derived at the call, a mutation that // hands any forward a video stream flips it. - const auto forward = [&](const Ltx2ModalityInput* video, const Ltx2ModalityInput* audio, - const Ltx2DitPerturbation* p) { + const auto x0_model = [&](const Ltx2ModalityInput* video, const Ltx2ModalityInput* audio, + const Ltx2DitPerturbation* p, + std::vector* velocity_out = nullptr) { if (video != nullptr) result.video_stream_present = true; - return Ltx2DitForward(req.device, params, *req.dit_weights, video, audio, req.compute_dtype, - /*cache=*/nullptr, p); + const Ltx2DitOutputs out = Ltx2DitForward(req.device, params, *req.dit_weights, video, audio, + req.compute_dtype, /*cache=*/nullptr, p); + // The RAW velocity, before the conversion, recorded only where a caller + // asked for it. It is the other half of the pair that makes "which space + // did the guider combine" an arithmetic question — see the header. + if (velocity_out != nullptr) *velocity_out = out.audio; + // `to_denoised(audio.latent, ax, audio.timesteps)` (model.py:603). + return ToDenoised(latent, out.audio, timesteps, tokens, width); }; - const Ltx2DitOutputs cond = forward(/*video=*/nullptr, &ain, /*p=*/nullptr); + const std::vector cond = x0_model( + /*video=*/nullptr, &ain, /*p=*/nullptr, + step == 0 ? &result.first_step_velocity : nullptr); ++result.cond_forwards; - std::vector velocity = cond.audio; + std::vector denoised = cond; if (want_uncond || want_perturbed) { std::vector uncond_text; std::vector uncond_perturbed; if (want_uncond) { Ltx2ModalityInput nin = ain; nin.context = req.negative_context; - uncond_text = forward(/*video=*/nullptr, &nin, /*p=*/nullptr).audio; + uncond_text = x0_model(/*video=*/nullptr, &nin, /*p=*/nullptr); ++result.uncond_forwards; } if (want_perturbed) { @@ -331,26 +353,31 @@ Ltx2T2aResult Ltx2T2aGenerate(const Ltx2T2aRequest& req) { // perturbs the MODEL, never the conditioning (guiders.py:244-273 takes // `uncond_perturbed` from a forward whose `perturbations` differ and // whose context does not). - uncond_perturbed = forward(/*video=*/nullptr, &ain, &perturbation).audio; + uncond_perturbed = x0_model(/*video=*/nullptr, &ain, &perturbation); ++result.perturbed_forwards; } - velocity = Ltx2MultiModalGuidance(g, cond.audio.data(), + denoised = Ltx2MultiModalGuidance(g, cond.data(), want_uncond ? uncond_text.data() : nullptr, want_perturbed ? uncond_perturbed.data() : nullptr, /*uncond_modality=*/nullptr, - static_cast(cond.audio.size())); + static_cast(cond.size())); } // Kept for the next step's `should_skip_step` branch, which reuses it rather // than recomputing (utils/denoisers.py:85-91). - last_denoised = ToDenoised(latent, velocity, timesteps, tokens, width); - const std::vector& denoised = last_denoised; + if (step == 0) { + result.first_step_latent = latent; + result.first_step_cond = cond; + result.first_step_denoised = denoised; + result.first_step_sigma = static_cast(sigma); + } + last_denoised = std::move(denoised); // `EulerDiffusionStep()` — `DiffusionStage.__call__`'s own default // (utils/blocks.py:524-527), which T2A does not override (it passes no // `stepper`, t2a_one_stage.py:154-170). The ancestral sampler that // `distilled.py` selects for generation 2.5 reaches this pipeline through // nothing. - latent = Ltx2EulerStep(latent.data(), denoised.data(), sigmas.data(), sigma_count, step, + latent = Ltx2EulerStep(latent.data(), last_denoised.data(), sigmas.data(), sigma_count, step, static_cast(latent.size())); } diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index 30f939d7d..8651d3620 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -3662,6 +3662,11 @@ VideoResult Ltx2VideoEngine::GenerateAudioOnly(Impl& im, const VideoGenParams& g im.trace.t2a_uncond_forwards = rendered.uncond_forwards; im.trace.t2a_perturbed_forwards = rendered.perturbed_forwards; im.trace.t2a_perturbed_blocks = rendered.perturbed_blocks; + im.trace.t2a_first_latent = rendered.first_step_latent; + im.trace.t2a_first_velocity = rendered.first_step_velocity; + im.trace.t2a_first_cond = rendered.first_step_cond; + im.trace.t2a_first_denoised = rendered.first_step_denoised; + im.trace.t2a_first_sigma = rendered.first_step_sigma; im.trace.audio_tokens = rendered.audio_tokens; im.trace.audio_latent_digest = rendered.latent_digest; im.trace.audio_latent_absmax = rendered.latent_absmax; diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index a7e40fa59..9c9cb1657 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -21,6 +21,7 @@ #include #include +#include #include #include #include @@ -5259,3 +5260,208 @@ TEST_CASE("ltx2 t2a: the one_stage recipes noise their initial latent (#1013)") CHECK_FALSE(one.audio_only); } } + +TEST_CASE("ltx2 t2a: the guider is handed x0 predictions and not raw velocities") { + // #1039. Upstream hands the denoiser an `X0Model` (ltx-pipelines + // utils/blocks.py:480-482), so `_guided_denoise` combines DENOISED tensors: + // `all_v, all_a = transformer(...)` at utils/denoisers.py:188 and + // `audio_guider.calculate(cond_a, uncond_a, ptb_a, mod_a)` at `:203`, over an + // `X0Model.forward` that already applied `to_denoised(latent, v, timesteps)` + // (ltx-core model/transformer/model.py:590-604, `to_denoised` at + // ltx-core utils.py:39-52 — `sample - velocity * sigma`). + // + // This port combined raw DiT VELOCITIES and converted once afterwards. That is + // the same function only while `rescale_scale == 0`, because `calculate`'s + // linear terms are invariant under `x0 = latent - sigma*v`. The rescale branch + // is not: scaling the x0 by `factor` gives `factor*(latent - sigma*v)`, + // scaling the velocity gives `latent - sigma*factor*v`, and the two differ by + // `(factor - 1) * latent`. + // + // WHAT THIS CASE ASSERTS, AND WHY IT IS NOT THE RESCALE ARITHMETIC ITSELF. + // The rescale's numeric consequence is NOT resolvable on the reduced fixture, + // and that was MEASURED rather than assumed. The first draft of this case + // computed both candidate step-0 predictions in full — `factor * x0_pred` and + // `latent - sigma*factor_v*v_pred` — and its own separation guard refused + // them: this fixture's DiT responds to the conditioning at ~1e-5 of its own + // output, so `std(cond)/std(pred)` is 1.0 to 1e-5 in BOTH spaces, both + // factors land within 1e-5 of 1.0, and the two candidates sit 7.6e-07 apart + // against a span of 3.41. An assertion on that difference would be an + // assertion about f32 noise, and it would have been GREEN either way. + // + // So the rescale's consequence is gated at the seam by the case below, which + // measures 0.35 relative disagreement at `rescale_scale = 0.7` against + // 1.5e-07 at 0.0. This case gates what the fixture CAN decide exactly, which + // is the same defect one step earlier: WHICH TENSOR THE GUIDER WAS HANDED. + // + // WHAT MAKES THAT UNREACHABLE BY ACCIDENT — the sibling trap on this campaign + // was a test whose expectation a zero-filled stub also met. + // `cond == latent - sigma*velocity` is an equation between three recorded + // tensors, not a magnitude. It is exact in x0 space; in velocity space `cond` + // IS the velocity and the residual is `|latent - 2*sigma*velocity|`, i.e. the + // whole sample. No fixture scale satisfies it by accident, a zeroed velocity + // collapses it to `cond == latent` and is refused by the lower bound below, + // and a zeroed `cond` fails it outright. + Workspace ws; + const vllm::multimodal::VideoModelParams mp = T2aParams(ws.paths); + + // The arm this case sits on, pinned as a LOCAL fact before anything is read + // off a render: `rescale_scale = 0.7` on the 2.3/2.4/2.5 lineage + // (ltx-pipelines utils/constants.py:63, and the `--audio-rescale-scale` + // default at utils/args.py:1101-1106). + { + const vllm::Ltx2PipelineRecipe t2a = vllm::ResolveLtx2PipelineRecipe("t2a_one_stage", "2.5"); + REQUIRE(t2a.phases.size() == 1); + CHECK(t2a.phases[0].audio_guidance.rescale_scale == 0.7); + } + + // Through the production entry point — `LoadVideoEngine` then + // `VideoEngine::Generate`, which is what `vllm_video_generate` calls. Nothing + // here constructs a guider, a DiT or a modality by hand. NO extra is touched + // either, so the render takes the recipe's own guider and the assertion sits + // on the shipped path rather than on one this test configured into existence. + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + REQUIRE(engine != nullptr); + engine->Generate(T2aGen(ws.root + "/x0_space", "a b c")); + const auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + REQUIRE(t.t2a_rendered); + + const size_t n = t.t2a_first_latent.size(); + REQUIRE(n > 0); + REQUIRE(t.t2a_first_velocity.size() == n); + REQUIRE(t.t2a_first_cond.size() == n); + REQUIRE(t.t2a_first_denoised.size() == n); + const double sigma = t.t2a_first_sigma; + REQUIRE(sigma > 0.0); + + double latent_span = 0.0; + double velocity_span = 0.0; + double err_x0 = 0.0; // |cond - (latent - sigma*velocity)| -> 0 in x0 space + double err_v = 0.0; // |cond - velocity| -> 0 in velocity space + for (size_t i = 0; i < n; ++i) { + const double lat = static_cast(t.t2a_first_latent[i]); + const double vel = static_cast(t.t2a_first_velocity[i]); + const double cond = static_cast(t.t2a_first_cond[i]); + latent_span = std::max(latent_span, std::abs(lat)); + velocity_span = std::max(velocity_span, std::abs(vel)); + err_x0 = std::max(err_x0, std::abs(cond - (lat - sigma * vel))); + err_v = std::max(err_v, std::abs(cond - vel)); + } + + INFO("sigma = " << sigma << " max|latent| = " << latent_span + << " max|velocity| = " << velocity_span + << " |cond - (latent - sigma*velocity)| = " << err_x0 + << " |cond - velocity| = " << err_v << " elements = " << n); + + // 1. THE FIXTURE CAN DECIDE THIS. The two candidate tensors are + // `latent - sigma*velocity` and `velocity`; they coincide when the sample + // is zero and `to_denoised` is the identity when the velocity is. Both are + // REQUIREs, because the checks below mean nothing once either fails. + REQUIRE_MESSAGE(latent_span > 1e-3, + "the step-0 sample is zero, so the two candidate tensors coincide and nothing " + "below discriminates"); + REQUIRE_MESSAGE(sigma * velocity_span > 1e-6, + "the DiT returned no velocity, so `to_denoised` is the identity here and the " + "two candidate tensors coincide"); + // 2. THE GUIDER WAS HANDED THE X0 PREDICTION, exactly — `to_denoised` on the + // way out of the forward, which is `X0Model.forward` (model.py:602-603). + CHECK_MESSAGE(err_x0 <= 1e-5 * latent_span, + "the tensor handed to `Ltx2MultiModalGuidance` is not `latent - sigma*velocity`, " + "which is what `X0Model.forward` returns (#1039): residual " + << err_x0 << " against a tolerance of " << (1e-5 * latent_span)); + // 3. AND IT WAS NOT THE RAW VELOCITY. Said separately from check 2, because a + // build handing the guider some THIRD tensor fails 2 and would pass a lone + // "not the velocity" check; the pair says which of the two happened. + CHECK_MESSAGE(err_v > 1e-2 * latent_span, + "the tensor handed to `Ltx2MultiModalGuidance` IS the raw DiT velocity, so the " + "guidance is combined in velocity space and converted once afterwards (#1039)"); + // 4. And the guider MOVED what it was handed, so the tensor checked above is + // a real input to the combination rather than a recorded value beside one. + CHECK(t.t2a_first_denoised != t.t2a_first_cond); +} + +TEST_CASE("ltx2 t2a: rescale_scale 0 is the control because both spaces agree there") { + // #1039's control, executable rather than asserted in prose. The case above + // would be testing something OTHER than the defect if it also fired at + // `rescale_scale = 0`, because `MultiModalGuider.calculate`'s linear terms + // (guiders.py:261-266) are invariant under `x0 = latent - sigma*v`: + // + // latent - sigma*(c + a(c-u) + b(c-p)) == x0c + a(x0c-x0u) + b(x0c-x0p) + // + // The rescale at `:268-271` is the only part that is not. This case measures + // both, on the real seam, with a latent that makes the difference visible. + const int64_t n = 512; + std::vector latent(static_cast(n)); + std::vector v_cond(static_cast(n)); + std::vector v_uncond(static_cast(n)); + std::vector v_ptb(static_cast(n)); + // Deterministic and NON-CONSTANT. A zero latent erases `(factor - 1) * latent` + // entirely and a constant one reduces it to a uniform offset; either would + // make the disagreement below unmeasurable and the control meaningless. + uint64_t s = 0x9E3779B97F4A7C15ULL; + const auto next = [&s]() { + s ^= s << 13; + s ^= s >> 7; + s ^= s << 17; + return static_cast(static_cast(s >> 11) / 9007199254740992.0 * 2.0 - 1.0); + }; + for (int64_t i = 0; i < n; ++i) { + const size_t j = static_cast(i); + latent[j] = 2.0F * next(); + v_cond[j] = next(); + v_uncond[j] = next(); + v_ptb[j] = next(); + } + const float sigma = 0.83F; + std::vector x_cond(static_cast(n)); + std::vector x_uncond(static_cast(n)); + std::vector x_ptb(static_cast(n)); + for (int64_t i = 0; i < n; ++i) { + const size_t j = static_cast(i); + x_cond[j] = latent[j] - sigma * v_cond[j]; + x_uncond[j] = latent[j] - sigma * v_uncond[j]; + x_ptb[j] = latent[j] - sigma * v_ptb[j]; + } + + vllm::Ltx2MultiModalGuiderParams params; + params.cfg_scale = 7.0; // the T2A defaults (utils/constants.py:58-66) + params.stg_scale = 1.0; + params.modality_scale = 1.0; + params.skip_step = 0; + + const auto compare = [&](double rescale) { + params.rescale_scale = rescale; + // Upstream's shape: combine the X0 predictions. + const std::vector x0_space = vllm::Ltx2MultiModalGuidance( + params, x_cond.data(), x_uncond.data(), x_ptb.data(), /*uncond_modality=*/nullptr, n); + // The shape this port shipped: combine the VELOCITIES and convert once after. + const std::vector v_space = vllm::Ltx2MultiModalGuidance( + params, v_cond.data(), v_uncond.data(), v_ptb.data(), /*uncond_modality=*/nullptr, n); + double worst = 0.0; + double scale = 0.0; + for (int64_t i = 0; i < n; ++i) { + const size_t j = static_cast(i); + const double converted = static_cast(latent[j]) - + static_cast(sigma) * static_cast(v_space[j]); + worst = std::max(worst, std::abs(static_cast(x0_space[j]) - converted)); + scale = std::max(scale, std::abs(static_cast(x0_space[j]))); + } + REQUIRE(scale > 1e-3); + return worst / scale; + }; + + const double at_zero = compare(0.0); + const double at_default = compare(0.7); + INFO("relative disagreement: at rescale 0.0 = " << at_zero + << " at rescale 0.7 = " << at_default); + // AT 0.0 THE TWO SPACES ARE THE SAME FUNCTION, to f32 rounding. An assertion + // that fires here is not about #1039. + CHECK(at_zero < 1e-4); + // AT THE SHIPPED 0.7 THEY ARE NOT, by orders of magnitude more. That is the + // whole of the defect, and it is why the case above can sit on the default. + CHECK(at_default > 1e-2); + CHECK(at_default > 100.0 * at_zero); +} From 22267d794c6f4c1b9a9613b2f6aab31f66ed9138 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 16 Aug 2026 18:49:18 +0000 Subject: [PATCH 8/9] fix(LTX25-T2A-ONE-STAGE): the #1039 gate covered one of three guidance arms (#1005, #1039) `ltx2_t2a.cpp:41-43` says `to_denoised` is applied to EVERY PASS. The gate that landed with the #1039 fix held that claim for the CONDITIONAL pass only: it recorded `first_step_velocity` and `first_step_cond` for that arm, nothing observed the unconditional or perturbed forwards, and nothing pinned what `Ltx2EulerStep` consumed. The default T2A arm runs three forwards per step, so a build that converts `cond` correctly and leaves either other arm in velocity space renders a different waveform through a guider whose `cond` term is impeccable, with a healthy forward count and nothing else to see it by. That is #1039 again, one arm over. Found by the fresh review of `c1fe35592`, which passed on the correctness of the fix and returned this as its one blocking finding. Reproduced before the repair on the same comma-free filter as the green run (`--test-case=ltx2 t2a*`, 10 cases / 526 assertions / exit 0 at `c1fe35592`). Each mutation applied to ONE file, rebuilt with the `: error:` count printed beside the verdict, exit code captured directly, and restored from a content SNAPSHOT with `os.utime(now)` and a sha256 compare. | Mutation | git diff --stat | BUILT | before | after | |---|---|---|---|---| | A1 the PERTURBED pass alone left in velocity space | 4 ++-- | YES (0 errors) | SURVIVED exit 0, 10/526 | DETECTED exit 1, 10/548 | | A2 the UNCONDITIONAL pass alone left in velocity space | 4 ++-- | YES (0 errors) | SURVIVED exit 0, 10/526 | DETECTED exit 1, 10/548 | | A3b `ToDenoised` applied twice, BELOW the step-0 record | 2 +- | YES (0 errors) | SURVIVED exit 0, 10/526 | DETECTED exit 1, 10/548 | | A3c `ToDenoised` applied twice, ABOVE the step-0 record | 1 + | YES (0 errors) | SURVIVED exit 0, 10/526 | DETECTED exit 1, 10/548 | | A4 the perturbed arm's recorded velocity ZEROED (the guard) | 1 + | YES (0 errors) | field did not exist | DETECTED exit 1, 10/538 | | N1 the original #1039 shape, restored in full | 5 ++--- | YES (0 errors) | DETECTED | DETECTED exit 1, 10/548 | A3c is not from the review. It was found while closing A3b: the reviewer's placement sits between the step-0 record and the Euler step, so recovering the Euler input sees it, and moving the same edit one statement earlier does not. Closing both needs two independent checks rather than one. N1's first draft dropped `ToDenoised`'s only call site and failed to build on `-Werror=unused-function`. A mutation that does not build reads as a passing test, so it is written as two edits that keep the function used. The same trap took the reviewer's own R1'. OBSERVABILITY, NOT A CHANGE TO THE FIX. `Ltx2T2aResult` and `Ltx2ConditioningTrace` gain a (raw velocity, x0 prediction) pair for the unconditional and perturbed arms, and the latent the Euler step wrote. The uncond and perturbed vectors stay EMPTY when the guider does not ask for that arm, because the forward did not run; a zero-filled one of the right length would be indistinguishable from a forward that returned zeros. Three checks over them, all in the existing end-to-end case through `LoadVideoEngine` and `VideoEngine::Generate`: - the SAME equation `x0 == latent - sigma*velocity` on every arm the render ran, exact in x0 space and off by the whole sample in velocity space, with `t2a_uncond_forwards > 0` and `t2a_perturbed_forwards > 0` asserted first so a silently skipped arm cannot vacate its own check; - the guider's output REPLAYED through the shipped `Ltx2MultiModalGuidance` over the three recorded arms, required bit-equal to `t2a_first_denoised`. This does not gate the guider's arithmetic; it gates that the pipeline handed it these tensors and passed its result on untouched, which is what A3c moves and no per-arm check can see; - `t2a_first_next_latent` recovered from `t2a_first_denoised` through `x + (x - denoised)/sigma * (sigma_next - sigma)`, the schedule re-derived from `Ltx2SigmaSchedule` and tied to the render by the sigma it recorded. That is what A3b moves. NON-VACUITY, PER ARM RATHER THAN ONCE. `latent_span > 1e-3` stays shared, since a zero sample makes the two candidate tensors coincide on every arm. Its partner `sigma * velocity_span > 1e-6` moves INSIDE the per-arm loop, because a zero velocity makes `to_denoised` the identity for that arm alone, and "expected zero, and a stub also produces zero" is the trap this campaign has already hit twice. A4 is the mutation that proves the guard is armed rather than decorative: zeroing one arm's recorded velocity takes the case red through the `REQUIRE`, at 538 assertions rather than 548 because the `REQUIRE` aborts the case. The replay check carries its own control (`t2a_first_denoised != t2a_first_cond`, so the guider MOVED what it was handed) and the Euler check carries two (`|dt| > 1e-3` and `scale > 1e-3`). THE RESCALE'S NUMERIC DIFFERENCE IS STILL NOT ASSERTED, and the reason was re-measured rather than inherited. `std(cond)/std(pred)` is 1 to printed precision on this fixture, so `factor = 0.7*1 + 0.3` is exactly 1, the rescale is a no-op in BOTH spaces, and the difference term `(factor - 1) * latent` is identically zero. Owed against the real-checkpoint render, unchanged. RECORDS, all verified against the tree before they were written. The LTX-2.5 CHECKPOINT PIN is missing campaign-wide, and #1048 now says so. `docs/USAGE.md` names six LTX-2.5 artifacts by bare file name with no repo, no revision and no sha256, at `:663-670` and `:2183-2188` on `origin/main`, plus the text-to-audio recipe at `:853-857`. `grep -n sha256 docs/USAGE.md` returns two checkpoint hashes and both belong to MiniMax-Music3. Recorded and NOT fabricated: no LTX-2.5 row claims a render on real weights, so there is no checkpoint any of them was gated against to pin. The recipe's `--audio-vae` is corrected to `ltx-2.5-audio-vae-bf16.safetensors`, which is what the other two LTX-2.5 recipes on the page name. SPEC 6b OVERCLAIMED. It said this row ends a test-only driver for four symbols. Only `Ltx2MultiModalGuidance` gains a production call site. `Ltx2CfgDelta` and `Ltx2StgDelta` are reachable solely through `Ltx2Guidance`, whose only caller is `tests/vllm/models/test_ltx2_pipeline.cpp:710`, and `Ltx2BatchedPerturbationConfig` is constructed nowhere outside that same file. Three of the four remain dead. This row does not owe the wiring, which is pre-existing from #641; the spec must not assert what `git grep` refutes. #1049. THE READER ANCHORS RELOCATION REASON WAS FALSE. The change does not only append below line 1231: the `ltx2_t2a.h` include at `@@ -36,6 +36,7` shifts every anchor by one and the audio-only video-VAE exception at `@@ -974,8 +975,21` adds thirteen more, which is why the list moved from `781 791 792 ...` to `782 792 793 ...`. The anchors were correctly re-derived and the gate passes 23/23; only the stated reason was wrong, and a false reason is what makes the next reader skip the re-derivation. #1050 files a defect this repair deliberately does NOT fix. `ltx2_pipeline.cpp:505-506` says torch's unbiased (N-1) `std` matters and the biased one "would be a small, everywhere, resolution-dependent gain error". `factor = std(cond)/std(pred)` divides two `std`s over the same count, so the `(n-1)` cancels exactly. The review's mutation from unbiased to biased survived because it is an IDENTITY, not because the gate is blind. The code is right; the comment is the defect, and it is outside this row's scope. No GPU result is claimed. `dgx.casa` is down, so there is no render on real weights here. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 3 + .agents/specs/ltx25-t2a-one-stage.md | 126 +++++++++- docs/FEATURES.md | 4 +- docs/USAGE.md | 14 +- include/vllm/model_executor/models/ltx2_t2a.h | 47 +++- include/vllm/multimodal/ltx2_video.h | 23 ++ src/vllm/model_executor/models/ltx2_t2a.cpp | 17 +- src/vllm/multimodal/ltx2_video.cpp | 5 + tests/vllm/multimodal/test_ltx2_video.cpp | 225 ++++++++++++++---- 9 files changed, 398 insertions(+), 66 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 6cf34d5ac..d64bcd803 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -277,3 +277,6 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1013](https://github.com/mudler/vllm.cpp/issues/1013) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 `one_stage` denoised from ZEROS. `OneStagePhase` (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1066 @ 332aed738`) left `Ltx2PhaseRecipe::noise_scale` at the struct default of 0.0, and 0.0 is not "no extra noise": `Ltx2GaussianNoise` is `latent + noise_scale * (noise - latent)` (`:218 @ 332aed738`), so the state stayed exactly as `create_initial_state` wrote it, which with no initial latent is all zeros. Upstream `ModalitySpec.noise_scale` defaults to 1.0 (`ltx-pipelines/utils/types.py:110 @ fd4ded7f`) and `TI2VidOneStagePipeline.__call__` constructs BOTH specs without it (`ti2vid_one_stage.py:233-239`); the two neighbouring recipes set it explicitly, which is what made the omission legible. No gate saw it because every end-to-end test loads `distilled_two_stage`, and a zero-initialized denoise still returns a finite clip of the right size, frame count and sample rate. FOUND and FIXED in flow by row `LTX25-T2A-ONE-STAGE`, whose `t2a_one_stage` rows are built FROM `OneStageRecipe` and would have inherited it. `dmd2` leaves the same field at 0.0 and is deliberately NOT corrected by analogy (its source is vLLM-Omni, not checked out here); that half is listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | | [#1031](https://github.com/mudler/vllm.cpp/issues/1031) | `ENG-EXPERT-STREAM` | CLOSED as a duplicate of [#1022](https://github.com/mudler/vllm.cpp/issues/1022), and this row is the corrected one rather than the filed one. As filed it said `check-agent-record` and `test_check_agent_record` are RED on `origin/main` because `.agents/issue-index.md` lists issue #995 twice, and that the repair needed a contract decision plus a checker-semantics spec. It did not: [#1022](https://github.com/mudler/vllm.cpp/issues/1022) had already read the two #995 rows and found neither WELL-FORMED, and `ff264cb82` (PR [#1025](https://github.com/mudler/vllm.cpp/pull/1025)) landed the repair on `main` before this branch merged it. Measured at this branch's head rather than inferred from the merge: `python3 scripts/check-agent-record.py` prints `agent record OK: ENGINE=156 MODEL=377 QUANT=82 KERNEL=51 BACKEND=83` and exits 0. CORRECTED IN PLACE, and that is a narrow exception argued here rather than a licence: the row had not landed, this branch added it, and the net diff against `origin/main` is still additions only — which `scripts/check-issue-index-append-only.py --base origin/main` is what checks. Once it lands `merge=union` makes it permanent and un-correctable, so leaving a filed-and-refuted claim in the record was the more expensive option | bug | | [#1039](https://github.com/mudler/vllm.cpp/issues/1039) | `LTX25-T2A-ONE-STAGE` | LTX-2.5 T2A combined its guidance passes in VELOCITY space. Upstream hands the denoiser an `X0Model` (`ltx-pipelines/utils/blocks.py:480-482 @ fd4ded7f`) whose `forward` returns `to_denoised(latent, v, timesteps)` (`ltx-core/model/transformer/model.py:601-604`, `to_denoised` at `ltx-core/utils.py:39-52`), so `_guided_denoise` combines DENOISED tensors — `all_v, all_a = transformer(...)` at `utils/denoisers.py:188` and `audio_guider.calculate(...)` at `:203`. `Ltx2T2aGenerate` combined raw DiT velocities and applied `ToDenoised` once afterwards. `MultiModalGuider.calculate`'s LINEAR terms are invariant under `x0 = latent - sigma*v`, so the two forms agree exactly while `rescale_scale == 0`; the RESCALE branch (`guiders.py:268-271`) is not, because upstream's `std(x0_cond)/std(x0_pred)` scales the whole x0 to `f*(latent - sigma*v)` where scaling the velocity gives `latent - sigma*f*v` — a difference of `(f - 1) * latent`, non-zero wherever the latent is, which on this path is everywhere. `rescale_scale = 0.7` is the SHIPPED T2A default (`utils/constants.py:63`, `utils/args.py:1101-1106`), so every default render took the divergent branch. No gate saw it: the forward counts, perturbed blocks, latent absmax, waveform length, channel count and sample rate are identical between the two forms. FOUND by review of PR [#1032](https://github.com/mudler/vllm.cpp/pull/1032) and FIXED in flow on the same branch, before it landed. The VIDEO arm is unaffected and the reason is recorded rather than assumed: `Ltx2MultiModalGuidance` has exactly ONE production caller and the joint driver runs a single unguided forward per step. Spec [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | +| [#1048](https://github.com/mudler/vllm.cpp/issues/1048) | — | LTX-2.5 ships with NO checkpoint pin. `docs/USAGE.md` names six LTX-2.5 artifacts by bare file name and gives no HuggingFace repo, no revision and no sha256 for any of them: `:663-670` and `:2183-2188` on `origin/main` at `d1b0ea3a8`, plus the text-to-audio recipe [#1005](https://github.com/mudler/vllm.cpp/issues/1005) added at `:853-857`. AGENTS.md § *Say which weights, and from where* requires file name, size, exact repo AND revision, grouped by arm, with a sha256 for a quantized artifact. Campaign-wide and PRE-EXISTING rather than introduced by that row, measured rather than asserted: `grep -n sha256 docs/USAGE.md` returns two checkpoint hashes and BOTH belong to MiniMax-Music3 (`:3127`, `:3269`), while MiniMax-H3 (`:1950-1993`) and MiniMax-Music3 (`:3123-3149`) each carry a full repo + revision + sha256 table and LTX-2.5 carries none anywhere. RECORDED AND DELIBERATELY NOT FABRICATED: no LTX-2.5 row in the campaign claims a render on real weights — `dgx.casa` is down and every LTX-2.5 gate runs on a reduced fixture — so there is no checkpoint any of them was gated against to pin, and inventing a repo id would be worse than the gap. The real-checkpoint render owed by [#644](https://github.com/mudler/vllm.cpp/issues/644) is what closes it. Two smaller things fall out of the same gap and belong to the same change: the t2a recipe names `ltx-2.5-dit.safetensors`, a file name appearing nowhere else in the tree, where every other LTX-2.5 recipe names `ltx-2.5-22b-distilled-fp8.safetensors` plus the `--dit-config` its missing `__metadata__` requires; and whether that recipe runs at all without `--dit-config` is unverified. Found repairing the fresh review of [#1039](https://github.com/mudler/vllm.cpp/issues/1039) on PR [#1032](https://github.com/mudler/vllm.cpp/pull/1032). Listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | +| [#1049](https://github.com/mudler/vllm.cpp/issues/1049) | — | `Ltx2Guidance` (`src/vllm/model_executor/models/ltx2_pipeline.cpp:526 @ c1fe35592`) is dead in production: `git grep -n 'Ltx2Guidance(' -- src include examples tests` returns the declaration (`ltx2_pipeline.h:330`), the definition, and ONE call, at `tests/vllm/models/test_ltx2_pipeline.cpp:710`. It is also the only path to two ported guiders — `Ltx2CfgDelta` (`ltx2_pipeline.cpp:532`, plus `test_ltx2_pipeline.cpp:615`) and `Ltx2StgDelta` (`:534`, plus `:630`) — so both are reachable from no product entry point either. `Ltx2BatchedPerturbationConfig` (`ltx2_pipeline.h:380`) is the same shape from a different direction: constructed only at `test_ltx2_pipeline.cpp:832-859`, while the LTX-2.5 text-to-audio path perturbs through `Ltx2DitPerturbation`, a different type, and no other path perturbs at all. This is the test-only-driver shape [`reachability.md`](reachability.md) names. All four landed with #641. `Ltx2MultiModalGuidance` was the fourth member of the set and is no longer one: [#1005](https://github.com/mudler/vllm.cpp/issues/1005) gave it a production call site in `ltx2_t2a.cpp`. Closing this means either routing a product path through `Ltx2Guidance` with a configured `Ltx2GuiderKind` — `Ltx2GuiderSigmaBin` and `Ltx2GuiderParamsForSigma` are already ported beside it — or retiring the unreached arms into `.agents/completed/` with their provenance. Found repairing the fresh review of [#1039](https://github.com/mudler/vllm.cpp/issues/1039) on PR [#1032](https://github.com/mudler/vllm.cpp/pull/1032), where an earlier draft of that row's spec §6b claimed the row ended all four test-only drivers; §6b now carries the measured table. Listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | +| [#1050](https://github.com/mudler/vllm.cpp/issues/1050) | — | The guider rescale's `std` comment states a consequence that cannot exist. `src/vllm/model_executor/models/ltx2_pipeline.cpp:505-506 @ c1fe35592`, repeated at `include/vllm/model_executor/models/ltx2_pipeline.h:319-322`, says torch's `std` is the UNBIASED (N-1) estimator by default and "the biased one would be a small, everywhere, resolution-dependent gain error that no shape or finiteness check can see". `factor_raw` is `unbiased_std(cond) / unbiased_std(pred)`, two `std`s over the SAME `count`, so the divisor cancels exactly: `sqrt(ss_c/(n-1))/sqrt(ss_p/(n-1)) == sqrt(ss_c/ss_p) == sqrt(ss_c/n)/sqrt(ss_p/n)`. There is no gain error, small or otherwise, and nothing about it is resolution-dependent; the two forms differ only by f32 rounding in the divide. Worth a record rather than a silent correction because the comment tells the next reader a gate is needed there and it is not: the fresh review of [#1039](https://github.com/mudler/vllm.cpp/issues/1039) mutated the estimator to the biased form and it survived — correctly, because it is an IDENTITY — and a survivor at that site otherwise reads as a blind instrument and costs another investigation. The CODE is right as written and should stay `unbiased_std`, because the name is what mirrors torch even where the ratio does not care; the COMMENT is the defect. Pre-existing from `cefacd2d0` (#641). Found repairing that review on PR [#1032](https://github.com/mudler/vllm.cpp/pull/1032) and out of scope there under its explicit exclusions. Listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | diff --git a/.agents/specs/ltx25-t2a-one-stage.md b/.agents/specs/ltx25-t2a-one-stage.md index 4d4f1d6a0..7749461a3 100644 --- a/.agents/specs/ltx25-t2a-one-stage.md +++ b/.agents/specs/ltx25-t2a-one-stage.md @@ -326,11 +326,22 @@ a checkpoint the pipeline cannot use. `audio_vae_path` stays required, because **The `READER ANCHORS` gate.** `ltx2_video.cpp` carries a derived line-number list re-derived and string-compared by `test_ltx2_video.cpp`. Any line inserted above the last anchored line shifts it, and a clean `git merge` will not warn. -Mitigation: the engine edit is a branch at the TOP of `Generate`, which is below -every anchor; the list is re-derived at the final tree after the last merge of +Mitigation: the list is re-derived at the final tree after the last merge of `origin/main` with the test's own walk rather than by arithmetic, and it is named as a merge hazard in the PR body. +**AND THE ANCHORS DID MOVE.** The mitigation above once claimed the engine edit +is "a branch at the TOP of `Generate`, which is below every anchor", and the +list is `781 791 792 854 950 966 968 1046 1071 1176 1217` on `origin/main` +against `782 792 793 855 951 967 969 1060 1085 1190 1231` here. Two hunks sit +ABOVE the last anchor: the `ltx2_t2a.h` include at `@@ -36,6 +36,7`, which shifts +every anchor by one, and the audio-only video-VAE exception at +`@@ -974,8 +975,21`, which adds thirteen more and moves the last four by +fourteen. `@@ -1018,7 +1032,7` is above 1231 as well and is net zero. The +anchors were correctly RE-DERIVED with the test's own walk and the gate passes +23/23, so the outcome is right; only the stated reason was false, and a false +reason is what makes the next reader skip the re-derivation. + **A silently wrong render.** Every failure mode here produces a playable WAV of the right length: a guider that never runs its uncond pass, an STG pass that perturbs the wrong block, a `video = nullptr` path that silently reintroduces v2a @@ -527,10 +538,26 @@ defect of the route rather than of this row, and the reach claim above is worded to exclude it. Whether the route's LOAD path was exercised end to end here is recorded in the final report as measured or unverified — it is not asserted. -**This row ends a test-only-driver.** `Ltx2MultiModalGuidance`, `Ltx2CfgDelta`, -`Ltx2StgDelta` and `Ltx2BatchedPerturbationConfig` have had test call sites and -no product one since they landed, which is exactly the shape -`.agents/reachability.md` names. +**This row ends ONE test-only driver, and leaves three standing.** An earlier +draft of this section claimed all four, and the tree contradicts it: + +| Symbol | Production call site after this row | +|---|---| +| `Ltx2MultiModalGuidance` | **yes** — `src/vllm/model_executor/models/ltx2_t2a.cpp`, in the guided step. This is the one this row ends | +| `Ltx2CfgDelta` | no — reached only through `Ltx2Guidance` (`ltx2_pipeline.cpp:532`) and from `tests/vllm/models/test_ltx2_pipeline.cpp:615` | +| `Ltx2StgDelta` | no — the same, at `ltx2_pipeline.cpp:534` and `test_ltx2_pipeline.cpp:630` | +| `Ltx2BatchedPerturbationConfig` | no — constructed nowhere outside `test_ltx2_pipeline.cpp:832-859`. T2A uses `Ltx2DitPerturbation`, which is a different type | + +`Ltx2Guidance` itself is the reason the two deltas stay dead: its only caller in +the tree is `tests/vllm/models/test_ltx2_pipeline.cpp:710`, so the switch that +would route a configured guider kind to them is reached from no product path. + +**This row does not owe that wiring.** All four landed with #641, before this row +existed, and none of them is on the T2A path — T2A resolves a +`Ltx2MultiModalGuiderParams` and calls `Ltx2MultiModalGuidance` directly, exactly +as `_guided_denoise` does upstream. Recorded here because the spec must not +assert what `git grep` refutes, and listed under `## Owed` with the issue that +tracks it. ## 7. Quantized arms @@ -696,6 +723,63 @@ tests/vllm/multimodal/test_ltx2_video.cpp:5378: ERROR: `Ltx2MultiModalGuidance` WAS the raw DiT velocity. Green after, on the same filter: 1 case, 16 assertions, 0 failed, exit 0. +### The #1039 gate covered ONE of the three guidance arms + +The fresh review of `c1fe35592` passed on the correctness of the fix and +returned one blocking finding: the case above recorded `first_step_velocity` and +`first_step_cond` for the CONDITIONAL pass, nothing observed the other two arms, +and nothing pinned what `Ltx2EulerStep` consumed. `ltx2_t2a.cpp:41-43` claims the +conversion is applied to EVERY PASS, and the gate held that claim for one third +of them. + +Reproduced before the repair, on the same filter as the green run +(`--test-case=ltx2 t2a*`, comma-free, 10 cases / 526 assertions at +`c1fe35592`). Each mutation applied to ONE file, `git diff --stat` taken against +the pre-mutation working tree rather than against `HEAD`, rebuilt with the +`: error:` count printed beside the verdict, exit code captured directly, and +restored from a content snapshot with `os.utime(now)` and a sha256 compare. + +| Mutation | before the repair | after | +|---|---|---| +| A1 the PERTURBED (STG) pass alone left in velocity space | SURVIVED, exit 0, 10/526 | DETECTED, exit 1 | +| A2 the UNCONDITIONAL pass alone left in velocity space | SURVIVED, exit 0, 10/526 | DETECTED, exit 1 | +| R1b `ToDenoised` applied a SECOND time to the guider's output, between the step-0 record and the Euler step | SURVIVED, exit 0, 10/526 | DETECTED, exit 1 | +| R1c the same double application ABOVE the step-0 record, so the recorded `t2a_first_denoised` is itself doubly converted | SURVIVED, exit 0, 10/526 | DETECTED, exit 1 | +| A4 the perturbed arm's recorded velocity ZEROED — the guard, not a defect | n/a (the field did not exist) | DETECTED, exit 1, by the `REQUIRE` | + +R1c is not from the review. It was found while closing R1b: the reviewer's R1b +sits between the record and the step, so the recovered-Euler-input check sees +it, and moving the same edit one statement earlier does not. That is why the +repair adds a second, independent check rather than one. + +**The repair is observability plus three checks, not a change to the fix.** +`Ltx2T2aResult` and the trace gain a (velocity, x0) pair for the unconditional +and perturbed arms and the latent the Euler step wrote. The case then applies the +SAME equation to every arm, replays `Ltx2MultiModalGuidance` over the three +recorded arms and requires bit equality with `t2a_first_denoised`, and recovers +`t2a_first_next_latent` from `t2a_first_denoised` through the Euler formula. + +**Non-vacuity, per arm rather than once.** `latent_span > 1e-3` stays shared — +a zero sample makes the two candidate tensors coincide on every arm. Its partner +`sigma * velocity_span > 1e-6` moves INSIDE the per-arm loop, because a zero +velocity makes `to_denoised` the identity for that arm alone, and "expected zero, +and a stub also produces zero" is the trap this campaign has hit twice. A4 is the +mutation that proves that guard is armed: zeroing one arm's recorded velocity +takes the case red through the `REQUIRE`, at 538 assertions rather than 548 +because the `REQUIRE` aborts the case. The replay check adds its own +(`t2a_first_denoised != t2a_first_cond`, the guider MOVED what it was handed) and +the Euler check adds two (`|dt| > 1e-3`, so the step is not the identity, and +`scale > 1e-3`, so the residual bounds something). + +**The rescale's numeric difference is still NOT asserted, and the reason was +re-measured.** `std(cond)/std(pred)` is 1 to printed precision on this fixture, +so `factor = 0.7*1 + 0.3 = 1`, the rescale is an exact no-op in BOTH spaces, and +the difference term `(factor - 1) * latent` is identically zero. Owed against the +real-checkpoint render, as before. + +Green after: `--test-case=ltx2 t2a*` at 10 cases / **548** assertions / 0 failed +/ exit 0, up from 526. + ## Owed - **The rescale's numeric consequence END TO END.** Gated at the seam (0.352 @@ -738,6 +822,36 @@ filter: 1 case, 16 assertions, 0 failed, exit 0. - **The `dmd2` recipe's `noise_scale`**, see §7b. - **A real-checkpoint T2A render.** Gated on the reduced fixture only; the GPU was out of bounds for this row. +- **THE LTX-2.5 CHECKPOINT PIN.** `docs/USAGE.md` names six LTX-2.5 artifacts by + bare file name and gives no HuggingFace repo, no revision and no sha256 for + any of them — `:663-670` and `:2183-2188` on `origin/main` at `d1b0ea3a8`, + plus the text-to-audio recipe this row added at `:853-857`. AGENTS.md + § *Say which weights, and from where* requires all three, per arm. It is + campaign-wide and pre-existing rather than introduced here: `grep -n sha256 + docs/USAGE.md` returns two checkpoint hashes and both belong to + MiniMax-Music3 (`:3127`, `:3269`), while MiniMax-H3 (`:1950-1993`) and + MiniMax-Music3 (`:3123-3149`) each carry a full table and LTX-2.5 carries + none. RECORDED AND NOT FABRICATED: this row claims no render on real weights, + so there is no checkpoint it was gated against to pin, and inventing a repo id + would be worse than the gap. The real-checkpoint render owed above is what + closes it. Tracked by + [#1048](https://github.com/mudler/vllm.cpp/issues/1048). +- **`Ltx2Guidance` and the two deltas it gates are DEAD in production**, and so + is `Ltx2BatchedPerturbationConfig`. See §6b for the measured table. All four + landed with #641, none is on the T2A path, and this row ends only + `Ltx2MultiModalGuidance`'s test-only-driver state. Tracked by + [#1049](https://github.com/mudler/vllm.cpp/issues/1049). +- **The guider rescale's `std` comment states an impossible consequence.** + `ltx2_pipeline.cpp:505-506` and `ltx2_pipeline.h:319-322` say torch's unbiased + (N-1) `std` matters and the biased one "would be a small, everywhere, + resolution-dependent gain error". `factor = std(cond)/std(pred)` divides two + `std`s over the same count, so the `(n-1)` cancels exactly and the two + estimators give the same ratio. A review mutation from biased to unbiased + survived because it is an IDENTITY, not because the gate is blind — which is + worth writing down, because a survivor at that site otherwise reads as a blind + instrument and costs another investigation. The CODE is right; the COMMENT is + the defect. Pre-existing from `cefacd2d0` (#641) and out of this row's scope. + Tracked by [#1050](https://github.com/mudler/vllm.cpp/issues/1050). - **Value goldens from executed upstream for the T2A COMPOSITION.** The bricks either side have them; the chain does not. What would close it is a section in `scripts/gen-ltx2-pipeline-goldens.py` that instantiates diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 66e1dac85..b6e1d9864 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -170,8 +170,8 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 tiled + streaming Conv VAE decode | LTX-2.5 video VAE | gated vs executed upstream `ltx_core` @ `fd4ded7f` (`test_ltx2_tiling` 10/10, 915 assertions); one-tile and untiled-spatial controls BIT-EXACT vs untiled on both causality arms; an untiled frames axis is REFUSED | Streams temporal chunks through upstream's AUTO layout (768/64 px, 80/24 frames); above one tile the pixel volume is never materialized. NO-OP below 768px and 81 frames; 81-120 IS tiled, differing 6.70% of range | | LTX-2.5 Conv VAE decode arithmetic width | LTX-2.5 video VAE | `test_ltx2_vae` "the decode's convolution accumulates in f32", entering through `Ltx2VideoDecodeStreaming`; widening the accumulator to `double`, or deleting the production call site, each turns it RED | **f32**, the width `F.conv3d` uses at f32 AND bf16 (MEASURED). Was f64 at 8 sites ([#1008](https://github.com/mudler/vllm.cpp/issues/1008)). Conv sums BLOCKED per input channel, as torch's. STORAGE stays f32; bf16 owed | | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated | -| LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 13 mutations, 12 DETECTED (one of them a skipped guidance step that recomputes instead of reusing) and the 13th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only GUIDED arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | -| LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions and not raw velocities" through `Generate`, plus a seam case measuring the two spaces 1.50e-07 apart at `rescale_scale = 0.0` and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 ([#1039](https://github.com/mudler/vllm.cpp/issues/1039)) | +| LTX-2.5 text-to-audio (`T2AOneStagePipeline`) | LTX-2.5 DiT + audio VAE, no video VAE | `test_ltx2_video`'s `ltx2 t2a:` cases, entering through `Generate`; 18 mutations, 17 DETECTED (four by review of a conditional-only #1039 gate) and the 18th proven an identity, not a blind spot | `--pipeline-kind t2a_one_stage`. NO picture: 0 frames, no mux argv. The only GUIDED arm (CFG + STG, 3 forwards/step), so it needs a text tower. CPU only; the device forward is refused by name | +| LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 ([#1039](https://github.com/mudler/vllm.cpp/issues/1039)) | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending | diff --git a/docs/USAGE.md b/docs/USAGE.md index 17e3bd7ee..320d1e6b2 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -851,13 +851,25 @@ is nothing to mux. ```sh ltx2-gen --dit ltx-2.5-dit.safetensors \ - --audio-vae ltx-2.5-audio-vae.safetensors \ + --audio-vae ltx-2.5-audio-vae-bf16.safetensors \ --encoder gemma4-12b-with-proj.safetensors --encoder-config gemma4.json \ --pipeline-kind t2a_one_stage --device cpu \ --frames 121 --steps 30 --prompt "rain on a tin roof, distant thunder" \ --workdir /tmp/t2a ``` +**These file names are not a checkpoint pin, and no LTX-2.5 recipe in this +document is.** None of them names a HuggingFace repo, a revision or a sha256, +which AGENTS.md § *Say which weights, and from where* requires; MiniMax-H3 and +MiniMax-Music3 below each carry a full table and LTX-2.5 carries none. That is +campaign-wide and pre-existing rather than particular to this recipe, and it is +recorded rather than invented, because no LTX-2.5 arm here has been rendered on +real weights yet. Tracked by +[#1048](https://github.com/mudler/vllm.cpp/issues/1048); read `--dit` above as +"the LTX-2.5 transformer", which the other recipes on this page spell as +`ltx-2.5-22b-distilled-fp8.safetensors` together with the `--dit-config` its +missing `__metadata__` requires. + **No `--video-vae` is needed**, and none is loaded: upstream's pipeline never constructs a video VAE. `--width` and `--height` are **refused** rather than ignored — upstream passes a 512x512 placeholder whose height and width it diff --git a/include/vllm/model_executor/models/ltx2_t2a.h b/include/vllm/model_executor/models/ltx2_t2a.h index 12b13ae23..0fc064663 100644 --- a/include/vllm/model_executor/models/ltx2_t2a.h +++ b/include/vllm/model_executor/models/ltx2_t2a.h @@ -175,15 +175,17 @@ struct Ltx2T2aResult { double latent_absmax = 0.0; // EVERYTHING STEP 0 PRODUCED, in the order it produced it: the sampler's - // input, the conditional pass's RAW DiT output, the tensor that pass handed - // the guider, and the guider's result. They exist because #1039 is invisible - // in every other field here — combining the guidance passes in VELOCITY space - // and converting once afterwards produces a waveform of the right length, the - // right channel count, the right sample rate and a perfectly healthy forward - // count — and together they make the question decidable by arithmetic rather - // than by magnitude: + // input, EVERY GUIDANCE PASS as a (raw velocity, x0 prediction) pair, the + // guider's result, and the latent the Euler step wrote. They exist because + // #1039 is invisible in every other field here — combining the guidance passes + // in VELOCITY space and converting once afterwards produces a waveform of the + // right length, the right channel count, the right sample rate and a perfectly + // healthy forward count — and together they make the question decidable by + // arithmetic rather than by magnitude: // - // first_step_cond == first_step_latent - sigma * first_step_velocity + // first_step_cond == first_step_latent - sigma * first_step_velocity + // first_step_uncond == first_step_latent - sigma * first_step_uncond_velocity + // first_step_perturbed == first_step_latent - sigma * first_step_perturbed_velocity // // holds when the guider is handed X0 PREDICTIONS, as `X0Model.forward` does // (model.py:590-604, over the `X0Model(...)` that utils/blocks.py:480-482 @@ -191,13 +193,42 @@ struct Ltx2T2aResult { // also upstream's own `DenoisedLatentResult.cond` (utils/denoisers.py:206), // rather than a field invented for a test. // + // ONE PAIR PER ARM, AND NOT ONLY THE CONDITIONAL ONE. The default T2A arm runs + // THREE forwards per step (header item 2), and a build that converts the + // conditional pass and leaves either of the other two in velocity space + // renders a different waveform through a guider whose cond term is impeccable. + // A single recorded pair holds the claim "`to_denoised` on the way out of the + // forward" for one third of the passes it is made about; the review that found + // #1039 mutated exactly those other two arms and the gate stayed green. + // + // `first_step_next_latent` is what `Ltx2EulerStep` WROTE, and it is here so + // that what the sampler CONSUMED is checkable rather than assumed: + // + // first_step_next_latent + // == latent + (latent - first_step_denoised)/sigma * (sigma_next - sigma) + // + // A second `ToDenoised` applied to the guider's output on the way into the + // step — the residue a partial #1039 repair leaves behind — moves this and + // nothing else. + // // STEP 0 SPECIFICALLY, because it is the one step whose inputs do not depend // on any earlier step, so two renders that differ only in a guider parameter // share a bit-identical step-0 latent and bit-identical DiT passes. + // + // The uncond and perturbed pairs stay EMPTY when the guider does not ask for + // that arm (`cfg_scale == 1.0`, `stg_scale == 0.0`), because the forward did + // not run. An empty vector is the honest record of a pass that never happened; + // a zero-filled one of the right length would be indistinguishable from a + // forward that returned zeros. std::vector first_step_latent; std::vector first_step_velocity; std::vector first_step_cond; + std::vector first_step_uncond_velocity; + std::vector first_step_uncond; + std::vector first_step_perturbed_velocity; + std::vector first_step_perturbed; std::vector first_step_denoised; + std::vector first_step_next_latent; double first_step_sigma = 0.0; }; diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index 7c8d84b58..fe3007e6c 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -738,6 +738,24 @@ struct Ltx2ConditioningTrace { // // holds in x0 space (`X0Model.forward`, ltx-core model/transformer/ // model.py:590-604) and fails in velocity space. + // + // ONE (velocity, x0) PAIR PER ARM. The default T2A guider runs THREE forwards + // per step, and the equation above decides only the pass it names. A build + // that converts the conditional pass and leaves the UNCONDITIONAL or the + // PERTURBED one in velocity space renders a different waveform with a healthy + // forward count, a correct `t2a_first_cond`, and nothing else to see it by — + // which is #1039 again, one arm over. + // + // `t2a_first_next_latent` is `Ltx2EulerStep`'s output, and it makes what the + // sampler CONSUMED checkable: + // + // next == latent + (latent - denoised)/sigma * (sigma_next - sigma) + // + // A second `to_denoised` applied to the guider's result on the way into the + // step moves this field and no other. + // + // The uncond and perturbed vectors are EMPTY when the guider does not ask for + // that arm, because the forward did not run. bool t2a_rendered = false; bool t2a_video_stream_present = false; int64_t t2a_cond_forwards = 0; @@ -747,7 +765,12 @@ struct Ltx2ConditioningTrace { std::vector t2a_first_latent; std::vector t2a_first_velocity; std::vector t2a_first_cond; + std::vector t2a_first_uncond_velocity; + std::vector t2a_first_uncond; + std::vector t2a_first_perturbed_velocity; + std::vector t2a_first_perturbed; std::vector t2a_first_denoised; + std::vector t2a_first_next_latent; double t2a_first_sigma = 0.0; // True only once the `Generate` that produced this conditioning RETURNED. The diff --git a/src/vllm/model_executor/models/ltx2_t2a.cpp b/src/vllm/model_executor/models/ltx2_t2a.cpp index 44a85d4ce..aea6d9e62 100644 --- a/src/vllm/model_executor/models/ltx2_t2a.cpp +++ b/src/vllm/model_executor/models/ltx2_t2a.cpp @@ -345,7 +345,13 @@ Ltx2T2aResult Ltx2T2aGenerate(const Ltx2T2aRequest& req) { if (want_uncond) { Ltx2ModalityInput nin = ain; nin.context = req.negative_context; - uncond_text = x0_model(/*video=*/nullptr, &nin, /*p=*/nullptr); + // The velocity of THIS arm, recorded beside its x0 exactly as the + // conditional pass's is. Recorded per arm rather than once, because the + // conversion is per pass and a claim made about "every pass" from one + // recorded pass is a claim about a third of them (see the header). + uncond_text = x0_model(/*video=*/nullptr, &nin, /*p=*/nullptr, + step == 0 ? &result.first_step_uncond_velocity : nullptr); + if (step == 0) result.first_step_uncond = uncond_text; ++result.uncond_forwards; } if (want_perturbed) { @@ -353,7 +359,9 @@ Ltx2T2aResult Ltx2T2aGenerate(const Ltx2T2aRequest& req) { // perturbs the MODEL, never the conditioning (guiders.py:244-273 takes // `uncond_perturbed` from a forward whose `perturbations` differ and // whose context does not). - uncond_perturbed = x0_model(/*video=*/nullptr, &ain, &perturbation); + uncond_perturbed = x0_model(/*video=*/nullptr, &ain, &perturbation, + step == 0 ? &result.first_step_perturbed_velocity : nullptr); + if (step == 0) result.first_step_perturbed = uncond_perturbed; ++result.perturbed_forwards; } denoised = Ltx2MultiModalGuidance(g, cond.data(), @@ -379,6 +387,11 @@ Ltx2T2aResult Ltx2T2aGenerate(const Ltx2T2aRequest& req) { // nothing. latent = Ltx2EulerStep(latent.data(), last_denoised.data(), sigmas.data(), sigma_count, step, static_cast(latent.size())); + // What the sampler WROTE, recorded after the step rather than derived from + // what was recorded before it. It is the only observable that says which + // tensor `Ltx2EulerStep` was actually handed: a second `ToDenoised` applied + // to `denoised` on the way in leaves every other field here untouched. + if (step == 0) result.first_step_next_latent = latent; } result.latent_digest = DigestF32(latent); diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index 8651d3620..bf59f5f35 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -3665,7 +3665,12 @@ VideoResult Ltx2VideoEngine::GenerateAudioOnly(Impl& im, const VideoGenParams& g im.trace.t2a_first_latent = rendered.first_step_latent; im.trace.t2a_first_velocity = rendered.first_step_velocity; im.trace.t2a_first_cond = rendered.first_step_cond; + im.trace.t2a_first_uncond_velocity = rendered.first_step_uncond_velocity; + im.trace.t2a_first_uncond = rendered.first_step_uncond; + im.trace.t2a_first_perturbed_velocity = rendered.first_step_perturbed_velocity; + im.trace.t2a_first_perturbed = rendered.first_step_perturbed; im.trace.t2a_first_denoised = rendered.first_step_denoised; + im.trace.t2a_first_next_latent = rendered.first_step_next_latent; im.trace.t2a_first_sigma = rendered.first_step_sigma; im.trace.audio_tokens = rendered.audio_tokens; im.trace.audio_latent_digest = rendered.latent_digest; diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 9c9cb1657..9c11de380 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -5301,6 +5301,28 @@ TEST_CASE("ltx2 t2a: the guider is handed x0 predictions and not raw velocities" // whole sample. No fixture scale satisfies it by accident, a zeroed velocity // collapses it to `cond == latent` and is refused by the lower bound below, // and a zeroed `cond` fails it outright. + // + // ALL THREE ARMS, AND THE STEP THAT CONSUMES THEM. An earlier draft of this + // case asserted the equation for the CONDITIONAL pass alone. The default T2A + // arm runs three forwards per step, so that draft held the file's own + // "applied to EVERY PASS" claim for one third of the passes, and three + // mutations survived it at 10 cases / 526 assertions / exit 0: + // + // A1 the PERTURBED pass alone left in velocity space + // A2 the UNCONDITIONAL pass alone left in velocity space + // R1b `ToDenoised` applied a SECOND time to the guider's output, between the + // step-0 recording and the Euler step + // + // and a fourth found while closing them: + // + // R1c the same double application placed ABOVE the step-0 recording, so the + // recorded `t2a_first_denoised` is itself doubly converted + // + // Each renders a different waveform of exactly the right length, through a + // guider whose `cond` term is impeccable. So the equation is applied to every + // recorded arm; the guider's own output is reproduced from the three recorded + // arms through the shipped seam, which is what R1c moves; and the Euler step's + // input is recovered from the latent it wrote, which is what R1b moves. Workspace ws; const vllm::multimodal::VideoModelParams mp = T2aParams(ws.paths); @@ -5308,17 +5330,19 @@ TEST_CASE("ltx2 t2a: the guider is handed x0 predictions and not raw velocities" // off a render: `rescale_scale = 0.7` on the 2.3/2.4/2.5 lineage // (ltx-pipelines utils/constants.py:63, and the `--audio-rescale-scale` // default at utils/args.py:1101-1106). - { - const vllm::Ltx2PipelineRecipe t2a = vllm::ResolveLtx2PipelineRecipe("t2a_one_stage", "2.5"); - REQUIRE(t2a.phases.size() == 1); - CHECK(t2a.phases[0].audio_guidance.rescale_scale == 0.7); - } + const vllm::Ltx2PipelineRecipe t2a_recipe = + vllm::ResolveLtx2PipelineRecipe("t2a_one_stage", "2.5"); + REQUIRE(t2a_recipe.phases.size() == 1); + CHECK(t2a_recipe.phases[0].audio_guidance.rescale_scale == 0.7); // Through the production entry point — `LoadVideoEngine` then // `VideoEngine::Generate`, which is what `vllm_video_generate` calls. Nothing - // here constructs a guider, a DiT or a modality by hand. NO extra is touched - // either, so the render takes the recipe's own guider and the assertion sits - // on the shipped path rather than on one this test configured into existence. + // here constructs a guider, a DiT or a modality by hand. `rescale_scale` is + // the recipe's own 0.7, pinned just above and left untouched by `T2aGen`, + // which is the field this case turns on. (`T2aGen` does set + // `audio_stg_blocks`, and that IS a guider field — the two-block fixture + // cannot take the params table's `[28]` — but it selects WHICH block the + // perturbed forward skips, not how the arms are combined.) const std::unique_ptr engine = vllm::multimodal::LoadVideoEngine(mp); REQUIRE(engine != nullptr); @@ -5334,53 +5358,160 @@ TEST_CASE("ltx2 t2a: the guider is handed x0 predictions and not raw velocities" REQUIRE(t.t2a_first_velocity.size() == n); REQUIRE(t.t2a_first_cond.size() == n); REQUIRE(t.t2a_first_denoised.size() == n); + REQUIRE(t.t2a_first_next_latent.size() == n); const double sigma = t.t2a_first_sigma; REQUIRE(sigma > 0.0); double latent_span = 0.0; - double velocity_span = 0.0; - double err_x0 = 0.0; // |cond - (latent - sigma*velocity)| -> 0 in x0 space - double err_v = 0.0; // |cond - velocity| -> 0 in velocity space for (size_t i = 0; i < n; ++i) { - const double lat = static_cast(t.t2a_first_latent[i]); - const double vel = static_cast(t.t2a_first_velocity[i]); - const double cond = static_cast(t.t2a_first_cond[i]); - latent_span = std::max(latent_span, std::abs(lat)); - velocity_span = std::max(velocity_span, std::abs(vel)); - err_x0 = std::max(err_x0, std::abs(cond - (lat - sigma * vel))); - err_v = std::max(err_v, std::abs(cond - vel)); - } - - INFO("sigma = " << sigma << " max|latent| = " << latent_span - << " max|velocity| = " << velocity_span - << " |cond - (latent - sigma*velocity)| = " << err_x0 - << " |cond - velocity| = " << err_v << " elements = " << n); - - // 1. THE FIXTURE CAN DECIDE THIS. The two candidate tensors are - // `latent - sigma*velocity` and `velocity`; they coincide when the sample - // is zero and `to_denoised` is the identity when the velocity is. Both are - // REQUIREs, because the checks below mean nothing once either fails. + latent_span = std::max(latent_span, std::abs(static_cast(t.t2a_first_latent[i]))); + } + // THE FIXTURE CAN DECIDE THIS AT ALL. The two candidate tensors for every arm + // are `latent - sigma*velocity` and `velocity`, and they coincide when the + // sample is zero. A REQUIRE, because nothing below discriminates once it + // fails. (Its per-arm partner, "the DiT returned no velocity", is next to each + // arm's own check: a zero velocity makes `to_denoised` the identity for THAT + // arm alone.) REQUIRE_MESSAGE(latent_span > 1e-3, "the step-0 sample is zero, so the two candidate tensors coincide and nothing " "below discriminates"); - REQUIRE_MESSAGE(sigma * velocity_span > 1e-6, - "the DiT returned no velocity, so `to_denoised` is the identity here and the " - "two candidate tensors coincide"); - // 2. THE GUIDER WAS HANDED THE X0 PREDICTION, exactly — `to_denoised` on the - // way out of the forward, which is `X0Model.forward` (model.py:602-603). - CHECK_MESSAGE(err_x0 <= 1e-5 * latent_span, - "the tensor handed to `Ltx2MultiModalGuidance` is not `latent - sigma*velocity`, " - "which is what `X0Model.forward` returns (#1039): residual " - << err_x0 << " against a tolerance of " << (1e-5 * latent_span)); - // 3. AND IT WAS NOT THE RAW VELOCITY. Said separately from check 2, because a - // build handing the guider some THIRD tensor fails 2 and would pass a lone - // "not the velocity" check; the pair says which of the two happened. - CHECK_MESSAGE(err_v > 1e-2 * latent_span, - "the tensor handed to `Ltx2MultiModalGuidance` IS the raw DiT velocity, so the " - "guidance is combined in velocity space and converted once afterwards (#1039)"); - // 4. And the guider MOVED what it was handed, so the tensor checked above is - // a real input to the combination rather than a recorded value beside one. - CHECK(t.t2a_first_denoised != t.t2a_first_cond); + + // ── the equation, once per guidance pass ────────────────────────────────── + // + // EVERY ARM THE RENDER RAN, not only the conditional one. The default T2A + // guider runs three forwards per step (ltx2_t2a.h item 2), `x0_model` claims + // to convert EVERY PASS, and a conditional-only assertion holds that claim for + // one of the three. `t2a_first_uncond` / `t2a_first_perturbed` are empty when + // the guider did not ask for that arm; this render asks for both, which is + // asserted rather than assumed — an arm silently skipped would otherwise + // vacate its own check. + REQUIRE(t.t2a_uncond_forwards > 0); + REQUIRE(t.t2a_perturbed_forwards > 0); + struct Arm { + const char* name; + const std::vector& velocity; + const std::vector& x0; + }; + const Arm arms[] = { + {"cond", t.t2a_first_velocity, t.t2a_first_cond}, + {"uncond", t.t2a_first_uncond_velocity, t.t2a_first_uncond}, + {"perturbed", t.t2a_first_perturbed_velocity, t.t2a_first_perturbed}, + }; + for (const Arm& arm : arms) { + INFO("arm = " << arm.name); + REQUIRE(arm.velocity.size() == n); + REQUIRE(arm.x0.size() == n); + + double velocity_span = 0.0; + double err_x0 = 0.0; // |x0 - (latent - sigma*velocity)| -> 0 in x0 space + double err_v = 0.0; // |x0 - velocity| -> 0 in velocity space + for (size_t i = 0; i < n; ++i) { + const double lat = static_cast(t.t2a_first_latent[i]); + const double vel = static_cast(arm.velocity[i]); + const double x0 = static_cast(arm.x0[i]); + velocity_span = std::max(velocity_span, std::abs(vel)); + err_x0 = std::max(err_x0, std::abs(x0 - (lat - sigma * vel))); + err_v = std::max(err_v, std::abs(x0 - vel)); + } + INFO("sigma = " << sigma << " max|latent| = " << latent_span + << " max|velocity| = " << velocity_span + << " |x0 - (latent - sigma*velocity)| = " << err_x0 + << " |x0 - velocity| = " << err_v << " elements = " << n); + + // 1. `to_denoised` IS NOT THE IDENTITY ON THIS ARM. The second half of the + // non-vacuity guard, per arm: a zeroed velocity collapses the equation to + // `x0 == latent` and would let a stub satisfy it. + REQUIRE_MESSAGE(sigma * velocity_span > 1e-6, + "the DiT returned no velocity on this arm, so `to_denoised` is the identity " + "here and the two candidate tensors coincide"); + // 2. THE GUIDER WAS HANDED THE X0 PREDICTION, exactly — `to_denoised` on the + // way out of the forward, which is `X0Model.forward` (model.py:602-603). + CHECK_MESSAGE(err_x0 <= 1e-5 * latent_span, + "the tensor handed to `Ltx2MultiModalGuidance` on this arm is not " + "`latent - sigma*velocity`, which is what `X0Model.forward` returns (#1039): " + "residual " + << err_x0 << " against a tolerance of " << (1e-5 * latent_span)); + // 3. AND IT WAS NOT THE RAW VELOCITY. Said separately from check 2, because a + // build handing the guider some THIRD tensor fails 2 and would pass a lone + // "not the velocity" check; the pair says which of the two happened. + CHECK_MESSAGE(err_v > 1e-2 * latent_span, + "the tensor handed to `Ltx2MultiModalGuidance` on this arm IS the raw DiT " + "velocity, so the guidance is combined in velocity space and converted once " + "afterwards (#1039)"); + } + + // ── the guider's output is the guider's output ──────────────────────────── + // + // The three recorded arms, through the SHIPPED `Ltx2MultiModalGuidance` on the + // recipe's own params, must reproduce `t2a_first_denoised` bit for bit. This + // does not gate the guider's arithmetic — `Ltx2Rescale`'s own cases and the + // seam case below do that — it gates that the pipeline handed the guider these + // tensors and passed its result on UNTOUCHED. A second `to_denoised` applied + // to the combination is invisible in every per-arm check above, because it + // moves nothing the guider was handed. + // + // `stg_blocks` is the one guider field `T2aGen` overrides and the one + // `Ltx2MultiModalGuidance` does not read (it selects the perturbed forward's + // blocks, not the combination), so the recipe's params are the render's params + // for this call. + { + const std::vector replayed = vllm::Ltx2MultiModalGuidance( + t2a_recipe.phases[0].audio_guidance, t.t2a_first_cond.data(), t.t2a_first_uncond.data(), + t.t2a_first_perturbed.data(), /*uncond_modality=*/nullptr, static_cast(n)); + REQUIRE(replayed.size() == n); + double worst = 0.0; + for (size_t i = 0; i < n; ++i) { + worst = std::max(worst, std::abs(static_cast(replayed[i]) - + static_cast(t.t2a_first_denoised[i]))); + } + INFO("max|replayed guidance - t2a_first_denoised| = " << worst); + // EXACT, not a tolerance: it is the same function over the same f32 inputs, + // so any non-zero residual is another operation this pipeline applied. + CHECK_MESSAGE(worst == 0.0, + "`t2a_first_denoised` is not `Ltx2MultiModalGuidance` over the three recorded " + "arms, so something else was applied to the guider's result (#1039)"); + // And the combination MOVED what it was handed, so the arms checked above are + // real inputs to it rather than recorded values beside one. + CHECK(t.t2a_first_denoised != t.t2a_first_cond); + } + + // ── and the sampler consumed exactly that ───────────────────────────────── + // + // `Ltx2EulerStep` is `x + (x - denoised)/sigma * (sigma_next - sigma)` + // (ltx2_pipeline.cpp, `EulerDiffusionStep` at ltx-pipelines + // utils/blocks.py:524-527). Recovering `t2a_first_next_latent` from + // `t2a_first_denoised` pins WHICH tensor the step was handed. `ToDenoised` + // applied a second time between the recording and the step leaves every field + // above untouched and moves only this one. + // + // The schedule is re-derived from the shared seam rather than read off the + // render, and tied to it by the sigma the render recorded. + { + const std::vector sigmas = vllm::Ltx2SigmaSchedule(/*steps=*/2, /*tokens=*/0); + REQUIRE(sigmas.size() == 3); // `T2aGen` renders two steps + REQUIRE(static_cast(sigmas[0]) == sigma); + const double dt = static_cast(sigmas[1]) - static_cast(sigmas[0]); + REQUIRE_MESSAGE(std::abs(dt) > 1e-3, + "the first two sigmas coincide, so the Euler step is the identity and this " + "check cannot see what it consumed"); + double worst = 0.0; + double scale = 0.0; + for (size_t i = 0; i < n; ++i) { + const double lat = static_cast(t.t2a_first_latent[i]); + const double den = static_cast(t.t2a_first_denoised[i]); + const double expected = lat + (lat - den) / sigma * dt; + worst = std::max(worst, std::abs(static_cast(t.t2a_first_next_latent[i]) - expected)); + scale = std::max(scale, std::abs(expected)); + } + INFO("sigma = " << sigma << " -> " << sigmas[1] << " max|next - Euler(latent, denoised)| = " + << worst << " scale = " << scale); + REQUIRE_MESSAGE(scale > 1e-3, + "the recomputed Euler output is zero, so the residual below bounds nothing"); + CHECK_MESSAGE(worst <= 1e-5 * scale, + "the latent `Ltx2EulerStep` wrote is not the step over `t2a_first_denoised`, so " + "the sampler was handed some other tensor (#1039): residual " + << worst << " against a tolerance of " << (1e-5 * scale)); + } } TEST_CASE("ltx2 t2a: rescale_scale 0 is the control because both spaces agree there") { From b70f630a5fca41e74b04572c46e9cc7cda335a03 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 16 Aug 2026 19:21:10 +0000 Subject: [PATCH 9/9] record(#1052): test_engine_core_proc's shutdown case is load-dependent, and the issue it was blamed on is a different test The full gate on the merged tree failed `test_engine_core_proc` twice in three `ctest -j4` runs, and the third run failed `test_cpu_threadpool` instead. Both pass alone. Measured rather than dismissed, because "load-dependent" is what an inherited excuse looks like when nobody re-ran it. `tests/vllm/v1/test_engine_core_proc.cpp:481` searches for the abort frame over a FIXED budget of 1000 dequeues while a `max_tokens=100000` request keeps the busy loop emitting token deltas. Nothing bounds how many frames precede the abort, so the budget is a bet on scheduling. MEASURED at `37e680cab`, same binary throughout, CPU-only Release on 20 cores: | Regime | Runs | Failures | |---|---|---| | `ctest -j4`, full 492-test suite | 3 | 2, `CHECK( abort_seen ) is NOT correct!` | | the binary alone, idle box (load 3.34) | 25 | 0 | | the binary alone, against 20 spinning processes | 25 | 0 | | `ctest -R '^test_engine_core_proc$'` | 2 | 0, `Passed 0.03 sec` | CPU pressure alone does not reproduce it, so the regime is the `-j4` harness rather than load as such, and that is stated instead of the usual shorthand. The rotation between the two failing tests is the strongest single fact: the binary did not change between the runs. NO ISSUE NAMED THIS TEST. The earlier revision of PR #1032's body attributed the flake to #294, and #294 is a different defect in a different test: "test_async_llm: reusing an aborted request id races the core abort". A misattributed flake is worse than an untracked one, because the next reader checks the citation, finds an open issue about something else, and stops looking. The PR body is corrected in the same flow. Not fixed here. It is unrelated engine code, this branch touches no file under `tests/vllm/v1/` or `src/vllm/v1/`, and the assertion guards a real guarantee: an in-flight request must get a `kAbort` finish on immediate shutdown. The 1000-frame budget is the part that is a guess. Owned under `## Owed` in `.agents/specs/ltx25-t2a-one-stage.md`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 1 + .agents/specs/ltx25-t2a-one-stage.md | 12 ++++++++++++ 2 files changed, 13 insertions(+) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 3da8d9fdc..36a4a41b8 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -293,3 +293,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1048](https://github.com/mudler/vllm.cpp/issues/1048) | — | LTX-2.5 ships with NO checkpoint pin. `docs/USAGE.md` names six LTX-2.5 artifacts by bare file name and gives no HuggingFace repo, no revision and no sha256 for any of them: `:663-670` and `:2183-2188` on `origin/main` at `d1b0ea3a8`, plus the text-to-audio recipe [#1005](https://github.com/mudler/vllm.cpp/issues/1005) added at `:853-857`. AGENTS.md § *Say which weights, and from where* requires file name, size, exact repo AND revision, grouped by arm, with a sha256 for a quantized artifact. Campaign-wide and PRE-EXISTING rather than introduced by that row, measured rather than asserted: `grep -n sha256 docs/USAGE.md` returns two checkpoint hashes and BOTH belong to MiniMax-Music3 (`:3127`, `:3269`), while MiniMax-H3 (`:1950-1993`) and MiniMax-Music3 (`:3123-3149`) each carry a full repo + revision + sha256 table and LTX-2.5 carries none anywhere. RECORDED AND DELIBERATELY NOT FABRICATED: no LTX-2.5 row in the campaign claims a render on real weights — `dgx.casa` is down and every LTX-2.5 gate runs on a reduced fixture — so there is no checkpoint any of them was gated against to pin, and inventing a repo id would be worse than the gap. The real-checkpoint render owed by [#644](https://github.com/mudler/vllm.cpp/issues/644) is what closes it. Two smaller things fall out of the same gap and belong to the same change: the t2a recipe names `ltx-2.5-dit.safetensors`, a file name appearing nowhere else in the tree, where every other LTX-2.5 recipe names `ltx-2.5-22b-distilled-fp8.safetensors` plus the `--dit-config` its missing `__metadata__` requires; and whether that recipe runs at all without `--dit-config` is unverified. Found repairing the fresh review of [#1039](https://github.com/mudler/vllm.cpp/issues/1039) on PR [#1032](https://github.com/mudler/vllm.cpp/pull/1032). Listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | | [#1049](https://github.com/mudler/vllm.cpp/issues/1049) | — | `Ltx2Guidance` (`src/vllm/model_executor/models/ltx2_pipeline.cpp:526 @ c1fe35592`) is dead in production: `git grep -n 'Ltx2Guidance(' -- src include examples tests` returns the declaration (`ltx2_pipeline.h:330`), the definition, and ONE call, at `tests/vllm/models/test_ltx2_pipeline.cpp:710`. It is also the only path to two ported guiders — `Ltx2CfgDelta` (`ltx2_pipeline.cpp:532`, plus `test_ltx2_pipeline.cpp:615`) and `Ltx2StgDelta` (`:534`, plus `:630`) — so both are reachable from no product entry point either. `Ltx2BatchedPerturbationConfig` (`ltx2_pipeline.h:380`) is the same shape from a different direction: constructed only at `test_ltx2_pipeline.cpp:832-859`, while the LTX-2.5 text-to-audio path perturbs through `Ltx2DitPerturbation`, a different type, and no other path perturbs at all. This is the test-only-driver shape [`reachability.md`](reachability.md) names. All four landed with #641. `Ltx2MultiModalGuidance` was the fourth member of the set and is no longer one: [#1005](https://github.com/mudler/vllm.cpp/issues/1005) gave it a production call site in `ltx2_t2a.cpp`. Closing this means either routing a product path through `Ltx2Guidance` with a configured `Ltx2GuiderKind` — `Ltx2GuiderSigmaBin` and `Ltx2GuiderParamsForSigma` are already ported beside it — or retiring the unreached arms into `.agents/completed/` with their provenance. Found repairing the fresh review of [#1039](https://github.com/mudler/vllm.cpp/issues/1039) on PR [#1032](https://github.com/mudler/vllm.cpp/pull/1032), where an earlier draft of that row's spec §6b claimed the row ended all four test-only drivers; §6b now carries the measured table. Listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | | [#1050](https://github.com/mudler/vllm.cpp/issues/1050) | — | The guider rescale's `std` comment states a consequence that cannot exist. `src/vllm/model_executor/models/ltx2_pipeline.cpp:505-506 @ c1fe35592`, repeated at `include/vllm/model_executor/models/ltx2_pipeline.h:319-322`, says torch's `std` is the UNBIASED (N-1) estimator by default and "the biased one would be a small, everywhere, resolution-dependent gain error that no shape or finiteness check can see". `factor_raw` is `unbiased_std(cond) / unbiased_std(pred)`, two `std`s over the SAME `count`, so the divisor cancels exactly: `sqrt(ss_c/(n-1))/sqrt(ss_p/(n-1)) == sqrt(ss_c/ss_p) == sqrt(ss_c/n)/sqrt(ss_p/n)`. There is no gain error, small or otherwise, and nothing about it is resolution-dependent; the two forms differ only by f32 rounding in the divide. Worth a record rather than a silent correction because the comment tells the next reader a gate is needed there and it is not: the fresh review of [#1039](https://github.com/mudler/vllm.cpp/issues/1039) mutated the estimator to the biased form and it survived — correctly, because it is an IDENTITY — and a survivor at that site otherwise reads as a blind instrument and costs another investigation. The CODE is right as written and should stay `unbiased_std`, because the name is what mirrors torch even where the ratio does not care; the COMMENT is the defect. Pre-existing from `cefacd2d0` (#641). Found repairing that review on PR [#1032](https://github.com/mudler/vllm.cpp/pull/1032) and out of scope there under its explicit exclusions. Listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | +| [#1052](https://github.com/mudler/vllm.cpp/issues/1052) | — | `tests/vllm/v1/test_engine_core_proc.cpp:481` ("EngineCoreProc: immediate shutdown aborts in-flight requests") searches for the abort frame over a FIXED budget of 1000 dequeues while a `max_tokens=100000` request keeps the busy loop producing token deltas, so nothing bounds how many frames precede the abort and the budget is a bet on scheduling. MEASURED at `37e680cab`, same binary throughout, CPU-only Release on 20 cores: **2 failures in 3 `ctest -j4` runs** of the full 492-test suite (`CHECK( abort_seen ) is NOT correct!`), **0 in 25 solo runs** on an idle box at load 3.34, **0 in 25 solo runs against 20 spinning processes**, and 0 in two `ctest -R '^test_engine_core_proc$'` runs (`Passed 0.03 sec`). So CPU pressure alone does not reproduce it; it needs the `-j4` harness. The third `-j4` run failed `test_cpu_threadpool` INSTEAD, which is on the same load-dependent list, so the IDENTITY of the failing test rotates between runs of an unchanged binary and both pass alone with exit 0. NO ISSUE NAMED THIS TEST: PR [#1032](https://github.com/mudler/vllm.cpp/pull/1032)'s body attributed its flake to [#294](https://github.com/mudler/vllm.cpp/issues/294), which is a different defect in a different test (`test_async_llm` reusing an aborted request id), and a misattributed flake is worse than an untracked one because the next reader checks the citation, finds an open issue about something else, and stops looking. The assertion guards a real guarantee (an in-flight request gets a `kAbort` finish on immediate shutdown); the 1000-frame budget is the part that is a guess. Found repairing the fresh review of [#1039](https://github.com/mudler/vllm.cpp/issues/1039), on a branch that touches no file under `tests/vllm/v1/` or `src/vllm/v1/`. Listed under `## Owed` in [`ltx25-t2a-one-stage.md`](specs/ltx25-t2a-one-stage.md) | bug | diff --git a/.agents/specs/ltx25-t2a-one-stage.md b/.agents/specs/ltx25-t2a-one-stage.md index 7749461a3..d3d4a5628 100644 --- a/.agents/specs/ltx25-t2a-one-stage.md +++ b/.agents/specs/ltx25-t2a-one-stage.md @@ -841,6 +841,18 @@ Green after: `--test-case=ltx2 t2a*` at 10 cases / **548** assertions / 0 failed landed with #641, none is on the T2A path, and this row ends only `Ltx2MultiModalGuidance`'s test-only-driver state. Tracked by [#1049](https://github.com/mudler/vllm.cpp/issues/1049). +- **`test_engine_core_proc`'s immediate-shutdown case is load-dependent**, and + no issue named it until now. Its abort frame is searched for over a FIXED 1000 + dequeues while a `max_tokens=100000` request keeps producing token deltas, so + the budget is a bet on scheduling. MEASURED at `37e680cab`, same binary + throughout: 2 failures in 3 `ctest -j4` runs of the full suite, 0 in 25 solo + runs on an idle box, 0 in 25 solo runs against 20 spinning processes, and 0 in + two `ctest -R` runs. The third `-j4` run failed `test_cpu_threadpool` INSTEAD, + so the identity of the failing test rotates between runs of an unchanged + binary. This branch touches no file under `tests/vllm/v1/` or `src/vllm/v1/`. + The earlier revision of the PR body attributed it to #294, which is a + DIFFERENT defect in `test_async_llm`. Tracked by + [#1052](https://github.com/mudler/vllm.cpp/issues/1052). - **The guider rescale's `std` comment states an impossible consequence.** `ltx2_pipeline.cpp:505-506` and `ltx2_pipeline.h:319-322` say torch's unbiased (N-1) `std` matters and the biased one "would be a small, everywhere,