From baa92ccf7be05012a307489a42895c0b4400ac6e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 17:05:09 +0000 Subject: [PATCH 01/10] spec(ltx-2.5): the prompt-side AdaLN path is DROPPED on every render MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `use_prompt_adaln_single` defaults TRUE in both references — diffusers `transformer_ltx2.py:1185` and LTX-2 `model.py:77` / `model_configurator.py:76,138` — and the shipped FP8 DiT carries the 18 tensors the flag builds. `ltx2_loader.cpp:988` (and `:573`, `:626`) cleared it unconditionally, so every render kept only the static `prompt_scale_shift_table` and dropped the timestep-conditioned half of the prompt K/V modulation. Shapes, finiteness and every committed golden agree with the defect, because the goldens were generated with the flag OFF. Spec only; no code. Records the upstream anchors for the module (`model.py:222-227`, `:252-257`), the producer (`transformer_args.py:274-277`, driven by `modality.sigma` and NOT `timesteps`), the consumer (`transformer.py:441-446`), and what replaces the cleared flag so `allow_unported_modules` can no longer silently disable a ported feature. Issue #644, row 0. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/roadmap_v1.md | 1 + .agents/specs/ltx25-prompt-adaln.md | 259 ++++++++++++++++++++++++++++ 2 files changed, 260 insertions(+) create mode 100644 .agents/specs/ltx25-prompt-adaln.md diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 3cc1d0234..df766a903 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -147,6 +147,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#435](https://github.com/mudler/vllm.cpp/issues/435) | `ROAD-V1-LTX25` | LTX-2.5: port the 21B joint video+audio DiT and generalize the video seam | feature | | [#560](https://github.com/mudler/vllm.cpp/issues/560) | `ROAD-V1-LTX25` | Stabilizing constants unreachable by any gate: FIVE instances found by sweep, incl. the DiT's own `norm_eps` field default | bug | | [#567](https://github.com/mudler/vllm.cpp/issues/567) | `ROAD-V1-LTX25` | `OpNameImpl` makes a SECOND file every new `OpId` must edit; collapse the enum and its names into one derived list | bug | +| [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)): `use_prompt_adaln_single` defaults TRUE in BOTH references and the shipped DiT carries its 18 tensors, but `ltx2_loader.cpp:988` cleared the flag unconditionally, so every render dropped the timestep-conditioned half of the prompt K/V modulation — invisible to every shape and finiteness check | feature | | [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug | | [#264](https://github.com/mudler/vllm.cpp/issues/264) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprob_token_ids`: generative scoring over an explicit token set is unported | feature | | [#365](https://github.com/mudler/vllm.cpp/issues/365) | `PERF-27B-DENSE-MARLIN-GATEUP` | 27B gap decomposed vs vLLM's own decode profile; dense W4A16 MLP bypassed the fused gate_up seam (spec `specs/perf-27b-dense-marlin-gateup.md`) | bug | diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md new file mode 100644 index 000000000..70cb486b5 --- /dev/null +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -0,0 +1,259 @@ +# LTX-2.5 — the prompt-side AdaLN path (`use_prompt_adaln_single`) + +Row: `LTX25-PROMPT-ADALN`. Campaign: [`ltx-2-5.md`](ltx-2-5.md) (operator-owned; not +edited by this row). Issue: +[#644](https://github.com/mudler/vllm.cpp/issues/644), row 0. + +Upstream pins: + +| Reference | Revision | +|---|---| +| Lightricks/LTX-2 (`packages/ltx-core`) | `fd4ded7f` | +| huggingface/diffusers | `3a2f35d4` | + +Both are read from local checkouts at those revisions, and the golden generator +IMPORTS and EXECUTES the LTX-2 modules rather than restating them. + +## 0. What is wrong today + +`src/vllm/model_executor/models/ltx2_loader.cpp:988` sets, unconditionally: + +```cpp +declared.use_prompt_adaln_single = false; +``` + +and `:573` / `:626` do the same on the two manifest paths. The flag defaults +**TRUE** in both references: + +- diffusers `src/diffusers/models/transformers/transformer_ltx2.py:1185` — + `use_prompt_adaln_single: bool = True` +- LTX-2 `packages/ltx-core/src/ltx_core/model/transformer/model.py:77` — same + default, and `model_configurator.py:76` / `:138` read it as + `config.get("use_prompt_adaln_single", True)` + +The shipped FP8 DiT carries the 18 tensors the flag builds (12 +`prompt_adaln_single.*`, 6 `audio_prompt_adaln_single.*`; see +`tests/vllm/models/ltx2_fp8_dit_manifest.inc:232-240,286-294`), so the flag is +TRUE for the checkpoint this campaign renders. `ltx2.cpp:274-276` refuses those +tensors by name, so a real render needs `allow_unported_modules=1` +(`src/vllm/multimodal/ltx2_video.cpp:570`) — which reaches the loader lines above +and **silently clears the flag**. + +Net effect: every render drops the timestep-conditioned half of the prompt K/V +modulation, keeping only the static `prompt_scale_shift_table`. Nothing observes +it: shapes are unchanged, values stay finite, and the goldens were generated with +`use_prompt_adaln_single=False` (`scripts/gen-ltx2-goldens.py:149`), so the gate +agrees with the defect. + +Campaign history this row does **not** re-derive: `ltx-2-5.md` §1.2 already +RETRACTED the "prompt K/V carry no timestep term" claim and recorded that the +shipped checkpoint carries a `[4096, 256]` prompt timestep embedder. What was +never closed is *using* the tensors. + +## 1. What upstream does, with anchors + +### 1.1 The module + +`model.py:222-227` (video) and `:252-257` (audio): + +```python +self.prompt_adaln_single = ( + AdaLayerNormSingle(self.inner_dim, embedding_coefficient=2) + if self.cross_attention_adaln and self.use_prompt_adaln_single + else None +) +``` + +`AdaLayerNormSingle` (`adaln.py:19-45`) is the same brick the port already has +(`Ltx2AdaLayerNormSingle`): `emb.timestep_embedder.linear_1 [dim, 256]`, +`linear_2 [dim, dim]`, `linear [coefficient * dim, dim]`. Coefficient **2** here, +not `adaln_embedding_coefficient()` — shift and scale for the K/V only. + +diffusers twin: `transformer_ltx2.py:1255-1259`, `num_mod_params=2`. + +Registration order inside `_init_video` puts it between `adaln_single` and +`proj_out`, which is where `EnumerateLtx2DitTensors` already reserves its slot +(the `VT_CHECK` at `ltx2.cpp:274-276`). + +### 1.2 The producer + +`transformer_args.py:274-277`, inside `TransformerArgsPreprocessor.prepare`: + +```python +prompt_timestep = None +if self.prompt_adaln is not None: + prompt_timestep, _ = self._prepare_timestep( + modality.sigma, self.prompt_adaln, batch_size, modality.latent.dtype + ) +``` + +Three things this fixes in one line, each of which a shape check cannot see: + +1. The input is **`modality.sigma`**, `(B,)` (`modality.py:54`) — the per-sample + scalar noise level — **not** `modality.timesteps`, which is per-token `(B, T)`. +2. `_prepare_timestep` (`transformer_args.py:173-186`) multiplies by + `timestep_scale_multiplier` before the embedder, exactly as the port's + `PrepareTimestep` already does for the main AdaLN. +3. The result is viewed to `(B, -1, 2 * dim)`, i.e. `(B, 1, 2 * dim)` — one row + broadcast over the prompt tokens. + +Wired into both preprocessor kinds at `model.py:313`, `:333` (multimodal) and +`:348`, `:364` (single-modality). + +diffusers twin: `transformer_ltx2.py:1536-1547`. diffusers passes `sigma` already +scaled from the pipeline (`pipeline_ltx2_image2video.py:1481` — `sigma=timestep`, +and `timestep` is the scheduler's 0..1000 value), so the two references agree on +the value reaching the embedder; only the place the x1000 happens differs. This +port mirrors LTX-2, so the multiply happens here. + +### 1.3 The consumer + +`transformer.py:427-447` (`apply_cross_attention_adaln`): + +```python +kv_modulation = prompt_scale_shift_table[None, None].to(...) # :441 +if prompt_timestep is not None: # :442 + kv_modulation = kv_modulation + prompt_timestep.reshape( + batch_size, prompt_timestep.shape[1], 2, -1) # :443 +shift_kv, scale_kv = kv_modulation.unbind(dim=2) # :444 +... +encoder_hidden_states = context * (1 + scale_kv) + shift_kv # :446 +``` + +Reached from `_apply_text_cross_attention` (`:223-251`), which is called for the +video stream at `:288-296` and the audio stream at `:317-325`, passing +`video.prompt_timestep` / `audio.prompt_timestep`. Every block, both streams. + +diffusers twin: `transformer_ltx2.py:677-693` (`get_mod_params` over +`prompt_scale_shift_table`), threaded at `:1648-1649`. + +Layout consequence: the flat `[B, 1, 2 * dim]` row is read as `[2, dim]` with +**shift first, scale second** — the same order the static table already uses in +`ModulateContext` (`ltx2_dit.cpp:118-129`). + +## 2. Scope + +**In.** + +1. `Ltx2DitParams::use_prompt_adaln_single` is honoured end to end: contract, + binding, host forward, device forward. +2. The 18 tensors enter `EnumerateLtx2DitTensors` / `BindLtx2DitWeights` when + `cross_attention_adaln && use_prompt_adaln_single`. +3. `temb_prompt` / `temb_prompt_audio` computed from each stream's own `sigma` + and threaded into every block's text cross-attention on both the host + (`ltx2_dit.cpp`) and device (`ltx2_device.cpp`) paths. +4. The three loader `= false` assignments are deleted, and replaced by a guard + (§3.2) that makes a future silent clear impossible. +5. The `ltx2.cpp:274-276` refusal is deleted for these two families. +6. Goldens executed from upstream at reduced dims, with a mutation proving the + new term is load-bearing, and a measured magnitude. + +**Out.** Anything the campaign already records as owed: keyframe absolute +position embedding (still genuinely unported, still what +`allow_unported_modules` is for), the caption projections, guidance +perturbations, and the bf16/FP8/NVFP4 stream dtypes on the host forward. + +## 3. Design + +### 3.1 The seam + +`Ltx2DitWeights` gains two `Ltx2AdaLayerNormSingleWeights` members; +`Ltx2BlockArgs` / `BlockArgsDev` gain a `[batch, 1, 2 * width]` prompt-modulation +pointer per stream, `nullptr` when the flag is off. `ModulateContext` and its +device twin take that pointer and add `prompt_mod[b, {0,1} * width + c]` to the +table row before applying `context * (1 + scale) + shift`. `nullptr` gives the +existing static-only behaviour byte-for-byte, which is what keeps every current +golden valid. + +The **order of the two additions** is upstream's: the table and the timestep row +are summed FIRST (`:443`), and only then does `(1 + scale)` apply. Folding it the +other way would round differently — the same trap `ProcessOutput` +(`ltx2_dit.cpp:530-540`) already documents. + +### 3.2 The prompt-K/V cache, and what replaces the cleared flag + +`Ltx2DitForward` already refuses a cache when the flag is on +(`ltx2_dit.cpp:672-676`); with the flag no longer cleared, that refusal becomes +*reachable* rather than dead, and it is correct: the K/V now carry a timestep +term. Nothing in the shipped pipeline passes a cache (`grep` over +`ltx2_pipeline.cpp` finds none), so no caller regresses. + +**What replaced the `= false`.** The loader clearing existed so +`EnumerateLtx2DitTensors` would not throw. With the tensors ported the contract +simply includes them, so the assignment has no job left. In its place the loader +asserts the invariant the clearing used to violate: + +> the resolved `use_prompt_adaln_single` must equal whether the FILE carries +> `prompt_adaln_single.linear.weight` + +A future edit that re-clears the flag then hits a named refusal instead of +quietly dropping 18 tensors. This is deliberately an *equality*, not a one-sided +check: clearing the flag with the tensors present is the defect this row fixes, +and setting it with the tensors absent would bind missing weights. + +### 3.3 `allow_unported_modules` + +After this row it is scoped to genuinely-unported modules only: the sole flag it +still clears in a config copy is `use_keyframes_abs_pos_embedding` +(`ltx2_loader.cpp:979-984`), whose module really is unported. The guard in §3.2 +is what makes that scoping structural rather than a comment — the extra `= false` +cannot come back without going red. + +`Ltx2AdoptDeclaredDitParams`'s contract-equality check becomes load-bearing in a +second way: a checkpoint whose config declares `use_prompt_adaln_single=false` +while its shapes carry the tensors now produces two DIFFERENT contracts and is +refused, instead of both sides being forced to the same cleared value. + +## 4. Memory format + +Mirrors the existing L2 parity forward exactly: f32 host, f32/bf16 device stream +with the `prompt_scale_shift_table` read at F32 (`ltx2_device.cpp:508-511`). The +prompt modulation is one `[batch, 1, 2 * width]` buffer per stream per forward — +`2 * 4096 * batch` floats for the video stream at full size, computed once +outside the block loop, not per block. No new per-token buffer, so no per-token +byte cost. + +## 5. Tests + +1. `EnumerateLtx2DitTensors` with the flag ON reproduces upstream + `named_parameters()` verbatim — names, order, ranks, dims — for a model built + with `use_prompt_adaln_single=True`. This is the 18-tensor contract. +2. Full dual-stream DiT forward, flag ON, against upstream's executed output. +3. **The mutation**: the same forward with the prompt-AdaLN contribution zeroed + must go RED against that golden. A term that is present but inert is not a + port. +4. The flag-OFF goldens stay byte-identical, proving the new path is off when + upstream's is off. +5. The prompt-K/V cache stays refused with the flag on (existing case). + +## 6. Measured magnitude + +Recorded in §Outcome: the relative change in the modulated prompt context and in +the DiT's outputs, flag ON vs OFF, at the reduced dimensions. This is the answer +to "does this matter"; a number below round-off would mean the term is inert and +the whole row is decoration. + +## 7. Risks + +- **The goldens agree with the defect.** Every existing LTX golden was generated + with the flag off, so no existing case can fail whatever this row does. The new + flag-ON case is the only instrument, which is why §5.3 mutates it rather than + asserting it. +- **`sigma` vs `timesteps`.** Using the per-token `timesteps` instead of the + per-sample `sigma` produces a same-shaped, finite, wrong result at batch 1 with + uniform timesteps. The golden runs a batch of 2 with per-token timesteps that + differ from sigma, so the two are distinguishable. +- **Order within the `[2, dim]` row.** Swapping shift and scale is finite and + same-shaped. The golden's random weights make it observable. + +## 8. Stop conditions + +- The flag-ON forward cannot be made to match upstream to the existing + `kRoundOff` bound: stop and report rather than widening the bound. +- The mutation in §5.3 stays green: the path is not reached, and the row is not + done. Escalate the mutation's magnitude before concluding anything about + reachability (issue #604). + +## Now + +`ACTIVE` — implementation on `row/LTX25-PROMPT-ADALN`. From 02038167677dcd8d0c0bece21d11f976130193df Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 17:54:16 +0000 Subject: [PATCH 02/10] feat(ltx-2.5): port the prompt-side AdaLN, and stop the opt-in from clearing it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `use_prompt_adaln_single` defaults TRUE in both references — diffusers `transformer_ltx2.py:1185`, LTX-2 `model.py:77` and `model_configurator.py:76`/`:138` — and the shipped DiT carries the module's tensors. `ltx2_loader.cpp` cleared it in three places, so every render kept only the static `prompt_scale_shift_table` and dropped the timestep-conditioned half of the prompt K/V modulation. Nothing saw it: shapes unchanged, values finite, and every committed golden was generated with the flag OFF. MEASURED, flag ON vs OFF over the same shared weights: the timestep term is 51.7% the magnitude of the static table it is added to, moves the block-0 modulated prompt K/V by 5.82%, and moves the DiT output by 1.46e-4 — 73x the gate's f32 floor. Ported, citing what each piece came from: the module <- model.py:222-227 (video), :252-257 (audio); AdaLayerNormSingle(dim, embedding_coefficient=2), adaln.py:19-45 the producer <- transformer_args.py:274-277 driven by `modality.sigma` [batch] and NOT the per-token `timesteps`, scaled by timestep_scale_multiplier at :173-186 the consumer <- transformer.py:441-446, summed into the table BEFORE `(1 + scale)` applies, threaded per stream at :288-296/:317-325 diffusers twin <- transformer_ltx2.py:1536-1547, :1648-1649, :677-693 Both arms: `Ltx2DitForward` and `Ltx2DitForwardDevice` (f32 and the bf16 production stream). The three `= false` assignments are replaced by an invariant: the resolved flag must equal whether the FILE carries `prompt_adaln_single`. And `Ltx2AdoptDeclaredDitParams` now clears exactly ONE flag in its config copy — `use_keyframes_abs_pos_embedding`, whose module really is unported — so `allow_unported_modules=1`, which a real render still needs, can no longer switch a ported feature off. A config that disagrees with the shapes about the flag is refused rather than forced into agreement. Both shipped DiTs were re-read off the NAS to check this refuses neither: the FP8 file carries no `__metadata__` at all and the NVFP4 file's config OMITS the key, so upstream's TRUE default decides on both, matching their tensors. Evidence. Goldens executed from upstream LTX-2 @fd4ded7f at reduced dims (`gen-ltx2-goldens.py` section 6); every pre-existing golden VALUE is byte-identical. Five mutations, each RED and each restored byte-for-byte: dropping the term host-side (3/35 cases), dropping it device-side (1/15), re-adding the cleared flag (the guard throws; assertion count drops 4826->4815), driving the MLP from `timesteps` instead of `sigma` (2/35), and swapping the shift and scale rows (2/35). A permanent in-tree case asserts the flag-ON forward MISSES the flag-OFF golden, so a bound-but-unused module cannot pass. Suites vs `cefacd2d0`: test_ltx2 30/1627 -> 35/2435, test_ltx2_loader 24/4817 -> 26/4826, test_ltx2_device 13/498 -> 15/523, test_ltx2_video 30/502 -> 30/502 (its fixture now carries the module, which is the shipped shape). Full `ctest` 423 registered, 422 pass under -j8 and `test_serve_low_tools` passes on its serial re-run. Spec .agents/specs/ltx25-prompt-adaln.md @baa92ccf7. Issue #644, row 0. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-prompt-adaln.md | 104 ++++- docs/FEATURES.md | 2 +- docs/USAGE.md | 11 +- include/vllm/model_executor/models/ltx2.h | 51 ++- .../vllm/model_executor/models/ltx2_loader.h | 45 +- include/vllm/multimodal/ltx2_video.h | 17 +- scripts/gen-ltx2-goldens.py | 151 +++++- src/vllm/model_executor/models/ltx2.cpp | 16 +- .../model_executor/models/ltx2_device.cpp | 79 +++- src/vllm/model_executor/models/ltx2_dit.cpp | 90 +++- .../model_executor/models/ltx2_loader.cpp | 67 ++- tests/vllm/models/ltx2_goldens.inc | 432 +++++++++++++++++- tests/vllm/models/test_ltx2.cpp | 141 ++++++ tests/vllm/models/test_ltx2_device.cpp | 88 +++- tests/vllm/models/test_ltx2_loader.cpp | 184 ++++++-- tests/vllm/multimodal/ltx2_video_fixture.h | 9 +- tests/vllm/multimodal/test_ltx2_video.cpp | 18 +- 17 files changed, 1381 insertions(+), 124 deletions(-) diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md index 70cb486b5..a24f127ce 100644 --- a/.agents/specs/ltx25-prompt-adaln.md +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -254,6 +254,108 @@ the whole row is decoration. done. Escalate the mutation's magnitude before concluding anything about reachability (issue #604). +## Outcome + +### What was measured + +The generator emits these into `tests/vllm/models/ltx2_goldens.inc` and prints +them on stderr, from the SAME shared weight stream on both arms (keyed by +parameter name, so every common weight is bit-identical and the difference is the +term and nothing else): + +| Quantity | Flag ON vs OFF | +|---|---| +| **timestep term vs the static table it is added to** | `max\|term\|` 0.0252 vs `max\|table\|` 0.0487 — **51.7%** | +| **block-0 modulated prompt K/V** | `max\|on-off\|` 0.0310 — **5.82%** of `max\|off\|` | +| DiT video output (2 blocks) | 1.4567e-4 — 0.04% of `max\|off\|`, **73x** the gate's 2e-6 floor | +| DiT audio output (2 blocks) | 7.367e-5 — 0.03%, **37x** the floor | + +**The answer to "does this matter" is the first two rows.** Roughly half the +magnitude of the prompt K/V modulation is the timestep-conditioned term, and +including it moves the modulated prompt context by ~6%. Every render before this +row discarded that. + +The two output rows are the GATE's floor, not a claim about the trained +checkpoint: they are bounded by the generator's synthetic weight scale (0.05) and +by a 2-block stack rather than 48. They are reported because a mutation must be +shown to move something, and 73x/37x above round-off is what makes the mutation +below meaningful. + +### The mutations + +All five run on the committed head, restored byte-for-byte afterwards (source +md5s re-checked). Exit status is the authority; assertion COUNTS are recorded +because doctest's summary and the exit code disagree in both directions. + +| # | Mutation | Result | +|---|---|---| +| M1 | host `ModulateContext` ignores `prompt_mod` | RED — 3/35 cases, 6/2435 assertions, exit 1 | +| M2 | device `TextCrossAttentionDev` takes the static-only branch always | RED — 1/15 cases, 6/523 assertions, exit 1 | +| M3 | re-add `use_prompt_adaln_single = false` before the loader guard | RED — the guard throws by name; assertion count DROPS 4826 → 4815, exit 1 | +| M4 | prompt AdaLN driven by `m.timesteps` instead of `m.sigma` | RED — 2/35 cases, 4/2435 assertions, exit 1 | +| M5 | shift and scale rows swapped within the `[2, width]` row | RED — 2/35 cases, 4/2435 assertions, exit 1 | + +M3 is the one that cannot be reached by any INPUT, and that is stated rather than +papered over: `ParseLtx2DitParamsFromManifest` derives the flag from the same +manifest the guard reads, so they agree by construction unless an assignment +intervenes — which is exactly the edit the guard exists to catch. The +input-driven half of the same rule lives in `Ltx2AdoptDeclaredDitParams`, where a +config that disagrees with the shapes now produces two different contracts and is +refused; that one is gated by a test with real inputs in both directions. + +### The gate + +`BUILD_EXIT=0` on every build; build logs grepped for `No space left|BFD assertion` +(0 hits) and `df -h /` logged (88% used, 52G free at the end). Case AND assertion +counts against the `cefacd2d0` baseline, measured by reverting the working tree to +HEAD, rebuilding the four targets and running them, then re-applying the diff and +re-checking its md5 (`03324d42…`, identical before and after): + +| Suite | HEAD `cefacd2d0` | this row | delta | +|---|---|---|---| +| `test_ltx2` | 30 cases / 1627 assertions | 35 / 2435 | +5 cases, +808 assertions | +| `test_ltx2_loader` | 24 / 4817 | 26 / 4826 | +2 cases, +9 (new cases minus the assertions the retired unported-family claims took with them) | +| `test_ltx2_device` | 13 / 498 | 15 / 523 | +2 cases, +25 assertions | +| `test_ltx2_video` | 30 / 502 | 30 / 502 | unchanged — the fixture now carries the module, and no assertion counted it | + +The `test_ltx2_video` fixture had to move: it declared a config that omits +`use_prompt_adaln_single` (mirroring the shipped NVFP4 DiT) while its SHAPES said +false, so the config/shape equality check refused it — correctly. It now carries +the module, which is the shipped shape and puts the whole video engine on the new +path. + +### What was rejected + +- **Widening `modulate`'s kernel contract** with a `rows_per_src_row` divisor, to + express "one row per batch element broadcast over that element's tokens" in one + launch. Rejected: it changes a kernel's semantics for a dimension that is 1 or 2 + in every shipped call, and would owe its own red-before evidence. The device + path loops over batch and offsets the pointers instead. +- **Narrowing the static prompt table to the stream dtype** on the flag-ON path, + which is literally what upstream's `.to(dtype=x_normed.dtype)` does. Rejected as + out of scope: the existing static-only path deliberately keeps the table at F32 + (`ltx2_device.cpp`, "a narrowed table would be the dtype rule applied + backwards"), and changing that polarity is a separate decision. The flag-ON path + routes the sum through `ada_value`, which stores at the stream dtype — the same + rounding every other table+modulation sum in that file already has. + +### Why the defaults are what they are + +`Ltx2DitParams::use_prompt_adaln_single` keeps its `true` default, which is now +honoured rather than overwritten. It matches `model.py:77`, +`model_configurator.py:76`/`:138` and diffusers `transformer_ltx2.py:1185`, and +it matches both shipped DiTs: the FP8 file carries no config at all (so the +default decides), and the NVFP4 file's config OMITS the key — verified by reading +both headers off the NAS. So neither shipped checkpoint is refused by the new +config/shape equality check. + +`allow_unported_modules` keeps existing, because +`keyframes_abs_pos_embedding` is genuinely unported and a real render still needs +the opt-in for it. What changed is that it can no longer switch a ported feature +off: the loader asserts the flag against the file instead of clearing it, and +`Ltx2AdoptDeclaredDitParams` clears exactly one flag, for a module nothing +applies. + ## Now -`ACTIVE` — implementation on `row/LTX25-PROMPT-ADALN`. +`DONE` — landed on `row/LTX25-PROMPT-ADALN`. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 7586e32ac..3c7ba3e68 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -161,7 +161,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder FORWARD 15.90x of vLLM's whole TTFT (pin 46.02 ms), or 2.89x with opt-in `VT_WHISPER_ENC_FA2=1` (costs 3 near-tie divergences vs 0). Not a TTFT ratio. Pending | | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending | | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams | -| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. The 320x192/25f scene was register-conditioned; a prompted render is OWED | Family `ltx-2.5` via `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8, torchao and first-party NVFP4 all load. DiffVAE, LoRA, image conditioning refused AT THE ENGINE. Speed PENDING | +| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt-side AdaLN ported, host+device. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. A prompted render is OWED | Family `ltx-2.5`, `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8/torchao/1st-party NVFP4 load; `keyframes_abs_pos_embedding` alone needs `allow_unported`. DiffVAE, LoRA, image cond refused. Speed PENDING | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending | diff --git a/docs/USAGE.md b/docs/USAGE.md index 142cd6a17..4f80a43ef 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2073,11 +2073,14 @@ other marker-less NVFP4 checkpoint as unsupported until it is. See `.agents/specs/nvfp4-nibble-order.md`. Two behaviours a caller has to know. `Ltx2LoadDitFromSafetensors` REFUSES the -shipped DiT by default, because that file carries **three** module families phase -L2 does not port (`prompt_adaln_single`, `audio_prompt_adaln_single` and -`keyframes_abs_pos_embedding`); pass `Ltx2DitLoadOptions::allow_unported_modules` +shipped DiT by default, because that file carries **one** module family this port +does not carry (`keyframes_abs_pos_embedding`); pass +`Ltx2DitLoadOptions::allow_unported_modules` to load the ported subset, which still reports every one of them in -`Ltx2DitCheckpoint::unported`. The two `*_embeddings_connector` towers are +`Ltx2DitCheckpoint::unported`. `prompt_adaln_single` and +`audio_prompt_adaln_single` were on that list until 2026-08-13 and are now +PORTED, so a checkpoint carrying them needs no opt-in on their account, and the +opt-in no longer disables them. The two `*_embeddings_connector` towers are **not** among them and never will be: `UnportedFamilies` filters them out at `ltx2_loader.cpp:439` (`LoadedElsewhere`), `RefuseUnported`'s own message says so in capitals at `ltx2_loader.cpp:461-464`, diff --git a/include/vllm/model_executor/models/ltx2.h b/include/vllm/model_executor/models/ltx2.h index f71a1a404..99898406b 100644 --- a/include/vllm/model_executor/models/ltx2.h +++ b/include/vllm/model_executor/models/ltx2.h @@ -51,8 +51,14 @@ // checkpoint: `caption_proj_before_connector=true` puts them in the TEXT // ENCODER, so the DiT has none (model_configurator.py:199-219). They are // phase L3. -// - `prompt_adaln_single` (model.py:223-227). LTX-2.5 sets -// `use_prompt_adaln_single=false`; see Ltx2PromptKvCache. +// +// PORTED 2026-08-13 — `prompt_adaln_single` / `audio_prompt_adaln_single` +// (model.py:222-227, :252-257), which this list previously carried as unported on +// the strength of "LTX-2.5 sets use_prompt_adaln_single=false". It does not: the +// flag defaults TRUE in both references (model.py:77, +// model_configurator.py:76/:138, diffusers transformer_ltx2.py:1185) and the +// shipped DiT carries the module's tensors. See +// .agents/specs/ltx25-prompt-adaln.md and issue #644. #pragma once #include @@ -112,8 +118,12 @@ struct Ltx2DitParams { bool double_precision_rope = false; bool apply_gated_attention = false; bool cross_attention_adaln = false; - // model_configurator.py:74-76. FALSE on LTX-2.5, which is what makes the - // cross-attention K/V timestep-independent — see Ltx2PromptKvCache. + // model_configurator.py:74-76 (`config.get("use_prompt_adaln_single", True)`), + // model.py:77, diffusers transformer_ltx2.py:1185 — TRUE by default in every + // reference, and TRUE for the shipped LTX-2.5 DiT, which carries the module's + // tensors. When true a prompt-side AdaLN MLP adds a timestep term to the + // cross-attention K/V modulation (transformer.py:441-443), which is what makes + // those K/V timestep-DEPENDENT and so uncacheable — see Ltx2PromptKvCache. bool use_prompt_adaln_single = true; // model_configurator.py:77-80. LTX-2.5 (gemma4) sets ff_bias=false and leaves // audio_ff_bias at its true default; the checkpoint's shapes agree. @@ -214,6 +224,11 @@ struct Ltx2BlockWeights { struct Ltx2DitWeights { Ltx2LinearWeight patchify_proj, proj_out; Ltx2AdaLayerNormSingleWeights adaln_single; + // model.py:222-227 / :252-257 — built only when `cross_attention_adaln AND + // use_prompt_adaln_single`, with embedding_coefficient 2 (shift + scale for the + // prompt K/V), NOT `adaln_embedding_coefficient()`. Left unbound otherwise. + Ltx2AdaLayerNormSingleWeights prompt_adaln_single; + Ltx2AdaLayerNormSingleWeights audio_prompt_adaln_single; vt::Tensor scale_shift_table; // [2, dim] — the OUTPUT table (:230), not the block's [9, dim] Ltx2LinearWeight audio_patchify_proj, audio_proj_out; Ltx2AdaLayerNormSingleWeights audio_adaln_single; @@ -295,13 +310,19 @@ Ltx2AdalnOut Ltx2AdaLayerNormSingle(vt::Device device, const Ltx2AdaLayerNormSin std::vector Ltx2FeedForward(vt::Device device, const Ltx2FeedForwardWeights& w, const float* x, int64_t rows, int64_t dim, int64_t inner); -// The K/V half of Attention.forward, split out because LTX-2.5 can CACHE it: -// with `use_prompt_adaln_single=false` the prompt modulation carries no timestep -// term (transformer.py:441), so `to_k`/`to_v` over the modulated context — and -// their k_norm, and the absence of RoPE on the text path — depend only on the -// prompt. The denoise loop computes them ONCE PER REQUEST and reuses them for -// every step. Layout: k/v are [batch * context_tokens, heads * dim_head], held -// exactly as the attention op consumes them (post-norm, post-RoPE). +// The K/V half of Attention.forward, split out because a checkpoint that sets +// `use_prompt_adaln_single=false` can CACHE it: the prompt modulation then +// carries no timestep term (transformer.py:441-443), so `to_k`/`to_v` over the +// modulated context — and their k_norm, and the absence of RoPE on the text path +// — depend only on the prompt. The denoise loop computes them ONCE PER REQUEST +// and reuses them for every step. Layout: k/v are +// [batch * context_tokens, heads * dim_head], held exactly as the attention op +// consumes them (post-norm, post-RoPE). +// +// THIS DOES NOT APPLY TO THE SHIPPED LTX-2.5 DiT, which sets the flag TRUE +// (.agents/specs/ltx-2-5.md §1.2, and .agents/specs/ltx25-prompt-adaln.md). The +// mechanism stays here, gated bit-identical, for a checkpoint that does set it +// false; `Ltx2DitForward` refuses a cache when the flag is on. struct Ltx2CrossKv { std::vector k, v; }; @@ -463,6 +484,14 @@ struct Ltx2BlockArgs { // Per-token AdaLN modulation, [batch, tokens, coefficient * dim]. const float* video_timestep_modulation = nullptr; const float* audio_timestep_modulation = nullptr; + // The PROMPT-side AdaLN modulation, [batch, 1, 2 * dim] — shift then scale, one + // row per batch element broadcast over the prompt tokens (transformer.py:443, + // whose `prompt_timestep` has token dimension 1 because `_prepare_timestep` ran + // on the modality's per-sample `sigma`). `nullptr` is upstream's + // `prompt_timestep is None`, i.e. `use_prompt_adaln_single=false`, in which case + // only the static per-block table applies (:441). + const float* video_prompt_modulation = nullptr; + const float* audio_prompt_modulation = nullptr; // Audio<->video cross-attention AdaLN inputs (transformer_args.py:388-411). const float* video_cross_scale_shift = nullptr; // [batch, video tokens, 4 * dim] const float* video_cross_gate = nullptr; // [batch, 1, dim] diff --git a/include/vllm/model_executor/models/ltx2_loader.h b/include/vllm/model_executor/models/ltx2_loader.h index 1c2da38a5..46aa57faa 100644 --- a/include/vllm/model_executor/models/ltx2_loader.h +++ b/include/vllm/model_executor/models/ltx2_loader.h @@ -98,27 +98,38 @@ // holds both copies at once, which is what wedged the box during MiniMax-H3's // port (minimax_h3.h:1598-1606). Same shape, same reason. // -// ─── WHAT THE SHIPPED DiT CARRIES THAT PHASE L2 DOES NOT PORT ──────────────── +// ─── WHAT THE SHIPPED DiT CARRIES THAT THIS PORT DOES NOT CARRY ────────────── // // MEASURED 2026-08-12 from the FP8 checkpoint's own header, and reported rather -// than absorbed. The file carries four families outside the L2 contract, and -// they fall into TWO groups that this comment used to conflate — corrected -// 2026-08-13, because the conflation is what made a downstream refusal state -// something untrue about the tree for a whole phase. +// than absorbed. The file carries four families outside the ORIGINAL L2 contract, +// and they fall into THREE groups. Two of them this comment once conflated +// (corrected 2026-08-13, because the conflation made a downstream refusal state +// something untrue about the tree for a whole phase); the third is a family that +// has since been ported. // // UNPORTED. `Ltx2LoadDitFromSafetensors` REFUSES the load by naming these, and // only an explicit `allow_unported_modules` — which exists so the ported subset // stays gateable — proceeds, still reporting every one of them in `unported`: // -// prompt_adaln_single.*, audio_prompt_adaln_single.* -// Upstream builds these only when `cross_attention_adaln AND -// use_prompt_adaln_single` (model.py:222-226, :253-257). Their presence -// means the shipped LTX-2.5 sets `use_prompt_adaln_single = TRUE`, which -// contradicts .agents/specs/ltx-2-5.md §1.2 and ltx2.h:115-117 — and with -// it the prompt-K/V "free win", whose whole premise is that the prompt -// modulation carries no timestep term. // keyframes_abs_pos_embedding [1, 4096] // So `use_keyframes_abs_pos_embedding = TRUE`, contradicting ltx2.h:47-49. +// This is now the ONLY flag `Ltx2AdoptDeclaredDitParams` clears in its +// config copy, and it must stay that way: a flag cleared there is invisible +// to the contract-equality check, so clearing a PORTED one silently drops +// its tensors. +// +// PORTED 2026-08-13 — no longer named in that refusal: +// +// prompt_adaln_single.*, audio_prompt_adaln_single.* +// Upstream builds these exactly when `cross_attention_adaln AND +// use_prompt_adaln_single` (model.py:222-226, :252-256), and the flag +// defaults TRUE in every reference (model.py:77, +// model_configurator.py:76/:138, diffusers transformer_ltx2.py:1185). Their +// presence means the shipped LTX-2.5 sets it TRUE — which is what +// .agents/specs/ltx-2-5.md §1.2 established and what row LTX25-PROMPT-ADALN +// (.agents/specs/ltx25-prompt-adaln.md, issue #644) implemented. The +// prompt-K/V cache stays correct-and-inapplicable for this checkpoint, and +// is refused by name rather than served. // // LOADED ELSEWHERE — NOT UNPORTED, and never named in that refusal: // @@ -515,11 +526,17 @@ nlohmann::json Ltx2ReadCheckpointConfig(const SafetensorsFile& file); // `Ltx2StreamDitToDevice` directly and therefore owes the same adoption. // // `allow_unported_modules` clears `use_keyframes_abs_pos_embedding` IN A COPY of -// the config before parsing, mirroring what the loader does for -// `use_prompt_adaln_single`: the flag is cleared for the CONTRACT, the module +// the config before parsing: the flag is cleared for the CONTRACT, the module // stays unported, and the checkpoint's `unported` list still names it. Without // the opt-in `ParseLtx2DitParams` throws, which is the refusal. // +// IT CLEARS EXACTLY THAT ONE FLAG, and the scoping is the rule, not an accident. +// It also used to clear `use_prompt_adaln_single`, whose module has been ported +// since 2026-08-13 — so the opt-in a real render REQUIRES was silently turning a +// correctness setting off, and the contract-equality check below could not see it +// because both sides had been forced to the same cleared value. A ported module's +// flag belongs in the contract; only a module nothing applies may be cleared here. +// // `source` names the config in every refusal, so a reader knows whether the // checkpoint declared it or a caller supplied it. Ltx2DitParams Ltx2AdoptDeclaredDitParams(const nlohmann::json& config, diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index fb683ce54..0953aebff 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -170,11 +170,20 @@ inline constexpr char kLtx2ModelVersionExtra[] = "model_version"; // belonging to another checkpoint is refused rather than bound. inline constexpr char kLtx2DitConfigPathExtra[] = "dit_config_path"; -// Proceed past the module families the L2 contract does not carry — -// `prompt_adaln_single` / `audio_prompt_adaln_single` and +// Proceed past the module families this port does not carry — // `keyframes_abs_pos_embedding` (ltx2_loader.h). "1" opts in; anything else -// leaves the loader's refusal in place. The shipped DiTs all carry at least one -// of them, so this is the flag that says "gate the ported subset knowingly". +// leaves the loader's refusal in place. The shipped DiTs carry it, so this is the +// flag that says "gate the ported subset knowingly". +// +// IT MUST NEVER DISABLE A PORTED FEATURE, and until 2026-08-13 it did: +// `prompt_adaln_single` / `audio_prompt_adaln_single` were on this list, and +// setting the extra reached three loader assignments that cleared +// `use_prompt_adaln_single`, so every real render dropped the timestep half of +// the prompt K/V modulation — finite, same-shaped, and invisible to every gate. +// Those families are ported now (.agents/specs/ltx25-prompt-adaln.md, issue +// #644), the loader asserts the flag against the file rather than clearing it, +// and this extra is scoped to the one module nothing applies. Adding a family +// here only ever means "the forward genuinely has no code for this". // // The two `*_embeddings_connector` families are NOT in that set and this extra // has nothing to do with them: they are outside the DiT contract by design and diff --git a/scripts/gen-ltx2-goldens.py b/scripts/gen-ltx2-goldens.py index 367cc4ddb..4c5d3e54b 100644 --- a/scripts/gen-ltx2-goldens.py +++ b/scripts/gen-ltx2-goldens.py @@ -316,10 +316,19 @@ def build_modalities(masked: bool, audio_enabled: bool = True, dense_self_mask: return video, audio -def build_model(rope_type_name: str, double_rope: bool): +def build_model(rope_type_name: str, double_rope: bool, prompt_adaln: bool | None = None): + """`prompt_adaln` overrides ARCH's `use_prompt_adaln_single` for the flag-ON arm. + + Every OTHER parameter keeps its name, and the deterministic stream is keyed by + NAME alone, so the two arms share their common weights bit-for-bit. That is + what makes the flag-ON and flag-OFF forwards directly subtractable, which is + how the magnitude in section 6 is measured. + """ from ltx_core.model.transformer.model import LTXModel, LTXModelType # noqa: PLC0415 from ltx_core.model.transformer.rope import LTXRopeType # noqa: PLC0415 + if prompt_adaln is None: + prompt_adaln = ARCH["use_prompt_adaln_single"] model = LTXModel( model_type=LTXModelType.AudioVideo, num_attention_heads=ARCH["num_attention_heads"], @@ -344,7 +353,7 @@ def build_model(rope_type_name: str, double_rope: bool): double_precision_rope=double_rope, apply_gated_attention=ARCH["apply_gated_attention"], cross_attention_adaln=ARCH["cross_attention_adaln"], - use_prompt_adaln_single=ARCH["use_prompt_adaln_single"], + use_prompt_adaln_single=prompt_adaln, ff_bias=ARCH["ff_bias"], audio_ff_bias=ARCH["audio_ff_bias"], ) @@ -617,12 +626,15 @@ def emit_bricks(out, model) -> None: def emit_forward( out, tag: str, rope_type_name: str, double_rope: bool, masked: bool, audio_enabled: bool = True, dense_self_mask: bool = False, + prompt_adaln: bool | None = None, ) -> None: out.write( f"// --- forward case {tag}: rope={rope_type_name} float64_freqs={double_rope} " - f"masked={masked} audio_enabled={audio_enabled} dense_self_mask={dense_self_mask} ---\n" + f"masked={masked} audio_enabled={audio_enabled} dense_self_mask={dense_self_mask} " + f"prompt_adaln={ARCH['use_prompt_adaln_single'] if prompt_adaln is None else prompt_adaln}" + " ---\n" ) - model = build_model(rope_type_name, double_rope) + model = build_model(rope_type_name, double_rope, prompt_adaln) video, audio = build_modalities(masked, audio_enabled, dense_self_mask) with torch.no_grad(): vx, ax = model(video=video, audio=audio, perturbations=None) @@ -631,6 +643,129 @@ def emit_forward( return model +def emit_prompt_adaln(out) -> None: + """Section 6 — the PROMPT-SIDE AdaLN arm (`use_prompt_adaln_single=True`). + + This is upstream's DEFAULT (model.py:77, model_configurator.py:76 and :138; + diffusers transformer_ltx2.py:1185) and what the shipped LTX-2.5 DiT carries. + Everything above this point runs the flag OFF, so nothing above can observe a + port that drops the term — which is exactly what happened. + + Emitted here: + * the flag-ON parameter list, i.e. the 18-tensor contract the flag adds + (12 video + 6 audio), in upstream's own registration order; + * the prompt AdaLN MLP run STANDALONE on `sigma * timestep_scale_multiplier` + (transformer_args.py:274-277 -> :173-186), so a failure localizes to the + MLP rather than to the threading; + * two full forwards, unmasked and masked; + * the MAGNITUDE: flag-ON minus flag-OFF over the same shared weights. + """ + out.write("// --- section 6: the prompt-side AdaLN arm (use_prompt_adaln_single=TRUE) ---\n") + model = build_model("split", False, prompt_adaln=True) + + names, ranks, dims = [], [], [] + for name, param in model.named_parameters(): + names.append(name) + ranks.append(len(param.shape)) + dims.extend(int(d) for d in param.shape) + out.write("inline constexpr const char* kLtx2PromptAdalnParamNames[] = {\n") + for name in names: + out.write(f' "{name}",\n') + out.write("};\n\n") + emit_i64(out, "kLtx2PromptAdalnParamRanks", ranks) + emit_i64(out, "kLtx2PromptAdalnParamDims", dims) + emit_scalar(out, "kLtx2PromptAdalnParamCount", len(names)) + out.write("\n") + + # The MLP alone. `_prepare_timestep` scales by timestep_scale_multiplier and + # feeds the modality's SIGMA -- (B,), one scalar per sample -- not its + # per-token `timesteps`. Emitting both halves keeps that distinction gateable. + scale = float(ARCH["timestep_scale_multiplier"]) + vsigma = rand_input("input.video.sigma", (BATCH,), 0.25, 0.5) * scale + asigma = rand_input("input.audio.sigma", (BATCH,), 0.25, 0.5) * scale + vmod, _ = model.prompt_adaln_single(vsigma.flatten(), hidden_dtype=torch.float32) + amod, _ = model.audio_prompt_adaln_single(asigma.flatten(), hidden_dtype=torch.float32) + emit_f32(out, "kLtx2PromptAdalnVideoTimesteps", tensor(vsigma)) + emit_f32(out, "kLtx2PromptAdalnAudioTimesteps", tensor(asigma)) + emit_f32(out, "kLtx2PromptAdalnVideoModulation", tensor(vmod)) + emit_f32(out, "kLtx2PromptAdalnAudioModulation", tensor(amod)) + + emit_forward(out, "PromptAdaln", "split", False, False, prompt_adaln=True) + emit_forward(out, "PromptAdalnMasked", "split", False, True, prompt_adaln=True) + + +def measure_prompt_adaln_magnitude() -> str: + """How far the conditioning MOVES when the term is included, vs when it is not. + + Both arms share every common parameter bit-for-bit (the stream is keyed by + parameter NAME), so the difference below is the term itself and nothing else. + Reported as a comment in the generated header AND on stderr, because "does + this matter" is a number, not an argument. + """ + lines = [] + off = build_model("split", False, prompt_adaln=False) + on = build_model("split", False, prompt_adaln=True) + video, audio = build_modalities(False) + with torch.no_grad(): + vx_off, ax_off = off(video=video, audio=audio, perturbations=None) + vx_on, ax_on = on(video=video, audio=audio, perturbations=None) + + def rel(a, b): + a32 = a.to(torch.float32) + b32 = b.to(torch.float32) + denom = float(a32.abs().max()) + return float((b32 - a32).abs().max()), ( + float((b32 - a32).abs().max()) / denom if denom > 0 else float("nan") + ) + + vabs, vrel = rel(vx_off, vx_on) + aabs, arel = rel(ax_off, ax_on) + lines.append(f"// DiT video output: max|on-off| = {vabs:.6g} ({vrel * 100:.2f}% of max|off|)") + lines.append(f"// DiT audio output: max|on-off| = {aabs:.6g} ({arel * 100:.2f}% of max|off|)") + + # The modulated prompt context of block 0, which is where the term enters. + scale = float(ARCH["timestep_scale_multiplier"]) + with torch.no_grad(): + vmod, _ = on.prompt_adaln_single( + (video.sigma * scale).flatten(), hidden_dtype=torch.float32 + ) + vmod = vmod.view(BATCH, -1, vmod.size(-1)) + table = on.transformer_blocks[0].prompt_scale_shift_table[None, None] + static = table.expand(BATCH, 1, 2, video_dim()) + full = static + vmod.reshape(BATCH, vmod.shape[1], 2, -1) + shift_off, scale_off = static.unbind(dim=2) + shift_on, scale_on = full.unbind(dim=2) + ctx = video.context + kv_off = ctx * (1 + scale_off) + shift_off + kv_on = ctx * (1 + scale_on) + shift_on + kabs, krel = rel(kv_off, kv_on) + lines.append( + f"// block 0 modulated prompt K/V: max|on-off| = {kabs:.6g} " + f"({krel * 100:.2f}% of max|off|)" + ) + # How much of the K/V modulation is timestep-conditioned at all: the MLP row + # against the static per-block table it is added to (transformer.py:441-443). + static_max = float(table.abs().max()) + term_max = float(vmod.abs().max()) + lines.append( + f"// timestep term vs static table: max|term| = {term_max:.6g} vs " + f"max|table| = {static_max:.6g} ({term_max / static_max * 100:.1f}%)" + ) + lines.append( + "// The two output rows are bounded by this generator's SYNTHETIC weight scale" + ) + lines.append( + "// (0.05, param_spec above) and by a 2-block stack; they are the FLOOR the gate" + ) + lines.append( + "// needs, not a claim about the trained checkpoint. The K/V row is where the" + ) + lines.append("// term actually enters and is the number that answers 'does this matter'.") + text = "\n".join(lines) + print("prompt-AdaLN magnitude:\n" + text, file=sys.stderr) + return text + + def emit_masks(out) -> None: out.write("// --- section 5: the prompt and self-attention masks the masked case runs ---\n") emit_i64(out, "kLtx2VideoContextMask", video_context_mask()) @@ -680,6 +815,14 @@ def main() -> int: # only this case — separates a kernel that indexes the bias by query from # one that reads bias row 0 for every query. emit_forward(out, "DenseMask", "split", False, True, dense_self_mask=True) + # Upstream's DEFAULT arm, and the one the shipped checkpoint runs. + emit_prompt_adaln(out) + out.write( + "// --- the MEASURED magnitude of the prompt-AdaLN term ---\n" + "// Same shared weights, same inputs, flag ON vs OFF:\n" + + measure_prompt_adaln_magnitude() + + "\n" + ) out.write("} // namespace vllm_test\n") print(f"wrote {args.out}", file=sys.stderr) return 0 diff --git a/src/vllm/model_executor/models/ltx2.cpp b/src/vllm/model_executor/models/ltx2.cpp index 7e5a4cffc..3d9af69ce 100644 --- a/src/vllm/model_executor/models/ltx2.cpp +++ b/src/vllm/model_executor/models/ltx2.cpp @@ -268,17 +268,21 @@ std::vector EnumerateLtx2DitTensors(const Ltx2DitParams& p) { out.push_back({"scale_shift_table", {2, dim}}); out.push_back({"audio_scale_shift_table", {2, adim}}); + // model.py:222-226 / :252-256 — built only when BOTH flags hold, and always + // with embedding_coefficient 2 (shift + scale for the prompt K/V), never + // `adaln_embedding_coefficient()`. + const bool prompt_adaln = p.cross_attention_adaln && p.use_prompt_adaln_single; + // _init_video (model.py:202-232), in child-registration order. PushLinear(out, "patchify_proj", dim, p.in_channels, true); PushAdaLayerNormSingle(out, "adaln_single", dim, coefficient); - VT_CHECK(!p.use_prompt_adaln_single, - "ltx2: use_prompt_adaln_single=true adds a prompt AdaLN MLP (model.py:223-227) whose " - "timestep term makes the cross-attention K/V uncacheable; not ported in phase L2"); + if (prompt_adaln) PushAdaLayerNormSingle(out, "prompt_adaln_single", dim, 2); PushLinear(out, "proj_out", p.out_channels, dim, true); // _init_audio (model.py:234-262). PushLinear(out, "audio_patchify_proj", adim, p.audio_in_channels, true); PushAdaLayerNormSingle(out, "audio_adaln_single", adim, coefficient); + if (prompt_adaln) PushAdaLayerNormSingle(out, "audio_prompt_adaln_single", adim, 2); PushLinear(out, "audio_proj_out", p.audio_out_channels, adim, true); // _init_audio_video (model.py:264-287); num_scale_shift_values is 4 (:133). @@ -482,9 +486,15 @@ Ltx2DitWeights BindLtx2DitWeights(const Ltx2DitParams& p, w.audio_scale_shift_table = Lookup(t, "audio_scale_shift_table"); w.patchify_proj = BindLinear(t, "patchify_proj", true); w.adaln_single = BindAdaln(t, "adaln_single"); + // model.py:222-226 / :252-256 — bound only when the module exists, exactly as + // the optional block tables above are. Left default-constructed otherwise, and + // the forward reads it only when `use_prompt_adaln_single` says it is there. + const bool prompt_adaln = p.cross_attention_adaln && p.use_prompt_adaln_single; + if (prompt_adaln) w.prompt_adaln_single = BindAdaln(t, "prompt_adaln_single"); w.proj_out = BindLinear(t, "proj_out", true); w.audio_patchify_proj = BindLinear(t, "audio_patchify_proj", true); w.audio_adaln_single = BindAdaln(t, "audio_adaln_single"); + if (prompt_adaln) w.audio_prompt_adaln_single = BindAdaln(t, "audio_prompt_adaln_single"); w.audio_proj_out = BindLinear(t, "audio_proj_out", true); w.av_ca_video_scale_shift = BindAdaln(t, "av_ca_video_scale_shift_adaln_single"); w.av_ca_audio_scale_shift = BindAdaln(t, "av_ca_audio_scale_shift_adaln_single"); diff --git a/src/vllm/model_executor/models/ltx2_device.cpp b/src/vllm/model_executor/models/ltx2_device.cpp index 3408007ab..e54436e4e 100644 --- a/src/vllm/model_executor/models/ltx2_device.cpp +++ b/src/vllm/model_executor/models/ltx2_device.cpp @@ -458,6 +458,10 @@ struct BlockArgsDev { bool video_enabled = true, audio_enabled = true; const Tensor* video_timestep_modulation = nullptr; const Tensor* audio_timestep_modulation = nullptr; + // The prompt-side AdaLN modulation, [batch, 2 * width] — nullptr is upstream's + // `prompt_timestep is None` (transformer.py:442), i.e. the flag is off. + const Tensor* video_prompt_modulation = nullptr; + const Tensor* audio_prompt_modulation = nullptr; const Tensor* video_cross_scale_shift = nullptr; const Tensor* video_cross_gate = nullptr; const Tensor* audio_cross_scale_shift = nullptr; @@ -480,6 +484,7 @@ struct BlockArgsDev { // device twin of ltx2_dit.cpp's TextCrossAttention. void TextCrossAttentionDev(Ctx& c, const Ltx2AttentionWeights& attn, const Tensor& sst, const Tensor& prompt_table, const Tensor& modulation, + const Tensor* prompt_modulation, const Tensor& x_normed, const Tensor& context, const Tensor* context_bias, int64_t batch, int64_t tokens, int64_t context_tokens, int64_t width, int64_t heads, @@ -501,18 +506,44 @@ void TextCrossAttentionDev(Ctx& c, const Ltx2AttentionWeights& attn, const Tenso c.s, c.s); // apply_cross_attention_adaln (transformer.py:420-447): the STATIC [2, dim] - // per-block table, with no timestep term at all when use_prompt_adaln_single is - // false (:441-443). `src_row_stride = 0` is that broadcast, and the table is - // read at F32 while the stream stays at c.s — which is the whole reason - // `modulate` carries a separate src_dtype. + // per-block table (:441), plus the prompt-side AdaLN row when the flag is on + // (:442-443). VT_CHECK(prompt_table.dtype == DType::kF32, "ltx2 device: the prompt scale-shift table is F32 in the checkpoint and is read as " "F32 here; a narrowed table would be the dtype rule applied backwards"); DBuf encoder(c.d, c.s, {batch * context_tokens, width}); c.d.b.Copy(c.d.q, encoder.ptr(), context.data, encoder.bytes()); - auto* table = prompt_table.Ptr(); - c.k->modulate(c.d.q, encoder.t().data, table + width, table, batch * context_tokens, width, 0, - c.s, DType::kF32); + if (prompt_modulation == nullptr) { + // No timestep term at all. `src_row_stride = 0` is the broadcast of the + // table's single row over every token, and the table is read at F32 while the + // stream stays at c.s — which is the whole reason `modulate` carries a + // separate src_dtype. + auto* table = prompt_table.Ptr(); + c.k->modulate(c.d.q, encoder.t().data, table + width, table, batch * context_tokens, width, 0, + c.s, DType::kF32); + } else { + // `kv_modulation = table[None, None] + prompt_timestep.reshape(B, 1, 2, -1)` + // (:441-443) is EXACTLY `ada_value`'s `table[row] + modulation[r, row]` over + // a 2-parameter modulation with one row per BATCH element, so the sum is + // formed here the way every other table+modulation sum in this file is — + // before `(1 + scale)` applies, which is the order upstream rounds in. + DBuf shift_kv = AdaValueDev(c, prompt_table, *prompt_modulation, batch, width, + /*num_params=*/2, /*index=*/0); + DBuf scale_kv = AdaValueDev(c, prompt_table, *prompt_modulation, batch, width, + /*num_params=*/2, /*index=*/1); + // `modulate`'s `src_row_stride` is a single stride, so it can broadcast ONE + // row over every token (stride 0) or give every row its own (stride width) — + // but not "row b for this batch element's context_tokens rows". The batch + // loop supplies that offset rather than widening the kernel's contract for a + // dimension that is 1 or 2 in every shipped call. + const int64_t elem = static_cast(vt::SizeOf(c.s)); + for (int64_t b = 0; b < batch; ++b) { + void* dst = static_cast(encoder.ptr()) + b * context_tokens * width * elem; + const void* sc = static_cast(scale_kv.t().data) + b * width * elem; + const void* sh = static_cast(shift_kv.t().data) + b * width * elem; + c.k->modulate(c.d.q, dst, sc, sh, context_tokens, width, 0, c.s, c.s); + } + } AttnArgsDev a; a.batch = batch; @@ -575,7 +606,8 @@ void BlockForwardDev(Ctx& c, const Ltx2BlockWeights& w, const BlockArgsDev& args RmsNormNoWeight(c, vx_normed->t(), *video_x, rows, dim); TextCrossAttentionDev(c, w.attn2, w.scale_shift_table, w.prompt_scale_shift_table, - *args.video_timestep_modulation, vx_normed->t(), *args.video_context, + *args.video_timestep_modulation, args.video_prompt_modulation, + vx_normed->t(), *args.video_context, args.video_context_bias, batch, tv, args.video_context_tokens, dim, c.p->num_attention_heads, c.p->attention_head_dim, *video_x); } @@ -609,6 +641,7 @@ void BlockForwardDev(Ctx& c, const Ltx2BlockWeights& w, const BlockArgsDev& args TextCrossAttentionDev(c, w.audio_attn2, w.audio_scale_shift_table, w.audio_prompt_scale_shift_table, *args.audio_timestep_modulation, + args.audio_prompt_modulation, ax_normed->t(), *args.audio_context, args.audio_context_bias, batch, ta, args.audio_context_tokens, adim, c.p->audio_num_attention_heads, c.p->audio_attention_head_dim, *audio_x); @@ -740,6 +773,8 @@ struct PreparedStreamDev { DevFreqs pe, cross_pe; std::optional cross_scale_shift; std::optional cross_gate; + // [batch, 2 * width] — empty when use_prompt_adaln_single is false. + std::optional prompt_modulation; }; // _prepare_timestep (transformer_args.py:173-186) + AdaLayerNormSingle. @@ -757,6 +792,7 @@ PreparedStreamDev PrepareStreamDev(Ctx& c, const Ltx2LinearWeight& patchify, const Ltx2AdaLayerNormSingleWeights& adaln, const Ltx2AdaLayerNormSingleWeights& cross_scale_shift_adaln, const Ltx2AdaLayerNormSingleWeights& cross_gate_adaln, + const Ltx2AdaLayerNormSingleWeights* prompt_adaln, const Ltx2ModalityInput& m, int64_t width, int64_t in_channels, int64_t n_pos_dims, const std::vector& max_pos, int64_t heads, const Ltx2ModalityInput* cross, @@ -774,6 +810,19 @@ PreparedStreamDev PrepareStreamDev(Ctx& c, const Ltx2LinearWeight& patchify, out.modulation = std::move(ada.modulation); out.embedded = std::move(ada.embedded); + // transformer_args.py:274-277 — the PROMPT-side AdaLN runs on this modality's + // own SIGMA, [batch], not on its per-token `timesteps`. + if (prompt_adaln != nullptr) { + VT_CHECK(m.sigma != nullptr, + "ltx2: use_prompt_adaln_single=true needs this modality's sigma " + "(transformer_args.py:274-277); it drives the prompt-side AdaLN MLP whose output is " + "added to the cross-attention K/V modulation, and a missing sigma would silently " + "fall back to the static table"); + AdalnOutDev prompt = PrepareTimestepDev(c, *prompt_adaln, m.sigma, m.batch, width, + c.p->timestep_scale_multiplier); + out.prompt_modulation = std::move(prompt.modulation); + } + if (m.context != nullptr && m.context_tokens > 0) { out.context = UploadStream(c.d, c.s, m.context, {m.batch * m.context_tokens, context_dim}); } @@ -892,6 +941,10 @@ void CheckWeightsResident(const Ltx2DitWeights& w, vt::Device dev) { CheckLinearResident(w.audio_proj_out, dev, "audio_proj_out"); CheckAdalnResident(w.adaln_single, dev, "adaln_single"); CheckAdalnResident(w.audio_adaln_single, dev, "audio_adaln_single"); + // Bound only when use_prompt_adaln_single is on; `CheckResident` no-ops on an + // unbound view, so this needs no flag and cannot go stale against one. + CheckAdalnResident(w.prompt_adaln_single, dev, "prompt_adaln_single"); + CheckAdalnResident(w.audio_prompt_adaln_single, dev, "audio_prompt_adaln_single"); CheckAdalnResident(w.av_ca_video_scale_shift, dev, "av_ca_video_scale_shift"); CheckAdalnResident(w.av_ca_audio_scale_shift, dev, "av_ca_audio_scale_shift"); CheckAdalnResident(w.av_ca_a2v_gate, dev, "av_ca_a2v_gate"); @@ -1044,14 +1097,18 @@ Ltx2DitOutputs Ltx2DitForwardDevice(vt::Queue& queue, const Ltx2DitParams& param OnesCache ones(d); Ctx c{d, compute_dtype, Glue(d), ¶ms, &ones}; + // model.py:222-226 / :252-256 — the module exists only when BOTH flags hold. + const bool prompt_adaln = params.cross_attention_adaln && params.use_prompt_adaln_single; PreparedStreamDev vs = PrepareStreamDev( c, weights.patchify_proj, weights.adaln_single, weights.av_ca_video_scale_shift, - weights.av_ca_a2v_gate, *video, dim, params.in_channels, 3, + weights.av_ca_a2v_gate, prompt_adaln ? &weights.prompt_adaln_single : nullptr, *video, dim, + params.in_channels, 3, params.positional_embedding_max_pos, params.num_attention_heads, audio, params.cross_attention_dim); PreparedStreamDev as = PrepareStreamDev( c, weights.audio_patchify_proj, weights.audio_adaln_single, weights.av_ca_audio_scale_shift, - weights.av_ca_v2a_gate, *audio, adim, params.audio_in_channels, 1, + weights.av_ca_v2a_gate, prompt_adaln ? &weights.audio_prompt_adaln_single : nullptr, *audio, + adim, params.audio_in_channels, 1, params.audio_positional_embedding_max_pos, params.audio_num_attention_heads, video, params.audio_cross_attention_dim); @@ -1066,6 +1123,8 @@ Ltx2DitOutputs Ltx2DitForwardDevice(vt::Queue& queue, const Ltx2DitParams& param a.audio_enabled = audio->enabled; a.video_timestep_modulation = &vs.modulation->t(); a.audio_timestep_modulation = &as.modulation->t(); + a.video_prompt_modulation = vs.prompt_modulation ? &vs.prompt_modulation->t() : nullptr; + a.audio_prompt_modulation = as.prompt_modulation ? &as.prompt_modulation->t() : nullptr; a.video_cross_scale_shift = vs.cross_scale_shift ? &vs.cross_scale_shift->t() : nullptr; a.video_cross_gate = vs.cross_gate ? &vs.cross_gate->t() : nullptr; a.audio_cross_scale_shift = as.cross_scale_shift ? &as.cross_scale_shift->t() : nullptr; diff --git a/src/vllm/model_executor/models/ltx2_dit.cpp b/src/vllm/model_executor/models/ltx2_dit.cpp index 0abd15fe3..b21655c64 100644 --- a/src/vllm/model_executor/models/ltx2_dit.cpp +++ b/src/vllm/model_executor/models/ltx2_dit.cpp @@ -114,18 +114,45 @@ void PostSelfAttention(float* x, const float* y, const std::vector& gate, } // apply_cross_attention_adaln (transformer.py:420-447). `prompt_table` is the -// STATIC [2, dim] per-block table; with use_prompt_adaln_single=false there is no -// timestep term at all (:441-443), which is exactly what makes the resulting K/V -// cacheable across denoise steps. +// STATIC [2, width] per-block table (:441). `prompt_mod` is the prompt-side AdaLN +// MLP's output for this stream, [batch, 2 * width] — shift row then scale row, +// one row per BATCH element because `_prepare_timestep` ran on the modality's +// per-sample `sigma` (transformer_args.py:274-277). It is nullptr exactly when +// upstream's `prompt_timestep is None` (:442), i.e. use_prompt_adaln_single=false +// — which is what makes the resulting K/V cacheable across denoise steps. +// +// ORDER. Upstream sums the table and the timestep row FIRST and only then +// applies `(1 + scale)` (:441-446). Folding the two additions the other way round +// would round differently, so the sum is materialized here as upstream forms it. std::vector ModulateContext(const float* context, const vt::Tensor& prompt_table, - int64_t rows, int64_t width) { - const float* shift_kv = prompt_table.Ptr(); - const float* scale_kv = prompt_table.Ptr() + width; - std::vector out(static_cast(rows * width)); - for (int64_t r = 0; r < rows; ++r) { - const float* src = context + r * width; - float* dst = out.data() + r * width; - for (int64_t c = 0; c < width; ++c) dst[c] = src[c] * (1.0f + scale_kv[c]) + shift_kv[c]; + const float* prompt_mod, int64_t batch, + int64_t context_tokens, int64_t width) { + const float* table_shift = prompt_table.Ptr(); + const float* table_scale = prompt_table.Ptr() + width; + std::vector out(static_cast(batch * context_tokens * width)); + std::vector shift_kv(static_cast(width)); + std::vector scale_kv(static_cast(width)); + for (int64_t b = 0; b < batch; ++b) { + for (int64_t c = 0; c < width; ++c) { + shift_kv[static_cast(c)] = table_shift[c]; + scale_kv[static_cast(c)] = table_scale[c]; + } + if (prompt_mod != nullptr) { + const float* m = prompt_mod + b * 2 * width; + for (int64_t c = 0; c < width; ++c) { + shift_kv[static_cast(c)] += m[c]; + scale_kv[static_cast(c)] += m[width + c]; + } + } + for (int64_t s = 0; s < context_tokens; ++s) { + const int64_t r = b * context_tokens + s; + const float* src = context + r * width; + float* dst = out.data() + r * width; + for (int64_t c = 0; c < width; ++c) { + dst[c] = src[c] * (1.0f + scale_kv[static_cast(c)]) + + shift_kv[static_cast(c)]; + } + } } return out; } @@ -148,7 +175,8 @@ void AddGatedBroadcast(float* x, const std::vector& y, const std::vector< void TextCrossAttention(vt::Device device, const Ltx2DitParams& params, const Ltx2AttentionWeights& attn, const vt::Tensor& sst, const vt::Tensor& prompt_table, const float* modulation, - const float* x_normed, const float* context, const float* context_bias, + const float* prompt_modulation, const float* x_normed, + const float* context, const float* context_bias, int64_t batch, int64_t tokens, int64_t context_tokens, int64_t width, int64_t heads, int64_t dim_head, const Ltx2CrossKv* kv_in, Ltx2CrossKv* kv_out, float* x) { @@ -174,7 +202,8 @@ void TextCrossAttention(vt::Device device, const Ltx2DitParams& params, // The modulated context is only needed when the K/V are actually recomputed. std::vector encoder; if (kv_in == nullptr) { - encoder = ModulateContext(context, prompt_table, batch * context_tokens, width); + encoder = ModulateContext(context, prompt_table, prompt_modulation, batch, context_tokens, + width); } Ltx2AttentionArgs a; @@ -250,7 +279,8 @@ void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, PostSelfAttention(video_x, msa.data(), gate, batch * tv, dim, eps, &vx_normed); TextCrossAttention(device, params, w.attn2, w.scale_shift_table, w.prompt_scale_shift_table, - args.video_timestep_modulation, vx_normed.data(), args.video_context, + args.video_timestep_modulation, args.video_prompt_modulation, + vx_normed.data(), args.video_context, args.video_context_bias, batch, tv, args.video_context_tokens, dim, params.num_attention_heads, params.attention_head_dim, args.prompt_kv_filled ? args.video_prompt_kv : nullptr, @@ -287,6 +317,7 @@ void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, TextCrossAttention(device, params, w.audio_attn2, w.audio_scale_shift_table, w.audio_prompt_scale_shift_table, args.audio_timestep_modulation, + args.audio_prompt_modulation, ax_normed.data(), args.audio_context, args.audio_context_bias, batch, ta, args.audio_context_tokens, adim, params.audio_num_attention_heads, params.audio_attention_head_dim, @@ -432,6 +463,7 @@ struct PreparedStream { Ltx2FreqsCis cross_pe; std::vector cross_scale_shift; // [batch, tokens, 4 * width] std::vector cross_gate; // [batch, 1, width] + std::vector prompt_modulation; // [batch, 1, 2 * width], empty when the flag is off }; // _prepare_timestep (transformer_args.py:173-186) + AdaLayerNormSingle. @@ -451,6 +483,7 @@ PreparedStream PrepareStream(vt::Device device, const Ltx2DitParams& params, const Ltx2AdaLayerNormSingleWeights& adaln, const Ltx2AdaLayerNormSingleWeights& cross_scale_shift_adaln, const Ltx2AdaLayerNormSingleWeights& cross_gate_adaln, + const Ltx2AdaLayerNormSingleWeights* prompt_adaln, const Ltx2ModalityInput& m, int64_t width, int64_t in_channels, int64_t n_pos_dims, const std::vector& max_pos, int64_t heads, const Ltx2ModalityInput* cross) { @@ -478,6 +511,21 @@ PreparedStream PrepareStream(vt::Device device, const Ltx2DitParams& params, PrepareTimestep(device, adaln, m.timesteps, rows, width, params.timestep_scale_multiplier, &out.modulation, &out.embedded); + // transformer_args.py:274-277 — the PROMPT-side AdaLN runs on this modality's + // own SIGMA, [batch], not on its per-token `timesteps`. `_prepare_timestep` + // applies the same timestep_scale_multiplier, and the result views to + // [batch, 1, 2 * width]: one row per sample, broadcast over the prompt tokens. + if (prompt_adaln != nullptr) { + VT_CHECK(m.sigma != nullptr, + "ltx2: use_prompt_adaln_single=true needs this modality's sigma " + "(transformer_args.py:274-277); it drives the prompt-side AdaLN MLP whose output is " + "added to the cross-attention K/V modulation, and a missing sigma would silently " + "fall back to the static table"); + std::vector unused; + PrepareTimestep(device, *prompt_adaln, m.sigma, m.batch, width, + params.timestep_scale_multiplier, &out.prompt_modulation, &unused); + } + if (m.context_mask != nullptr) { out.context_bias = Ltx2PrepareContextMask(m.context_mask, m.batch, m.context_tokens); } @@ -675,19 +723,25 @@ Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, "timestep term and caching them would be wrong"); } + // model.py:222-226 / :252-256 — the module exists only when BOTH flags hold, and + // `prompt_adaln=getattr(self, "prompt_adaln_single", None)` (:313, :333) is how + // upstream turns its absence into `prompt_timestep is None`. + const bool prompt_adaln = params.cross_attention_adaln && params.use_prompt_adaln_single; const bool have_both = video != nullptr && audio != nullptr; PreparedStream vs, as; if (video != nullptr) { VT_CHECK(video->context_tokens == 0 || video->context != nullptr, "ltx2: the video stream needs a context when context_tokens > 0"); vs = PrepareStream(device, params, weights.patchify_proj, weights.adaln_single, - weights.av_ca_video_scale_shift, weights.av_ca_a2v_gate, *video, dim, + weights.av_ca_video_scale_shift, weights.av_ca_a2v_gate, + prompt_adaln ? &weights.prompt_adaln_single : nullptr, *video, dim, params.in_channels, 3, params.positional_embedding_max_pos, params.num_attention_heads, have_both ? audio : nullptr); } if (audio != nullptr) { as = PrepareStream(device, params, weights.audio_patchify_proj, weights.audio_adaln_single, - weights.av_ca_audio_scale_shift, weights.av_ca_v2a_gate, *audio, adim, + weights.av_ca_audio_scale_shift, weights.av_ca_v2a_gate, + prompt_adaln ? &weights.audio_prompt_adaln_single : nullptr, *audio, adim, params.audio_in_channels, 1, params.audio_positional_embedding_max_pos, params.audio_num_attention_heads, have_both ? video : nullptr); } @@ -719,6 +773,10 @@ Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, a.audio_enabled = audio != nullptr && audio->enabled; a.video_timestep_modulation = vs.modulation.empty() ? nullptr : vs.modulation.data(); a.audio_timestep_modulation = as.modulation.empty() ? nullptr : as.modulation.data(); + a.video_prompt_modulation = + vs.prompt_modulation.empty() ? nullptr : vs.prompt_modulation.data(); + a.audio_prompt_modulation = + as.prompt_modulation.empty() ? nullptr : as.prompt_modulation.data(); a.video_cross_scale_shift = vs.cross_scale_shift.empty() ? nullptr : vs.cross_scale_shift.data(); a.video_cross_gate = vs.cross_gate.empty() ? nullptr : vs.cross_gate.data(); a.audio_cross_scale_shift = as.cross_scale_shift.empty() ? nullptr : as.cross_scale_shift.data(); diff --git a/src/vllm/model_executor/models/ltx2_loader.cpp b/src/vllm/model_executor/models/ltx2_loader.cpp index 946771019..bfc85075c 100644 --- a/src/vllm/model_executor/models/ltx2_loader.cpp +++ b/src/vllm/model_executor/models/ltx2_loader.cpp @@ -540,18 +540,53 @@ std::vector ContractOf(const Ltx2DitParams& params) { list += families[i]; } Fail( - "the checkpoint carries modules phase L2 does NOT port: " + list + - ". They are not dropped silently: prompt_adaln_single / " - "audio_prompt_adaln_single mean use_prompt_adaln_single is TRUE, which " - "contradicts .agents/specs/ltx-2-5.md section 1.2 and voids the prompt-K/V " - "cache's premise; keyframes_abs_pos_embedding contradicts ltx2.h:47-49. The " - "two *_embeddings_connector families are NOT in this list and never will be " - "— they are outside the DiT contract by design and are loaded by " + "the checkpoint carries modules this port does NOT carry: " + list + + ". They are not dropped silently: keyframes_abs_pos_embedding means " + "use_keyframes_abs_pos_embedding is TRUE, and nothing here applies it. " + "prompt_adaln_single / audio_prompt_adaln_single are NO LONGER in this list " + "— they were ported by row LTX25-PROMPT-ADALN " + "(.agents/specs/ltx25-prompt-adaln.md, issue #644) and are now part of the " + "contract whenever the checkpoint carries them. The two " + "*_embeddings_connector families are not in this list either and never will " + "be — they are outside the DiT contract by design and are loaded by " "Ltx2LoadConnectorWeights, which is what the video engine calls. Pass " "Ltx2DitLoadOptions::allow_unported_modules to load the ported SUBSET, which " "still reports every one of them."); } +// THE GUARD THAT REPLACED THREE `use_prompt_adaln_single = false` ASSIGNMENTS. +// +// Those assignments existed only so `EnumerateLtx2DitTensors` would not throw on +// a module this port did not carry. The module is carried now, so the contract +// simply includes it — and the assignments would have become a silent DROP of the +// module's 12 parameters (18 entries in the shipped FP8 manifest, which carries a +// `weight_scale` per quantized weight), reachable through +// `allow_unported_modules=1`, which is exactly the defect issue #644 row 0 fixes. +// +// So the invariant they violated is asserted instead: the resolved flag must say +// what the FILE says. Deliberately an EQUALITY. Clearing it with the tensors +// present is the old defect; setting it with them absent would bind weights that +// are not there. Either way this refuses by name rather than rendering. +void CheckPromptAdalnAgreesWithFile(const DitPlan& plan, const Ltx2DitParams& params, + const char* where) { + bool file_has = false; + for (const Ltx2TensorSpec& spec : plan.manifest) { + if (spec.name == "prompt_adaln_single.linear.weight") { + file_has = true; + break; + } + } + if (file_has == params.use_prompt_adaln_single) return; + Fail(std::string(where) + ": use_prompt_adaln_single resolved to " + + (params.use_prompt_adaln_single ? "TRUE" : "FALSE") + " while the file " + + (file_has ? "DOES" : "does NOT") + + " carry prompt_adaln_single. Upstream builds that module exactly when the flag is set " + "(model.py:222-226), so the two cannot disagree. A FALSE flag over a file that carries " + "the module would drop the timestep term from every cross-attention K/V modulation " + "(transformer.py:441-443) and render with only the static table — finite, same-shaped, " + "and invisible to every gate, which is why this is checked rather than assumed."); +} + } // namespace Ltx2DitParams Ltx2ParseDitParamsFromCheckpoint(const SafetensorsFile& file, @@ -568,9 +603,7 @@ Ltx2DitCheckpoint Ltx2LoadDitFromSafetensors(const SafetensorsFile& file, out.quant = plan.quant; out.checkpoint_params = ParseLtx2DitParamsFromManifest(plan.manifest); out.params = out.checkpoint_params; - // The one flag whose module this port does not carry. Cleared for the CONTRACT - // only; `checkpoint_params` keeps what the file actually says. - out.params.use_prompt_adaln_single = false; + CheckPromptAdalnAgreesWithFile(plan, out.params, "Ltx2LoadDitFromSafetensors"); const std::vector contract = ContractOf(out.params); out.unported = UnportedFamilies(plan, contract); @@ -623,7 +656,7 @@ Ltx2DitCheckpoint Ltx2StreamDitToDevice(vt::Queue& queue, const SafetensorsFile& out.quant = plan.quant; out.checkpoint_params = ParseLtx2DitParamsFromManifest(plan.manifest); out.params = out.checkpoint_params; - out.params.use_prompt_adaln_single = false; + CheckPromptAdalnAgreesWithFile(plan, out.params, "Ltx2StreamDitToDevice"); const std::vector contract = ContractOf(out.params); out.unported = UnportedFamilies(plan, contract); @@ -976,16 +1009,20 @@ Ltx2DitParams Ltx2AdoptDeclaredDitParams(const nlohmann::json& config, // first-party LTX-2.5 DiT declares it, so reading the declared config verbatim // would refuse a real checkpoint the loader has just accepted under // `allow_unported_modules`. + // + // EXACTLY ONE FLAG, and that is now structural rather than a comment. This block + // also cleared `use_prompt_adaln_single`, whose module IS ported + // (.agents/specs/ltx25-prompt-adaln.md, issue #644) — so `allow_unported=1`, + // which a real render needs, silently turned off a correctness setting. Whatever + // is cleared here must be a module nothing below applies; a ported one belongs + // in the contract, where the equality check further down can see it. if (allow_unported_modules && copy.contains("transformer") && copy["transformer"].is_object()) { copy["transformer"]["use_keyframes_abs_pos_embedding"] = false; } nlohmann::json wrapper; wrapper["config"] = copy; - Ltx2DitParams declared = ParseLtx2DitParams(wrapper); - // The one flag the L2 contract clears, mirroring the manifest path above - // ("cleared for the CONTRACT only"). - declared.use_prompt_adaln_single = false; + const Ltx2DitParams declared = ParseLtx2DitParams(wrapper); const std::vector a = EnumerateLtx2DitTensors(from_shapes); const std::vector b = EnumerateLtx2DitTensors(declared); diff --git a/tests/vllm/models/ltx2_goldens.inc b/tests/vllm/models/ltx2_goldens.inc index ff1c964eb..964047788 100644 --- a/tests/vllm/models/ltx2_goldens.inc +++ b/tests/vllm/models/ltx2_goldens.inc @@ -2583,7 +2583,7 @@ inline constexpr float kLtx2AudioSelfMaskDense[] = { }; // --- section 4: the full DiT forward --- -// --- forward case Split: rope=split float64_freqs=False masked=False audio_enabled=True dense_self_mask=False --- +// --- forward case Split: rope=split float64_freqs=False masked=False audio_enabled=True dense_self_mask=False prompt_adaln=False --- inline constexpr float kLtx2ForwardSplitVideo[] = { -0.112128824f, -0.0224671941f, -0.0198019743f, -0.0801180974f, -0.00492008496f, -0.0997069329f, -0.141085327f, -0.110280119f, 0.242049828f, 0.0161605272f, 0.0691268668f, -0.128027037f, @@ -2622,7 +2622,7 @@ inline constexpr float kLtx2ForwardSplitAudio[] = { -0.0946981385f, 0.0834118277f, 0.000482047704f, -0.244372234f, 0.0668168291f, 0.076439999f, }; -// --- forward case Interleaved: rope=interleaved float64_freqs=False masked=False audio_enabled=True dense_self_mask=False --- +// --- forward case Interleaved: rope=interleaved float64_freqs=False masked=False audio_enabled=True dense_self_mask=False prompt_adaln=False --- inline constexpr float kLtx2ForwardInterleavedVideo[] = { -0.111679666f, -0.0226908159f, -0.019702062f, -0.0793902054f, -0.00504480582f, -0.0992663726f, -0.141819194f, -0.110411108f, 0.240116999f, 0.0159209836f, 0.0681923628f, -0.130362436f, @@ -2661,7 +2661,7 @@ inline constexpr float kLtx2ForwardInterleavedAudio[] = { -0.0936588645f, 0.0829930827f, -0.000500017079f, -0.244381592f, 0.0684131756f, 0.0764773339f, }; -// --- forward case Double: rope=split float64_freqs=True masked=False audio_enabled=True dense_self_mask=False --- +// --- forward case Double: rope=split float64_freqs=True masked=False audio_enabled=True dense_self_mask=False prompt_adaln=False --- inline constexpr float kLtx2ForwardDoubleVideo[] = { -0.112128653f, -0.0224669855f, -0.0198022574f, -0.0801180974f, -0.00491989311f, -0.0997067019f, -0.141085491f, -0.110280029f, 0.242049828f, 0.0161606614f, 0.0691268519f, -0.128027067f, @@ -2700,7 +2700,7 @@ inline constexpr float kLtx2ForwardDoubleAudio[] = { -0.0946978033f, 0.0834115744f, 0.000482207892f, -0.244371802f, 0.0668166652f, 0.0764404312f, }; -// --- forward case Masked: rope=split float64_freqs=False masked=True audio_enabled=True dense_self_mask=False --- +// --- forward case Masked: rope=split float64_freqs=False masked=True audio_enabled=True dense_self_mask=False prompt_adaln=False --- inline constexpr float kLtx2ForwardMaskedVideo[] = { -0.110146172f, -0.0195347276f, -0.0187677145f, -0.0824092105f, -0.00312865246f, -0.102393486f, -0.140672266f, -0.111012168f, 0.238716885f, 0.0195573885f, 0.0744788125f, -0.129781172f, @@ -2739,7 +2739,7 @@ inline constexpr float kLtx2ForwardMaskedAudio[] = { -0.0945852846f, 0.0829883665f, 0.000665711937f, -0.24394846f, 0.0671078935f, 0.0765393525f, }; -// --- forward case AudioOff: rope=split float64_freqs=False masked=False audio_enabled=False dense_self_mask=False --- +// --- forward case AudioOff: rope=split float64_freqs=False masked=False audio_enabled=False dense_self_mask=False prompt_adaln=False --- inline constexpr float kLtx2ForwardAudioOffVideo[] = { -0.11211022f, -0.0225408655f, -0.0197366029f, -0.0800089315f, -0.00495309103f, -0.099537693f, -0.141150832f, -0.110280804f, 0.242115393f, 0.0160462279f, 0.0691971257f, -0.127955094f, @@ -2778,7 +2778,7 @@ inline constexpr float kLtx2ForwardAudioOffAudio[] = { -0.0867422372f, 0.0840105489f, -0.000306644681f, -0.244602948f, 0.0686045066f, 0.0746730566f, }; -// --- forward case DenseMask: rope=split float64_freqs=False masked=True audio_enabled=True dense_self_mask=True --- +// --- forward case DenseMask: rope=split float64_freqs=False masked=True audio_enabled=True dense_self_mask=True prompt_adaln=False --- inline constexpr float kLtx2ForwardDenseMaskVideo[] = { -0.112229049f, -0.0196566191f, -0.0203980505f, -0.08254648f, -0.00351214409f, -0.102313384f, -0.139395878f, -0.10990762f, 0.244468585f, 0.0156435464f, 0.0681985766f, -0.126972139f, @@ -2817,4 +2817,424 @@ inline constexpr float kLtx2ForwardDenseMaskAudio[] = { -0.0944952741f, 0.0835703015f, 6.12687072e-05f, -0.244034111f, 0.0671091005f, 0.0762160644f, }; +// --- section 6: the prompt-side AdaLN arm (use_prompt_adaln_single=TRUE) --- +inline constexpr const char* kLtx2PromptAdalnParamNames[] = { + "scale_shift_table", + "audio_scale_shift_table", + "patchify_proj.weight", + "patchify_proj.bias", + "adaln_single.emb.timestep_embedder.linear_1.weight", + "adaln_single.emb.timestep_embedder.linear_1.bias", + "adaln_single.emb.timestep_embedder.linear_2.weight", + "adaln_single.emb.timestep_embedder.linear_2.bias", + "adaln_single.linear.weight", + "adaln_single.linear.bias", + "prompt_adaln_single.emb.timestep_embedder.linear_1.weight", + "prompt_adaln_single.emb.timestep_embedder.linear_1.bias", + "prompt_adaln_single.emb.timestep_embedder.linear_2.weight", + "prompt_adaln_single.emb.timestep_embedder.linear_2.bias", + "prompt_adaln_single.linear.weight", + "prompt_adaln_single.linear.bias", + "proj_out.weight", + "proj_out.bias", + "audio_patchify_proj.weight", + "audio_patchify_proj.bias", + "audio_adaln_single.emb.timestep_embedder.linear_1.weight", + "audio_adaln_single.emb.timestep_embedder.linear_1.bias", + "audio_adaln_single.emb.timestep_embedder.linear_2.weight", + "audio_adaln_single.emb.timestep_embedder.linear_2.bias", + "audio_adaln_single.linear.weight", + "audio_adaln_single.linear.bias", + "audio_prompt_adaln_single.emb.timestep_embedder.linear_1.weight", + "audio_prompt_adaln_single.emb.timestep_embedder.linear_1.bias", + "audio_prompt_adaln_single.emb.timestep_embedder.linear_2.weight", + "audio_prompt_adaln_single.emb.timestep_embedder.linear_2.bias", + "audio_prompt_adaln_single.linear.weight", + "audio_prompt_adaln_single.linear.bias", + "audio_proj_out.weight", + "audio_proj_out.bias", + "av_ca_video_scale_shift_adaln_single.emb.timestep_embedder.linear_1.weight", + "av_ca_video_scale_shift_adaln_single.emb.timestep_embedder.linear_1.bias", + "av_ca_video_scale_shift_adaln_single.emb.timestep_embedder.linear_2.weight", + "av_ca_video_scale_shift_adaln_single.emb.timestep_embedder.linear_2.bias", + "av_ca_video_scale_shift_adaln_single.linear.weight", + "av_ca_video_scale_shift_adaln_single.linear.bias", + "av_ca_audio_scale_shift_adaln_single.emb.timestep_embedder.linear_1.weight", + "av_ca_audio_scale_shift_adaln_single.emb.timestep_embedder.linear_1.bias", + "av_ca_audio_scale_shift_adaln_single.emb.timestep_embedder.linear_2.weight", + "av_ca_audio_scale_shift_adaln_single.emb.timestep_embedder.linear_2.bias", + "av_ca_audio_scale_shift_adaln_single.linear.weight", + "av_ca_audio_scale_shift_adaln_single.linear.bias", + "av_ca_a2v_gate_adaln_single.emb.timestep_embedder.linear_1.weight", + "av_ca_a2v_gate_adaln_single.emb.timestep_embedder.linear_1.bias", + "av_ca_a2v_gate_adaln_single.emb.timestep_embedder.linear_2.weight", + "av_ca_a2v_gate_adaln_single.emb.timestep_embedder.linear_2.bias", + "av_ca_a2v_gate_adaln_single.linear.weight", + "av_ca_a2v_gate_adaln_single.linear.bias", + "av_ca_v2a_gate_adaln_single.emb.timestep_embedder.linear_1.weight", + "av_ca_v2a_gate_adaln_single.emb.timestep_embedder.linear_1.bias", + "av_ca_v2a_gate_adaln_single.emb.timestep_embedder.linear_2.weight", + "av_ca_v2a_gate_adaln_single.emb.timestep_embedder.linear_2.bias", + "av_ca_v2a_gate_adaln_single.linear.weight", + "av_ca_v2a_gate_adaln_single.linear.bias", + "transformer_blocks.0.scale_shift_table", + "transformer_blocks.0.audio_scale_shift_table", + "transformer_blocks.0.scale_shift_table_a2v_ca_audio", + "transformer_blocks.0.scale_shift_table_a2v_ca_video", + "transformer_blocks.0.prompt_scale_shift_table", + "transformer_blocks.0.audio_prompt_scale_shift_table", + "transformer_blocks.0.attn1.q_norm.weight", + "transformer_blocks.0.attn1.k_norm.weight", + "transformer_blocks.0.attn1.to_q.weight", + "transformer_blocks.0.attn1.to_q.bias", + "transformer_blocks.0.attn1.to_k.weight", + "transformer_blocks.0.attn1.to_k.bias", + "transformer_blocks.0.attn1.to_v.weight", + "transformer_blocks.0.attn1.to_v.bias", + "transformer_blocks.0.attn1.to_gate_logits.weight", + "transformer_blocks.0.attn1.to_gate_logits.bias", + "transformer_blocks.0.attn1.to_out.0.weight", + "transformer_blocks.0.attn1.to_out.0.bias", + "transformer_blocks.0.attn2.q_norm.weight", + "transformer_blocks.0.attn2.k_norm.weight", + "transformer_blocks.0.attn2.to_q.weight", + "transformer_blocks.0.attn2.to_q.bias", + "transformer_blocks.0.attn2.to_k.weight", + "transformer_blocks.0.attn2.to_k.bias", + "transformer_blocks.0.attn2.to_v.weight", + "transformer_blocks.0.attn2.to_v.bias", + "transformer_blocks.0.attn2.to_gate_logits.weight", + "transformer_blocks.0.attn2.to_gate_logits.bias", + "transformer_blocks.0.attn2.to_out.0.weight", + "transformer_blocks.0.attn2.to_out.0.bias", + "transformer_blocks.0.ff.net.0.proj.weight", + "transformer_blocks.0.ff.net.2.weight", + "transformer_blocks.0.audio_attn1.q_norm.weight", + "transformer_blocks.0.audio_attn1.k_norm.weight", + "transformer_blocks.0.audio_attn1.to_q.weight", + "transformer_blocks.0.audio_attn1.to_q.bias", + "transformer_blocks.0.audio_attn1.to_k.weight", + "transformer_blocks.0.audio_attn1.to_k.bias", + "transformer_blocks.0.audio_attn1.to_v.weight", + "transformer_blocks.0.audio_attn1.to_v.bias", + "transformer_blocks.0.audio_attn1.to_gate_logits.weight", + "transformer_blocks.0.audio_attn1.to_gate_logits.bias", + "transformer_blocks.0.audio_attn1.to_out.0.weight", + "transformer_blocks.0.audio_attn1.to_out.0.bias", + "transformer_blocks.0.audio_attn2.q_norm.weight", + "transformer_blocks.0.audio_attn2.k_norm.weight", + "transformer_blocks.0.audio_attn2.to_q.weight", + "transformer_blocks.0.audio_attn2.to_q.bias", + "transformer_blocks.0.audio_attn2.to_k.weight", + "transformer_blocks.0.audio_attn2.to_k.bias", + "transformer_blocks.0.audio_attn2.to_v.weight", + "transformer_blocks.0.audio_attn2.to_v.bias", + "transformer_blocks.0.audio_attn2.to_gate_logits.weight", + "transformer_blocks.0.audio_attn2.to_gate_logits.bias", + "transformer_blocks.0.audio_attn2.to_out.0.weight", + "transformer_blocks.0.audio_attn2.to_out.0.bias", + "transformer_blocks.0.audio_ff.net.0.proj.weight", + "transformer_blocks.0.audio_ff.net.0.proj.bias", + "transformer_blocks.0.audio_ff.net.2.weight", + "transformer_blocks.0.audio_ff.net.2.bias", + "transformer_blocks.0.audio_to_video_attn.q_norm.weight", + "transformer_blocks.0.audio_to_video_attn.k_norm.weight", + "transformer_blocks.0.audio_to_video_attn.to_q.weight", + "transformer_blocks.0.audio_to_video_attn.to_q.bias", + "transformer_blocks.0.audio_to_video_attn.to_k.weight", + "transformer_blocks.0.audio_to_video_attn.to_k.bias", + "transformer_blocks.0.audio_to_video_attn.to_v.weight", + "transformer_blocks.0.audio_to_video_attn.to_v.bias", + "transformer_blocks.0.audio_to_video_attn.to_gate_logits.weight", + "transformer_blocks.0.audio_to_video_attn.to_gate_logits.bias", + "transformer_blocks.0.audio_to_video_attn.to_out.0.weight", + "transformer_blocks.0.audio_to_video_attn.to_out.0.bias", + "transformer_blocks.0.video_to_audio_attn.q_norm.weight", + "transformer_blocks.0.video_to_audio_attn.k_norm.weight", + "transformer_blocks.0.video_to_audio_attn.to_q.weight", + "transformer_blocks.0.video_to_audio_attn.to_q.bias", + "transformer_blocks.0.video_to_audio_attn.to_k.weight", + "transformer_blocks.0.video_to_audio_attn.to_k.bias", + "transformer_blocks.0.video_to_audio_attn.to_v.weight", + "transformer_blocks.0.video_to_audio_attn.to_v.bias", + "transformer_blocks.0.video_to_audio_attn.to_gate_logits.weight", + "transformer_blocks.0.video_to_audio_attn.to_gate_logits.bias", + "transformer_blocks.0.video_to_audio_attn.to_out.0.weight", + "transformer_blocks.0.video_to_audio_attn.to_out.0.bias", + "transformer_blocks.1.scale_shift_table", + "transformer_blocks.1.audio_scale_shift_table", + "transformer_blocks.1.scale_shift_table_a2v_ca_audio", + "transformer_blocks.1.scale_shift_table_a2v_ca_video", + "transformer_blocks.1.prompt_scale_shift_table", + "transformer_blocks.1.audio_prompt_scale_shift_table", + "transformer_blocks.1.attn1.q_norm.weight", + "transformer_blocks.1.attn1.k_norm.weight", + "transformer_blocks.1.attn1.to_q.weight", + "transformer_blocks.1.attn1.to_q.bias", + "transformer_blocks.1.attn1.to_k.weight", + "transformer_blocks.1.attn1.to_k.bias", + "transformer_blocks.1.attn1.to_v.weight", + "transformer_blocks.1.attn1.to_v.bias", + "transformer_blocks.1.attn1.to_gate_logits.weight", + "transformer_blocks.1.attn1.to_gate_logits.bias", + "transformer_blocks.1.attn1.to_out.0.weight", + "transformer_blocks.1.attn1.to_out.0.bias", + "transformer_blocks.1.attn2.q_norm.weight", + "transformer_blocks.1.attn2.k_norm.weight", + "transformer_blocks.1.attn2.to_q.weight", + "transformer_blocks.1.attn2.to_q.bias", + "transformer_blocks.1.attn2.to_k.weight", + "transformer_blocks.1.attn2.to_k.bias", + "transformer_blocks.1.attn2.to_v.weight", + "transformer_blocks.1.attn2.to_v.bias", + "transformer_blocks.1.attn2.to_gate_logits.weight", + "transformer_blocks.1.attn2.to_gate_logits.bias", + "transformer_blocks.1.attn2.to_out.0.weight", + "transformer_blocks.1.attn2.to_out.0.bias", + "transformer_blocks.1.ff.net.0.proj.weight", + "transformer_blocks.1.ff.net.2.weight", + "transformer_blocks.1.audio_attn1.q_norm.weight", + "transformer_blocks.1.audio_attn1.k_norm.weight", + "transformer_blocks.1.audio_attn1.to_q.weight", + "transformer_blocks.1.audio_attn1.to_q.bias", + "transformer_blocks.1.audio_attn1.to_k.weight", + "transformer_blocks.1.audio_attn1.to_k.bias", + "transformer_blocks.1.audio_attn1.to_v.weight", + "transformer_blocks.1.audio_attn1.to_v.bias", + "transformer_blocks.1.audio_attn1.to_gate_logits.weight", + "transformer_blocks.1.audio_attn1.to_gate_logits.bias", + "transformer_blocks.1.audio_attn1.to_out.0.weight", + "transformer_blocks.1.audio_attn1.to_out.0.bias", + "transformer_blocks.1.audio_attn2.q_norm.weight", + "transformer_blocks.1.audio_attn2.k_norm.weight", + "transformer_blocks.1.audio_attn2.to_q.weight", + "transformer_blocks.1.audio_attn2.to_q.bias", + "transformer_blocks.1.audio_attn2.to_k.weight", + "transformer_blocks.1.audio_attn2.to_k.bias", + "transformer_blocks.1.audio_attn2.to_v.weight", + "transformer_blocks.1.audio_attn2.to_v.bias", + "transformer_blocks.1.audio_attn2.to_gate_logits.weight", + "transformer_blocks.1.audio_attn2.to_gate_logits.bias", + "transformer_blocks.1.audio_attn2.to_out.0.weight", + "transformer_blocks.1.audio_attn2.to_out.0.bias", + "transformer_blocks.1.audio_ff.net.0.proj.weight", + "transformer_blocks.1.audio_ff.net.0.proj.bias", + "transformer_blocks.1.audio_ff.net.2.weight", + "transformer_blocks.1.audio_ff.net.2.bias", + "transformer_blocks.1.audio_to_video_attn.q_norm.weight", + "transformer_blocks.1.audio_to_video_attn.k_norm.weight", + "transformer_blocks.1.audio_to_video_attn.to_q.weight", + "transformer_blocks.1.audio_to_video_attn.to_q.bias", + "transformer_blocks.1.audio_to_video_attn.to_k.weight", + "transformer_blocks.1.audio_to_video_attn.to_k.bias", + "transformer_blocks.1.audio_to_video_attn.to_v.weight", + "transformer_blocks.1.audio_to_video_attn.to_v.bias", + "transformer_blocks.1.audio_to_video_attn.to_gate_logits.weight", + "transformer_blocks.1.audio_to_video_attn.to_gate_logits.bias", + "transformer_blocks.1.audio_to_video_attn.to_out.0.weight", + "transformer_blocks.1.audio_to_video_attn.to_out.0.bias", + "transformer_blocks.1.video_to_audio_attn.q_norm.weight", + "transformer_blocks.1.video_to_audio_attn.k_norm.weight", + "transformer_blocks.1.video_to_audio_attn.to_q.weight", + "transformer_blocks.1.video_to_audio_attn.to_q.bias", + "transformer_blocks.1.video_to_audio_attn.to_k.weight", + "transformer_blocks.1.video_to_audio_attn.to_k.bias", + "transformer_blocks.1.video_to_audio_attn.to_v.weight", + "transformer_blocks.1.video_to_audio_attn.to_v.bias", + "transformer_blocks.1.video_to_audio_attn.to_gate_logits.weight", + "transformer_blocks.1.video_to_audio_attn.to_gate_logits.bias", + "transformer_blocks.1.video_to_audio_attn.to_out.0.weight", + "transformer_blocks.1.video_to_audio_attn.to_out.0.bias", +}; + +inline constexpr int64_t kLtx2PromptAdalnParamRanks[] = { + 2, 2, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 2, + 2, 2, 2, 2, 1, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 1, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 2, 1, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 1, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 2, + 2, 2, 2, 2, 1, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 1, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 2, 1, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 1, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, +}; + +inline constexpr int64_t kLtx2PromptAdalnParamDims[] = { + 2, 32, 2, 16, 32, 8, 32, 32, 256, 32, 32, 32, + 32, 288, 32, 288, 32, 256, 32, 32, 32, 32, 64, 32, + 64, 8, 32, 8, 16, 6, 16, 16, 256, 16, 16, 16, + 16, 144, 16, 144, 16, 256, 16, 16, 16, 16, 32, 16, + 32, 6, 16, 6, 32, 256, 32, 32, 32, 32, 128, 32, + 128, 16, 256, 16, 16, 16, 16, 64, 16, 64, 32, 256, + 32, 32, 32, 32, 32, 32, 32, 16, 256, 16, 16, 16, + 16, 16, 16, 16, 9, 32, 9, 16, 5, 16, 5, 32, + 2, 32, 2, 16, 32, 32, 32, 32, 32, 32, 32, 32, + 32, 32, 32, 4, 32, 4, 32, 32, 32, 32, 32, 32, + 32, 32, 32, 32, 32, 32, 32, 32, 4, 32, 4, 32, + 32, 32, 128, 32, 32, 128, 16, 16, 16, 16, 16, 16, + 16, 16, 16, 16, 16, 4, 16, 4, 16, 16, 16, 16, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 4, 16, + 4, 16, 16, 16, 64, 16, 64, 16, 64, 16, 16, 16, + 16, 32, 16, 16, 16, 16, 16, 16, 16, 4, 32, 4, + 32, 16, 32, 16, 16, 16, 16, 16, 16, 32, 16, 16, + 32, 16, 4, 16, 4, 16, 16, 16, 9, 32, 9, 16, + 5, 16, 5, 32, 2, 32, 2, 16, 32, 32, 32, 32, + 32, 32, 32, 32, 32, 32, 32, 4, 32, 4, 32, 32, + 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, + 4, 32, 4, 32, 32, 32, 128, 32, 32, 128, 16, 16, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 4, 16, 4, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, + 16, 16, 4, 16, 4, 16, 16, 16, 64, 16, 64, 16, + 64, 16, 16, 16, 16, 32, 16, 16, 16, 16, 16, 16, + 16, 4, 32, 4, 32, 16, 32, 16, 16, 16, 16, 16, + 16, 32, 16, 16, 32, 16, 4, 16, 4, 16, 16, 16, +}; + +inline constexpr int64_t kLtx2PromptAdalnParamCount = 226; + +inline constexpr float kLtx2PromptAdalnVideoTimesteps[] = { + 619.314514f, 424.717133f, +}; + +inline constexpr float kLtx2PromptAdalnAudioTimesteps[] = { + 473.940033f, 471.390289f, +}; + +inline constexpr float kLtx2PromptAdalnVideoModulation[] = { + 0.00287261885f, 0.0172509346f, 0.0149077997f, 0.00213927543f, 0.016918283f, 0.00749109872f, + 0.00878333207f, -0.00887233205f, -0.0209352672f, 0.0127304699f, 0.0109749483f, -0.0130382832f, + -0.00827212445f, -0.00580324279f, -0.0134638241f, 0.00964590162f, -0.016003741f, -0.0158182271f, + -0.00942324009f, -0.0159717295f, -0.00885281898f, -0.00412264559f, -0.00142920704f, -0.0168404803f, + 0.0133440131f, -0.0136787146f, 0.0163368266f, 0.00325167179f, 0.0156327151f, 0.000542653725f, + -0.0113121318f, -0.00703316741f, -0.0195021238f, 0.0151473908f, 0.0180853736f, 0.0211514644f, + 0.000123613572f, -0.00695834123f, -0.00497911917f, -0.0171213523f, 0.0199449155f, -0.0122921225f, + 0.0164053645f, 0.000113433227f, 0.00943795405f, -0.0121485656f, 0.00261268346f, 0.00783557631f, + -0.00460897852f, 0.0153093971f, 0.0194506291f, 0.014570266f, -0.0162125602f, -0.00318641192f, + -0.013586767f, 0.00904892385f, -0.0157831647f, -0.0182319973f, 0.0170140117f, 0.00303793233f, + -0.0181853306f, -0.00718653016f, 0.0158703439f, 0.00020046602f, 0.00623062998f, 0.0252011884f, + 0.0163403824f, 0.000290016062f, 0.014817901f, 0.00552449375f, 0.00866025314f, -0.00905592274f, + -0.0175954159f, 0.0110378982f, 0.0159901753f, -0.0115343379f, -0.0125738885f, -0.00668510282f, + -0.0144314552f, 0.0128096947f, -0.0149616096f, -0.0173609052f, -0.0133896163f, -0.0159855876f, + -0.00701024011f, -0.00505556632f, -0.0011741718f, -0.0174259786f, 0.00958774798f, -0.0130143911f, + 0.0113152023f, 0.00158250146f, 0.0137644717f, 0.00236273953f, -0.0104373507f, -0.000455798348f, + -0.0210349094f, 0.0129983025f, 0.0220505167f, 0.0202870686f, 0.000495404762f, -0.00149141857f, + -0.00326095428f, -0.0143762864f, 0.0244831145f, -0.01158146f, 0.00986448769f, 0.000591582619f, + 0.013319497f, -0.0125091095f, 0.00394352386f, 0.0113571035f, -0.00364819146f, 0.013945261f, + 0.0238441341f, 0.0148018515f, -0.0194255728f, -0.00321829435f, -0.0185660385f, 0.0112335477f, + -0.0152186789f, -0.0156262573f, 0.0185506362f, 0.00343540264f, -0.0173967108f, -0.00660126004f, + 0.0198770091f, -0.000762573676f, +}; + +inline constexpr float kLtx2PromptAdalnAudioModulation[] = { + 0.0187722091f, -0.00105935254f, -0.00853663217f, 0.00841550902f, -0.0137612084f, -0.0173807591f, + -0.0139704188f, -0.0109588308f, 0.00186792947f, -0.00266267685f, -0.00698805135f, -0.0129980808f, + 0.00670515466f, -0.0131825786f, 0.0064191781f, 0.00632017246f, -0.0122615304f, 0.00804066937f, + 0.0122477906f, 0.0157948378f, 0.0128126917f, -0.00316316052f, -0.00883637834f, -0.0100117959f, + -0.0133155594f, 0.00456165429f, 0.0183124971f, -0.0130120357f, 0.018200513f, 0.00201772549f, + 0.0119188903f, -0.0103679141f, 0.0201762542f, -0.000246715499f, -0.00900617056f, 0.00820883084f, + -0.0132663827f, -0.0180532821f, -0.0145739987f, -0.0113234669f, 0.00138104265f, -0.00278934441f, + -0.00686988048f, -0.013545204f, 0.00675359881f, -0.0138096726f, 0.00569557305f, 0.00596735114f, + -0.0114674894f, 0.00765935751f, 0.0126606431f, 0.0161463916f, 0.0141690942f, -0.00382941356f, + -0.0089492118f, -0.00970011577f, -0.0132083837f, 0.00381002505f, 0.0185851343f, -0.0123645896f, + 0.0179123916f, 0.00177870644f, 0.0127155371f, -0.0096385479f, +}; + +// --- forward case PromptAdaln: rope=split float64_freqs=False masked=False audio_enabled=True dense_self_mask=False prompt_adaln=True --- +inline constexpr float kLtx2ForwardPromptAdalnVideo[] = { + -0.112144604f, -0.0224400591f, -0.0198658258f, -0.0801219419f, -0.00492565054f, -0.0996806622f, + -0.141095862f, -0.110312767f, 0.242095038f, 0.0162332151f, 0.0690327138f, -0.127996832f, + -0.327992439f, -0.067813158f, -0.0181498341f, 0.0133159151f, -0.285277247f, 0.0288095567f, + -0.103010967f, 0.0120137185f, 0.175781652f, -0.133603424f, -0.20695813f, -0.182875797f, + -0.275936812f, -0.0216972958f, 0.194740638f, -0.250197232f, -0.196145147f, -0.0676094294f, + -0.115069151f, -0.0885243863f, -0.0418822765f, 0.120427184f, -0.0467535444f, -0.166882023f, + -0.145086944f, -0.127097756f, -0.125268698f, -0.0937946662f, -0.10143815f, -0.0346935503f, + 0.347262383f, -0.0154392701f, 0.0253714621f, -0.151571572f, 0.0893032998f, -0.217105299f, + -0.113787532f, 0.107526913f, 0.111182854f, -0.0174661856f, 0.0259091742f, -0.199445173f, + -0.0387741439f, -0.157782555f, 0.159147635f, 0.0321634449f, 0.232984737f, -0.122400187f, + -0.27637139f, -0.0666981712f, 0.0339653119f, -0.140525416f, 0.0675620735f, -0.120439962f, + 0.0212921053f, 0.112628169f, 0.190132841f, -0.144298837f, -0.0116734616f, -0.159263715f, + 0.0784623772f, -0.179399416f, -0.328205913f, -0.0860405788f, -0.0919508785f, -0.0483957306f, + -0.30280292f, 0.158935979f, -0.127778456f, -0.174414024f, 0.315983474f, -0.0398868211f, + -0.11850293f, 0.0652811453f, -0.0993542522f, -0.234619945f, -0.284468323f, -0.183528736f, + 0.225024208f, -0.049902156f, -0.0650495142f, -0.0445851982f, -0.193018377f, -0.149498582f, + -0.0479458123f, -0.0666453466f, -0.0906915069f, 0.0227309261f, 0.0560047664f, -0.146587655f, + -0.172089651f, -0.0502583086f, 0.0590079799f, 0.19477278f, 0.218966454f, -0.112773709f, + -0.16504398f, -0.111266837f, 0.15254502f, -0.120294176f, 0.114031032f, -0.0505410358f, + -0.150275424f, -0.024936026f, -0.139517337f, -0.00708480086f, -0.160885945f, 0.0312023032f, + 0.27590549f, 0.0162312202f, -0.389412105f, -0.066493772f, -0.005505112f, -0.114737555f, + -0.136138007f, 0.0470205061f, +}; + +inline constexpr float kLtx2ForwardPromptAdalnAudio[] = { + -0.00688289292f, 0.0557257049f, -0.169716612f, 0.136525676f, -0.0794674978f, -0.106710114f, + -0.109623641f, 0.0357984938f, -0.0222658273f, -0.0561756417f, 0.145198658f, 0.0478727296f, + 0.0297143031f, 0.0544307008f, -0.205607757f, 0.122264318f, -0.105997331f, -0.191682264f, + -0.0355675966f, 0.0435401984f, -0.107154176f, 0.192362309f, -0.0537107848f, -0.080251269f, + -0.0339412056f, -0.0676042289f, -0.161326066f, -0.0567146167f, 0.166392133f, -0.0363745391f, + -0.00992792659f, 0.0993538871f, -0.196374968f, 0.146272391f, -0.0314919427f, -0.147439599f, + 0.00373091619f, 0.119020537f, -0.16169925f, 0.149572372f, -0.0535437092f, -0.17913793f, + -0.0729758665f, 0.0560201257f, -0.103747681f, 0.0615522824f, 0.147424534f, -0.0767256021f, + -0.0455891043f, -0.074104555f, -0.0908109695f, 0.14269349f, 0.00909528229f, 0.000857369974f, + -0.094678238f, 0.0833884105f, 0.000516908942f, -0.244370729f, 0.0668335408f, 0.0764729604f, +}; + +// --- forward case PromptAdalnMasked: rope=split float64_freqs=False masked=True audio_enabled=True dense_self_mask=False prompt_adaln=True --- +inline constexpr float kLtx2ForwardPromptAdalnMaskedVideo[] = { + -0.110144161f, -0.0195061471f, -0.0188482404f, -0.0824035481f, -0.00313981529f, -0.102331489f, + -0.140647486f, -0.111037448f, 0.238786951f, 0.019598389f, 0.0744047314f, -0.129760861f, + -0.326935619f, -0.0698587522f, -0.0180070139f, 0.0129257301f, -0.282629013f, 0.0306178499f, + -0.100754961f, 0.00948145799f, 0.175529778f, -0.135220498f, -0.206774727f, -0.182601333f, + -0.275893211f, -0.0184815805f, 0.198321715f, -0.251868665f, -0.194811657f, -0.0697973669f, + -0.114349291f, -0.089783825f, -0.0404667407f, 0.121526636f, -0.0449657179f, -0.170914561f, + -0.144608453f, -0.129623294f, -0.126025274f, -0.0928756446f, -0.0987707227f, -0.0303842705f, + 0.349234581f, -0.0163635593f, 0.0282549523f, -0.152388036f, 0.0889601111f, -0.216030046f, + -0.112322196f, 0.108959578f, 0.112902433f, -0.0188521128f, 0.0273712836f, -0.20055984f, + -0.0387175046f, -0.156802967f, 0.156722441f, 0.0369930752f, 0.239927039f, -0.124359809f, + -0.273358971f, -0.0692257583f, 0.0352873057f, -0.141093776f, 0.0673692524f, -0.120450936f, + 0.0237387791f, 0.111967109f, 0.187116802f, -0.139871866f, -0.011316698f, -0.153532162f, + 0.0786181018f, -0.177252695f, -0.331035018f, -0.0842280909f, -0.0938364938f, -0.0409911275f, + -0.301013172f, 0.167324454f, -0.124323212f, -0.174904421f, 0.316507936f, -0.0429520756f, + -0.120540164f, 0.067215845f, -0.101051688f, -0.225887731f, -0.286542743f, -0.182181612f, + 0.224588186f, -0.049729839f, -0.0685416833f, -0.0383053645f, -0.189322844f, -0.138665244f, + -0.0496713445f, -0.068905741f, -0.093425855f, 0.0234508831f, 0.0515494831f, -0.140059441f, + -0.166952908f, -0.041921705f, 0.0591960102f, 0.194034085f, 0.216313154f, -0.111525752f, + -0.165100992f, -0.10801211f, 0.152435049f, -0.115520097f, 0.110954985f, -0.0485403165f, + -0.152800724f, -0.0228928309f, -0.138780624f, -0.00368307461f, -0.157897815f, 0.0359253138f, + 0.277324349f, 0.0178819038f, -0.393882304f, -0.0665938482f, -0.00776502956f, -0.109243833f, + -0.134122461f, 0.0536325052f, +}; + +inline constexpr float kLtx2ForwardPromptAdalnMaskedAudio[] = { + -0.00623655878f, 0.0550939925f, -0.169768706f, 0.134756476f, -0.0800285041f, -0.106801011f, + -0.109707221f, 0.0349297225f, -0.0223909412f, -0.0580404438f, 0.145440862f, 0.0479248241f, + 0.0300626066f, 0.0543725975f, -0.205051288f, 0.121049382f, -0.106699832f, -0.19155553f, + -0.0357390642f, 0.043708574f, -0.106640562f, 0.191323146f, -0.0545797087f, -0.0801633745f, + -0.0338453911f, -0.067704387f, -0.161020443f, -0.0588866882f, 0.16616787f, -0.0362463593f, + -0.00972275995f, 0.0983785167f, -0.197862387f, 0.146195069f, -0.0301159211f, -0.147994176f, + 0.00437853113f, 0.118179567f, -0.162635818f, 0.14993386f, -0.0523924604f, -0.179551288f, + -0.0724292099f, 0.0543062873f, -0.103864521f, 0.0613403916f, 0.14883852f, -0.0763024017f, + -0.0452010371f, -0.0744607076f, -0.0918518007f, 0.142673746f, 0.0106131276f, 5.35193831e-05f, + -0.09456487f, 0.0829611197f, 0.000701400219f, -0.243942767f, 0.0671276823f, 0.0765738413f, +}; + +// --- the MEASURED magnitude of the prompt-AdaLN term --- +// Same shared weights, same inputs, flag ON vs OFF: +// DiT video output: max|on-off| = 0.000145674 (0.04% of max|off|) +// DiT audio output: max|on-off| = 7.36713e-05 (0.03% of max|off|) +// block 0 modulated prompt K/V: max|on-off| = 0.0309528 (5.82% of max|off|) +// timestep term vs static table: max|term| = 0.0252012 vs max|table| = 0.0487142 (51.7%) +// The two output rows are bounded by this generator's SYNTHETIC weight scale +// (0.05, param_spec above) and by a 2-block stack; they are the FLOOR the gate +// needs, not a claim about the trained checkpoint. The K/V row is where the +// term actually enters and is the number that answers 'does this matter'. } // namespace vllm_test diff --git a/tests/vllm/models/test_ltx2.cpp b/tests/vllm/models/test_ltx2.cpp index dd6a8e077..f00771f38 100644 --- a/tests/vllm/models/test_ltx2.cpp +++ b/tests/vllm/models/test_ltx2.cpp @@ -135,6 +135,16 @@ Ltx2DitParams ReducedParams(Ltx2RopeType rope_type, bool double_precision) { return p; } +// Upstream's DEFAULT arm (model.py:77, model_configurator.py:76/:138, diffusers +// transformer_ltx2.py:1185) and the one the shipped LTX-2.5 DiT carries: the +// prompt-side AdaLN MLP is built, so the cross-attention K/V modulation carries a +// timestep term on top of the static per-block table (transformer.py:441-443). +Ltx2DitParams ReducedParamsPromptAdaln(Ltx2RopeType rope_type, bool double_precision) { + Ltx2DitParams p = ReducedParams(rope_type, double_precision); + p.use_prompt_adaln_single = true; + return p; +} + // A materialized weight set: owned f32 storage plus the views the forward takes. struct WeightSet { std::map> storage; @@ -831,6 +841,137 @@ TEST_CASE("ltx2 forward: a disabled audio stream still feeds audio->video") { /*audio_enabled=*/false); } +// --------------------------------------------------------------------------- +// The prompt-side AdaLN arm — upstream's DEFAULT +// (.agents/specs/ltx25-prompt-adaln.md, issue #644) +// --------------------------------------------------------------------------- + +namespace { + +// The same forward, run with `use_prompt_adaln_single = true`. Kept separate from +// CheckForward rather than folded into it, because the two arms have DIFFERENT +// weight contracts (12 extra parameters) and sharing one helper would hide which +// contract a case ran under. +vllm::Ltx2DitOutputs RunPromptAdalnForward(const Ltx2DitParams& p, WeightSet& set, + Modalities* m, bool masked) { + BuildModalities(m, masked); + return Ltx2DitForward(Cpu(), p, set.weights, &m->video, &m->audio, vt::DType::kF32); +} + +void CheckPromptAdalnForward(bool masked, const float* want_video, const float* want_audio, + const char* label) { + INFO("prompt-AdaLN forward case: " << std::string(label)); + const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); + WeightSet set = BuildWeights(p); + Modalities m; + const vllm::Ltx2DitOutputs out = RunPromptAdalnForward(p, set, &m, masked); + const size_t vcount = static_cast(m.video.batch * m.video.tokens * p.out_channels); + const size_t acount = + static_cast(m.audio.batch * m.audio.tokens * p.audio_out_channels); + const double vdiff = MaxAbsDiff(out.video, want_video, vcount); + const double adiff = MaxAbsDiff(out.audio, want_audio, acount); + MESSAGE("max|diff| video=" << vdiff << " audio=" << adiff); + CHECK(vdiff < kRoundOff); + CHECK(adiff < kRoundOff); +} + +} // namespace + +// The 12 parameters the flag adds (6 per stream: three linears x weight+bias), +// which the shipped FP8 checkpoint carries as 18 manifest entries because each +// quantized weight also has a `weight_scale`. Order matters as much as presence: +// upstream registers `prompt_adaln_single` between `adaln_single` and `proj_out` +// inside `_init_video` (model.py:222-232), and the audio twin in `_init_audio` +// (:252-262). +TEST_CASE("ltx2 layout: the flag-ON contract matches upstream named_parameters()") { + const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); + const std::vector manifest = EnumerateLtx2DitTensors(p); + REQUIRE(static_cast(manifest.size()) == vllm_test::kLtx2PromptAdalnParamCount); + // The flag is the ONLY difference, so the count delta is exactly the module. + CHECK(vllm_test::kLtx2PromptAdalnParamCount - vllm_test::kLtx2ParamCount == 12); + size_t dim_cursor = 0; + for (size_t i = 0; i < manifest.size(); ++i) { + CAPTURE(i); + CAPTURE(manifest[i].name); + CHECK(manifest[i].name == std::string(vllm_test::kLtx2PromptAdalnParamNames[i])); + const int64_t rank = vllm_test::kLtx2PromptAdalnParamRanks[i]; + REQUIRE(static_cast(manifest[i].shape.size()) == rank); + for (int64_t d = 0; d < rank; ++d) { + CHECK(manifest[i].shape[static_cast(d)] == + vllm_test::kLtx2PromptAdalnParamDims[dim_cursor]); + ++dim_cursor; + } + } +} + +// The MLP on its own, so a failure localizes here rather than in the threading. +// The input is the modality's SIGMA scaled by timestep_scale_multiplier +// (transformer_args.py:274-277 -> :173-186) — one scalar per batch element, NOT +// the per-token `timesteps` the main AdaLN consumes. +TEST_CASE("ltx2 brick: the prompt AdaLN MLP runs on sigma, not on the per-token timesteps") { + const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); + WeightSet set = BuildWeights(p); + const int64_t b = vllm_test::kLtx2Batch; + const int64_t dim = p.inner_dim(); + const int64_t adim = p.audio_inner_dim(); + + std::vector vts(vllm_test::kLtx2PromptAdalnVideoTimesteps, + vllm_test::kLtx2PromptAdalnVideoTimesteps + b); + const vllm::Ltx2AdalnOut vout = vllm::Ltx2AdaLayerNormSingle( + Cpu(), set.weights.prompt_adaln_single, vts.data(), b, dim); + REQUIRE(vout.modulation.size() == static_cast(b * 2 * dim)); + CHECK(MaxAbsDiff(vout.modulation, vllm_test::kLtx2PromptAdalnVideoModulation, + static_cast(b * 2 * dim)) < kRoundOff); + + std::vector ats(vllm_test::kLtx2PromptAdalnAudioTimesteps, + vllm_test::kLtx2PromptAdalnAudioTimesteps + b); + const vllm::Ltx2AdalnOut aout = vllm::Ltx2AdaLayerNormSingle( + Cpu(), set.weights.audio_prompt_adaln_single, ats.data(), b, adim); + REQUIRE(aout.modulation.size() == static_cast(b * 2 * adim)); + CHECK(MaxAbsDiff(aout.modulation, vllm_test::kLtx2PromptAdalnAudioModulation, + static_cast(b * 2 * adim)) < kRoundOff); +} + +TEST_CASE("ltx2 forward: the prompt-side AdaLN arm") { + CheckPromptAdalnForward(false, vllm_test::kLtx2ForwardPromptAdalnVideo, + vllm_test::kLtx2ForwardPromptAdalnAudio, "prompt AdaLN"); +} + +TEST_CASE("ltx2 forward: the prompt-side AdaLN arm, with both masks") { + CheckPromptAdalnForward(true, vllm_test::kLtx2ForwardPromptAdalnMaskedVideo, + vllm_test::kLtx2ForwardPromptAdalnMaskedAudio, + "prompt AdaLN + masks"); +} + +// THE INSTRUMENT THAT MAKES THE ARM ABOVE MEAN SOMETHING. +// +// Both goldens come from the same deterministic weight stream, keyed by parameter +// NAME, so every weight the two arms share is bit-identical and the ONLY thing +// separating `kLtx2ForwardPromptAdaln*` from `kLtx2ForwardSplit*` is the timestep +// term. A port that accepted the flag, bound the 12 tensors and then never added +// their output would reproduce the flag-OFF numbers exactly and pass nothing here. +// +// Measured on this fixture (generator stderr, and the comment block at the end of +// ltx2_goldens.inc): the term is 51.7% the magnitude of the static per-block +// table it is added to, moves the block-0 modulated prompt K/V by 5.82%, and +// moves the DiT's own output by 1.46e-4 — 73x the kRoundOff floor. The bound +// below is set at 20x kRoundOff so it is comfortably inside the measured signal +// and comfortably outside f32 noise. +TEST_CASE("ltx2 forward: the prompt-AdaLN term is LOAD-BEARING, not decoration") { + const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); + WeightSet set = BuildWeights(p); + Modalities m; + const vllm::Ltx2DitOutputs out = RunPromptAdalnForward(p, set, &m, false); + const size_t vcount = static_cast(m.video.batch * m.video.tokens * p.out_channels); + const size_t acount = + static_cast(m.audio.batch * m.audio.tokens * p.audio_out_channels); + const double vdiff = MaxAbsDiff(out.video, vllm_test::kLtx2ForwardSplitVideo, vcount); + const double adiff = MaxAbsDiff(out.audio, vllm_test::kLtx2ForwardSplitAudio, acount); + MESSAGE("flag-ON vs flag-OFF: video=" << vdiff << " audio=" << adiff); + CHECK(vdiff > 20.0 * kRoundOff); + CHECK(adiff > 20.0 * kRoundOff); +} + TEST_CASE("ltx2 forward: a single-stream model type is REFUSED") { const Ltx2DitParams p = ReducedParams(Ltx2RopeType::kSplit, false); WeightSet set = BuildWeights(p); diff --git a/tests/vllm/models/test_ltx2_device.cpp b/tests/vllm/models/test_ltx2_device.cpp index 46bea0b78..cafef9ceb 100644 --- a/tests/vllm/models/test_ltx2_device.cpp +++ b/tests/vllm/models/test_ltx2_device.cpp @@ -304,8 +304,9 @@ struct CaseResult { CaseResult RunDeviceCase(vt::Queue& q, vt::DType stream, Ltx2RopeType rope_type, bool double_precision, bool masked, const float* want_video, const float* want_audio, bool audio_enabled = true, - bool dense_self_mask = false) { - const Ltx2DitParams p = ReducedParams(rope_type, double_precision); + bool dense_self_mask = false, bool prompt_adaln = false) { + Ltx2DitParams p = ReducedParams(rope_type, double_precision); + p.use_prompt_adaln_single = prompt_adaln; WeightSet set = BuildWeights(p); // Staged at the SAME dtype the stream computes in. Staging f32 weights under a // bf16 stream would compare a DIFFERENT MODEL, not a different dtype policy. @@ -406,6 +407,74 @@ void CheckBf16Stream(vt::Queue& q, const char* label) { CHECK(bf16.video > kDeviceRoundOff); } +// The prompt-side AdaLN arm — upstream's DEFAULT and what the shipped DiT runs +// (.agents/specs/ltx25-prompt-adaln.md, issue #644). The device path forms +// `kv_modulation = table + prompt_timestep` (transformer.py:441-443) through +// `ada_value` and then broadcasts one row per BATCH element over that element's +// prompt tokens, which is a DIFFERENT `modulate` call shape from the static +// arm — so it needs its own case rather than riding on the loop above. +void CheckPromptAdalnCases(vt::Queue& q, const char* label) { + { + INFO(label << " / prompt AdaLN, f32"); + const CaseResult r = + RunDeviceCase(q, vt::DType::kF32, Ltx2RopeType::kSplit, false, false, + vllm_test::kLtx2ForwardPromptAdalnVideo, + vllm_test::kLtx2ForwardPromptAdalnAudio, /*audio_enabled=*/true, + /*dense_self_mask=*/false, /*prompt_adaln=*/true); + MESSAGE("prompt AdaLN f32: max|diff| video=" << r.video << " audio=" << r.audio); + CHECK(r.video < kDeviceRoundOff); + CHECK(r.audio < kDeviceRoundOff); + } + { + // The masked case: the prompt mask and the prompt modulation act on the SAME + // context tensor, and a per-batch broadcast written as a per-token one would + // survive the unmasked case at batch 1. + INFO(label << " / prompt AdaLN + masks, f32"); + const CaseResult r = + RunDeviceCase(q, vt::DType::kF32, Ltx2RopeType::kSplit, false, true, + vllm_test::kLtx2ForwardPromptAdalnMaskedVideo, + vllm_test::kLtx2ForwardPromptAdalnMaskedAudio, /*audio_enabled=*/true, + /*dense_self_mask=*/false, /*prompt_adaln=*/true); + MESSAGE("prompt AdaLN masked f32: max|diff| video=" << r.video << " audio=" << r.audio); + CHECK(r.video < kDeviceRoundOff); + CHECK(r.audio < kDeviceRoundOff); + } + { + // AND IT IS LOAD-BEARING. The flag-ON and flag-OFF goldens share every common + // weight bit-for-bit, so a device path that bound the module and never added + // its output would reproduce the flag-OFF numbers exactly. + // + // The bound is `kDeviceRoundOff` itself, and that is the precise statement: + // the flag-ON run must miss the flag-OFF golden by MORE than the tolerance + // the flag-OFF case is held to, or a dropped term would pass that gate. + // Measured here: video 1.46e-4 (7.3x the bound), audio 7.37e-5 (3.7x). + INFO(label << " / prompt AdaLN is load-bearing"); + const CaseResult r = + RunDeviceCase(q, vt::DType::kF32, Ltx2RopeType::kSplit, false, false, + vllm_test::kLtx2ForwardSplitVideo, vllm_test::kLtx2ForwardSplitAudio, + /*audio_enabled=*/true, /*dense_self_mask=*/false, + /*prompt_adaln=*/true); + MESSAGE("prompt AdaLN vs flag-OFF golden: video=" << r.video << " audio=" << r.audio); + CHECK(r.video > kDeviceRoundOff); + CHECK(r.audio > kDeviceRoundOff); + } + { + // The bf16 PRODUCTION stream on the same arm: `ada_value` stores the + // table+timestep sum at the stream dtype, so this is where a bf16 store of the + // new sum is exercised at all. + INFO(label << " / prompt AdaLN, bf16"); + const CaseResult r = + RunDeviceCase(q, vt::DType::kBF16, Ltx2RopeType::kSplit, false, false, + vllm_test::kLtx2ForwardPromptAdalnVideo, + vllm_test::kLtx2ForwardPromptAdalnAudio, /*audio_enabled=*/true, + /*dense_self_mask=*/false, /*prompt_adaln=*/true); + MESSAGE("prompt AdaLN bf16: max|diff| video=" << r.video << " audio=" << r.audio); + CHECK(r.video < kBf16RoundOff); + CHECK(r.audio < kBf16RoundOff); + CHECK(r.video > kDeviceRoundOff); + } +} + vt::Backend* TryCuda() { return vt::TryGetBackend(vt::DeviceType::kCUDA); } } // namespace @@ -527,6 +596,21 @@ TEST_CASE("ltx2 device: the DEVICE-RESIDENT forward matches upstream on CUDA") { CheckAllForwardCases(q, "cuda"); } +TEST_CASE("ltx2 device: the prompt-side AdaLN arm matches upstream (CPU backend)") { + vt::Queue q{Cpu(), nullptr}; + CheckPromptAdalnCases(q, "cpu-backend"); +} + +TEST_CASE("ltx2 device: the prompt-side AdaLN arm matches upstream on CUDA") { + vt::Backend* cuda = TryCuda(); + if (cuda == nullptr) { + MESSAGE("SKIP: no CUDA backend registered"); + return; + } + vt::Queue q = cuda->CreateQueue(); + CheckPromptAdalnCases(q, "cuda"); +} + TEST_CASE("ltx2 device: the bf16 PRODUCTION stream matches upstream (CPU backend)") { vt::Queue q{Cpu(), nullptr}; CheckBf16Stream(q, "cpu-backend bf16"); diff --git a/tests/vllm/models/test_ltx2_loader.cpp b/tests/vllm/models/test_ltx2_loader.cpp index 654ecd134..e191d958b 100644 --- a/tests/vllm/models/test_ltx2_loader.cpp +++ b/tests/vllm/models/test_ltx2_loader.cpp @@ -827,8 +827,10 @@ TEST_CASE("ltx2 loader: the shipped FP8 DiT manifest is fully accounted for") { CHECK(bf16 == 2284); CHECK(f32 == 290); // the tables; the 1775 scalar scales were counted above - // The four families outside the phase-L2 contract, named so their absence - // from the port cannot be discovered later. + // The five families the file carries beyond the ORIGINAL L2 contract, named so + // their status cannot be discovered later. Two are ported (the prompt AdaLN + // pair), two are loaded elsewhere (the connectors), one is genuinely unported + // (keyframes). CHECK(families.count("prompt_adaln_single") == 1); CHECK(families.count("audio_prompt_adaln_single") == 1); CHECK(families.count("keyframes_abs_pos_embedding") == 1); @@ -886,18 +888,19 @@ TEST_CASE("ltx2 loader: the L2 contract's every name is present in the shipped D CHECK(p.cross_attention_adaln); CHECK_FALSE(p.ff_bias); CHECK(p.audio_ff_bias); - // MEASURED, and it contradicts .agents/specs/ltx-2-5.md section 1.2 and - // ltx2.h:115-117: the SHIPPED checkpoint carries prompt_adaln_single, which - // upstream builds only when use_prompt_adaln_single is TRUE (model.py:222-226). - // The prompt-K/V cache's premise does not hold for this checkpoint. + // MEASURED: the SHIPPED checkpoint carries prompt_adaln_single, which upstream + // builds only when use_prompt_adaln_single is TRUE (model.py:222-226), so the + // prompt-K/V cache's premise does not hold for this checkpoint + // (.agents/specs/ltx-2-5.md §1.2) and the module is PORTED + // (.agents/specs/ltx25-prompt-adaln.md, issue #644). CHECK(p.use_prompt_adaln_single); - // Enumerate the contract for the subset this port DOES carry, and require - // every one of its names in the file. - Ltx2DitParams contract = p; - contract.use_prompt_adaln_single = false; - const std::vector want = vllm::EnumerateLtx2DitTensors(contract); - CHECK(want.size() == 4078); + // Enumerate the contract AS THE FILE DESCRIBES IT — no flag is forced here any + // more — and require every one of its names in the file. This line used to read + // `contract.use_prompt_adaln_single = false`, which is the shape of the defect: + // the gate agreed with the port about a module they were both dropping. + const std::vector want = vllm::EnumerateLtx2DitTensors(p); + CHECK(want.size() == 4078 + 12); int64_t missing = 0; std::string first_missing; for (const Ltx2TensorSpec& spec : want) { @@ -913,15 +916,16 @@ TEST_CASE("ltx2 loader: the L2 contract's every name is present in the shipped D // ... and account for every name the file has that the contract does not, so // "the rest is fine" is a counted claim rather than a hope. // 258 = 2 connectors x (8 blocks x 16 + 1 learnable_registers) - // 12 = prompt_adaln_single + audio_prompt_adaln_single, 6 tensors each // 1 = keyframes_abs_pos_embedding + // The 12 prompt-AdaLN tensors are no longer here: they moved INTO `want` when + // the module was ported, which is the whole delta this row landed. std::set want_set; for (const Ltx2TensorSpec& spec : want) want_set.insert(spec.name); int64_t extra = 0; for (const std::string& name : present) { if (want_set.count(name) == 0) ++extra; } - CHECK(extra == 258 + 12 + 1); + CHECK(extra == 258 + 1); } // =========================================================================== @@ -1542,8 +1546,7 @@ TEST_CASE("ltx2 loader: the unported families are refused by name, not absorbed" const Ltx2DitParams p = TinyParams(); const SyntheticDit syn = BuildSyntheticDit( p, Ltx2DitQuant::kFp8, - {"prompt_adaln_single.linear.weight", "audio_prompt_adaln_single.linear.weight", - "keyframes_abs_pos_embedding", "video_embeddings_connector.learnable_registers", + {"keyframes_abs_pos_embedding", "video_embeddings_connector.learnable_registers", "audio_embeddings_connector.learnable_registers"}); const std::string path = TmpPath("unported"); WriteSafetensors(syn.entries, path); @@ -1557,12 +1560,18 @@ TEST_CASE("ltx2 loader: the unported families are refused by name, not absorbed" } const std::string what_msg = "what: " + what; INFO(what_msg); - // THREE families now, not five. `audio_prompt_adaln_single` is here because it - // goes through the identical generic FamilyOf path as its video twin and - // covering it literally is one line. - CHECK(what.find("prompt_adaln_single") != std::string::npos); - CHECK(what.find("audio_prompt_adaln_single") != std::string::npos); - CHECK(what.find("keyframes_abs_pos_embedding") != std::string::npos); + // ONE family in the LIST now, not five. `prompt_adaln_single` / + // `audio_prompt_adaln_single` left it when they were PORTED (issue #644); the + // case below proves a checkpoint carrying them needs no opt-in at all. + // + // The list is asserted as a whole rather than by substring, because the message + // deliberately goes on to NAME the families that are not in it — a + // `find(...) == npos` over the whole message would only test the prose. + const std::string head = "does NOT carry: "; + const size_t at = what.find(head); + REQUIRE(at != std::string::npos); + const std::string list = what.substr(at + head.size(), what.find('.', at) - at - head.size()); + CHECK(list == "keyframes_abs_pos_embedding"); // THE TWO CONNECTOR FAMILIES ARE NOT UNPORTED AS OF PHASE L9c. They are // outside the DiT's contract by design — upstream loads them into the text @@ -1574,17 +1583,15 @@ TEST_CASE("ltx2 loader: the unported families are refused by name, not absorbed" CHECK(what.find("video_embeddings_connector") == std::string::npos); CHECK(what.find("audio_embeddings_connector") == std::string::npos); - // The opt-in still REPORTS every one of the three; it does not make them vanish. + // The opt-in still REPORTS it; it does not make it vanish. Ltx2DitLoadOptions options; options.allow_unported_modules = true; const vllm::Ltx2DitCheckpoint ck = vllm::Ltx2LoadDitFromSafetensors(file, options); - CHECK(ck.unported.size() == 3); + CHECK(ck.unported.size() == 1); for (const std::string& family : ck.unported) { CHECK(family != "video_embeddings_connector"); CHECK(family != "audio_embeddings_connector"); } - CHECK(ck.checkpoint_params.use_prompt_adaln_single); - CHECK_FALSE(ck.params.use_prompt_adaln_single); std::remove(path.c_str()); // AND THE CONNECTOR-ONLY CHECKPOINT LOADS WITH NO OPT-IN AT ALL, which is the @@ -1604,6 +1611,131 @@ TEST_CASE("ltx2 loader: the unported families are refused by name, not absorbed" std::remove(conn_path.c_str()); } +// THE REGRESSION GATE FOR ISSUE #644 ROW 0. +// +// A checkpoint that carries `prompt_adaln_single` — which the shipped LTX-2.5 DiT +// does — used to be refused without `allow_unported_modules=1`, and setting that +// extra reached three assignments that CLEARED `use_prompt_adaln_single`. So the +// only way to load a real DiT was also the way to silently drop the timestep half +// of every cross-attention K/V modulation. +// +// Both halves are asserted here, and neither is true by construction: the load +// with no opt-in exercises the contract, and the resolved flag is read back off +// the checkpoint the loader actually bound. +TEST_CASE("ltx2 loader: a DiT carrying prompt_adaln_single loads with NO opt-in") { + Ltx2DitParams p = TinyParams(); + p.use_prompt_adaln_single = true; + const SyntheticDit syn = BuildSyntheticDit(p, Ltx2DitQuant::kFp8, {}); + const std::string path = TmpPath("prompt_adaln"); + WriteSafetensors(syn.entries, path); + const SafetensorsFile file = SafetensorsFile::Open(path); + + // No options at all: the families are in the contract, so nothing is unported. + const vllm::Ltx2DitCheckpoint ck = vllm::Ltx2LoadDitFromSafetensors(file); + CHECK(ck.unported.empty()); + CHECK(ck.checkpoint_params.use_prompt_adaln_single); + // The flag the FORWARD reads. This is the assertion the defect broke: it was + // false while `checkpoint_params` said true. + CHECK(ck.params.use_prompt_adaln_single); + // And the weights are bound, not merely enumerated — a contract that listed the + // tensors while `BindLtx2DitWeights` left the views null would render the same + // wrong picture with a different failure mode. + CHECK(ck.weights.prompt_adaln_single.linear.weight.data != nullptr); + CHECK(ck.weights.audio_prompt_adaln_single.linear.weight.data != nullptr); + CHECK(ck.weights.prompt_adaln_single.linear.weight.shape[0] == 2 * p.inner_dim()); + CHECK(ck.weights.audio_prompt_adaln_single.linear.weight.shape[0] == 2 * p.audio_inner_dim()); + // The 12 tensors, counted rather than assumed. + Ltx2DitParams off = p; + off.use_prompt_adaln_single = false; + CHECK(vllm::EnumerateLtx2DitTensors(p).size() - + vllm::EnumerateLtx2DitTensors(off).size() == + 12); + std::remove(path.c_str()); + + // AND THE OPT-IN CANNOT UNDO IT. `allow_unported_modules` is what a real render + // still passes for `keyframes_abs_pos_embedding`; it must leave a ported feature + // alone. Same bytes, opt-in set, same resolved flag. + const std::string path2 = TmpPath("prompt_adaln_optin"); + WriteSafetensors(syn.entries, path2); + const SafetensorsFile file2 = SafetensorsFile::Open(path2); + Ltx2DitLoadOptions options; + options.allow_unported_modules = true; + const vllm::Ltx2DitCheckpoint ck2 = vllm::Ltx2LoadDitFromSafetensors(file2, options); + CHECK(ck2.params.use_prompt_adaln_single); + CHECK(ck2.weights.prompt_adaln_single.linear.weight.data != nullptr); + std::remove(path2.c_str()); +} + +// The DECLARED config path. `Ltx2AdoptDeclaredDitParams` no longer forces both +// sides of its comparison to a cleared flag, so a config that disagrees with the +// file's shapes about `use_prompt_adaln_single` now produces two DIFFERENT +// contracts and is refused — which is an INPUT-driven gate, not a mutation-only +// one. Both directions are checked, because the equality is the point. +TEST_CASE("ltx2 loader: a config that disagrees about use_prompt_adaln_single is REFUSED") { + Ltx2DitParams shapes = TinyParams(); + shapes.use_prompt_adaln_single = true; + + nlohmann::json t; + t["num_attention_heads"] = shapes.num_attention_heads; + t["attention_head_dim"] = shapes.attention_head_dim; + t["in_channels"] = shapes.in_channels; + t["out_channels"] = shapes.out_channels; + t["num_layers"] = shapes.num_layers; + t["cross_attention_dim"] = shapes.cross_attention_dim; + t["audio_num_attention_heads"] = shapes.audio_num_attention_heads; + t["audio_attention_head_dim"] = shapes.audio_attention_head_dim; + t["audio_in_channels"] = shapes.audio_in_channels; + t["audio_out_channels"] = shapes.audio_out_channels; + t["audio_cross_attention_dim"] = shapes.audio_cross_attention_dim; + t["apply_gated_attention"] = true; + t["cross_attention_adaln"] = true; + t["ff_bias"] = false; + t["audio_ff_bias"] = true; + t["rope_type"] = "split"; + // The checks ParseLtx2DitParams asserts verbatim (model_configurator.py:26-44). + t["dropout"] = 0.0; + t["attention_bias"] = true; + t["num_vector_embeds"] = nullptr; + t["activation_fn"] = "gelu-approximate"; + t["num_embeds_ada_norm"] = 1000; + t["use_linear_projection"] = false; + t["only_cross_attention"] = false; + t["cross_attention_norm"] = true; + t["double_self_attention"] = false; + t["upcast_attention"] = false; + t["standardization_norm"] = "rms_norm"; + t["norm_elementwise_affine"] = false; + t["qk_norm"] = "rms_norm"; + t["positional_embedding_type"] = "rope"; + t["use_audio_video_cross_attention"] = true; + t["share_ff"] = false; + t["av_cross_ada_norm"] = true; + t["use_middle_indices_grid"] = true; + t["caption_proj_before_connector"] = true; + + nlohmann::json agreeing; + agreeing["transformer"] = t; + agreeing["transformer"]["use_prompt_adaln_single"] = true; + // Agreement is adopted, and carries the flag through. + const Ltx2DitParams adopted = + vllm::Ltx2AdoptDeclaredDitParams(agreeing, shapes, false, "the test config"); + CHECK(adopted.use_prompt_adaln_single); + + nlohmann::json disagreeing; + disagreeing["transformer"] = t; + disagreeing["transformer"]["use_prompt_adaln_single"] = false; + CHECK_THROWS(vllm::Ltx2AdoptDeclaredDitParams(disagreeing, shapes, false, "the test config")); + // The opt-in must not rescue it: clearing a PORTED flag in the config copy is + // exactly what made this comparison blind before. + CHECK_THROWS(vllm::Ltx2AdoptDeclaredDitParams(disagreeing, shapes, true, "the test config")); + + // And the other direction: shapes WITHOUT the module against a config that + // declares it. + Ltx2DitParams shapes_off = shapes; + shapes_off.use_prompt_adaln_single = false; + CHECK_THROWS(vllm::Ltx2AdoptDeclaredDitParams(agreeing, shapes_off, false, "the test config")); +} + TEST_CASE("ltx2 loader: the f32 widening is OPT-IN and bit-exact over bf16") { const Ltx2DitParams p = TinyParams(); const SyntheticDit syn = BuildSyntheticDit(p, Ltx2DitQuant::kFp8, {}); diff --git a/tests/vllm/multimodal/ltx2_video_fixture.h b/tests/vllm/multimodal/ltx2_video_fixture.h index e2191ed9b..ec3fca7af 100644 --- a/tests/vllm/multimodal/ltx2_video_fixture.h +++ b/tests/vllm/multimodal/ltx2_video_fixture.h @@ -248,7 +248,14 @@ inline vllm::Ltx2DitParams ReducedDitParams() { p.use_middle_indices_grid = true; p.apply_gated_attention = true; // every LTX-2.5 attention is gated p.cross_attention_adaln = true; // the shipped config sets it - p.use_prompt_adaln_single = false; + // TRUE, exactly as both shipped DiTs resolve it: the FP8 file carries no + // config at all and the NVFP4 file's config OMITS the key, so upstream's + // default (model.py:77, model_configurator.py:76) decides — and both files + // carry `prompt_adaln_single`'s tensors. `ReducedDitTransformerConfig` below + // likewise omits the key, so this fixture reproduces the shipped shape rather + // than a configuration nothing ships, and the whole video engine renders + // through the prompt-side AdaLN path (.agents/specs/ltx25-prompt-adaln.md). + p.use_prompt_adaln_single = true; p.ff_bias = false; // LTX-2.5 (gemma4) sets ff_bias=false p.audio_ff_bias = true; return p; diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index e8eaf1220..85b47025a 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -927,10 +927,12 @@ TEST_CASE("ltx2 video: the SHIPPED Lightricks checkpoints parse and load") { const vllm::SafetensorsFile file = vllm::SafetensorsFile::Open(path); vllm::Ltx2DitQuant quant = vllm::Ltx2DitQuant::kFp8; vllm::Ltx2DitParams from_shapes = vllm::Ltx2ParseDitParamsFromCheckpoint(file, &quant); - // The manifest parser leaves `use_prompt_adaln_single` at its default; the - // LOADER clears it for the contract (ltx2_loader.cpp), and so does the - // engine. Mirror that here so the two contracts are compared like for like. - from_shapes.use_prompt_adaln_single = false; + // MEASURED from this file's own header: it carries `prompt_adaln_single`, so + // the manifest parser resolves `use_prompt_adaln_single = TRUE` — and nothing + // clears it any more (.agents/specs/ltx25-prompt-adaln.md, issue #644). This + // line used to force it false on BOTH sides of the comparison below, which is + // what made a config/shape disagreement about it unobservable. + CHECK(from_shapes.use_prompt_adaln_single); CHECK(quant == vllm::Ltx2DitQuant::kNvfp4); CHECK(from_shapes.num_layers == 48); CHECK(from_shapes.inner_dim() == 4096); @@ -950,8 +952,12 @@ TEST_CASE("ltx2 video: the SHIPPED Lightricks checkpoints parse and load") { config["transformer"]["use_keyframes_abs_pos_embedding"] = false; nlohmann::json wrapper; wrapper["config"] = config; - vllm::Ltx2DitParams declared = vllm::ParseLtx2DitParams(wrapper); - declared.use_prompt_adaln_single = false; + const vllm::Ltx2DitParams declared = vllm::ParseLtx2DitParams(wrapper); + // The shipped config OMITS `use_prompt_adaln_single`, so it resolves to + // upstream's TRUE default (model_configurator.py:76) — which is what the + // file's own tensors say. Asserted rather than forced: the two sides of the + // contract comparison below must AGREE about it, not be made to. + CHECK(declared.use_prompt_adaln_single); CHECK(declared.double_precision_rope); // frequencies_precision float64 CHECK(declared.av_ca_timestep_scale_multiplier == 1000); CHECK(declared.apply_gated_attention); From 2990ed81b7b9075c5bace4ef0b93210f554c1b43 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 20:00:50 +0000 Subject: [PATCH 03/10] record(LTX25-PROMPT-ADALN): the prompt-AdaLN defect was understated ~138x, and the keyframes claim next door was false (#644) The port landed and reviewed PASS. What we WROTE about it did not. ## The headline numbers were fixture artifacts The spec Outcome, the PR body and test_ltx2.cpp all quoted 51.7% ("term vs table") and 5.82% (block-0 K/V) as the answer to "does this matter". Both come from the generator's synthetic init: `prompt_scale_shift_table` AND every prompt-AdaLN MLP parameter are drawn from the same `param_spec` rule at scale=0.05 (gen-ltx2-goldens.py:100-106), so the ratio between them is a property of the FIXTURE. It moves with the init scale: 0.005 gives 4.1%, 0.2 gives 1450%. The Outcome disclaimed only the two OUTPUT rows as synthetic-bounded and then called the other two the answer, though all four have identical provenance. Measured on the SHIPPED weights instead, through upstream's own AdaLayerNormSingle(dim, embedding_coefficient=2) evaluated on sigma*1000 exactly as transformer_args.py:274-278/:177 do, against all 48 prompt_scale_shift_table tensors of `ltx-2.5-22b-distilled-transformer-nvfp4.safetensors`: video audio rms|table| (48 blocks) 0.017553 0.021925 rms|term| (uniform s) 0.236446 0.347171 term/table RMS 1347% 1583% max|term|/max|table| 7119% 2817% Like for like -- 51.7% is a max-vs-max ratio, and the same ratio on the real weights is 7119% / 2817% -- the fixture UNDERSTATED the defect by 138x (video) and 54x (audio). On the shipped model the timestep term DOMINATES the static table; the table is the perturbation. Pre-row renders applied `context*(1 + ~0.018 rms) + ~0.018` where upstream applies `context*(1 + ~0.32 rms) + ~0.10`; on a unit-rms context the modulated context moves +8.8% (video) / +31.3% (audio). Both dispositions the finding offered are taken: the four fixture rows are now labelled GATE-FLOOR numbers from synthetic weights in the spec, the goldens' comment block and test_ltx2.cpp, AND the real-checkpoint measurement is stated as the answer. `scripts/measure-ltx2-prompt-adaln.py` is committed so the number is re-runnable rather than transcribed; it asserts `ltx_core.__file__` under the named checkout before reading anything. ## A false claim about a checkpoint, the same class this row just fixed ltx2.h:47-49 -- a line surviving inside the paragraph this row rewrote -- said "LTX-2.5's checkpoint does not carry the parameter" about keyframes_abs_pos_embedding. Both shipped DiTs contradict it, each on a different half, read off their headers and run through upstream's own configurator: FP8 (vonkaiser): CARRIES keyframes_abs_pos_embedding F8_E4M3 [1,4096] + scale, and declares NO __metadata__ at all -- so LTXModelConfigurator.from_metadata cannot configure it: KeyError: 'caption_channels', raised by _build_caption_projections on the empty dict, BEFORE the flag at model_configurator.py:82 is ever read. NVFP4 (1st-party): DECLARES the flag true and does NOT carry the tensor, so the parameter stays torch.zeros(1, inner_dim) (model.py:217-219) through load_state_dict(..., strict=False) (single_gpu_model_builder.py:98) -- a genuine no-op there. It is also not keyframe-only: transformer_args.py:269 applies it on every prepare whose keyframes_mask is set, and tools.py:186-196 sets that mask unconditionally. REFUSAL KEYING: unchanged, deliberately. It fires on the TENSORS the file carries. Keying it on the resolved flag would, on the FP8 DiT, read a DEFAULT rather than the file -- that file declares nothing -- and load it while silently discarding a trained [1,4096] parameter. No behaviour changed, so no new gate is owed; three refusal MESSAGES changed, because they asserted an implication that is false in both directions. ## Records the lifecycle change did not move model-matrix.md's inventory row still listed "timestep-INDEPENDENT prompt K/V (cacheable once per request)" -- precisely what this row disproved. docs/STATUS.md's LTX-2.5 row did not mention the prompt-side AdaLN at all; FEATURES.md and USAGE.md described the keyframes opt-in as if only one file were involved. ## Two things recorded honestly rather than fixed - test_ltx2_video's shipped-checkpoint case is env-gated on LTX2_CHECKPOINT_ROOT and SKIPS by default, so the row's "unchanged 30/502" meant the whole real-header case did not run. With the variable set the same binary measures 30/8734 -- verified here, exit 0 both ways. Noted in the spec and at the skip site. - Upstream loads with strict=False, so a config declaring the prompt-AdaLN flag false over a file that carries the module would silently run flag-OFF upstream where we now refuse. Ours is better; it is a DOCUMENTED DIVERGENCE, not a mirror, and it is named as one. ## Gate BUILD_EXIT=0; build logs grepped for `No space left|BFD assertion` (0 hits); df -h / 92% used, 37G free. ctest -N = 423; full ctest -j8 422/423 with test_serve_low_tools starved under -j (known parallel flake), 1/1 PASS serially, exit 0; 2 skipped as on the baseline. Suite counts unchanged from the row: test_ltx2 35/2435, test_ltx2_loader 26/4826, test_ltx2_device 15/523, test_ltx2_video 30/502 (skipped default) and 30/8734 with the checkpoint root. ltx2_goldens.inc REGENERATED from gen-ltx2-goldens.py against ltx_core fd4ded7f: every golden VALUE byte-identical, the diff is the comment block alone. An earlier full run was VOIDED rather than reported: another session's disk-pressure cleanup deleted build/ while ctest was at 421/423 and the last two tests recorded "Not Run -- Failed to change working directory". A run whose tree vanished under it is not a result; it was rebuilt and re-run from scratch. One finding detail did not hold on inspection: it cited `.agents/model-matrix.md:98` as already saying the two DiTs "differ in a TRAINED keyframes_abs_pos_embedding". That file contains no occurrence of "keyframes" at all on this branch; the corroborating anchors are `.agents/specs/ltx-2-5.md` section 3.1 and `tests/vllm/multimodal/test_ltx2_video.cpp:913-914`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/model-matrix.md | 2 +- .agents/specs/ltx25-prompt-adaln.md | 187 ++++++++++++++++-- docs/FEATURES.md | 2 +- docs/STATUS.md | 2 +- docs/USAGE.md | 17 +- include/vllm/model_executor/models/ltx2.h | 27 ++- .../vllm/model_executor/models/ltx2_loader.h | 6 +- scripts/gen-ltx2-goldens.py | 25 ++- scripts/measure-ltx2-prompt-adaln.py | 164 +++++++++++++++ src/vllm/model_executor/models/ltx2.cpp | 4 +- .../model_executor/models/ltx2_loader.cpp | 7 +- tests/vllm/models/ltx2_goldens.inc | 15 +- tests/vllm/models/test_ltx2.cpp | 21 +- tests/vllm/multimodal/test_ltx2_video.cpp | 5 + 14 files changed, 435 insertions(+), 49 deletions(-) create mode 100755 scripts/measure-ltx2-prompt-adaln.py diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index e63964f10..ec14bf4bd 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -429,7 +429,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-higgs-audio-v3-higgs-audio-v3-talker-for-conditional-generation` | `HiggsMultimodalQwen3ForConditionalGeneration`, `HiggsAudioV3TalkerForConditionalGeneration` | **OUT-OF-REPO** — not in `555967922`, and absent from `registry.py` on vLLM `main`; registered by `vllm-project/vllm-omni` at `vllm_omni/model_executor/models/registry.py:213-217` (and the explicit `HiggsAudioV3TalkerForConditionalGeneration` spelling at `:218`) @ `bbe6ccc512a404a2df8c977ea29003002f2683e8` → module `vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_talker.py`, class `HiggsAudioV3TalkerForConditionalGeneration` (separate module and class fields rather than the compact colon-pair form, per the out-of-repo convention), with code2wav at `registry.py:223-227` → module `higgs_audio_v3/higgs_audio_v3_code2wav.py`, class `HiggsAudioV3Code2WavForConditionalGeneration`. Documented at `vllm-omni` `docs/models/supported_models.md:81`. Official recipe: `bosonai/higgs-audio-v3-tts-4b` (published at `recipes.vllm.ai`), whose `config.json` declares `architectures: ["HiggsMultimodalQwen3ForConditionalGeneration"]`. The distinct v2 lineage (`HiggsAudioV2ForConditionalGeneration`, `higgs_audio_v2/`) is a DIFFERENT target and has no recipe in the audited set, so it gets no row here | conditional generation / text+audio in, AUDIO OUT (TTS) | MM processor; audio detokenizer / code2wav; audio tokenizer; Qwen3-derived multimodal decoder; streaming audio output | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-voxtral-realtime-voxtral-realtime-for-conditional-generation` | `VoxtralRealtimeForConditionalGeneration` (target-pending) | **TARGET-PENDING — the architecture string is registered NOWHERE we can find.** Recipe `mistralai/Voxtral-Mini-4B-Realtime-2602` (one of the 157 recipes at `vllm-project/recipes` `86c7777a`) declares `architectures: ["VoxtralRealtimeForConditionalGeneration"]` with an inner `audio_config.model_type: "voxtral_realtime_encoder"`, but that exact string appears in none of: (a) the pinned registry `555967922`; (b) any of the ten dictionaries of `vllm/model_executor/models/registry.py` on vLLM `main`; (c) `_OMNI_MODELS` in `vllm_omni/model_executor/models/registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8`; (d) `vllm-omni` `docs/models/supported_models.md`. **Related but NOT equal, recorded so nobody re-derives it:** `VoxtralRealtimeGeneration` — a DIFFERENT string — is registered at the pin (`registry.py:585`, `main` `:603`) and already has its own row here, `MODEL-MM-voxtral-realtime-voxtral-realtime-generation`; `VoxtralForConditionalGeneration` (pin `registry.py:584`) is the audio→TEXT model; and `vllm-omni` `registry.py:305-309` registers `VoxtralTTSForConditionalGeneration` for the SEPARATE `mistralai/Voxtral-4B-TTS-2603` recipe. The most likely reading is that this checkpoint's `config.json` names a renamed or not-yet-landed spelling of the existing `VoxtralRealtimeGeneration` target, but that is a hypothesis: no anchor is asserted until one of those is shown to be what actually resolves. Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / audio in, AUDIO OUT (realtime speech-to-speech) — modality inferred from the recipe and config, NOT from an upstream implementation | unresolved until the target is located | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-bailing-mm-native-bailing-mm-native-for-conditional-generation` | `BailingMMNativeForConditionalGeneration` (target-pending) | **TARGET-PENDING — the architecture string is registered NOWHERE we can find.** Recipe `inclusionAI/Ming-omni-tts-0.5B` (one of the 157 recipes at `vllm-project/recipes` `86c7777a`) declares `architectures: ["BailingMMNativeForConditionalGeneration"]`, but that exact string appears in none of: (a) the pinned registry `555967922`; (b) any dictionary of `vllm/model_executor/models/registry.py` on vLLM `main`; (c) `_OMNI_MODELS` in `vllm_omni/model_executor/models/registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8`; (d) `vllm-omni` `docs/models/supported_models.md`. **Related but NOT equal:** `vllm-omni` `docs/models/supported_models.md:75` lists that SAME checkpoint under a different architecture name, `MingTTSForConditionalGeneration` (`registry.py:250-254` → module `ming_tts/ming_tts.py`, class `MingTTSForConditionalGeneration`); and `vllm-omni` `registry.py:417-421` carries a near-miss alias `BailingMM2NativeForConditionalGeneration` — note the `2` — commented "HF repo currently ships this architecture name in config.json", pointing at module `ming_flash_omni/ming_flash_omni.py`, class `MingFlashOmniForConditionalGeneration`, for the DIFFERENT Ming-flash-omni-2.0 checkpoint. So the shape of the gap is a missing config-string alias upstream, but that is an observation, not an anchor, and none is asserted here. Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / text+image+audio in, AUDIO OUT (TTS) — modality from the recipe and config, NOT from a resolved upstream target | unresolved until the target is located | ☐ required | `INVENTORIED` | none | unassigned | -| `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` | `LTX2VideoTransformer3DModel` | **BEYOND-PIN AND OUT-OF-REPO** — not in `555967922` (LTX-2.5 released 2026-08) and not in the vLLM repository at all. Architecture reference is Lightricks' own `LTX-2` (`packages/ltx-core/src/ltx_core/`: `model/transformer/{transformer,model,attention,rope,model_configurator}.py`, `model/audio_vae/`, `components/`, `guidance/`). vLLM-Omni carries an `ltx2` module but its recipes stop at 2.3 (`vllm_omni/diffusion/models/ltx2/ltx2_recipes.py:162-166`); 2.5 is upstream-OPEN at [vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066) and [#4985](https://github.com/vllm-project/vllm-omni/issues/4985). Serving oracle reaches 2.5 through the GENERIC `DiffusersAdapterPipeline` (`pipeline_diffusers_adapter.py:116`) against `Lightricks/LTX-2.5-Diffusers`. | diffusion generation / video + audio (text/image/video in, video+audio out) | flow-matching denoise loop; TWO coupled modality streams with audio<->video cross-attention; per-head gated attention; AdaLN-single with 9-vector per-block tables; timestep-INDEPENDENT prompt K/V (cacheable once per request); split/interleaved 3D RoPE; gelu-approximate FFN; Gemma-4 12B text encoder with dual caption projections (4096 video / 2048 audio); Conv video VAE + DiffVAE; audio VAE + vocoder; latent spatial/temporal x2 upsamplers; duration head; distilled two-stage recipe | [ltx-2.5](specs/ltx-2-5.md) | `SPIKE` | **L0 LANDED 2026-08-11 (#435).** Spec committed BEFORE implementation per AGENTS.md. Geometry measured from the ungated `vonkaiser/LTX-2.5-FP8-NVFP4` FP8 checkpoint header by range request — the same no-payload technique used for H3's manifests: 21.00B params, 48 blocks, 1775 F8_E4M3 tensors each with a per-tensor F32 `weight_scale`, biases/norms BF16. Cross-modal projection shapes are ASYMMETRIC and are a named test trap: `audio_to_video_attn.to_q` is [2048, 4096] while `to_k`/`to_v` are [2048, 2048] and `to_out` is [4096, 2048], so a transposition still type-checks against a square assumption. **CHECKPOINT ACCESS (verified against the HF API 2026-08-11):** `Lightricks/LTX-2.5` is `gated: auto` (accepting the license opens it; holds the first-party NVFP4 DiT at 18.72 GB), `Lightricks/LTX-2.5-Diffusers` is RESTRICTED behind manual approval (needed for the binding oracle), and `vonkaiser/LTX-2.5-FP8-NVFP4` is UNGATED and unblocks L1-L2 today. All artifacts land under `$CHECKPOINT_ROOT` (`/mnt/nas_share/checkpoints`) so dgx.casa and the cluster nodes share one copy. **PHASES (one PR, developer-directed):** L1 `vllm::multimodal::VideoEngine` interface + checkpoint-detected registry with MiniMax-H3 moved behind it UNCHANGED (gated on frames+WAV byte-identical to the pre-refactor fold fixture) and ABI **v13 by ADDING fields only**; L2 DiT forward; L3 Gemma-4 TE reusing `gemma4.cpp`; L4 Conv video VAE + audio VAE + vocoder; L5 pipeline/recipes/upsampler/duration head; L6 NVFP4 arms + GB10 load-time residency; L7 e2e on dgx.casa under `flock $HOME/gpu.lock`. **OPEN, same as H3:** there is no vllm-omni parity PIN — `.agents/upstream-sync.md` covers the vLLM repo only, so each golden records the vllm-omni SHA inline. | `CLAIM-LTX25-L0-L7` | +| `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` | `LTX2VideoTransformer3DModel` | **BEYOND-PIN AND OUT-OF-REPO** — not in `555967922` (LTX-2.5 released 2026-08) and not in the vLLM repository at all. Architecture reference is Lightricks' own `LTX-2` (`packages/ltx-core/src/ltx_core/`: `model/transformer/{transformer,model,attention,rope,model_configurator}.py`, `model/audio_vae/`, `components/`, `guidance/`). vLLM-Omni carries an `ltx2` module but its recipes stop at 2.3 (`vllm_omni/diffusion/models/ltx2/ltx2_recipes.py:162-166`); 2.5 is upstream-OPEN at [vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066) and [#4985](https://github.com/vllm-project/vllm-omni/issues/4985). Serving oracle reaches 2.5 through the GENERIC `DiffusersAdapterPipeline` (`pipeline_diffusers_adapter.py:116`) against `Lightricks/LTX-2.5-Diffusers`. | diffusion generation / video + audio (text/image/video in, video+audio out) | flow-matching denoise loop; TWO coupled modality streams with audio<->video cross-attention; per-head gated attention; AdaLN-single with 9-vector per-block tables; timestep-CONDITIONED prompt K/V via `prompt_adaln_single` (model.py:223-227, transformer.py:441-446), so the K/V cache is inapplicable on the shipped DiT -- the row's first claim of cacheability was RETRACTED 2026-08-12 and the module PORTED 2026-08-13 (#644, [spec](specs/ltx25-prompt-adaln.md)); on the real weights the timestep term DOMINATES the static table, RMS ratio 1347% video / 1583% audio; split/interleaved 3D RoPE; gelu-approximate FFN; Gemma-4 12B text encoder with dual caption projections (4096 video / 2048 audio); Conv video VAE + DiffVAE; audio VAE + vocoder; latent spatial/temporal x2 upsamplers; duration head; distilled two-stage recipe | [ltx-2.5](specs/ltx-2-5.md) | `SPIKE` | **L0 LANDED 2026-08-11 (#435).** Spec committed BEFORE implementation per AGENTS.md. Geometry measured from the ungated `vonkaiser/LTX-2.5-FP8-NVFP4` FP8 checkpoint header by range request — the same no-payload technique used for H3's manifests: 21.00B params, 48 blocks, 1775 F8_E4M3 tensors each with a per-tensor F32 `weight_scale`, biases/norms BF16. Cross-modal projection shapes are ASYMMETRIC and are a named test trap: `audio_to_video_attn.to_q` is [2048, 4096] while `to_k`/`to_v` are [2048, 2048] and `to_out` is [4096, 2048], so a transposition still type-checks against a square assumption. **CHECKPOINT ACCESS (verified against the HF API 2026-08-11):** `Lightricks/LTX-2.5` is `gated: auto` (accepting the license opens it; holds the first-party NVFP4 DiT at 18.72 GB), `Lightricks/LTX-2.5-Diffusers` is RESTRICTED behind manual approval (needed for the binding oracle), and `vonkaiser/LTX-2.5-FP8-NVFP4` is UNGATED and unblocks L1-L2 today. All artifacts land under `$CHECKPOINT_ROOT` (`/mnt/nas_share/checkpoints`) so dgx.casa and the cluster nodes share one copy. **PHASES (one PR, developer-directed):** L1 `vllm::multimodal::VideoEngine` interface + checkpoint-detected registry with MiniMax-H3 moved behind it UNCHANGED (gated on frames+WAV byte-identical to the pre-refactor fold fixture) and ABI **v13 by ADDING fields only**; L2 DiT forward; L3 Gemma-4 TE reusing `gemma4.cpp`; L4 Conv video VAE + audio VAE + vocoder; L5 pipeline/recipes/upsampler/duration head; L6 NVFP4 arms + GB10 load-time residency; L7 e2e on dgx.casa under `flock $HOME/gpu.lock`. **OPEN, same as H3:** there is no vllm-omni parity PIN — `.agents/upstream-sync.md` covers the vLLM repo only, so each golden records the vllm-omni SHA inline. | `CLAIM-LTX25-L0-L7` | | `MODEL-MM-moss-transcribe-diarize-moss-transcribe-diarize-for-conditional-generation` | `MossTranscribeDiarizeForConditionalGeneration` (v0.25.0 target-pending) | v0.25.0 target `registry.py:450-453`; `vllm/model_executor/models/moss_transcribe_diarize.py::MossTranscribeDiarizeForConditionalGeneration` @ `702f481` | conditional generation / audio | MM processor; Whisper encoder; VQ adaptor; Qwen3 decoder; speech-to-text/diarization frontend | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-lightonocr-light-on-ocrfor-conditional-generation` | `LightOnOCRForConditionalGeneration` | `registry.py:450-453`; `vllm/model_executor/models/lightonocr.py::LightOnOCRForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-lfm2-vl-lfm2-vlfor-conditional-generation` | `Lfm2VlForConditionalGeneration` | `registry.py:454`; `vllm/model_executor/models/lfm2_vl.py::Lfm2VLForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; Mamba/SSM state; vision encoder; video path | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md index a24f127ce..724566583 100644 --- a/.agents/specs/ltx25-prompt-adaln.md +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -228,10 +228,18 @@ byte cost. ## 6. Measured magnitude -Recorded in §Outcome: the relative change in the modulated prompt context and in -the DiT's outputs, flag ON vs OFF, at the reduced dimensions. This is the answer -to "does this matter"; a number below round-off would mean the term is inert and -the whole row is decoration. +Recorded in §Outcome, on two fixtures that answer two different questions. + +The reduced-dimension generator gives the relative change in the modulated prompt +context and in the DiT's outputs, flag ON vs OFF. That is a GATE FLOOR: a number +below round-off would mean the term is inert and the mutation in §5.3 could not +bite. It is **not** the answer to "does this matter", because both the static +table and the prompt-AdaLN MLP are drawn from the same synthetic init scale, so +every ratio it produces is a property of the fixture. + +"Does this matter" is answered on the SHIPPED checkpoint's own weights, run +through upstream's `AdaLayerNormSingle`. That measurement is required before the +row's Outcome may state a magnitude. ## 7. Risks @@ -256,7 +264,7 @@ the whole row is decoration. ## Outcome -### What was measured +### What was measured — (a) the gate floor, on SYNTHETIC weights The generator emits these into `tests/vllm/models/ltx2_goldens.inc` and prints them on stderr, from the SAME shared weight stream on both arms (keyed by @@ -265,21 +273,71 @@ term and nothing else): | Quantity | Flag ON vs OFF | |---|---| -| **timestep term vs the static table it is added to** | `max\|term\|` 0.0252 vs `max\|table\|` 0.0487 — **51.7%** | -| **block-0 modulated prompt K/V** | `max\|on-off\|` 0.0310 — **5.82%** of `max\|off\|` | +| timestep term vs the static table it is added to | `max\|term\|` 0.0252 vs `max\|table\|` 0.0487 — 51.7% | +| block-0 modulated prompt K/V | `max\|on-off\|` 0.0310 — 5.82% of `max\|off\|` | | DiT video output (2 blocks) | 1.4567e-4 — 0.04% of `max\|off\|`, **73x** the gate's 2e-6 floor | | DiT audio output (2 blocks) | 7.367e-5 — 0.03%, **37x** the floor | -**The answer to "does this matter" is the first two rows.** Roughly half the -magnitude of the prompt K/V modulation is the timestep-conditioned term, and -including it moves the modulated prompt context by ~6%. Every render before this -row discarded that. - -The two output rows are the GATE's floor, not a claim about the trained -checkpoint: they are bounded by the generator's synthetic weight scale (0.05) and -by a 2-block stack rather than 48. They are reported because a mutation must be -shown to move something, and 73x/37x above round-off is what makes the mutation -below meaningful. +**ALL FOUR ROWS ARE GATE-FLOOR NUMBERS, AND NONE OF THEM ANSWERS "DOES THIS +MATTER".** Corrected 2026-08-13 (issue #644) — this section originally billed the +first two as the answer and disclaimed only the last two as synthetic-bounded. +They have identical provenance: `prompt_scale_shift_table` and every +prompt-AdaLN MLP parameter are drawn from the same `param_spec` rule at +`scale=0.05` (`scripts/gen-ltx2-goldens.py:100-106`), so the ratio between them +is a property of the FIXTURE, not of the conditioning. Vary only the MLP init and +it moves with it: 0.005 → 4.1% / 0.49%, 0.05 (committed) → 51.7% / 5.82%, +0.2 → 1450% / 142%. What these rows are FOR is the mutation below: 73x and 37x +above round-off is what makes a zeroed term detectable. + +### What was measured — (b) the SHIPPED checkpoint, which is the answer + +Measured 2026-08-13 by loading the real tensors into upstream's own +`AdaLayerNormSingle(inner_dim, embedding_coefficient=2)` (`adaln.py:19-45`, built +by `model.py:223-227` / `:253-257`) and evaluating it on `sigma * +timestep_scale_multiplier` — the file's own config gives 1000 — exactly as +`transformer_args.py:274-278` and `:177` do, then comparing against all 48 +`prompt_scale_shift_table` / `audio_prompt_scale_shift_table` tensors it is +summed with at `transformer.py:441-443`. + +- File: `/mnt/nas_share/checkpoints/ltx-2.5/lightricks-ltx-2.5/diffusion_models/ltx-2.5-22b-distilled-transformer-nvfp4.safetensors` + (7876 tensors, 1,179,408-byte header; the prompt-AdaLN tensors are BF16 there, + so they are read directly with no dequantisation step of ours in the path). +- Upstream: `ltx_core` at `fd4ded7f`, imported BY PATH from + `/home/mudler/_git/LTX-2` with `ltx_core.__file__` asserted under that checkout. +- Sigmas: a uniform grid over the whole range, `linspace(0, 1, 101)`, and + separately the sampler the file's own scheduler config names — + `LinearQuadraticScheduler().execute(8)` (`components/schedulers.py:60-88`). + +| | video (dim 4096) | audio (dim 2048) | +|---|---|---| +| `rms\|table\|`, 48 blocks | 0.017553 | 0.021925 | +| `rms\|term\|`, uniform σ | 0.236446 | 0.347171 | +| **term/table, RMS** | **1347%** | **1583%** | +| `max\|term\|` / `max\|table\|` | **7119%** | **2817%** | +| term/table RMS, LinearQuadratic 8-step | 1275% | 1492% | + +Split by row, on the uniform grid: shift 605% (video) / 461% (audio), scale 1732% +/ 2044%. The scale row is where it concentrates, which is the row that multiplies. + +**On the shipped model the timestep term DOMINATES the static table; the table is +the perturbation.** What the pre-row renders applied was +`context * (1 + ~0.018 rms) + ~0.018` where upstream applies +`context * (1 + ~0.32 rms) + ~0.10`. Stated as the quantity actually consumed at +`transformer.py:446`, for a context of unit rms: the modulated context has rms +1.0035 static-only against 1.0915 upstream (video, **+8.8%**) and 1.0033 against +1.3170 (audio, **+31.3%**). + +**The synthetic fixture UNDERSTATES the real defect by two orders of magnitude.** +Comparing like with like — the fixture's 51.7% is a `max\|term\|` / `max\|table\|` +ratio, and that same ratio on the shipped weights is 7119% (video) and 2817% +(audio): **138x and 54x** larger. Recorded because the original Outcome quoted +"roughly half the magnitude" from the fixture as if it described the checkpoint. + +Reproduce with `scripts/measure-ltx2-prompt-adaln.py --ltx2 +--checkpoint `, committed by this repair so the number is +re-runnable rather than transcribed. It asserts `ltx_core.__file__` under the +named checkout before it reads anything, and nothing of ours is in its numeric +path — the only vllm.cpp input is which tensors to read. ### The mutations @@ -316,7 +374,16 @@ re-checking its md5 (`03324d42…`, identical before and after): | `test_ltx2` | 30 cases / 1627 assertions | 35 / 2435 | +5 cases, +808 assertions | | `test_ltx2_loader` | 24 / 4817 | 26 / 4826 | +2 cases, +9 (new cases minus the assertions the retired unported-family claims took with them) | | `test_ltx2_device` | 13 / 498 | 15 / 523 | +2 cases, +25 assertions | -| `test_ltx2_video` | 30 / 502 | 30 / 502 | unchanged — the fixture now carries the module, and no assertion counted it | +| `test_ltx2_video` | 30 / 502 | 30 / 502 | unchanged — see below; 502 is the SKIPPED default | + +**What `30 / 502` does and does not say (corrected 2026-08-13).** The +shipped-checkpoint case is env-gated: with `LTX2_CHECKPOINT_ROOT` unset it prints +`SKIPPED` and returns at `test_ltx2_video.cpp:917-921`, so `30 / 502` means the +whole real-header case DID NOT RUN — not "it ran and no assertion counted the +module". With the variable pointing at the Lightricks tree the same binary +measures **30 cases / 8734 assertions**, both before and after this repair +(re-measured on this branch, exit 0 in both configurations). Any future quote of +this suite's count owes the configuration alongside it. The `test_ltx2_video` fixture had to move: it declared a config that omits `use_prompt_adaln_single` (mirroring the shipped NVFP4 DiT) while its SHAPES said @@ -356,6 +423,90 @@ off: the loader asserts the flag against the file instead of clearing it, and `Ltx2AdoptDeclaredDitParams` clears exactly one flag, for a module nothing applies. +### The keyframes claim next door, corrected 2026-08-13 + +`ltx2.h` carried, in the same paragraph this row rewrote, *"LTX-2.5's checkpoint +does not carry the parameter"* about `keyframes_abs_pos_embedding`. It is FALSE — +the same class of claim as the `use_prompt_adaln_single=false` assertion this row +exists to remove — and the tree already contradicted it twice +(`.agents/model-matrix.md`, `tests/vllm/multimodal/test_ltx2_video.cpp:913-914`). +Read straight off both files' headers, and run through upstream's own loader and +configurator: + +| | FP8 (`vonkaiser`) | NVFP4 (first-party) | +|---|---|---| +| carries `keyframes_abs_pos_embedding` | YES — `F8_E4M3 [1, 4096]` + F32 scale | NO | +| declares the flag in `__metadata__` | **no `__metadata__` AT ALL** | `true` | +| `LTXModelConfigurator.from_metadata` | **RAISES** `KeyError: 'caption_channels'` | builds it, `[1, 4096]` | + +So the two files each contradict one half of the retired claim, and neither +supports it. Two corrections to the reasoning that came with the finding, both +measured rather than read: + +- The FP8 file does not "resolve the flag `False` at `model_configurator.py:82`". + Upstream never reaches line 82 on it: `_build_caption_projections` indexes + `caption_channels` on the empty config first and raises. That file ships no + config, so what its flag resolves to is decided entirely out of band — and the + tensor it carries is trained (`.agents/specs/ltx-2-5.md` §3.1 reads its bytes). +- On the NVFP4 file the flag IS on and the module IS built, but the tensor is + absent, so it keeps `torch.zeros(1, inner_dim)` (`model.py:217-219`) through + `load_state_dict(..., strict=False)` + (`loader/single_gpu_model_builder.py:98`) — a genuine no-op there. + +It is also not a keyframe-only feature: `transformer_args.py:269` applies it on +every `prepare` whose `keyframes_mask` is set, and `tools.py:186-196` sets that +mask unconditionally on the target's first latent frame. (Diffusers' own pipeline +does not consume it — `.agents/specs/ltx-2-5.md` §3.1 records that — but `ltx_core` +is what this campaign ports, and `ltx_core` does.) + +**The refusal keying does NOT change, and that is the decision, not an omission.** +`ltx2_loader.cpp`'s `RefuseUnported` fires on the TENSORS the file carries. Keying +it on the resolved flag instead would, on the FP8 DiT, read a DEFAULT rather than +the file — because that file declares nothing — and would therefore load it +silently while discarding a trained `[1, 4096]` parameter. Tensor presence is the +only signal that file actually carries, and refusing loudly with an opt-in is +strictly safer than resolving quietly. No behaviour changed, so no new gate is +owed; the refusal MESSAGE changed, because it asserted the implication that is +false in both directions. + +### The claims repair's own gate (2026-08-13) + +Nothing executable changed except three refusal MESSAGES, so the numbers are +expected to be identical and the point is that they are: + +- `BUILD_EXIT=0`; build logs grepped for `No space left|BFD assertion` — 0 hits; + `df -h /` 92% used, 37G free at the end. +- `ctest -N` = **423**. Full `ctest -j8` = 422/423 with `test_serve_low_tools` + starved under `-j` (a known parallel flake); serially **1/1 PASS, exit 0**. + 2 skipped (`test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`) as + on the baseline. +- Suite counts, unchanged from the row above: `test_ltx2` 35/2435, + `test_ltx2_loader` 26/4826, `test_ltx2_device` 15/523, `test_ltx2_video` + 30/502 skipped-default and **30/8734** with `LTX2_CHECKPOINT_ROOT` set. Exit 0 + on all five runs. +- `tests/vllm/models/ltx2_goldens.inc` REGENERATED from + `scripts/gen-ltx2-goldens.py` against `ltx_core` `fd4ded7f`: every golden VALUE + byte-identical, the diff is the comment block alone. That re-proves provenance + as well as the wording. + +An earlier full run was voided rather than reported: another session's +disk-pressure cleanup deleted `build/` while ctest was at 421/423, and the last +two tests recorded `Not Run — Failed to change working directory`. A run whose +tree vanished under it is not a result; it was rebuilt and re-run from scratch. + +### Two divergences from upstream, recorded rather than fixed + +- **We are stricter than upstream about a config that disagrees with its file.** + Upstream loads with `load_state_dict(..., strict=False)` + (`loader/single_gpu_model_builder.py:98`), so a config declaring + `use_prompt_adaln_single=false` over a file that carries the module would build + no module, drop 18 tensors on the floor and run flag-OFF without a word. §3.2's + equality check refuses that. Ours is better; it is still a DIVERGENCE, not a + mirror, and it is named here so it is not later mistaken for ported behaviour. +- **We refuse `keyframes_abs_pos_embedding` by tensor presence** where upstream + would take an out-of-band config's word for it (above). Same shape of + divergence, same reason it stands. + ## Now `DONE` — landed on `row/LTX25-PROMPT-ADALN`. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 3c7ba3e68..cd6498160 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -161,7 +161,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder FORWARD 15.90x of vLLM's whole TTFT (pin 46.02 ms), or 2.89x with opt-in `VT_WHISPER_ENC_FA2=1` (costs 3 near-tie divergences vs 0). Not a TTFT ratio. Pending | | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending | | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams | -| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt-side AdaLN ported, host+device. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. A prompted render is OWED | Family `ltx-2.5`, `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8/torchao/1st-party NVFP4 load; `keyframes_abs_pos_embedding` alone needs `allow_unported`. DiffVAE, LoRA, image cond refused. Speed PENDING | +| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt-side AdaLN ported, host+device. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. A prompted render is OWED | Family `ltx-2.5`, `ltx2-gen`. ~29 GB NVFP4, ~44 GB FP8, +~24 GB tower. FP8/torchao/NVFP4 load; `keyframes_abs_pos_embedding` needs `allow_unported` on BOTH DiTs. DiffVAE/LoRA/image cond refused. Speed PENDING | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending | diff --git a/docs/STATUS.md b/docs/STATUS.md index 2acbe72f1..70b830b60 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -139,7 +139,7 @@ token-for-token correctness against the pinned oracle. | Laguna-S-2.1 MoE (`LagunaForCausalLM`, 118B/8B) | **BINDING 2026-08-04: was 87% of vLLM (37.55 vs 43.10, same-tool nsys)**; root cause was bf16 projections on UNIFIED/ATS host memory, and device-resident staging (byte-exact) gives 44.6, parity+ vs 43.1, default-ON | 48 layers (12 global + 36 SWA-512), 256 routed top-10 + 1 shared expert, per-head softplus attn out-gate, sigmoid `noaux_tc` router, dual per-layer RoPE, GQA 8 KV / 128 head-dim, 1M ctx. History: benchmark-record | | InternLM2 dense (fused-`wqkv` interleaved split) | Correctness-complete, speed-pending | Token-exact 16/16 (internlm2-chat-1_8b): 12/16 strict + 4/16 bf16 near-tie (max gap 0.0 nats), 0 divergent; first InternLM model; ZERO new compute kernel (reuses the Llama dense forward; the only delta is a loader-side de-interleave of the fused `wqkv`, which packs q/k/v interleaved by KV-group) | | MiniMax-H3 (`MiniMaxH3DiTModel`, video+audio DIFFUSION) | **ABI v12 ONE SURFACE; device selector uses generic `DeviceType`; DSR 32.** t2va+fl2va COHERENT; bf16 shards STREAM | ref2va ckpt fidelity §8.12; encoder A/B §8.15; GB10 re-verify residual; CPU fold 6/137 (one queue + device provenance mutation-gated) | -| LTX-2.5 (`LTX2VideoTransformer3DModel`, video+audio DIFFUSION) | **L1-L9c landed (#435).** 21.00B / 48 blocks. `VideoEngine` seam + ABI **v18**, DiT forward (CPU f32 parity, bf16 device-resident), Gemma-4 TE, both VAEs, the embeddings connector, pipeline, NVFP4/FP8 arms, `/v1/videos` | A shipped 21.00B FP8 DiT runs device-resident on GB10. The 320x192/25f frames ARE a scene, register-conditioned. L13 encodes a typed prompt, FIXTURE-gated; a prompted render is OWED. Speed and oracle parity `PENDING` | +| LTX-2.5 (`LTX2VideoTransformer3DModel`, video+audio DIFFUSION) | **L1-L9c landed (#435); prompt-side AdaLN #644.** 21.00B / 48 blocks. `VideoEngine` seam + ABI **v18**, DiT forward (f32 host, bf16 device-resident), Gemma-4 TE, both VAEs, connector, pipeline, NVFP4/FP8, `/v1/videos` | A shipped 21.00B FP8 DiT runs device-resident on GB10. The 320x192/25f frames ARE a scene, register-conditioned. L13 encodes a typed prompt, FIXTURE-gated; a prompted render is OWED. Speed and oracle parity `PENDING` | | Command-R / Cohere dense (`CohereForCausalLM`) | Implemented, gate-blocked | ZERO-new-kernel port grounded in vLLM `commandr.py`: weight-only Cohere LayerNorm + GPT-J full-width RoPE + PARALLEL residual + `logit_scale` + tied embeddings, all reuse; compiles, links, self-registers. No SACRED gate yet (real checkpoints HF-gated, ungated ones tiny-random, GPU box disk-full); oracle run-verified at W0. See docs/BENCHMARKS.md | | Phi-1 / Phi-2 dense (`PhiForCausalLM`, parallel residual) | Correctness-complete, speed-pending | Token-exact 16/16 (microsoft/phi-2): 9/16 strict + 7/16 bf16 near-ties (max gap 0.25 nats), 0 forward-divergent; the OLDER Microsoft Phi arch, DISTINCT from Phi-3/Phi-4; ZERO new compute kernel (GPT-J parallel residual, LayerNorm-with-bias, biased qkv/dense, partial NeoX rope 32/80, non-gated NewGELU MLP reusing `vt::GeluTanh`, untied biased lm_head); F16 dtype-aware loader | | MiniCPM dense (`MiniCPMForCausalLM`, three scalars) | Correctness-complete, speed-pending | Token-exact 16/16 (openbmb/MiniCPM-2B-sft-bf16): 10/16 strict + 6/16 bf16 near-ties (max gap 0.0 nats), 0 forward-divergent; first OpenBMB MiniCPM model; ZERO new compute kernel (the Llama/Granite dense forward plus three scalars: scale_emb, scale_depth/sqrt(layers) residual, dim_model_base logit scaling), tied lm_head; `.bin`-only weights converted to safetensors via trusted torch | diff --git a/docs/USAGE.md b/docs/USAGE.md index 4f80a43ef..6cf9f4f0f 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2072,12 +2072,19 @@ Only the LTX-2.5 DiT is gated against an independent oracle here, so treat any other marker-less NVFP4 checkpoint as unsupported until it is. See `.agents/specs/nvfp4-nibble-order.md`. -Two behaviours a caller has to know. `Ltx2LoadDitFromSafetensors` REFUSES the -shipped DiT by default, because that file carries **one** module family this port -does not carry (`keyframes_abs_pos_embedding`); pass -`Ltx2DitLoadOptions::allow_unported_modules` +Two behaviours a caller has to know. `Ltx2LoadDitFromSafetensors` REFUSES a +shipped DiT by default over **one** module family this port does not carry, +`keyframes_abs_pos_embedding`; pass `Ltx2DitLoadOptions::allow_unported_modules` to load the ported subset, which still reports every one of them in -`Ltx2DitCheckpoint::unported`. `prompt_adaln_single` and +`Ltx2DitCheckpoint::unported`. Both shipped DiTs need the opt-in, for DIFFERENT +reasons, and the distinction matters if you are reading the refusal text +(corrected 2026-08-13): the **FP8** file CARRIES the tensor +(`F8_E4M3 [1, 4096]`) and declares no `__metadata__` at all, so the refusal is on +its TENSORS; the first-party **NVFP4** file does NOT carry the tensor but its +config DECLARES `use_keyframes_abs_pos_embedding: true`, so it is refused by the +FLAG, in `ParseLtx2DitParams`, and `allow_unported_modules` clears that flag in a +config copy. Neither file supports the retired claim that "LTX-2.5's checkpoint +does not carry the parameter". `prompt_adaln_single` and `audio_prompt_adaln_single` were on that list until 2026-08-13 and are now PORTED, so a checkpoint carrying them needs no opt-in on their account, and the opt-in no longer disables them. The two `*_embeddings_connector` towers are diff --git a/include/vllm/model_executor/models/ltx2.h b/include/vllm/model_executor/models/ltx2.h index 99898406b..87f4ddebb 100644 --- a/include/vllm/model_executor/models/ltx2.h +++ b/include/vllm/model_executor/models/ltx2.h @@ -44,9 +44,30 @@ // attention.py:545-552 `perturbation_mask`). L2 runs the no-perturbation // configuration, whose masks are all-ones and whose flags are all false — // upstream's own `perturbations=None` path (model.py:509-511). -// - `use_keyframes_abs_pos_embedding` (transformer_args.py:23-43). LTX-2.5's -// checkpoint does not carry the parameter; the enumeration refuses a config -// that asks for it. +// - `use_keyframes_abs_pos_embedding` (transformer_args.py:23-43). The +// enumeration refuses a config that asks for it. +// +// CORRECTED 2026-08-13 (issue #644). This line read "LTX-2.5's checkpoint +// does not carry the parameter", which is FALSE, and it is the same class of +// claim the prompt-AdaLN repair below exists to remove. The two shipped DiTs +// disagree with each other and each contradicts one half of it, read straight +// off their safetensors headers: +// * FP8 (`vonkaiser`, ltx-2.5-22b-distilled-fp8): CARRIES +// `keyframes_abs_pos_embedding` `F8_E4M3 [1, 4096]` plus its F32 scale, +// and declares NO `__metadata__` at all — so it ships no config, and +// upstream's own `LTXModelConfigurator.from_metadata` cannot configure it +// (run 2026-08-13: `KeyError: 'caption_channels'`, raised by +// `_build_caption_projections` on the empty dict, BEFORE the flag at +// model_configurator.py:82 is ever read). What that file's flag resolves +// to therefore depends on a config it does not ship. +// * NVFP4 (first-party Lightricks): DECLARES the flag `true` and does NOT +// carry the tensor, so upstream's parameter stays its +// `torch.zeros(1, inner_dim)` initialiser (model.py:217-219) and +// `load_state_dict(..., strict=False)` +// (loader/single_gpu_model_builder.py:98) leaves it zero — a no-op. +// It is also NOT a keyframe-only feature: `transformer_args.py:269` applies it +// on EVERY prepare whenever `keyframes_mask` is non-None, and `tools.py:186-196` +// marks the target's first latent frame unconditionally. // - The caption projections (text_projection.py:31-38). LTX-2.5 is a 22B-form // checkpoint: `caption_proj_before_connector=true` puts them in the TEXT // ENCODER, so the DiT has none (model_configurator.py:199-219). They are diff --git a/include/vllm/model_executor/models/ltx2_loader.h b/include/vllm/model_executor/models/ltx2_loader.h index 46aa57faa..8d472913d 100644 --- a/include/vllm/model_executor/models/ltx2_loader.h +++ b/include/vllm/model_executor/models/ltx2_loader.h @@ -112,7 +112,11 @@ // stays gateable — proceeds, still reporting every one of them in `unported`: // // keyframes_abs_pos_embedding [1, 4096] -// So `use_keyframes_abs_pos_embedding = TRUE`, contradicting ltx2.h:47-49. +// Present in the FP8 DiT and ABSENT from the NVFP4 one, whose config +// nonetheless DECLARES the flag `true` — so tensors and flag do not imply +// each other in either direction, and the refusal is keyed on the tensors. +// Corrected 2026-08-13 (#644); the measured behaviour of BOTH files, and +// why the keying stays on tensors, is in ltx2.h. // This is now the ONLY flag `Ltx2AdoptDeclaredDitParams` clears in its // config copy, and it must stay that way: a flag cleared there is invisible // to the contract-equality check, so clearing a PORTED one silently drops diff --git a/scripts/gen-ltx2-goldens.py b/scripts/gen-ltx2-goldens.py index 4c5d3e54b..119374ae9 100644 --- a/scripts/gen-ltx2-goldens.py +++ b/scripts/gen-ltx2-goldens.py @@ -752,15 +752,30 @@ def rel(a, b): f"max|table| = {static_max:.6g} ({term_max / static_max * 100:.1f}%)" ) lines.append( - "// The two output rows are bounded by this generator's SYNTHETIC weight scale" + "// ALL FOUR ROWS ARE GATE-FLOOR NUMBERS FROM SYNTHETIC WEIGHTS. The table and" ) lines.append( - "// (0.05, param_spec above) and by a 2-block stack; they are the FLOOR the gate" + "// the prompt-AdaLN MLP are BOTH drawn at param_spec's scale=0.05 above, so every" ) lines.append( - "// needs, not a claim about the trained checkpoint. The K/V row is where the" + "// ratio here is a property of THIS FIXTURE and moves with that scale; the output" ) - lines.append("// term actually enters and is the number that answers 'does this matter'.") + lines.append( + "// rows are bounded by it AND by a 2-block stack. They are reported because a" + ) + lines.append( + "// mutation has to be shown to move something -- NOT as a claim about the trained" + ) + lines.append( + "// checkpoint. On the SHIPPED DiT the term DOMINATES the table it is added to:" + ) + lines.append( + "// rms|term|/rms|table| = 1347% video, 1583% audio, measured through upstream's" + ) + lines.append( + "// own AdaLayerNormSingle on the real weights. See" + ) + lines.append("// .agents/specs/ltx25-prompt-adaln.md section Outcome.") text = "\n".join(lines) print("prompt-AdaLN magnitude:\n" + text, file=sys.stderr) return text @@ -818,7 +833,7 @@ def main() -> int: # Upstream's DEFAULT arm, and the one the shipped checkpoint runs. emit_prompt_adaln(out) out.write( - "// --- the MEASURED magnitude of the prompt-AdaLN term ---\n" + "// --- the prompt-AdaLN term's magnitude ON THIS SYNTHETIC FIXTURE ---\n" "// Same shared weights, same inputs, flag ON vs OFF:\n" + measure_prompt_adaln_magnitude() + "\n" diff --git a/scripts/measure-ltx2-prompt-adaln.py b/scripts/measure-ltx2-prompt-adaln.py new file mode 100755 index 000000000..8961f845c --- /dev/null +++ b/scripts/measure-ltx2-prompt-adaln.py @@ -0,0 +1,164 @@ +#!/usr/bin/env python3 +"""Measure the LTX-2.5 prompt-AdaLN timestep term against the static per-block +table, ON THE SHIPPED CHECKPOINT'S OWN WEIGHTS. + +Why this exists. `scripts/gen-ltx2-goldens.py` reports the same ratio at reduced +dimensions, but there the table AND the prompt-AdaLN MLP are both drawn from +`param_spec`'s `scale=0.05`, so the ratio it prints is a property of that fixture +and moves with the init scale. It is a gate FLOOR, not a statement about the +trained model. This script answers the other question — how much of the prompt +K/V modulation is timestep-conditioned on the model people actually run — and its +output is what `.agents/specs/ltx25-prompt-adaln.md` §Outcome quotes (issue #644). + +Nothing of ours is in the numeric path. It reads the safetensors header, pulls +six tensors per stream plus the 96 per-block tables, and runs UPSTREAM's own +module: + + ltx_core/model/transformer/adaln.py:19-45 AdaLayerNormSingle + ltx_core/model/transformer/model.py:223-227,253-257 built at embedding_coefficient=2 + ltx_core/model/transformer/transformer_args.py:274-278 driven by SIGMA, not timesteps + ltx_core/model/transformer/transformer_args.py:177 scaled by timestep_scale_multiplier + ltx_core/model/transformer/transformer.py:441-446 table + term, then context*(1+scale)+shift + ltx_core/components/schedulers.py:60-88 the sampler the file's config names + +Usage: + python3 scripts/measure-ltx2-prompt-adaln.py \ + --ltx2 ~/_git/LTX-2 \ + --checkpoint $CHECKPOINT_ROOT/ltx-2.5/lightricks-ltx-2.5/diffusion_models/\ +ltx-2.5-22b-distilled-transformer-nvfp4.safetensors +""" + +from __future__ import annotations + +import argparse +import json +import struct +import sys +from pathlib import Path + +PREFIX = "model.diffusion_model." +ADALN_KEYS = ( + "emb.timestep_embedder.linear_1.weight", + "emb.timestep_embedder.linear_1.bias", + "emb.timestep_embedder.linear_2.weight", + "emb.timestep_embedder.linear_2.bias", + "linear.weight", + "linear.bias", +) + + +def load_upstream(root: Path): + """Import `ltx_core` BY PATH from `root`, and PROVE that is what resolved. + + A pip-installed or otherwise-shadowing `ltx_core` would import silently and + every number below would describe an upstream nobody pinned — the failure + mode `.agents/specs/ltx-2-5.md` records at its "(b) byte-identical goldens" + entry. Identity is asserted, not assumed. + """ + src = root / "packages" / "ltx-core" / "src" + if not (src / "ltx_core").is_dir(): + raise SystemExit(f"no ltx_core under {src}; point --ltx2 at a Lightricks/LTX-2 checkout") + sys.path.insert(0, str(src)) + import ltx_core # noqa: PLC0415 + + resolved = Path(ltx_core.__file__).resolve() + if resolved.parent != (src / "ltx_core").resolve(): + raise SystemExit(f"ltx_core resolved to {resolved}, NOT to the checkout at {src}") + return resolved + + +def read_header(path: Path) -> dict: + """The safetensors header alone: an 8-byte length prefix and its JSON. No payload.""" + with path.open("rb") as fh: + length = struct.unpack(" float: + return float(t.pow(2).mean().sqrt()) + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--ltx2", required=True, type=Path, help="path to a Lightricks/LTX-2 checkout") + ap.add_argument("--checkpoint", required=True, type=Path, help="a shipped LTX-2.5 DiT .safetensors") + ap.add_argument("--steps", type=int, default=8, help="step count for the shipped sampler arm") + args = ap.parse_args() + + resolved = load_upstream(args.ltx2.expanduser()) + import torch # noqa: PLC0415 + from safetensors import safe_open # noqa: PLC0415 + + from ltx_core.components.schedulers import LinearQuadraticScheduler # noqa: PLC0415 + from ltx_core.model.transformer.adaln import AdaLayerNormSingle # noqa: PLC0415 + + ckpt = args.checkpoint.expanduser() + print(f"ltx_core: {resolved}") + print(f"checkpoint: {ckpt}") + + header = read_header(ckpt) + meta = header.get("__metadata__") + if not meta or "config" not in meta: + raise SystemExit(f"{ckpt} declares no __metadata__ config; this script needs the file's own geometry") + config = json.loads(meta["config"]) + tcfg = config["transformer"] + mult = tcfg.get("timestep_scale_multiplier", 1000) + blocks = tcfg["num_layers"] + dims = { + "video": (tcfg["num_attention_heads"] * tcfg["attention_head_dim"], + "prompt_adaln_single.", "prompt_scale_shift_table"), + "audio": (tcfg["audio_num_attention_heads"] * tcfg["audio_attention_head_dim"], + "audio_prompt_adaln_single.", "audio_prompt_scale_shift_table"), + } + print(f"config: num_layers={blocks} timestep_scale_multiplier={mult} " + f"cross_attention_adaln={tcfg.get('cross_attention_adaln')} " + f"use_prompt_adaln_single={tcfg.get('use_prompt_adaln_single', ' upstream default True>')} " + f"sampler={config.get('scheduler', {}).get('sampler')}") + + arms = { + "uniform sigma [0,1] x101": torch.linspace(0.0, 1.0, 101, dtype=torch.float32), + f"shipped LinearQuadratic, {args.steps} steps": LinearQuadraticScheduler().execute(args.steps), + } + + torch.set_grad_enabled(False) + with safe_open(str(ckpt), framework="pt") as fh: + for stream, (dim, sub, table_name) in dims.items(): + module = AdaLayerNormSingle(dim, embedding_coefficient=2) + module.load_state_dict( + {k: fh.get_tensor(PREFIX + sub + k).float() for k in ADALN_KEYS}, strict=True + ) + module.eval() + tables = torch.stack([ + fh.get_tensor(f"{PREFIX}transformer_blocks.{i}.{table_name}").float() + for i in range(blocks) + ]) # [blocks, 2, dim]; row 0 shift, row 1 scale (transformer.py:444) + + t_rms, t_max = rms(tables), float(tables.abs().max()) + print(f"\n=== {stream} dim={dim} blocks={blocks}") + print(f" rms|table|={t_rms:.6f} max|table|={t_max:.6f}") + for arm, sigmas in arms.items(): + term = module(sigmas * mult, hidden_dtype=torch.float32)[0] + term = term.reshape(sigmas.numel(), 2, dim) # transformer.py:443 + e_rms, e_max = rms(term), float(term.abs().max()) + print(f" [{arm}] rms|term|={e_rms:.6f} -> {100.0 * e_rms / t_rms:.0f}% of rms|table|; " + f"max|term|={e_max:.6f} -> {100.0 * e_max / t_max:.0f}% of max|table|") + for row, name in ((0, "shift"), (1, "scale")): + print(f" {name}: rms|table|={rms(tables[:, row]):.6f} " + f"rms|term|={rms(term[:, row]):.6f} " + f"-> {100.0 * rms(term[:, row]) / rms(tables[:, row]):.0f}%") + # What is actually consumed (transformer.py:446): context*(1+scale)+shift. + # For a context of unit rms with independent entries the modulated + # context has rms sqrt(mean((1+scale)^2) + mean(shift^2)). + def modulated(scale, shift) -> float: + return float(((1.0 + scale).pow(2).mean() + shift.pow(2).mean()).sqrt()) + + static = modulated(tables[:, 1], tables[:, 0]) + full = modulated(tables[:, 1].unsqueeze(0) + term[:, 1].unsqueeze(1), + tables[:, 0].unsqueeze(0) + term[:, 0].unsqueeze(1)) + print(f" modulated context on a UNIT-rms context: static-only={static:.4f} " + f"upstream={full:.4f} ({100.0 * (full - static) / static:+.1f}%)") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/vllm/model_executor/models/ltx2.cpp b/src/vllm/model_executor/models/ltx2.cpp index 3d9af69ce..344155af1 100644 --- a/src/vllm/model_executor/models/ltx2.cpp +++ b/src/vllm/model_executor/models/ltx2.cpp @@ -193,7 +193,9 @@ Ltx2DitParams ParseLtx2DitParams(const nlohmann::json& metadata) { !cfg.at("use_keyframes_abs_pos_embedding").is_boolean() || !cfg.at("use_keyframes_abs_pos_embedding").get(), "ltx2: use_keyframes_abs_pos_embedding is not ported (transformer_args.py:23-43); " - "the LTX-2.5 checkpoint does not carry keyframes_abs_pos_embedding"); + "upstream applies it on every prepare whose keyframes_mask is set, which " + "tools.py:186-196 populates unconditionally, so this is refused rather than " + "ignored (see ltx2.h for what each shipped DiT actually declares and carries)"); VT_CHECK(p.num_attention_heads > 0 && p.attention_head_dim > 0 && p.audio_num_attention_heads > 0 && p.audio_attention_head_dim > 0 && diff --git a/src/vllm/model_executor/models/ltx2_loader.cpp b/src/vllm/model_executor/models/ltx2_loader.cpp index bfc85075c..398d578a9 100644 --- a/src/vllm/model_executor/models/ltx2_loader.cpp +++ b/src/vllm/model_executor/models/ltx2_loader.cpp @@ -541,8 +541,11 @@ std::vector ContractOf(const Ltx2DitParams& params) { } Fail( "the checkpoint carries modules this port does NOT carry: " + list + - ". They are not dropped silently: keyframes_abs_pos_embedding means " - "use_keyframes_abs_pos_embedding is TRUE, and nothing here applies it. " + ". They are not dropped silently. This refusal is keyed on the TENSORS the " + "file carries, NOT on a declared flag, and that is deliberate: the FP8 DiT " + "carries a trained keyframes_abs_pos_embedding while declaring no " + "__metadata__ at all, so a flag-keyed refusal would read a DEFAULT rather " + "than the file and would silently discard it (see ltx2.h). " "prompt_adaln_single / audio_prompt_adaln_single are NO LONGER in this list " "— they were ported by row LTX25-PROMPT-ADALN " "(.agents/specs/ltx25-prompt-adaln.md, issue #644) and are now part of the " diff --git a/tests/vllm/models/ltx2_goldens.inc b/tests/vllm/models/ltx2_goldens.inc index 964047788..d7f73a4ce 100644 --- a/tests/vllm/models/ltx2_goldens.inc +++ b/tests/vllm/models/ltx2_goldens.inc @@ -3227,14 +3227,19 @@ inline constexpr float kLtx2ForwardPromptAdalnMaskedAudio[] = { -0.09456487f, 0.0829611197f, 0.000701400219f, -0.243942767f, 0.0671276823f, 0.0765738413f, }; -// --- the MEASURED magnitude of the prompt-AdaLN term --- +// --- the prompt-AdaLN term's magnitude ON THIS SYNTHETIC FIXTURE --- // Same shared weights, same inputs, flag ON vs OFF: // DiT video output: max|on-off| = 0.000145674 (0.04% of max|off|) // DiT audio output: max|on-off| = 7.36713e-05 (0.03% of max|off|) // block 0 modulated prompt K/V: max|on-off| = 0.0309528 (5.82% of max|off|) // timestep term vs static table: max|term| = 0.0252012 vs max|table| = 0.0487142 (51.7%) -// The two output rows are bounded by this generator's SYNTHETIC weight scale -// (0.05, param_spec above) and by a 2-block stack; they are the FLOOR the gate -// needs, not a claim about the trained checkpoint. The K/V row is where the -// term actually enters and is the number that answers 'does this matter'. +// ALL FOUR ROWS ARE GATE-FLOOR NUMBERS FROM SYNTHETIC WEIGHTS. The table and +// the prompt-AdaLN MLP are BOTH drawn at param_spec's scale=0.05 above, so every +// ratio here is a property of THIS FIXTURE and moves with that scale; the output +// rows are bounded by it AND by a 2-block stack. They are reported because a +// mutation has to be shown to move something -- NOT as a claim about the trained +// checkpoint. On the SHIPPED DiT the term DOMINATES the table it is added to: +// rms|term|/rms|table| = 1347% video, 1583% audio, measured through upstream's +// own AdaLayerNormSingle on the real weights. See +// .agents/specs/ltx25-prompt-adaln.md section Outcome. } // namespace vllm_test diff --git a/tests/vllm/models/test_ltx2.cpp b/tests/vllm/models/test_ltx2.cpp index f00771f38..b4896462f 100644 --- a/tests/vllm/models/test_ltx2.cpp +++ b/tests/vllm/models/test_ltx2.cpp @@ -951,12 +951,21 @@ TEST_CASE("ltx2 forward: the prompt-side AdaLN arm, with both masks") { // term. A port that accepted the flag, bound the 12 tensors and then never added // their output would reproduce the flag-OFF numbers exactly and pass nothing here. // -// Measured on this fixture (generator stderr, and the comment block at the end of -// ltx2_goldens.inc): the term is 51.7% the magnitude of the static per-block -// table it is added to, moves the block-0 modulated prompt K/V by 5.82%, and -// moves the DiT's own output by 1.46e-4 — 73x the kRoundOff floor. The bound -// below is set at 20x kRoundOff so it is comfortably inside the measured signal -// and comfortably outside f32 noise. +// WHAT THIS FIXTURE'S NUMBERS ARE, AND ARE NOT. The generator's stderr and the +// comment at the end of ltx2_goldens.inc report the term at 51.7% of the static +// per-block table, the block-0 prompt K/V moving 5.82%, and the DiT output moving +// 1.46e-4 (73x kRoundOff). ALL FOUR are GATE-FLOOR numbers from SYNTHETIC weights, +// not a claim about the trained checkpoint: the table and the prompt-AdaLN MLP are +// both drawn at `param_spec`'s scale=0.05 (gen-ltx2-goldens.py:100-106), so every +// ratio is a property of THIS FIXTURE and moves with the init scale. +// +// On the SHIPPED DiT the term DOMINATES the table rather than halving it — +// rms|term|/rms|table| = 1347% video, 1583% audio, measured through upstream's own +// AdaLayerNormSingle on the real weights (.agents/specs/ltx25-prompt-adaln.md +// §Outcome). So this fixture UNDERSTATES the defect; it does not bound it. +// +// The bound below is set at 20x kRoundOff: comfortably inside the signal this +// fixture does produce, and comfortably outside f32 noise. TEST_CASE("ltx2 forward: the prompt-AdaLN term is LOAD-BEARING, not decoration") { const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); WeightSet set = BuildWeights(p); diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 85b47025a..ad1e39e6d 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -913,6 +913,11 @@ TEST_CASE("ltx2 video: an ABI client loads, detects and generates through vllm.h // It is NOT interchangeable with the `vonkaiser` FP8 copy — they differ in a // TRAINED `keyframes_abs_pos_embedding` (spec section 3.1) — so the file this // case reads is named here and in every report of its result. +// WHEN QUOTING THIS SUITE'S ASSERTION COUNT, QUOTE THE CONFIGURATION WITH IT. +// This case skips by default, and it is most of the suite: unset, the binary +// measures 30 cases / 502 assertions; with LTX2_CHECKPOINT_ROOT set it measures +// 30 / 8734. An "unchanged 30 / 502" across a change therefore says the real +// headers were never read, not that they were read and nothing counted them. TEST_CASE("ltx2 video: the SHIPPED Lightricks checkpoints parse and load") { const char* root_env = std::getenv("LTX2_CHECKPOINT_ROOT"); if (root_env == nullptr) { From 8531fd0eb2610cea4eec00b4ce84ed8535219c5e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 21:28:03 +0000 Subject: [PATCH 04/10] record(LTX25-ADALN-CLAIMS): the keyframes parameter is left on META, not zeroed, and the audio factor had the wrong denominator (#644, #673) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review findings F-1 and F-2 on PR #663. The reviewer re-ran every measurement in that PR and every published digit reproduced; this repairs the two claims that did not. ## F-1 (blocking) — a claim about upstream that upstream's own code contradicts Three places said that on the NVFP4 DiT the absent `keyframes_abs_pos_embedding` means the parameter "keeps `torch.zeros(1, inner_dim)` (model.py:217-219) through `load_state_dict(..., strict=False)` (single_gpu_model_builder.py:98) — a genuine no-op there": `include/vllm/model_executor/models/ltx2.h` (THE PRODUCT TREE), the spec, and the PR body. IT IS NOT A NO-OP. The claim quoted `single_gpu_model_builder.py:98` while dropping the `assign=True` that is on that same line, and upstream builds on the META DEVICE -- `loader/helpers.py:84-95`, `create_meta_model`: `with torch.device("meta"): configurator.from_metadata(...)` at `:90-91`. A key absent from the state dict is therefore never materialised at all. RUN HERE, not transcribed, through upstream's own `create_meta_model` on the shipped file's real `__metadata__` (read with upstream's own `read_model_metadata` / `SafetensorsModelStateDictLoader`, sft_loader.py:58-74, which JSON-decodes each value -- the flag lives at `config.transformer`, not at the top level, so a raw `__metadata__` read returns None for it): keys matching 'keyframes_abs_pos' in the file : [] (0 of 7876 entries) config.transformer['use_keyframes_abs_pos_embedding'] = True keyframes_abs_pos_embedding: shape=(1, 4096) f32 device=meta is_meta=True supports_keyframes_abs_pos_embedding (BEFORE load) : False after load_state_dict(sd, strict=False, assign=True): neighbour patchify_proj.weight : device=cpu is_meta=False <- materialised keyframes_abs_pos_embedding : device=meta is_meta=True in missing_keys : True reading the value RAISES : RuntimeError: Tensor.item() cannot be called on meta tensors supports_keyframes_abs_pos_embedding (AFTER load) : False Upstream states this itself and then never asks. `supports_keyframes_abs_pos_embedding` (model.py:166-173) returns False for "a model whose config set the flag but whose checkpoint carried no weight for it (the parameter would still be on ``meta``)" (:170), and `enable_keyframes_abs_pos_embedding` (model.py:175-200) exists because such a parameter "would fail at the first forward" (:182). BOTH ARE DEFINED AND NEVER CALLED: `grep -rn` over the whole Lightricks/LTX-2 checkout at fd4ded7f returns exactly 1 hit each, the definition, re-confirmed here against a positive control rather than asserted from a null grep. Polarity is not what fails: `apply_keyframes_absolute_embedding` is `hidden_states + mask * embedding` (transformer_args.py:23-43, the sum at :43), so REAL zeros would be inert. The mechanism claim fails on `meta`, not on additivity. This STRENGTHENS the row's conclusion and changes nothing downstream: the refusal stays keyed on TENSOR PRESENCE for the FP8 file, and the NVFP4 file is refused by FLAG in `ParseLtx2DitParams` (ltx2.cpp:192-198). NOTHING RUNTIME CHANGED IN THIS COMMIT. `ltx2.h` and `test_ltx2.cpp` move comment text only; the one script edit (`gen-ltx2-goldens.py`) alters only the COMMENT the generator emits; `ltx2_goldens.inc` moves two comment lines with every value byte-identical. No `src/` file is touched at all. Suite counts are therefore expected to be identical, and the point of the gate below is that they are. The same anchor is now quoted in full at the OTHER place it appears (the documented-divergence bullet), because dropping the `assign=True` is precisely what made the claim above wrong. ## F-2 — the audio factor's denominator The Outcome read "138x (video) and 54x (audio)" as if both were per-stream. The video figure is right and like-for-like. The audio one divided the shipped AUDIO ratio by the VIDEO fixture denominator, because `gen-ltx2-goldens.py` emits exactly ONE fixture ratio and it is the video stream's. 54x is literally true about that single published figure and errs CONSERVATIVE. Recomputed by IMPORTING the generator, so the weight stream is bit-identical and no weight is re-drawn: video: max|term| = 0.0252012 max|table| = 0.0487142 -> 51.7327% (7119/51.73 = 137.6x) audio: max|term| = 0.0201763 max|table| = 0.0496868 -> 40.6068% (2817/40.61 = 69.4x) FIXED AT THE SOURCE, not only in prose. The generator emitted BOTH video-only rows unlabelled -- "timestep term vs static table" and "block 0 modulated prompt K/V" -- which is exactly how a video denominator came to be applied to an audio numerator. `gen-ltx2-goldens.py` now names the stream on both, and `ltx2_goldens.inc` was REGENERATED against ltx_core fd4ded7f to carry them (REGEN_EXIT=0). The regeneration diff is those two comment lines and NOTHING else: every golden VALUE byte-identical, and the "Regenerate with:" header unchanged because the committed command was re-run verbatim -- so this re-proves provenance as well as the labels. The Outcome now carries a per-stream table with each row against its own denominator, and `test_ltx2.cpp` and the spec's gate-floor table say which stream they mean. No fifth golden was added: the audio figure is a denominator for this record, not a gate floor, so "ALL FOUR ROWS ARE GATE-FLOOR NUMBERS" still holds and the three files that label those four are otherwise untouched. ## Recorded, not fixed — issue #673 `LTX2_CHECKPOINT_ROOT` is set by NO workflow: `grep -rn CHECKPOINT_ROOT .github/` exits 1 with zero hits while the same pattern matches in `tests/` and `.agents/` (positive control run in the same command). CI therefore executes 502 of 8734 assertions -- 5.7% -- of `test_ltx2_video`, at an UNCHANGED case count of 30, and `scripts/measure-ltx2-prompt-adaln.py` is a manual tool no gate invokes. This row's checkpoint-derived evidence is manual and host-local. Filed as visible debt and linked in the roadmap issue table and the spec; wiring checkpoints into CI is explicitly not in this row. ## No test asserts this text, and none was invented Searched in the subject's own vocabulary: the corrected claim lives only in a `ltx2.h` comment, the spec and the PR body. The runtime refusal MESSAGES (`ltx2.cpp:195-198`, `ltx2_loader.cpp` `RefuseUnported`) never carried the "zeros"/"no-op" implication, and no checker reads the header comment. Writing a test here would be writing one that cannot fail, so this change adds none. The tensor-keyed refusal it defends is already pinned by a mutation the reviewer ran: making `LoadedElsewhere` swallow the tensor takes `test_ltx2_loader` to 25/26 cases, 4817/4818 assertions, exit 1. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/roadmap_v1.md | 1 + .agents/specs/ltx25-prompt-adaln.md | 125 ++++++++++++++++++++-- docs/USAGE.md | 19 +++- include/vllm/model_executor/models/ltx2.h | 40 ++++++- scripts/gen-ltx2-goldens.py | 11 +- tests/vllm/models/ltx2_goldens.inc | 4 +- tests/vllm/models/test_ltx2.cpp | 7 +- 7 files changed, 183 insertions(+), 24 deletions(-) diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 29ab578da..f0fd89fa6 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -151,6 +151,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#560](https://github.com/mudler/vllm.cpp/issues/560) | `ROAD-V1-LTX25` | Stabilizing constants unreachable by any gate: FIVE instances found by sweep, incl. the DiT's own `norm_eps` field default | bug | | [#567](https://github.com/mudler/vllm.cpp/issues/567) | `ROAD-V1-LTX25` | `OpNameImpl` makes a SECOND file every new `OpId` must edit; collapse the enum and its names into one derived list | bug | | [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)): `use_prompt_adaln_single` defaults TRUE in BOTH references and the shipped DiT carries its 18 tensors, but `ltx2_loader.cpp:988` cleared the flag unconditionally, so every render dropped the timestep-conditioned half of the prompt K/V modulation — invisible to every shape and finiteness check | feature | +| [#673](https://github.com/mudler/vllm.cpp/issues/673) | `ROAD-V1-LTX25` | #644's checkpoint-derived evidence is MANUAL and host-local: `LTX2_CHECKPOINT_ROOT` is set by no workflow (`grep -rn CHECKPOINT_ROOT .github/` exits 1, positive control in `tests/`), so CI runs **502 of 8734 assertions — 5.7%** of `test_ltx2_video` at an unchanged case count of 30, and `scripts/measure-ltx2-prompt-adaln.py` is a manual tool no gate invokes. Recorded as visible debt; the skip itself is already announced in four places and that disposition stands | bug | | [#615](https://github.com/mudler/vllm.cpp/issues/615) | `GATE-PR-SIZE-BINARY` | `check-pr-size` fail-closes on every binary path with no exemption route, so no golden-bearing PR can merge: it blocks #431 and post-dates the golden precedent it rejects, spec [`gate-pr-size-binary.md`](specs/gate-pr-size-binary.md) | bug | | [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug | | [#264](https://github.com/mudler/vllm.cpp/issues/264) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprob_token_ids`: generative scoring over an explicit token set is unported | feature | diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md index 724566583..4ffcf7335 100644 --- a/.agents/specs/ltx25-prompt-adaln.md +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -273,8 +273,8 @@ term and nothing else): | Quantity | Flag ON vs OFF | |---|---| -| timestep term vs the static table it is added to | `max\|term\|` 0.0252 vs `max\|table\|` 0.0487 — 51.7% | -| block-0 modulated prompt K/V | `max\|on-off\|` 0.0310 — 5.82% of `max\|off\|` | +| timestep term vs the static table it is added to (**VIDEO stream**) | `max\|term\|` 0.0252 vs `max\|table\|` 0.0487 — 51.7% | +| block-0 modulated prompt K/V (**VIDEO stream**) | `max\|on-off\|` 0.0310 — 5.82% of `max\|off\|` | | DiT video output (2 blocks) | 1.4567e-4 — 0.04% of `max\|off\|`, **73x** the gate's 2e-6 floor | | DiT audio output (2 blocks) | 7.367e-5 — 0.03%, **37x** the floor | @@ -328,10 +328,53 @@ the perturbation.** What the pre-row renders applied was 1.3170 (audio, **+31.3%**). **The synthetic fixture UNDERSTATES the real defect by two orders of magnitude.** -Comparing like with like — the fixture's 51.7% is a `max\|term\|` / `max\|table\|` -ratio, and that same ratio on the shipped weights is 7119% (video) and 2817% -(audio): **138x and 54x** larger. Recorded because the original Outcome quoted -"roughly half the magnitude" from the fixture as if it described the checkpoint. +Comparing like with like — the fixture ratio is `max\|term\|` / `max\|table\|`, +and so is the shipped-weights ratio — **per stream, each against its own fixture +denominator**: + +| | fixture `max\|term\|`/`max\|table\|` | shipped | understatement | +|---|---|---|---| +| video (`prompt_adaln_single` vs `prompt_scale_shift_table`) | 51.7% | 7119% | **138x** | +| audio (`audio_prompt_adaln_single` vs `audio_prompt_scale_shift_table`) | 40.6% | 2817% | **69x** | + +**CORRECTED 2026-08-13 (issue #644): this read "138x and 54x", which is not a +per-stream pair.** The generator emits exactly ONE fixture ratio and it is the +VIDEO stream's (`gen-ltx2-goldens.py`, `measure_prompt_adaln_magnitude`), so 54x +divided the shipped AUDIO number by the VIDEO denominator. It is literally true +about the single published figure and it errs CONSERVATIVE — the real audio +understatement is larger. The audio stream's own fixture ratio, recomputed with +the identical statistic on the identical fixture (the generator is imported, so +the weight stream is bit-identical), is **40.6%** — `max|term|` 0.0201763 vs +`max|table|` 0.0496868 — giving **2817 / 40.6 = 69x**. The video row is +unchanged and was always like-for-like: `max|term|` 0.0252012 vs `max|table|` +0.0487142 = 51.7327%, and 7119 / 51.73 = 137.6. + +Reproduce the audio denominator by importing the generator and asking it the same +question the video row is built from (`measure_prompt_adaln_magnitude`), so no +weight is re-drawn: + +```python +gen.load_upstream(LTX2); torch.set_grad_enabled(False) +on = gen.build_model("split", False, prompt_adaln=True) +video, audio = gen.build_modalities(False) +scale = float(gen.ARCH["timestep_scale_multiplier"]) +amod, _ = on.audio_prompt_adaln_single((audio.sigma * scale).flatten(), hidden_dtype=torch.float32) +amod.abs().max() / on.transformer_blocks[0].audio_prompt_scale_shift_table.abs().max() +``` + +**Fixed at the source, not only in prose.** The generator emitted both video-only +rows unlabelled, which is how a video denominator came to be applied to an audio +numerator; `gen-ltx2-goldens.py` now names the stream on both, and +`ltx2_goldens.inc` was REGENERATED against `ltx_core` `fd4ded7f` to carry them. +The regeneration diff is those two comment lines and nothing else — every golden +VALUE byte-identical, and the `Regenerate with:` header unchanged because the +committed command was re-run verbatim, so this re-proves provenance as well as +the labels. `test_ltx2.cpp` says the same. No fifth golden was added: the audio +figure is a denominator for this record, not a gate floor, so "ALL FOUR ROWS" +still holds. + +Recorded because the original Outcome quoted "roughly half the magnitude" from +the fixture as if it described the checkpoint. Reproduce with `scripts/measure-ltx2-prompt-adaln.py --ltx2 --checkpoint `, committed by this repair so the number is @@ -385,6 +428,19 @@ measures **30 cases / 8734 assertions**, both before and after this repair (re-measured on this branch, exit 0 in both configurations). Any future quote of this suite's count owes the configuration alongside it. +**And CI never sets it — [#673](https://github.com/mudler/vllm.cpp/issues/673), +filed 2026-08-13 as visible debt rather than repaired here.** `grep -rn +CHECKPOINT_ROOT .github/` exits 1 with zero hits while the same pattern matches in +`tests/` and `.agents/` (positive control run in the same command, so this is not +an assertion from a failed grep). CI therefore executes **502 of 8734 assertions — +5.7%** of this suite, at an unchanged case count of 30, and +`scripts/measure-ltx2-prompt-adaln.py` — which produces every shipped-weights +number in this Outcome — is a manual tool no gate invokes (`grep -rn +measure-ltx2-prompt-adaln` hits only its own usage string and this file). So this +row's checkpoint-derived evidence is **manual and host-local**: reproducible only +on a box carrying the 18.72 GB NVFP4 DiT and the 23 GB FP8 DiT under +`$CHECKPOINT_ROOT`. Wiring checkpoints into CI is explicitly NOT in this row. + The `test_ltx2_video` fixture had to move: it declared a config that omits `use_prompt_adaln_single` (mirroring the shipped NVFP4 DiT) while its SHAPES said false, so the config/shape equality check refused it — correctly. It now carries @@ -449,9 +505,54 @@ measured rather than read: config, so what its flag resolves to is decided entirely out of band — and the tensor it carries is trained (`.agents/specs/ltx-2-5.md` §3.1 reads its bytes). - On the NVFP4 file the flag IS on and the module IS built, but the tensor is - absent, so it keeps `torch.zeros(1, inner_dim)` (`model.py:217-219`) through - `load_state_dict(..., strict=False)` - (`loader/single_gpu_model_builder.py:98`) — a genuine no-op there. + absent — 0 of its 7876 entries match `keyframes_abs_pos`. + + **CORRECTED AGAIN 2026-08-13, same issue.** This bullet said the parameter + "keeps `torch.zeros(1, inner_dim)` (`model.py:217-219`) through + `load_state_dict(..., strict=False)` (`loader/single_gpu_model_builder.py:98`) + — a genuine no-op there". **It is not a no-op.** The claim quoted that line + while dropping the `assign=True` that is ON THE SAME LINE, and upstream builds + on the **meta device** — `loader/helpers.py:84-95`, `create_meta_model`: + `with torch.device("meta"): configurator.from_metadata(...)` at `:90-91`. A key + absent from the state dict is therefore never materialised at all; it is not a + zero, it is an **unmaterialised meta parameter**. + + Run 2026-08-13 through upstream's own `create_meta_model` on this file's real + `__metadata__` (read with upstream's own `read_model_metadata` / + `SafetensorsModelStateDictLoader`, which JSON-decodes each value — + `sft_loader.py:58-74`; the flag lives at `config.transformer`, not at the top + level): + + ```text + config.transformer['use_keyframes_abs_pos_embedding'] = True + keyframes_abs_pos_embedding: shape=(1, 4096) dtype=torch.float32 device=meta is_meta=True + supports_keyframes_abs_pos_embedding (BEFORE load) : False + after load_state_dict(sd, strict=False, assign=True): + neighbour patchify_proj.weight : device=cpu is_meta=False <- materialised + keyframes_abs_pos_embedding : device=meta is_meta=True + in missing_keys : True + reading the value RAISES : RuntimeError: Tensor.item() cannot be called on meta tensors + supports_keyframes_abs_pos_embedding (AFTER load) : False + ``` + + **Upstream says exactly this itself**, and then never asks. + `supports_keyframes_abs_pos_embedding` (`model.py:166-173`) returns `False` for + "a model whose config set the flag but whose checkpoint carried no weight for it + (the parameter would still be on `meta`)", and + `enable_keyframes_abs_pos_embedding` (`model.py:175-200`) exists because such a + parameter "would fail at the first forward". **Both are defined and never + called** — one `grep -rn` hit each across the whole `Lightricks/LTX-2` checkout + at `fd4ded7f`, the definition itself, re-confirmed here rather than transcribed. + + Polarity is not what fails. `apply_keyframes_absolute_embedding` is + `hidden_states + mask * embedding` (`transformer_args.py:23-43`, the sum at + `:43`), so **real** zeros would be inert — the mechanism claim fails on `meta`, + not on additivity. + + This **strengthens** the row's conclusion and changes nothing downstream: the + refusal stays keyed on tensor presence for the FP8 file, and the NVFP4 file is + refused by flag in `ParseLtx2DitParams` (`ltx2.cpp:192-198`) — where upstream, + had it loaded, would carry a parameter its own guard reports as unsupported. It is also not a keyframe-only feature: `transformer_args.py:269` applies it on every `prepare` whose `keyframes_mask` is set, and `tools.py:186-196` sets that @@ -497,8 +598,10 @@ tree vanished under it is not a result; it was rebuilt and re-run from scratch. ### Two divergences from upstream, recorded rather than fixed - **We are stricter than upstream about a config that disagrees with its file.** - Upstream loads with `load_state_dict(..., strict=False)` - (`loader/single_gpu_model_builder.py:98`), so a config declaring + Upstream loads with `load_state_dict(..., strict=False, assign=True)` + (`loader/single_gpu_model_builder.py:98` — quoted in full, because dropping the + `assign=True` is exactly what made the keyframes claim above wrong), so a config + declaring `use_prompt_adaln_single=false` over a file that carries the module would build no module, drop 18 tensors on the floor and run flag-OFF without a word. §3.2's equality check refuses that. Ours is better; it is still a DIVERGENCE, not a diff --git a/docs/USAGE.md b/docs/USAGE.md index 6cf9f4f0f..2249df95d 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2084,7 +2084,24 @@ its TENSORS; the first-party **NVFP4** file does NOT carry the tensor but its config DECLARES `use_keyframes_abs_pos_embedding: true`, so it is refused by the FLAG, in `ParseLtx2DitParams`, and `allow_unported_modules` clears that flag in a config copy. Neither file supports the retired claim that "LTX-2.5's checkpoint -does not carry the parameter". `prompt_adaln_single` and +does not carry the parameter". + +**What the opt-in costs you on the NVFP4 file: nothing upstream has either.** +Corrected 2026-08-13 — an earlier note said upstream keeps a zero-initialised +parameter there, "a no-op". It does not. Upstream builds on the **meta device** +(`create_meta_model`, `loader/helpers.py:90-91`) and loads with +`load_state_dict(..., strict=False, assign=True)` +(`loader/single_gpu_model_builder.py:98`), so a tensor the file does not carry is +never materialised: the parameter stays on `meta`, lands in `missing_keys`, and +reading it raises `RuntimeError: Tensor.item() cannot be called on meta tensors`. +Upstream's own `supports_keyframes_abs_pos_embedding` (`model.py:166-173`) reports +such a model as NOT supporting the feature — and nothing in the LTX-2 checkout +ever calls that guard, or the `enable_…` repair next to it. So passing +`allow_unported_modules` on the NVFP4 DiT does not drop a capability upstream +would have given you; it declines a marker upstream cannot use on that file +either. On the **FP8** file the opt-in does drop something real — a trained +`[1, 4096]` tensor — which is why the refusal there is keyed on tensor presence +and is worth taking seriously. `prompt_adaln_single` and `audio_prompt_adaln_single` were on that list until 2026-08-13 and are now PORTED, so a checkpoint carrying them needs no opt-in on their account, and the opt-in no longer disables them. The two `*_embeddings_connector` towers are diff --git a/include/vllm/model_executor/models/ltx2.h b/include/vllm/model_executor/models/ltx2.h index 87f4ddebb..5429f0555 100644 --- a/include/vllm/model_executor/models/ltx2.h +++ b/include/vllm/model_executor/models/ltx2.h @@ -60,11 +60,41 @@ // `_build_caption_projections` on the empty dict, BEFORE the flag at // model_configurator.py:82 is ever read). What that file's flag resolves // to therefore depends on a config it does not ship. -// * NVFP4 (first-party Lightricks): DECLARES the flag `true` and does NOT -// carry the tensor, so upstream's parameter stays its -// `torch.zeros(1, inner_dim)` initialiser (model.py:217-219) and -// `load_state_dict(..., strict=False)` -// (loader/single_gpu_model_builder.py:98) leaves it zero — a no-op. +// * NVFP4 (first-party Lightricks): DECLARES the flag `true` (at +// `config.transformer`, not at the top level) and does NOT carry the +// tensor — 0 of its 7876 entries match `keyframes_abs_pos`. +// +// CORRECTED AGAIN 2026-08-13, same issue. This bullet said the parameter +// "stays its `torch.zeros(1, inner_dim)` initialiser (model.py:217-219)" +// and that `load_state_dict(..., strict=False)` +// (loader/single_gpu_model_builder.py:98) "leaves it zero — a no-op". +// IT IS NOT A NO-OP. That quote dropped the `assign=True` that is ON THE +// SAME LINE, and upstream builds on the META DEVICE — helpers.py:84-95, +// `create_meta_model`, `with torch.device("meta"): +// configurator.from_metadata(...)` at :90-91. A key absent from the state +// dict is never materialised at all. What this path leaves is an +// UNMATERIALISED META PARAMETER, not a silent zero. +// RUN 2026-08-13 through upstream's own `create_meta_model` on this +// file's real `__metadata__`: `[1, 4096]` f32 `device=meta`; after +// `load_state_dict(sd, strict=False, assign=True)` still `is_meta=True`, +// present in `missing_keys`, and reading it raises `RuntimeError: +// Tensor.item() cannot be called on meta tensors`. +// Upstream states this itself and then never asks: +// `supports_keyframes_abs_pos_embedding` (model.py:166-173) returns FALSE +// for "a model whose config set the flag but whose checkpoint carried no +// weight for it (the parameter would still be on `meta`)" (:170), and +// `enable_keyframes_abs_pos_embedding` (model.py:175-200) exists because +// such a parameter "would fail at the first forward" (:182). BOTH ARE +// DEFINED AND NEVER CALLED — one `grep -rn` hit each across the whole +// Lightricks/LTX-2 checkout at fd4ded7f, the definition itself. +// Polarity is not what fails: `apply_keyframes_absolute_embedding` is +// `hidden_states + mask * embedding` (transformer_args.py:23-43, the sum +// at :43), so REAL zeros would be inert. The claim fails on `meta`, not +// on additivity. +// This STRENGTHENS the refusals below rather than changing them: this +// file is refused by FLAG in `ParseLtx2DitParams` (ltx2.cpp:192-198) and +// the FP8 file by TENSOR PRESENCE, and upstream's own guard would report +// a model loaded this way as not supporting the feature at all. // It is also NOT a keyframe-only feature: `transformer_args.py:269` applies it // on EVERY prepare whenever `keyframes_mask` is non-None, and `tools.py:186-196` // marks the target's first latent frame unconditionally. diff --git a/scripts/gen-ltx2-goldens.py b/scripts/gen-ltx2-goldens.py index 119374ae9..d09108634 100644 --- a/scripts/gen-ltx2-goldens.py +++ b/scripts/gen-ltx2-goldens.py @@ -739,8 +739,13 @@ def rel(a, b): kv_off = ctx * (1 + scale_off) + shift_off kv_on = ctx * (1 + scale_on) + shift_on kabs, krel = rel(kv_off, kv_on) + # NAME THE STREAM. Both rows below are computed from `vmod` and the VIDEO + # `prompt_scale_shift_table`, and neither said so, which is how the Outcome came + # to divide a shipped AUDIO ratio by this VIDEO denominator (issue #644). The + # audio stream's own value differs -- 40.6% against this row's 51.7% -- so an + # unlabelled ratio here is a denominator waiting to be misapplied. lines.append( - f"// block 0 modulated prompt K/V: max|on-off| = {kabs:.6g} " + f"// block 0 modulated prompt K/V (VIDEO stream): max|on-off| = {kabs:.6g} " f"({krel * 100:.2f}% of max|off|)" ) # How much of the K/V modulation is timestep-conditioned at all: the MLP row @@ -748,8 +753,8 @@ def rel(a, b): static_max = float(table.abs().max()) term_max = float(vmod.abs().max()) lines.append( - f"// timestep term vs static table: max|term| = {term_max:.6g} vs " - f"max|table| = {static_max:.6g} ({term_max / static_max * 100:.1f}%)" + f"// timestep term vs static table (VIDEO stream): max|term| = {term_max:.6g} " + f"vs max|table| = {static_max:.6g} ({term_max / static_max * 100:.1f}%)" ) lines.append( "// ALL FOUR ROWS ARE GATE-FLOOR NUMBERS FROM SYNTHETIC WEIGHTS. The table and" diff --git a/tests/vllm/models/ltx2_goldens.inc b/tests/vllm/models/ltx2_goldens.inc index d7f73a4ce..b56693b48 100644 --- a/tests/vllm/models/ltx2_goldens.inc +++ b/tests/vllm/models/ltx2_goldens.inc @@ -3231,8 +3231,8 @@ inline constexpr float kLtx2ForwardPromptAdalnMaskedAudio[] = { // Same shared weights, same inputs, flag ON vs OFF: // DiT video output: max|on-off| = 0.000145674 (0.04% of max|off|) // DiT audio output: max|on-off| = 7.36713e-05 (0.03% of max|off|) -// block 0 modulated prompt K/V: max|on-off| = 0.0309528 (5.82% of max|off|) -// timestep term vs static table: max|term| = 0.0252012 vs max|table| = 0.0487142 (51.7%) +// block 0 modulated prompt K/V (VIDEO stream): max|on-off| = 0.0309528 (5.82% of max|off|) +// timestep term vs static table (VIDEO stream): max|term| = 0.0252012 vs max|table| = 0.0487142 (51.7%) // ALL FOUR ROWS ARE GATE-FLOOR NUMBERS FROM SYNTHETIC WEIGHTS. The table and // the prompt-AdaLN MLP are BOTH drawn at param_spec's scale=0.05 above, so every // ratio here is a property of THIS FIXTURE and moves with that scale; the output diff --git a/tests/vllm/models/test_ltx2.cpp b/tests/vllm/models/test_ltx2.cpp index b4896462f..4afa40777 100644 --- a/tests/vllm/models/test_ltx2.cpp +++ b/tests/vllm/models/test_ltx2.cpp @@ -952,8 +952,11 @@ TEST_CASE("ltx2 forward: the prompt-side AdaLN arm, with both masks") { // their output would reproduce the flag-OFF numbers exactly and pass nothing here. // // WHAT THIS FIXTURE'S NUMBERS ARE, AND ARE NOT. The generator's stderr and the -// comment at the end of ltx2_goldens.inc report the term at 51.7% of the static -// per-block table, the block-0 prompt K/V moving 5.82%, and the DiT output moving +// comment at the end of ltx2_goldens.inc report the VIDEO stream's term at 51.7% +// of its static per-block table — that ratio is emitted for the video stream ONLY, +// and the audio stream's own value on the same fixture is 40.6%, so the 51.7% is +// not a denominator for anything audio — the block-0 prompt K/V moving 5.82%, and +// the DiT output moving // 1.46e-4 (73x kRoundOff). ALL FOUR are GATE-FLOOR numbers from SYNTHETIC weights, // not a claim about the trained checkpoint: the table and the prompt-AdaLN MLP are // both drawn at `param_spec`'s scale=0.05 (gen-ltx2-goldens.py:100-106), so every From d3f846768b0dfecd4854293db5dd14ca910595bd Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 12:14:39 +0000 Subject: [PATCH 05/10] record(LTX25-PROMPT-ADALN): the term is reached and no test enters through the door, plus the counts and claims the keyframe port moved (#644, #658, #886, #900) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The merge before this one reconciled the TREE onto `98f8e046d`. This reconciles what the row SAYS, files the issue it was carrying outside the index, and answers a rule that landed after the branch was written. ## `.agents/issue-index.md` — #673 was never indexed `AGENTS.md` `## Every change starts from an issue` requires the number in three places that agree: the index, the row's spec, and the pull request body. #673 was open, named in the spec and in the pull request body, and absent from the index on this branch AND on `main` — checked both, not one. The branch had appended it to the roadmap's inline issue table instead, and `main` has since retired that table into the index. So the row moves to the surface that still exists. Two rows are appended, #673 and the new #900 below. Appended, nothing edited and nothing removed: `scripts/check-issue-index-append-only.py --base origin/main --head HEAD` exits 0, and `scripts/check-agent-record.py` exits 0 at `ENGINE=156 MODEL=377 QUANT=82 KERNEL=51 BACKEND=82`. The row carries `—` in the `Row` column and is listed under `## Owed` in the spec, which is the shape the checker's unowned ratchet requires; the high-water mark is untouched. ## The spec's keyframes section argued for a disposition that #658 retired `LTX25-KEYFRAMES-ABS-POS` (#658, `98f8e046d`) ported `keyframes_abs_pos_embedding` on 2026-08-14 and retired BOTH refusals this row's Outcome reasoned about. Three passages were left asserting the opposite, and one of them is a forward-looking rule rather than history: - §3.3 said `use_keyframes_abs_pos_embedding` is "the sole flag `Ltx2AdoptDeclaredDitParams` still clears". It clears none: the declared flag is now RESOLVED against what the file carries. - "The refusal keying does NOT change" described a refusal that no longer exists. - The second of the two recorded divergences — "we refuse `keyframes_abs_pos_embedding` by tensor presence" — is gone with the refusal. The first divergence survives and is unchanged. Each is dated and marked SUPERSEDED in place rather than deleted, with the current disposition pointed at `.agents/specs/ltx25-keyframes-abs-pos.md` §2. `AGENTS.md` `## Records`: never delete evidence to reduce context. The FINDING — that the absent parameter stays on `meta` rather than becoming a zero — is not retracted; #658 acted on it, and `scripts/measure-ltx2-keyframes-meta.py` now re-runs it on demand instead of quoting it, which is the review's meta-device probe already landed on `main` rather than owed here. ## The reachability question, asked because the rule post-dates the branch `AGENTS.md` `## Nothing lands dead` and `.agents/reachability.md` landed in `8f49ac3be` (#886), after this branch was written, so the answer is derived rather than assumed. It ships NO checker — `8f49ac3be` touches seven files, all documents, and `ls scripts/ | grep -iE 'reach|dead'` returns nothing against a positive control of 37 `check-*` scripts — so there is no exit code to report and the guide says why. **Reached: YES**, and the chain is in the spec with every anchor re-derived at this merge commit and asserted UNIQUE rather than merely present: `include/vllm.h:962` (`vllm_video_generate`) to `src/capi/vllm_c.cpp:1646` to `ltx2_video.cpp:1063` to `:1730`/`:1732` to `ltx2_dit.cpp:785` / `ltx2_device.cpp:1152`, where the flag DEFAULTS TRUE at `ltx2.h:133`, and on to `ltx2_dit.cpp:581` and `:205`/`:140`. The loader half is `ltx2_video.cpp:533` to `:624`. Default configuration, not an opt-in, and as of #658 both shipped DiTs take it with no `allow_unported_modules`. **Entered by the smallest failing test: NO — filed as [#900](https://github.com/mudler/vllm.cpp/issues/900).** The rule's second half is the one that fails, and the reachability mutation says so rather than an argument. Delete the production call site — `ltx2_dit.cpp:140` `if (prompt_mod != nullptr) {` becomes `if (false) {` — and rebuild: BUILT=YES, compile_err=NO, so this is a test result and not a compiler refusal. `test_ltx2`, which calls `Ltx2DitForward` by hand, goes RED at 3 of 43 cases and 6 of 4388 assertions. `test_ltx2_video`, which enters through the ABI, stays GREEN at 37 of 37 and 784 of 784, exit 0 — although its fixture sets the flag TRUE (`ltx2_video_fixture.h:258`) and every render it drives executes the path. It asserts no value the term can move. Filed rather than fixed in flow because closing it needs an ABI-level value oracle designed red-first, which is a row with a spec, not a record repair. The LOADER half passes both halves, and the same harness shows it: M3 re-adds `use_prompt_adaln_single = false` in front of §3.2's guard and takes `test_ltx2_video` to 29 of 37 cases failing, exit 1. ## The numbers were re-measured, not carried forward Every published digit was re-derived on this tree rather than copied. The shipped NVFP4 DiT through `scripts/measure-ltx2-prompt-adaln.py` (`ltx_core` at the pinned `fd4ded7f`, `__file__` asserted, exit 0): video `rms|term|/rms|table|` 1347%, audio 1583%, `max` ratios 7119% / 2817%, `+8.8%` / `+31.3%` on a unit-rms context, and 1275% / 1492% under the file's own LinearQuadratic 8-step sampler. The audio fixture denominator by importing the generator so no weight is re-drawn: 0.0201763 / 0.0496868 = 40.6068%, giving 2817 / 40.61 = 69.4x, against the video 0.0252012 / 0.0487142 = 51.7327% and 7119 / 51.73 = 137.6x. Unchanged, so the headline stands as written. **Two counts DID move, and #658 moved them.** `test_ltx2_video` measures 37 cases / 784 assertions with `LTX2_CHECKPOINT_ROOT` unset and 37 / 9031 with it set, where the row recorded 30 / 502 and 30 / 8734 on 2026-08-13. The figure #673 quotes is therefore 784 of 9031 — 8.7%, not 5.7%. Corrected in the spec, in the index row and at the skip site in `tests/vllm/multimodal/test_ltx2_video.cpp`, each carrying its date, because a bare count in a comment is exactly the thing that goes stale under somebody else's merge. Note also that the CASE count is now IDENTICAL in both configurations, so the "unchanged case count" reading the original note relied on no longer distinguishes them at all. **M5's first run was a silent NO-OP and is recorded rather than dropped.** The harness's substitution for the shift/scale swap spanned two lines and never matched, so the tree was unchanged, the build succeeded, and `test_ltx2` printed 43/43 SUCCESS — a green that says nothing about the guarantee. It was caught only because the harness prints `git diff --stat` for every mutation and that line was missing. Re-applied as a real edit (`1 file changed, 2 insertions(+), 2 deletions(-)`) it is RED at 2 of 43 cases, exit 1. All five mutations are in the spec with their BUILT status, because a mutation that fails to build reads as a passing test. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/issue-index.md | 2 + .agents/specs/ltx25-prompt-adaln.md | 211 ++++++++++++++++++++-- tests/vllm/multimodal/test_ltx2_video.cpp | 12 +- 3 files changed, 204 insertions(+), 21 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 6a9221b8f..1664d02cd 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -221,3 +221,5 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#513](https://github.com/mudler/vllm.cpp/issues/513) | `SPEC-DSPARK` | DFlash/DSpark draft loading hardcodes the VL weight prefix, so a text-only Qwen3.5 target cannot share its embedding | bug | | [#536](https://github.com/mudler/vllm.cpp/issues/536) | `ROAD-V1-C3` | C3's named tail records DSpark as unspiked three days after it landed, and files TLI under the wrong lane | bug | | [#886](https://github.com/mudler/vllm.cpp/issues/886) | `POLICY-NOTHING-LANDS-DEAD` | The seam checkers police where a capability routes and nothing polices whether anything reaches it, so a change can land wired downward and driven only by its own test, as tensor parallelism did (guide [`reachability.md`](reachability.md)) | feature | +| [#673](https://github.com/mudler/vllm.cpp/issues/673) | — | LTX-2.5 prompt-AdaLN: the row's checkpoint-derived evidence is MANUAL and host-local. `LTX2_CHECKPOINT_ROOT` is set by no workflow (`grep -rn CHECKPOINT_ROOT .github/` exits 1 with zero hits, positive control matches in `tests/` and `.agents/`), so CI executes 784 of 9031 assertions — 8.7%, measured 2026-08-15 — of `test_ltx2_video`, at a case count identical in both configurations, and `scripts/measure-ltx2-prompt-adaln.py` is a manual tool no gate invokes. Listed under `## Owed` in [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md) | bug | +| [#900](https://github.com/mudler/vllm.cpp/issues/900) | — | LTX-2.5 prompt-AdaLN is REACHED from `vllm_video_generate` but no test ENTERS through that entry point and observes it: deleting the term's only consumer (`ltx2_dit.cpp:140`) takes `test_ltx2` red at 3 of 43 cases while `test_ltx2_video` stays GREEN at 37 of 37, so the ABI-level gate measures that the pipeline runs rather than that this capability is in it. The second half of `AGENTS.md` `## Nothing lands dead`, whose rule post-dates the row (#886). Listed under `## Owed` in [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md) | bug | diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md index 4ffcf7335..8baf91f23 100644 --- a/.agents/specs/ltx25-prompt-adaln.md +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -204,6 +204,18 @@ second way: a checkpoint whose config declares `use_prompt_adaln_single=false` while its shapes carry the tensors now produces two DIFFERENT contracts and is refused, instead of both sides being forced to the same cleared value. +**SUPERSEDED 2026-08-15 by `LTX25-KEYFRAMES-ABS-POS` +([#658](https://github.com/mudler/vllm.cpp/issues/658), landed as `98f8e046d`).** +`keyframes_abs_pos_embedding` is PORTED, so the "sole flag still cleared" above +is now cleared by nothing: `Ltx2AdoptDeclaredDitParams` resolves the declared +flag against what the file carries instead of force-clearing it, and neither +shipped DiT needs `allow_unported_modules` any more. This paragraph is kept as +the state at the time this row ran; the current disposition is +[`ltx25-keyframes-abs-pos.md`](ltx25-keyframes-abs-pos.md) §2. What is unchanged +is the guard in §3.2, which is what makes the scoping structural — and which is +why the keyframes port could retire the clear without re-opening this row's +defect. + ## 4. Memory format Mirrors the existing L2 parity forward exactly: f32 host, f32/bf16 device stream @@ -428,12 +440,20 @@ measures **30 cases / 8734 assertions**, both before and after this repair (re-measured on this branch, exit 0 in both configurations). Any future quote of this suite's count owes the configuration alongside it. +**Both figures MOVED, and that is the point of quoting the date with them.** +Re-measured 2026-08-15 on this merge commit, after `LTX25-KEYFRAMES-ABS-POS` +(#658) added cases here: `LTX2_CHECKPOINT_ROOT` unset gives **37 cases / 784 +assertions**, set gives **37 / 9031**, exit 0 both ways. The CASE count is the +same in both configurations, so an unchanged case count never distinguishes them. +The 2026-08-13 figures above are kept as what was measured then. + **And CI never sets it — [#673](https://github.com/mudler/vllm.cpp/issues/673), filed 2026-08-13 as visible debt rather than repaired here.** `grep -rn CHECKPOINT_ROOT .github/` exits 1 with zero hits while the same pattern matches in `tests/` and `.agents/` (positive control run in the same command, so this is not -an assertion from a failed grep). CI therefore executes **502 of 8734 assertions — -5.7%** of this suite, at an unchanged case count of 30, and +an assertion from a failed grep). CI therefore executes **784 of 9031 assertions — +8.7%** of this suite (2026-08-15; it was 502 of 8734, 5.7%, on 2026-08-13), at an +identical case count in both configurations, and `scripts/measure-ltx2-prompt-adaln.py` — which produces every shipped-weights number in this Outcome — is a manual tool no gate invokes (`grep -rn measure-ltx2-prompt-adaln` hits only its own usage string and this file). So this @@ -477,10 +497,23 @@ config/shape equality check. the opt-in for it. What changed is that it can no longer switch a ported feature off: the loader asserts the flag against the file instead of clearing it, and `Ltx2AdoptDeclaredDitParams` clears exactly one flag, for a module nothing -applies. +applies. (**SUPERSEDED 2026-08-15**, §3.3: that last module is ported, the opt-in +is needed by neither shipped DiT, and `Ltx2AdoptDeclaredDitParams` clears +nothing.) ### The keyframes claim next door, corrected 2026-08-13 +**READ THE SUPERSESSION FIRST.** Everything below is the state on 2026-08-13, +when `keyframes_abs_pos_embedding` was unported and both shipped DiTs were +refused. `LTX25-KEYFRAMES-ABS-POS` +([#658](https://github.com/mudler/vllm.cpp/issues/658), `98f8e046d`) ported the +module on 2026-08-14, retired both refusals and committed +`scripts/measure-ltx2-keyframes-meta.py`, which re-runs the meta-device +observation below on demand instead of quoting it. The FINDING stands and that +port confirms it; the DISPOSITION it argues for — refuse by tensor presence, keep +the opt-in — is retired. Current disposition: +[`ltx25-keyframes-abs-pos.md`](ltx25-keyframes-abs-pos.md) §2. + `ltx2.h` carried, in the same paragraph this row rewrote, *"LTX-2.5's checkpoint does not carry the parameter"* about `keyframes_abs_pos_embedding`. It is FALSE — the same class of claim as the `use_prompt_adaln_single=false` assertion this row @@ -549,10 +582,21 @@ measured rather than read: `:43`), so **real** zeros would be inert — the mechanism claim fails on `meta`, not on additivity. - This **strengthens** the row's conclusion and changes nothing downstream: the - refusal stays keyed on tensor presence for the FP8 file, and the NVFP4 file is - refused by flag in `ParseLtx2DitParams` (`ltx2.cpp:192-198`) — where upstream, - had it loaded, would carry a parameter its own guard reports as unsupported. + This **strengthens** the row's conclusion and changed nothing downstream on + 2026-08-13: the refusal stayed keyed on tensor presence for the FP8 file, and + the NVFP4 file was refused by flag in `ParseLtx2DitParams` — where upstream, had + it loaded, would carry a parameter its own guard reports as unsupported. Both + refusals are GONE as of `98f8e046d` (#658), and the meta observation above is + what let that row load the NVFP4 file and apply nothing rather than refuse it or + synthesise a zero. It is now re-runnable rather than quoted: + `scripts/measure-ltx2-keyframes-meta.py`, committed by #658 and RE-RUN on this + merge commit against both shipped files, exit 0. It reproduces every line of + the transcript above and carries its own positive control — a neighbour + (`scale_shift_table`) that materialises `device=cpu is_meta=False` on the same + load, so `is_meta=True` on `keyframes_abs_pos_embedding` is the parameter's + state and not a loader that never ran. It also settles the FP8 half by + execution: 2 of 6124 keys match, `[1, 4096]` `F8_E4M3` with 4096 of 4096 bytes + NON-ZERO plus a rank-0 `F32` scale, against `__metadata__ keys : NONE`. It is also not a keyframe-only feature: `transformer_args.py:269` applies it on every `prepare` whose `keyframes_mask` is set, and `tools.py:186-196` sets that @@ -560,15 +604,21 @@ mask unconditionally on the target's first latent frame. (Diffusers' own pipelin does not consume it — `.agents/specs/ltx-2-5.md` §3.1 records that — but `ltx_core` is what this campaign ports, and `ltx_core` does.) -**The refusal keying does NOT change, and that is the decision, not an omission.** -`ltx2_loader.cpp`'s `RefuseUnported` fires on the TENSORS the file carries. Keying -it on the resolved flag instead would, on the FP8 DiT, read a DEFAULT rather than -the file — because that file declares nothing — and would therefore load it -silently while discarding a trained `[1, 4096]` parameter. Tensor presence is the -only signal that file actually carries, and refusing loudly with an opt-in is -strictly safer than resolving quietly. No behaviour changed, so no new gate is -owed; the refusal MESSAGE changed, because it asserted the implication that is -false in both directions. +**The refusal keying did NOT change here, and that was the decision, not an +omission.** `ltx2_loader.cpp`'s `RefuseUnported` fired on the TENSORS the file +carries. Keying it on the resolved flag instead would, on the FP8 DiT, read a +DEFAULT rather than the file — because that file declares nothing — and would +therefore have loaded it silently while discarding a trained `[1, 4096]` +parameter. Tensor presence was the only signal that file actually carries, and +refusing loudly with an opt-in was strictly safer than resolving quietly. No +behaviour changed in this row, so no new gate was owed; the refusal MESSAGE +changed, because it asserted the implication that is false in both directions. + +**RETIRED 2026-08-14 by #658.** Both refusals are gone, because the module is +ported: nothing is keyed on the tensors and nothing is keyed on the flag, and the +declared flag is now RESOLVED against what the file carries +(`Ltx2AdoptDeclaredDitParams`). The reasoning above is why the port had to settle +the NVFP4 arm by EXECUTION before it could retire either refusal. ### The claims repair's own gate (2026-08-13) @@ -608,7 +658,134 @@ tree vanished under it is not a result; it was rebuilt and re-run from scratch. mirror, and it is named here so it is not later mistaken for ported behaviour. - **We refuse `keyframes_abs_pos_embedding` by tensor presence** where upstream would take an out-of-band config's word for it (above). Same shape of - divergence, same reason it stands. + divergence, same reason it stood. **RETIRED 2026-08-14 by #658**: the module is + ported, nothing refuses it, and the declared flag is resolved against the file + the way upstream's own `supports_keyframes_abs_pos_embedding` resolves it. Only + the first divergence survives. + +### The reconciliation onto `main`, and its gate (2026-08-15) + +The branch sat 120 commits behind and conflicted on ten files. Its port half had +already landed as `65e79eee5` (#654), and `98f8e046d` (#658) then ported +`keyframes_abs_pos_embedding`, which is why four of the five SOURCE conflicts were +this branch re-asserting retired refusals. All five took `main`'s side; +`tests/vllm/models/ltx2_goldens.inc` was REGENERATED with the committed command +against `ltx_core` `fd4ded7f` (`REGEN_EXIT=0`) rather than hand-merged, and +differs from `main` by 12 insertions and 7 deletions, all comment lines: every +golden VALUE on both sides byte-identical. + +Every published number was re-derived on the merged tree rather than carried +forward, and none moved: the shipped-weights table above via +`scripts/measure-ltx2-prompt-adaln.py` (exit 0), and the audio fixture +denominator 40.6068% by importing the generator so no weight is re-drawn. + +Suite counts on the merged tree, exit 0 each, against the 2026-08-13 figures: + +| Suite | 2026-08-13 | 2026-08-15 (this merge) | +|---|---|---| +| `test_ltx2` | 35 / 2435 | 43 / 4388 | +| `test_ltx2_loader` | 26 / 4826 | 28 / 4978 | +| `test_ltx2_device` | 15 / 523 | 18 / 546 | +| `test_ltx2_video` | 30 / 502 unset, 30 / 8734 set | 37 / 784 unset, 37 / 9031 set | + +The growth is #658's, not this reconciliation's: no `src/` or `include/` file +differs from `origin/main` on this branch. + +All five mutations re-run on this tree, each BUILT, each restored byte-for-byte: + +| # | Mutation | BUILT | Result | +|---|---|---|---| +| M1 | host `ModulateContext` ignores `prompt_mod` | YES | RED — `test_ltx2` 3/43 cases, 6/4388 assertions, exit 1 | +| M2 | device path takes the static-only branch always | YES | RED — `test_ltx2_device` 1/18, 6/546, exit 1 | +| M3 | re-add `use_prompt_adaln_single = false` before the guard | YES | RED — `test_ltx2_loader` 1/28 cases, exit 1, and `test_ltx2_video` 29/37, exit 1 | +| M4 | prompt AdaLN driven by `m.timesteps` instead of `m.sigma` | YES | RED — `test_ltx2` 2/43, 4/4388, exit 1 | +| M5 | shift and scale rows swapped in the `[2, width]` row | YES | RED — `test_ltx2` 2/43, 4/4388, exit 1 | + +**M5's first attempt was a silent NO-OP and is recorded rather than dropped.** The +harness's substitution spanned two lines and never matched, so the tree was +unchanged, the build succeeded and `test_ltx2` reported 43/43 SUCCESS — a GREEN +that says nothing about the guarantee. It was caught because the harness prints +`git diff --stat` for every mutation and that line was absent. Re-applied as a +real edit (`1 file changed, 2 insertions(+), 2 deletions(-)`) it is RED. An +unmoved mutation is not evidence of anything. + +M3 also shows the doctest counting trap: the case THROWS, so its assertions stop +being counted and the total DROPS 4978 → 4967 while `0 failed` is printed. The +exit code and the `test cases:` line are the authority, not `assertions:`. + +### Reachability, checked 2026-08-15 against `AGENTS.md` `## Nothing lands dead` + +That rule and [`reachability.md`](../reachability.md) post-date this row +(`8f49ac3be`, [#886](https://github.com/mudler/vllm.cpp/issues/886)), so the +question is answered here rather than assumed. The prompt-side AdaLN is REACHED +from a production entry point on its DEFAULT configuration, and the chain is +`file:line` at this merge commit: + +1. `include/vllm.h:962` — `vllm_video_generate`, the shipped C ABI entry point. +2. `src/capi/vllm_c.cpp:1646` — that entry point calls `VideoEngine::Generate` on + the registry-detected engine. +3. `src/vllm/multimodal/ltx2_video.cpp:1063` — `Ltx2VideoEngine::Generate`. +4. `src/vllm/multimodal/ltx2_video.cpp:1730` and `:1732` — the denoise loop calls + `Ltx2DitForwardDevice` or `Ltx2DitForward`. Both arms carry the term. +5. `src/vllm/model_executor/models/ltx2_dit.cpp:785` (host) and + `ltx2_device.cpp:1152` (device) — `prompt_adaln = params.cross_attention_adaln + && params.use_prompt_adaln_single`, and that flag DEFAULTS TRUE at + `include/vllm/model_executor/models/ltx2.h:133`. This is the default + configuration, not an opt-in. +6. `src/vllm/model_executor/models/ltx2_dit.cpp:581` — `PrepareTimestep` runs the + prompt-side MLP on the stream's own `sigma`. +7. `src/vllm/model_executor/models/ltx2_dit.cpp:205` → `:140` — + `ModulateContext` adds that row to the static table, on every block of both + streams. + +The loader half is reached the same way: `Ltx2VideoEngine::Load` +(`src/vllm/multimodal/ltx2_video.cpp:533`) calls `Ltx2LoadDitFromSafetensors` at +`:624`, so §3.2's guard runs for every real checkpoint. Both shipped DiTs take +that path with NO opt-in as of #658. + +Every anchor above was re-derived at this merge commit and asserted UNIQUE — the +quoted text matches exactly once in its file — against a positive control that +reports `STALE` on a deliberately wrong line. `git grep` alone would not have +answered this; the chain was followed by hand. + +**The second half of the rule is NOT satisfied, and the mutation says so.** The +rule also requires that the smallest failing test ENTER through that entry point. +It does not here. The reachability mutation from `reachability.md` — delete the +production call site, `ltx2_dit.cpp:140` `if (prompt_mod != nullptr) {` becomes +`if (false) {`, so `ModulateContext` ignores the term entirely — was run on a +scratch copy and **BUILT=YES, compile_err=NO**, so this is a test result: + +| suite | enters through | result | +|---|---|---| +| `test_ltx2` | `Ltx2DitForward`, by hand | RED — 3 of 43 cases, 6 of 4388 assertions, exit 1 | +| `test_ltx2_video` | `vllm_video_generate` / the ABI | **GREEN** — 37 of 37, 784 of 784, exit 0 | + +The entry-point suite drives the path — its fixture sets the flag TRUE +(`tests/vllm/multimodal/ltx2_video_fixture.h:258`) — and asserts no value the +term can move, so it measures that the pipeline runs rather than that this +capability is in it. Filed as +[#900](https://github.com/mudler/vllm.cpp/issues/900) and listed below rather +than repaired here: closing it means designing an ABI-level value oracle +red-first, which is a row with its own spec and not a record repair. + +The LOADER half does satisfy both halves. Re-adding +`use_prompt_adaln_single = false` in front of §3.2's guard (M3 below) takes +`test_ltx2_video` to 29 of 37 cases failing, exit 1 — that gate enters through the +production load path and observes the guard. + +## Owed + +- [#673](https://github.com/mudler/vllm.cpp/issues/673) — this row's + checkpoint-derived evidence is manual and host-local. `LTX2_CHECKPOINT_ROOT` is + set by no workflow, so the `test_ltx2_video` shipped-header case skips in CI and + `scripts/measure-ltx2-prompt-adaln.py` is a manual tool no gate invokes. Filed + as visible debt; wiring checkpoints into CI is explicitly not in this row. +- [#900](https://github.com/mudler/vllm.cpp/issues/900) — the prompt-AdaLN term is + REACHED from `vllm_video_generate` (chain above) but no test ENTERS through that + entry point and observes it: deleting the term's only consumer leaves + `test_ltx2_video` green at 37 of 37. That is the second half of `AGENTS.md` + `## Nothing lands dead`, whose rule and guide post-date this row (`8f49ac3be`, + #886). Closing it needs an ABI-level value oracle designed red-first. ## Now diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 09e43b2a6..2ca7fff4f 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -1303,10 +1303,14 @@ TEST_CASE("ltx2 video: an ABI client loads, detects and generates through vllm.h // TRAINED `keyframes_abs_pos_embedding` (spec section 3.1) — so the file this // case reads is named here and in every report of its result. // WHEN QUOTING THIS SUITE'S ASSERTION COUNT, QUOTE THE CONFIGURATION WITH IT. -// This case skips by default, and it is most of the suite: unset, the binary -// measures 30 cases / 502 assertions; with LTX2_CHECKPOINT_ROOT set it measures -// 30 / 8734. An "unchanged 30 / 502" across a change therefore says the real -// headers were never read, not that they were read and nothing counted them. +// This case skips by default, and it is most of the suite: with the variable +// UNSET the binary measures 37 cases / 784 assertions, and with it SET, 37 / +// 9031 — measured 2026-08-15, exit 0 both ways. So the CASE count is identical in +// both configurations and only the assertion count moves; an unchanged case count +// across a change therefore says nothing about whether the real headers were read. +// Quote the number WITH its configuration and its date: these figures move +// whenever a case is added here, and they already have (they read 30 / 502 and +// 30 / 8734 before the keyframe-bias port, issue #658). TEST_CASE("ltx2 video: the SHIPPED Lightricks checkpoints parse and load") { const char* root_env = std::getenv("LTX2_CHECKPOINT_ROOT"); if (root_env == nullptr) { From 7a6165dab9d2ee4ebd2618275d2ec7ffcf1371c2 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 13:14:29 +0000 Subject: [PATCH 06/10] fix(LTX25-ADALN-CLAIMS): two spec anchors into test_ltx2_video.cpp were dated but never re-derived (#644) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review of #663 returned FAIL on two `file:line` citations in `.agents/specs/ltx25-prompt-adaln.md`. Both pointed into `tests/vllm/multimodal/test_ltx2_video.cpp`, and both landed inside an unrelated CRF-refusal `SUBCASE` rather than the text they claim to cite. `:917-921` for the shipped-checkpoint skip-and-return becomes `:1316-1319`, and `:913-914` for the "differ in a TRAINED `keyframes_abs_pos_embedding`" comment becomes `:1302-1303`. Why they rotted is the point. Both were correct when written at `2990ed81b`. The row then re-measured its suite counts and dated them, but re-derived none of its own anchors into `test_ltx2_video.cpp` — the one file whose line numbers #658 moved most. `98f8e046d` inserted 169 lines above the cited text and this branch added 9 more, so the citations drifted 389 lines while the row was auditing the numbers next to them. Dating a count is not re-deriving an anchor; anchors here rot inside a single pull request, and these two did. The skip-and-return is also not uniquely identifiable by its text: the block at `:1316-1319` is byte-identical to the one at `:1451-1454` in `ltx2 video: the SHIPPED Lightricks VAEs and upsampler load`. A future re-derivation that greps for it gets two hits and cannot choose. The sentence therefore now names the enclosing case, so the anchor can be re-derived rather than only trusted. Verified at this tree with a uniqueness assertion, not an existence check: the cited comment occurs exactly once in the file, and the cited skip block sits inside the named case, which is itself declared exactly once. Four positive controls — the two old anchors, the identical VAE-case skip block, and an off-by-two span — each report STALE, so the clean result is a measurement and not silence. Documentation only. No `src/`, `include/`, `tests/` or record file is touched. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-prompt-adaln.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md index 8baf91f23..be8578d32 100644 --- a/.agents/specs/ltx25-prompt-adaln.md +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -432,8 +432,9 @@ re-checking its md5 (`03324d42…`, identical before and after): | `test_ltx2_video` | 30 / 502 | 30 / 502 | unchanged — see below; 502 is the SKIPPED default | **What `30 / 502` does and does not say (corrected 2026-08-13).** The -shipped-checkpoint case is env-gated: with `LTX2_CHECKPOINT_ROOT` unset it prints -`SKIPPED` and returns at `test_ltx2_video.cpp:917-921`, so `30 / 502` means the +shipped-checkpoint case `ltx2 video: the SHIPPED Lightricks checkpoints parse and +load` is env-gated: with `LTX2_CHECKPOINT_ROOT` unset it prints +`SKIPPED` and returns at `test_ltx2_video.cpp:1316-1319`, so `30 / 502` means the whole real-header case DID NOT RUN — not "it ran and no assertion counted the module". With the variable pointing at the Lightricks tree the same binary measures **30 cases / 8734 assertions**, both before and after this repair @@ -518,7 +519,7 @@ the opt-in — is retired. Current disposition: does not carry the parameter"* about `keyframes_abs_pos_embedding`. It is FALSE — the same class of claim as the `use_prompt_adaln_single=false` assertion this row exists to remove — and the tree already contradicted it twice -(`.agents/model-matrix.md`, `tests/vllm/multimodal/test_ltx2_video.cpp:913-914`). +(`.agents/model-matrix.md`, `tests/vllm/multimodal/test_ltx2_video.cpp:1302-1303`). Read straight off both files' headers, and run through upstream's own loader and configurator: From 00613767da5187d44c138d3d9958783607febf57 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 13:31:51 +0000 Subject: [PATCH 07/10] record(LTX25-ADALN-CLAIMS): five more spec anchors that moved after they were derived (#644) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The whole of `.agents/specs/ltx25-prompt-adaln.md` is branch-local — it does not exist on `main` — so every citation in it arrives with this pull request and gets no second chance to be right. Five anchors were verified correct at `baa92ccf7`, then moved out from under the prose by this row's own implementation commit `020381676` and by `98f8e046d` (#658) on top of it. That is exactly the window in which a citation goes wrong without anyone reading it again. Three are present-tense claims about the current tree and take a live line number: the modulation-order trap in `ProcessOutput` (`ltx2_dit.cpp:530-540` → `:640-642`), the prompt-K/V cache refusal in `Ltx2DitForward` (`:672-676` → `:775-780`), and the `use_keyframes_abs_pos_embedding` clear in `Ltx2AdoptDeclaredDitParams` (`ltx2_loader.cpp:979-984` → `:1033-1035`). Each now resolves to the text its sentence quotes, and to nothing else in its file. Two sit in section 0, "what is wrong today", and describe a state this row changed. They are decided separately rather than uniformly. `ltx2_loader.cpp:988` quoted `declared.use_prompt_adaln_single = false;`, which this row DELETED; no live line can carry it, so it is SHA-anchored `@ baa92ccf7`, where that assignment and the two manifest-path assignments the next sentence cites as `:573` / `:626` are all still present and the live file has none of the three. `ltx2_video.cpp:570` quoted the `allow_unported_modules` read, which still exists and merely moved, so it takes the live `:617`. A SHA anchor is the honest fix only when the cited text is gone. Reaching for one to avoid re-deriving a line that still exists would hide the anchor from every later sweep. Every anchor in the file was re-derived at this head and asserted UNIQUE — the quoted text matches exactly once in the cited file and the match starts at the cited line — against a positive control that reports STALE on a deliberately wrong line, so a clean result is not silence. 32 repo-local citations were checked: 26 distinct full-form anchors plus 5 bare `:NN` continuations. The 34 `diffusers` and `ltx-core` anchors are out of scope here. One repo-local citation is STALE and is deliberately NOT repaired, because this change carries authority for the five above and no more: `ltx2.cpp:274-276`, cited three times as the `VT_CHECK` that refuses the prompt-AdaLN tensors by name. It was exactly those three lines at `baa92ccf7`, this row deleted it (section 2 item 5 says so), and the live lines are the unrelated `keyframes_abs_pos_embedding` enumeration. Same defect class as `ltx2_loader.cpp:988`, and it wants the same SHA anchor. Recorded here rather than left for a later reader to rediscover. `ltx2_video.cpp:617` is the one anchor sensitive to a `main` merge: this branch adds nothing to `src/`, and `origin/main` has since moved that line to 660. It must be re-derived on the merged tree, not carried forward. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-prompt-adaln.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md index be8578d32..150954410 100644 --- a/.agents/specs/ltx25-prompt-adaln.md +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -16,7 +16,8 @@ IMPORTS and EXECUTES the LTX-2 modules rather than restating them. ## 0. What is wrong today -`src/vllm/model_executor/models/ltx2_loader.cpp:988` sets, unconditionally: +`src/vllm/model_executor/models/ltx2_loader.cpp:988 @ baa92ccf7` sets, +unconditionally: ```cpp declared.use_prompt_adaln_single = false; @@ -36,7 +37,7 @@ The shipped FP8 DiT carries the 18 tensors the flag builds (12 `tests/vllm/models/ltx2_fp8_dit_manifest.inc:232-240,286-294`), so the flag is TRUE for the checkpoint this campaign renders. `ltx2.cpp:274-276` refuses those tensors by name, so a real render needs `allow_unported_modules=1` -(`src/vllm/multimodal/ltx2_video.cpp:570`) — which reaches the loader lines above +(`src/vllm/multimodal/ltx2_video.cpp:617`) — which reaches the loader lines above and **silently clears the flag**. Net effect: every render drops the timestep-conditioned half of the prompt K/V @@ -168,12 +169,12 @@ golden valid. The **order of the two additions** is upstream's: the table and the timestep row are summed FIRST (`:443`), and only then does `(1 + scale)` apply. Folding it the other way would round differently — the same trap `ProcessOutput` -(`ltx2_dit.cpp:530-540`) already documents. +(`ltx2_dit.cpp:640-642`) already documents. ### 3.2 The prompt-K/V cache, and what replaces the cleared flag `Ltx2DitForward` already refuses a cache when the flag is on -(`ltx2_dit.cpp:672-676`); with the flag no longer cleared, that refusal becomes +(`ltx2_dit.cpp:775-780`); with the flag no longer cleared, that refusal becomes *reachable* rather than dead, and it is correct: the K/V now carry a timestep term. Nothing in the shipped pipeline passes a cache (`grep` over `ltx2_pipeline.cpp` finds none), so no caller regresses. @@ -195,7 +196,7 @@ and setting it with the tensors absent would bind missing weights. After this row it is scoped to genuinely-unported modules only: the sole flag it still clears in a config copy is `use_keyframes_abs_pos_embedding` -(`ltx2_loader.cpp:979-984`), whose module really is unported. The guard in §3.2 +(`ltx2_loader.cpp:1033-1035`), whose module really is unported. The guard in §3.2 is what makes that scoping structural rather than a comment — the extra `= false` cannot come back without going red. From 8f8997e341e983dba2d5beeecf047a288f35da9b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 13:46:51 +0000 Subject: [PATCH 08/10] record(LTX25-ADALN-CLAIMS): SHA-anchor the deleted refusal, re-derive all 33 citations, and file the class (#644, #911) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three things, all in one spec plus one append-only index row. **The refusal that no live line can carry.** `ltx2.cpp:274-276` was cited three times (§0, §1.1, §2) as the `VT_CHECK` refusing the prompt-AdaLN tensors by name. This row DELETED it — §2 item 5 says so — so the citation was not merely stale: live `274-276` is now an unrelated `keyframes_abs_pos_embedding` enumeration, which contradicts the sentence carrying it. All three are now `ltx2.cpp:274-276 @ baa92ccf7`, the same disposition §0 already uses for `ltx2_loader.cpp:988`. Proven both ways with positive controls: at `baa92ccf7`, `VT_CHECK(!p.use_prompt_adaln_single` matches once at `:274` and `not ported in phase L2` at `:276`; in the live tree both grep exit 1 across `src include tests`, while the control term `PushAdaLayerNormSingle` matches 7 times at that SHA and 9 today. A null grep on its own would only have proven the search terms wrong. **Seven anchors the merge moved.** The preceding merge commit explains the mechanism; this is the repair. `ltx2_video.cpp:617 → :660`, `:1063 → :1106`, `:1730/:1732 → :1784/:1786`, `:533 → :575`, `:624 → :667`, and in the test `:1316-1319 → :1620-1625`, `:1302-1303 → :1608-1609`. Every one was correct at `00613767d`. The env-gate anchor gained a line rather than just an offset. Its four gate lines are byte-identical to the gate of `ltx2 video: the SHIPPED Lightricks VAEs and upsampler load` at `:1757-1760`, so an anchor on the gate alone names two places and identifies neither — an ambiguity that was already there at `:1316-1319`. Starting the span on the `TEST_CASE` line, which occurs exactly once, is what makes it resolvable. **All 33 repo-local citations re-derived**, not only the ones known to be wrong: 27 live, 4 SHA-anchored occurrences, 2 bare continuations of the SHA-anchored loader claim. All FRESH. The expected text comes from the CLAIM, never from the file — a checker that reads the span out of the file it is validating is a tautology and reported 27 of 27 fresh on the same tree where five anchors pointed at a `VT_CHECK` message string and a comment about mask pointers. Armed, not decorative: shifting four anchors by one line gives 6 STALE of 33. `ltx2_loader.cpp:573`/`:626` is the deliberate exception. Both lines really do read `out.params.use_prompt_adaln_single = false;`, which is what §0 claims, so uniqueness there is asserted as "exactly these two lines". **One claim corrected while its anchor was being re-derived.** §0 said the FP8 DiT carries "12 `prompt_adaln_single.*`, 6 `audio_prompt_adaln_single.*`". It is 9 and 9 — `grep -c` returns 9 per family and each cited span is nine lines — 6 parameters plus 3 FP8 weight scales per stream. The TOTAL of 18 was always right and no conclusion drawn from it moves; the breakdown was not, and a citation whose own span contradicts its sentence is the defect this repair exists to remove. **Filed rather than swept: #911.** This spec shipped eight stale repo-local anchors across two earlier repair commits, every one introduced by its own commits and moved by its own `020381676` and by `98f8e046d` (#658), then seven more from the merge alone. Spec BODIES are checked by nothing: `check-agent-record.py`'s `MATRIX_PATHS` covers the five matrices, `feature-matrix.md` and `specs/model-family-inventory.md`, leaving 4772 line-carrying citations across 315 `.agents/specs/*.md` unexamined (positive control: 2314 line-less `.cpp` mentions match the same shape, so the count is not a grep artefact). The rule is one sentence — an anchor into a file the row is editing is stale until re-derived at the tree that LANDS, merge included — and both dispositions are already in use here unwritten. Narrower on surface than #632 and sharper on mechanism, so filed beside it rather than folded into it. A repo-wide sweep and a checker change each need their own spec and red-before evidence, which is not this PR. No `src/`, `include/` or `tests/` file changes. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 1 + .agents/specs/ltx25-prompt-adaln.md | 110 ++++++++++++++++++++++++---- 2 files changed, 95 insertions(+), 16 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 3987f41f9..558b9a921 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -233,3 +233,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#896](https://github.com/mudler/vllm.cpp/issues/896) | — | No gate refuses a new unchecked `static_cast<...LoadedModel&>` in a registry entry point. The #847 sweep decided the checker is warranted — the class is exactly grep-able, unlike the unaligned-read class in #627, and it regrows the moment a new model port copies its neighbour — and deliberately did not bundle a repository-wide refusal gate into a 30-file mechanical sweep. Listed under `## Owed` in [`registry-downcast-sweep.md`](specs/registry-downcast-sweep.md) §6 | feature | | [#897](https://github.com/mudler/vllm.cpp/issues/897) | — | `ModelAs`'s `const LoadedModel&` overload has no caller anywhere in the tree: every registered `prepare`/`forward` takes a NON-const `LoadedModel&`, so all 35 call sites select the non-const overload and its `Model&` result merely binds to the `const auto&`. Proven by deleting the overload and compiling all 30 swept TUs `-fsyntax-only`. It is dead in the way that reads as covered — #847 cited its existence as the reason the 14 `const` sites needed no thought. Listed under `## Owed` in [`registry-downcast-sweep.md`](specs/registry-downcast-sweep.md) §3.3 | bug | | [#904](https://github.com/mudler/vllm.cpp/issues/904) | `ROAD-V1-LTX25` | `main` red on `sanitize-cpu (address,undefined)` for a THIRD reason after the two [#730](https://github.com/mudler/vllm.cpp/issues/730) enumerates, both of which are now closed: `~Ltx2DitDeviceWeights` frees the staged DiT buffers on the main thread (`ltx2_device.cpp:1088`) while a `vt::cpu` threadpool worker is still reading one inside `AddKernel` (`cpu_layernorm.cpp:33`), so the staged weights' lifetime is not joined to the in-flight parallel op that reads them. Deterministic, 5 runs / 5 aborts on unmodified `e8048ef63`; the rest of the suite is 477/478 and `test_ltx2_video` now PASSES | bug | +| [#911](https://github.com/mudler/vllm.cpp/issues/911) | — | A `file:line` anchor into a file the row is ITSELF editing is stale by default, and spec BODIES are checked by nothing: `check-agent-record.py`'s `MATRIX_PATHS` (`:521`, `:529-530`) covers the five matrices, `feature-matrix.md` and `specs/model-family-inventory.md`, leaving 4772 line-carrying citations across 315 `.agents/specs/*.md` unexamined (positive control: 2314 line-less `.cpp` mentions match the same shape). `ltx25-prompt-adaln.md` shipped EIGHT stale repo-local anchors across two repair commits, moved by its own `020381676` and by `98f8e046d` (#658), then SEVEN more that were correct at `00613767d` and wrong at the merge of `origin/main`, because `0785cfc4d` (#882) added 70 lines to `ltx2_video.cpp` and 306 to `test_ltx2_video.cpp` ahead of every one. The obvious checker is a TAUTOLOGY — reading the span out of the file it validates reports 27 of 27 fresh on the same tree where reading the spans against their CLAIMS finds seven stale. Remedy is already in use and unwritten: `path:NN @ ` for a historical claim, claim-sourced uniqueness re-derivation for a live one, re-run after the merge. Narrower than [#632](https://github.com/mudler/vllm.cpp/issues/632) on surface and sharper on mechanism. Listed under `## Owed` in [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md) | bug | diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md index 150954410..c8f9bb854 100644 --- a/.agents/specs/ltx25-prompt-adaln.md +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -32,14 +32,24 @@ and `:573` / `:626` do the same on the two manifest paths. The flag defaults default, and `model_configurator.py:76` / `:138` read it as `config.get("use_prompt_adaln_single", True)` -The shipped FP8 DiT carries the 18 tensors the flag builds (12 -`prompt_adaln_single.*`, 6 `audio_prompt_adaln_single.*`; see -`tests/vllm/models/ltx2_fp8_dit_manifest.inc:232-240,286-294`), so the flag is -TRUE for the checkpoint this campaign renders. `ltx2.cpp:274-276` refuses those -tensors by name, so a real render needs `allow_unported_modules=1` -(`src/vllm/multimodal/ltx2_video.cpp:617`) — which reaches the loader lines above +The shipped FP8 DiT carries the 18 tensors the flag builds — **9 +`audio_prompt_adaln_single.*` at +`tests/vllm/models/ltx2_fp8_dit_manifest.inc:232-240` and 9 +`prompt_adaln_single.*` at `:286-294`**, each stream contributing 6 parameters +(`linear_1`, `linear_2`, `linear`, weight and bias) plus the 3 `F32` +`weight_scale` entries the FP8 file carries beside them. So the flag is TRUE for +the checkpoint this campaign renders. `ltx2.cpp:274-276 @ baa92ccf7` refuses +those tensors by name, so a real render needs `allow_unported_modules=1` +(`src/vllm/multimodal/ltx2_video.cpp:660`) — which reaches the loader lines above and **silently clears the flag**. +(The split read `12` / `6` until 2026-08-15. It is `9` / `9`: `grep -c` over the +two families in that file returns 9 apiece, and the two cited spans are nine +lines each. The TOTAL of 18 was always right, and so was every conclusion drawn +from it; the breakdown was not. Corrected here because the anchor beside it was +being re-derived anyway, and a citation whose own span contradicts the sentence +that carries it is the defect this repair exists to remove.) + Net effect: every render drops the timestep-conditioned half of the prompt K/V modulation, keeping only the static `prompt_scale_shift_table`. Nothing observes it: shapes are unchanged, values stay finite, and the goldens were generated with @@ -73,8 +83,8 @@ not `adaln_embedding_coefficient()` — shift and scale for the K/V only. diffusers twin: `transformer_ltx2.py:1255-1259`, `num_mod_params=2`. Registration order inside `_init_video` puts it between `adaln_single` and -`proj_out`, which is where `EnumerateLtx2DitTensors` already reserves its slot -(the `VT_CHECK` at `ltx2.cpp:274-276`). +`proj_out`, which is where `EnumerateLtx2DitTensors` already reserved its slot +(the `VT_CHECK` at `ltx2.cpp:274-276 @ baa92ccf7`). ### 1.2 The producer @@ -145,7 +155,7 @@ Layout consequence: the flat `[B, 1, 2 * dim]` row is read as `[2, dim]` with (`ltx2_dit.cpp`) and device (`ltx2_device.cpp`) paths. 4. The three loader `= false` assignments are deleted, and replaced by a guard (§3.2) that makes a future silent clear impossible. -5. The `ltx2.cpp:274-276` refusal is deleted for these two families. +5. The `ltx2.cpp:274-276 @ baa92ccf7` refusal is deleted for these two families. 6. Goldens executed from upstream at reduced dims, with a mutation proving the new term is load-bearing, and a measured magnitude. @@ -435,9 +445,16 @@ re-checking its md5 (`03324d42…`, identical before and after): **What `30 / 502` does and does not say (corrected 2026-08-13).** The shipped-checkpoint case `ltx2 video: the SHIPPED Lightricks checkpoints parse and load` is env-gated: with `LTX2_CHECKPOINT_ROOT` unset it prints -`SKIPPED` and returns at `test_ltx2_video.cpp:1316-1319`, so `30 / 502` means the +`SKIPPED` and returns at `test_ltx2_video.cpp:1620-1625`, so `30 / 502` means the whole real-header case DID NOT RUN — not "it ran and no assertion counted the -module". With the variable pointing at the Lightricks tree the same binary +module". (The span starts on the `TEST_CASE` line deliberately. The four gate +lines alone are **not unique** — `ltx2 video: the SHIPPED Lightricks VAEs and +upsampler load` carries a byte-identical `LTX2_CHECKPOINT_ROOT` gate at +`:1757-1760` — so an anchor on the gate alone names two places and identifies +neither. That ambiguity was already there when the anchor read `:1316-1319`; +widening by one line to the case name, which occurs exactly once, is what makes +the citation resolvable.) With the variable pointing at the Lightricks tree the +same binary measures **30 cases / 8734 assertions**, both before and after this repair (re-measured on this branch, exit 0 in both configurations). Any future quote of this suite's count owes the configuration alongside it. @@ -520,7 +537,7 @@ the opt-in — is retired. Current disposition: does not carry the parameter"* about `keyframes_abs_pos_embedding`. It is FALSE — the same class of claim as the `use_prompt_adaln_single=false` assertion this row exists to remove — and the tree already contradicted it twice -(`.agents/model-matrix.md`, `tests/vllm/multimodal/test_ltx2_video.cpp:1302-1303`). +(`.agents/model-matrix.md`, `tests/vllm/multimodal/test_ltx2_video.cpp:1608-1609`). Read straight off both files' headers, and run through upstream's own loader and configurator: @@ -726,8 +743,8 @@ from a production entry point on its DEFAULT configuration, and the chain is 1. `include/vllm.h:962` — `vllm_video_generate`, the shipped C ABI entry point. 2. `src/capi/vllm_c.cpp:1646` — that entry point calls `VideoEngine::Generate` on the registry-detected engine. -3. `src/vllm/multimodal/ltx2_video.cpp:1063` — `Ltx2VideoEngine::Generate`. -4. `src/vllm/multimodal/ltx2_video.cpp:1730` and `:1732` — the denoise loop calls +3. `src/vllm/multimodal/ltx2_video.cpp:1106` — `Ltx2VideoEngine::Generate`. +4. `src/vllm/multimodal/ltx2_video.cpp:1784` and `:1786` — the denoise loop calls `Ltx2DitForwardDevice` or `Ltx2DitForward`. Both arms carry the term. 5. `src/vllm/model_executor/models/ltx2_dit.cpp:785` (host) and `ltx2_device.cpp:1152` (device) — `prompt_adaln = params.cross_attention_adaln @@ -741,8 +758,8 @@ from a production entry point on its DEFAULT configuration, and the chain is streams. The loader half is reached the same way: `Ltx2VideoEngine::Load` -(`src/vllm/multimodal/ltx2_video.cpp:533`) calls `Ltx2LoadDitFromSafetensors` at -`:624`, so §3.2's guard runs for every real checkpoint. Both shipped DiTs take +(`src/vllm/multimodal/ltx2_video.cpp:575`) calls `Ltx2LoadDitFromSafetensors` at +`:667`, so §3.2's guard runs for every real checkpoint. Both shipped DiTs take that path with NO opt-in as of #658. Every anchor above was re-derived at this merge commit and asserted UNIQUE — the @@ -750,6 +767,51 @@ quoted text matches exactly once in its file — against a positive control that reports `STALE` on a deliberately wrong line. `git grep` alone would not have answered this; the chain was followed by hand. +**All seven `ltx2_video.cpp` / `test_ltx2_video.cpp` anchors in this section +MOVED under the 2026-08-15 merge of `origin/main`, and the numbers above are the +POST-merge ones.** `0785cfc4d` (#882) added 70 lines to `ltx2_video.cpp` and 306 +to `test_ltx2_video.cpp`, ahead of every anchor here: `:1063 → :1106`, +`:1730/:1732 → :1784/:1786`, `:533 → :575`, `:624 → :667`, and in the test +`:1316-1319 → :1622-1625`, `:1302-1303 → :1608-1609`. Each was CORRECT at +`00613767d` and each was WRONG the moment the merge landed, which is the point: +a re-derivation is only true of the tree it ran on, and the merge is part of +landing. They were caught by comparing each span's TEXT against the claim beside +it. A checker that reads the span out of the file and then looks for that span +in the same file is a tautology — it returns unique-and-at-the-cited-line for +every anchor, including the ones now pointing at unrelated code. The expected +text has to come from the CLAIM. + +### Every repo-local citation in this file, re-derived (2026-08-15) + +Not only this section's. **33 repo-local citations** were re-derived at the +merge commit that lands: 27 live (25 full-form plus the `:1786`, `:140` and +`:667` bare continuations), 4 SHA-anchored occurrences into `baa92ccf7` +(`ltx2_loader.cpp:988` once, `ltx2.cpp:274-276` three times) and 2 bare +continuations of the SHA-anchored loader claim (`:573`, `:626`). All 33 FRESH. + +The rule applied: the expected text is taken from the CLAIM, then required to +occur exactly once in the cited file at the cited revision, beginning on the +cited line. Two consequences worth stating, because each is a finding rather +than a formality: + +- `ltx2_loader.cpp:573` and `:626` are the one citation whose text is + DELIBERATELY not unique — both lines read + `out.params.use_prompt_adaln_single = false;`, which is exactly what §0 claims + ("`:573` / `:626` do the same"). Uniqueness is asserted as "exactly these two + lines", not "exactly one". +- The env-gate anchor was widened from the four gate lines to `:1620-1625` so + that it resolves at all; see the note in `### The gate`. + +The check is armed, not decorative: shifting four anchors by one line +(`ltx2_video.cpp:660`, `:1106`, `ltx2.h:133`, `ltx2.cpp:274-276`) takes it to +6 STALE of 33 — six because the `ltx2.cpp` anchor is cited three times, which is +also why SHA-anchoring it once fixed three sentences. + +Upstream citations (`model.py`, `transformer_args.py`, `transformer.py`, +`adaln.py`, `transformer_ltx2.py`, and the rest) are NOT covered by that run. +They are pinned to `fd4ded7f` / `3a2f35d4` and audited separately under +[#794](https://github.com/mudler/vllm.cpp/issues/794). + **The second half of the rule is NOT satisfied, and the mutation says so.** The rule also requires that the smallest failing test ENTER through that entry point. It does not here. The reachability mutation from `reachability.md` — delete the @@ -788,6 +850,22 @@ production load path and observes the guard. `test_ltx2_video` green at 37 of 37. That is the second half of `AGENTS.md` `## Nothing lands dead`, whose rule and guide post-date this row (`8f49ac3be`, #886). Closing it needs an ABI-level value oracle designed red-first. +- [#911](https://github.com/mudler/vllm.cpp/issues/911) — this spec is the worked + example for a class, not an outlier. It shipped EIGHT stale repo-local anchors + across two repair commits (`7a6165dab`, `00613767d`), every one introduced by + the row's own commits and moved by its own `020381676` and by `98f8e046d` + (#658), and then SEVEN more that were correct at `00613767d` and wrong at the + merge of `origin/main` in this commit. Spec BODIES are checked by nothing: + `check-agent-record.py`'s `MATRIX_PATHS` (`:521`, `:529-530`) covers the five + matrices, `feature-matrix.md` and `specs/model-family-inventory.md`, so 4772 + line-carrying citations across 315 `.agents/specs/*.md` are unexamined. The + rule the issue asks for is one sentence — an anchor into a file the row is + editing is stale until re-derived at the tree that LANDS, merge included — and + the two dispositions are already in use here unwritten: `path:NN @ ` for a + historical claim, claim-sourced uniqueness re-derivation for a live one. Filed + rather than fixed: a repo-wide sweep and a checker change each need their own + spec and red-before evidence, and #632's `ENG-RECORD-ANCHOR-RATCHET` is the row + that should absorb it. ## Now From 7bf2a92973813b08fc30ad4fd9d3abb03b3ba099 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 13:52:02 +0000 Subject: [PATCH 09/10] record(LTX25-ADALN-CLAIMS): move the re-derivation record out of the middle of the reachability argument (#644) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three editorial defects in the preceding commit, found by reading its own diff. The re-derivation subsection was inserted between the reachability chain and the paragraph that qualifies it, so `**The second half of the rule is NOT satisfied**` — which is about the chain above it — read as though it were about citation freshness. Moved to the end of `## Outcome`, immediately before `## Owed`, where the thing it qualifies is the whole file rather than one section. The migration arrow said `:1316-1319 → :1622-1625` while the anchor two hundred lines earlier reads `:1620-1625`, because the widening for uniqueness was decided after that sentence was written. An anchor repair that ships two different numbers for the same anchor is the failure it is repairing, so the arrow now names `:1620-1625` and says why it gained a line. A reflow left `same binary` alone on its own line. All 33 citations re-derived again after the move — the move changes spec line numbers, not anchors, but "it cannot have changed anything" is the reasoning this row exists to distrust. 33 FRESH, and the armed control still reports 6 STALE of 33. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-prompt-adaln.md | 56 ++++++++++++++--------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md index c8f9bb854..522132321 100644 --- a/.agents/specs/ltx25-prompt-adaln.md +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -454,8 +454,7 @@ upsampler load` carries a byte-identical `LTX2_CHECKPOINT_ROOT` gate at neither. That ambiguity was already there when the anchor read `:1316-1319`; widening by one line to the case name, which occurs exactly once, is what makes the citation resolvable.) With the variable pointing at the Lightricks tree the -same binary -measures **30 cases / 8734 assertions**, both before and after this repair +same binary measures **30 cases / 8734 assertions**, both before and after this repair (re-measured on this branch, exit 0 in both configurations). Any future quote of this suite's count owes the configuration alongside it. @@ -772,7 +771,8 @@ MOVED under the 2026-08-15 merge of `origin/main`, and the numbers above are the POST-merge ones.** `0785cfc4d` (#882) added 70 lines to `ltx2_video.cpp` and 306 to `test_ltx2_video.cpp`, ahead of every anchor here: `:1063 → :1106`, `:1730/:1732 → :1784/:1786`, `:533 → :575`, `:624 → :667`, and in the test -`:1316-1319 → :1622-1625`, `:1302-1303 → :1608-1609`. Each was CORRECT at +`:1316-1319 → :1620-1625` (widened by one line as well; see `### The gate`) and +`:1302-1303 → :1608-1609`. Each was CORRECT at `00613767d` and each was WRONG the moment the merge landed, which is the point: a re-derivation is only true of the tree it ran on, and the merge is part of landing. They were caught by comparing each span's TEXT against the claim beside @@ -781,6 +781,31 @@ in the same file is a tautology — it returns unique-and-at-the-cited-line for every anchor, including the ones now pointing at unrelated code. The expected text has to come from the CLAIM. +**The second half of the rule is NOT satisfied, and the mutation says so.** The +rule also requires that the smallest failing test ENTER through that entry point. +It does not here. The reachability mutation from `reachability.md` — delete the +production call site, `ltx2_dit.cpp:140` `if (prompt_mod != nullptr) {` becomes +`if (false) {`, so `ModulateContext` ignores the term entirely — was run on a +scratch copy and **BUILT=YES, compile_err=NO**, so this is a test result: + +| suite | enters through | result | +|---|---|---| +| `test_ltx2` | `Ltx2DitForward`, by hand | RED — 3 of 43 cases, 6 of 4388 assertions, exit 1 | +| `test_ltx2_video` | `vllm_video_generate` / the ABI | **GREEN** — 37 of 37, 784 of 784, exit 0 | + +The entry-point suite drives the path — its fixture sets the flag TRUE +(`tests/vllm/multimodal/ltx2_video_fixture.h:258`) — and asserts no value the +term can move, so it measures that the pipeline runs rather than that this +capability is in it. Filed as +[#900](https://github.com/mudler/vllm.cpp/issues/900) and listed below rather +than repaired here: closing it means designing an ABI-level value oracle +red-first, which is a row with its own spec and not a record repair. + +The LOADER half does satisfy both halves. Re-adding +`use_prompt_adaln_single = false` in front of §3.2's guard (M3 below) takes +`test_ltx2_video` to 29 of 37 cases failing, exit 1 — that gate enters through the +production load path and observes the guard. + ### Every repo-local citation in this file, re-derived (2026-08-15) Not only this section's. **33 repo-local citations** were re-derived at the @@ -812,31 +837,6 @@ Upstream citations (`model.py`, `transformer_args.py`, `transformer.py`, They are pinned to `fd4ded7f` / `3a2f35d4` and audited separately under [#794](https://github.com/mudler/vllm.cpp/issues/794). -**The second half of the rule is NOT satisfied, and the mutation says so.** The -rule also requires that the smallest failing test ENTER through that entry point. -It does not here. The reachability mutation from `reachability.md` — delete the -production call site, `ltx2_dit.cpp:140` `if (prompt_mod != nullptr) {` becomes -`if (false) {`, so `ModulateContext` ignores the term entirely — was run on a -scratch copy and **BUILT=YES, compile_err=NO**, so this is a test result: - -| suite | enters through | result | -|---|---|---| -| `test_ltx2` | `Ltx2DitForward`, by hand | RED — 3 of 43 cases, 6 of 4388 assertions, exit 1 | -| `test_ltx2_video` | `vllm_video_generate` / the ABI | **GREEN** — 37 of 37, 784 of 784, exit 0 | - -The entry-point suite drives the path — its fixture sets the flag TRUE -(`tests/vllm/multimodal/ltx2_video_fixture.h:258`) — and asserts no value the -term can move, so it measures that the pipeline runs rather than that this -capability is in it. Filed as -[#900](https://github.com/mudler/vllm.cpp/issues/900) and listed below rather -than repaired here: closing it means designing an ABI-level value oracle -red-first, which is a row with its own spec and not a record repair. - -The LOADER half does satisfy both halves. Re-adding -`use_prompt_adaln_single = false` in front of §3.2's guard (M3 below) takes -`test_ltx2_video` to 29 of 37 cases failing, exit 1 — that gate enters through the -production load path and observes the guard. - ## Owed - [#673](https://github.com/mudler/vllm.cpp/issues/673) — this row's From 25cba74915d4bc57a3db9e42e7e714996d675049 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 14:06:55 +0000 Subject: [PATCH 10/10] record(LTX25-ADALN-CLAIMS): the test_ltx2_video counts this merge invalidated, marked UNKNOWN rather than carried (#644, #673) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The merge that repairs seven anchors also invalidates a count, and the count was about to ride along unexamined — which is the same defect in its other form. An anchor and a measurement both name a tree, and this row's rule is that neither survives a merge on the argument that it probably did not move. `0785cfc4d` (#882) added 306 lines to `tests/vllm/multimodal/test_ltx2_video.cpp` between `00613767d` and this branch's tip, taking it from 37 to 40 `TEST_CASE`s. The Outcome quoted 37 / 784 with `LTX2_CHECKPOINT_ROOT` unset and 37 / 9031 with it set, dated to the PREVIOUS merge, in three places. What is measured here is the SOURCE count, 37 to 40, by `grep -c '^TEST_CASE'` on both revisions with the same grep returning matches on `origin/main` as its control. The doctest CASE and ASSERTION totals were NOT re-run, so they are recorded as UNKNOWN rather than adjusted, guessed, or left standing. Marking a number superseded is cheap; inventing its replacement is not available, because the `set` arm needs the 18.72 GB NVFP4 and 23 GB FP8 DiTs under `$CHECKPOINT_ROOT` and no CI host has them. That debt is #673's, where it already lives. Two claims that DO survive, checked rather than assumed. The #673 ratio finding — CI executes a single-digit percentage of this suite — does not depend on the exact totals. And the #900 reachability finding, that deleting the term leaves `test_ltx2_video` green, is not undone by adding three cases that also do not observe it. Scoped by measurement, not by assertion: `test_ltx2_loader.cpp` and `test_ltx2_device.cpp` are byte-identical across those two revisions — `git diff --numstat` reports nothing on either, against a positive control that reports `62 8` on `ltx2_video.cpp` — so their counts are untouched, and `test_ltx2.cpp` differs only by this branch's own edit. All 33 citations re-derived again on this tree: 33 FRESH, armed control still 6 STALE of 33. This commit changes one Markdown file and no build input, so the `BUILD_EXIT=0` / `CTEST_EXIT=0` / `100% tests passed, 0 tests failed out of 480` gate measured at `7bf2a9297` still describes the tree that lands; `git diff --name-only 7bf2a9297` naming only the spec is the evidence, and it is stated as an inherited gate rather than a fresh one. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-prompt-adaln.md | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md index 522132321..da276e20f 100644 --- a/.agents/specs/ltx25-prompt-adaln.md +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -465,12 +465,33 @@ assertions**, set gives **37 / 9031**, exit 0 both ways. The CASE count is the same in both configurations, so an unchanged case count never distinguishes them. The 2026-08-13 figures above are kept as what was measured then. +**AND THEY MOVED AGAIN, one merge later — the figures in this paragraph are +SUPERSEDED and are kept only as what was measured on the previous merge.** +`0785cfc4d` (#882) added 306 lines to `tests/vllm/multimodal/test_ltx2_video.cpp` +between `00613767d` and the merge that lands, taking it from **37 to 40** +`TEST_CASE`s (`grep -c '^TEST_CASE'` on both revisions; the same grep returns +matches on `origin/main`, so the count is not a failed pattern). That source +count is what is MEASURED here. The doctest CASE and ASSERTION totals in both +configurations were **NOT re-run** on this merge and are therefore UNKNOWN — not +37 / 784 and 37 / 9031. A number that has moved twice in three days is not one to +carry forward on the argument that it probably did not move a third time, and +this file's own rule is that a count owes its configuration AND its date. Owed +under [#673](https://github.com/mudler/vllm.cpp/issues/673), where this suite's +configuration debt already lives; the `set` arm needs the 18.72 GB NVFP4 and +23 GB FP8 DiTs under `$CHECKPOINT_ROOT`, so no CI host can close it. +`test_ltx2_loader.cpp` and `test_ltx2_device.cpp` are byte-identical across those +two revisions — `git diff --numstat` reports nothing on either, against a +positive control that reports `62 8` on `ltx2_video.cpp` — so their counts below +are unaffected, and `test_ltx2.cpp` differs only by this branch's own edit. + **And CI never sets it — [#673](https://github.com/mudler/vllm.cpp/issues/673), filed 2026-08-13 as visible debt rather than repaired here.** `grep -rn CHECKPOINT_ROOT .github/` exits 1 with zero hits while the same pattern matches in `tests/` and `.agents/` (positive control run in the same command, so this is not an assertion from a failed grep). CI therefore executes **784 of 9031 assertions — -8.7%** of this suite (2026-08-15; it was 502 of 8734, 5.7%, on 2026-08-13), at an +8.7%** of this suite (measured on the PREVIOUS merge, 2026-08-15; it was 502 of +8734, 5.7%, on 2026-08-13; both SUPERSEDED and un-remeasured after #882, see +above — the RATIO is the finding and it does not depend on the exact totals), at an identical case count in both configurations, and `scripts/measure-ltx2-prompt-adaln.py` — which produces every shipped-weights number in this Outcome — is a manual tool no gate invokes (`grep -rn @@ -704,7 +725,7 @@ Suite counts on the merged tree, exit 0 each, against the 2026-08-13 figures: | `test_ltx2` | 35 / 2435 | 43 / 4388 | | `test_ltx2_loader` | 26 / 4826 | 28 / 4978 | | `test_ltx2_device` | 15 / 523 | 18 / 546 | -| `test_ltx2_video` | 30 / 502 unset, 30 / 8734 set | 37 / 784 unset, 37 / 9031 set | +| `test_ltx2_video` | 30 / 502 unset, 30 / 8734 set | 37 / 784 unset, 37 / 9031 set — **SUPERSEDED, not re-run after #882 took the file to 40 `TEST_CASE`s** | The growth is #658's, not this reconciliation's: no `src/` or `include/` file differs from `origin/main` on this branch. @@ -791,7 +812,7 @@ scratch copy and **BUILT=YES, compile_err=NO**, so this is a test result: | suite | enters through | result | |---|---|---| | `test_ltx2` | `Ltx2DitForward`, by hand | RED — 3 of 43 cases, 6 of 4388 assertions, exit 1 | -| `test_ltx2_video` | `vllm_video_generate` / the ABI | **GREEN** — 37 of 37, 784 of 784, exit 0 | +| `test_ltx2_video` | `vllm_video_generate` / the ABI | **GREEN** — 37 of 37, 784 of 784, exit 0 (on the previous merge; #882 has since added 3 cases, and the mutation was NOT re-run — the finding is that the suite is green with the term deleted, which more cases cannot undo) | The entry-point suite drives the path — its fixture sets the flag TRUE (`tests/vllm/multimodal/ltx2_video_fixture.h:258`) — and asserts no value the