diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 641fb7c52..49e3fca0c 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -224,6 +224,8 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#691](https://github.com/mudler/vllm.cpp/issues/691) | `ROAD-V1-LTX25` | The LTX-2.5 out-of-scope ledger's reachable/marker split gates the MESSAGE TEXT, not reachability: give `kLoraFusion` a product call site and the ledger test still passes while the shipped refusal tells the caller `DECLARED, NOT REQUESTABLE` right after they requested it. Copy the anti-tautological shape of the extras inventory gate | bug | | [#692](https://github.com/mudler/vllm.cpp/issues/692) | `ROAD-V1-LTX25` | Campaign spec `ltx-2-5.md:291` still records the FABRICATED `multishot` arm as owed after row `LTX25-RETIRE-DEAD-ARMS` retired it; same line needs `CFG parallelism` renamed to multi-GPU. Operator-owned file, so filed rather than edited by that row | bug | | [#886](https://github.com/mudler/vllm.cpp/issues/886) | `POLICY-NOTHING-LANDS-DEAD` | The seam checkers police where a capability routes and nothing polices whether anything reaches it, so a change can land wired downward and driven only by its own test, as tensor parallelism did (guide [`reachability.md`](reachability.md)) | feature | +| [#673](https://github.com/mudler/vllm.cpp/issues/673) | — | LTX-2.5 prompt-AdaLN: the row's checkpoint-derived evidence is MANUAL and host-local. `LTX2_CHECKPOINT_ROOT` is set by no workflow (`grep -rn CHECKPOINT_ROOT .github/` exits 1 with zero hits, positive control matches in `tests/` and `.agents/`), so CI executes 784 of 9031 assertions — 8.7%, measured 2026-08-15 — of `test_ltx2_video`, at a case count identical in both configurations, and `scripts/measure-ltx2-prompt-adaln.py` is a manual tool no gate invokes. Listed under `## Owed` in [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md) | bug | +| [#900](https://github.com/mudler/vllm.cpp/issues/900) | — | LTX-2.5 prompt-AdaLN is REACHED from `vllm_video_generate` but no test ENTERS through that entry point and observes it: deleting the term's only consumer (`ltx2_dit.cpp:140`) takes `test_ltx2` red at 3 of 43 cases while `test_ltx2_video` stays GREEN at 37 of 37, so the ABI-level gate measures that the pipeline runs rather than that this capability is in it. The second half of `AGENTS.md` `## Nothing lands dead`, whose rule post-dates the row (#886). Listed under `## Owed` in [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md) | bug | | [#891](https://github.com/mudler/vllm.cpp/issues/891) | `MODEL-MM-qwen3-5-qwen3-5-moe-for-conditional-generation` | MoE vision tower (M2/M3): image and video for `Qwen3_5MoeForConditionalGeneration` — the dense arm has it, the MoE arm does not | feature | | [#889](https://github.com/mudler/vllm.cpp/issues/889) | `LTX25-RETIRE-DEAD-ARMS` | `kBetaScheduler` shipped as a REACHABLE refusal in the header, `docs/FEATURES.md` and `docs/USAGE.md` with zero product callers: its site sits inside `Ltx2Schedule`, which nothing calls, and the engine calls `Ltx2SigmaSchedule` directly. Upstream constructs `BetaScheduler` nowhere either — all seven ltx-pipelines entry points hard-code `LTX2Scheduler()` — so it is reclassified as a marker rather than wired | bug | | [#890](https://github.com/mudler/vllm.cpp/issues/890) | `LTX25-RETIRE-DEAD-ARMS` | `ltx2_video.cpp`'s temporal-upsampler guard tested `temporal_upsample` alone, which every BOTH-flags config also satisfies, so a genuine spatiotemporal checkpoint was told it is the temporal x2 arm and the ledger refusal naming the spatiotemporal one was unreachable from a request. Narrowed to `temporal_upsample && !spatial_upsample`, with a both-flags fixture driven through `LoadVideoEngine` | bug | @@ -232,3 +234,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#896](https://github.com/mudler/vllm.cpp/issues/896) | — | No gate refuses a new unchecked `static_cast<...LoadedModel&>` in a registry entry point. The #847 sweep decided the checker is warranted — the class is exactly grep-able, unlike the unaligned-read class in #627, and it regrows the moment a new model port copies its neighbour — and deliberately did not bundle a repository-wide refusal gate into a 30-file mechanical sweep. Listed under `## Owed` in [`registry-downcast-sweep.md`](specs/registry-downcast-sweep.md) §6 | feature | | [#897](https://github.com/mudler/vllm.cpp/issues/897) | — | `ModelAs`'s `const LoadedModel&` overload has no caller anywhere in the tree: every registered `prepare`/`forward` takes a NON-const `LoadedModel&`, so all 35 call sites select the non-const overload and its `Model&` result merely binds to the `const auto&`. Proven by deleting the overload and compiling all 30 swept TUs `-fsyntax-only`. It is dead in the way that reads as covered — #847 cited its existence as the reason the 14 `const` sites needed no thought. Listed under `## Owed` in [`registry-downcast-sweep.md`](specs/registry-downcast-sweep.md) §3.3 | bug | | [#904](https://github.com/mudler/vllm.cpp/issues/904) | `ROAD-V1-LTX25` | `main` red on `sanitize-cpu (address,undefined)` for a THIRD reason after the two [#730](https://github.com/mudler/vllm.cpp/issues/730) enumerates, both of which are now closed: `~Ltx2DitDeviceWeights` frees the staged DiT buffers on the main thread (`ltx2_device.cpp:1088`) while a `vt::cpu` threadpool worker is still reading one inside `AddKernel` (`cpu_layernorm.cpp:33`), so the staged weights' lifetime is not joined to the in-flight parallel op that reads them. Deterministic, 5 runs / 5 aborts on unmodified `e8048ef63`; the rest of the suite is 477/478 and `test_ltx2_video` now PASSES | bug | +| [#911](https://github.com/mudler/vllm.cpp/issues/911) | — | A `file:line` anchor into a file the row is ITSELF editing is stale by default, and spec BODIES are checked by nothing: `check-agent-record.py`'s `MATRIX_PATHS` (`:521`, `:529-530`) covers the five matrices, `feature-matrix.md` and `specs/model-family-inventory.md`, leaving 4772 line-carrying citations across 315 `.agents/specs/*.md` unexamined (positive control: 2314 line-less `.cpp` mentions match the same shape). `ltx25-prompt-adaln.md` shipped EIGHT stale repo-local anchors across two repair commits, moved by its own `020381676` and by `98f8e046d` (#658), then SEVEN more that were correct at `00613767d` and wrong at the merge of `origin/main`, because `0785cfc4d` (#882) added 70 lines to `ltx2_video.cpp` and 306 to `test_ltx2_video.cpp` ahead of every one. The obvious checker is a TAUTOLOGY — reading the span out of the file it validates reports 27 of 27 fresh on the same tree where reading the spans against their CLAIMS finds seven stale. Remedy is already in use and unwritten: `path:NN @ ` for a historical claim, claim-sourced uniqueness re-derivation for a live one, re-run after the merge. Narrower than [#632](https://github.com/mudler/vllm.cpp/issues/632) on surface and sharper on mechanism. Listed under `## Owed` in [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md) | bug | diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 6a2d024a5..344d620b9 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -461,7 +461,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-higgs-audio-v3-higgs-audio-v3-talker-for-conditional-generation` | `HiggsMultimodalQwen3ForConditionalGeneration`, `HiggsAudioV3TalkerForConditionalGeneration` | **OUT-OF-REPO** — not in `555967922`, and absent from `registry.py` on vLLM `main`; registered by `vllm-project/vllm-omni` at `vllm_omni/model_executor/models/registry.py:213-217` (and the explicit `HiggsAudioV3TalkerForConditionalGeneration` spelling at `:218`) @ `bbe6ccc512a404a2df8c977ea29003002f2683e8` → module `vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_talker.py`, class `HiggsAudioV3TalkerForConditionalGeneration` (separate module and class fields rather than the compact colon-pair form, per the out-of-repo convention), with code2wav at `registry.py:223-227` → module `higgs_audio_v3/higgs_audio_v3_code2wav.py`, class `HiggsAudioV3Code2WavForConditionalGeneration`. Documented at `vllm-omni` `docs/models/supported_models.md:81`. Official recipe: `bosonai/higgs-audio-v3-tts-4b` (published at `recipes.vllm.ai`), whose `config.json` declares `architectures: ["HiggsMultimodalQwen3ForConditionalGeneration"]`. The distinct v2 lineage (`HiggsAudioV2ForConditionalGeneration`, `higgs_audio_v2/`) is a DIFFERENT target and has no recipe in the audited set, so it gets no row here | conditional generation / text+audio in, AUDIO OUT (TTS) | MM processor; audio detokenizer / code2wav; audio tokenizer; Qwen3-derived multimodal decoder; streaming audio output | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-voxtral-realtime-voxtral-realtime-for-conditional-generation` | `VoxtralRealtimeForConditionalGeneration` (target-pending) | **TARGET-PENDING — the architecture string is registered NOWHERE we can find.** Recipe `mistralai/Voxtral-Mini-4B-Realtime-2602` (one of the 157 recipes at `vllm-project/recipes` `86c7777a`) declares `architectures: ["VoxtralRealtimeForConditionalGeneration"]` with an inner `audio_config.model_type: "voxtral_realtime_encoder"`, but that exact string appears in none of: (a) the pinned registry `555967922`; (b) any of the ten dictionaries of `vllm/model_executor/models/registry.py` on vLLM `main`; (c) `_OMNI_MODELS` in `vllm_omni/model_executor/models/registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8`; (d) `vllm-omni` `docs/models/supported_models.md`. **Related but NOT equal, recorded so nobody re-derives it:** `VoxtralRealtimeGeneration` — a DIFFERENT string — is registered at the pin (`registry.py:585`, `main` `:603`) and already has its own row here, `MODEL-MM-voxtral-realtime-voxtral-realtime-generation`; `VoxtralForConditionalGeneration` (pin `registry.py:584`) is the audio→TEXT model; and `vllm-omni` `registry.py:305-309` registers `VoxtralTTSForConditionalGeneration` for the SEPARATE `mistralai/Voxtral-4B-TTS-2603` recipe. The most likely reading is that this checkpoint's `config.json` names a renamed or not-yet-landed spelling of the existing `VoxtralRealtimeGeneration` target, but that is a hypothesis: no anchor is asserted until one of those is shown to be what actually resolves. Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / audio in, AUDIO OUT (realtime speech-to-speech) — modality inferred from the recipe and config, NOT from an upstream implementation | unresolved until the target is located | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-bailing-mm-native-bailing-mm-native-for-conditional-generation` | `BailingMMNativeForConditionalGeneration` (target-pending) | **TARGET-PENDING — the architecture string is registered NOWHERE we can find.** Recipe `inclusionAI/Ming-omni-tts-0.5B` (one of the 157 recipes at `vllm-project/recipes` `86c7777a`) declares `architectures: ["BailingMMNativeForConditionalGeneration"]`, but that exact string appears in none of: (a) the pinned registry `555967922`; (b) any dictionary of `vllm/model_executor/models/registry.py` on vLLM `main`; (c) `_OMNI_MODELS` in `vllm_omni/model_executor/models/registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8`; (d) `vllm-omni` `docs/models/supported_models.md`. **Related but NOT equal:** `vllm-omni` `docs/models/supported_models.md:75` lists that SAME checkpoint under a different architecture name, `MingTTSForConditionalGeneration` (`registry.py:250-254` → module `ming_tts/ming_tts.py`, class `MingTTSForConditionalGeneration`); and `vllm-omni` `registry.py:417-421` carries a near-miss alias `BailingMM2NativeForConditionalGeneration` — note the `2` — commented "HF repo currently ships this architecture name in config.json", pointing at module `ming_flash_omni/ming_flash_omni.py`, class `MingFlashOmniForConditionalGeneration`, for the DIFFERENT Ming-flash-omni-2.0 checkpoint. So the shape of the gap is a missing config-string alias upstream, but that is an observation, not an anchor, and none is asserted here. Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / text+image+audio in, AUDIO OUT (TTS) — modality from the recipe and config, NOT from a resolved upstream target | unresolved until the target is located | ☐ required | `INVENTORIED` | none | unassigned | -| `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` | `LTX2VideoTransformer3DModel` | **BEYOND-PIN AND OUT-OF-REPO** — not in `555967922` (LTX-2.5 released 2026-08) and not in the vLLM repository at all. Architecture reference is Lightricks' own `LTX-2` (`packages/ltx-core/src/ltx_core/`: `model/transformer/{transformer,model,attention,rope,model_configurator}.py`, `model/audio_vae/`, `components/`, `guidance/`). vLLM-Omni carries an `ltx2` module but its recipes stop at 2.3 (`vllm_omni/diffusion/models/ltx2/ltx2_recipes.py:162-166`); 2.5 is upstream-OPEN at [vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066) and [#4985](https://github.com/vllm-project/vllm-omni/issues/4985). Serving oracle reaches 2.5 through the GENERIC `DiffusersAdapterPipeline` (`pipeline_diffusers_adapter.py:116`) against `Lightricks/LTX-2.5-Diffusers`. | diffusion generation / video + audio (text/image/video in, video+audio out) | flow-matching denoise loop; TWO coupled modality streams with audio<->video cross-attention; per-head gated attention; AdaLN-single with 9-vector per-block tables; timestep-INDEPENDENT prompt K/V (cacheable once per request); split/interleaved 3D RoPE; gelu-approximate FFN; Gemma-4 12B text encoder with dual caption projections (4096 video / 2048 audio); Conv video VAE + DiffVAE; audio VAE + vocoder; latent spatial/temporal x2 upsamplers; duration head; distilled two-stage recipe | [ltx-2.5](specs/ltx-2-5.md) | `SPIKE` | **L0 LANDED 2026-08-11 (#435).** Spec committed BEFORE implementation per AGENTS.md. Geometry measured from the ungated `vonkaiser/LTX-2.5-FP8-NVFP4` FP8 checkpoint header by range request — the same no-payload technique used for H3's manifests: 21.00B params, 48 blocks, 1775 F8_E4M3 tensors each with a per-tensor F32 `weight_scale`, biases/norms BF16. Cross-modal projection shapes are ASYMMETRIC and are a named test trap: `audio_to_video_attn.to_q` is [2048, 4096] while `to_k`/`to_v` are [2048, 2048] and `to_out` is [4096, 2048], so a transposition still type-checks against a square assumption. **CHECKPOINT ACCESS (verified against the HF API 2026-08-11):** `Lightricks/LTX-2.5` is `gated: auto` (accepting the license opens it; holds the first-party NVFP4 DiT at 18.72 GB), `Lightricks/LTX-2.5-Diffusers` is RESTRICTED behind manual approval (needed for the binding oracle), and `vonkaiser/LTX-2.5-FP8-NVFP4` is UNGATED and unblocks L1-L2 today. All artifacts land under `$CHECKPOINT_ROOT` (`/mnt/nas_share/checkpoints`) so dgx.casa and the cluster nodes share one copy. **PHASES (one PR, developer-directed):** L1 `vllm::multimodal::VideoEngine` interface + checkpoint-detected registry with MiniMax-H3 moved behind it UNCHANGED (gated on frames+WAV byte-identical to the pre-refactor fold fixture) and ABI **v13 by ADDING fields only**; L2 DiT forward; L3 Gemma-4 TE reusing `gemma4.cpp`; L4 Conv video VAE + audio VAE + vocoder; L5 pipeline/recipes/upsampler/duration head; L6 NVFP4 arms + GB10 load-time residency; L7 e2e on dgx.casa under `flock $HOME/gpu.lock`. **OPEN, same as H3:** there is no vllm-omni parity PIN — `.agents/upstream-sync.md` covers the vLLM repo only, so each golden records the vllm-omni SHA inline. | `CLAIM-LTX25-L0-L7` | +| `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` | `LTX2VideoTransformer3DModel` | **BEYOND-PIN AND OUT-OF-REPO** — not in `555967922` (LTX-2.5 released 2026-08) and not in the vLLM repository at all. Architecture reference is Lightricks' own `LTX-2` (`packages/ltx-core/src/ltx_core/`: `model/transformer/{transformer,model,attention,rope,model_configurator}.py`, `model/audio_vae/`, `components/`, `guidance/`). vLLM-Omni carries an `ltx2` module but its recipes stop at 2.3 (`vllm_omni/diffusion/models/ltx2/ltx2_recipes.py:162-166`); 2.5 is upstream-OPEN at [vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066) and [#4985](https://github.com/vllm-project/vllm-omni/issues/4985). Serving oracle reaches 2.5 through the GENERIC `DiffusersAdapterPipeline` (`pipeline_diffusers_adapter.py:116`) against `Lightricks/LTX-2.5-Diffusers`. | diffusion generation / video + audio (text/image/video in, video+audio out) | flow-matching denoise loop; TWO coupled modality streams with audio<->video cross-attention; per-head gated attention; AdaLN-single with 9-vector per-block tables; timestep-CONDITIONED prompt K/V via `prompt_adaln_single` (model.py:223-227, transformer.py:441-446), so the K/V cache is inapplicable on the shipped DiT -- the row's first claim of cacheability was RETRACTED 2026-08-12 and the module PORTED 2026-08-13 (#644, [spec](specs/ltx25-prompt-adaln.md)); on the real weights the timestep term DOMINATES the static table, RMS ratio 1347% video / 1583% audio; split/interleaved 3D RoPE; gelu-approximate FFN; Gemma-4 12B text encoder with dual caption projections (4096 video / 2048 audio); Conv video VAE + DiffVAE; audio VAE + vocoder; latent spatial/temporal x2 upsamplers; duration head; distilled two-stage recipe | [ltx-2.5](specs/ltx-2-5.md) | `SPIKE` | **L0 LANDED 2026-08-11 (#435).** Spec committed BEFORE implementation per AGENTS.md. Geometry measured from the ungated `vonkaiser/LTX-2.5-FP8-NVFP4` FP8 checkpoint header by range request — the same no-payload technique used for H3's manifests: 21.00B params, 48 blocks, 1775 F8_E4M3 tensors each with a per-tensor F32 `weight_scale`, biases/norms BF16. Cross-modal projection shapes are ASYMMETRIC and are a named test trap: `audio_to_video_attn.to_q` is [2048, 4096] while `to_k`/`to_v` are [2048, 2048] and `to_out` is [4096, 2048], so a transposition still type-checks against a square assumption. **CHECKPOINT ACCESS (verified against the HF API 2026-08-11):** `Lightricks/LTX-2.5` is `gated: auto` (accepting the license opens it; holds the first-party NVFP4 DiT at 18.72 GB), `Lightricks/LTX-2.5-Diffusers` is RESTRICTED behind manual approval (needed for the binding oracle), and `vonkaiser/LTX-2.5-FP8-NVFP4` is UNGATED and unblocks L1-L2 today. All artifacts land under `$CHECKPOINT_ROOT` (`/mnt/nas_share/checkpoints`) so dgx.casa and the cluster nodes share one copy. **PHASES (one PR, developer-directed):** L1 `vllm::multimodal::VideoEngine` interface + checkpoint-detected registry with MiniMax-H3 moved behind it UNCHANGED (gated on frames+WAV byte-identical to the pre-refactor fold fixture) and ABI **v13 by ADDING fields only**; L2 DiT forward; L3 Gemma-4 TE reusing `gemma4.cpp`; L4 Conv video VAE + audio VAE + vocoder; L5 pipeline/recipes/upsampler/duration head; L6 NVFP4 arms + GB10 load-time residency; L7 e2e on dgx.casa under `flock $HOME/gpu.lock`. **OPEN, same as H3:** there is no vllm-omni parity PIN — `.agents/upstream-sync.md` covers the vLLM repo only, so each golden records the vllm-omni SHA inline. | `CLAIM-LTX25-L0-L7` | | `MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation` | `MiniMaxMusic3ForConditionalGeneration` | **BEYOND-PIN AND OUT-OF-REPO** — not in `555967922` and not in the vLLM repository at all; absent from `vllm-omni` too. Its reference implementation is the OPEN `diffusers` PR [#14456](https://github.com/huggingface/diffusers/pull/14456) head `c6da9936` (`src/diffusers/models/transformers/transformer_minimax_music3.py`, `.../minimax_music3_rvq_depth_decoder.py`, `src/diffusers/models/autoencoders/minimax_music3_vocoder.py`, `src/diffusers/models/condition_embedders/condition_embedder_minimax_music3.py`, `src/diffusers/modular_pipelines/minimax_music3/`), with SGLang-Omni `748a0b43` (`sglang_omni/models/minimax_music3/`) serving the NATIVE layout as the e2e cross-check. **The first row to exercise the fallback-oracle rule (#647)** | music generation / text -> audio (lyrics + structured description in, multi-minute stereo song out) | Qwen3 dense forward + paged KV (LANDED); flow-matching denoise loop (H3/LTX-2.5 precedent); DAC-style audio VAE decode + WAV writer (H3/LTX-2 precedent); NEW: 8-codebook RVQ frame path, 4-layer depth decoder, learned 8-layer condition mix, snake activations with weight-norm folding, and an LLM->diffusion handoff on CONTINUOUS hidden states rather than discrete tokens | [minimax-music3](specs/minimax-music3.md) | `ACTIVE` | **W0 DONE + W1 LANDED (#672).** Oracle GATEABLE: `tools/oracle/music3_oracle.py` generated 0.998 s of 44100 Hz stereo at diffusers `c6da9936` (identity asserted at runtime, proved red first), with 13 per-stage goldens under `tests/parity/goldens/minimax_music3_oracle/` (#708). W1 modular loader `minimax_music3_loader.{h,cpp}`: 1413/1413 assertions against the real 27 GB tree, all 1012 tensors accounted, 50 mutations proven to fire, native arm refused by name; `MaterializeWeightNorm` relocated into the shared `vocoder1d` primitives (H3 79/79 cases, 57,395 assertions unchanged) rather than forked (#714). Spec committed BEFORE implementation per AGENTS.md. Geometry MEASURED, not inferred: every component `config.json` read, plus safetensors headers by HTTP range request — `transformer` 231 tensors/1.240B per shard **all F32** (so the card's 2.4B is right and 9.73 GB is fp32 storage, NOT a 4.9B bf16 model), `rvq_depth_decoder` 47/0.646B BF16, `vocoder` 121/0.054B F32 with `weight_g`/`weight_v` weight-norm, `condition_encoder` **4 tensors** (`layer_scale`, `layer_weight_logits`, `proj`) which makes it a learned weighted mix over 8 LLM layers rather than an encoder tower. `language_model` is our LANDED `Qwen3ForCausalLM` architecture exactly (36L/4096/32-8/head_dim 128) at `vocab_size 200000`. The repo ships TWO packagings of ONE set of weights (native `qwen_7B/`+`flowmatching_vae.pth`+`dav.pth` vs the diffusers six); `scripts/convert_minimax_music3_to_diffusers.py@c6da9936` `:29-38,47-189` proves the conversion is a pure re-layout, so the diffusers arm is the port target at **~28.5 GB resident** (no pickle, no `trust_remote_code`) and SGLang-Omni stays a valid e2e cross-check. **Sample rate RESOLVED 2026-08-13** (spec §1.1): not a contradiction but a stage boundary. The vocoder natively emits 44100 stereo (86.133 Hz latent frames x 512 upsampling, derived from `minimax_music3_vocoder.py:84,92-95` and confirmed by `dav.py:94,115`); diffusers returns that unresampled, while SGLang-Omni's SERVER resamples 44100 -> 32000 on delivery (`constants.py:18-19`, `acoustic.py:55-58,422-431`). Goldens are captured at 44100 stereo, the native resample-free rate; 32 kHz is a downstream delivery transform gated separately, and NOT free — reproducing its bytes needs torchaudio's default sinc filter, and a latent-parity check sits upstream of that call and cannot see it. **W2/W3 landed 2026-08-14 (#672)**: the autoregressive half's compute — prompt assembly, the semantic CFG logit pipeline, the learned 8-layer condition mix and the 4-layer RVQ depth decoder — gated at reduced dimensions in float32 against upstream's own executed classes (25 cases / 338 assertions, no checkpoint) AND at full scale in bf16 against the committed oracle capture (condition mix 175 989 of 176 128 values BIT-IDENTICAL; depth decoder 716 800 values over 25 frames x 7 depth steps). **Spec §5's token-exact claim is WITHDRAWN and the reason is measured**: upstream's AR stage has NO greedy path (`_sample_top_k` ends in `torch.multinomial` against a seeded generator, `encoders.py:94-103`) and both stages sample from a CFG mix whose unconditional row the goldens do not store, so `rvq_codes.npy` is a seeded sample consumed as an INPUT. The full-scale bound is calibrated against a MATCHED CONTROL — torch's own `sdpa_kernel(MATH)` arm reproduces the goldens to 46.34% bit-identical at mean absolute error 1.659e-03, ours 43.61% / 1.824e-03 — because torch's CPU attention kernel runs a blocked online softmax no closed-form rounding model reproduced (its bf16 Linear reproduces 32 759 of 32 768). **W4/W5 landed 2026-08-14 (#672)**: the ACOUSTIC half — `minimax_music3_acoustic.{h,cpp}` — carrying the 2.4B fp32 flow-matching DiT (partial rotary over 32 of 64 head dims, a prepended timestep token, two RESIDUAL 1x1 convolutions, a gated FFN whose FIRST half is the value), the `FlowMatchEulerDiscreteScheduler` with `invert_sigmas` (shift -> invert -> train-scale, in float32 because the shift3 golden can tell float32 from double), `ClassifierFreeGuidance` at scale 1.7 based on the UNCONDITIONAL row, the denoise loop's 200-frame/100-hop window bookkeeping, and the DAC Flow-VAE vocoder routed through the shared `vocoder1d` primitives with `AliasFreeActivation1d` explicitly NOT applied (Music3 uses plain snake). Gated at reduced dimensions in float32 against upstream's own EXECUTED classes (27 cases / 265 assertions, no checkpoint; 16 of 18 mutations fire) AND at full scale against the committed capture: the Euler step reproduces `denoise_{first,last}_latents_out` **BIT-EXACTLY** (22 016 of 22 016), `denoise_last_latents_out` is bit-identical to `vocoder_input_chunk0`, and the vocoder reproduces `waveform.npy` over 88 064 samples with 0 outside tolerance (mean abs-diff 3.19e-08). **Bounds calibrated against a MEASURED control** — upstream's own modules at `torch.set_num_threads(1)` against goldens captured at 20 threads reproduce to 1.911 % bit-identical (vocoder) and 15.416 %/5.596 % (DiT first/last), so the ABSOLUTE floor binds and the relative one does not (the control's own max relative deviation is 7.4e-02 on near-zero samples). **Three findings**: W1's header documented `folded == 20` where the checkpoint yields 30 (corrected in the same change); a 1e-5 relative tolerance HIDES upstream's `(1 - 1e-6)` overlap-blend factor, so that assertion is now bit-exact; and CFG at scale 1 does not recover the conditional row bit-for-bit in float32, so the exact discriminator is scale 0 against the unconditional row. **W6 landed 2026-08-14 (#672)**: the model reaches the SHARED SURFACE. `minimax_music3_speech.{h,cpp}` registers it as a `SpeechRegistry` family named `minimax-music3` whose detector INSPECTS the artifact (`modular_model_index.json` must name `MiniMaxMusic3ModularPipeline` AND all seven component directories must exist), declares 44100 Hz stereo and `requires_reference_audio() == false`, and composes upstream's four modular-pipeline blocks that nothing had composed: `before_denoise.py` -> `Music3ChunkPlan`, `denoise.py` -> `Music3DenoiseChunks`, `decoders.py` -> `Music3DecodeChunks`. `multimodal::SpeechGenParams` grew ADDITIVELY by `lyrics`, `description`, `audio_duration_s`, `num_inference_steps` and `guidance_scale` — two texts because upstream normalizes lyrics and description differently (`encoders.py:54-91`), so one `text` behind a separator would be a private protocol inside a shared struct — and IndexTTS-2.5 is BYTE-IDENTICAL (0 lines changed in `indextts2.{h,cpp}` or `speech_engine.cpp`; its 4 cases / 8 assertions and the seam's 7 cases / 20 assertions unchanged). The new C-ABI surface is **v20, not v19**: `origin/main` took v19 for the multimodal input limits (#607 L2) while this was in flight, so the number was renumbered rather than contested. `POST /v1/audio/speech` is OpenAI's createSpeech shape with the two music inputs as additional named fields; it is registered ONLY when a synthesizer is attached, gated over a REAL SOCKET in both directions (absent => 404 from the route table with no envelope leaked; present => `audio/wav` bytes at 44100 Hz stereo), and `voice`, `speed`, streaming and any non-`wav` `response_format` are refused BY NAME rather than ignored. The `requires_reference_audio()` refusal happens BEFORE the runner is called, which is the reason that method exists. **Gated full-scale against the capture**: the vocoder + crop + clamp + stereo split reproduce `waveform.npy` over 88 064 values with 0 outside W5's own bound, the WAV payload is BIT-EXACT against the quantization of that golden (88 064 int16 samples, 0 mismatched), and the WHOLE TAIL — condition mix -> 4 guided 2.4B DiT steps -> vocoder, driven from `frame_hiddens.npy` and the capture's own `denoise_first_sample_in.npy` — lands at max abs-diff 4.523e-06 / mean abs-diff 1.225e-07 on the waveform and max abs-diff 2.396e-05 on the latents, exactly where the DiT's measured per-step error propagated over four Euler steps says it should. **No request's waveform can ever equal the golden**, twice over: the AR codes are a seeded `torch.multinomial` draw and the denoise loop's initial latents are a seeded `randn_tensor`, so `Music3NoiseSource` is a PARAMETER and the gate supplies the capture's own noise — that is the only entry at which this loop is comparable at all. **Formerly owed on W2, now LANDED (this PR).** The dense path grew `Qwen3DenseModel::ForwardEmbeds` - the `inputs_embeds` door upstream's own `Qwen3Model.forward` has and that only the DENSE registration had never wired, already present on `qwen3_vl.h:145,159`, `gemma4.h:210-218` and `muse_glimmer.h:369-380`. It is needed because `_embed_audio_frame` (`encoders.py:106-115`) is a SUM of one embedding row and seven depth-decoder rows scaled by `num_codebooks^-0.5`, a continuous vector no token id spells. ADDITIVE and proved so: the embedding OF THE SAME IDS reproduces `Forward` bit for bit in the logits AND in the paged KV, which matters because five registrations ride that one forward. **`num_condition_layers: 8` is NOT eight transformer layers** - the eight rows are `cat(last_hidden, depth_hidden_1..7)` (`encoders.py:343`), so no per-layer capture is needed and none is taken. **Gated**: 102 400 values, 9337 bit-identical, mean abs-diff 1.763e-02, inside a MATCHED CONTROL (12 036 / 1.475e-02) and with a NEGATIVE CONTROL 46x outside; the oracle's own codes rank 2.48 where chance is 8191.5. **End to end: WRITTEN, WIRED, NOT OBSERVED** - two runs stopped at 85 and 34 minutes, both inside the SHARED `LoadQwen3ForCausalLMWeights` rather than in Music3 code, which is what makes it a performance question; the same load and forward DO complete in test_minimax_music3_llm_real. Owed and named. Previously owed: the 8.6B `Qwen3ForCausalLM` forward teacher-forced on the golden codes, which needs an `inputs_embeds` entry the landed Qwen3 dense path does not have — it is REFUSED BY NAME from `Synthesize`, so a real request gets that refusal rather than silence. **W7 landed 2026-08-14 (#672): ONE arm implemented and value-gated, the rest refused BY NAME.** The survey (spec §9.1, every query with its result count) found MiniMaxAI ships bf16/fp32 ONLY while the community published 14 quantized repositories in 5 formats within days. NOT found by those queries: AWQ, GPTQ, compressed-tensors, fp8 or bitsandbytes. Ten published GGUF headers were then MEASURED by HTTP range request (56 MiB, no weight bytes), which is what turned a repo list into a contract and produced the finding that "the GGUF arm" is THREE MUTUALLY INCOMPATIBLE LINEAGES: the ComfyUI lineage ships the DiT + condition encoder ONLY so it can never generate audio; `mm3` (scragnog) needs a rename table PLUS fused QKV to split and folded weight-norm to invert; `comfy.gguf.orig_shape.*`, which the H3 GGUF arm depends on, is absent from all ten. The IMPLEMENTED arm is the audio-cpp `native` lineage's `rvq_depth_decoder_q4_k.gguf` — chosen because it is the ONLY component whose bound can be DERIVED today (W3 already gates that tensor full-scale and its bf16 control is already measured), and because `audiocpp.tensor_name_format = native` means all 47 names bind to `EnumerateMiniMaxMusic3RvqDepthDecoderTensors` EXACTLY, so no rename table can hide a mis-binding. Everything else — the other 4 components, the other 2 lineages, NVFP4, MXFP4, FP8, INT8, AWQ/GPTQ, bitsandbytes, MLX and an UNIDENTIFIED case naming all 3 candidate schemes rather than picking one — is refused with the format, the evidence, the missing piece, the supported arm, the phase and the issue. Before it, a GGUF tree was told it was "missing transformer, condition_encoder, ..." (seven directories its owner does not have) and a real NVFP4 `condition_encoder` refused on `layer_scale`, a tensor that is not quantized, is not wrong, and merely sorts first. Gated: 29 cases / 125 assertions with NO checkpoint and no network, plus 6 cases / 319 assertions against the pinned artifact. RED captured first at 8 of 8 probe checks. **18 mutations, all 18 fire**, and three are recorded rather than counted — one STAYED GREEN (hardcoding `matched = 1` passed every case because each carried exactly one marker, so the count was reported but never DISCRIMINATED; a 36-marker case closed it), one was a BAD MUTATION (deleting the config hook's only caller tripped `-Werror` unused-function, so the COMPILER refused it and the gate never spoke — a build failure is not a red gate), and QM4 (neutering the GGUF lineage guard) exposed a real COVERAGE HOLE: the suite's only GGUF was the accepted lineage, so nothing proved the guard rejected the other two — closed with synthetic in-memory GGUF fixtures, after which it fires. NO speed number is claimed and none was measured. The negatives are gated too, because a detector that fired on the shipped checkpoint would refuse every real load: the bf16/fp32 dtypes, the real transformer config, a `null` quantization_config, and the vocoder's 30 legacy `weight_g`/`weight_v` weight-norm pairs, which are a PARAMETERIZATION and not a quantization | `CLAIM-MODEL-MUSIC3-W0` | | `MODEL-MM-moss-transcribe-diarize-moss-transcribe-diarize-for-conditional-generation` | `MossTranscribeDiarizeForConditionalGeneration` (v0.25.0 target-pending) | v0.25.0 target `registry.py:450-453`; `vllm/model_executor/models/moss_transcribe_diarize.py::MossTranscribeDiarizeForConditionalGeneration` @ `702f481` | conditional generation / audio | MM processor; Whisper encoder; VQ adaptor; Qwen3 decoder; speech-to-text/diarization frontend | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-lightonocr-light-on-ocrfor-conditional-generation` | `LightOnOCRForConditionalGeneration` | `registry.py:450-453`; `vllm/model_executor/models/lightonocr.py::LightOnOCRForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md index a24f127ce..da276e20f 100644 --- a/.agents/specs/ltx25-prompt-adaln.md +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -16,7 +16,8 @@ IMPORTS and EXECUTES the LTX-2 modules rather than restating them. ## 0. What is wrong today -`src/vllm/model_executor/models/ltx2_loader.cpp:988` sets, unconditionally: +`src/vllm/model_executor/models/ltx2_loader.cpp:988 @ baa92ccf7` sets, +unconditionally: ```cpp declared.use_prompt_adaln_single = false; @@ -31,14 +32,24 @@ and `:573` / `:626` do the same on the two manifest paths. The flag defaults default, and `model_configurator.py:76` / `:138` read it as `config.get("use_prompt_adaln_single", True)` -The shipped FP8 DiT carries the 18 tensors the flag builds (12 -`prompt_adaln_single.*`, 6 `audio_prompt_adaln_single.*`; see -`tests/vllm/models/ltx2_fp8_dit_manifest.inc:232-240,286-294`), so the flag is -TRUE for the checkpoint this campaign renders. `ltx2.cpp:274-276` refuses those -tensors by name, so a real render needs `allow_unported_modules=1` -(`src/vllm/multimodal/ltx2_video.cpp:570`) — which reaches the loader lines above +The shipped FP8 DiT carries the 18 tensors the flag builds — **9 +`audio_prompt_adaln_single.*` at +`tests/vllm/models/ltx2_fp8_dit_manifest.inc:232-240` and 9 +`prompt_adaln_single.*` at `:286-294`**, each stream contributing 6 parameters +(`linear_1`, `linear_2`, `linear`, weight and bias) plus the 3 `F32` +`weight_scale` entries the FP8 file carries beside them. So the flag is TRUE for +the checkpoint this campaign renders. `ltx2.cpp:274-276 @ baa92ccf7` refuses +those tensors by name, so a real render needs `allow_unported_modules=1` +(`src/vllm/multimodal/ltx2_video.cpp:660`) — which reaches the loader lines above and **silently clears the flag**. +(The split read `12` / `6` until 2026-08-15. It is `9` / `9`: `grep -c` over the +two families in that file returns 9 apiece, and the two cited spans are nine +lines each. The TOTAL of 18 was always right, and so was every conclusion drawn +from it; the breakdown was not. Corrected here because the anchor beside it was +being re-derived anyway, and a citation whose own span contradicts the sentence +that carries it is the defect this repair exists to remove.) + Net effect: every render drops the timestep-conditioned half of the prompt K/V modulation, keeping only the static `prompt_scale_shift_table`. Nothing observes it: shapes are unchanged, values stay finite, and the goldens were generated with @@ -72,8 +83,8 @@ not `adaln_embedding_coefficient()` — shift and scale for the K/V only. diffusers twin: `transformer_ltx2.py:1255-1259`, `num_mod_params=2`. Registration order inside `_init_video` puts it between `adaln_single` and -`proj_out`, which is where `EnumerateLtx2DitTensors` already reserves its slot -(the `VT_CHECK` at `ltx2.cpp:274-276`). +`proj_out`, which is where `EnumerateLtx2DitTensors` already reserved its slot +(the `VT_CHECK` at `ltx2.cpp:274-276 @ baa92ccf7`). ### 1.2 The producer @@ -144,7 +155,7 @@ Layout consequence: the flat `[B, 1, 2 * dim]` row is read as `[2, dim]` with (`ltx2_dit.cpp`) and device (`ltx2_device.cpp`) paths. 4. The three loader `= false` assignments are deleted, and replaced by a guard (§3.2) that makes a future silent clear impossible. -5. The `ltx2.cpp:274-276` refusal is deleted for these two families. +5. The `ltx2.cpp:274-276 @ baa92ccf7` refusal is deleted for these two families. 6. Goldens executed from upstream at reduced dims, with a mutation proving the new term is load-bearing, and a measured magnitude. @@ -168,12 +179,12 @@ golden valid. The **order of the two additions** is upstream's: the table and the timestep row are summed FIRST (`:443`), and only then does `(1 + scale)` apply. Folding it the other way would round differently — the same trap `ProcessOutput` -(`ltx2_dit.cpp:530-540`) already documents. +(`ltx2_dit.cpp:640-642`) already documents. ### 3.2 The prompt-K/V cache, and what replaces the cleared flag `Ltx2DitForward` already refuses a cache when the flag is on -(`ltx2_dit.cpp:672-676`); with the flag no longer cleared, that refusal becomes +(`ltx2_dit.cpp:775-780`); with the flag no longer cleared, that refusal becomes *reachable* rather than dead, and it is correct: the K/V now carry a timestep term. Nothing in the shipped pipeline passes a cache (`grep` over `ltx2_pipeline.cpp` finds none), so no caller regresses. @@ -195,7 +206,7 @@ and setting it with the tensors absent would bind missing weights. After this row it is scoped to genuinely-unported modules only: the sole flag it still clears in a config copy is `use_keyframes_abs_pos_embedding` -(`ltx2_loader.cpp:979-984`), whose module really is unported. The guard in §3.2 +(`ltx2_loader.cpp:1033-1035`), whose module really is unported. The guard in §3.2 is what makes that scoping structural rather than a comment — the extra `= false` cannot come back without going red. @@ -204,6 +215,18 @@ second way: a checkpoint whose config declares `use_prompt_adaln_single=false` while its shapes carry the tensors now produces two DIFFERENT contracts and is refused, instead of both sides being forced to the same cleared value. +**SUPERSEDED 2026-08-15 by `LTX25-KEYFRAMES-ABS-POS` +([#658](https://github.com/mudler/vllm.cpp/issues/658), landed as `98f8e046d`).** +`keyframes_abs_pos_embedding` is PORTED, so the "sole flag still cleared" above +is now cleared by nothing: `Ltx2AdoptDeclaredDitParams` resolves the declared +flag against what the file carries instead of force-clearing it, and neither +shipped DiT needs `allow_unported_modules` any more. This paragraph is kept as +the state at the time this row ran; the current disposition is +[`ltx25-keyframes-abs-pos.md`](ltx25-keyframes-abs-pos.md) §2. What is unchanged +is the guard in §3.2, which is what makes the scoping structural — and which is +why the keyframes port could retire the clear without re-opening this row's +defect. + ## 4. Memory format Mirrors the existing L2 parity forward exactly: f32 host, f32/bf16 device stream @@ -228,10 +251,18 @@ byte cost. ## 6. Measured magnitude -Recorded in §Outcome: the relative change in the modulated prompt context and in -the DiT's outputs, flag ON vs OFF, at the reduced dimensions. This is the answer -to "does this matter"; a number below round-off would mean the term is inert and -the whole row is decoration. +Recorded in §Outcome, on two fixtures that answer two different questions. + +The reduced-dimension generator gives the relative change in the modulated prompt +context and in the DiT's outputs, flag ON vs OFF. That is a GATE FLOOR: a number +below round-off would mean the term is inert and the mutation in §5.3 could not +bite. It is **not** the answer to "does this matter", because both the static +table and the prompt-AdaLN MLP are drawn from the same synthetic init scale, so +every ratio it produces is a property of the fixture. + +"Does this matter" is answered on the SHIPPED checkpoint's own weights, run +through upstream's `AdaLayerNormSingle`. That measurement is required before the +row's Outcome may state a magnitude. ## 7. Risks @@ -256,7 +287,7 @@ the whole row is decoration. ## Outcome -### What was measured +### What was measured — (a) the gate floor, on SYNTHETIC weights The generator emits these into `tests/vllm/models/ltx2_goldens.inc` and prints them on stderr, from the SAME shared weight stream on both arms (keyed by @@ -265,21 +296,114 @@ term and nothing else): | Quantity | Flag ON vs OFF | |---|---| -| **timestep term vs the static table it is added to** | `max\|term\|` 0.0252 vs `max\|table\|` 0.0487 — **51.7%** | -| **block-0 modulated prompt K/V** | `max\|on-off\|` 0.0310 — **5.82%** of `max\|off\|` | +| timestep term vs the static table it is added to (**VIDEO stream**) | `max\|term\|` 0.0252 vs `max\|table\|` 0.0487 — 51.7% | +| block-0 modulated prompt K/V (**VIDEO stream**) | `max\|on-off\|` 0.0310 — 5.82% of `max\|off\|` | | DiT video output (2 blocks) | 1.4567e-4 — 0.04% of `max\|off\|`, **73x** the gate's 2e-6 floor | | DiT audio output (2 blocks) | 7.367e-5 — 0.03%, **37x** the floor | -**The answer to "does this matter" is the first two rows.** Roughly half the -magnitude of the prompt K/V modulation is the timestep-conditioned term, and -including it moves the modulated prompt context by ~6%. Every render before this -row discarded that. +**ALL FOUR ROWS ARE GATE-FLOOR NUMBERS, AND NONE OF THEM ANSWERS "DOES THIS +MATTER".** Corrected 2026-08-13 (issue #644) — this section originally billed the +first two as the answer and disclaimed only the last two as synthetic-bounded. +They have identical provenance: `prompt_scale_shift_table` and every +prompt-AdaLN MLP parameter are drawn from the same `param_spec` rule at +`scale=0.05` (`scripts/gen-ltx2-goldens.py:100-106`), so the ratio between them +is a property of the FIXTURE, not of the conditioning. Vary only the MLP init and +it moves with it: 0.005 → 4.1% / 0.49%, 0.05 (committed) → 51.7% / 5.82%, +0.2 → 1450% / 142%. What these rows are FOR is the mutation below: 73x and 37x +above round-off is what makes a zeroed term detectable. + +### What was measured — (b) the SHIPPED checkpoint, which is the answer + +Measured 2026-08-13 by loading the real tensors into upstream's own +`AdaLayerNormSingle(inner_dim, embedding_coefficient=2)` (`adaln.py:19-45`, built +by `model.py:223-227` / `:253-257`) and evaluating it on `sigma * +timestep_scale_multiplier` — the file's own config gives 1000 — exactly as +`transformer_args.py:274-278` and `:177` do, then comparing against all 48 +`prompt_scale_shift_table` / `audio_prompt_scale_shift_table` tensors it is +summed with at `transformer.py:441-443`. + +- File: `/mnt/nas_share/checkpoints/ltx-2.5/lightricks-ltx-2.5/diffusion_models/ltx-2.5-22b-distilled-transformer-nvfp4.safetensors` + (7876 tensors, 1,179,408-byte header; the prompt-AdaLN tensors are BF16 there, + so they are read directly with no dequantisation step of ours in the path). +- Upstream: `ltx_core` at `fd4ded7f`, imported BY PATH from + `/home/mudler/_git/LTX-2` with `ltx_core.__file__` asserted under that checkout. +- Sigmas: a uniform grid over the whole range, `linspace(0, 1, 101)`, and + separately the sampler the file's own scheduler config names — + `LinearQuadraticScheduler().execute(8)` (`components/schedulers.py:60-88`). + +| | video (dim 4096) | audio (dim 2048) | +|---|---|---| +| `rms\|table\|`, 48 blocks | 0.017553 | 0.021925 | +| `rms\|term\|`, uniform σ | 0.236446 | 0.347171 | +| **term/table, RMS** | **1347%** | **1583%** | +| `max\|term\|` / `max\|table\|` | **7119%** | **2817%** | +| term/table RMS, LinearQuadratic 8-step | 1275% | 1492% | + +Split by row, on the uniform grid: shift 605% (video) / 461% (audio), scale 1732% +/ 2044%. The scale row is where it concentrates, which is the row that multiplies. + +**On the shipped model the timestep term DOMINATES the static table; the table is +the perturbation.** What the pre-row renders applied was +`context * (1 + ~0.018 rms) + ~0.018` where upstream applies +`context * (1 + ~0.32 rms) + ~0.10`. Stated as the quantity actually consumed at +`transformer.py:446`, for a context of unit rms: the modulated context has rms +1.0035 static-only against 1.0915 upstream (video, **+8.8%**) and 1.0033 against +1.3170 (audio, **+31.3%**). + +**The synthetic fixture UNDERSTATES the real defect by two orders of magnitude.** +Comparing like with like — the fixture ratio is `max\|term\|` / `max\|table\|`, +and so is the shipped-weights ratio — **per stream, each against its own fixture +denominator**: + +| | fixture `max\|term\|`/`max\|table\|` | shipped | understatement | +|---|---|---|---| +| video (`prompt_adaln_single` vs `prompt_scale_shift_table`) | 51.7% | 7119% | **138x** | +| audio (`audio_prompt_adaln_single` vs `audio_prompt_scale_shift_table`) | 40.6% | 2817% | **69x** | + +**CORRECTED 2026-08-13 (issue #644): this read "138x and 54x", which is not a +per-stream pair.** The generator emits exactly ONE fixture ratio and it is the +VIDEO stream's (`gen-ltx2-goldens.py`, `measure_prompt_adaln_magnitude`), so 54x +divided the shipped AUDIO number by the VIDEO denominator. It is literally true +about the single published figure and it errs CONSERVATIVE — the real audio +understatement is larger. The audio stream's own fixture ratio, recomputed with +the identical statistic on the identical fixture (the generator is imported, so +the weight stream is bit-identical), is **40.6%** — `max|term|` 0.0201763 vs +`max|table|` 0.0496868 — giving **2817 / 40.6 = 69x**. The video row is +unchanged and was always like-for-like: `max|term|` 0.0252012 vs `max|table|` +0.0487142 = 51.7327%, and 7119 / 51.73 = 137.6. + +Reproduce the audio denominator by importing the generator and asking it the same +question the video row is built from (`measure_prompt_adaln_magnitude`), so no +weight is re-drawn: + +```python +gen.load_upstream(LTX2); torch.set_grad_enabled(False) +on = gen.build_model("split", False, prompt_adaln=True) +video, audio = gen.build_modalities(False) +scale = float(gen.ARCH["timestep_scale_multiplier"]) +amod, _ = on.audio_prompt_adaln_single((audio.sigma * scale).flatten(), hidden_dtype=torch.float32) +amod.abs().max() / on.transformer_blocks[0].audio_prompt_scale_shift_table.abs().max() +``` -The two output rows are the GATE's floor, not a claim about the trained -checkpoint: they are bounded by the generator's synthetic weight scale (0.05) and -by a 2-block stack rather than 48. They are reported because a mutation must be -shown to move something, and 73x/37x above round-off is what makes the mutation -below meaningful. +**Fixed at the source, not only in prose.** The generator emitted both video-only +rows unlabelled, which is how a video denominator came to be applied to an audio +numerator; `gen-ltx2-goldens.py` now names the stream on both, and +`ltx2_goldens.inc` was REGENERATED against `ltx_core` `fd4ded7f` to carry them. +The regeneration diff is those two comment lines and nothing else — every golden +VALUE byte-identical, and the `Regenerate with:` header unchanged because the +committed command was re-run verbatim, so this re-proves provenance as well as +the labels. `test_ltx2.cpp` says the same. No fifth golden was added: the audio +figure is a denominator for this record, not a gate floor, so "ALL FOUR ROWS" +still holds. + +Recorded because the original Outcome quoted "roughly half the magnitude" from +the fixture as if it described the checkpoint. + +Reproduce with `scripts/measure-ltx2-prompt-adaln.py --ltx2 +--checkpoint `, committed by this repair so the number is +re-runnable rather than transcribed. It asserts `ltx_core.__file__` under the +named checkout before it reads anything, and nothing of ours is in its numeric +path — the only vllm.cpp input is which tensors to read. ### The mutations @@ -316,7 +440,65 @@ re-checking its md5 (`03324d42…`, identical before and after): | `test_ltx2` | 30 cases / 1627 assertions | 35 / 2435 | +5 cases, +808 assertions | | `test_ltx2_loader` | 24 / 4817 | 26 / 4826 | +2 cases, +9 (new cases minus the assertions the retired unported-family claims took with them) | | `test_ltx2_device` | 13 / 498 | 15 / 523 | +2 cases, +25 assertions | -| `test_ltx2_video` | 30 / 502 | 30 / 502 | unchanged — the fixture now carries the module, and no assertion counted it | +| `test_ltx2_video` | 30 / 502 | 30 / 502 | unchanged — see below; 502 is the SKIPPED default | + +**What `30 / 502` does and does not say (corrected 2026-08-13).** The +shipped-checkpoint case `ltx2 video: the SHIPPED Lightricks checkpoints parse and +load` is env-gated: with `LTX2_CHECKPOINT_ROOT` unset it prints +`SKIPPED` and returns at `test_ltx2_video.cpp:1620-1625`, so `30 / 502` means the +whole real-header case DID NOT RUN — not "it ran and no assertion counted the +module". (The span starts on the `TEST_CASE` line deliberately. The four gate +lines alone are **not unique** — `ltx2 video: the SHIPPED Lightricks VAEs and +upsampler load` carries a byte-identical `LTX2_CHECKPOINT_ROOT` gate at +`:1757-1760` — so an anchor on the gate alone names two places and identifies +neither. That ambiguity was already there when the anchor read `:1316-1319`; +widening by one line to the case name, which occurs exactly once, is what makes +the citation resolvable.) With the variable pointing at the Lightricks tree the +same binary measures **30 cases / 8734 assertions**, both before and after this repair +(re-measured on this branch, exit 0 in both configurations). Any future quote of +this suite's count owes the configuration alongside it. + +**Both figures MOVED, and that is the point of quoting the date with them.** +Re-measured 2026-08-15 on this merge commit, after `LTX25-KEYFRAMES-ABS-POS` +(#658) added cases here: `LTX2_CHECKPOINT_ROOT` unset gives **37 cases / 784 +assertions**, set gives **37 / 9031**, exit 0 both ways. The CASE count is the +same in both configurations, so an unchanged case count never distinguishes them. +The 2026-08-13 figures above are kept as what was measured then. + +**AND THEY MOVED AGAIN, one merge later — the figures in this paragraph are +SUPERSEDED and are kept only as what was measured on the previous merge.** +`0785cfc4d` (#882) added 306 lines to `tests/vllm/multimodal/test_ltx2_video.cpp` +between `00613767d` and the merge that lands, taking it from **37 to 40** +`TEST_CASE`s (`grep -c '^TEST_CASE'` on both revisions; the same grep returns +matches on `origin/main`, so the count is not a failed pattern). That source +count is what is MEASURED here. The doctest CASE and ASSERTION totals in both +configurations were **NOT re-run** on this merge and are therefore UNKNOWN — not +37 / 784 and 37 / 9031. A number that has moved twice in three days is not one to +carry forward on the argument that it probably did not move a third time, and +this file's own rule is that a count owes its configuration AND its date. Owed +under [#673](https://github.com/mudler/vllm.cpp/issues/673), where this suite's +configuration debt already lives; the `set` arm needs the 18.72 GB NVFP4 and +23 GB FP8 DiTs under `$CHECKPOINT_ROOT`, so no CI host can close it. +`test_ltx2_loader.cpp` and `test_ltx2_device.cpp` are byte-identical across those +two revisions — `git diff --numstat` reports nothing on either, against a +positive control that reports `62 8` on `ltx2_video.cpp` — so their counts below +are unaffected, and `test_ltx2.cpp` differs only by this branch's own edit. + +**And CI never sets it — [#673](https://github.com/mudler/vllm.cpp/issues/673), +filed 2026-08-13 as visible debt rather than repaired here.** `grep -rn +CHECKPOINT_ROOT .github/` exits 1 with zero hits while the same pattern matches in +`tests/` and `.agents/` (positive control run in the same command, so this is not +an assertion from a failed grep). CI therefore executes **784 of 9031 assertions — +8.7%** of this suite (measured on the PREVIOUS merge, 2026-08-15; it was 502 of +8734, 5.7%, on 2026-08-13; both SUPERSEDED and un-remeasured after #882, see +above — the RATIO is the finding and it does not depend on the exact totals), at an +identical case count in both configurations, and +`scripts/measure-ltx2-prompt-adaln.py` — which produces every shipped-weights +number in this Outcome — is a manual tool no gate invokes (`grep -rn +measure-ltx2-prompt-adaln` hits only its own usage string and this file). So this +row's checkpoint-derived evidence is **manual and host-local**: reproducible only +on a box carrying the 18.72 GB NVFP4 DiT and the 23 GB FP8 DiT under +`$CHECKPOINT_ROOT`. Wiring checkpoints into CI is explicitly NOT in this row. The `test_ltx2_video` fixture had to move: it declared a config that omits `use_prompt_adaln_single` (mirroring the shipped NVFP4 DiT) while its SHAPES said @@ -354,7 +536,357 @@ config/shape equality check. the opt-in for it. What changed is that it can no longer switch a ported feature off: the loader asserts the flag against the file instead of clearing it, and `Ltx2AdoptDeclaredDitParams` clears exactly one flag, for a module nothing -applies. +applies. (**SUPERSEDED 2026-08-15**, §3.3: that last module is ported, the opt-in +is needed by neither shipped DiT, and `Ltx2AdoptDeclaredDitParams` clears +nothing.) + +### The keyframes claim next door, corrected 2026-08-13 + +**READ THE SUPERSESSION FIRST.** Everything below is the state on 2026-08-13, +when `keyframes_abs_pos_embedding` was unported and both shipped DiTs were +refused. `LTX25-KEYFRAMES-ABS-POS` +([#658](https://github.com/mudler/vllm.cpp/issues/658), `98f8e046d`) ported the +module on 2026-08-14, retired both refusals and committed +`scripts/measure-ltx2-keyframes-meta.py`, which re-runs the meta-device +observation below on demand instead of quoting it. The FINDING stands and that +port confirms it; the DISPOSITION it argues for — refuse by tensor presence, keep +the opt-in — is retired. Current disposition: +[`ltx25-keyframes-abs-pos.md`](ltx25-keyframes-abs-pos.md) §2. + +`ltx2.h` carried, in the same paragraph this row rewrote, *"LTX-2.5's checkpoint +does not carry the parameter"* about `keyframes_abs_pos_embedding`. It is FALSE — +the same class of claim as the `use_prompt_adaln_single=false` assertion this row +exists to remove — and the tree already contradicted it twice +(`.agents/model-matrix.md`, `tests/vllm/multimodal/test_ltx2_video.cpp:1608-1609`). +Read straight off both files' headers, and run through upstream's own loader and +configurator: + +| | FP8 (`vonkaiser`) | NVFP4 (first-party) | +|---|---|---| +| carries `keyframes_abs_pos_embedding` | YES — `F8_E4M3 [1, 4096]` + F32 scale | NO | +| declares the flag in `__metadata__` | **no `__metadata__` AT ALL** | `true` | +| `LTXModelConfigurator.from_metadata` | **RAISES** `KeyError: 'caption_channels'` | builds it, `[1, 4096]` | + +So the two files each contradict one half of the retired claim, and neither +supports it. Two corrections to the reasoning that came with the finding, both +measured rather than read: + +- The FP8 file does not "resolve the flag `False` at `model_configurator.py:82`". + Upstream never reaches line 82 on it: `_build_caption_projections` indexes + `caption_channels` on the empty config first and raises. That file ships no + config, so what its flag resolves to is decided entirely out of band — and the + tensor it carries is trained (`.agents/specs/ltx-2-5.md` §3.1 reads its bytes). +- On the NVFP4 file the flag IS on and the module IS built, but the tensor is + absent — 0 of its 7876 entries match `keyframes_abs_pos`. + + **CORRECTED AGAIN 2026-08-13, same issue.** This bullet said the parameter + "keeps `torch.zeros(1, inner_dim)` (`model.py:217-219`) through + `load_state_dict(..., strict=False)` (`loader/single_gpu_model_builder.py:98`) + — a genuine no-op there". **It is not a no-op.** The claim quoted that line + while dropping the `assign=True` that is ON THE SAME LINE, and upstream builds + on the **meta device** — `loader/helpers.py:84-95`, `create_meta_model`: + `with torch.device("meta"): configurator.from_metadata(...)` at `:90-91`. A key + absent from the state dict is therefore never materialised at all; it is not a + zero, it is an **unmaterialised meta parameter**. + + Run 2026-08-13 through upstream's own `create_meta_model` on this file's real + `__metadata__` (read with upstream's own `read_model_metadata` / + `SafetensorsModelStateDictLoader`, which JSON-decodes each value — + `sft_loader.py:58-74`; the flag lives at `config.transformer`, not at the top + level): + + ```text + config.transformer['use_keyframes_abs_pos_embedding'] = True + keyframes_abs_pos_embedding: shape=(1, 4096) dtype=torch.float32 device=meta is_meta=True + supports_keyframes_abs_pos_embedding (BEFORE load) : False + after load_state_dict(sd, strict=False, assign=True): + neighbour patchify_proj.weight : device=cpu is_meta=False <- materialised + keyframes_abs_pos_embedding : device=meta is_meta=True + in missing_keys : True + reading the value RAISES : RuntimeError: Tensor.item() cannot be called on meta tensors + supports_keyframes_abs_pos_embedding (AFTER load) : False + ``` + + **Upstream says exactly this itself**, and then never asks. + `supports_keyframes_abs_pos_embedding` (`model.py:166-173`) returns `False` for + "a model whose config set the flag but whose checkpoint carried no weight for it + (the parameter would still be on `meta`)", and + `enable_keyframes_abs_pos_embedding` (`model.py:175-200`) exists because such a + parameter "would fail at the first forward". **Both are defined and never + called** — one `grep -rn` hit each across the whole `Lightricks/LTX-2` checkout + at `fd4ded7f`, the definition itself, re-confirmed here rather than transcribed. + + Polarity is not what fails. `apply_keyframes_absolute_embedding` is + `hidden_states + mask * embedding` (`transformer_args.py:23-43`, the sum at + `:43`), so **real** zeros would be inert — the mechanism claim fails on `meta`, + not on additivity. + + This **strengthens** the row's conclusion and changed nothing downstream on + 2026-08-13: the refusal stayed keyed on tensor presence for the FP8 file, and + the NVFP4 file was refused by flag in `ParseLtx2DitParams` — where upstream, had + it loaded, would carry a parameter its own guard reports as unsupported. Both + refusals are GONE as of `98f8e046d` (#658), and the meta observation above is + what let that row load the NVFP4 file and apply nothing rather than refuse it or + synthesise a zero. It is now re-runnable rather than quoted: + `scripts/measure-ltx2-keyframes-meta.py`, committed by #658 and RE-RUN on this + merge commit against both shipped files, exit 0. It reproduces every line of + the transcript above and carries its own positive control — a neighbour + (`scale_shift_table`) that materialises `device=cpu is_meta=False` on the same + load, so `is_meta=True` on `keyframes_abs_pos_embedding` is the parameter's + state and not a loader that never ran. It also settles the FP8 half by + execution: 2 of 6124 keys match, `[1, 4096]` `F8_E4M3` with 4096 of 4096 bytes + NON-ZERO plus a rank-0 `F32` scale, against `__metadata__ keys : NONE`. + +It is also not a keyframe-only feature: `transformer_args.py:269` applies it on +every `prepare` whose `keyframes_mask` is set, and `tools.py:186-196` sets that +mask unconditionally on the target's first latent frame. (Diffusers' own pipeline +does not consume it — `.agents/specs/ltx-2-5.md` §3.1 records that — but `ltx_core` +is what this campaign ports, and `ltx_core` does.) + +**The refusal keying did NOT change here, and that was the decision, not an +omission.** `ltx2_loader.cpp`'s `RefuseUnported` fired on the TENSORS the file +carries. Keying it on the resolved flag instead would, on the FP8 DiT, read a +DEFAULT rather than the file — because that file declares nothing — and would +therefore have loaded it silently while discarding a trained `[1, 4096]` +parameter. Tensor presence was the only signal that file actually carries, and +refusing loudly with an opt-in was strictly safer than resolving quietly. No +behaviour changed in this row, so no new gate was owed; the refusal MESSAGE +changed, because it asserted the implication that is false in both directions. + +**RETIRED 2026-08-14 by #658.** Both refusals are gone, because the module is +ported: nothing is keyed on the tensors and nothing is keyed on the flag, and the +declared flag is now RESOLVED against what the file carries +(`Ltx2AdoptDeclaredDitParams`). The reasoning above is why the port had to settle +the NVFP4 arm by EXECUTION before it could retire either refusal. + +### The claims repair's own gate (2026-08-13) + +Nothing executable changed except three refusal MESSAGES, so the numbers are +expected to be identical and the point is that they are: + +- `BUILD_EXIT=0`; build logs grepped for `No space left|BFD assertion` — 0 hits; + `df -h /` 92% used, 37G free at the end. +- `ctest -N` = **423**. Full `ctest -j8` = 422/423 with `test_serve_low_tools` + starved under `-j` (a known parallel flake); serially **1/1 PASS, exit 0**. + 2 skipped (`test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`) as + on the baseline. +- Suite counts, unchanged from the row above: `test_ltx2` 35/2435, + `test_ltx2_loader` 26/4826, `test_ltx2_device` 15/523, `test_ltx2_video` + 30/502 skipped-default and **30/8734** with `LTX2_CHECKPOINT_ROOT` set. Exit 0 + on all five runs. +- `tests/vllm/models/ltx2_goldens.inc` REGENERATED from + `scripts/gen-ltx2-goldens.py` against `ltx_core` `fd4ded7f`: every golden VALUE + byte-identical, the diff is the comment block alone. That re-proves provenance + as well as the wording. + +An earlier full run was voided rather than reported: another session's +disk-pressure cleanup deleted `build/` while ctest was at 421/423, and the last +two tests recorded `Not Run — Failed to change working directory`. A run whose +tree vanished under it is not a result; it was rebuilt and re-run from scratch. + +### Two divergences from upstream, recorded rather than fixed + +- **We are stricter than upstream about a config that disagrees with its file.** + Upstream loads with `load_state_dict(..., strict=False, assign=True)` + (`loader/single_gpu_model_builder.py:98` — quoted in full, because dropping the + `assign=True` is exactly what made the keyframes claim above wrong), so a config + declaring + `use_prompt_adaln_single=false` over a file that carries the module would build + no module, drop 18 tensors on the floor and run flag-OFF without a word. §3.2's + equality check refuses that. Ours is better; it is still a DIVERGENCE, not a + mirror, and it is named here so it is not later mistaken for ported behaviour. +- **We refuse `keyframes_abs_pos_embedding` by tensor presence** where upstream + would take an out-of-band config's word for it (above). Same shape of + divergence, same reason it stood. **RETIRED 2026-08-14 by #658**: the module is + ported, nothing refuses it, and the declared flag is resolved against the file + the way upstream's own `supports_keyframes_abs_pos_embedding` resolves it. Only + the first divergence survives. + +### The reconciliation onto `main`, and its gate (2026-08-15) + +The branch sat 120 commits behind and conflicted on ten files. Its port half had +already landed as `65e79eee5` (#654), and `98f8e046d` (#658) then ported +`keyframes_abs_pos_embedding`, which is why four of the five SOURCE conflicts were +this branch re-asserting retired refusals. All five took `main`'s side; +`tests/vllm/models/ltx2_goldens.inc` was REGENERATED with the committed command +against `ltx_core` `fd4ded7f` (`REGEN_EXIT=0`) rather than hand-merged, and +differs from `main` by 12 insertions and 7 deletions, all comment lines: every +golden VALUE on both sides byte-identical. + +Every published number was re-derived on the merged tree rather than carried +forward, and none moved: the shipped-weights table above via +`scripts/measure-ltx2-prompt-adaln.py` (exit 0), and the audio fixture +denominator 40.6068% by importing the generator so no weight is re-drawn. + +Suite counts on the merged tree, exit 0 each, against the 2026-08-13 figures: + +| Suite | 2026-08-13 | 2026-08-15 (this merge) | +|---|---|---| +| `test_ltx2` | 35 / 2435 | 43 / 4388 | +| `test_ltx2_loader` | 26 / 4826 | 28 / 4978 | +| `test_ltx2_device` | 15 / 523 | 18 / 546 | +| `test_ltx2_video` | 30 / 502 unset, 30 / 8734 set | 37 / 784 unset, 37 / 9031 set — **SUPERSEDED, not re-run after #882 took the file to 40 `TEST_CASE`s** | + +The growth is #658's, not this reconciliation's: no `src/` or `include/` file +differs from `origin/main` on this branch. + +All five mutations re-run on this tree, each BUILT, each restored byte-for-byte: + +| # | Mutation | BUILT | Result | +|---|---|---|---| +| M1 | host `ModulateContext` ignores `prompt_mod` | YES | RED — `test_ltx2` 3/43 cases, 6/4388 assertions, exit 1 | +| M2 | device path takes the static-only branch always | YES | RED — `test_ltx2_device` 1/18, 6/546, exit 1 | +| M3 | re-add `use_prompt_adaln_single = false` before the guard | YES | RED — `test_ltx2_loader` 1/28 cases, exit 1, and `test_ltx2_video` 29/37, exit 1 | +| M4 | prompt AdaLN driven by `m.timesteps` instead of `m.sigma` | YES | RED — `test_ltx2` 2/43, 4/4388, exit 1 | +| M5 | shift and scale rows swapped in the `[2, width]` row | YES | RED — `test_ltx2` 2/43, 4/4388, exit 1 | + +**M5's first attempt was a silent NO-OP and is recorded rather than dropped.** The +harness's substitution spanned two lines and never matched, so the tree was +unchanged, the build succeeded and `test_ltx2` reported 43/43 SUCCESS — a GREEN +that says nothing about the guarantee. It was caught because the harness prints +`git diff --stat` for every mutation and that line was absent. Re-applied as a +real edit (`1 file changed, 2 insertions(+), 2 deletions(-)`) it is RED. An +unmoved mutation is not evidence of anything. + +M3 also shows the doctest counting trap: the case THROWS, so its assertions stop +being counted and the total DROPS 4978 → 4967 while `0 failed` is printed. The +exit code and the `test cases:` line are the authority, not `assertions:`. + +### Reachability, checked 2026-08-15 against `AGENTS.md` `## Nothing lands dead` + +That rule and [`reachability.md`](../reachability.md) post-date this row +(`8f49ac3be`, [#886](https://github.com/mudler/vllm.cpp/issues/886)), so the +question is answered here rather than assumed. The prompt-side AdaLN is REACHED +from a production entry point on its DEFAULT configuration, and the chain is +`file:line` at this merge commit: + +1. `include/vllm.h:962` — `vllm_video_generate`, the shipped C ABI entry point. +2. `src/capi/vllm_c.cpp:1646` — that entry point calls `VideoEngine::Generate` on + the registry-detected engine. +3. `src/vllm/multimodal/ltx2_video.cpp:1106` — `Ltx2VideoEngine::Generate`. +4. `src/vllm/multimodal/ltx2_video.cpp:1784` and `:1786` — the denoise loop calls + `Ltx2DitForwardDevice` or `Ltx2DitForward`. Both arms carry the term. +5. `src/vllm/model_executor/models/ltx2_dit.cpp:785` (host) and + `ltx2_device.cpp:1152` (device) — `prompt_adaln = params.cross_attention_adaln + && params.use_prompt_adaln_single`, and that flag DEFAULTS TRUE at + `include/vllm/model_executor/models/ltx2.h:133`. This is the default + configuration, not an opt-in. +6. `src/vllm/model_executor/models/ltx2_dit.cpp:581` — `PrepareTimestep` runs the + prompt-side MLP on the stream's own `sigma`. +7. `src/vllm/model_executor/models/ltx2_dit.cpp:205` → `:140` — + `ModulateContext` adds that row to the static table, on every block of both + streams. + +The loader half is reached the same way: `Ltx2VideoEngine::Load` +(`src/vllm/multimodal/ltx2_video.cpp:575`) calls `Ltx2LoadDitFromSafetensors` at +`:667`, so §3.2's guard runs for every real checkpoint. Both shipped DiTs take +that path with NO opt-in as of #658. + +Every anchor above was re-derived at this merge commit and asserted UNIQUE — the +quoted text matches exactly once in its file — against a positive control that +reports `STALE` on a deliberately wrong line. `git grep` alone would not have +answered this; the chain was followed by hand. + +**All seven `ltx2_video.cpp` / `test_ltx2_video.cpp` anchors in this section +MOVED under the 2026-08-15 merge of `origin/main`, and the numbers above are the +POST-merge ones.** `0785cfc4d` (#882) added 70 lines to `ltx2_video.cpp` and 306 +to `test_ltx2_video.cpp`, ahead of every anchor here: `:1063 → :1106`, +`:1730/:1732 → :1784/:1786`, `:533 → :575`, `:624 → :667`, and in the test +`:1316-1319 → :1620-1625` (widened by one line as well; see `### The gate`) and +`:1302-1303 → :1608-1609`. Each was CORRECT at +`00613767d` and each was WRONG the moment the merge landed, which is the point: +a re-derivation is only true of the tree it ran on, and the merge is part of +landing. They were caught by comparing each span's TEXT against the claim beside +it. A checker that reads the span out of the file and then looks for that span +in the same file is a tautology — it returns unique-and-at-the-cited-line for +every anchor, including the ones now pointing at unrelated code. The expected +text has to come from the CLAIM. + +**The second half of the rule is NOT satisfied, and the mutation says so.** The +rule also requires that the smallest failing test ENTER through that entry point. +It does not here. The reachability mutation from `reachability.md` — delete the +production call site, `ltx2_dit.cpp:140` `if (prompt_mod != nullptr) {` becomes +`if (false) {`, so `ModulateContext` ignores the term entirely — was run on a +scratch copy and **BUILT=YES, compile_err=NO**, so this is a test result: + +| suite | enters through | result | +|---|---|---| +| `test_ltx2` | `Ltx2DitForward`, by hand | RED — 3 of 43 cases, 6 of 4388 assertions, exit 1 | +| `test_ltx2_video` | `vllm_video_generate` / the ABI | **GREEN** — 37 of 37, 784 of 784, exit 0 (on the previous merge; #882 has since added 3 cases, and the mutation was NOT re-run — the finding is that the suite is green with the term deleted, which more cases cannot undo) | + +The entry-point suite drives the path — its fixture sets the flag TRUE +(`tests/vllm/multimodal/ltx2_video_fixture.h:258`) — and asserts no value the +term can move, so it measures that the pipeline runs rather than that this +capability is in it. Filed as +[#900](https://github.com/mudler/vllm.cpp/issues/900) and listed below rather +than repaired here: closing it means designing an ABI-level value oracle +red-first, which is a row with its own spec and not a record repair. + +The LOADER half does satisfy both halves. Re-adding +`use_prompt_adaln_single = false` in front of §3.2's guard (M3 below) takes +`test_ltx2_video` to 29 of 37 cases failing, exit 1 — that gate enters through the +production load path and observes the guard. + +### Every repo-local citation in this file, re-derived (2026-08-15) + +Not only this section's. **33 repo-local citations** were re-derived at the +merge commit that lands: 27 live (25 full-form plus the `:1786`, `:140` and +`:667` bare continuations), 4 SHA-anchored occurrences into `baa92ccf7` +(`ltx2_loader.cpp:988` once, `ltx2.cpp:274-276` three times) and 2 bare +continuations of the SHA-anchored loader claim (`:573`, `:626`). All 33 FRESH. + +The rule applied: the expected text is taken from the CLAIM, then required to +occur exactly once in the cited file at the cited revision, beginning on the +cited line. Two consequences worth stating, because each is a finding rather +than a formality: + +- `ltx2_loader.cpp:573` and `:626` are the one citation whose text is + DELIBERATELY not unique — both lines read + `out.params.use_prompt_adaln_single = false;`, which is exactly what §0 claims + ("`:573` / `:626` do the same"). Uniqueness is asserted as "exactly these two + lines", not "exactly one". +- The env-gate anchor was widened from the four gate lines to `:1620-1625` so + that it resolves at all; see the note in `### The gate`. + +The check is armed, not decorative: shifting four anchors by one line +(`ltx2_video.cpp:660`, `:1106`, `ltx2.h:133`, `ltx2.cpp:274-276`) takes it to +6 STALE of 33 — six because the `ltx2.cpp` anchor is cited three times, which is +also why SHA-anchoring it once fixed three sentences. + +Upstream citations (`model.py`, `transformer_args.py`, `transformer.py`, +`adaln.py`, `transformer_ltx2.py`, and the rest) are NOT covered by that run. +They are pinned to `fd4ded7f` / `3a2f35d4` and audited separately under +[#794](https://github.com/mudler/vllm.cpp/issues/794). + +## Owed + +- [#673](https://github.com/mudler/vllm.cpp/issues/673) — this row's + checkpoint-derived evidence is manual and host-local. `LTX2_CHECKPOINT_ROOT` is + set by no workflow, so the `test_ltx2_video` shipped-header case skips in CI and + `scripts/measure-ltx2-prompt-adaln.py` is a manual tool no gate invokes. Filed + as visible debt; wiring checkpoints into CI is explicitly not in this row. +- [#900](https://github.com/mudler/vllm.cpp/issues/900) — the prompt-AdaLN term is + REACHED from `vllm_video_generate` (chain above) but no test ENTERS through that + entry point and observes it: deleting the term's only consumer leaves + `test_ltx2_video` green at 37 of 37. That is the second half of `AGENTS.md` + `## Nothing lands dead`, whose rule and guide post-date this row (`8f49ac3be`, + #886). Closing it needs an ABI-level value oracle designed red-first. +- [#911](https://github.com/mudler/vllm.cpp/issues/911) — this spec is the worked + example for a class, not an outlier. It shipped EIGHT stale repo-local anchors + across two repair commits (`7a6165dab`, `00613767d`), every one introduced by + the row's own commits and moved by its own `020381676` and by `98f8e046d` + (#658), and then SEVEN more that were correct at `00613767d` and wrong at the + merge of `origin/main` in this commit. Spec BODIES are checked by nothing: + `check-agent-record.py`'s `MATRIX_PATHS` (`:521`, `:529-530`) covers the five + matrices, `feature-matrix.md` and `specs/model-family-inventory.md`, so 4772 + line-carrying citations across 315 `.agents/specs/*.md` are unexamined. The + rule the issue asks for is one sentence — an anchor into a file the row is + editing is stale until re-derived at the tree that LANDS, merge included — and + the two dispositions are already in use here unwritten: `path:NN @ ` for a + historical claim, claim-sourced uniqueness re-derivation for a live one. Filed + rather than fixed: a repo-wide sweep and a checker change each need their own + spec and red-before evidence, and #632's `ENG-RECORD-ANCHOR-RATCHET` is the row + that should absorb it. ## Now diff --git a/scripts/gen-ltx2-goldens.py b/scripts/gen-ltx2-goldens.py index 264aec643..ae1e03606 100644 --- a/scripts/gen-ltx2-goldens.py +++ b/scripts/gen-ltx2-goldens.py @@ -912,8 +912,13 @@ def rel(a, b): kv_off = ctx * (1 + scale_off) + shift_off kv_on = ctx * (1 + scale_on) + shift_on kabs, krel = rel(kv_off, kv_on) + # NAME THE STREAM. Both rows below are computed from `vmod` and the VIDEO + # `prompt_scale_shift_table`, and neither said so, which is how the Outcome came + # to divide a shipped AUDIO ratio by this VIDEO denominator (issue #644). The + # audio stream's own value differs -- 40.6% against this row's 51.7% -- so an + # unlabelled ratio here is a denominator waiting to be misapplied. lines.append( - f"// block 0 modulated prompt K/V: max|on-off| = {kabs:.6g} " + f"// block 0 modulated prompt K/V (VIDEO stream): max|on-off| = {kabs:.6g} " f"({krel * 100:.2f}% of max|off|)" ) # How much of the K/V modulation is timestep-conditioned at all: the MLP row @@ -921,19 +926,34 @@ def rel(a, b): static_max = float(table.abs().max()) term_max = float(vmod.abs().max()) lines.append( - f"// timestep term vs static table: max|term| = {term_max:.6g} vs " - f"max|table| = {static_max:.6g} ({term_max / static_max * 100:.1f}%)" + f"// timestep term vs static table (VIDEO stream): max|term| = {term_max:.6g} " + f"vs max|table| = {static_max:.6g} ({term_max / static_max * 100:.1f}%)" ) lines.append( - "// The two output rows are bounded by this generator's SYNTHETIC weight scale" + "// ALL FOUR ROWS ARE GATE-FLOOR NUMBERS FROM SYNTHETIC WEIGHTS. The table and" ) lines.append( - "// (0.05, param_spec above) and by a 2-block stack; they are the FLOOR the gate" + "// the prompt-AdaLN MLP are BOTH drawn at param_spec's scale=0.05 above, so every" ) lines.append( - "// needs, not a claim about the trained checkpoint. The K/V row is where the" + "// ratio here is a property of THIS FIXTURE and moves with that scale; the output" ) - lines.append("// term actually enters and is the number that answers 'does this matter'.") + lines.append( + "// rows are bounded by it AND by a 2-block stack. They are reported because a" + ) + lines.append( + "// mutation has to be shown to move something -- NOT as a claim about the trained" + ) + lines.append( + "// checkpoint. On the SHIPPED DiT the term DOMINATES the table it is added to:" + ) + lines.append( + "// rms|term|/rms|table| = 1347% video, 1583% audio, measured through upstream's" + ) + lines.append( + "// own AdaLayerNormSingle on the real weights. See" + ) + lines.append("// .agents/specs/ltx25-prompt-adaln.md section Outcome.") text = "\n".join(lines) print("prompt-AdaLN magnitude:\n" + text, file=sys.stderr) return text @@ -991,7 +1011,7 @@ def main() -> int: # Upstream's DEFAULT arm, and the one the shipped checkpoint runs. emit_prompt_adaln(out) out.write( - "// --- the MEASURED magnitude of the prompt-AdaLN term ---\n" + "// --- the prompt-AdaLN term's magnitude ON THIS SYNTHETIC FIXTURE ---\n" "// Same shared weights, same inputs, flag ON vs OFF:\n" + measure_prompt_adaln_magnitude() + "\n" diff --git a/scripts/measure-ltx2-prompt-adaln.py b/scripts/measure-ltx2-prompt-adaln.py new file mode 100755 index 000000000..8961f845c --- /dev/null +++ b/scripts/measure-ltx2-prompt-adaln.py @@ -0,0 +1,164 @@ +#!/usr/bin/env python3 +"""Measure the LTX-2.5 prompt-AdaLN timestep term against the static per-block +table, ON THE SHIPPED CHECKPOINT'S OWN WEIGHTS. + +Why this exists. `scripts/gen-ltx2-goldens.py` reports the same ratio at reduced +dimensions, but there the table AND the prompt-AdaLN MLP are both drawn from +`param_spec`'s `scale=0.05`, so the ratio it prints is a property of that fixture +and moves with the init scale. It is a gate FLOOR, not a statement about the +trained model. This script answers the other question — how much of the prompt +K/V modulation is timestep-conditioned on the model people actually run — and its +output is what `.agents/specs/ltx25-prompt-adaln.md` §Outcome quotes (issue #644). + +Nothing of ours is in the numeric path. It reads the safetensors header, pulls +six tensors per stream plus the 96 per-block tables, and runs UPSTREAM's own +module: + + ltx_core/model/transformer/adaln.py:19-45 AdaLayerNormSingle + ltx_core/model/transformer/model.py:223-227,253-257 built at embedding_coefficient=2 + ltx_core/model/transformer/transformer_args.py:274-278 driven by SIGMA, not timesteps + ltx_core/model/transformer/transformer_args.py:177 scaled by timestep_scale_multiplier + ltx_core/model/transformer/transformer.py:441-446 table + term, then context*(1+scale)+shift + ltx_core/components/schedulers.py:60-88 the sampler the file's config names + +Usage: + python3 scripts/measure-ltx2-prompt-adaln.py \ + --ltx2 ~/_git/LTX-2 \ + --checkpoint $CHECKPOINT_ROOT/ltx-2.5/lightricks-ltx-2.5/diffusion_models/\ +ltx-2.5-22b-distilled-transformer-nvfp4.safetensors +""" + +from __future__ import annotations + +import argparse +import json +import struct +import sys +from pathlib import Path + +PREFIX = "model.diffusion_model." +ADALN_KEYS = ( + "emb.timestep_embedder.linear_1.weight", + "emb.timestep_embedder.linear_1.bias", + "emb.timestep_embedder.linear_2.weight", + "emb.timestep_embedder.linear_2.bias", + "linear.weight", + "linear.bias", +) + + +def load_upstream(root: Path): + """Import `ltx_core` BY PATH from `root`, and PROVE that is what resolved. + + A pip-installed or otherwise-shadowing `ltx_core` would import silently and + every number below would describe an upstream nobody pinned — the failure + mode `.agents/specs/ltx-2-5.md` records at its "(b) byte-identical goldens" + entry. Identity is asserted, not assumed. + """ + src = root / "packages" / "ltx-core" / "src" + if not (src / "ltx_core").is_dir(): + raise SystemExit(f"no ltx_core under {src}; point --ltx2 at a Lightricks/LTX-2 checkout") + sys.path.insert(0, str(src)) + import ltx_core # noqa: PLC0415 + + resolved = Path(ltx_core.__file__).resolve() + if resolved.parent != (src / "ltx_core").resolve(): + raise SystemExit(f"ltx_core resolved to {resolved}, NOT to the checkout at {src}") + return resolved + + +def read_header(path: Path) -> dict: + """The safetensors header alone: an 8-byte length prefix and its JSON. No payload.""" + with path.open("rb") as fh: + length = struct.unpack(" float: + return float(t.pow(2).mean().sqrt()) + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--ltx2", required=True, type=Path, help="path to a Lightricks/LTX-2 checkout") + ap.add_argument("--checkpoint", required=True, type=Path, help="a shipped LTX-2.5 DiT .safetensors") + ap.add_argument("--steps", type=int, default=8, help="step count for the shipped sampler arm") + args = ap.parse_args() + + resolved = load_upstream(args.ltx2.expanduser()) + import torch # noqa: PLC0415 + from safetensors import safe_open # noqa: PLC0415 + + from ltx_core.components.schedulers import LinearQuadraticScheduler # noqa: PLC0415 + from ltx_core.model.transformer.adaln import AdaLayerNormSingle # noqa: PLC0415 + + ckpt = args.checkpoint.expanduser() + print(f"ltx_core: {resolved}") + print(f"checkpoint: {ckpt}") + + header = read_header(ckpt) + meta = header.get("__metadata__") + if not meta or "config" not in meta: + raise SystemExit(f"{ckpt} declares no __metadata__ config; this script needs the file's own geometry") + config = json.loads(meta["config"]) + tcfg = config["transformer"] + mult = tcfg.get("timestep_scale_multiplier", 1000) + blocks = tcfg["num_layers"] + dims = { + "video": (tcfg["num_attention_heads"] * tcfg["attention_head_dim"], + "prompt_adaln_single.", "prompt_scale_shift_table"), + "audio": (tcfg["audio_num_attention_heads"] * tcfg["audio_attention_head_dim"], + "audio_prompt_adaln_single.", "audio_prompt_scale_shift_table"), + } + print(f"config: num_layers={blocks} timestep_scale_multiplier={mult} " + f"cross_attention_adaln={tcfg.get('cross_attention_adaln')} " + f"use_prompt_adaln_single={tcfg.get('use_prompt_adaln_single', ' upstream default True>')} " + f"sampler={config.get('scheduler', {}).get('sampler')}") + + arms = { + "uniform sigma [0,1] x101": torch.linspace(0.0, 1.0, 101, dtype=torch.float32), + f"shipped LinearQuadratic, {args.steps} steps": LinearQuadraticScheduler().execute(args.steps), + } + + torch.set_grad_enabled(False) + with safe_open(str(ckpt), framework="pt") as fh: + for stream, (dim, sub, table_name) in dims.items(): + module = AdaLayerNormSingle(dim, embedding_coefficient=2) + module.load_state_dict( + {k: fh.get_tensor(PREFIX + sub + k).float() for k in ADALN_KEYS}, strict=True + ) + module.eval() + tables = torch.stack([ + fh.get_tensor(f"{PREFIX}transformer_blocks.{i}.{table_name}").float() + for i in range(blocks) + ]) # [blocks, 2, dim]; row 0 shift, row 1 scale (transformer.py:444) + + t_rms, t_max = rms(tables), float(tables.abs().max()) + print(f"\n=== {stream} dim={dim} blocks={blocks}") + print(f" rms|table|={t_rms:.6f} max|table|={t_max:.6f}") + for arm, sigmas in arms.items(): + term = module(sigmas * mult, hidden_dtype=torch.float32)[0] + term = term.reshape(sigmas.numel(), 2, dim) # transformer.py:443 + e_rms, e_max = rms(term), float(term.abs().max()) + print(f" [{arm}] rms|term|={e_rms:.6f} -> {100.0 * e_rms / t_rms:.0f}% of rms|table|; " + f"max|term|={e_max:.6f} -> {100.0 * e_max / t_max:.0f}% of max|table|") + for row, name in ((0, "shift"), (1, "scale")): + print(f" {name}: rms|table|={rms(tables[:, row]):.6f} " + f"rms|term|={rms(term[:, row]):.6f} " + f"-> {100.0 * rms(term[:, row]) / rms(tables[:, row]):.0f}%") + # What is actually consumed (transformer.py:446): context*(1+scale)+shift. + # For a context of unit rms with independent entries the modulated + # context has rms sqrt(mean((1+scale)^2) + mean(shift^2)). + def modulated(scale, shift) -> float: + return float(((1.0 + scale).pow(2).mean() + shift.pow(2).mean()).sqrt()) + + static = modulated(tables[:, 1], tables[:, 0]) + full = modulated(tables[:, 1].unsqueeze(0) + term[:, 1].unsqueeze(1), + tables[:, 0].unsqueeze(0) + term[:, 0].unsqueeze(1)) + print(f" modulated context on a UNIT-rms context: static-only={static:.4f} " + f"upstream={full:.4f} ({100.0 * (full - static) / static:+.1f}%)") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/vllm/models/ltx2_goldens.inc b/tests/vllm/models/ltx2_goldens.inc index 67f851dff..5769b7345 100644 --- a/tests/vllm/models/ltx2_goldens.inc +++ b/tests/vllm/models/ltx2_goldens.inc @@ -3227,16 +3227,21 @@ inline constexpr float kLtx2ForwardPromptAdalnMaskedAudio[] = { -0.09456487f, 0.0829611197f, 0.000701400219f, -0.243942767f, 0.0671276823f, 0.0765738413f, }; -// --- the MEASURED magnitude of the prompt-AdaLN term --- +// --- the prompt-AdaLN term's magnitude ON THIS SYNTHETIC FIXTURE --- // Same shared weights, same inputs, flag ON vs OFF: // DiT video output: max|on-off| = 0.000145674 (0.04% of max|off|) // DiT audio output: max|on-off| = 7.36713e-05 (0.03% of max|off|) -// block 0 modulated prompt K/V: max|on-off| = 0.0309528 (5.82% of max|off|) -// timestep term vs static table: max|term| = 0.0252012 vs max|table| = 0.0487142 (51.7%) -// The two output rows are bounded by this generator's SYNTHETIC weight scale -// (0.05, param_spec above) and by a 2-block stack; they are the FLOOR the gate -// needs, not a claim about the trained checkpoint. The K/V row is where the -// term actually enters and is the number that answers 'does this matter'. +// block 0 modulated prompt K/V (VIDEO stream): max|on-off| = 0.0309528 (5.82% of max|off|) +// timestep term vs static table (VIDEO stream): max|term| = 0.0252012 vs max|table| = 0.0487142 (51.7%) +// ALL FOUR ROWS ARE GATE-FLOOR NUMBERS FROM SYNTHETIC WEIGHTS. The table and +// the prompt-AdaLN MLP are BOTH drawn at param_spec's scale=0.05 above, so every +// ratio here is a property of THIS FIXTURE and moves with that scale; the output +// rows are bounded by it AND by a 2-block stack. They are reported because a +// mutation has to be shown to move something -- NOT as a claim about the trained +// checkpoint. On the SHIPPED DiT the term DOMINATES the table it is added to: +// rms|term|/rms|table| = 1347% video, 1583% audio, measured through upstream's +// own AdaLayerNormSingle on the real weights. See +// .agents/specs/ltx25-prompt-adaln.md section Outcome. // --- section 7: the keyframe absolute-position embedding arm --- inline constexpr const char* kLtx2KeyframesParamNames[] = { "keyframes_abs_pos_embedding", diff --git a/tests/vllm/models/test_ltx2.cpp b/tests/vllm/models/test_ltx2.cpp index 4ce9bce4a..5dd0fc1a9 100644 --- a/tests/vllm/models/test_ltx2.cpp +++ b/tests/vllm/models/test_ltx2.cpp @@ -968,12 +968,24 @@ TEST_CASE("ltx2 forward: the prompt-side AdaLN arm, with both masks") { // term. A port that accepted the flag, bound the 12 tensors and then never added // their output would reproduce the flag-OFF numbers exactly and pass nothing here. // -// Measured on this fixture (generator stderr, and the comment block at the end of -// ltx2_goldens.inc): the term is 51.7% the magnitude of the static per-block -// table it is added to, moves the block-0 modulated prompt K/V by 5.82%, and -// moves the DiT's own output by 1.46e-4 — 73x the kRoundOff floor. The bound -// below is set at 20x kRoundOff so it is comfortably inside the measured signal -// and comfortably outside f32 noise. +// WHAT THIS FIXTURE'S NUMBERS ARE, AND ARE NOT. The generator's stderr and the +// comment at the end of ltx2_goldens.inc report the VIDEO stream's term at 51.7% +// of its static per-block table — that ratio is emitted for the video stream ONLY, +// and the audio stream's own value on the same fixture is 40.6%, so the 51.7% is +// not a denominator for anything audio — the block-0 prompt K/V moving 5.82%, and +// the DiT output moving +// 1.46e-4 (73x kRoundOff). ALL FOUR are GATE-FLOOR numbers from SYNTHETIC weights, +// not a claim about the trained checkpoint: the table and the prompt-AdaLN MLP are +// both drawn at `param_spec`'s scale=0.05 (gen-ltx2-goldens.py:100-106), so every +// ratio is a property of THIS FIXTURE and moves with the init scale. +// +// On the SHIPPED DiT the term DOMINATES the table rather than halving it — +// rms|term|/rms|table| = 1347% video, 1583% audio, measured through upstream's own +// AdaLayerNormSingle on the real weights (.agents/specs/ltx25-prompt-adaln.md +// §Outcome). So this fixture UNDERSTATES the defect; it does not bound it. +// +// The bound below is set at 20x kRoundOff: comfortably inside the signal this +// fixture does produce, and comfortably outside f32 noise. TEST_CASE("ltx2 forward: the prompt-AdaLN term is LOAD-BEARING, not decoration") { const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); WeightSet set = BuildWeights(p); diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 60de0d69d..c997b61b6 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -1608,6 +1608,15 @@ TEST_CASE("ltx2 video: an ABI client loads, detects and generates through vllm.h // It is NOT interchangeable with the `vonkaiser` FP8 copy — they differ in a // TRAINED `keyframes_abs_pos_embedding` (spec section 3.1) — so the file this // case reads is named here and in every report of its result. +// WHEN QUOTING THIS SUITE'S ASSERTION COUNT, QUOTE THE CONFIGURATION WITH IT. +// This case skips by default, and it is most of the suite: with the variable +// UNSET the binary measures 37 cases / 784 assertions, and with it SET, 37 / +// 9031 — measured 2026-08-15, exit 0 both ways. So the CASE count is identical in +// both configurations and only the assertion count moves; an unchanged case count +// across a change therefore says nothing about whether the real headers were read. +// Quote the number WITH its configuration and its date: these figures move +// whenever a case is added here, and they already have (they read 30 / 502 and +// 30 / 8734 before the keyframe-bias port, issue #658). TEST_CASE("ltx2 video: the SHIPPED Lightricks checkpoints parse and load") { const char* root_env = std::getenv("LTX2_CHECKPOINT_ROOT"); if (root_env == nullptr) {