From 86422587ff0df97effcc8ad9151ad1d7f80e1608 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 14:08:33 +0000 Subject: [PATCH 1/5] spec(LTX25-PHASE-LORA): the adapter set belongs to a phase, not to the load (#1118) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four upstream pipelines build two `DiffusionStage`s from one checkpoint and give them different adapter sets — `a2vid_two_stage.py:107` against `:114`, `ti2vid_two_stages.py:140` against `:151`, `ti2vid_two_stages_hq.py:154` against `:165`, and `ic_lora.py:108` against `:119`, all read at Lightricks/LTX-2 `fd4ded7f`. This engine has exactly one placement: `dit_options.loras` is pushed at `ltx2_video.cpp:831` and nowhere else, so every phase of every recipe runs the same fused weights. The spec costs three shapes and chooses the third. A second resident `Ltx2DitWeights` doubles the DiT and is not viable on one GB10, which has no swap and no OOM line. Unfused runtime LoRA is `Wx + s*B(Ax)` where upstream is `round_bf16(W + s*BA)x`, so it would change every arm's numerics to serve one recipe. Re-materializing the affected tensors at the phase boundary is exact and costs no resident memory, and on this fleet wall-clock is not the constraint that reboots the box. Committed before the implementation so the order proves the spec came first. The proving consumer is `a2vid_two_stage`, not the IC-LoRA refusal the dispatch proposed: that refusal gives two reasons, and its first one — the reference clip's pixel path — is untouched by this row. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-a2vid-recipe.md | 11 +- .agents/specs/ltx25-phase-lora.md | 312 ++++++++++++++++++++++++++++ 2 files changed, 320 insertions(+), 3 deletions(-) create mode 100644 .agents/specs/ltx25-phase-lora.md diff --git a/.agents/specs/ltx25-a2vid-recipe.md b/.agents/specs/ltx25-a2vid-recipe.md index e360677e8..024a9ceee 100644 --- a/.agents/specs/ltx25-a2vid-recipe.md +++ b/.agents/specs/ltx25-a2vid-recipe.md @@ -439,9 +439,14 @@ Report `NEEDS_DECISION` rather than narrowing silently if: ## Owed - **[#1118](https://github.com/mudler/vllm.cpp/issues/1118) — the per-phase - adapter.** §4.4. Owned by this row. It also bounds - [#1093](https://github.com/mudler/vllm.cpp/issues/1093) and - [#921](https://github.com/mudler/vllm.cpp/issues/921). + adapter.** §4.4. Filed by this row and CLOSED by row `LTX25-PHASE-LORA` + ([`ltx25-phase-lora.md`](ltx25-phase-lora.md)), which puts the distilled + adapter on stage 2 alone and makes `A2VidTwoStageRecipe`'s stage 1 run the + base weights, as `a2vid_two_stage.py:107` does against `:114`. That row also + unbounds [#1093](https://github.com/mudler/vllm.cpp/issues/1093) and + [#921](https://github.com/mudler/vllm.cpp/issues/921) on this seam; both + remain blocked on their own checkpoints and, for #921, on a per-phase adapter + STRENGTH that the seam deliberately does not carry. - **A real-checkpoint A2V render.** Gated on fixtures only. The artifacts exist now — `/usr/local/nas_share/checkpoints/ltx-2.5/lightricks-ltx-2.5/` holds the NVFP4 DiT, both video VAEs, the audio VAE, both upscalers and diff --git a/.agents/specs/ltx25-phase-lora.md b/.agents/specs/ltx25-phase-lora.md new file mode 100644 index 000000000..8b9777b1a --- /dev/null +++ b/.agents/specs/ltx25-phase-lora.md @@ -0,0 +1,312 @@ +# LTX25-PHASE-LORA — the adapter set belongs to a PHASE, not to the load + +Row `LTX25-PHASE-LORA`. Issue +[#1118](https://github.com/mudler/vllm.cpp/issues/1118). Campaign +[`ltx-2-5.md`](ltx-2-5.md), under roadmap row `ROAD-V1-LTX25`. + +Upstream pin: Lightricks/LTX-2 `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca`, +verified with `git rev-parse HEAD` in `/home/mudler/_git/LTX-2` on 2026-08-17. + +## Now + +`ACTIVE` -> `DONE` with this change. The row exists because #1118 blocks #1093 +and #1096 and bounds #921; it is the seam those three need, and it closes the +divergence #1117 shipped and filed in the same breath. + +## Scope + +**In.** One seam: a phase declares which of the load's adapters it runs, and the +engine brings the resident DiT to that state at the phase boundary. One +consumer: `a2vid_two_stage`, whose stage 1 upstream runs WITHOUT the distilled +adapter. One record repair: the reference-conditioning refusal in +`ltx2_video.cpp` gives two reasons and the second becomes false here. + +**Out.** `ti2vid_two_stages` (#1093) and `keyframe_interpolation` (#1096) — both +additionally need checkpoints that are not on the NAS. The HQ arm's per-phase +STRENGTH (#921). N-adapter subsets. The reference clip's pixel path, which is +the reference refusal's FIRST reason and stays refused. + +## Upstream chain + +Four pipelines build two `DiffusionStage`s from ONE checkpoint and give them +DIFFERENT adapter sets. Each line below was read at the pin. + +| Pipeline | stage 1 | stage 2 | +|---|---|---| +| `a2vid_two_stage.py` | `loras=tuple(loras)` (`:107`) | `(*loras, *distilled_lora)` (`:114`) | +| `ti2vid_two_stages.py` | `loras=tuple(loras)` (`:140`) | `(*tuple(loras), *distilled_lora)` (`:151`) | +| `ti2vid_two_stages_hq.py` | `(*loras, distilled_lora_stage_1)` (`:154`) | `(*loras, distilled_lora_stage_2)` (`:165`) | +| `ic_lora.py` | `loras=tuple(loras)` (`:108`) | `loras=()` (`:119`) | + +The two `DiffusionStage.from_checkpoint` calls are `ti2vid_two_stages.py:136` +and `:147`, and `ic_lora.py:104` and `:115`. `ic_lora.py` is the mirror image of +the other three: the adapter rides stage 1 and stage 2 runs bare. + +`distilled.py:131` builds ONE stage set, which is why `distilled_two_stage`, +`dfr` and `retake` have never needed this seam and why no gate here has ever +asked the question. **Those recipes must not change**, and the default value of +the new field is chosen so that they cannot. + +The fusion arithmetic itself is unchanged and already ported: +`loader/fuse_loras.py:99-116` and `:119-150`, anchored in +[`ltx2_lora.h`](../../include/vllm/model_executor/models/ltx2_lora.h). + +## Our baseline + +Measured on `d1e5e9bc0`, the base this row branches from. + +- `git grep -n 'dit_options.loras' -- src` returns exactly ONE line, + `src/vllm/multimodal/ltx2_video.cpp:831`. Positive control: + `git grep -c loras -- src` returns five FILES, seven hits of them in + `ltx2_video.cpp`. (The dispatch that opened this row said "5 in that same + file"; the five is a file count.) +- That one site runs inside `Ltx2VideoEngine::Load`, under + `if (!lora_path.empty())`, and feeds + `Ltx2LoadDitFromSafetensors` / `Ltx2StreamDitToDevice` at `:833-834`. +- `Ltx2DitCheckpoint im.dit` (`ltx2_video.cpp:593`) is built once and outlives + every generation. The phase loop (`:2775`) and the DiT forward inside it + (`:3599-3602`) read `im.dit.weights` with no notion of a phase. +- `Ltx2PhaseRecipe` (`ltx2_pipeline.h:606-629`) carries thirteen per-phase + fields and none of them names an adapter. +- `ltx2_pipeline.h:686-691` already states the gap in the tree's own words, and + names #1118: "What this flag CANNOT express is upstream's placement". + +So the defect is not that the adapter is unreachable. It is that the adapter has +exactly one placement, and upstream has four. + +## Port map + +### The field + +`Ltx2PhaseRecipe` gains `Ltx2PhaseLoraScope loras = kAllAdapters`. + +The enum has two enumerators, and two is the whole space this engine can select +because `Ltx2ResolveLoraReferenceFactors` refuses more than one adapter by name +(`ltx2_lora.h:167-172`, mirroring `dubit.py:364-365` and +`hdr_ic_lora.py:271-272`): + +- `kAllAdapters` — the phase runs every adapter the load supplied. This is the + DEFAULT, so every recipe that exists today keeps the behaviour it was gated + with, and `distilled.py:131`'s single stage set stays single. +- `kNoAdapters` — the phase runs the base weights. + +**A per-phase STRENGTH is deliberately NOT added.** `ti2vid_two_stages_hq.py` +needs one (0.25 at `:92-96`, 0.5 at `:97-101`), and no recipe this row ships +would set it, so adding the field now lands a branch nothing can select — the +argument `ltx2_lora.h:41-44` already makes for the second product form. Owed +below, against #921. + +### The mechanism + +`Ltx2RebindDitLoras`, new in `ltx2_loader.h` / `.cpp`, brings an +ALREADY-LOADED checkpoint to the adapter state a given phase wants: + +```cpp +void Ltx2RebindDitLoras(vt::Queue* queue, const SafetensorsFile& file, + const Ltx2DitLoadOptions& options, bool fuse, + Ltx2DitCheckpoint& checkpoint); +``` + +For every contract tensor the adapters target it re-materializes the tensor from +`file` through the same `MaterializeDitTensor` the load uses, fuses the adapters +into it when `fuse`, and writes the result back **into the buffer the view +already points at**. The view pointer never moves, so `checkpoint.weights` — a +pure view struct (`ltx2.h:273-295`) — stays valid and is not re-bound. `queue` +is non-null exactly when the checkpoint was staged to a device, in which case the +write back is a `Copy` into the same device allocation. + +Three properties make this the shape this row chose: + +1. **It is EXACT.** Each phase gets `round_bf16(W + delta)` computed from the + pristine base, through the same code path the load uses. Not + `Wx + s*B(Ax)`. +2. **It costs no extra RESIDENT memory.** One tensor of scratch, plus the + adapter's own A/B factors for the duration of the rebind. No second weight + set ever exists. +3. **It is one call site.** `FuseLorasInto` (`ltx2_loader.cpp:536-547`) already + collapses both load arms onto one `Ltx2FuseLoraIntoTensor` call; the rebind + reuses it rather than writing a parallel fuse. + +### The consumer + +`A2VidTwoStageRecipe` (`ltx2_pipeline.cpp:1510`) sets +`stage1.loras = kNoAdapters` (`a2vid_two_stage.py:107`) and leaves stage 2 at +the default (`:114`). + +Reachable from a production entry point on its default configuration: +`include/vllm.h` -> `LoadVideoEngine` with the documented `pipeline_kind` and +`lora_path` load extras -> `Ltx2VideoEngine::Generate`. `ltx2-gen +--pipeline-kind a2vid_two_stage --lora-path ... --audio-path ...` is the same +two calls through the ABI. The `/v1/videos` route cannot drive it, because +`VideoGenParamsFromRequest` never writes `gen.extras` (#928) — stated so the +reach claim excludes it rather than overstating it. + +**Which upstream list our one slot IS.** For this recipe the supplied adapter is +upstream's `distilled_lora`, which is what `requires_distilled_lora` +(`ltx2_pipeline.h:692`) already declares by mirroring `--distilled-lora +required=True` (`utils/args.py:1140-1153`). Upstream's separate user `loras` +list, which rides BOTH stages, has no spelling here and cannot until the +adapter arity refusal lifts. Owed below. + +### The record repair + +`ltx2_video.cpp`'s reference-conditioning refusal gives TWO reasons. Reason 2 — +"THE REFERENCE ITEM BELONGS TO STAGE 1, AND STAGE 2 MUST RUN UNFUSED ... this +engine holds ONE `Ltx2Dit`, fused at load, that every phase of the recipe runs" +— becomes FALSE with this change and is rewritten to say what closed it, in the +form that block already uses for its three earlier ruled-out reasons. + +**The refusal is NOT retired.** Reason 1, the reference clip's pixel path +(`iclora_utils.py:112-117`, `:87-89`, `:144-148`), is untouched by this row and +still holds. Retiring the whole refusal here would ship an arm whose geometry +nothing supplies. The dispatch that opened this row proposed retiring it as the +proving consumer; that is rejected on the evidence above and `a2vid_two_stage` is +the consumer instead. + +## Tests to port + +Upstream has no test for adapter placement — the placement IS the pipeline +constructor, and `ltx-pipelines` ships no unit test over it. So these are ported +in the sense that every assertion cites the upstream line it mirrors, and the +harness is ours. + +### The distinguishing gate + +`tests/vllm/multimodal/test_ltx2_video.cpp`, in the a2vid section, driven +entirely through `LoadVideoEngine` and `Generate`: + +| Render | `lora_strength` | `max_phase` | Assertion | +|---|---|---|---| +| A | 1.0 | 0 | `A == B` | +| B | 0.0 | 0 | — | +| C | 1.0 | (none) | `C != D` | +| D | 0.0 | (none) | — | + +`max_phase` is a documented LOAD extra (`kLtx2MaxPhaseExtra`, +`ltx2_video.h:202`, read at `ltx2_video.cpp:1029`), so `max_phase = 0` renders +stage 1 ALONE through the production path. Strength `0.0` is the control rather +than "no adapter", because `requires_distilled_lora` refuses an a2vid load with +no `lora_path` at all; strength 0 fuses a zero delta and is already gated as +equal to the base model ("the IC-LoRA strength reaches the PIXELS, and 0 is a +no-op"). + +**Both rows are load-bearing, and either one alone is passable by a defect.** +`A == B` says the adapter is NOT on stage 1 — it REDs under today's load-time +fusion. `C != D` says the adapter IS on stage 2 — it REDs under an +implementation that simply stopped fusing. A gate asserting only "a LoRA was +applied" passes on the current defect, which is why neither row stands alone. + +### The exactness gate + +`tests/vllm/models/test_ltx2_loader.cpp`: load a checkpoint with the adapter and +without it, keep both byte images, then on a third checkpoint rebind off and +compare byte-for-byte against the unfused image, and rebind on and compare +byte-for-byte against the fused image. This is what makes the "exact, not +`Wx + s*B(Ax)`" claim executable rather than asserted, and it fails on any +rebind that reconstructs the base by SUBTRACTING the delta instead of +re-materializing it. + +### Mutations required to pass + +1. `stage1.loras = kNoAdapters` deleted from `A2VidTwoStageRecipe` — reverts to + single fusion. The distinguishing gate must go RED. +2. The rebind call deleted from the phase loop. Must go RED. +3. `dit_options.loras.push_back` deleted (the standing reachability mutation, + `ltx25-ic-lora.md` section 5.3). Must go RED. + +Every mutation prints four facts: `git diff --stat`, whether it BUILT, the +compile-error count, and the exit code captured directly. A mutation that fails +to build, or that never applied, reads as a passing test. + +## Gates + +```sh +cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF +cmake --build build -j6 && ctest --test-dir build -j4 --output-on-failure +``` + +Whole binaries, never a `--test-case` filter: a filter matching zero cases +prints `SUCCESS!` at exit 0, and at least one LTX case name contains a comma, +which doctest `-tc` splits on. Report `CONFIGURE_EXIT`, `BUILD_EXIT`, the +`: error:` count, `ctest -N`, `CTEST_EXIT`, the pass/fail line, and +`No space left` / `BFD` greps with positive controls, plus load and free disk. + +`READER ANCHORS` (`ltx2_video.cpp:386-387`) is gated by `test_ltx2_video` and +shifts whenever the readers above line 1335 move. `Load` is edited by this row, +so the list WILL move. Re-derive with the test's own walk and paste the list it +prints; arm the instrument first by inserting a line above the anchors and +confirming MISMATCH. + +**No GPU.** This fleet is leased with `rc`, and a weight-lifetime seam is +correctly gated by the CPU goldens. No real-weights result is claimed by this +row. + +## Dependencies + +- #1117 (`LTX25-A2VID-RECIPE`), landed at `d1e5e9bc0`. This row's consumer is + its recipe, and its `requires_distilled_lora` flag is what makes the supplied + adapter identifiable as the distilled one. +- #923 (`LTX25-IC-LORA`), landed. Supplies `ltx2_lora.h` and the one fuse site. + +Blocks #1093, #1096. Bounds #921. + +## Work breakdown + +1. Spec, committed before any product code. +2. The enum and the field, defaulted so no existing recipe moves. +3. `Ltx2RebindDitLoras` and its exactness test. +4. The a2vid consumer and the distinguishing gate. +5. The reference-refusal reason-2 rewrite. +6. Anchors re-derived, mutations run, full gate. + +## Risks and decisions + +**Three shapes were costed. The third was chosen.** + +| Shape | Resident memory | Exactness | Verdict | +|---|---|---|---| +| A second resident `Ltx2DitWeights` | DOUBLES the DiT: 18.7 GB nvfp4, 21.0 GB fp8, ~39 GB bf16 | exact | REJECTED | +| Unfused runtime LoRA, phase-selectable | + the adapter | `Wx + s*B(Ax)` against upstream's `round_bf16(W + s*BA)x` | REJECTED | +| Re-materialize the affected tensors at the phase boundary | none | exact | CHOSEN | + +Shape A is not viable on one GB10: 119 GB unified, `vm.overcommit_memory=1`, +zero swap, and no OOM line — the box goes down when memory is oversubscribed, +twice in the week this row was written. Shape B is a rounding divergence AND a +different GEMM path, so it would change every arm's numerics to serve one +recipe. Shape C spends wall-clock at a boundary that happens once or twice per +render, and on this fleet wall-clock is not the constraint that reboots the box. + +**What shape C actually costs, stated rather than waved at.** The rebind +re-reads the adapter file and re-materializes only the tensors that adapter +targets, so the transient peak is the resident DiT plus the adapter's A/B +factors plus one tensor. It is not free, and it is not a full reload: every +tensor no adapter names is untouched, because for those the fused and unfused +images are equal by construction. + +**Why re-materialize rather than subtract the delta.** Subtracting would be one +pass over the target tensors with no file read, and it would be WRONG: +`round_bf16(round_bf16(W + d) - d)` is not `W`. The exactness gate is written to +fail that implementation specifically. + +**Why the default is `kAllAdapters`.** Any other default silently changes +`distilled_two_stage`, `dfr`, `retake`, `one_stage` and `res2s`, all of which are +gated, and upstream gives all of them one stage set (`distilled.py:131`). A +default that moves a landed arm would be the same class of defect this row +exists to fix. + +## Owed + +- **Per-phase adapter STRENGTH**, which `ti2vid_two_stages_hq.py:92-101` needs + and this field cannot express. Owned by #921. +- **N-adapter per-phase SUBSETS**, upstream's `(*loras, *distilled_lora)` where + `loras` is the user's own list and rides both stages. Blocked on the adapter + arity refusal (`ltx2_lora.h:167-172`), which is upstream-faithful for the + pipelines that take exactly one and is not lifted here. +- **A real-weights comparison against upstream's own render** on the same + checkpoint, take and seed — upstream's stage 1 on base weights against ours. + This is the instrument `ltx25-a2vid-recipe.md` section 4.4 named, and it needs + the distilled adapter checkpoint, which `find /mnt/nas_share/checkpoints + -iname '*lora*'` returns nothing for. Owed against #1093's checkpoint + dependency. +- **The reference clip's PIXEL path**, reason 1 of the refusal this row + narrows. Still owed by #975. From 851b15da7cd64c23bd9be40641898e5931b8d67d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 14:40:43 +0000 Subject: [PATCH 2/5] feat(LTX25-PHASE-LORA): the adapter set belongs to a phase, and the DiT moves between them (#1118) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Ltx2PhaseRecipe::loras` carries upstream's per-stage adapter set, and `A2VidTwoStageRecipe` gives stage 1 `kNoAdapters` — `loras=tuple(loras)` at `a2vid_two_stage.py:107` against `(*tuple(loras), *tuple(distilled_lora))` at `:114`. Until now `dit_options.loras` was pushed once at load and every phase of every recipe ran the same fused weights, so stage 1's guided schedule ran against base + distilled where upstream runs it against the base alone. That rendered: the frames differed while the frame count, the shapes and the sample rate did not, which is why nothing caught it. UPSTREAM HOLDS ONE TRANSFORMER, NOT TWO. Both `DiffusionStage.from_checkpoint` calls name the same `model_paths.transformer()` (`:104` and `:116`, `ti2vid_two_stages.py:137` and `:148`) and differ only in the adapter tuple, so what upstream pays is a second MATERIALIZATION rather than a second resident model. `Ltx2RebindDitLoras` is that: it re-materializes from the pristine file the tensors an adapter targets, fuses the phase's set back in, and writes into the buffer the view already points at, so no pointer moves and no second weight set ever exists. A second resident `Ltx2DitWeights` would have been a heavier architecture than the reference and would not fit one GB10; unfused runtime LoRA would be `Wx + s*B(Ax)` against upstream's `round_bf16(W + s*BA)x`, a rounding divergence and a different GEMM path. The rebind reconstructs the base by RE-READING it, never by subtracting the delta: `round_bf16(round_bf16(W + d) - d)` is not `W`, and the loader gate compares byte-for-byte against a fresh load in both directions so that implementation fails. The scope field is a SET and not a boolean because upstream needs two placements: stage 2 only for TI2Vid, A2Vid and Keyframe (`ltx-pipelines/CLAUDE.md:48`), both stages for HQ and DFR (`:49`, `:50-51`). Two enumerators are the complete space while the adapter arity is capped at one. `kAllAdapters` is the default, so `distilled_two_stage`, `dfr`, `retake`, `one_stage`, `res2s` and `t2a_one_stage` keep the behaviour they were gated with, which is upstream-correct since `distilled.py:131` builds one stage set. The reference-conditioning refusal gave two reasons and the second is now false; it says so and names what closed it, keeping reason one, the reference clip's pixel path, which this row does not touch. The refusal therefore stands. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-phase-lora.md | 50 ++++- docs/FEATURES.md | 2 +- docs/USAGE.md | 25 ++- .../vllm/model_executor/models/ltx2_loader.h | 64 +++++++ .../model_executor/models/ltx2_pipeline.h | 70 ++++++- .../model_executor/models/ltx2_loader.cpp | 106 +++++++++++ .../model_executor/models/ltx2_pipeline.cpp | 14 ++ src/vllm/multimodal/ltx2_video.cpp | 114 ++++++++++-- tests/vllm/models/test_ltx2_loader.cpp | 176 ++++++++++++++++++ tests/vllm/multimodal/test_ltx2_video.cpp | 122 ++++++++++++ 10 files changed, 700 insertions(+), 43 deletions(-) diff --git a/.agents/specs/ltx25-phase-lora.md b/.agents/specs/ltx25-phase-lora.md index 8b9777b1a..d4cd9947a 100644 --- a/.agents/specs/ltx25-phase-lora.md +++ b/.agents/specs/ltx25-phase-lora.md @@ -269,12 +269,50 @@ Blocks #1093, #1096. Bounds #921. | Unfused runtime LoRA, phase-selectable | + the adapter | `Wx + s*B(Ax)` against upstream's `round_bf16(W + s*BA)x` | REJECTED | | Re-materialize the affected tensors at the phase boundary | none | exact | CHOSEN | -Shape A is not viable on one GB10: 119 GB unified, `vm.overcommit_memory=1`, -zero swap, and no OOM line — the box goes down when memory is oversubscribed, -twice in the week this row was written. Shape B is a rounding divergence AND a -different GEMM path, so it would change every arm's numerics to serve one -recipe. Shape C spends wall-clock at a boundary that happens once or twice per -render, and on this fleet wall-clock is not the constraint that reboots the box. +**The premise that shape A mirrors upstream is FALSE, and correcting it removes +the only argument that made shape A look principled.** The dispatch that opened +this row, and #1118 itself, both describe upstream as paying "two +`DiffusionStage.from_checkpoint` calls" as though it held two models. It does +not. Both calls name the SAME file: + +```text +a2vid_two_stage.py:103 self.stage_1 = DiffusionStage.from_checkpoint( +a2vid_two_stage.py:104 model_paths.transformer(), +a2vid_two_stage.py:107 loras=tuple(loras), +a2vid_two_stage.py:114 stage_2_loras = (*tuple(loras), *tuple(distilled_lora)) +a2vid_two_stage.py:115 self.stage_2 = DiffusionStage.from_checkpoint( +a2vid_two_stage.py:116 model_paths.transformer(), +a2vid_two_stage.py:119 loras=stage_2_loras, +``` + +`ti2vid_two_stages.py:136`/`:147` has the identical shape. So upstream pays two +MATERIALIZATIONS of one checkpoint with different adapter tuples, which is what +shape C is. Shape C is therefore the faithful mirror rather than the cheap +substitute, and shape A would invent a heavier architecture than the reference. + +Shape A is also not viable on one GB10 regardless: 119 GB unified, +`vm.overcommit_memory=1`, zero swap, and no OOM line — the box goes down when +memory is oversubscribed, twice in the week this row was written. Shape B is a +rounding divergence AND a different GEMM path, so it would change every arm's +numerics to serve one recipe. Shape C spends wall-clock at a boundary that +happens once or twice per render, and on this fleet wall-clock is not the +constraint that reboots the box. + +**The field is a per-phase SET, not a per-phase boolean, and the difference is +load-bearing.** Upstream needs two placements, not one: + +| Placement | Pipelines | `ltx-pipelines/CLAUDE.md` | +|---|---|---| +| adapter on stage 2 ONLY | TI2Vid, A2Vid, Keyframe | `:48` | +| adapter on BOTH stages | HQ, DFR | `:49`, `:50-51` | + +`Ltx2PhaseLoraScope` expresses both: stage 1 `kNoAdapters` with stage 2 +defaulted gives the first, and both phases defaulted gives the second. Two +enumerators are the COMPLETE space here rather than a boolean standing in for a +set, because the adapter arity is capped at one by a gated refusal +(`ltx2_lora.h:167-172`, case "ltx2 lora: more than one adapter refuses BY +NAME"), so the powerset of the load's adapters has exactly two members. The day +that cap lifts, the third value goes here. **What shape C actually costs, stated rather than waved at.** The rebind re-reads the adapter file and re-materializes only the tensors that adapter diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 70a1b5f78..4f0b6b15f 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -178,7 +178,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 ([#1039](https://github.com/mudler/vllm.cpp/issues/1039)) | | LTX-2.5 VIDEO guidance | LTX-2.5 DiT, joint video+audio | `test_ltx2_video`'s `ltx2 one_stage:` cases through `Generate`; all FOUR arms carry the x0 invariant and the guider output replays EXACTLY | `--pipeline-kind one_stage` runs `_guided_denoise`: 4 forwards/step, combined per modality in **x0**. Was ONE unguided forward, every `video_guidance` field dead ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) | | LTX-2.5 cross-attention perturbations | LTX-2.5 DiT | `test_ltx2_video` gates each direction ALONE, on a forward where one stream is PRESENT but DISABLED so only that one runs: the flag moves the stream it writes, the other leaves it bit-identical. Swapping the two is RED | `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN` ported, which `modality_scale = 3.0` selects on every video row. The DEVICE forward takes no perturbations, so that pass is refused there by name | -| LTX-2.5 audio-to-video (`A2VidPipelineTwoStage`) | LTX-2.5 DiT + audio VAE encoder + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video`'s `ltx2 a2vid:` cases through `LoadVideoEngine`+`Generate`; the take's latent is bit-identical across SEEDS and moves with the WINDOW | `--pipeline-kind a2vid_two_stage`. Guided half-res stage 1, DERIVED schedule, plain Euler; distilled 3-sigma stage 2. `--audio-path` and `--lora` REQUIRED; adapter placement diverges (#1118) | +| LTX-2.5 audio-to-video (`A2VidPipelineTwoStage`) | LTX-2.5 DiT + audio VAE encoder + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video`'s `ltx2 a2vid:` cases through `LoadVideoEngine`+`Generate`; the take's latent is bit-identical across SEEDS and moves with the WINDOW | `--pipeline-kind a2vid_two_stage`. Guided half-res stage 1, DERIVED schedule, plain Euler; distilled 3-sigma stage 2. `--audio-path` and `--lora` REQUIRED; the distilled adapter rides stage 2 ALONE (#1118) | | LTX-2.5 guidance knobs | LTX-2.5 request surface | `test_ltx2_video` renders with an override and refuses one on a fixed recipe | Seven video/audio guider extras mirroring `default_1_stage_arg_parser`, plus a negative embeds pair for a tower-less engine. Refused whole on `distilled_two_stage` and `retake`, whose guidance is distilled in | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | MTP speculation DEPTH (`num_speculative_tokens` > 1) | Qwen3.5/3.6 `mtp.*` heads | k=1..4 through the loader, greedy tokens unmoved, two witnesses per arm: the draft decode forwards the propose RAN, and whether the DELIVERED draft row varied with depth. `test_mtp_depth` 5/5, 63 assertions | Default stays k=1. NO speed claim at k>1. Drafts are proposed and verified, never ACCEPTED, and neither witness proves per-column provenance. Both await the owed DGX gate (#81) | diff --git a/docs/USAGE.md b/docs/USAGE.md index 3816f41b7..51f0327a6 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -862,8 +862,10 @@ existing key in place, so `--lora a --lora b` leaves one `lora_path` extra holding `b`, silently fuses `b`, and exits 0. Pass one adapter. The C ABI cannot reach it either, and that is the wider half of the finding: -`ltx2_video.cpp:813` is the ONLY `dit_options.loras.push_back` in the tree and it -runs at most once, under `if (!lora_path.empty())`. So `loras.size()` is 0 or 1 +`Ltx2VideoEngine::Load` carries the ONLY `dit_options.loras.push_back` in the +tree and it runs at most once, under `if (!lora_path.empty())` — named by symbol +rather than by line, because the line moved with #1118 and a stale anchor is what +this paragraph already had to correct once. So `loras.size()` is 0 or 1 on every production path — CLI, `vllm_video_engine_load` and the server alike — and the more-than-one refusal is reached only by `test_ltx2_lora`. It is correct code guarding a state nothing can currently construct, which is the shape @@ -3301,14 +3303,17 @@ Three things this kind demands, each refused by name rather than defaulted: defaults to the clip's own duration. A take shorter than the clip is refused rather than padded, and a longer one keeps its leading frames. -**One divergence, and it is not repairable from the request.** Upstream fuses -the distilled adapter into stage 2 alone and leaves stage 1 on the base weights; -this engine fuses adapters once at load, so stage 1 sees it too. Expect frames -that differ from the ones upstream renders for the same checkpoint, take and -seed. Nothing in the shape of the output shows it — the clip comes back at the -size, frame count and sample rate you asked for, and no error is raised — so the -only instrument that sees this is a side-by-side render against upstream. -Tracked as [#1118](https://github.com/mudler/vllm.cpp/issues/1118). +**The distilled adapter rides stage 2 alone**, as upstream's does: stage 1 is +built with `loras=tuple(loras)` (`a2vid_two_stage.py:107`) and stage 2 with +`(*tuple(loras), *tuple(distilled_lora))` (`:114`), and +`ltx-pipelines/CLAUDE.md:48` states the convention for TI2Vid, A2Vid and +Keyframe alike. Until 2026-08-17 this page recorded the opposite as an +unrepairable divergence, because adapters fused once at load and every phase saw +them; [#1118](https://github.com/mudler/vllm.cpp/issues/1118) closed that. The +engine still holds ONE DiT — upstream does too, since both of its +`from_checkpoint` calls name the same `model_paths.transformer()` — and +re-materializes the adapter's target tensors at the phase boundary instead of +keeping a second weight set. The guider flags (`--video-cfg-guidance-scale` and the rest, spelled as the `video_cfg_guidance_scale` extras over the C API) reach stage 1 and are ignored diff --git a/include/vllm/model_executor/models/ltx2_loader.h b/include/vllm/model_executor/models/ltx2_loader.h index ab96b7f0a..f4424fbc7 100644 --- a/include/vllm/model_executor/models/ltx2_loader.h +++ b/include/vllm/model_executor/models/ltx2_loader.h @@ -461,6 +461,70 @@ void Ltx2WidenDitToF32(Ltx2DitCheckpoint& checkpoint); Ltx2DitCheckpoint Ltx2StreamDitToDevice(vt::Queue& queue, const SafetensorsFile& file, const Ltx2DitLoadOptions& options = {}); +// Bring an ALREADY-LOADED checkpoint to the adapter state one PHASE wants. +// +// ─── WHY THIS EXISTS ───────────────────────────────────────────────────────── +// +// `options.loras` is a property of the LOAD, and upstream's is a property of the +// STAGE. Four pipelines build two `DiffusionStage`s from one checkpoint and hand +// them different adapter sets, read at Lightricks/LTX-2 fd4ded7f: +// +// a2vid_two_stage.py:107 `loras=tuple(loras)` +// a2vid_two_stage.py:114 vs `(*loras, *distilled_lora)` +// ti2vid_two_stages.py:140 `loras=tuple(loras)` +// ti2vid_two_stages.py:151 vs `(*tuple(loras), *distilled_lora)` +// ti2vid_two_stages_hq.py:154 `(*loras, distilled_lora_stage_1)` +// ti2vid_two_stages_hq.py:165 vs `(*loras, distilled_lora_stage_2)` +// ic_lora.py:108 `loras=tuple(loras)` +// ic_lora.py:119 vs `loras=()` <- the mirror image +// +// Upstream pays for that with a SECOND `from_checkpoint` call against THE SAME +// FILE — `model_paths.transformer()` at `a2vid_two_stage.py:104` and `:116`, +// `ti2vid_two_stages.py:137` and `:148` — differing only in the adapter tuple. +// It is a second MATERIALIZATION, not a second model, and that is exactly what +// this function is. A second resident weight set would be a heavier +// architecture than the reference: the DiT is 18.7 GB nvfp4, 21.0 GB fp8 and +// ~39 GB bf16, against one GB10's 119 GB of unified memory with no swap. +// +// ─── WHAT IT DOES, AND WHY NOT THE TWO CHEAPER THINGS ──────────────────────── +// +// For every contract tensor the adapters target it re-materializes the tensor +// from `file` through the SAME `MaterializeDitTensor` the load uses, fuses the +// adapters back in when `fuse` is true, and writes the result into the buffer +// the view ALREADY points at. So: +// +// * the view pointer never moves, and `checkpoint.weights` — a pure view +// struct — stays valid without being re-bound; +// * peak residency rises by ONE tensor plus the adapter's own A/B factors, +// never by a second weight set; +// * each phase gets `round_bf16(W + delta)` computed from the PRISTINE base, +// which is upstream's arithmetic rather than an approximation of it. +// +// NOT by SUBTRACTING the delta, which needs no file read and is wrong: +// `round_bf16(round_bf16(W + d) - d)` is not `W`. The rounding is the whole +// reason the accumulator dtype is pinned in `ltx2_lora.h`, and a subtract would +// spend it twice. `test_ltx2_loader` compares byte-for-byte against a fresh +// unfused load precisely so that implementation fails. +// +// NOT by applying the adapter UNFUSED at run time, which is `Wx + s*B(Ax)` +// against upstream's `round_bf16(W + s*BA)x`. That is a rounding divergence AND +// a different GEMM path, and it would change every arm's numerics to serve one +// recipe. +// +// A tensor no adapter names is not touched at all, because for those the fused +// and unfused images are equal by construction. +// +// `queue` is non-null exactly when the checkpoint was staged to a device, in +// which case the write back is a `Copy` into the same device allocation. Passing +// a queue for a host checkpoint, or none for a staged one, refuses by name +// rather than writing to the wrong address space. +// +// Calling this with the state the checkpoint is already in is a no-op it detects +// itself, so a single-phase recipe pays nothing. +void Ltx2RebindDitLoras(vt::Queue* queue, const SafetensorsFile& file, + const Ltx2DitLoadOptions& options, bool fuse, + Ltx2DitCheckpoint& checkpoint); + // --------------------------------------------------------------------------- // The text encoder // --------------------------------------------------------------------------- diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index c611af23d..f56643e6f 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -603,6 +603,55 @@ enum class Ltx2StepperKind { kEuler, kEulerAncestral, kRes2s }; enum class Ltx2PhaseDenoiser { kGuided, kSimple }; // LTXPhaseRecipe (ltx2_recipes.py:29-50). +// WHICH of the load's adapters a PHASE runs. +// +// Upstream states this by building a second `DiffusionStage` from the same +// checkpoint with a different `loras=` argument, read at Lightricks/LTX-2 +// fd4ded7f: `a2vid_two_stage.py:107` against `:114`, `ti2vid_two_stages.py:140` +// against `:151`, `ti2vid_two_stages_hq.py:154` against `:165`, and — the mirror +// image — `ic_lora.py:108` against `:119`, where the adapter rides stage 1 and +// stage 2 runs bare. +// +// UPSTREAM NEEDS TWO PLACEMENTS, not one, which is why this is a SET and not a +// "does this phase get the distilled adapter" boolean. +// `ltx-pipelines/CLAUDE.md:48` scopes the adapter to "stage 2 only in +// TI2Vid/A2Vid/Keyframe", while `:49` has HQ apply it to BOTH stages and +// `:50-51` says the same of DFR. Stage 1 `kNoAdapters` with stage 2 defaulted is +// the first; both phases defaulted is the second. +// +// TWO ENUMERATORS, and two is the COMPLETE space rather than a boolean wearing +// an enum's clothes: `Ltx2ResolveLoraReferenceFactors` refuses more than one +// adapter by name (`ltx2_lora.h:167-172`, mirroring `dubit.py:364-365` and +// `hdr_ic_lora.py:271-272`), so the powerset of the load's adapters has exactly +// two members. "Some of them" has no spelling here because it has no spelling +// anywhere in this engine yet; the day that arity cap lifts, the third value +// goes here. +// +// AND UPSTREAM HOLDS ONE TRANSFORMER, not two. Both `from_checkpoint` calls name +// the same `model_paths.transformer()` (`a2vid_two_stage.py:104` and `:116`, +// `ti2vid_two_stages.py:137` and `:148`) and differ only in the adapter tuple. +// So a phase-scoped adapter over one resident DiT is what upstream does, and a +// second resident weight set would be a heavier architecture than the reference +// rather than a faithful port of it. +// +// NO PER-PHASE STRENGTH, deliberately. `ti2vid_two_stages_hq.py` needs one — +// 0.25 at `:92-96` and 0.5 at `:97-101` — and no recipe this tree ships would +// set it, so adding the field now lands a branch nothing can select. That is the +// argument `ltx2_lora.h:41-44` already makes for the second product form. Owed +// by https://github.com/mudler/vllm.cpp/issues/921. +enum class Ltx2PhaseLoraScope { + // Every adapter the load supplied. The DEFAULT, because `distilled.py:131` + // builds ONE stage set and so every recipe that predates this field — + // `distilled_two_stage`, `dfr`, `retake`, `one_stage`, `res2s`, `t2a_one_stage` + // — is upstream-correct running the adapters on all of its phases. A different + // default would silently move six gated arms. + kAllAdapters, + // The base weights. `ic_lora.py:119`'s `loras=()`, and + // `a2vid_two_stage.py:107`'s stage 1 relative to the distilled adapter that + // `requires_distilled_lora` identifies. + kNoAdapters, +}; + struct Ltx2PhaseRecipe { std::string name; Ltx2MultiModalGuiderParams video_guidance; @@ -619,6 +668,11 @@ struct Ltx2PhaseRecipe { // recipe that refuses an override can reach it. Ltx2PhaseDenoiser denoiser = Ltx2PhaseDenoiser::kGuided; bool use_official_sigma_schedule = true; + // The adapter set this phase runs. Read in exactly one place — the phase + // loop's rebind, immediately before the phase's first DiT forward — and + // honoured by `Ltx2RebindDitLoras`, which re-materializes only the tensors an + // adapter targets so that no second weight set ever exists. + Ltx2PhaseLoraScope loras = Ltx2PhaseLoraScope::kAllAdapters; Ltx2StepperKind stepper = Ltx2StepperKind::kEuler; double stepper_eta = 0.0; double stepper_s_noise = 1.0; @@ -683,12 +737,16 @@ struct Ltx2PipelineRecipe { // carrying no adapter without running a distilled schedule on undistilled // weights. // - // What this flag CANNOT express is upstream's placement: - // `stage_2_loras = (*loras, *distilled_lora)` (a2vid_two_stage.py:114) puts - // the adapter on stage 2 ALONE, against `loras=tuple(loras)` for stage 1 - // (`:107`), and this engine fuses at load into one weight set. Owed by - // https://github.com/mudler/vllm.cpp/issues/1118 and recorded in - // .agents/specs/ltx25-a2vid-recipe.md section 4.4. + // THE PLACEMENT IS NOT THIS FLAG'S JOB, and it is no longer missing. This + // comment used to end "this engine fuses at load into one weight set", owed by + // #1118. Row LTX25-PHASE-LORA closed that: `Ltx2PhaseRecipe::loras` carries + // upstream's per-stage adapter set, and `A2VidTwoStageRecipe` gives stage 1 + // `kNoAdapters` (`a2vid_two_stage.py:107`) against stage 2's default + // (`:114`, `stage_2_loras = (*loras, *distilled_lora)`). + // + // What this flag says is only that the load must CARRY an adapter, mirroring + // `--distilled-lora required=True`. What the phase field says is which stage + // runs it. The two were conflated while only one placement existed. bool requires_distilled_lora = false; int64_t max_spatial_downscale() const; diff --git a/src/vllm/model_executor/models/ltx2_loader.cpp b/src/vllm/model_executor/models/ltx2_loader.cpp index a606cd1ca..05f993123 100644 --- a/src/vllm/model_executor/models/ltx2_loader.cpp +++ b/src/vllm/model_executor/models/ltx2_loader.cpp @@ -760,6 +760,112 @@ Ltx2DitCheckpoint Ltx2StreamDitToDevice(vt::Queue& queue, const SafetensorsFile& return out; } +void Ltx2RebindDitLoras(vt::Queue* queue, const SafetensorsFile& file, + const Ltx2DitLoadOptions& options, bool fuse, + Ltx2DitCheckpoint& checkpoint) { + // No adapter was ever supplied, so every phase runs the same base weights and + // there is no state to move. Checked first so that a `kNoAdapters` phase on a + // load with no `lora_path` costs nothing at all. + if (options.loras.empty()) return; + + // The state the checkpoint is in. `CheckLorasWereApplied` refuses a load whose + // adapters fused into ZERO tensors, so with an adapter present a non-zero + // count means fused and zero means rebound-off — the count is a faithful state + // bit and needs no field of its own. + const bool currently_fused = checkpoint.lora_fused_tensors > 0; + if (currently_fused == fuse) return; + + const bool staged = !checkpoint.device_storage.empty(); + if (staged != (queue != nullptr)) { + Fail(std::string("Ltx2RebindDitLoras was given ") + + (queue != nullptr ? "a queue for a HOST-resident checkpoint" + : "no queue for a DEVICE-STAGED checkpoint") + + ". The two write to different address spaces, and guessing which one a " + "view points at is how a rebind would corrupt the weights silently " + "instead of refusing."); + } + + const DitPlan plan = PlanDit(file); + const std::vector contract = ContractOf(checkpoint.params); + // Opened per rebind rather than cached on the checkpoint: the A/B factors are + // the adapter's whole payload, and holding them resident for the life of the + // engine would spend most of what the second-weight-set shape was rejected + // for. This is the wall-clock half of that trade, paid once per phase + // boundary. + const std::vector loras = OpenDitLoras(options, contract); + + vt::Backend* backend = queue != nullptr ? &vt::GetBackend(queue->device.type) : nullptr; + std::vector host; + int64_t fused_count = 0; + for (const Ltx2TensorSpec& spec : contract) { + // Only a tensor some adapter TARGETS can differ between the two states. For + // every other tensor the fused and unfused images are equal by construction, + // so re-materializing it would be work with no observable result. + bool targeted = false; + for (const Ltx2LoraAdapter& lora : loras) { + if (lora.Find(spec.name) != nullptr) { + targeted = true; + break; + } + } + if (!targeted) continue; + + const auto it = checkpoint.views.find(spec.name); + if (it == checkpoint.views.end()) { + Fail("Ltx2RebindDitLoras: '" + spec.name + + "' is a LoRA target in this checkpoint's own contract but is not bound. " + "Refusing rather than rebinding a subset and reporting success."); + } + vt::Tensor& view = it->second; + + // The SAME materialize the load uses, from the pristine file, so the base + // this fuses onto is the base the load fused onto — bit for bit. + const vt::DType dtype = MaterializeDitTensor(file, plan, spec, host); + if (fuse && FuseLorasInto(loras, spec, dtype, host)) ++fused_count; + + const int64_t numel = view.Numel(); + if (view.dtype == dtype) { + const size_t bytes = static_cast(numel) * vt::SizeOf(dtype); + if (bytes != host.size()) { + Fail("Ltx2RebindDitLoras: '" + spec.name + "' re-materialized to " + + std::to_string(host.size()) + " bytes but the bound view holds " + + std::to_string(bytes)); + } + if (backend != nullptr) { + backend->Copy(*queue, view.data, host.data(), bytes); + backend->Synchronize(*queue); // `host` is reused by the next iteration + } else { + std::memcpy(view.data, host.data(), bytes); + } + } else if (view.dtype == vt::DType::kF32 && dtype == vt::DType::kBF16) { + // `Ltx2WidenDitToF32` ran on this checkpoint, so the bound view is the f32 + // copy and the materialization is still bf16. Widening HERE keeps the + // rebind's arithmetic identical to the load's: fuse in bf16 first, widen + // second, exactly as `Ltx2LoadDitFromSafetensors` then `Ltx2WidenDitToF32` + // do. Widening before the fuse would accumulate in f32 and quietly undo + // the dtype `ltx2_lora.h` pins. + if (static_cast(numel) * sizeof(uint16_t) != host.size()) { + Fail("Ltx2RebindDitLoras: '" + spec.name + "' re-materialized to " + + std::to_string(host.size()) + " bf16 bytes but the bound f32 view holds " + + std::to_string(numel) + " elements"); + } + const uint16_t* src = reinterpret_cast(host.data()); + float* dst = static_cast(view.data); + for (int64_t i = 0; i < numel; ++i) dst[i] = Bf16ToF32(src[static_cast(i)]); + } else { + Fail("Ltx2RebindDitLoras: '" + spec.name + "' materializes as " + + std::string(vt::Name(dtype)) + " but its bound view is " + + std::string(vt::Name(view.dtype)) + + ", and this rebind knows no conversion between them"); + } + } + + // The same refusal the load makes, for the same reason: an adapter that fused + // into nothing renders identically to no adapter while reporting success. + if (fuse) CheckLorasWereApplied(loras, fused_count); + checkpoint.lora_fused_tensors = fused_count; +} + // --------------------------------------------------------------------------- // The text encoder // --------------------------------------------------------------------------- diff --git a/src/vllm/model_executor/models/ltx2_pipeline.cpp b/src/vllm/model_executor/models/ltx2_pipeline.cpp index ddf8772a4..d3ea2818a 100644 --- a/src/vllm/model_executor/models/ltx2_pipeline.cpp +++ b/src/vllm/model_executor/models/ltx2_pipeline.cpp @@ -1541,10 +1541,24 @@ Ltx2PipelineRecipe A2VidTwoStageRecipe(const Ltx2PipelineParams& params, // per request at `:353-360`. stage1.denoiser = Ltx2PhaseDenoiser::kGuided; stage1.allow_guidance_override = true; + // `loras=tuple(loras)` (`:107`) — stage 1 runs WITHOUT the distilled adapter, + // against stage 2's `(*loras, *distilled_lora)` (`:114`). The adapter this + // engine's single `lora_path` slot carries for this recipe IS that + // `distilled_lora`, which is what `requires_distilled_lora` declares by + // mirroring `--distilled-lora required=True` (utils/args.py:1140-1153). + // + // THIS LINE IS THE ROW. Deleting it reverts to one fused weight set for both + // stages, which is #1118 exactly, and it renders: stage 1's 40-step guided + // schedule would run against base + distilled where upstream runs it against + // the base alone. The gate that sees it is "the distilled adapter rides stage + // 2 ALONE" in test_ltx2_video. + stage1.loras = Ltx2PhaseLoraScope::kNoAdapters; stage1.stepper = Ltx2StepperKind::kEuler; Ltx2PhaseRecipe stage2; stage2.name = "stage_2"; + // Left at `kAllAdapters`: `(*loras, *distilled_lora)` (`:114`) is every + // adapter this engine holds. stage2.spatial_downscale = 1; // `stage_2_sigmas: torch.Tensor = STAGE_2_DISTILLED_SIGMAS` (`:164`). stage2.sigmas = Stage2DistilledSigmas(); diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index 473de69e9..5e16c2140 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -384,7 +384,7 @@ constexpr char kLtx2DurationHeadPathExtra[] = "duration_head_path"; // they are no longer trusted: the list below is derived from this file on every // run and compared, and the failure prints the replacement to paste in. // READER ANCHORS (derived and gated by test_ltx2_video): -// 809 819 820 882 978 994 1029 1120 1145 1250 1291 1333 1335 +// 823 833 834 896 992 1008 1043 1134 1159 1264 1305 1347 1349 const char* const kKnownLoadExtras[] = { kLtx2AudioPromptEmbedsExtra, kLtx2PipelineKindExtra, kLtx2ModelVersionExtra, @@ -591,6 +591,18 @@ struct Ltx2VideoEngine::Impl { std::optional queue; Ltx2DitCheckpoint dit; + // The adapter set the LOAD supplied, kept so the phase loop can put the DiT + // into the state each phase asks for (`Ltx2PhaseRecipe::loras`). Upstream + // instead builds a second `DiffusionStage` per adapter set + // (a2vid_two_stage.py:103 and :115); this engine holds one DiT and + // re-materializes the adapter's target tensors at the boundary, because a + // second resident weight set is 18.7-39 GB and one GB10 has 119 GB with no + // swap. `Ltx2RebindDitLoras` carries the whole argument. + // + // The SPECS only — the adapter file itself is re-read per rebind rather than + // held, since its A/B factors are its whole payload and keeping them resident + // would spend most of what the second-weight-set shape was rejected for. + Ltx2DitLoadOptions dit_options; std::string model_version, pipeline_kind; Ltx2PipelineRecipe recipe; int64_t max_phase = -1; // -1 => every phase @@ -805,7 +817,9 @@ std::unique_ptr Ltx2VideoEngine::Load(const VideoModelParams& p // ── the DiT ─────────────────────────────────────────────────────────────── const SafetensorsFile dit_file = SafetensorsFile::Open(params.dit_path); - Ltx2DitLoadOptions dit_options; + // ON THE IMPL, not a local: the phase loop re-reads these to put the DiT into + // the adapter state each phase declares (`Ltx2PhaseRecipe::loras`). + Ltx2DitLoadOptions& dit_options = im.dit_options; dit_options.allow_unported_modules = VideoExtra(params.extras, kLtx2AllowUnportedExtra) == "1"; // On the CPU, f32 is what `Ltx2DitForward` requires: it is the PARITY dtype, // not a widening of a bf16 path. On an accelerator nothing is widened at all — @@ -2219,16 +2233,34 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // temporal subsample, neither of which retake performs and neither of // which any reader supplies. // - // 2. THE REFERENCE ITEM BELONGS TO STAGE 1, AND STAGE 2 MUST RUN UNFUSED. - // `ICLoraPipeline` builds two `DiffusionStage`s from the same checkpoint - // and gives stage 1 `loras=tuple(loras)` (ic_lora.py:108) and stage 2 - // `loras=()` (:119); stage 1 takes `_create_conditionings`, which appends - // the reference item (:269-278, :377-402), and stage 2 takes plain - // `combined_image_conditionings` with no reference item at all - // (:314-321). This engine holds ONE `Ltx2Dit`, fused at load, that every - // phase of the recipe runs. Serving the arm on a two-phase recipe needs a - // second unfused DiT or a phase-scoped adapter, and serving it on one - // phase only is upstream's `skip_stage_2` (:302-308), a different request. + // THE SECOND REASON THIS MESSAGE GAVE IS NOW FALSE, and it is recorded + // here rather than deleted because it is the third reason in this block + // to come true and a reader needs to know which. It said: "the reference + // item belongs to stage 1 and stage 2 must run unfused — + // `ICLoraPipeline` gives stage 1 `loras=tuple(loras)` (ic_lora.py:108) + // and stage 2 `loras=()` (:119), and this engine holds ONE `Ltx2Dit`, + // fused at load, that every phase of the recipe runs. Serving the arm + // needs a second unfused DiT or a phase-scoped adapter." + // + // Row LTX25-PHASE-LORA (#1118) landed the phase-scoped adapter. + // `Ltx2PhaseRecipe::loras` (ltx2_pipeline.h) carries upstream's per-stage + // set and the phase loop in this file honours it through + // `Ltx2RebindDitLoras`, which re-materializes only the tensors an adapter + // targets — so a two-phase recipe CAN now give stage 1 the adapter and + // stage 2 none, which is exactly `ic_lora.py:108` against `:119`, and it + // does so without a second resident weight set. `A2VidTwoStageRecipe` is + // the executable proof it exists: it gives stage 1 `kNoAdapters` and the + // gate "the distilled adapter rides stage 2 ALONE" renders both states + // through this ABI and compares the pixels. + // + // What that leaves is reason 1 ALONE, and reason 1 is unrelated to + // weights: it is the reference clip's own geometry. The conditioning + // split is also still upstream's — stage 1 takes `_create_conditionings`, + // which appends the reference item (:269-278, :377-402), and stage 2 + // takes plain `combined_image_conditionings` with no reference item + // (:314-321) — but that is a conditioning question, not a fused-weight + // one, and serving the arm on one phase only is upstream's `skip_stage_2` + // (:302-308), a different request. std::string factors = "no adapter was supplied, so none were read"; if (im.dit.lora_fused_tensors > 0) { factors = "the supplied adapter declares downscale=" + @@ -2250,13 +2282,13 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { "`minimax_h3_video.cpp:650` — and is doubly false now that row LTX25-RETAKE (#924) " "reads it on this side through `Ltx2ReadFrameDirectory`. What is missing is the " "reference item's own geometry, the downscale resize and the temporal subsample, " - "which no reader supplies. SECOND, the reference item is a STAGE-1 item " - "and stage 2 must run with NO adapter: `ICLoraPipeline` gives stage 1 " - "`loras=tuple(loras)` (ic_lora.py:108) and the reference conditioning (:269-278), " - "and gives stage 2 `loras=()` (:119) and `combined_image_conditionings` with no " - "reference item (:314-321) — and this engine holds one DiT, fused at load, that " - "every phase runs. WHAT IS *NOT* THE REASON, because this refusal has now given two " - "reasons that later became false: (a) the IC-LoRA METADATA. Row LTX25-IC-LORA (#923) " + "which no reader supplies. SECOND, the reference item is a STAGE-1 item and stage 2 " + "takes `combined_image_conditionings` with no reference item at all: `ICLoraPipeline` " + "gives stage 1 the reference conditioning (ic_lora.py:269-278) and stage 2 none " + "(:314-321), and this phase loop appends the same conditioning set to every phase. " + "That is a CONDITIONING gap and not a weights one. WHAT IS *NOT* THE REASON, because " + "this refusal has now given THREE reasons that later became false: (a) the IC-LoRA " + "METADATA. Row LTX25-IC-LORA (#923) " "closed that; supply `lora_path` and the factors are read at load " "(iclora_utils.py:30-49) — right now, " + factors + ". (b) the TOKEN-APPEND machinery. This message blamed it on 2026-08-15 and row " @@ -2270,7 +2302,15 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { "`conditioning_attention_strength >= 1.0` with no latent mask `attn_mask` is None " "(iclora_utils.py:159-160) and `ConditioningItemAttentionStrengthWrapper` is " "applied only `if attn_mask is not None` (:168-169). The sub-1.0 arm is owed by " - "#932, and it is not what blocks this one. Use first_frame_ppm / first_frame_path " + "#932, and it is not what blocks this one. (d) the FUSED-AT-LOAD adapter. This " + "message said until 2026-08-17 that stage 2 must run with no adapter while \"this " + "engine holds one DiT, fused at load, that every phase runs\", and row " + "LTX25-PHASE-LORA (#1118) closed it: `Ltx2PhaseRecipe::loras` carries upstream's " + "per-stage set and the phase loop rebinds the DiT through `Ltx2RebindDitLoras`, so " + "`loras=tuple(loras)` on stage 1 against `loras=()` on stage 2 (ic_lora.py:108, " + ":119) is now expressible with no second weight set. `a2vid_two_stage`'s stage 1 " + "runs `kNoAdapters` on exactly that machinery, which is the executable proof it " + "exists. Use first_frame_ppm / first_frame_path " "for image-to-video, and last_frame_path for a closing keyframe."); } if (!gen.ref_audio_path.empty() || !gen.ref_audio_wav.empty()) { @@ -2774,6 +2814,31 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { for (int64_t phase_index = 0; phase_index <= last_phase; ++phase_index) { const Ltx2PhaseRecipe& phase = recipe.phases[static_cast(phase_index)]; + + // THE PER-PHASE ADAPTER SET (row LTX25-PHASE-LORA, issue #1118). Upstream + // hands each `DiffusionStage` its own `loras=` argument + // (a2vid_two_stage.py:107 against :114) and pays for it with a second + // resident weight set; this engine holds ONE DiT and moves it between the + // two states here, re-materializing only the tensors an adapter targets. + // + // BEFORE any conditioning, encode or forward of this phase, so no work is + // ever paid against weights the phase did not ask for. A no-op when the + // load supplied no adapter, and a no-op when the DiT is already in the + // requested state — so a one-stage recipe and every recipe that predates + // this field cost nothing, and a two-stage render pays exactly one rebind. + // + // The emptiness test is HERE as well as inside the rebind so that a load + // with no adapter does not re-open the checkpoint once per phase to be told + // there is nothing to do. + { + const bool want_fused = phase.loras == Ltx2PhaseLoraScope::kAllAdapters; + if (!im.dit_options.loras.empty() && want_fused != (im.dit.lora_fused_tensors > 0)) { + Ltx2RebindDitLoras(im.on_device ? &*im.queue : nullptr, + SafetensorsFile::Open(im.params.dit_path), im.dit_options, + want_fused, im.dit); + } + } + const int64_t phase_h = height / phase.spatial_downscale; const int64_t phase_w = width / phase.spatial_downscale; @@ -4253,6 +4318,15 @@ VideoResult Ltx2VideoEngine::GenerateAudioOnly(Impl& im, const VideoGenParams& g int64_t context_tokens) { const Ltx2PipelineRecipe& recipe = im.recipe; VT_CHECK(recipe.audio_only, "ltx2 t2a: reached the audio-only path on a video recipe"); + // This path does NOT run the phase loop — it reads `phases.front()` and + // denoises once — so it is the one place a per-phase adapter set could be + // declared and silently ignored. Upstream's `T2AOneStagePipeline` builds ONE + // stage set (t2a_one_stage.py:67), so no audio-only recipe has a reason to ask + // for anything but the load's own adapters, and this asserts that rather than + // assuming it. Row LTX25-PHASE-LORA (#1118). + VT_CHECK(recipe.phases.front().loras == Ltx2PhaseLoraScope::kAllAdapters, + "ltx2 t2a: an audio-only phase asked for a per-phase adapter set, and this path " + "never runs the rebind that would honour it"); // Upstream's T2A CLI has no --height/--width, and its pipeline substitutes a // 512x512 PLACEHOLDER whose height and width it documents as unused diff --git a/tests/vllm/models/test_ltx2_loader.cpp b/tests/vllm/models/test_ltx2_loader.cpp index 5999b6911..24f5045b4 100644 --- a/tests/vllm/models/test_ltx2_loader.cpp +++ b/tests/vllm/models/test_ltx2_loader.cpp @@ -2370,3 +2370,179 @@ TEST_CASE("ltx2 loader: an adapter that fuses into NOTHING refuses rather than l std::remove(dit_path.c_str()); std::remove(lora_path.c_str()); } + +// ─── LTX25-PHASE-LORA (#1118) ──────────────────────────────────────────────── + +namespace { + +// Every bound view of `a` against `b`, byte for byte. Returns the number of +// views that DIFFER, so a caller can assert both directions and neither +// assertion is vacuous. +// +// Byte-for-byte and not a tolerance: the claim `Ltx2RebindDitLoras` makes is +// that a rebound checkpoint is INDISTINGUISHABLE from one the loader produced, +// and a tolerance would pass an implementation that reconstructs the base by +// subtracting the delta — which is the wrong implementation this case exists to +// catch. +int64_t ViewsDiffering(const vllm::Ltx2DitCheckpoint& a, const vllm::Ltx2DitCheckpoint& b) { + REQUIRE(a.views.size() == b.views.size()); + int64_t differing = 0; + for (const auto& kv : a.views) { + const auto it = b.views.find(kv.first); + REQUIRE(it != b.views.end()); + const vt::Tensor& x = kv.second; + const vt::Tensor& y = it->second; + REQUIRE(x.dtype == y.dtype); + REQUIRE(x.Numel() == y.Numel()); + const size_t bytes = static_cast(x.Numel()) * vt::SizeOf(x.dtype); + if (std::memcmp(x.data, y.data, bytes) != 0) ++differing; + } + return differing; +} + +} // namespace + +TEST_CASE("ltx2 loader: a phase rebind reproduces the load BYTE-FOR-BYTE in both directions") { + // THE EXACTNESS CLAIM of row LTX25-PHASE-LORA (#1118), made executable. + // + // Upstream gives each `DiffusionStage` its own adapter set by building a + // SECOND stage from the same checkpoint (ic_lora.py:104 and :115). This port + // holds one DiT and moves it between the two states, so the thing that has to + // be proved is that "moved back" and "loaded that way" are the same bytes — + // otherwise the seam buys per-phase adapters at the cost of a silent numeric + // drift, which is the trade this row rejected when it rejected unfused runtime + // LoRA. + const Ltx2DitParams p = TinyParams(); + const SyntheticDit syn = BuildSyntheticDit(p, Ltx2DitQuant::kFp8, {}); + const std::string dit_path = TmpPath("rebind_dit"); + WriteSafetensors(syn.entries, dit_path); + + const std::string target = "transformer_blocks.0.attn1.to_q.weight"; + const std::string lora_path = WriteLoraFor(p, target, kLoraScale, TmpPath("rebind_lora")); + + const SafetensorsFile file = SafetensorsFile::Open(dit_path); + + vllm::Ltx2DitLoadOptions options; + vllm::Ltx2LoraSpec spec; + spec.path = lora_path; + spec.strength = 1.0; + options.loras.push_back(spec); + + // The two REFERENCE images, each produced by the loader itself. + const vllm::Ltx2DitCheckpoint plain = vllm::Ltx2LoadDitFromSafetensors(file); + const vllm::Ltx2DitCheckpoint fused = vllm::Ltx2LoadDitFromSafetensors(file, options); + REQUIRE(plain.lora_fused_tensors == 0); + REQUIRE(fused.lora_fused_tensors == 1); + // The instrument is armed: the two references genuinely differ, in exactly the + // one tensor the adapter targets. Without this line every equality below could + // pass on a checkpoint where the adapter did nothing. + REQUIRE(ViewsDiffering(plain, fused) == 1); + + // The one under test, loaded FUSED exactly as `Ltx2VideoEngine::Load` loads it. + vllm::Ltx2DitCheckpoint live = vllm::Ltx2LoadDitFromSafetensors(file, options); + REQUIRE(live.lora_fused_tensors == 1); + // The pointer the bound weights read through. `Ltx2DitWeights` is a pure view + // struct, so a rebind that reallocated would leave `live.weights` dangling and + // every forward would read freed memory. Captured before, checked after. + const void* const target_before = live.views.at(target).data; + + SUBCASE("rebound OFF, it is the checkpoint the loader builds with no adapter") { + vllm::Ltx2RebindDitLoras(/*queue=*/nullptr, file, options, /*fuse=*/false, live); + CHECK(live.lora_fused_tensors == 0); + CHECK(live.views.at(target).data == target_before); + // The whole claim, in one number: nothing distinguishes it from `plain`. + CHECK(ViewsDiffering(live, plain) == 0); + // And it really moved — this is what fails if the rebind quietly did nothing. + CHECK(ViewsDiffering(live, fused) == 1); + } + + SUBCASE("rebound OFF then ON, it is the checkpoint the loader builds WITH the adapter") { + vllm::Ltx2RebindDitLoras(/*queue=*/nullptr, file, options, /*fuse=*/false, live); + vllm::Ltx2RebindDitLoras(/*queue=*/nullptr, file, options, /*fuse=*/true, live); + CHECK(live.lora_fused_tensors == 1); + CHECK(live.views.at(target).data == target_before); + // THE ROUND TRIP. An implementation that reconstructed the base by + // SUBTRACTING the delta would land here at + // `round_bf16(round_bf16(W + d) - d) + d`, which is not `round_bf16(W + d)` + // for every element, and this line is what tells the two apart. + CHECK(ViewsDiffering(live, fused) == 0); + CHECK(ViewsDiffering(live, plain) == 1); + } + + SUBCASE("a rebind to the state it is already in changes nothing") { + // The no-op a one-stage recipe and every recipe predating the phase field + // relies on: they never ask for a different set, and they must not pay a + // re-materialization to be told so. + vllm::Ltx2RebindDitLoras(/*queue=*/nullptr, file, options, /*fuse=*/true, live); + CHECK(live.lora_fused_tensors == 1); + CHECK(ViewsDiffering(live, fused) == 0); + } + + SUBCASE("a queue for a HOST checkpoint refuses by name") { + // The two address spaces. Guessing which one a view points at is how a + // rebind would corrupt weights silently instead of refusing. + vt::Queue queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + std::string what; + try { + vllm::Ltx2RebindDitLoras(&queue, file, options, /*fuse=*/false, live); + } catch (const std::exception& e) { + what = e.what(); + } + INFO("what: ", what); + CHECK(what.find("a queue for a HOST-resident checkpoint") != std::string::npos); + // It refused BEFORE touching anything. + CHECK(live.lora_fused_tensors == 1); + CHECK(ViewsDiffering(live, fused) == 0); + } + + std::remove(dit_path.c_str()); + std::remove(lora_path.c_str()); +} + +TEST_CASE("ltx2 loader: a rebind of a WIDENED checkpoint fuses in bf16 and stores f32") { + // The host arm the CPU parity forward actually runs: `widen_to_f32` is set + // whenever the engine is off-device (`ltx2_video.cpp`), so the bound view is + // an f32 copy while `MaterializeDitTensor` still returns bf16. If the rebind + // widened BEFORE fusing it would accumulate the delta in f32 and quietly undo + // the accumulator dtype `ltx2_lora.h` pins — which no token gate and no + // golden could see, because the numbers would still be finite and close. + const Ltx2DitParams p = TinyParams(); + const SyntheticDit syn = BuildSyntheticDit(p, Ltx2DitQuant::kFp8, {}); + const std::string dit_path = TmpPath("rebind_wide_dit"); + WriteSafetensors(syn.entries, dit_path); + + const std::string target = "transformer_blocks.0.attn1.to_q.weight"; + const std::string lora_path = WriteLoraFor(p, target, kLoraScale, TmpPath("rebind_wide_lora")); + + const SafetensorsFile file = SafetensorsFile::Open(dit_path); + + vllm::Ltx2DitLoadOptions options; + options.widen_to_f32 = true; + vllm::Ltx2LoraSpec spec; + spec.path = lora_path; + spec.strength = 1.0; + options.loras.push_back(spec); + + vllm::Ltx2DitLoadOptions plain_options; + plain_options.widen_to_f32 = true; + + const vllm::Ltx2DitCheckpoint plain = vllm::Ltx2LoadDitFromSafetensors(file, plain_options); + const vllm::Ltx2DitCheckpoint fused = vllm::Ltx2LoadDitFromSafetensors(file, options); + REQUIRE(plain.views.at(target).dtype == vt::DType::kF32); + REQUIRE(ViewsDiffering(plain, fused) == 1); + + vllm::Ltx2DitCheckpoint live = vllm::Ltx2LoadDitFromSafetensors(file, options); + const void* const target_before = live.views.at(target).data; + + vllm::Ltx2RebindDitLoras(/*queue=*/nullptr, file, options, /*fuse=*/false, live); + CHECK(live.views.at(target).data == target_before); + CHECK(live.views.at(target).dtype == vt::DType::kF32); + CHECK(ViewsDiffering(live, plain) == 0); + + vllm::Ltx2RebindDitLoras(/*queue=*/nullptr, file, options, /*fuse=*/true, live); + CHECK(ViewsDiffering(live, fused) == 0); + CHECK(ViewsDiffering(live, plain) == 1); + + std::remove(dit_path.c_str()); + std::remove(lora_path.c_str()); +} diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 5d59d6bae..197aa4c14 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -7342,3 +7342,125 @@ TEST_CASE("ltx2 a2vid: every requirement the recipe adds refuses BY WHAT IS MISS CHECK_THROWS((void)fixed->Generate(gen_fixed)); } } + +// ─── LTX25-PHASE-LORA (#1118) ──────────────────────────────────────────────── + +TEST_CASE("ltx2 a2vid: the distilled adapter rides stage 2 ALONE") { + // THE ROW, and the one case that separates a PER-PHASE adapter set from + // fusion at load. A gate that only asserted "a LoRA was applied" passes on the + // defect this row fixes, because the defect DOES apply the LoRA — to every + // phase. + // + // Upstream, read at Lightricks/LTX-2 fd4ded7f: two `DiffusionStage`s are built + // from the SAME `model_paths.transformer()` (a2vid_two_stage.py:104, :116) and + // differ only in their adapter tuple — stage 1 takes `loras=tuple(loras)` + // (`:107`) and stage 2 takes `(*tuple(loras), *tuple(distilled_lora))` + // (`:114`, passed at `:119`). `ltx-pipelines/CLAUDE.md:48` states the same + // convention in prose: the distilled adapter is "applied to stage 2 only in + // TI2Vid/A2Vid/Keyframe". + // + // ENTRY POINT: `LoadVideoEngine` with the documented `pipeline_kind`, + // `lora_path`, `lora_strength` and `max_phase` LOAD extras, then `Generate`. + // Nothing here constructs a recipe, a phase or a checkpoint by hand. + // + // WHY STRENGTH 0 IS THE CONTROL and not "no adapter": `requires_distilled_lora` + // refuses an a2vid load carrying no `lora_path` at all (upstream's + // `--distilled-lora required=True`, utils/args.py:1140-1153), so the base- + // weights arm has to be spelled some other way. Strength 0 fuses a ZERO delta, + // and that it reproduces the base model is already gated independently by + // "ltx2 video: the IC-LoRA strength reaches the PIXELS, and 0 is a no-op". + Workspace ws; + const std::string wav = WriteWav(ws.root + "/take.wav", 2, kFixtureAudioRate, 2.0); + const std::string lora = + WriteFixtureLora(ws.root + "/distilled.safetensors", kFixtureLoraTarget, 1.0F); + + // `max_phase` is a LOAD extra, so each arm is its own engine. + const auto render = [&](const char* strength, const char* max_phase, const char* out) { + vllm::multimodal::VideoModelParams mp = A2VidParams(ws.paths, lora); + mp.extras[vllm::multimodal::kLtx2LoraStrengthExtra] = strength; + if (max_phase != nullptr) mp.extras[vllm::multimodal::kLtx2MaxPhaseExtra] = max_phase; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + REQUIRE(engine != nullptr); + const std::string dir = std::string(ws.root) + "/" + out; + const vllm::multimodal::VideoResult result = engine->Generate(A2VidGen(dir, wav)); + return A2VidArtifacts(dir, result); + }; + + // ── stage 1 ALONE, adapter at full strength against a zero delta ─────────── + const std::string s1_full = render("1.0", "0", "s1_full"); + const std::string s1_zero = render("0.0", "0", "s1_zero"); + REQUIRE(s1_full.size() > 0); + REQUIRE(s1_full.size() == s1_zero.size()); + + size_t s1_differing = 0; + for (size_t i = 0; i < s1_full.size(); ++i) { + if (s1_full[i] != s1_zero[i]) ++s1_differing; + } + MESSAGE("stage 1 alone: the adapter moves " << s1_differing << " of " << s1_full.size() + << " artifact bytes"); + // THE HALF THAT REDS ON TODAY'S DEFECT. Under fusion at load, stage 1 runs + // base + distilled and this count is non-zero. `loras=tuple(loras)` at + // `:107` names no distilled adapter, so stage 1 must be the base model and + // the adapter's strength must be invisible to it. + CHECK(s1_differing == 0); + + // ── both stages, the same two strengths ─────────────────────────────────── + const std::string both_full = render("1.0", nullptr, "both_full"); + const std::string both_zero = render("0.0", nullptr, "both_zero"); + REQUIRE(both_full.size() == both_zero.size()); + + size_t both_differing = 0; + for (size_t i = 0; i < both_full.size(); ++i) { + if (both_full[i] != both_zero[i]) ++both_differing; + } + MESSAGE("both stages: the adapter moves " << both_differing << " of " << both_full.size() + << " artifact bytes"); + // THE HALF THAT REDS ON "STOPPED FUSING ALTOGETHER". `stage_2_loras` at `:114` + // DOES carry the distilled adapter, so it must reach the pixels through stage + // 2. Without this line the case above is satisfied by an engine that ignores + // `lora_path` entirely, which is the same shape of green-but-proves-nothing + // the row's spec rejects. + // + // Strictly greater than zero and no count floor above it: a count-based + // tolerance would bound nothing. + CHECK(both_differing > 0); + + // ── and the two arms are not the same render ────────────────────────────── + // Stage 2 upsamples, so a stage-1-only artifact cannot equal a two-stage one. + // This is what proves `max_phase = 0` actually stopped after stage 1 rather + // than the whole comparison having run twice on the same pixels. + CHECK(s1_full != both_full); +} + +TEST_CASE("ltx2 a2vid: the rebind leaves the DiT where the NEXT generation expects it") { + // A phase-scoped adapter mutates weights the engine keeps across calls, so the + // question load-time fusion never had to answer is whether generation N+1 sees + // what generation N left behind. `Ltx2RebindDitLoras` is driven from the TOP + // of the phase loop and keys off the checkpoint's own state, so every + // generation re-establishes stage 1's before it denoises anything. + // + // Rendered through the ABI twice on ONE engine, which is the shape a server + // runs and the shape no single-generation case can see. + Workspace ws; + const std::string wav = WriteWav(ws.root + "/take.wav", 2, kFixtureAudioRate, 2.0); + const std::string lora = + WriteFixtureLora(ws.root + "/distilled.safetensors", kFixtureLoraTarget, 1.0F); + + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(A2VidParams(ws.paths, lora)); + REQUIRE(engine != nullptr); + + const vllm::multimodal::VideoResult first = + engine->Generate(A2VidGen(ws.root + "/gen_a", wav)); + const std::string a = A2VidArtifacts(ws.root + "/gen_a", first); + const vllm::multimodal::VideoResult second = + engine->Generate(A2VidGen(ws.root + "/gen_b", wav)); + const std::string b = A2VidArtifacts(ws.root + "/gen_b", second); + + REQUIRE(a.size() > 0); + // Same request, same seed, same engine: byte-identical. A rebind that left the + // DiT unfused after the first render would make the second render's stage 2 + // run on base weights, and these would differ. + CHECK(a == b); +} From 17b2113845429e5d808995a5260305a505166a1e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 15:50:57 +0000 Subject: [PATCH 3/5] record(LTX25-PHASE-LORA): the gate stands on one byte, and the no-op is the next row's trap (#1118) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three non-blocking findings from the fresh review of #1140, all of them records. No behaviour changes: a spec `## Outcome`, one header comment and one source comment. THE HEADLINE GATE'S MARGIN IS ONE BYTE. "ltx2 a2vid: the distilled adapter rides stage 2 ALONE" has two halves, and reverting to fusion at load moves exactly 1 of 63,809 artifact bytes — 0.0016%. `CHECK(s1_differing == 0)` cannot go falsely RED, because the render is deterministic and 0 is 0. It can go falsely GREEN: a fixture change to the block count, the 2-step schedule, the sigma table or the PPM's 8-bit quantization could take that 1 to 0, and the gate would then pass on the defect it exists to catch. The other half carries 11 of 146,753 and reds under "stopped fusing altogether", so the pair does not fail open together — which is why this is recorded rather than repaired. Both margins and the three mutation counts now sit in the spec's `## Outcome`, which the row also owed for reaching `DONE`. PER-PHASE STRENGTH NEEDS MORE THAN A FIELD, and the header comment as written points the next reader the wrong way. `Ltx2RebindDitLoras` early-returns on `currently_fused == fuse` and the header calls that a no-op it detects itself. The state is a BOOLEAN, so it means "already fused" and never "already fused at this strength". HQ is stage 1 at 0.25 and stage 2 at 0.5 (`ti2vid_two_stages_hq.py:92-101`, `:154`, `:165`; defaults at `utils/args.py:1174-1184`) with BOTH stages fused, so that early return would no-op the transition and stage 2 would silently render at stage 1's strength. #921 therefore needs `bool fuse` to become a type carrying a strength AND `Ltx2DitCheckpoint` to record WHICH state is applied. Written beside the early return, in the spec, and on #921. Recorded with it: `res2s_two_stage` already runs both stages at 1.0 where upstream runs 0.25/0.5 — pre-existing, not worsened here, and unstated until now. THE REAL-WEIGHTS BULLET WAS OUT OF DATE. It said the comparison needs an adapter that `find /mnt/nas_share/checkpoints -iname '*lora*'` returns nothing for. That control now returns two, including the one this row needs: `ltx-2.5-22b-distilled-lora-450-bf16.safetensors`, 8,899,889,568 bytes, 1660 A/B pairs at rank 450, `model_version` 2.5.0. The full/dev transformer has since completed as well, 42,018,190,584 bytes, 4349 tensors, 21.004 B params, BF16 4059 / F32 290, `keyframes_abs_pos_embedding` present, data end == file size. Both headers were read rather than assumed. So what is owed is the RUN, not the artifacts. The phase-loop comment claimed a two-stage render pays exactly one rebind. It pays two — load fused, phase 0 off, phase 1 on — and each re-opens the adapter and reads every A/B factor pair. That cost stays UNMEASURED and a later performance row owns it; saying so beats a comment that undercounts it. Also folded in, from the review's "checked and fine" list, so nobody derives them again: the f32-widen branch's direct write is unreachable with a device queue because `Ltx2StreamDitToDevice` refuses `widen_to_f32` by name; contract drift between load and rebind cannot occur because `Ltx2AdoptDeclaredDitParams` refuses unless names and shapes are identical; and after a `max_phase = 0` render the reference refusal prints "no adapter was supplied" because `lora_fused_tensors` doubles as the state bit — message-only, on a path that refuses either way. No build. The change is comment and prose, and no build can move a doctest `MESSAGE` count. `READER ANCHORS` was re-derived with a faithful port of the test's own walk and is unchanged at `823 833 834 896 992 1008 1043 1134 1159 1264 1305 1347 1349`; the port was armed first against a one-line insertion and reported MISMATCH at exit 1. `check-public-doc-tables.py`, `check-issue-index-append-only.py`, `check-agent-record.py`, `check-doc-checkpoint.py` and `agent-preflight.sh` all exit 0, and the first was armed with an added prose line and refused at exit 1 before the file was restored byte-for-byte. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-phase-lora.md | 143 +++++++++++++++++- .../vllm/model_executor/models/ltx2_loader.h | 23 +++ src/vllm/multimodal/ltx2_video.cpp | 12 +- 3 files changed, 173 insertions(+), 5 deletions(-) diff --git a/.agents/specs/ltx25-phase-lora.md b/.agents/specs/ltx25-phase-lora.md index d4cd9947a..51325c605 100644 --- a/.agents/specs/ltx25-phase-lora.md +++ b/.agents/specs/ltx25-phase-lora.md @@ -332,19 +332,154 @@ gated, and upstream gives all of them one stage set (`distilled.py:131`). A default that moves a landed arm would be the same class of defect this row exists to fix. +## Outcome + +Row `DONE`. Recorded here because neither the code nor the Git history carries +it: what the gate MEASURED, what it is standing on, and what the fresh review +checked and found sound so that nobody derives it a second time. + +### The distinguishing gate's margins are ONE byte and ELEVEN bytes + +The case "ltx2 a2vid: the distilled adapter rides stage 2 ALONE" prints both +counts with `MESSAGE`, and the fresh review ran the mutations against head +`851b15da7`. The numbers below are those doctest lines, not a summary of them. + +| Run | stage 1 alone | both stages | Result | +|---|---|---|---| +| head, unmutated | **0** of 63809 | **11** of 146753 | `SUCCESS!`, 2420 assertions, 0 failed | +| M1 — `stage1.loras = kNoAdapters` deleted | **1** of 63809 | 14 of 146753 | `FAILURE!`, `CHECK(s1_differing == 0)` RED | +| M2 — the phase loop's rebind call deleted | **1** of 63809 | 14 of 146753 | `FAILURE!`, same assertion RED | +| M3 — the rebind never fuses | 0 of 63809 | **0** of 146753 | `FAILURE!`, `CHECK(both_differing > 0)` RED | + +**So the half that catches this row's own defect catches it by ONE byte in +63,809 — 0.0016%.** Reverting to fusion at load moves exactly that much of the +artifact, and the assertion is the difference between 0 and 1. + +That assertion cannot go falsely RED. The render is deterministic, so the count +is 0 or it is not, and there is no tolerance to drift. What it can do is go +falsely GREEN: anything that lowers the artifact's sensitivity to a stage-1 +weight change takes the 1 to 0 and the gate then passes on the defect it exists +to catch. The candidates are the fixture's block count, its 2-step schedule, its +sigma table, and the PPM's 8-bit quantization, which is where a sub-LSB +trajectory difference is rounded away. **Anyone who moves the a2vid fixture owes +a re-run of M1 and a look at the printed count, not a look at the green.** + +**The pair does not fail open together**, which is why this is recorded rather +than repaired here. `CHECK(both_differing > 0)` carries 11 of 146,753 and M3 +drives it to 0, so an implementation that simply stopped fusing is still caught +after the first half has gone blind. The two counts are one instrument. A +stage-1 count of 0 is evidence only while the both-stages count is above 0. + +### The wall-clock trade is stated and UNMEASURED + +A two-stage a2vid render pays **two** rebinds, not one: the load fuses, phase 0 +rebinds off, phase 1 rebinds back on, and the DiT is left fused so the next +render pays the same two. Each rebind re-opens the adapter and reads every A/B +factor pair (`Ltx2LoraAdapter::Open` -> `ReadFactorAsBf16`), and the shipped +distilled adapter is 8,899,889,568 bytes. Upstream pays its second +materialization ONCE per process instead, because it keeps two +`DiffusionStage`s. That is the trade `## Risks and decisions` accepted, and **no +number was measured for it** — this row is gated on CPU fixtures and claims no +wall-clock result. A later performance row owns it. + +### Checked by the fresh review and found sound + +Recorded so they are not re-derived. + +- **The f32-widen branch's direct write is not a missing `Copy`.** It writes + `static_cast(view.data)` rather than going through `backend->Copy`, + and it cannot be reached with a device queue: `Ltx2StreamDitToDevice` refuses + `widen_to_f32` by name (`ltx2_loader.cpp:714-720`) and `Load` sets + `widen_to_f32 = !im.on_device`, so an f32 view implies a HOST checkpoint, + which the rebind's own address-space refusal already pins to `queue == + nullptr`. +- **Contract drift between the load and the rebind cannot occur.** + `Ltx2AdoptDeclaredDitParams` refuses unless the enumerated tensor names AND + shapes are identical (`ltx2_loader.cpp:1218-1231`), so `ContractOf( + checkpoint.params)` inside the rebind enumerates what the load bound. +- **One cosmetic message drift, on a path that refuses anyway.** + `lora_fused_tensors` doubles as the state bit, so after a `max_phase = 0` + render it is 0 and a later reference-conditioning refusal prints "no adapter + was supplied, so none were read" for a load that did supply one. + `lora_reference` itself is untouched by the rebind and stays correct; it is + simply not read on that branch. Message-only, and the refusal refuses either + way. + +### Not re-measured by this repair + +The margins above are read from the review's own doctest logs at head +`851b15da7`, not re-run. The repair that recorded them changes prose, one header +comment and one source comment, and no build can change a `MESSAGE` count. The +`READER ANCHORS` list was re-derived with a faithful port of the test's own walk +and is unchanged, with an armed control proving the port detects a one-line +shift. + ## Owed - **Per-phase adapter STRENGTH**, which `ti2vid_two_stages_hq.py:92-101` needs and this field cannot express. Owned by #921. + + **A NEW FIELD IS NOT ENOUGH, and the seam's own no-op is why.** + `Ltx2RebindDitLoras` early-returns on `if (currently_fused == fuse) return;`, + and the header says calling it with the state the checkpoint is already in is + a no-op it detects itself. That state is a BOOLEAN, so it means "already + fused" and never "already fused AT THIS STRENGTH". HQ's case is stage 1 at + 0.25 and stage 2 at 0.5 (`ti2vid_two_stages_hq.py:92-101`, handed to `:154` + and `:165`; the CLI defaults are at `utils/args.py:1174-1184`) — **both + fused**. The early return would therefore no-op the transition and stage 2 + would silently render at stage 1's strength, with no refusal and nothing wrong + in the shape of the output. + + So #921 needs two changes beyond the field: `bool fuse` must become a type + that can carry a strength, and `Ltx2DitCheckpoint` must record WHICH adapter + state is applied rather than merely whether one is. That is modest growth, not + a redesign — the re-materialize-and-write-back mechanism is untouched, because + re-materializing from the pristine file already reaches any strength in one + pass. The trap is written beside the early return in `ltx2_loader.h` and + noted on #921 so it is inherited rather than rediscovered. + +- **`res2s_two_stage` runs BOTH stages at strength 1.0 where upstream runs 0.25 + and 0.5.** `Res2sTwoStageRecipe` is this tree's port of + `ti2vid_two_stages_hq.py`, neither of its phases sets `loras`, so both default + to `kAllAdapters`, and the load carries ONE strength for the whole engine + (`lora_strength` absent is 1.0, `ltx2_video.h:214-218`). Pre-existing and not + worsened by this row — the field this row adds is a set, and a set cannot + express a strength — but it was unstated anywhere until now. Owned by #921 + with the item above. - **N-adapter per-phase SUBSETS**, upstream's `(*loras, *distilled_lora)` where `loras` is the user's own list and rides both stages. Blocked on the adapter arity refusal (`ltx2_lora.h:167-172`), which is upstream-faithful for the pipelines that take exactly one and is not lifted here. - **A real-weights comparison against upstream's own render** on the same checkpoint, take and seed — upstream's stage 1 on base weights against ours. - This is the instrument `ltx25-a2vid-recipe.md` section 4.4 named, and it needs - the distilled adapter checkpoint, which `find /mnt/nas_share/checkpoints - -iname '*lora*'` returns nothing for. Owed against #1093's checkpoint - dependency. + This is the instrument `ltx25-a2vid-recipe.md` section 4.4 named. + + **WHAT IS OWED IS THE RUN, NOT THE ARTIFACTS.** An earlier draft of this + bullet said the comparison needs the distilled adapter checkpoint, "which + `find /mnt/nas_share/checkpoints -iname '*lora*'` returns nothing for". That + control now returns two adapters, and both were verified by reading the + safetensors header on 2026-08-17: + + | File | Bytes | Header | + |---|---|---| + | `ltx-2.5/lightricks-ltx-2.5/loras/ltx-2.5-22b-distilled-lora-450-bf16.safetensors` | 8,899,889,568 | 3320 BF16 tensors = 1660 `lora_A`/`lora_B` pairs, 4.450 B params, `lora_rank`/`lora_alpha` 450, `model_version` 2.5.0, data end == file size | + | `ltx-2.5/lightricks-ltx-2.5/ic_loras/ltx-2.5-22b-ic-lora-pixel-spatial-upscaler-x2-1.0.safetensors` | 327,322,640 | the IC-LoRA, a different adapter and not this row's | + + The first IS the file this row needs: the a2vid recipe's single `lora_path` + slot is upstream's `distilled_lora` (`--distilled-lora required=True`, + `utils/args.py:1140-1153`), which is what `requires_distilled_lora` mirrors. + + The full/dev transformer that upstream's stage 1 runs has since completed too, + at `ltx-2.5/lightricks-ltx-2.5/diffusion_models/ltx-2.5-22b-dev-transformer-bf16.safetensors`, + 42,018,190,584 bytes, header verified the same day: 4349 tensors, 21.004 B + params, BF16 4059 / F32 290, `model_version` 2.5.0, + `model.diffusion_model.keyframes_abs_pos_embedding` present, data end == + file size. + + So nothing is blocked on an artifact any more. What is owed is a GPU lease, a + build, both renders and the comparison. Note that the append-only index rows + for #1093 (`issue-index.md:318`) and #1118 (`:330`) still carry the + pre-arrival control, and those rows cannot be edited — this is the correction, + and it lives here. - **The reference clip's PIXEL path**, reason 1 of the refusal this row narrows. Still owed by #975. diff --git a/include/vllm/model_executor/models/ltx2_loader.h b/include/vllm/model_executor/models/ltx2_loader.h index f4424fbc7..f390e4ef1 100644 --- a/include/vllm/model_executor/models/ltx2_loader.h +++ b/include/vllm/model_executor/models/ltx2_loader.h @@ -521,6 +521,29 @@ Ltx2DitCheckpoint Ltx2StreamDitToDevice(vt::Queue& queue, const SafetensorsFile& // // Calling this with the state the checkpoint is already in is a no-op it detects // itself, so a single-phase recipe pays nothing. +// +// ─── AND THAT NO-OP IS THE TRAP THE NEXT ROW WALKS INTO ────────────────────── +// +// The state this detects is a BOOLEAN — `fuse` here, `lora_fused_tensors > 0` on +// the checkpoint — so "already in that state" means "already FUSED" and never +// "already fused AT THIS STRENGTH". Adding a per-phase strength on top of this +// signature therefore does not work, and it fails SILENTLY rather than by name. +// +// `TI2VidTwoStagesHQPipeline` is the case: it builds the SAME adapter twice, at +// `distilled_lora_strength_stage_1` and `distilled_lora_strength_stage_2` +// (`ti2vid_two_stages_hq.py:92-101`), and hands one to each stage (`:154`, +// `:165`). The CLI defaults them 0.25 and 0.5 (`utils/args.py:1174-1184`). BOTH +// stages are fused, so `currently_fused == fuse` holds at the stage boundary and +// this function returns having done nothing — stage 2 renders at stage 1's +// strength, with no refusal, no shape change and no wrong-looking output. +// +// So growing this seam for https://github.com/mudler/vllm.cpp/issues/921 needs +// TWO changes beyond a new field on `Ltx2PhaseRecipe`: `fuse` must become a type +// that can carry a STRENGTH, and `Ltx2DitCheckpoint` must record WHICH adapter +// state is applied rather than merely whether one is. That is modest growth, not +// a redesign — the re-materialize-and-write-back mechanism above is untouched by +// it, because re-materializing from the pristine file already reaches any +// strength in one pass. void Ltx2RebindDitLoras(vt::Queue* queue, const SafetensorsFile& file, const Ltx2DitLoadOptions& options, bool fuse, Ltx2DitCheckpoint& checkpoint); diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index 5e16c2140..d54945e57 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -2825,7 +2825,17 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // ever paid against weights the phase did not ask for. A no-op when the // load supplied no adapter, and a no-op when the DiT is already in the // requested state — so a one-stage recipe and every recipe that predates - // this field cost nothing, and a two-stage render pays exactly one rebind. + // this field cost nothing. + // + // WHAT A TWO-STAGE RENDER PAYS IS TWO REBINDS, not one, and the count is + // written out because it is the wall-clock half of the trade the row's spec + // accepted. `a2vid_two_stage` loads FUSED, phase 0 asks `kNoAdapters` and + // rebinds off, phase 1 asks `kAllAdapters` and rebinds back on; the DiT is + // left fused, so the NEXT render pays the same two. Each one re-opens the + // adapter and reads every A/B factor pair (`Ltx2LoraAdapter::Open` -> + // `ReadFactorAsBf16`), and the shipped distilled adapter is 8.9 GB. That + // cost is UNMEASURED on real weights; the row claims no wall-clock result + // and a later perf row owns the number. // // The emptiness test is HERE as well as inside the rebind so that a load // with no adapter does not re-open the checkpoint once per phase to be told From 10d9d010ee533f4f886676617320bc29e054bbeb Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 15:58:47 +0000 Subject: [PATCH 4/5] record(LTX25-PHASE-LORA): #921 is closed, so the per-phase strength debt gets an open owner (#1144) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The repair one commit back wrote the per-phase strength trap into `ltx2_loader.h` and this row's `## Owed` and said the note would also go on #921, which the fresh review named as the owner. #921 is CLOSED. It was closed as completed on 2026-08-17 at 12:28Z by `LTX25-RES2S-LOOP` (`4d7748646`, PR #1125), which landed the res_2s sampler four hours before this row's head. That row was right about its own scope: it names "the distilled LoRA per stage" as out of scope and unchanged by it. What it did not do is list the item under its own `## Owed`, so when the issue closed the debt outlived its owner. An `## Owed` bullet pointing at a closed issue does not own anything; it only looks like it does, and `check-agent-record.py` cannot see the difference because GitHub holds the open/closed state and the index deliberately does not. So #1144 is filed and this points at it: the spec's two strength bullets, the `Ltx2RebindDitLoras` header, and `Ltx2PhaseLoraScope`'s "NO PER-PHASE STRENGTH, deliberately" note, which was the one pointer at a closed issue sitting in a shipped header. The two "#921 when this was written" mentions in Scope and the port map keep the original number beside the new one, because the row genuinely was bounded by #921 on the day it was written and rewriting that would be rewriting history rather than the record. #1144 is a `bug` and not an enhancement, and the reason is the second half of the finding: `res2s_two_stage` — this tree's port of `TI2VidTwoStagesHQPipeline` — sets `loras` on neither phase, so both take `kAllAdapters`, and the load carries ONE strength (`lora_strength` absent is 1.0). It therefore runs both stages at 1.0 where upstream runs 0.25 and 0.5 (`utils/args.py:1174-1184`, placed at `ti2vid_two_stages_hq.py:92-101` and handed to `:154`, `:165`). That is a live divergence on a shipped arm, pre-existing and not worsened by this row, and it was unstated anywhere until now. It is not fixed in flow because the fix changes a seam signature and a checkpoint's state representation, which needs its own spec and its own fresh review. `.agents/issue-index.md` was taken from `origin/main` wholesale and the one row re-appended, per the record rule for a union-merged append-only file. Verified after writing: `origin/main`'s bytes are an exact prefix of the result, the row list is main's 315 plus exactly one, all 316 ids are unique, and no row that was on this branch before is lost. No build. `READER ANCHORS` re-derived unchanged at `823 833 834 896 992 1008 1043 1134 1159 1264 1305 1347 1349` with the same armed port of the test's own walk. The two source-text gates over `ltx2_pipeline.h` match flattened comment TEXT and are explicitly "derived rather than pinned to a line number", and neither of the strings they hold is touched here. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 4 +++ .agents/specs/ltx25-phase-lora.md | 26 +++++++++++++------ .../vllm/model_executor/models/ltx2_loader.h | 2 +- .../model_executor/models/ltx2_pipeline.h | 6 ++++- 4 files changed, 28 insertions(+), 10 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 9512f191a..f049d8d2c 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -325,6 +325,10 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1068](https://github.com/mudler/vllm.cpp/issues/1068) | `ENG-RELEASE-WINDOWS` | `main` stopped compiling under MSVC at `e34d71379` (#1054), which dropped the `[&kRequired]` capture from the `refuse` lambda in `qwen3_5_weights.cpp` as "the redundant namespace-scope capture". `kMoeExpertLayoutHelp` (`:894`) is namespace-scope and needs no capture; `kRequired` (`:929`) is a function-local `const std::string&` bound to it and IS odr-used in the lambda body, so MSVC rejects it (`error C3493`). Fixed by naming the namespace-scope constant inside the lambda, which satisfies MSVC and keeps the AppleClang diagnostic #1054 removed. It landed green because the guarding gate is a source-TEXT assertion ("rejects `[&kRequired]`, finds `[]`") that passes whether or not the TU compiles, and because `windows-msvc-*` are skipped on `main` (#503) so no baseline existed to regress. An instance of [#503](https://github.com/mudler/vllm.cpp/issues/503) | bug | | [#1092](https://github.com/mudler/vllm.cpp/issues/1092) | `LTX25-GUIDED-VIDEO` | The LTX-2.5 **video** denoise loop runs one UNGUIDED forward per step: `Ltx2PhaseRecipe::video_guidance` is set by every recipe (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1069 @ b5756ea8c`) and read by nothing, so a `pipeline_kind = one_stage` render ignores `cfg_scale = 3.0`, `stg_scale = 1.0`, `rescale_scale = 0.7` and `modality_scale = 3.0` and denoises along a different trajectory than `ti2vid_one_stage.py:221-226 @ fd4ded7f`, which builds a `FactoryGuidedDenoiser` from exactly those. `allow_guidance_override` (`ltx2_pipeline.h:534`) is dead the same way. Positive control for the grep: the same command for `audio_guidance` returns the T2A consumer at `ltx2_video.cpp:3527`. Blocks four more pipelines on one missing seam (`a2vid_two_stage.py:230`, `ti2vid_two_stages.py:248`, `ti2vid_two_stages_hq.py:271`, `keyframe_interpolation.py:232`). Spec [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) | bug | | [#1111](https://github.com/mudler/vllm.cpp/issues/1111) | `LTX25-GUIDED-VIDEO` | `Ltx2T2aGuidedStep` (`src/vllm/model_executor/models/ltx2_t2a.cpp:203-214` @ `420f6b474`) builds the STG block mask and refuses when NO bit is set, and that one predicate catches two different things. One is a request that disagrees with the CHECKPOINT - `[28]` on a two-block DiT - which is a LOCAL condition, because upstream only ever runs 48-block checkpoints and this port runs reduced ones. The other is an EMPTY list, which is upstream's OWN documented spelling for "STG off" and is refused here. Measured at Lightricks/LTX-2 `fd4ded7f`: `ltx-pipelines/docs/multimodal-guidance.md:13` says "Set to `[]` to disable STG", in the same table and the same idiom as `stg_scale` -> 0.0 and `cfg_scale` -> 1.0; `MultiModalGuiderParams.stg_blocks` DEFAULTS to `[]` (`guiders.py:204`, `field(default_factory=list)`); `--audio-stg-blocks` is `nargs="*"` (`args.py:1039-1045`, `:1107-1113`) so the empty list has a CLI spelling, and `nargs="+"` was the one-character way to forbid it; `LTX_2_3_HQ_PARAMS` SHIPS `stg_blocks=[]` on both modalities (`constants.py:105,:113`); and a whole-tree search found NO validation of `stg_blocks` anywhere - no emptiness, length or range check, no assert, no raise - with the null results recorded. `blocks=None` means EVERY block and `blocks=[]` means NO block (`perturbations.py:26-33`), so the empty list is how a caller says the second thing, and both this tree's parsers go out of their way to keep PRESENT-and-empty distinct from ABSENT - then refuse the one configuration that distinction exists to express. Upstream does not skip the pass either: `do_perturbed_generation` reads `stg_scale` alone (`guiders.py:279-281`), so an empty list disables the STG SIGNAL and not the STG COST. FOUND while repairing the fresh review of [#1092](https://github.com/mudler/vllm.cpp/issues/1092), whose B3 finding is the same divergence on the VIDEO path; that half IS fixed in flow (`ltx2_video.cpp` `ApplyGuidanceOverrides`, `ltx2_denoisers.cpp` `check_reaches_a_block`). NOT fixed here because this half landed with [#1032](https://github.com/mudler/vllm.cpp/issues/1032) and moving it changes a landed row's gated behaviour and its case "a perturbed pass over NO block is refused", which is the normal row-spec-review path rather than an in-flow repair. Until then the two arms disagree about the same request. Listed under `## Owed` in [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) | bug | +| [#1129](https://github.com/mudler/vllm.cpp/issues/1129) | `SPEC-MTP-K-GT-1` | The pinned vLLM oracle has NO lease-compliant path on `dgx.casa`, confirmed from two independent directions. The HOST has carried no toolchain since the 2026-08-14 reimage (no `gcc`, `cc`, `clang`, `ninja` or `nvcc`, no `/usr/include/stdio.h`), so the recorded cure is `sudo -n docker run` against `vllmcpp-build:gb10` or `nvidia/cuda:13.0.1-devel-ubuntu24.04`, reached over `ssh`, which bypasses the `rc` lease and makes the fleet report `dgx:gpu0` free while somebody is on it. `994d30b5d` (PR [#1116](https://github.com/mudler/vllm.cpp/pull/1116)) recorded that half and left the container re-check owed. The re-check is now DONE and CONFIRMS the blocker: probed through `rc run -d dgx:gpu0` on 2026-08-17 (job `ff28ada1-0cd3-4867-bf9b-f67050d0608b`), the leased worker runs as user `rc` in a k3s pod and carries no `gcc`, `cc`, `clang`, `nvcc`, `ninja`, `cmake`, `make`, `python3`, `pip`, `docker`, `sudo`, `git`, `ssh` or `curl`, no `/usr/include/stdio.h` and no `/usr/local/cuda*`, and `/home/mudler` does not exist inside it, so it cannot reach `~/venvs/vllm-oracle-pin-555967922` and could not start it if it could. `rc run` has no `--image` flag. So NO vLLM leg of any row can currently run on `dgx.casa` by any lease-compliant path, which blocks every oracle-dependent row: [#1003](https://github.com/mudler/vllm.cpp/issues/1003)'s twelve re-takes, [#915](https://github.com/mudler/vllm.cpp/issues/915)'s withheld cells, [#821](https://github.com/mudler/vllm.cpp/issues/821), and the MTP adjudication plus vLLM leg owed by [#81](https://github.com/mudler/vllm.cpp/issues/81). The only recorded lever is `/workspace`, which is NAS-backed, writable from the worker and the same folder on `dgx` and `thor`. The migration is deliberately NOT designed there. Listed under `## Owed` in [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) | bug | +| [#1130](https://github.com/mudler/vllm.cpp/issues/1130) | `ENV-GPU-LEASE-METHODOLOGY` | A merged pull request is necessary before removing a worktree and is NOT sufficient. The worktree for PR [#1035](https://github.com/mudler/vllm.cpp/pull/1035) was reaped while its branch carried two commits that were on neither `main` nor its own remote branch, preserved only by `rescue/es-cuda-grouped-unpushed` (tip `3ca7c23d8`). On a squash-only `main` ancestry carries no information: `git merge-base --is-ancestor` returns false for work that landed perfectly and `git cherry origin/main` marks landed commits `+`. Measured on that case, four instruments gave three different answers, and only `git log @{u}..HEAD` plus a CONTENT check agreed with the truth, which is that all three commits' content DID reach `main` in squash `b493f4981`. So the rule is two-part: verify `@{u}..HEAD` is empty AND that the content reached `main` by `git diff`/`git log -S`, because step 1 alone blocks a safe reap and step 2 alone allows an unsafe one. Fixed in flow in [`workflow.md`](workflow.md) `## Isolation`. Two sibling rescue refs remain unadjudicated, `rescue/cuda-breadth-sm75-audit` and `rescue/fp8-native` | bug | | [#837](https://github.com/mudler/vllm.cpp/issues/837) | `ROCM-GEMMA4-GETBLAS-DUALSLOT` | ROCm: dual-slot GetBlas TLS + host lifetime so two HIP streams do not share one handle | bug | | [#1117](https://github.com/mudler/vllm.cpp/issues/1117) | `LTX25-A2VID-RECIPE` | `A2VidPipelineTwoStage` (`a2vid_two_stage.py:53` @ `fd4ded7f`) has no recipe row, so `pipeline_kind = a2vid_two_stage` gets the generic table refusal (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1328-1332`) naming the pair rather than the missing machinery. [#922](https://github.com/mudler/vllm.cpp/issues/922) is CLOSED and closed the audio CONDITIONING, not the recipe: a supplied take rides `distilled_two_stage`, which [`ltx25-a2v-audio-input.md`](specs/ltx25-a2v-audio-input.md):438-446 already records as a different trajectory. Measured at `daeff67f2`: `git grep -n '"a2vid' -- src include tests docs examples` returns TWO hits, both upstream anchors inside `Fail`-message assertions (`tests/vllm/multimodal/test_ltx2_video.cpp:4363,:4427`), against a control of 4 for `"one_stage"` in `include/` alone. Four differences from the recipe it rides, each read at the pin: stage 1 is CFG/STG/modality-guided and caller-configured (`:230-240`, fed from `utils/args.py:947-1006`, `--a2v-guidance-scale` defaulting to `video_guider.modality_scale` = 3.0 at `utils/constants.py:54,:64`) where `distilled_two_stage` fixes `allow_guidance_override = false`; stage 1's schedule is scheduler-derived (`:225-227`) against our fixed `DistilledSigmas()`; stage 1 is plain Euler (`:229-258` passes no `stepper`, `utils/blocks.py:526-527`) against our `kEulerAncestral` on 2.5; and the AUDIO guider is the DEFAULT positive-only one (`:237-239`, `ltx-core components/guiders.py:200-210`) rather than the params table's cfg-7.0 row. Two non-schedule facts that must not be guessed: `--audio-path` is `required=True` (`:312-317`), and the distilled LoRA rides stage 2 ALONE (`:114` against `:107`) with `--distilled-lora` `required=True` (`utils/args.py:1140-1153`). Unblocked by `Ltx2GuidedDenoise` landing at `daeff67f2` (#1092/#1102), which [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) `## Owed` names this arm against. Spec [`ltx25-a2vid-recipe.md`](specs/ltx25-a2vid-recipe.md) | enhancement | | [#1118](https://github.com/mudler/vllm.cpp/issues/1118) | `LTX25-A2VID-RECIPE` | LoRA adapters fuse ONCE, at load, into ONE weight set: `src/vllm/multimodal/ltx2_video.cpp:816-820` is the only `dit_options.loras.push_back` in the tree and it runs under `if (!lora_path.empty())`, so every phase of every recipe sees the same fused `im.dit.weights`. Three upstream pipelines build TWO `DiffusionStage`s from the same checkpoint with DIFFERENT adapter sets at `fd4ded7f`: `a2vid_two_stage.py:107` against `:114-119`, `ti2vid_two_stages.py:140` against `:151`, and `ti2vid_two_stages_hq.py:154,:165` at two separate strengths (`:92-101`). `distilled.py:131` builds ONE stage set, which is why `distilled_two_stage`, `dfr` and `retake` have never needed this and no gate has ever asked. Consequence for the arm landing with #1117: that row mirrors `--distilled-lora required=True` (`utils/args.py:1140-1153`) by refusing an `a2vid_two_stage` load with no `lora_path`, so the 3-step stage-2 schedule cannot run on a checkpoint with no distilled adapter — but it CANNOT mirror upstream leaving stage 1 WITHOUT that adapter, so stage 1's guided schedule runs against base + distilled LoRA where upstream runs it against the base alone. That divergence RENDERS, and the PIXELS it renders are not upstream's: it moves the trajectory, so the frames themselves differ, while the frame count, the shapes, the sample rate and the errors are all exactly what they were — nothing in the SHAPE of the result says anything is wrong, which is why it is filed rather than left in a comment. It is not undetectable, and saying so would be the more damaging error: the instrument that WOULD see it is a real-weights comparison against upstream's own render on the same checkpoint, take and seed, upstream's stage 1 on the base weights against ours on base + distilled. Two fix shapes, neither chosen: a second `Ltx2DitWeights` per adapter set (what upstream pays, two `from_checkpoint` calls at `a2vid_two_stage.py:103,:115`) or unfused runtime LoRA selectable per phase; the first doubles resident DiT weights and the second changes the GEMM path. Bounds #1093 and #921, which need the same seam. Listed under `## Owed` in [`ltx25-a2vid-recipe.md`](specs/ltx25-a2vid-recipe.md) | enhancement | +| [#1134](https://github.com/mudler/vllm.cpp/issues/1134) | `BACKEND-ROCM` | The `VT_ATTN_DECODE_D128=1` ctest registration added by #767 cannot show the ROCm `d=128` decode arm REACHED the new kernel, and is empty off ROCm. `RegisteredDevices()` (`tests/vt/test_backend_cross_device.cpp:84-96`) enumerates `{kCUDA, kMETAL, kVULKAN, kXPU, kROCM}` and excludes `kCPU`, so on a CPU-only runner — which is what CI has — the new "Qwen3 geometry (bf16, GQA 2, head_dim 128)" case reports 1 test case, 0 assertions, exit 0, for BOTH registrations. On ROCm hardware the case's only backend assertion is `OpProviderStats::declines == 0`, and `OpProviderStats` counts at PROVIDER granularity, so it is identical with the flag set and unset; the NMSE bound passes on either kernel because the arm is correctness-complete. The two compose: there is no machine in this project on which the flag-ON registration distinguishes itself from the flag-OFF one. Disclosed in [`specs/rocm-decode-attn-d128.md`](specs/rocm-decode-attn-d128.md) §4, its `## Owed` section and its result banner, and §9 stop condition 2 is left OPEN rather than claimed discharged. Closing it needs a kernel-selection counter in `src/vt/rocm/rocm_paged_attn.hip` asserted to DIFFER between the two registrations; the CPU-runner half wants `kCPU` in `RegisteredDevices()` or a non-zero-assertion floor per [#463](https://github.com/mudler/vllm.cpp/issues/463). Not a duplicate of #463 (that is the unset-weights-env-var shape and does not describe the `declines` granularity half), #785 (a kernel that never LAUNCHES behind a dead `#if`, a code defect not a coverage one) or #900 (same family, LTX-2.5 subject) | bug | +| [#1144](https://github.com/mudler/vllm.cpp/issues/1144) | — | `Res2sTwoStageRecipe` is this tree's port of `TI2VidTwoStagesHQPipeline` (`ti2vid_two_stages_hq.py:59` @ `fd4ded7f`), the one in-scope pipeline that runs the SAME adapter on BOTH stages at DIFFERENT strengths: it builds `distilled_lora_stage_1` and `distilled_lora_stage_2` from one path (`:92-101`) and hands one to each `DiffusionStage.from_checkpoint` (`:154`, `:165`), CLI-defaulted 0.25 and 0.5 (`utils/args.py:1174-1184`). Here neither phase sets `Ltx2PhaseRecipe::loras`, so both take the `kAllAdapters` default, and the engine carries ONE strength for the whole load (`lora_strength` absent is 1.0, `include/vllm/multimodal/ltx2_video.h:214-218`) — so both stages run at 1.0. Nothing refuses and nothing changes shape; only a render against upstream on the same checkpoint, take and seed sees it. A NEW FIELD ON `Ltx2PhaseRecipe` DOES NOT CLOSE IT: `Ltx2RebindDitLoras` early-returns on `currently_fused == fuse` where `currently_fused` is `checkpoint.lora_fused_tensors > 0`, a BOOLEAN, so it detects "already fused" and never "already fused AT THIS STRENGTH" — and HQ has both stages fused, so the no-op its own header advertises would swallow the transition and stage 2 would render at stage 1's strength. Closing it needs `bool fuse` to become a type carrying a strength AND `Ltx2DitCheckpoint` to record WHICH adapter state is applied, plus a `--distilled-lora-strength-stage-1`/`-stage-2` request pair, since one `lora_strength` extra cannot spell two values. The re-materialize-and-write-back mechanism #1118 landed is untouched by that change. Filed because [#921](https://github.com/mudler/vllm.cpp/issues/921), which owned the per-phase strength in [`ltx25-phase-lora.md`](specs/ltx25-phase-lora.md), was CLOSED as completed on 2026-08-17 by `LTX25-RES2S-LOOP` (`4d7748646`, PR [#1125](https://github.com/mudler/vllm.cpp/pull/1125)); that row named the distilled LoRA per stage as out of scope and correct to leave, but did not list it under its own `## Owed`, so the debt outlived its issue with no open owner. Found during the review repair of PR [#1140](https://github.com/mudler/vllm.cpp/pull/1140). Listed under `## Owed` in [`ltx25-phase-lora.md`](specs/ltx25-phase-lora.md) | bug | diff --git a/.agents/specs/ltx25-phase-lora.md b/.agents/specs/ltx25-phase-lora.md index 51325c605..429fe49c3 100644 --- a/.agents/specs/ltx25-phase-lora.md +++ b/.agents/specs/ltx25-phase-lora.md @@ -23,7 +23,8 @@ adapter. One record repair: the reference-conditioning refusal in **Out.** `ti2vid_two_stages` (#1093) and `keyframe_interpolation` (#1096) — both additionally need checkpoints that are not on the NAS. The HQ arm's per-phase -STRENGTH (#921). N-adapter subsets. The reference clip's pixel path, which is +STRENGTH (#921 when this was written; #1144 since that issue closed). +N-adapter subsets. The reference clip's pixel path, which is the reference refusal's FIRST reason and stays refused. ## Upstream chain @@ -94,7 +95,7 @@ because `Ltx2ResolveLoraReferenceFactors` refuses more than one adapter by name needs one (0.25 at `:92-96`, 0.5 at `:97-101`), and no recipe this row ships would set it, so adding the field now lands a branch nothing can select — the argument `ltx2_lora.h:41-44` already makes for the second product form. Owed -below, against #921. +below, against #1144 (#921 when this was written). ### The mechanism @@ -417,7 +418,14 @@ shift. ## Owed - **Per-phase adapter STRENGTH**, which `ti2vid_two_stages_hq.py:92-101` needs - and this field cannot express. Owned by #921. + and this field cannot express. **Owned by + [#1144](https://github.com/mudler/vllm.cpp/issues/1144), not by #921.** This + spec said #921 until 2026-08-17, and #921 was closed as completed that same + day by `LTX25-RES2S-LOOP` (`4d7748646`, PR #1125), which landed the res_2s + sampler. That row named the distilled LoRA per stage as out of scope and was + right to leave it, but did not list it under its own `## Owed` — so the debt + outlived its issue with no open owner until #1144 was filed. An `## Owed` item + pointing at a closed issue is not owned; it only looks owned. **A NEW FIELD IS NOT ENOUGH, and the seam's own no-op is why.** `Ltx2RebindDitLoras` early-returns on `if (currently_fused == fuse) return;`, @@ -430,13 +438,13 @@ shift. would silently render at stage 1's strength, with no refusal and nothing wrong in the shape of the output. - So #921 needs two changes beyond the field: `bool fuse` must become a type + So #1144 needs two changes beyond the field: `bool fuse` must become a type that can carry a strength, and `Ltx2DitCheckpoint` must record WHICH adapter state is applied rather than merely whether one is. That is modest growth, not a redesign — the re-materialize-and-write-back mechanism is untouched, because re-materializing from the pristine file already reaches any strength in one - pass. The trap is written beside the early return in `ltx2_loader.h` and - noted on #921 so it is inherited rather than rediscovered. + pass. The trap is written beside the early return in `ltx2_loader.h`, in + #1144's body, and on #921 so it is inherited rather than rediscovered. - **`res2s_two_stage` runs BOTH stages at strength 1.0 where upstream runs 0.25 and 0.5.** `Res2sTwoStageRecipe` is this tree's port of @@ -444,8 +452,10 @@ shift. to `kAllAdapters`, and the load carries ONE strength for the whole engine (`lora_strength` absent is 1.0, `ltx2_video.h:214-218`). Pre-existing and not worsened by this row — the field this row adds is a set, and a set cannot - express a strength — but it was unstated anywhere until now. Owned by #921 - with the item above. + express a strength — but it was unstated anywhere until now. Owned by + [#1144](https://github.com/mudler/vllm.cpp/issues/1144) with the item above, + and it is the reason that issue is a `bug` rather than an enhancement: this is + a live divergence from upstream on a shipped arm, not a missing feature. - **N-adapter per-phase SUBSETS**, upstream's `(*loras, *distilled_lora)` where `loras` is the user's own list and rides both stages. Blocked on the adapter arity refusal (`ltx2_lora.h:167-172`), which is upstream-faithful for the diff --git a/include/vllm/model_executor/models/ltx2_loader.h b/include/vllm/model_executor/models/ltx2_loader.h index f390e4ef1..2181b0cee 100644 --- a/include/vllm/model_executor/models/ltx2_loader.h +++ b/include/vllm/model_executor/models/ltx2_loader.h @@ -537,7 +537,7 @@ Ltx2DitCheckpoint Ltx2StreamDitToDevice(vt::Queue& queue, const SafetensorsFile& // this function returns having done nothing — stage 2 renders at stage 1's // strength, with no refusal, no shape change and no wrong-looking output. // -// So growing this seam for https://github.com/mudler/vllm.cpp/issues/921 needs +// So growing this seam for https://github.com/mudler/vllm.cpp/issues/1144 needs // TWO changes beyond a new field on `Ltx2PhaseRecipe`: `fuse` must become a type // that can carry a STRENGTH, and `Ltx2DitCheckpoint` must record WHICH adapter // state is applied rather than merely whether one is. That is modest growth, not diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index f56643e6f..0656581f8 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -638,7 +638,11 @@ enum class Ltx2PhaseDenoiser { kGuided, kSimple }; // 0.25 at `:92-96` and 0.5 at `:97-101` — and no recipe this tree ships would // set it, so adding the field now lands a branch nothing can select. That is the // argument `ltx2_lora.h:41-44` already makes for the second product form. Owed -// by https://github.com/mudler/vllm.cpp/issues/921. +// by https://github.com/mudler/vllm.cpp/issues/1144 — NOT #921, which was closed +// as completed the same day this landed and would have left the debt looking +// owned while owning nothing. The trap that makes it more than a new field is +// written beside `Ltx2RebindDitLoras` in `ltx2_loader.h`: that function's no-op +// test is a BOOLEAN, and HQ needs both stages fused at different strengths. enum class Ltx2PhaseLoraScope { // Every adapter the load supplied. The DEFAULT, because `distilled.py:131` // builds ONE stage set and so every recipe that predates this field — From 08ea64b15ed2ba4f53ec27bd9fbd97da8dac377b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 16:06:58 +0000 Subject: [PATCH 5/5] docs(LTX25-PHASE-LORA): what the phase-scoped adapter costs per render, and which adapter --lora wants (#1118) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two facts a reader of the a2vid section can act on, both new with #1118 and neither of them stated anywhere until now. THE COST. Moving one DiT between two adapter states is paid in wall-clock rather than memory: a two-stage render does TWO rebinds, one per phase boundary, each re-opening `--lora` and reading every `lora_A`/`lora_B` factor pair before re-materializing the tensors they target. The DiT is left in stage 2's state, so the next render pays the same two. No number is published, because this recipe is gated on reduced fixtures and nothing has timed the boundary on real weights — saying so is the point, since the alternative is a reader assuming the seam is free. WHICH ADAPTER. The sample command has named `ltx-2.5-22b-distilled-lora-450-bf16.safetensors` since #1117 without saying how to know you have it. It is now pinned by CONTENT — 8,899,889,568 bytes, 3320 BF16 tensors forming 1660 `lora_A`/`lora_B` pairs, `lora_rank` and `lora_alpha` both 450, `model_version` 2.5.0 — because a LoRA repository can be re-quantized in place under an unchanged filename, which is exactly why AGENTS.md asks for the bytes and not the name. The header was read on the artifact, not copied from a model card. The IC-LoRA is named beside it as the file this is NOT, with its own size, and the page says plainly that nothing in the engine tells them apart: `requires_distilled_lora` refuses a load carrying no `--lora` and that is the whole of its check. ON THE SHAPE OF THIS SERIES, because the reader of `git log` deserves it rather than a puzzle. `check-doc-checkpoint.py` classifies any `include/vllm/` path as `user_usage` and requires `docs/USAGE.md` in the SAME commit, by path and without reading content. The two commits before this one changed only COMMENTS in two headers and carried no `docs/USAGE.md`, so both fail that gate per-commit. That is a real rule and they really broke it: the obligation is per-commit by design, because a diff-scoped range is never re-covered later. Repairing it in place would mean rewriting two pushed commits, and this session may not force-push, so the red is left standing and reported rather than hidden. `main` is squash-merge only, so the commit that lands carries this file and satisfies the gate on the branch that matters; the per-commit red is on the pull-request lane alone. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- docs/USAGE.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/USAGE.md b/docs/USAGE.md index 51f0327a6..a0dfcdbcb 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -3315,6 +3315,29 @@ engine still holds ONE DiT — upstream does too, since both of its re-materializes the adapter's target tensors at the phase boundary instead of keeping a second weight set. +**What that costs you, per render.** Moving one DiT between the two states is +paid in wall-clock rather than in memory: a two-stage render does **two** +rebinds, one at each phase boundary, and each re-opens `--lora` and reads every +`lora_A`/`lora_B` factor pair before re-materializing the tensors they target. +The adapter above is 8,899,889,568 bytes, so this is not free, and the DiT is +left in stage 2's state so the next render pays the same two. **No number is +published for it** — this recipe is gated on reduced fixtures and nothing has +timed the boundary on real weights. Upstream spends memory here instead, holding +two `DiffusionStage`s over one checkpoint, which does not fit one GB10. + +**The adapter `--lora` wants**, pinned by content rather than by name, because a +LoRA repository can be re-quantized in place under an unchanged filename: +`ltx-2.5-22b-distilled-lora-450-bf16.safetensors`, 8,899,889,568 bytes, 3320 +BF16 tensors forming 1660 `lora_A`/`lora_B` pairs, +`__metadata__` `lora_rank` and `lora_alpha` both `450` and `model_version` +`2.5.0`. This is upstream's `distilled_lora`, the one `--distilled-lora` +(`required=True`) names. It is **not** the IC-LoRA +(`ltx-2.5-22b-ic-lora-pixel-spatial-upscaler-x2-1.0.safetensors`, 327,322,640 +bytes), which is a different adapter for a different arm. Nothing here checks +which one you passed: `requires_distilled_lora` refuses a load carrying **no** +`--lora`, and that is the whole of it, so the two are told apart by the header +facts above and not by the engine. + The guider flags (`--video-cfg-guidance-scale` and the rest, spelled as the `video_cfg_guidance_scale` extras over the C API) reach stage 1 and are ignored by stage 2, which runs no guider at all — unlike `distilled_two_stage` and