diff --git a/.agents/issue-index.md b/.agents/issue-index.md index d7a25b34c..ef5ab4e71 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -365,12 +365,17 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1171](https://github.com/mudler/vllm.cpp/issues/1171) | `KERNEL-GDN-REPLAYSSM` | GDN decode rewrites the whole `[HV,V,K]` fp32 state every step (`src/vt/cuda/cuda_gdn.cu:2393` reads the tile, `:2425` writes it back), which at the 27B shape `HV=32, V=128, K=128` is 2,097,152 bytes read plus the same written per layer, per request, per token. ReplaySSM keeps a per-slot ring of the last `L` steps' rank-1 factors `(d, k, g)`, reconstructs the state in registers, and writes it back only every `L` steps. vLLM implements the algorithm at the pin `555967922` for Mamba2 selective-state ONLY (`layers/mamba/ops/selective_state_update_replayssm_output_only.py`, ring shapes/dtypes `mamba_utils.py:84-93,202-221`, `use_replayssm` default `False` at `config/cache.py:152`, introduced `866fea2b` #48018) and it cannot reach GDN: `config/vllm.py:2318-2322` refuses any model not setting `supports_replayssm`, only `NemotronHForCausalLM` sets it (`models/nemotron_h.py:711`), `GDNAttentionMetadataBuilder` does not subclass the Mamba builder that derives the cursor (`v1/attention/backends/gdn_attn.py:82` vs `mamba_attn.py:575-638`), and the kernel hard-requires a scalar-per-head `A` (`:540-542`) with the Mamba2 `(B,C)` group structure (`:529`). Still true 877 commits past the pin. SGLang ported it to GDN at our recorded pin `f63458b5` (`layers/attention/fla/fused_recurrent_linear_replayssm.py`, whose `:50` credits vLLM; `--enable-linear-replayssm` default `False` and `--linear-replayssm-cache-len` default 16 at `server_args.py:1972-1986`; rings `memory_pool.py:465-483`; commit `a10a24e9` #28451), so the algorithm is a vLLM mirror and the GDN application is a secondary-oracle port. PAYOFF UNMEASURED HERE and deliberately not claimed: ReplaySSM removes the state WRITE and not the read, the flush step reads the checkpoint a SECOND time, so the honest state ratio is `(1+2/L)/2` = 0.5625 at `L=16` against SGLang's published 0.53x which models neither the flush re-read nor any ring read; the ring itself adds 395,264 bytes per slot per layer at the 27B shape = **+18.9% KV page**, worse than vLLM's ~7% on Nemotron because GDN's state is `V*K` while the ring is `L*(V+K)`; and SGLang's own end-to-end figure is ~2.3% TPOT at 128 concurrency on an MoE model. Neither upstream is bit-exact against its unbuffered path and neither claims to be. Motivation is the open Qwen3.8-27B bf16 decode gap (c4 total 0.918x, output 0.963x, `docs/BENCHMARKS.md:192-205`). Spec [`gdn-replayssm.md`](specs/gdn-replayssm.md) | perf | | [#1179](https://github.com/mudler/vllm.cpp/issues/1179) | `ENG-CUDAGRAPH-BREAK` | The hand-rolled decode-graph driver count recorded in `9bc4d7f44` is **eight** and is actually **nine**, and the row it feeds was framed as coverage-only when it is also correctness. The ninth is the DFlash draft graph, file-local with no header declaration, at `src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106` — its own `int g_state = 0` three-state machine (`:771`), its own `VT_DFLASH_GRAPH` kill switch (`:870`) instead of the `VLLM_CPP_CUDAGRAPH` the six batched drivers read, its own invalidate-on-block-width-change (`:1038-1047`) and its own `try { EndCaptureGraph(); } catch (...) {}` drain (`:1106`). The eight-count is stated in four places, all corrected here: [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) §4 and `## Owed`, [`.agents/engine-matrix.md`](engine-matrix.md) rows `ENG-CUDAGRAPH-BREAK` and `ENG-CUDAGRAPH-DEDUP` ("times eight drivers", which sizes #1162's signature table), and [`.agents/roadmap_v1.md`](roadmap_v1.md) track `C12`. The reframing is the substantive half: `ENG-CUDAGRAPH-BREAK` was recorded as a COVERAGE row, and the duplication has already cost a SHIPPED model its decode graph. `src/vllm/model_executor/models/qwen3.cpp:961-986` declines the decode graph outright whenever the asynchronous device-token mirror is live, on its own measured battery — `depth-1, graph ON PASS 78/78`; `depth-2, graph OFF PASS 82/82`; `depth-2, graph ON FAIL, slots 1-3 degenerate` — because `Step()` replays against the HOST `input.token_ids` and the combine has patched the DEVICE ids. The comment names the real fix as reading the identifiers at replay time from a stable device buffer, and that fix exists, in exactly one sibling driver, as `StepDevInputs` (`src/vllm/model_executor/models/qwen3_5.cpp:3894`): `grep -c StepDevInputs` returns 41 lines there and 0 in each of `qwen3_moe.cpp`, `qwen3.cpp`, `deepseek_v2.cpp` and `voxtral.cpp`. One capability, written once, unavailable to four models, with a live mitigation standing in its place. This does NOT weaken the framing rule that `ENG-CUDAGRAPH` established: the row still makes no throughput claim, and the prefill refutation (GB10 3.8% host-idle between launches, GPU-busy >96%, 27B prefill gap 92.5% non-GEMM glue) stands unchanged. Coverage AND correctness, never speed. Fixed in flow with the [`eng-cudagraph-break.md`](specs/eng-cudagraph-break.md) review repair ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)) | record | | [#1181](https://github.com/mudler/vllm.cpp/issues/1181) | `FIX-READ-F32-SCALAR-GUARD` | `ReadF32Scalar` (`src/vllm/model_executor/models/qwen3_5_weights.cpp:312-318` @ `ab6e65216`) bounds its input with `t.data != nullptr && t.nbytes >= sizeof(float)`, a LOWER bound, and then `memcpy`s four bytes into a `float`. Two silent wrong-value paths follow and neither fails: an ARRAY is reduced to element 0, so a block-wise FP8 scale grid of shape `[ceil(N/128), ceil(K/128)]` passes and stands in for the whole weight (measured under [#1166](https://github.com/mudler/vllm.cpp/issues/1166) on `Qwen/Qwen3.8-27B-FP8` @ `017b9c7af6b5689d5dd426a76e0bc077eb5ca20a`, `q_proj.weight_scale_inv` is `[96, 40]`), and ANY dtype is reinterpreted, since that same tensor is `BF16` and its four bytes are two bf16 values read as one float. Both return a finite plausible float, so the output is fluent, plausible and wrong, which is what a token gate cannot see. Upstream makes both facts structural rather than optional: a per-tensor scale is a distinct parameter TYPE that asserts `loaded_weight.shape[0] == 1` (`vllm/model_executor/parameter.py:260-272,304-309` @ `555967922`, plus the `_assert_and_load` shape assert at `:93-96`), the slot is allocated `torch.float32` so a narrow on-disk dtype is VALUE-converted rather than reinterpreted (`utils/fp8_utils.py:1276`), and the declared strategy TENSOR/CHANNEL/BLOCK picks the parameter type before a byte is read (`compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py:63,128`). The AUDIT corrects the issue's own framing twice. The 27 grep hits across five files are 5 definitions, 20 call sites and 2 comment references, and both counts are short: `ReadCtF32Scalar` (`include/vllm/model_executor/models/dense_weight_loaders.h:376`) is a SIXTH copy of the same defect under another name, reached from a SIXTH model file (`src/vllm/model_executor/models/qwen3_weights.cpp:100,126-128` through `LoadCtNvfp4W4A16`). Of the six, three check nothing, `LnReadF32Scalar`/`ShReadF32Scalar` check dtype but not count, and only `nemotron_h_weights.cpp:557-573` is correct, which makes it the model the shared guard generalizes. No call site legitimately passes a multi-element or non-F32 tensor, and every existing fixture emits rank-0 or `{1}` `F32`, so nothing in the tree needed the leniency. It is NOT merely latent: `dense_weight_loaders.h:73-74` and `docs/BENCHMARKS.md:52` both record `unsloth/Qwen3.6-27B-NVFP4` @ `ccdaab7e` as FP8 W8A8 throughout with BF16 PER-OUTPUT-CHANNEL scales, and `LoadAttnDense` branches on the weight dtype alone (`qwen3_5_dense_weights.cpp:478-480`), so those projections enter the per-tensor arm and hit both defects at once under the tensor name the loader actually asked for, with no misspelling to stop them. Fixed in flow by one `dense_loaders::ReadF32Scalar(get, name)` that refuses `numel != 1` naming the shape, refuses a non-`F32` dtype naming the dtype, and requires exactly four readable bytes, with the other five copies deleted onto it and `nemotron_h`'s `Loader`-based twin kept as the one tracked exception. A narrow dtype is refused rather than converted, because a one-element BF16 scale has never been read correctly here and the BF16 layout that IS shipped is per-channel, which the count check refuses first. Per-channel FP8, block-wise FP8 and any explicit narrow-dtype conversion stay owed. Spec [`read-f32-scalar-guard.md`](specs/read-f32-scalar-guard.md) | bug | +| [#1187](https://github.com/mudler/vllm.cpp/issues/1187) | — | The image-conditioning request surface is TWO FIXED SLOTS against upstream's repeatable `--image PATH FRAME_IDX STRENGTH [CRF]` (`packages/ltx-pipelines/src/ltx_pipelines/utils/args.py:805-817` @ `fd4ded7f`), which both conditioning builders expand per item (`utils/helpers.py:272-308` and `:343-367`). This ABI carries `first_frame` / `first_frame_ppm` and `last_frame` (`include/vllm.h:947-948`) at the FIXED indices `latent_idx = 0` and `frame_idx = frames - 1`, with one shared `noise_aug` and one shared `image_crf`, so an interior keyframe, a third keyframe, and two keyframes at different strengths or CRFs are each unrequestable; `ltx2-gen` exposes only `--first-frame` and `/v1/videos` forwards only a first frame. `num_generated_keyframes` is a DIFFERENT feature — model-invented interior slots (`keyframe_slots.py:71-150`) — and is not a substitute. SECOND HALF, filed together because it is the same surface: the sigma-BINNED guider factory. `FactoryGuidedDenoiser` resolves a guider per step from sigma (`utils/denoisers.py:332-343`) through `MultiModalGuiderFactory.build_from_sigma` (`ltx-core/components/guiders.py:337-342`), but every `main()` in `ltx-pipelines` passes plain `MultiModalGuiderParams`, so `create_multimodal_guider_factory` takes `constant()` and builds ONE `(inf, params)` bin (`guiders.py:312-315`) — which is exactly what `Ltx2GuidedDenoise` mirrors. The unported arm is `MultiModalGuiderFactory.from_dict` (`guiders.py:317-330`), reachable only by a caller who builds a factory and passes it as `video_guider_params`; this engine holds one `Ltx2MultiModalGuiderParams` per phase and has no sigma-keyed spelling. This CORRECTS the framing of [#1096](https://github.com/mudler/vllm.cpp/issues/1096), which named the per-sigma denoiser as a blocker on `KeyframeInterpolationPipeline` without deriving that the pipeline's own default path never resolves more than one guider. Filed rather than fixed in flow because it changes `include/vllm.h`, the video-engine request struct, `ltx2-gen`, the server request mapping and the conditioning loop, needs its own goldens for per-item strength and CRF, and narrows EVERY pipeline that takes `images` rather than one recipe. Listed under `## Owed` in [`ltx25-keyframe-interp.md`](specs/ltx25-keyframe-interp.md) | feature | +| [#1191](https://github.com/mudler/vllm.cpp/issues/1191) | `LTX25-KEYFRAME-INTERP` | `ltx2-gen` parsed `--first-frame` (`examples/ltx2_gen/main.cpp:333`) and assigned `vp.first_frame` (`:413`) while never reading `last_frame` at all, though `vllm_video_params` has carried the field since `include/vllm.h:948` and the LTX-2.5 engine has SERVED it since row `LTX25-TOKEN-APPEND` ([#930](https://github.com/mudler/vllm.cpp/issues/930)): the `wants_last_frame` arm of the phase loop places the image as a `VideoConditionByKeyframeIndex` at pixel frame `frames - 1`, gated by `test_ltx2_video`. So a closing keyframe was reachable from the C API and not from the shipped CLI. Latent until row `LTX25-KEYFRAME-INTERP` ([#1096](https://github.com/mudler/vllm.cpp/issues/1096)), which is the first caller it actually narrows: `KeyframeInterpolationPipeline` exists to generate the motion BETWEEN pinned keyframes, and with one slot the CLI can only ask for half of that. Fixed in flow by that row — one `--last-frame` flag parsed and assigned beside `--first-frame`, sharing the same `--image-crf` and strength the two slots already share, plus the usage and help text. An INTERIOR keyframe stays unrequestable and is [#1187](https://github.com/mudler/vllm.cpp/issues/1187) | bug | | [#1185](https://github.com/mudler/vllm.cpp/issues/1185) | `ENV-ORACLE-WHEEL-IN-LEASE` | The pinned vLLM oracle BUILDS, installs, imports and sees the GPU inside an `rc` lease on `dgx:gpu0`, measured 2026-08-18, which falsifies the `nvcc` clause four records carried. [`lease-runtime-staging.md`](specs/lease-runtime-staging.md) said the oracle "needs `nvcc`, which the worker still lacks", and `.agents/environment.md`, [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) and [`gpu-lease-methodology.md`](specs/gpu-lease-methodology.md) each derived a blocker from it. The build job (`buildvllm.sh`, staged sha256 prefix `15e140d41f44e7c2`) asserted the checkout against the pin BEFORE compiling, printing `PIN CONFIRMED` at `5559679229bc961848b121ccdeaa8fa5d79bec98` and aborting otherwise, took `nvcc` from the toolkit row `MODEL-NEMOTRON-H-ABI-A3-E2E` staged (`NVCC_RC=0`, CUDA `release 13.3, V13.3.73`) and produced `WHEEL_RC=0`, `PERSIST_RC=0` and a 434 MiB `vllm-0.1.dev1+g555967922.cu133-cp312-cp312-linux_aarch64.whl`, sha256 `7c58b339741a288fbb313f4f5196c9c92a9e3b3c3ebe2ea970b0ff50bb9bcba4`. The identity job (`oracleenv.sh`, prefix `6119f5223f5d818c`) asserted from `cd /`, outside any source tree: `vllm.__version__ = 0.1.dev1+g555967922`, `IDENTITY_RC=0`, `cuda True NVIDIA GB10`, `CUDA_RC=0`. SCOPE, and it carries the same weight as the result: RUNNING A MODEL IS UNTESTED. Only build, install, import and `torch.cuda.is_available()` are measured, and [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) records that the last time an oracle reached this far it consumed the host in the step AFTER `torch.compile` and REBOOTED the box, at `gpu_memory_utilization` 0.75 and again at 0.30, so the fraction is not the lever. The version string is an OPEN discrepancy: `.agents/upstream-sync.md` records `vllm_runtime_version = 0.23.1rc1.dev1511+g555967922`, the commit segment matches and satisfies the pin's binding `+g` rule, and the prefix differs because a shallow fetch stops `setuptools_scm` counting commits since the last tag, so a full-string gate needs a deeper fetch or a recorded pretend-version. The venv is NOT staged, because that job was killed at a 90-minute ceiling mid-copy and its partial tree was removed, so only the WHEEL is durable. Four staging walls, all artifacts of the NAS rather than of CUDA: `cp -a` preserves `file_mode=0664` so `nvcc` exited 126. CIFS `nounix` stores no symlink so `include` and `lib64` vanished and CMake reported `Could NOT find CUDA (missing: CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) (found version "13.3")`, naming the version and denying the toolkit in one line. 32 library links `libfoo.so` and `libfoo.so.MAJOR` had to be rebuilt because only the `libfoo.so.X.Y.Z` real files survived. And `markupsafe` existed as a dist-info with NO package files from a `pip --target` killed at a 35-minute ceiling, so Marlin codegen died on `ModuleNotFoundError`. The `rc` worker container is REUSED between jobs, so a repair inside a staging branch is skipped on the next run (`nvcc already in place`) and an environment repair must be unconditional and assert its postcondition. CONSEQUENCE for the rows #1129 blocked, [#1003](https://github.com/mudler/vllm.cpp/issues/1003), [#915](https://github.com/mudler/vllm.cpp/issues/915), [#821](https://github.com/mudler/vllm.cpp/issues/821) and [#81](https://github.com/mudler/vllm.cpp/issues/81): UNBLOCKED FOR THE BUILD STEP and STILL BLOCKED FOR A MODEL RUN. None can take a measurement until a model run is demonstrated. Job details, walls and non-claims in [`oracle-wheel-in-lease.md`](specs/oracle-wheel-in-lease.md) | verification | | [#1184](https://github.com/mudler/vllm.cpp/issues/1184) | `ENG-CUDAGRAPH-DEDUP` | `VT_CUDA_GRAPH_DEDUP=1` cannot complete a single decode step once a CUDA graph is actually captured. Measured 6/6 deterministic on GB10: `vt graph dedup: captured 1 graphs, deduped to 1 execs`, then `[Qwen3DenseDecodeGraph] captured dense decode graph for padded size S=8`, then `engine-fatal: EngineCore busy loop threw: vt cuda: greedy_argmax launch: invalid device function` -- from a `greedy_argmax` launch that had SUCCEEDED. The OFF and `=0` arms are clean and byte-identical across 7 runs on the identical binary and libraries, which is what makes the asymmetry arm-attributable. `greedy_argmax` is not at fault. `src/vt/graph_dedup_runtime.h` @ `3ed6253c6` walks the retained raw graph with CUDA runtime APIs and is DESIGNED to see several of them fail -- the `cudaGraphExecUpdate` probe refusing a fold is the feature working, not an exception, and the topology walk has five more escapes that degrade the key rather than abort inside a capture -- but it never consumed the runtime's sticky per-thread error: `grep -n 'cudaGetLastError\|hipGetLastError' src/vt/graph_dedup_runtime.h` returned nothing. Twelve sites swallowed a non-success return without clearing (`:98,100,115,130,137,146,154,173,179,281,307,317`); the two that mattered are the probe (`:307,:317`), whose refusal is routine, and `cudaGraphInstantiate` (`:281`). The next unrelated kernel, launched with the ordinary `kernel<<<>>>(); Check(cudaGetLastError())` pattern, then reported our refusal as its own failure. Every symptom follows: it needs BOTH `dedup=1` and a real capture, `CUDA_LAUNCH_BLOCKING=1` does not move it because the latch is host-side and synchronous rather than a deferred async error, and `cudaGraphLaunch` returns success because a return value does not consume the latch. Fixed in flow, structurally rather than site-by-site: twelve hand-placed clears are a fix the thirteenth fallible call misses, so the clear lives in `ScopedLatchClear`'s destructor (`src/vt/graph_dedup_latch.h`, new) installed at the binding's entry points, which are exactly the six `GraphDedupOps` members; `MakeLatchGuardedOps` is the table's only constructor and takes the raw functions as template arguments, so no raw address reaches a field and a seventh operation wired elsewhere leaves its field null, which the registry constructor refuses. One line covers CUDA and HIP because there is one source. The coverage gap that hid it is closed in the same change: the device-free half of the signature walk moves to `src/vt/graph_dedup_signature.h` and is gated by the new `tests/vt/test_graph_dedup_runtime.cpp` (13 cases, 51 assertions, 7/7 negative mutations detected) -- Kahn ordering, topological re-index, sorted edge emission, the depth-4 child bound and the four graph-level escapes, none of which was reached by any test on any tier before. STATED LIMIT: a CPU test drives a fake runtime and cannot observe the real latched error, so it proves the guard's structure and not the device outcome; the device A/B re-run is owed under `## Owed` in [`eng-cudagraph-dedup.md`](specs/eng-cudagraph-dedup.md), which also records that the shipped async serving path never reaches this feature at all | bug | | [#1192](https://github.com/mudler/vllm.cpp/issues/1192) | `ENG-CUDAGRAPH-BREAK` | W1 of the break-point capture seam: `vt::BreakableGraph`, `vt::GraphCaptureScope` and `vt::GraphBreak` land in `include/vt/breakable_graph.h` + `src/vt/breakable_graph.cpp`, with the SGLang unit suite ported case for case and ONE break point registered on ONE model. **The stage's exit criterion was answered first, on a leased GPU, because W0 deliberately did not answer it**: `cudaStreamEndCapture` followed by `cudaStreamBeginCapture` on the SAME stream mid-forward with EAGER work between them is LEGAL under `cudaStreamCaptureModeThreadLocal`, our backend's mode (`src/vt/cuda/cuda_backend.cu:204-206`) — measured on `orin:gpu0` through an `rc` lease, driver 12060, three replays with fresh inputs and 0 mismatches, plus the bare zero-work re-begin (`breakable_cuda_graph.py:370-374`). The first probe REFUSED and that refusal was the probe's, not CUDA's: `dlsym` on the bare symbol name binds the LEGACY v1 entry points, which are not capture-aware, so `cuMemcpyDtoDAsync` returned `CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED` and `cuStreamBeginCapture` (v1) silently took no capture-mode argument — a broken instrument failing toward a verdict about the code. Bound by exact versioned name, the criterion holds. The break point is the DENSE ATTENTION ENTRY of `Qwen3ForCausalLM` (`src/vllm/model_executor/models/qwen3.cpp`, inside `RunLayer`), which is vLLM's boundary and not one this row invents (`splitting_ops` defaulted to the attention family, `vllm/config/compilation.py:517,764-772,1145` @ `5559679229`) registered in SGLang's form because we have no compiler to match operation names against (`layers/radix_attention.py:256` @ `f63458b5be`). The G2 reachability gate is `tests/vllm/models/test_qwen3_break_point.cpp`: it drives the production `Qwen3DenseModel::Forward` with a scope open and counts `num_hidden_layers + 1` segments, so deleting the one call site takes it to 1 and turns the gate RED — a unit test that constructs the container by hand could not have asked that. It also holds G4 in the same case: the logits are compared BIT FOR BIT against the unscoped forward, 500 values, 0 differing. **STAGED SLICE, named rather than implied:** `GraphCaptureScope` and `BreakableGraph` are not yet ENTERED from a production step, because no driver opens a scope until W2 migrates `Qwen3DenseDecodeGraph`; the break point itself is on the production path on every forward and takes the pass-through arm. Listed under `## Owed` in [`eng-cudagraph-break.md`](specs/eng-cudagraph-break.md). The capture-failure drain is NOT owed: it landed here, as behaviour and as three gated arms. The destructor `catch` alone covered a throwing `EndCaptureGraph` and nothing else, so a break function or ordinary model code throwing mid-capture left a container reporting `captured() == true` over a PARTIAL forward; comparing `std::uncaught_exceptions()` against the depth recorded at scope entry is what tells those apart, and tests 13a, 13b and 13c gate the three arms. Still owed by later stages and NOT claimed here: the auxiliary-stream auto-join before every segment close (spec D10, live at `qwen3_5.cpp:6254-6255,6384` and `laguna.cpp:2572-2576,2612`, which W4 and W5 migrate), the non-capturing-backend arm on ROCm and Tenstorrent (G5), G1 bit-exactness on a real GPU over more than one replay, and one residual the fresh review surfaced: an exception CAUGHT INSIDE the capture scope leaves the segment closed and the rest of the forward uncaptured while `captured()` stays true, which the drain cannot see because nothing is unwinding at scope exit (spec `## Owed`, W2). No throughput claim is made anywhere in this change: the prefill refutation stands (GB10 3.8% host-idle between launches, GPU-busy above 96%, 27B prefill gap 92.5% non-GEMM glue) | feature | | [#1189](https://github.com/mudler/vllm.cpp/issues/1189) | `VT-QUANT-FP8-GROUP` | Block-wise (128x128) FP8 so `Qwen/Qwen3.8-27B-FP8` runs instead of being refused. `weight_block_size` appears nowhere in `src/` or `include/`, so `469f38395` refuses the arm by name (#1166). Six independently landable milestones; M1 lands here. M1 is `vt::QuantFp8Group`, the dynamic per-token per-group activation quant, CPU and CUDA. The numerics mirror the kernel that ACTUALLY EXECUTES on a CUDA-alike platform with a contiguous input, which is the C++ custom op at `csrc/libtorch_stable/quantization/w8a8/fp8/per_token_group_quant.cu:42-96` and NOT the Triton kernel at `fp8_utils.py:95-150`: `fp8_utils.py:635-650` calls the former and returns before the latter. The two arms differ, and the difference is measurable rather than cosmetic. The CUDA kernel divides twice, `local_absmax / max_8bit` at `:68` and `static_cast(src) / y_s` at `:85`; the Triton kernel multiplies by `(1.0 / fp8_max)` at `fp8_utils.py:145` under a comment that names the 1-ULP gap. Upstream's own test tolerates the gap with `rtol=0.15` (`test_block_fp8.py:112-114`), so a value comparison cannot tell the two apart and only a byte comparison against a spelled-out reference can. `eps` is the reduction's INITIAL value (`:47`), not a post-clamp, which is what keeps an all-zero group from dividing by zero. Scope refused here and owed to later milestones: the block-scaled GEMM (M2), `Fp8BlockWeight` and the loader (M3), `Fp8BlockLinearMethod` and the Qwen3.5 wiring (M4), the mainloop-scaled CUTLASS kernel and the column-major/TMA-aligned scale layouts (M5), merged `gate_up`/QKV (M6). M1 lands UNREACHED: no production entry point dispatches `vt::QuantFp8Group` at its merge commit, M4 owns the wiring, and `.agents/specs/vt-quant-fp8-group.md` lists it under `## Owed`. The CUDA arm compiles and its on-hardware leg is owed too, because the row took no GPU lease by design: the CPU arm is the gateable one | feature | | [#1190](https://github.com/mudler/vllm.cpp/issues/1190) | `ENV-AGNOSTIC-CAMPAIGN` | One operator's hosts, share paths and addresses are written into 227 tracked files where a `.env` placeholder belongs, so a second developer who follows the protocol documents is told to reach a box on one home network. Re-derived at `fd64c76ee` with `git grep -cIE 'dgx\.casa\|nas_share\|192\.168\.\|thor:gpu0'`: `dgx.casa` 203 files, `nas_share` 32, `192.168.` 24, `thor:gpu0` 11. The mechanism was already there and almost unused: `.env.example` is tracked, `.env` and `.agents/developer-preferences.md` are ignored, `scripts/agent-onboard.py --env-set` already records one answered value and refuses an undeclared key, and `${VLLM_SOURCE}` and `${GPU_LOCK}` already resolve in 59 and 50 files while `${DEVICE_ARCH}`, `${DEVICE_TOOLKIT_ROOT}` and `${DEVICE_COMPILER}` resolve in none. This SCOPING row lands the rule and the mechanism, not the sweep. The rule is a substitution test: replace the literal with a second developer's value and if the sentence stays true it is CONFIGURATION and becomes `${KEY}`, and if it becomes false it is PROVENANCE and stays literal. That reverses the issue's own ranking, because all 34 hits in the densest guide file `.agents/environment.md` are provenance or named-profile definition and none is configuration, so density does not predict the defect and a blind `sed` would falsify records. Landed: three new keys `GATE_CHECKOUT`, `SHARED_STORAGE_ROOT` and `GATE_DEVICE` derived from the literals that recur and map to no existing key; the create-on-first-use route in `scripts/agent-start.py`, which printed `environment: missing` as a status label and then listed next actions that never mentioned it, so the fallback in practice was a host name copied from a document; the matching obligation in `AGENTS.md`; and one worked example. The worked example `scripts/dgx-bringup.sh` found a live defect rather than a cosmetic one: it defaulted `CUTLASS_DIR` to `$HOME/cutlass_probe` while `.agents/environment.md:389` records `$HOME/cutlass-4.5.0` as mandatory on the same box, and a configure that misses CUTLASS silently drops the sm120a NVFP4 GEMM and FlashAttention-2, which that file measures as moving the SACRED `test_qwen27_paged_engine` from 235/235 to 234/235 with the source untouched, so a stale hard-coded default is a false green. Its new test also caught that `set -a; . ./.env; set +a` over an `.env.example` copy blanks a value the caller exported, so the process-environment contract is now executable. Waves `ENV-AGNOSTIC-W1-TOOLING` through `ENV-AGNOSTIC-W5-LEDGERS` own the sweep, partition all 227 files, and open their own issues. Spec [`env-agnostic.md`](specs/env-agnostic.md) | bug | | [#1193](https://github.com/mudler/vllm.cpp/issues/1193) | `SPEC-DSPARK-QWEN3-ROUTING` | A Qwen3 DSpark draft declaring `architectures=["DSparkDraftModel"]` with `model_type` `qwen3` has no route. The pin forces every DSpark draft that is not `Qwen3DSparkModel` or `Gemma4DSparkModel` onto `model_type` `deepseek_v4` (`vllm/config/speculative.py:934-944` @ `555967922`), and vLLM PR 52197 (merged 2026-08-17 at `7075ddac`) replaced that with a leading branch normalizing the pair to `Qwen3DSparkModel`. We diverge from BOTH: the forced rewrite was never ported, so nothing in `src/vllm/entrypoints/model_loader.cpp` reads a draft config's `architectures` key at all, and `SpeculativeConfig::IsDsparkDraft` (`include/vllm/config/speculative.h:120-136`) has no production caller — every reference outside its header is in `tests/vllm/config/test_speculative_dspark.cpp:132-140`, and `ResolveSpecConfig` branches on `cli.method` alone. The checkpoint is real and gateable here: `RadixArk/Qwen3.8-27B-DSpark` at revision `85ef153be924f17ce4bf62726954eeaa4a73e854` carries exactly that config shape in one 2718576122-byte shard, drafting five layers for a 64-layer Qwen3.8-27B target | bug | +| [#1219](https://github.com/mudler/vllm.cpp/issues/1219) | `LTX25-KEYFRAME-INTERP` | Pinning BOTH keyframes aborts the render on every recipe whose frame-0 item APPENDS, which is `keyframe_interpolation` -- the one pipeline that exists to interpolate between two pinned ends, whose `docs/USAGE.md` worked example passes `--first-frame open.ppm --last-frame close.ppm` and whose `ltx2-gen --help` says to use the two together. Reachable from the shipped CLI and from `vllm_video_engine_load` / `Generate`. The defect is a derived index that was correct only by APPEND ORDER: the last-frame arm of `src/vllm/multimodal/ltx2_video.cpp` located its own appended tokens at `video.positions[target_tokens * 2]`, and `target_tokens` is the phase's fixed target grid, so that index names the first token PAST THE GRID -- the first appended token in the sequence, whichever item appended it. Row `LTX25-KEYFRAME-INTERP` ([#1096](https://github.com/mudler/vllm.cpp/issues/1096)) put a second appending item in front of it, because `image_conditionings_by_adding_guiding_latent` sends frame 0 to `VideoConditionByKeyframeIndex`, which APPENDS (`keyframe_cond.py:79-82`), rather than to `VideoConditionByLatentIndex`, which replaces (`latent_cond.py:40-41`). With both ends pinned the index named the FIRST frame's keyframe at temporal 0 and the arm's own positional assertion threw `must carry the temporal position of pixel frame frames - 1 (0.333333), but the first appended token starts at 0.000000`. The `ti2vid_two_stage` control -- replace at frame 0, one append -- never reached it and stayed green, which is what says ordering rather than keyframes. The generated-keyframe-slot arm carried the same shape in its growth check. Neither assertion is weakened: fixed in flow by capturing the sequence length at the moment of each append and locating each item's tokens from that, plus a new assertion that the recorded `generated_keyframe_layout.first_token` equals that pre-append count. Gated by a permanent `test_ltx2_video` case pinning both ends on both builders with the bare grid and each single end as controls, captured RED at the pre-fix source with that exact message. Found by the fresh review of #1096 | bug | +| [#1220](https://github.com/mudler/vllm.cpp/issues/1220) | — | The two LTX schedule-anchor cases in `tests/vllm/multimodal/test_ltx2_video.cpp` return the REQUEST step count from their render lambda under a comment saying the count is read back out of the render, so the four `REQUIRE(x.steps == rendered_steps)` lines compare four copies of one request field rather than four renders. WEAKENED, not vacuous, and that is why it is filed rather than repaired in flow: the lambda pins the step count at 3 with a stated reason -- a 2-step schedule is `{1, 0.1, 0}` for every token count, so the 4096 anchor cannot reach the trajectory -- and the assertions below recompute `Ltx2SigmaSchedule` at it, so lowering that literal still reds them by name. What it cannot see is an engine that ignored the request and ran a different number of steps. `Ltx2ConditioningTrace::dit_evaluations` (`include/vllm/multimodal/ltx2_video.h:757-774`) already carries the observation, incremented at one site inside the shared `Evaluate` lambda, and is `steps` per phase on the Euler and ancestral arms against `2 * steps + 1` on res_2s. Not repaired during the review repair of [#1096](https://github.com/mudler/vllm.cpp/issues/1096) because it changes what a LANDED case measures on both pipelines and on the res_2s control, so it owes its own red-first evidence that the derived count reproduces the current literal on every arm rather than a quiet re-derivation of a passing assertion. Listed under `## Owed` in [`ltx25-keyframe-interp.md`](specs/ltx25-keyframe-interp.md) | bug | +| [#1230](https://github.com/mudler/vllm.cpp/issues/1230) | — | Twenty-two LTX-2.5 source citations name two upstream anchors short, and the SAME FILES already carry the corrected form beside the stale one, so a reader who greps either string finds both and has nothing that says which is right. Re-derived at the LTX-2 pin `fd4ded7f` by reading the pinned files: `latent_cond.py:38` is `latent_state = latent_state.clone()` and `:39` is blank, so the two writes every citation means are `:40-41`; `schedulers.py:31` is the return annotation `) -> torch.FloatTensor:`, so the `tokens = math.prod(latent.shape[2:])` read is `:32`. Eleven citations of each stale form, across `src/vllm/multimodal/ltx2_video.cpp`, `src/vllm/model_executor/models/ltx2_pipeline.cpp`, `include/vllm/model_executor/models/ltx2_pipeline.h`, `include/vllm/model_executor/models/ltx2_conditioning.h`, four test files and two specs; the corrected form is already landed in eight more, including `include/vllm/multimodal/ltx2_video.h:625` and `include/vllm/model_executor/models/ltx2_samplers.h:32`. The disagreement is INSIDE single files: `ltx2_video.cpp` cites `schedulers.py:31` at `:3618` and `:32` at `:3017` and `:3600`. **A partial correction is strictly worse than none**, which is why this is a row rather than an in-flow fix and why PR #1209's review-repair commit had its seven anchor corrections REVERTED rather than extended. Those seven left `ltx2_video.cpp` reading both forms a hundred lines apart with nothing recording which to believe -- correctable by a grep while uniform, not correctable while mixed -- and one of them lived in `include/vllm/`, a `USER_USAGE_PREFIXES` path in `scripts/check-doc-checkpoint.py:99`, which is a pure path match with no content analysis, so a comment-only anchor edit in a public header demanded a `docs/USAGE.md` edit the change did not owe. Both consequences are properties of doing it piecemeal and neither arises from one sweep over all 22. Filed by the review-repair flow of `LTX25-KEYFRAME-INTERP` ([#1096](https://github.com/mudler/vllm.cpp/issues/1096)), which owes it under `## Owed` rather than fixing it in flow: the sweep opens five files that row does not otherwise touch and a 22-site mechanical edit owes its own reviewer | bug | | [#1198](https://github.com/mudler/vllm.cpp/issues/1198) | — | Three specs assert `model_loader.cpp` behaviour the loader no longer has, found while verifying line citations for conversion under [#1143](https://github.com/mudler/vllm.cpp/issues/1143): `safetensors-windowed-load.md:63,108` quotes a `shards.clear()` that does not exist (the shard vector is a `shared_ptr` released by the deferred-expert closure, `model_loader.cpp:1636-1643`), `gguf-dflash-draft.md:17-18` calls `LoadDflashDraft` "still typed on `std::vector`" when it takes a `SharedHeadSource`, its `A5`/`B2` rows plan around a GGUF refusal the loader says at `:906-910` is GONE, and `model-factory-registry.md:91` cites `IsDenseArch` which survives only in a comment saying the registry superseded it. Filed rather than repaired because a citation sweep can see the claim is false and not what the true statement is — that is the owning row's judgement. Owned under `## Owed` in [`citation-anchor-freshness.md`](specs/citation-anchor-freshness.md) | bug | | [#1199](https://github.com/mudler/vllm.cpp/issues/1199) | — | `scripts/check-symbol-anchors.py --upstream-root` resolved 354 `vllm/...::Symbol` citations against the parity pin `555967922`: **343 fresh, 0 stale**, which is the measurement that decides the symbol convention over line anchors, since the same pin advance broke every line anchor [#1139](https://github.com/mudler/vllm.cpp/issues/1139) examined. The 11 that did not resolve name a FILE absent at the pin: six `.agents/model-matrix.md` rows (`olmo.py`, `olmo2.py`, `ouro.py`, `persimmon.py`, `plamo2.py`, `fuyu.py` — `registry.py:658` routes `OlmoForCausalLM` to the `transformers` fallback and `registry.py:765` lists `FuyuForCausalLM` as removed at 0.25.0, so our rows claim a mirror source vLLM deleted), two malformed `vllm/tests/kernels/...` paths, `vllm/v1/worker/gpu/worker.py` for `gpu_worker.py`, and one fixture-text false positive. Filed rather than repaired because the model-matrix half is a claim about vLLM, not a path edit. Owned under `## Owed` in [`citation-anchor-freshness.md`](specs/citation-anchor-freshness.md) | bug | | [#1213](https://github.com/mudler/vllm.cpp/issues/1213) | `ENV-LEASE-RUNTIME-STAGING` | `AGENTS.md` stated that a leased worker "has no compiler, no downloader and no Python, so it cannot produce a runtime in place", and `.agents/environment.md` carried the matching clause twice for `dgx:gpu0`. All three negatives are false. `rc describe dgx:gpu0` states that a job runs as root in an Ubuntu 24.04 container carrying `git`, `curl`, `wget`, `ssh`, `gcc`, `g++`, `make`, `cmake`, `ninja`, `pkg-config`, `python3`, `pip` and `venv`, and it instructs the reader to install anything missing; the one limit it names is the absent CUDA toolkit. Two jobs then compiled inside a lease on 2026-08-18: `claude/mudler-ubuntu-box/qwen38-gate` apt-installed `cuda-nvcc-13-0` from the `ubuntu2404/sbsa` lane and built this tree 1791/1791 to `BUILD_RC=0` (`/mnt/nas_share/rc/qwen38-gate/out-main/cfg.log` records `nvcc` 13.0.88 and `CUDA feature cutlass-fp8: ENABLED for [121a]`), and `/mnt/nas_share/rc/mtp_test/build.sh` cloned `github.com/mudler/llama.cpp` from inside a job and left a 97 MB `libggml-cuda.so` on the share. **Why it matters:** "the lease cannot produce a runtime in place" is the stated basis for treating the pinned vLLM oracle as unreachable from a lease, and that oracle is the denominator for every speed-parity number the project owes, so the premise needs re-testing rather than inheriting. This claims nothing about a model run; #1185 owns that and stays open. FIXED IN FLOW: the `AGENTS.md` paragraph and both `.agents/environment.md` clauses now say what the measurement supports, and the four real limits (no preinstalled CUDA toolkit, global installs leak until the pod restarts, CIFS `/workspace` holds no symlink so build in `/tmp` and `cp -rL`, and `-j 4` because unconstrained parallelism OOM-reboots the box) plus the host-versus-container egress distinction ride with the correction. | record | diff --git a/.agents/specs/ltx-2-5.md b/.agents/specs/ltx-2-5.md index e6090c2d3..975a5d9c3 100644 --- a/.agents/specs/ltx-2-5.md +++ b/.agents/specs/ltx-2-5.md @@ -971,13 +971,20 @@ each saying what is absent, what a future row starts from, and what blocks it. negative RoPE shift (`dubit.py:351-353`), which our one ported shift structurally cannot produce because it clamps at zero (`ltx2_conditioning.cpp:596-601`), and on the Dub-It IC-LoRA. -- [#1096](https://github.com/mudler/vllm.cpp/issues/1096) — - `KeyframeInterpolationPipeline` (`keyframe_interpolation.py`). `Ltx2ConditionVideoByKeyframe` - IS served and mutation-proven reachable (`ltx25-token-append.md:270`). Blocked on a - multi-keyframe request surface — the ABI carries two scalar slots - (`include/vllm.h:934-935`) against upstream's repeatable `--image PATH FRAME_IDX STRENGTH` - — and on a per-sigma guided denoiser, ours being fixed per phase - (`ltx2_pipeline.cpp:1069-1070`) and audio-only. +- ~~[#1096](https://github.com/mudler/vllm.cpp/issues/1096) — + `KeyframeInterpolationPipeline` (`keyframe_interpolation.py`).~~ LANDED as row + `LTX25-KEYFRAME-INTERP` ([`ltx25-keyframe-interp.md`](ltx25-keyframe-interp.md)), + and two of the three blockers recorded here were stale by the time it was picked + up. The per-sigma denoiser resolves ONE guider on this pipeline's default path — + `main()` passes plain `MultiModalGuiderParams`, so + `create_multimodal_guider_factory` takes `constant()` and builds a single + `(inf, params)` bin (`guiders.py:312-315`) — and both checkpoints are on the NAS + with #1148 closed at `40a796aa9`. The multi-keyframe surface is real, is NOT + what makes this pipeline different, and is now + [#1187](https://github.com/mudler/vllm.cpp/issues/1187). What WAS different, and + is named in none of the above, is the conditioning BUILDER: `:211` and `:260` + call `image_conditionings_by_adding_guiding_latent` (`helpers.py:343-367`), so + frame 0 is a keyframe that APPENDS rather than a latent that REPLACES. - [#1097](https://github.com/mudler/vllm.cpp/issues/1097) — `ltx2-gen` silently discards a second `--lora`, and `kKnownLoadExtras`' own comment still says "nine of these ten" over a twelve-entry array. Product code, so filed rather than fixed in this records change. diff --git a/.agents/specs/ltx25-keyframe-interp.md b/.agents/specs/ltx25-keyframe-interp.md new file mode 100644 index 000000000..ae38273dd --- /dev/null +++ b/.agents/specs/ltx25-keyframe-interp.md @@ -0,0 +1,685 @@ +# LTX25-KEYFRAME-INTERP — the interpolation pipeline, and the conditioning branch it deletes + +Row `LTX25-KEYFRAME-INTERP`. Issue +[#1096](https://github.com/mudler/vllm.cpp/issues/1096). Campaign +[`ltx-2-5.md`](ltx-2-5.md), under roadmap row `ROAD-V1-LTX25`. + +Upstream pin: Lightricks/LTX-2 `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca`, +verified with `git rev-parse HEAD` in `/home/mudler/_git/LTX-2` on 2026-08-18. + +Base: `8daf58e77`, pinned when the branch was created. + +## Now + +`ACTIVE` -> `DONE` with this change. `KeyframeInterpolationPipeline` +(`keyframe_interpolation.py:55`) becomes `pipeline_kind = "keyframe_interpolation"` +on the four generations the recipe table keys. + +## Scope + +**In.** One recipe row, `KeyframeInterpolationRecipe`, and its four +`(kind, version)` keys. One new recipe field, +`Ltx2PipelineRecipe::image_conditioning`, because this pipeline is the first arm +that uses upstream's OTHER image-conditioning builder and this engine can only +express the one. The docs and the CLI help that list the kinds. + +**Out.** The three arms #1150 leaves divergent on the schedule anchor — this row +sets its own phase and touches nothing else. Per-phase adapter STRENGTH (#1144). +An N-image request surface with per-image `frame_idx` — see `## Owed`. The +real-weights render, which needs a GPU lease — see `## What is NOT verified`. + +## What was genuinely missing, against what #1096 recorded + +`.agents/issue-index.md:321` is the audit this row inherited. It named two +blockers and a missing artifact. **Two of the three are stale, one is real but +is not what makes this pipeline different, and the difference that DOES make it +different is not in the audit at all.** Each is re-derived at the pin below, +because "the audit said so" is a search result and not a finding. + +### (a) "no multi-keyframe request surface" — TRUE, and not the blocker + +Still true as stated: the ABI carries two scalar slots, `first_frame` and +`last_frame` (`include/vllm.h:947-948`), against upstream's repeatable +`--image PATH FRAME_IDX STRENGTH [CRF]` (`utils/args.py:805-817`). The indices +are still fixed — `latent_idx=0` and `frame_idx=frames-1`. + +But two pinned keyframes at the two ENDS of the clip is what +"interpolate between keyframes" means at its default configuration, and this row +serves that. An arbitrary interior `frame_idx` is a request-surface row of its +own, owed below with an issue. **A pipeline that cannot be asked for a third +keyframe is narrower than upstream; a pipeline that maps its first keyframe onto +the wrong conditioning item is WRONG**, and only the second stops the row. + +### (b) "a per-sigma guided denoiser" — STALE on this pipeline's default path + +The audit is right that `FactoryGuidedDenoiser` resolves guiders per step from +sigma (`utils/denoisers.py:332-343`). It does not follow that this pipeline ever +uses more than one, and at the pin it does not: + +- `main()` builds plain `MultiModalGuiderParams` for both streams + (`keyframe_interpolation.py:325-340`), never a `MultiModalGuiderFactory`. +- `create_multimodal_guider_factory` therefore takes its last line, + `MultiModalGuiderFactory.constant(params, ...)` (`guiders.py:360`). +- `constant` builds `_params_by_sigma = ((inf, params),)` (`guiders.py:312-315`) + — ONE bin, so `build_from_sigma` returns the same guider at every sigma. +- `FactoryGuidedDenoiser.__call__` then delegates to `_guided_denoise` + (`denoisers.py:345-358`), which is what `Ltx2GuidedDenoise` ports. + +So on the configuration this row ships, `FactoryGuidedDenoiser` IS +`GuidedDenoiser`. `ltx25-ti2vid-recipe.md` recorded the same reduction for +`ti2vid_two_stages.py`, which selects the identical parser. The seam landed at +`daeff67f2` (#1092/#1102) and `kGuided` reaches it. + +The sigma-BINNED arm is real and is reachable only by a caller who constructs a +`MultiModalGuiderFactory.from_dict` and passes it as `video_guider_params` — no +CLI flag builds one. It is owed below, under the issue that owns it for every +arm rather than for this one. + +### (c) "neither file is on the NAS" — STALE + +Both are, and byte-verified: the dev transformer at 42,018,190,584 B and the +distilled adapter at 8,899,889,568 B, under +`/mnt/nas_share/checkpoints/ltx-2.5/lightricks-ltx-2.5/`. `PlanDit`'s refusal of +a pure-BF16 DiT (#1148) closed at `40a796aa9`. What is owed is the RUN. + +### (d) THE DIFFERENCE THE AUDIT DOES NOT NAME: the conditioning builder + +`ti2vid_two_stages.py:231` and `:276` call `combined_image_conditionings`; this +pipeline's `:211` and `:260` call `image_conditionings_by_adding_guiding_latent`. +(Both files carry a `:211`, and this line named the wrong one until the review +repair: `ti2vid_two_stages.py:211` is blank.) They are +different functions and the difference is one branch: + +| | `combined_image_conditionings` (`helpers.py:272-308`) | `image_conditionings_by_adding_guiding_latent` (`helpers.py:343-367`) | +|---|---|---| +| `frame_idx == 0` | `VideoConditionByLatentIndex` (`:295-300`) | `VideoConditionByKeyframeIndex` | +| any other `frame_idx` | `VideoConditionByKeyframeIndex` (`:301-305`) | `VideoConditionByKeyframeIndex` | +| what frame 0 does to the state | REPLACES latent frame 0's clean tokens; the token count NEVER changes (`latent_cond.py:40-41`) | APPENDS a latent frame of tokens at the end (`keyframe_cond.py:79-82`) | + +The second function has no branch at all: it is one loop and one item type. That +is the whole content of the diff between the two files' conditioning, and it is +this pipeline's name — the first image is a KEYFRAME to interpolate FROM, not a +frame to overwrite. + +**This engine hard-codes the branch.** `ltx2_video.cpp`'s `wants_first_frame` arm +calls `Ltx2ConditionVideoByLatentIndex` unconditionally and its comment cites +`helpers.py:295-300` by line. Both conditioning primitives are ported and +gated — `Ltx2ConditionVideoByKeyframe` is what the `wants_last_frame` arm below +it already drives — so what is missing is the SELECTION, not a capability. + +**And it is invisible to every gate this tree owns.** A `keyframe_interpolation` +render built on the replace arm returns a clip of the right size, the right frame +count and the right sample rate, with the supplied image pinned into it. It is +conditioned; it is conditioned as a different pipeline. The only observable is +the sequence LENGTH the DiT ran over, which is `Ltx2ConditioningTrace::video_tokens`, +and no pixel comparison and no shape check reads it. + +## Port map, each line read at the pin + +Anchors are +`packages/ltx-pipelines/src/ltx_pipelines/keyframe_interpolation.py` unless +another file is named. + +### Stage 1 — `stage_1` (`:100-110`, called at `:231-252`) + +| Field | Value | Upstream | +|---|---|---| +| `spatial_downscale` | 2 | `:203-209`, `width // 2` / `height // 2` | +| `sigmas` | empty (derived) | `:199-200`, `self._scheduler.execute(steps=num_inference_steps)` | +| `schedule_tokens` | `kSchedulerDefault` | the same call passes NO latent; `schedulers.py:32` reads that as `default_number_of_tokens` = `MAX_SHIFT_ANCHOR` = 4096 (`:11`, `:29`) | +| `noise_scale` | 1.0 | `:244-247` sets none; `ModalitySpec.noise_scale` defaults 1.0 (`utils/types.py:110`) | +| `video_guidance` | `params.video_guider` | `:222-225`, from `MultiModalGuiderParams` the CLI fills (`:325-332`) | +| `audio_guidance` | `params.audio_guider` | `:226-229`, filled from six `--audio-*` / `--v2a-guidance-scale` flags (`:333-340`) | +| `denoiser` | `kGuided` | `:232`, `FactoryGuidedDenoiser` — see (b) above for why that is `Ltx2GuidedDenoise` here | +| `allow_guidance_override` | true | `:301` selects `default_2_stage_arg_parser`, which carries the six guider flags (`utils/args.py:947-1006`) | +| `loras` | `kNoAdapters` | `:104`, `loras=tuple(loras)`, against `:111`'s `stage_2_loras` | +| `stepper` | `kEuler` | derived, below | + +### Stage 2 — `stage_2` (`:112-122`, called at `:271-290`) + +| Field | Value | Upstream | +|---|---|---| +| `sigmas` | `Stage2DistilledSigmas()` | `:166`, `stage_2_sigmas: torch.Tensor = STAGE_2_DISTILLED_SIGMAS` — a DEFAULT ARGUMENT, so frozen | +| `use_official_sigma_schedule` | false | the schedule is explicit | +| `noise_scale` | `Stage2DistilledSigmas().front()` | `:282` and `:287`, `stage_2_sigmas[0].item()` on BOTH modality specs | +| `input_transform` | `kSpatialUpsample` | `:255`, `self.upsampler(video_state.latent[:1])` | +| `denoiser` | `kSimple` | `:272`, `SimpleDenoiser(v_context_p, a_context_p)` — takes no params | +| `allow_guidance_override` | true | the flags are legal on this parser and reach stage 1's guider alone; `kSimple` is what makes them inert here | +| `loras` | `kAllAdapters` (default) | `:111`, `(*tuple(loras), *tuple(distilled_lora))`, passed at `:116` | +| `stepper` | `kEuler` | derived, below | + +### Recipe (`:147-168`, `:297-358`) + +| Field | Value | Upstream | +|---|---|---| +| `height` / `width` | `params.stage_2_*()` | `:301` sets the request geometry to the FINAL output (`utils/args.py:1128`) | +| `negative_prompt` | per version | `:178-186` encodes `[prompt, negative_prompt]` and reads `ctx_n` into both guider factories (`:224`, `:228`) | +| `video_output_phase` | 1 | `:292` decodes the name `:271` rebound | +| `audio_output_phase` | **1** | `:271` is `video_state, audio_state = self.stage_2(...)` — NOT a discard — and `:293` decodes that name | +| `allow_request_sigmas` | true | `:165` `stage_1_sigmas` is a real parameter and `:200` honours it | +| `allow_request_latents` | false | no `__call__` parameter carries one (`:147-168`) | +| `allow_negative_prompt` | true | `:150` | +| `requires_distilled_lora` | **true** | `distilled_lora` is POSITIONAL and non-defaulted (`:68`), and `--distilled-lora` is `required=True` (`utils/args.py:1140-1155`) on the parser `:301` selects | +| `requires_audio_input` | false (default) | there is no `--audio-path`; the soundtrack is generated | +| `audio_only` | false (default) | `:292-293` decodes both | +| `image_conditioning` | **`kAddGuidingLatent`** | `:211` and `:260` — the new field, section (d) above | + +### `audio_output_phase = 1`, and it is the opposite of `ti2vid_two_stage` + +These two pipelines share a parser, a stage layout, a stepper and a sigma set, +and they disagree here. `ti2vid_two_stages.py:287-289` carries upstream's own +comment — "Stage 2 refines video only; discard its audio" — and binds +`video_state, _`. `keyframe_interpolation.py:271` binds `video_state, +audio_state` and `:293` decodes it. There is no comment either way in this file; +the binding IS the statement. + +**So the field most likely to be copied from `ti2vid_two_stage` is the one that +must not be.** Writing 0 here decodes stage 1's soundtrack, which is finite, the +right length, at the right sample rate, and the wrong take. The recipe case +asserts both polarities side by side for that reason. + +### `stepper = kEuler` is derived, not assumed + +Neither `self.stage_1(...)` (`:231-252`) nor `self.stage_2(...)` (`:271-290`) +passes `stepper` or `loop`. `DiffusionStage.__call__` declares both as `None` +defaults (`utils/blocks.py:512-513`) and fills them at `:524-527` with +`euler_denoising_loop` and `EulerDiffusionStep()`. `distilled.py:76-84` selects +the ANCESTRAL stepper on generation 2.5 and reaches this pipeline through +nothing. + +### Four version keys + +`2`, `2.3`, `2.4`, `2.5`, mirroring `ti2vid_two_stage` and `a2vid_two_stage`. +`main()` calls `resolve_cli_params()` (`:300`) and hands the result to +`default_2_stage_arg_parser(params=params)` (`:301`), so the generation comes off +the CHECKPOINT and there is no "which generations support this pipeline" +question upstream. The 2 and 2.3 rows carry `kOmniNegativePrompt`; 2.4 and 2.5 +carry `LightricksNegativePrompt()`. + +### `num_frames` is REQUIRED here and optional on `ti2vid_two_stage` + +`:154` is a positional, non-defaulted `num_frames: int`, and this pipeline builds +no `DurationPredictor` — `ti2vid_two_stages.py:174` takes +`int | AutoDuration = DEFAULT_AUTO_DURATION` and resolves it from the caption. +Recorded rather than expressed: this engine has never auto-predicted a duration, +so every recipe already behaves as this pipeline does, and a field would have one +value on every row. + +## The new field, and why it is on the RECIPE and not the phase + +`Ltx2PipelineRecipe` gains +`Ltx2ImageConditioningBuilder image_conditioning = kCombined`: + +- `kCombined` — `combined_image_conditionings` (`helpers.py:272-308`). Frame 0 + REPLACES. **The DEFAULT, and it is today's behaviour, so no landed arm moves.** +- `kAddGuidingLatent` — `image_conditionings_by_adding_guiding_latent` + (`helpers.py:343-367`). Frame 0 APPENDS, like every other frame. + +**Per recipe, because upstream calls the same builder for both of this +pipeline's stages** (`:211` for stage 1 and `:260` for stage 2 — the only +difference between the two calls is the height and width). A per-phase field +would offer a combination upstream has no site for, and every phase field this +engine carries exists because some upstream pipeline sets it per stage. + +**A preserving default cannot fail silently.** With `kCombined` default, an arm +moves only where a line says so. The upstream-faithful-everywhere alternative +does not arise: `combined_image_conditionings` is what the other ten pipelines +call, so `kCombined` IS the majority as well as the incumbent. + +The engine reads it in one place, the `wants_first_frame` arm of the phase loop +in `ltx2_video.cpp`, which becomes a two-way branch on the field. The +`wants_last_frame` arm below it is unchanged: `frame_idx != 0` takes the keyframe +item under BOTH builders. + +`causal_fix` stays `true` at the call. `VideoConditionByKeyframeIndex` gates it +itself — `latent_tools.causal_fix if self.frame_idx == 0 else False` +(`keyframe_cond.py:49`) — and `Ltx2ConditionVideoByKeyframe` already mirrors that +at `ltx2_conditioning.cpp:548`. So passing `true` with `frame_idx = 0` is the +one combination in which the fix is APPLIED, which is upstream's, and passing +`false` would silently drop it. + +### The trace has to follow the tokens + +`im.trace.image_tokens` / `image_digest` / `image_absmax` are digested from the +tokens the item WROTE, not from the encoder's output, and the replace arm reads +them off the FRONT of `video.clean`. Under `kAddGuidingLatent` the written tokens +are at the TAIL. Reading the front there would digest unconditioned tokens, and +the instrument would report a healthy conditioning for a build that placed +nothing — the exact failure the field's own comment in `ltx2_video.h` says it +exists to prevent. The digest slice is therefore taken from the range the append +actually grew. + +## Tests + +Upstream ships no unit test over a pipeline's recipe — the recipe IS the +constructor — so these are ported in the sense that every assertion cites the +line it mirrors, and the harness is ours. + +### 1. The recipe, field by field (`test_ltx2_pipeline.cpp`) + +Mirrors `"ltx2 ti2vid: the recipe is the PLAIN two-stage pipeline, not the HQ +one"` (`:3521`). Every field in the Port map, each against a CONTROL drawn from +the recipe it would otherwise be confused with — `ti2vid_two_stage` above all, +which it is TWO fields from: + +- `audio_output_phase == 1`, control `ti2vid_two_stage` at 0 and + `res2s_two_stage` at 0. The two-against-two split is what stops this passing + because every two-stage recipe happens to agree; +- `image_conditioning == kAddGuidingLatent`, control `ti2vid_two_stage`, + `a2vid_two_stage`, `res2s_two_stage`, `distilled_two_stage` and `one_stage` all + at `kCombined`; +- stage 1 `loras == kNoAdapters`, control `res2s_two_stage` stage 1 at + `kAllAdapters`; +- stage 1 `schedule_tokens == kSchedulerDefault`, control `res2s_two_stage` at + `kTargetLatent`; +- stage 1 `audio_guidance.cfg_scale` == the params table's 7.0, control + `a2vid_two_stage` at the positive-only default; +- `stepper == kEuler` on both phases, controls `distilled_two_stage` at 2.5 on + `kEulerAncestral` and `res2s_two_stage` on `kRes2s`; +- `requires_distilled_lora` true, `requires_audio_input` FALSE, control + `a2vid_two_stage` true/true. + +Plus a version-key case: all four resolve, and `keyframe_interpolation_two_stage`, +`keyframe`, `keyframe_interp` and version `2.6` refuse by name. + +### 2. The conditioning builder REACHES the state (`test_ltx2_video.cpp`) + +The row's identity, gated through `LoadVideoEngine` + `Generate` rather than on +the recipe struct: case 1 proves the field is set, this one proves it is +CONSUMED (#1013). + +The SAME image, the SAME geometry and the SAME seed on two loads that differ +only in `pipeline_kind`. `Ltx2ConditioningTrace::video_tokens` is the length the +DiT ran over on the last phase, recorded before the trim: + +- `keyframe_interpolation` reports `video_tokens == ti.video_tokens + + kf.image_tokens` — one latent frame of tokens MORE than the plain arm, which + is what an append costs; +- `ti2vid_two_stage` reports `video_tokens` equal to the bare target grid, taken + from a third render of the same kind with NO image at all. Without that third + render the equality above would pass on a tree where both arms append. +- `image_tokens > 0` on both, so neither passes by placing nothing. + +The rendered pixels are compared too, and must differ from an unconditioned +render of the same kind — the arms place the same content, so a pixel comparison +between THEM is not the discriminator, and saying which comparison carries which +claim is the point. + +### 3. The per-arm x0 invariant on stage 1 (`test_ltx2_video.cpp`) + +The correctness trap this arm inherits: guidance combines **x0**, not velocity. +Every linear term is invariant under `x0 = latent - sigma*v`, so cfg, stg and +modality cannot see the difference; the RESCALE branch is not invariant and +`rescale_scale` defaults to 0.7, which is the default path (#1039, #1092). + +A magnitude assertion cannot gate it — on a reduced fixture `std(cond)/std(pred)` +is 1.0 to 1e-5 in BOTH spaces. The gate is the per-arm equation +`x0 == latent - sigma*velocity` over three recorded tensors, exact in x0 space +and off by the whole sample in velocity space, where it degenerates to +`|x0 - velocity| = 0`. Both residuals are printed on every arm so a RED says +which space it landed in. Over **all four** passes — cond, uncond, perturbed, +modality — each with its own non-vacuity guard, because a zeroed velocity +collapses the equation to `x0 == latent` for that arm alone. + +### 4. The schedule anchor REACHES the sigmas (`test_ltx2_video.cpp`) + +`trace.schedule_tokens == 4096` on a `keyframe_interpolation` render at two +geometries, and on the same fixture a `res2s_two_stage` render reports a +DIFFERENT, non-4096 count at each. Both halves are load-bearing: the equalities +alone pass on a build that hard-codes 4096, the inequalities alone on today's +tree. + +**The trajectory half must render at three steps or more and must recompute at +the RENDER's step count.** At two steps the schedule is `{1, 0.1, 0}` for every +token count — `stretch` pins sigma[0] at 1.0 and renormalises the last non-zero +sigma to exactly `terminal` = 0.1 (`schedulers.py:48-55`), and a 2-step schedule +has only those two non-zero entries. Three steps is the shortest schedule with an +interior sigma. The step count comes back OUT of the render lambda, and +`rendered_steps > 2` is asserted by name, so lowering it is refused rather than +silently turning the comparison into a value against itself. `ltx25-ti2vid-recipe.md` +shipped this case with a literal `3` and a fresh review found it vacuous; the +mutation table below re-runs that exact mutation here. + +### 5. Reachability (`test_ltx2_video.cpp`) + +`LoadVideoEngine` with `pipeline_kind = keyframe_interpolation` -> `Generate` -> +pixels, which is `include/vllm.h` plus the documented load extras and is what +`ltx2-gen --pipeline-kind keyframe_interpolation --lora-path ... --upsampler-path +...` does through the ABI. Deleting the dispatch row in +`ResolveLtx2PipelineRecipe` REDs it at the LOAD. + +`dit_forwards`, not an evaluation count, distinguishes the guided stage: a +denoiser call is ONE evaluation whether or not guidance ran, and only +`Ltx2ConditioningTrace::dit_forwards` counts actual `Ltx2DitForward` calls. + +All three knobs this recipe needs are LOAD extras, which a server supplies +through `--video-extra KEY=VALUE`, and `requires_audio_input` is false — so #928 +does not exclude the HTTP route, as it does on `a2vid_two_stage`. That is a claim +about the request SURFACE. Nothing here drives the HTTP route end to end, so it +is not measured and the reach claim rests on the ABI path alone. + +### 6. The `requires_distilled_lora` refusal + +A `keyframe_interpolation` load with no `lora_path` refuses BY WHAT IS MISSING, +and the message names the pipeline. Mirrors the positional `distilled_lora` +(`:68`) and `--distilled-lora required=True` (`utils/args.py:1140-1155`). Two +controls: the same load WITH `lora_path` renders, and the DEFAULT kind still +loads without one — otherwise the case passes on any load failure or on a new +global requirement. + +### Mutations required to pass + +1. `recipe.image_conditioning = kAddGuidingLatent` deleted. Test 2 RED. +2. The reader branch in `ltx2_video.cpp` forced to the replace arm. Test 2 RED. +3. `recipe.audio_output_phase = 1` -> 0. The recipe case RED. +4. `stage1.schedule_tokens = kSchedulerDefault` deleted. Test 4 RED. +5. `stage1.loras = kNoAdapters` deleted. The recipe case RED. +6. `recipe.requires_distilled_lora = true` deleted. Test 6 RED. +7. The `keyframe_interpolation` branch deleted from `ResolveLtx2PipelineRecipe` — + the standing reachability mutation. Tests 2, 3, 4, 5, 6 RED. +8. `Ltx2GuidedDenoise` left in velocity space on one arm. Test 3 RED. +9. Test 4's `gen.steps` 3 -> 2 — the mutation the ti2vid row's FIRST head passed. + Test 4 RED on both `at_anchor != at_target` and `rendered_steps > 2`. + +Every mutation prints FOUR facts: `git diff --stat`, whether it BUILT, the +compile-error count, and the exit code captured directly. The expected OLD line +content is asserted before each edit, because a mutation that landed on a comment +line printed a clean one-line diff, BUILT, 0 errors, exit 0, and read as a pass. + +**`stg_blocks = []` is legal.** Upstream defaults it to `[]` and validates it +nowhere. No refusal is added for it; one had to be removed already. + +## Gates + +```sh +cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF +cmake --build build -j6 && ctest --test-dir build -j4 --output-on-failure +``` + +Whole binaries, never a `--test-case` filter: a filter matching zero cases prints +`SUCCESS!` at exit 0, and LTX case names contain commas, which doctest `-tc` +splits on. Assert a non-zero case AND assertion count. A thrown case prints +`0 failed` beside `Status: FAILURE!`, so the exit code is the authority. + +Report `CONFIGURE_EXIT`, `BUILD_EXIT`, the `: error:` count, `ctest -N`, +`CTEST_EXIT`, the pass/fail line, and `No space left` / `BFD` greps WITH live +positive controls, plus load and free disk. 511 tests registered at this head. + +`READER ANCHORS` (`ltx2_video.cpp`) is gated by `test_ltx2_video` and shifts +whenever the readers above it move. This row edits the conditioning block, which +is BELOW the last anchor, so the list is not expected to move — re-derive with +the test's own walk anyway, and arm the instrument first by inserting a line and +confirming MISMATCH. + +**No GPU.** Another agent holds `dgx:gpu0` for a render, and a recipe row is +correctly gated by the CPU goldens. + +`ltx2-gen` gains `--last-frame` in this change ([#1191](https://github.com/mudler/vllm.cpp/issues/1191)), +fixed in flow. The ABI field and the engine's reader both predate it; only the +CLI was missing, and this is the first row a missing CLOSING keyframe narrows. + +## What is NOT verified + +**No real-weights render, and the reason is contention, not a missing artifact.** + +Upstream marks this arm `2 | Full + distilled LoRA | Euler | Keyframe +interpolation` (`packages/ltx-pipelines/CLAUDE.md:24`). Stage 1's identity is CFG +on the UNADAPTED model, so the checkpoint it needs is +`ltx-2.5-22b-dev-transformer-bf16.safetensors`. That file is on the NAS and +byte-verified, the distilled adapter is beside it, and #1148 made the pure-BF16 +DiT loadable at `40a796aa9`. What remains owed is the RUN: a GPU lease, a build +on that host, upstream's render and ours on the same checkpoint, prompt and seed, +and the comparison. + +**Running this arm against a DISTILLED checkpoint instead would be worse than not +running it.** The distilled scales are trained INTO those weights, so a +CFG-guided stage 1 on top samples a trajectory they were never trained for, and +it renders — right size, right frame count, right sample rate, plausible picture, +no diagnostic (#1137). + +So the claim this row makes is: gated on CPU goldens, correct against the +upstream SOURCE line by line, and NOT measured against upstream's own render. + +## Outcome + +Row `DONE`. Recorded here because neither the code nor the Git history carries +it: what the gate MEASURED, and the two things this row found that its scoping +pass did not know. + +### The audit was wrong about WHICH blocker, in both directions + +#1096 named a per-sigma denoiser and two absent checkpoints, both stale, and a +multi-keyframe surface, which is true and is not what stops the pipeline. It did +not name the conditioning builder, which is the one difference that makes +`KeyframeInterpolationPipeline` a different pipeline rather than a second +`ti2vid_two_stage`. + +Both errors have the same shape and it is worth naming: the audit read the +pipeline's IMPORTS and the machinery they reach, and inferred what would be hard. +`FactoryGuidedDenoiser` resolves guiders per sigma, so a per-sigma guider looked +required — and the same file's `main()` passes plain params, so the factory has +one bin. `image_conditionings_by_adding_guiding_latent` and +`combined_image_conditionings` are both one-line imports of a helper, so neither +looked like anything. **The import list says which machinery is reachable; only +the call site says which behaviour runs.** + +### What the gate measured + +Focused, at the head this row pushes: + +| Binary | cases | assertions | exit | +|---|---|---|---| +| `test_ltx2_pipeline` | 56 | 3316 | 0 | +| `test_ltx2_video` | 87 | 2713 | 0 | + +RED before the recipe landed, captured on the same binaries: `test_ltx2_pipeline` +56 cases / 54 passed / 3183 assertions, `Status: FAILURE!`, exit 1, both new +cases throwing `Unsupported LTX pipeline kind/version: +'keyframe_interpolation'/'2.5'`; `test_ltx2_video` 87 / 83 passed / 2593, exit 1. + +Full gate at the same head: `CONFIGURE_EXIT=0`, `BUILD_EXIT=0`, `: error:` count +0, `ctest -N` **511**, `CTEST_EXIT=0`, `100% tests passed, 0 tests failed out of +511`, `No space left` and `BFD` each 0 against injected controls that returned 1. +Load average 14 to 36 across the run and 13 GiB free at its start, and none of +the four load-dependent gates (#618, #294, #1052, #428) went red. + +The conditioning case's own `MESSAGE` line is the row in one measurement: + +```text +video_tokens keyframe 12 ti2vid 8 bare 8 +``` + +`image_tokens` is 4 on both arms — the same image, the same encode, 8 + 4 on one +and 8 on the other. And the anchor case reports `keyframe: 4096 / 4096 res2s: +2 / 8`, the same split `ltx25-ti2vid-recipe.md` measured. + +### The reachability mutation FAILED TO BUILD on its first shape, and that is the finding + +Deleting the whole `keyframe_interpolation` arm from `ResolveLtx2PipelineRecipe` +left `KeyframeInterpolationRecipe` unreferenced in an anonymous namespace, and +`-Werror` killed the build: `BUILT=NO`, 1 error, **no test result at all**. A +harness that printed only a diff stat and an exit code would have shown a clean +26-line deletion and nothing else, which is the third shape of green-that-proves- +nothing this campaign has paid for. + +The mutation that measures the thing is a rename of the dispatch KEY — +`"keyframe_interpolation"` to `"keyframe_interpolation__UNREACHABLE"`. Every line +stays compiled and referenced; the recipe simply stops being selectable through +the request surface, which is what "delete the production call site" is asking. +Under it all six new cases go RED **by name**, four in `test_ltx2_video` and two +in `test_ltx2_pipeline`. + +Both runs are in the pull request body. The first is not a failed attempt to be +tidied away: it is the reason the harness prints four facts. + +### The trace digest had to follow the tokens, and nothing would have said so + +`im.trace.image_tokens` / `image_digest` / `image_absmax` are digested from the +tokens the conditioning item WROTE, and the replace arm reads them off the FRONT +of `video.clean`. Under the append arm the written tokens are at the TAIL. + +Left as it was, the digest would have described unconditioned tokens on this +recipe and reported a healthy conditioning for a state the keyframe never +reached — which is precisely what the field's own comment in `ltx2_video.h` says +it exists to prevent, one arm down from where it was written. No test in this row +compares the digest's VALUE, so nothing here would have caught it; it is correct +because the range was derived, not because a gate held it. Recorded so the next +reader knows which half is measured. + +### Two things a later reader should not re-derive + +- **`causal_fix` is passed `true` at the frame-0 keyframe call and that is not a + copy of the last-frame arm.** `VideoConditionByKeyframeIndex` gates it itself + — `latent_tools.causal_fix if self.frame_idx == 0 else False` + (`keyframe_cond.py:49`) — and `Ltx2ConditionVideoByKeyframe` mirrors that gate + at `ltx2_conditioning.cpp:548`. So `frame_idx = 0` is the ONE combination in + which the fix applies, and passing `false` would silently drop it rather than + being refused. +- **`ltx2-gen` gained `--last-frame` (#1191) and the ABI did not change.** + `vllm_video_params::last_frame` has existed since #930 and the engine has + served it since; the CLI had simply never read the field. Nothing about this + row required an ABI change, and a reader who sees the flag appear in the same + commit should not infer one. + +### A number this row moved that no checker owns + +`docs/USAGE.md` says how many `(kind, version)` pairs resolve. It said +**twenty-four** and now says **twenty-eight**. It is derived by hand from +`ResolveLtx2PipelineRecipe` and nothing recomputes it, so the next four-key kind +has to move it again. + +### The review repair, and the one defect it found + +The fresh review returned eight findings. Six are repaired on this branch, one is +inherited and tree-wide, and one is filed above under `## Owed`. + +**The blocking one is [#1219](https://github.com/mudler/vllm.cpp/issues/1219), +and it is worth stating as a shape rather than as a line number.** This row's own +`## Tests` §2 gated that frame 0 APPENDS on this builder, and every case it +describes pins ONE end. Pinning both — which is the pipeline's name, the +`docs/USAGE.md` worked example and the `ltx2-gen --help` instruction — aborted the +render, because the LANDED last-frame arm located its own appended tokens at +`positions[target_tokens * 2]`. That index is the first token past the target +grid: correct while that arm owned the first append, and false the moment this row +put a second appending item in front of it. The generated-keyframe-slot arm +carried the same derivation. + +**Nothing in the diff touched either arm.** The row added an append two hundred +lines above them and changed what "the first appended token" meant. A derived +index that is right by ORDER survives every review of the code that contains it, +because the code that contains it is not the code that breaks it. What the case +this repair adds gates is the ORDER, on both builders, with `ti2vid_two_stage` — +replace plus one append — as the control that never reached the defect. + +**A second measurement the repair owes the next reader**: `causal_fix = true` at +the first-frame call site is INERT, not merely ungated. The reason and the probe +numbers are under `## Owed`, and the gate that could exist lives on the seam in +`test_ltx2_vae` rather than at the call site. + +## Dependencies + +- #1118 (`LTX25-PHASE-LORA`), landed at `4ae0f54ab`. Supplies + `Ltx2PhaseRecipe::loras` and `Ltx2RebindDitLoras`, without which stage 1 could + not run unadapted. +- #1092 / #1102 (`LTX25-GUIDED-VIDEO`), landed at `daeff67f2`. Supplies + `Ltx2GuidedDenoise`, without which stage 1's CFG has no seam. +- #1093 (`LTX25-TI2VID-RECIPE`), landed at `affc2a7fd`. Supplies + `Ltx2PhaseRecipe::schedule_tokens` and `Ltx2PhaseScheduleTokens`, and is the + reviewed template this row follows. +- #1117 (`LTX25-A2VID-RECIPE`), landed at `d1e5e9bc0`. Supplies + `requires_distilled_lora` and its refusal, keyed on the flag rather than on the + kind string. +- #1148 (`LTX25-BF16-DIT`), landed at `40a796aa9`. Makes the dev transformer + loadable, which is what the owed run needs. +- #930 (`LTX25-TOKEN-APPEND`), landed at `c7cb59fbb`. Supplies the append seam — + `Ltx2ExtendKeyframesMask` and `Ltx2ClearConditioning` — which the frame-0 + keyframe now drives at a second call site. + +## Owed + +- **An N-image request surface with per-image `frame_idx`, `strength` and + `crf`**, [#1187](https://github.com/mudler/vllm.cpp/issues/1187). Upstream's + `--image PATH FRAME_IDX STRENGTH [CRF]` is repeatable (`utils/args.py:805-817`) + and expands per keyframe (`helpers.py:343-367`); this ABI carries two scalar + slots at fixed indices 0 and `frames - 1`. Every pipeline that takes `images` + is narrowed by it, which is why it is filed against the request surface rather + than against this row. This row serves the two-endpoint case, which is + keyframe interpolation's default configuration. +- **The sigma-BINNED guider factory**, `MultiModalGuiderFactory.from_dict` + (`guiders.py:317-330`) resolved per step by `FactoryGuidedDenoiser` + (`denoisers.py:332-343`). No CLI flag builds one, so no upstream default path + reaches it — but a caller who passes a factory as `video_guider_params` does. + Owed under [#1187](https://github.com/mudler/vllm.cpp/issues/1187) as the + second half of the same request-surface gap: this engine's guider is one struct + per phase and there is no sigma-keyed spelling to fill. +- **The real-weights comparison against upstream's own render** on the dev + transformer, same checkpoint, prompt and seed. #644 owns the standing "close + every refused arm" sweep and every LTX arm shipped to date carries the same + debt. +- **Per-phase adapter STRENGTH**, [#1144](https://github.com/mudler/vllm.cpp/issues/1144). + Not needed here: upstream gives this pipeline `loras=tuple(loras)` against + `(*tuple(loras), *tuple(distilled_lora))`, i.e. ABSENT vs PRESENT, which + `Ltx2PhaseLoraScope` expresses exactly. +- **The other three divergent arms on the schedule anchor**, owned by + [#1150](https://github.com/mudler/vllm.cpp/issues/1150). This row sets its own + phase and touches none of them. +- **A trace-derived step count in the two schedule-anchor cases**, + [#1220](https://github.com/mudler/vllm.cpp/issues/1220). Both cases — the + `ti2vid` one and the copy this row made of it — return `gen.steps` from their + render lambda under a comment that says the count is read back out of the + render, so the four `REQUIRE(x.steps == rendered_steps)` lines compare four + copies of one request field. The guard is weakened rather than vacuous: it + still reds when the pinned `gen.steps = 3` is lowered, which is the mutation it + was built for. `Ltx2ConditioningTrace::dit_evaluations` already carries the + observation. Not repaired in this row's review-repair flow because it changes + what a LANDED case measures on both pipelines and the res_2s control, so it + owes its own red-first evidence rather than a quiet re-derivation of a passing + assertion. +- **The `causal_fix` argument at the first-frame call site is INERT**, and the + gate for it lives on the seam rather than at the call site. MEASURED on a probe + of `Ltx2ConditionVideoByKeyframe`: at `num_pixel_frames = 1`, which both + production arms pass, flipping `causal_fix` moves 0 of 48 position values at + `frame_idx` 0 and at `frame_idx` 8 alike, because `get_pixel_coords` leaves a + one-latent-frame keyframe's temporal START at 0 either way + (`max(0 + 1 - time, 0)`, patchifiers.py:166-169) and the `num_pixel_frames == 1` + narrow at `keyframe_cond.py:56-57` then overwrites the END the fix had moved. + So no call-site check can detect a flip, and one that appeared to would be a + false gate. The `frame_idx == 0` gate the argument passes through + (`keyframe_cond.py:49`) is what carries the risk, and it is gated in + `test_ltx2_vae` at `num_pixel_frames != 1`, where the fix shows: 4 of 48 + differing at `frame_idx` 0 and 0 of 48 at `frame_idx` 8. The pre-existing + negative-half assertion beside it (`no_fix.positions == state.positions` at + `frame_idx` 5, `num_pixel_frames = 1`) holds VACUOUSLY and is not this row's to + rewrite. + +- **The tree-wide correction of two off-by-N upstream anchors**, + [#1230](https://github.com/mudler/vllm.cpp/issues/1230). Re-derived at the + LTX-2 pin `fd4ded7f` by reading the pinned files rather than inheriting the + citation: `latent_cond.py:38` is `latent_state = latent_state.clone()` and + `:39` is blank, so the two writes are `:40-41`; `schedulers.py:31` is the + return annotation `) -> torch.FloatTensor:`, so the + `tokens = math.prod(latent.shape[2:])` read is `:32`. Twenty-two citations + carry the stale form and eight already carry the corrected one, inside the + same files: `ltx2_video.cpp` cites `schedulers.py:31` at `:3618` and `:32` at + `:3017` and `:3600`. This row's review repair corrected the seven its own new + lines restated and those corrections were then REVERTED, because a PARTIAL + correction is strictly worse than none. It left `ltx2_video.cpp` reading + `latent_cond.py:38-39` at `:2208` and `:3174` and `:40-41` a hundred lines + later with nothing recording which a reader should believe, and a uniform + wrong anchor is correctable by one grep while a mixed one is not. One of the + seven also lived in `include/vllm/`, a `USER_USAGE_PREFIXES` path in + `scripts/check-doc-checkpoint.py:99` -- a pure path match with no content + analysis -- so a comment-only anchor edit in a public header demanded a + `docs/USAGE.md` edit this change did not owe, and manufacturing one to turn a + gate green is what AGENTS.md forbids. Both effects are properties of doing it + piecemeal. The records this row writes state the anchors CORRECTLY -- the + table at `:97` and `:129` above, and #1219's index row -- so the record says + what is true while the source stays uniformly stale until #1230 sweeps it in + one commit with its own reviewer. + +## Stop conditions + +Return `NEEDS_DECISION` rather than narrowing scope if the conditioning-builder +field cannot be added without moving a landed arm, or if serving frame 0 through +the keyframe item turns out to need a second latent-state seam this engine does +not have. diff --git a/.agents/specs/ltx25-ti2vid-recipe.md b/.agents/specs/ltx25-ti2vid-recipe.md index 134307de7..fedf2b4cb 100644 --- a/.agents/specs/ltx25-ti2vid-recipe.md +++ b/.agents/specs/ltx25-ti2vid-recipe.md @@ -164,7 +164,7 @@ inverts how #1093 and `ltx25-res2s-loop.md:80-88` both framed it. | `retake.py:287` | no | `retake`, non-distilled arm | **no** | | `a2vid_two_stage.py:226` | no | `a2vid_two_stage` stage 1 | **no** | | `ti2vid_two_stages.py:244` | no | **this row** | this row makes it yes | -| `keyframe_interpolation.py:200` | no | unported (#1096) | n/a | +| `keyframe_interpolation.py:200` | no | `keyframe_interpolation` stage 1 | yes, since row LTX25-KEYFRAME-INTERP (#1096) | | `ti2vid_two_stages_hq.py:267` | **yes** | `res2s_two_stage` stage 1 | yes | So the engine mirrors the exception and diverges from the rule. That is #1150, @@ -538,7 +538,13 @@ RED before the recipe landed, captured on the same binaries: `test_ltx2_pipeline (`:151`'s `*tuple(loras)`). This engine's one adapter slot is upstream's `distilled_lora`; the second list has no spelling here until the adapter arity refusal lifts (`ltx2_lora.h:167-172`). -- **`keyframe_interpolation`** (#1096), the fourth pipeline on this parser. +- ~~**`keyframe_interpolation`** (#1096), the fourth pipeline on this parser.~~ + Landed as row `LTX25-KEYFRAME-INTERP`, which took this row as its template and + set `schedule_tokens = kSchedulerDefault` on its own stage 1. That moved the + keyframe line of the table above from `n/a` to `yes` and left the count owed to + #1150 at **three**, where it already stood: the six-to-one split is the + `latent` column, which no port can change, and the keyframe arm was never one + of the divergent three because it was unported rather than wrong. - **`allow_request_latents` has no reader in `src/`**, on any recipe, owned by [#1152](https://github.com/mudler/vllm.cpp/issues/1152). Four assignments on `origin/main` and five with this row's, zero readers in either count, against diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 121881ea8..df916c63c 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -181,6 +181,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 VIDEO guidance | LTX-2.5 DiT, joint video+audio | `test_ltx2_video`'s `ltx2 one_stage:` cases through `Generate`; all FOUR arms carry the x0 invariant and the guider output replays EXACTLY | `--pipeline-kind one_stage` runs `_guided_denoise`: 4 forwards/step, combined per modality in **x0**. Was ONE unguided forward, every `video_guidance` field dead ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) | | LTX-2.5 cross-attention perturbations | LTX-2.5 DiT | `test_ltx2_video` gates each direction ALONE, on a forward where one stream is PRESENT but DISABLED so only that one runs: the flag moves the stream it writes, the other leaves it bit-identical. Swapping the two is RED | `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN` ported, which `modality_scale = 3.0` selects on every video row. The DEVICE forward takes no perturbations, so that pass is refused there by name | | LTX-2.5 audio-to-video (`A2VidPipelineTwoStage`) | LTX-2.5 DiT + audio VAE encoder + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video`'s `ltx2 a2vid:` cases through `LoadVideoEngine`+`Generate`; the take's latent is bit-identical across SEEDS and moves with the WINDOW | `--pipeline-kind a2vid_two_stage`. Guided half-res stage 1, DERIVED schedule, plain Euler; distilled 3-sigma stage 2. `--audio-path` and `--lora` REQUIRED; the distilled adapter rides stage 2 ALONE (#1118) | +| LTX-2.5 keyframe interpolation (`KeyframeInterpolationPipeline`) | LTX-2.5 DiT + spatial upsampler | `ltx2 keyframe:` cases in `test_ltx2_pipeline` / `test_ltx2_video` via `LoadVideoEngine`+`Generate`: frame 0 APPENDS against a `ti2vid_two_stage` control, the x0 invariant on four arms, the 4096 anchor | `--pipeline-kind keyframe_interpolation`. No frame-0 special case, so `--first-frame` is guidance to interpolate FROM; stage 2's audio leaves. `--lora` REQUIRED. `--last-frame` new (#1191). CPU fixtures | | LTX-2.5 two-stage text/image-to-video (`TI2VidTwoStagesPipeline`) | LTX-2.5 DiT + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video` `ltx2 ti2vid:` cases through `LoadVideoEngine`+`Generate`; the x0 invariant on all FOUR arms, and the 4096 anchor read at two geometries against a res_2s control that moves | `--pipeline-kind ti2vid_two_stage`. Guided half-res stage 1 on the UNADAPTED model, plain Euler; distilled 3-sigma stage 2. `--lora` REQUIRED, no `--audio-path`; stage 1's audio leaves. CPU fixtures, Full-model run owed | | LTX-2.5 guidance knobs | LTX-2.5 request surface | `test_ltx2_video` renders with an override and refuses one on a fixed recipe | Seven video/audio guider extras mirroring `default_1_stage_arg_parser`, plus a negative embeds pair for a tower-less engine. Refused whole on `distilled_two_stage` and `retake`, whose guidance is distilled in | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | diff --git a/docs/USAGE.md b/docs/USAGE.md index 3c789f2aa..962360f55 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -3438,7 +3438,7 @@ CHECKPOINT_ROOT=... VLLM_CPP_LTX2_TOWER_E2E=1 \ Recipes resolve on an EXACT `(pipeline_kind, model_version)` pair and refuse anything else by name rather than defaulting, because a plausible but wrong sigma -schedule or guidance scale renders a video instead of failing. **Twenty-four** +schedule or guidance scale renders a video instead of failing. **Twenty-eight** pairs resolve, derived from `ResolveLtx2PipelineRecipe`: | `pipeline_kind` | resolving `model_version` | what it also needs | @@ -3452,6 +3452,7 @@ pairs resolve, derived from `ResolveLtx2PipelineRecipe`: | `t2a_one_stage` | 2, 2.3, 2.4, 2.5 | a text tower; no video VAE is asked for | | `a2vid_two_stage` | 2, 2.3, 2.4, 2.5 | `upsampler_path`, `lora_path`, and an `audio_path` on every request | | `ti2vid_two_stage` | 2, 2.3, 2.4, 2.5 | `upsampler_path` and `lora_path` | +| `keyframe_interpolation` | 2, 2.3, 2.4, 2.5 | `upsampler_path` and `lora_path` | This list ran to ten until 2026-08-17, omitting `dfr` entirely and all four `t2a_one_stage` rows. **`dfr` at 2 is refused deliberately, not by oversight**: @@ -3659,6 +3660,82 @@ of `/v1/videos`. That is a statement about the request surface: the gated path is `vllm_video_engine_load` plus `vllm_video_generate`, which is what `ltx2-gen` drives, and no test here exercises the HTTP route end to end. +### `keyframe_interpolation`: generating the motion between pinned frames + +`KeyframeInterpolationPipeline` — you supply the keyframes, the model generates +what happens between them. Its two stages are `ti2vid_two_stage`'s: a guided +half-resolution stage 1 on the **unadapted** model, then a 2x latent upsample and +a distilled three-sigma refinement. It needs the same `--lora-path` and +`--upsampler-path`, for the same reasons. + +```sh +ltx2-gen \ + --pipeline-kind keyframe_interpolation \ + --checkpoint "$CHECKPOINT_ROOT/ltx-2.5/..." \ + --upsampler-path "$CHECKPOINT_ROOT/ltx-2.5/.../spatial-upsampler.safetensors" \ + --lora-path "$CHECKPOINT_ROOT/ltx-2.5/.../ltx-2.5-22b-distilled-lora-450-bf16.safetensors" \ + --prompt 'the balloon drifts from the left ridge to the right one' \ + --first-frame open.ppm --last-frame close.ppm --image-crf 0 \ + --height 704 --width 1216 --num-frames 121 --steps 30 \ + --output-dir out/ +``` + +**Two fields separate it from `ti2vid_two_stage`, and both render either way.** + +**The first frame is a KEYFRAME, not a replacement.** Every other pipeline maps a +conditioning image at frame 0 onto a latent-index item, which overwrites the +tokens of latent frame 0 in place. This one drops that special case: the image is +appended as keyframe guidance the model interpolates *from*, and the sequence the +transformer runs over grows by one latent frame. Nothing about a rendered clip +shows which mapping was used — both return the right size, the right frame count +and the right sample rate with the image visibly present — so the difference is +gated on the token count the transformer actually ran over. + +**The soundtrack that leaves is stage 2's**, where `ti2vid_two_stage` keeps stage +1's and discards its refinement stage's audio. Upstream says so by what it binds +rather than in a comment, and the two pipelines bind opposite ways. + +Everything else is shared. `--lora-path` is **required** and the load is refused +without it: upstream makes the distilled adapter a positional, non-defaulted +constructor argument as well as a required flag, and the adapter rides **stage 2 +alone** while stage 1 runs the base weights. There is no `--audio-path`; the +soundtrack is generated. Height and width describe the FINAL output and must +divide 64, because stage 1 halves them. Its stage-1 sigma shift is fitted on the +scheduler's fixed 4096-token anchor rather than on the target latent grid, which +is what upstream's `execute(steps=...)` with no latent resolves to. + +**`--last-frame` is new with this kind** and works on every pipeline that takes +images: the ABI and the engine have served a closing keyframe since +[#930](https://github.com/mudler/vllm.cpp/issues/930), and `ltx2-gen` had never +read the field ([#1191](https://github.com/mudler/vllm.cpp/issues/1191)). Both +image slots share one `--image-crf` and one strength, and a keyframe at an +**interior** frame is not requestable — upstream's `--image PATH FRAME_IDX +STRENGTH [CRF]` is repeatable and this request surface carries two fixed slots +([#1187](https://github.com/mudler/vllm.cpp/issues/1187)). + +**Which weights this was gated against: reduced CPU fixtures, and nothing else.** +Upstream runs this pipeline on the FULL model +(`ltx-2.5-22b-dev-transformer-bf16.safetensors`, 42,018,190,584 bytes, 4349 +tensors, 21.004 B parameters, pure BF16, `model_version` `2.5.0`), which is on +the NAS and header-verified, and which `LTX25-BF16-DIT` +([#1148](https://github.com/mudler/vllm.cpp/issues/1148)) made loadable. **What +is owed is the run**: a comparison against upstream's own render on the same +checkpoint, prompt and seed. Do **not** substitute a distilled transformer to try +the arm out — the distilled scales are trained into those weights, so a +CFG-guided stage 1 on top samples a trajectory they were never trained for and +renders a plausible clip with nothing in its size, frame count, sample rate or +errors to show it ([#1137](https://github.com/mudler/vllm.cpp/issues/1137)). + +All three knobs this arm needs are LOAD extras, so a server supplies them with +`--video-extra pipeline_kind=keyframe_interpolation` and the same for +`lora_path` and `upsampler_path`. Like `ti2vid_two_stage` and unlike +`a2vid_two_stage` it needs no per-generation extra, so +[#928](https://github.com/mudler/vllm.cpp/issues/928) does not stand in the way +of `/v1/videos` — though `/v1/videos` forwards no image either, so a server +render is unconditioned. That is a statement about the request surface: the gated +path is `vllm_video_engine_load` plus `vllm_video_generate`, which is what +`ltx2-gen` drives, and no test here exercises the HTTP route end to end. + ### Retake: regenerating a time window of an existing clip `retake` is `RetakePipeline`: it keeps the source clip outside a window and diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index b447798e5..4a4efb121 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -101,7 +101,8 @@ const char* Need(int argc, char** argv, int i, const char* flag) { " [--lora [STRENGTH]] fused at load; 1.0\n" " [--prompt-valid-rows N] how many embed rows are real tokens\n" " [--frames N] [--width N] [--height N] [--seed N]\n" - " [--first-frame ] [--image-crf 0]\n" + " [--first-frame ] [--last-frame ]\n" + " [--image-crf 0]\n" " [--audio-path ] [--audio-start-time S]\n" " [--audio-max-duration S]\n" " [--device cpu|cuda]\n\n" @@ -135,7 +136,12 @@ const char* Need(int argc, char** argv, int i, const char* flag) { "round trip needs libx264 and none is vendored here, so leaving --image-crf out\n" "resolves 18 and REFUSES by name. --image-crf 0 is upstream-legal and OUT OF\n" "DISTRIBUTION: the model sees pixels it was not trained on. That is a quality\n" - "cost, and this tool states it rather than turning it on quietly.\n\n" + "cost, and this tool states it rather than turning it on quietly.\n" + "--last-frame takes a second PPM and pins the CLOSING frame. It is a KEYFRAME\n" + "rather than a replacement: its tokens are APPENDED to the sequence, carrying\n" + "the temporal position of pixel frame N-1, and are trimmed off again before the\n" + "clip is decoded. Both slots share one --image-crf and one strength, and a\n" + "keyframe at an INTERIOR frame is not requestable (#1187).\n\n" "AUDIO-TO-VIDEO. --audio-path takes a 16-bit PCM WAV and CONDITIONS the render\n" "on it: the take is decoded, encoded through the audio VAE\'s encoder, truncated\n" "to the clip\'s duration, and then held FROZEN through every denoise phase, so\n" @@ -221,7 +227,19 @@ const char* Need(int argc, char** argv, int i, const char* flag) { "divide 64, since stage 1 halves them. Upstream runs this on the FULL\n" "(non-distilled) transformer; pointing it at a distilled checkpoint renders a\n" "plausible clip on a trajectory those weights were never trained for, and\n" - "nothing in the output says so.\n"); + "nothing in the output says so.\n\n" + "KEYFRAME INTERPOLATION generates the motion BETWEEN keyframes you pin.\n" + "--pipeline-kind keyframe_interpolation selects it. Its two stages are the\n" + "ti2vid_two_stage ones -- guided half-res stage 1 on the UNADAPTED model, then a\n" + "distilled three-sigma refinement -- and it needs the same --lora and\n" + "--upsampler for the same reasons. TWO fields differ and both of them render\n" + "either way. First, --first-frame is a KEYFRAME here rather than a frame that\n" + "overwrites the opening latent: upstream drops the frame-0 special case, so the\n" + "image is appended as guidance the model interpolates FROM instead of replacing\n" + "what it would otherwise generate. Second, the audio.wav you get back is STAGE\n" + "2\'s, not stage 1\'s as on ti2vid_two_stage. Use --first-frame and --last-frame\n" + "together to pin both ends of the clip; a keyframe at an INTERIOR frame is not\n" + "requestable yet.\n"); std::exit(code); } @@ -233,7 +251,7 @@ int main(int argc, char** argv) { std::string workdir = "/tmp/ltx2_gen", out_path, ffmpeg = "ffmpeg", device = "cuda"; // BORROWED by `vllm_video_generate`, like the extras below, so it is owned // here and pointed at only after parsing. - std::string prompt, first_frame, image_crf; + std::string prompt, first_frame, last_frame, image_crf; std::string audio_path, audio_start_time, audio_max_duration; // RETAKE (row LTX25-RETAKE, #924): a source clip DIRECTORY and the window to // regenerate. `--ref-video` is a directory of frame_%06d.ppm, not a container. @@ -319,6 +337,15 @@ int main(int argc, char** argv) { // own 18 and refuse, which is the point: this CLI must not be the thing that // quietly turns an out-of-distribution render on. else if (f == "--first-frame") first_frame = Need(argc, argv, ++i, "--first-frame"); + // `--last-frame` pins the CLOSING keyframe, at pixel frame `frames - 1`. The + // ABI has carried `last_frame` and the engine has served it since row + // LTX25-TOKEN-APPEND (#930); this CLI simply never read the field, which + // #1191 records. It matters from `keyframe_interpolation` on, because a + // pipeline whose whole job is the motion BETWEEN two pinned frames could + // otherwise only be asked for one of them. Same `--image-crf` and the same + // strength as the first frame, because the request surface carries one of + // each (#1187). + else if (f == "--last-frame") last_frame = Need(argc, argv, ++i, "--last-frame"); else if (f == "--image-crf") image_crf = Need(argc, argv, ++i, "--image-crf"); // AUDIO-TO-VIDEO (#922). Upstream's `--audio-path` is REQUIRED because that // CLI drives the A2V pipeline and nothing else (a2vid_two_stage.py:312-317); @@ -399,6 +426,7 @@ int main(int argc, char** argv) { vp.output_dir = workdir.c_str(); if (!prompt.empty()) vp.prompt = prompt.c_str(); if (!first_frame.empty()) vp.first_frame = first_frame.c_str(); + if (!last_frame.empty()) vp.last_frame = last_frame.c_str(); if (!ref_video.empty()) vp.ref_video = ref_video.c_str(); // The PER-GENERATION extras are a SEPARATE array from the load-time ones, and diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index 69ebb281d..5733bc7f0 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -674,7 +674,7 @@ enum class Ltx2PhaseLoraScope { // retake.py:287 no latent our `retake`, non-distilled arm // a2vid_two_stage.py:226 no latent our `a2vid_two_stage` stage 1 // ti2vid_two_stages.py:244 no latent our `ti2vid_two_stage` stage 1 -// keyframe_interpolation.py:200 no latent unported (#1096) +// keyframe_interpolation.py:200 no latent our `keyframe_interpolation` stage 1 // ti2vid_two_stages_hq.py:267 latent=empty_latent our `res2s_two_stage` // // So the LATENT-DERIVED anchor is upstream's exception, not its rule — which is @@ -737,6 +737,50 @@ struct Ltx2PhaseRecipe { int64_t num_inference_steps() const; }; +// WHICH OF UPSTREAM'S TWO IMAGE-CONDITIONING BUILDERS this recipe runs over its +// `images` list. Row LTX25-KEYFRAME-INTERP, issue #1096. +// +// The two functions live side by side in +// `ltx-pipelines/utils/helpers.py` and differ by exactly one branch: +// +// combined_image_conditionings ..._by_adding_guiding_latent +// (:272-308) (:343-367) +// frame_idx == 0 VideoConditionByLatentIndex VideoConditionByKeyframeIndex +// any other frame_idx VideoConditionByKeyframeIndex VideoConditionByKeyframeIndex +// +// The second has no branch at all — one loop, one item type. And the two items +// do different things to the state: `VideoConditionByLatentIndex` REPLACES the +// clean tokens of latent frame 0 and the token count never changes +// (latent_cond.py:38-39), while `VideoConditionByKeyframeIndex` APPENDS a latent +// frame of tokens at the end (keyframe_cond.py:79-82). +// +// A RECIPE FIELD RATHER THAN A PHASE ONE, because upstream picks the builder per +// PIPELINE: `keyframe_interpolation.py` calls the same one for both of its +// stages (`:211` and `:260`, differing only in the height and width they pass). +// A per-phase field would offer a combination upstream has no call site for. +// +// AND NOT A `pipeline_kind` STRING COMPARE at the reader, for the reason +// `audio_only` and `requires_audio_input` give below: the recipe table is the +// one place that knows, and a string test at the call site is one more chance +// for the next recipe on this builder to be missed. +// +// WHY THIS IS INVISIBLE WITHOUT A DELIBERATE GATE. A `keyframe_interpolation` +// render built on the replace arm returns a clip of the right size, the right +// frame count and the right sample rate, with the supplied image pinned into it. +// It IS conditioned — it is conditioned as a different pipeline. The only +// observable is the sequence LENGTH the DiT ran over, which is +// `Ltx2ConditioningTrace::video_tokens`; no pixel comparison and no shape check +// reads it. +enum class Ltx2ImageConditioningBuilder { + // `combined_image_conditionings` (helpers.py:272-308). Frame 0 REPLACES. Every + // other pipeline in `ltx-pipelines`, so this is upstream's majority as well as + // this engine's incumbent behaviour — the default moves nothing. + kCombined, + // `image_conditionings_by_adding_guiding_latent` (helpers.py:343-367). Frame 0 + // APPENDS, like every other frame. `keyframe_interpolation.py:211`, `:260`. + kAddGuidingLatent, +}; + // LTXPipelineRecipe (ltx2_recipes.py:53-87). struct Ltx2PipelineRecipe { std::vector phases; @@ -804,6 +848,12 @@ struct Ltx2PipelineRecipe { // runs it. The two were conflated while only one placement existed. bool requires_distilled_lora = false; + // See `Ltx2ImageConditioningBuilder`. Read in exactly one place — the phase + // loop's first-frame arm, which is the only site where the two builders + // disagree. The last-frame arm below it is unchanged, because `frame_idx != 0` + // takes the keyframe item under BOTH of them. + Ltx2ImageConditioningBuilder image_conditioning = Ltx2ImageConditioningBuilder::kCombined; + int64_t max_spatial_downscale() const; }; diff --git a/src/vllm/model_executor/models/ltx2_pipeline.cpp b/src/vllm/model_executor/models/ltx2_pipeline.cpp index 0faa209d1..373fcd95e 100644 --- a/src/vllm/model_executor/models/ltx2_pipeline.cpp +++ b/src/vllm/model_executor/models/ltx2_pipeline.cpp @@ -1785,6 +1785,190 @@ Ltx2PipelineRecipe Ti2VidTwoStageRecipe(const Ltx2PipelineParams& params, return recipe; } +// `KeyframeInterpolationPipeline` (keyframe_interpolation.py:55, `__call__` at +// :147). Row LTX25-KEYFRAME-INTERP, issue #1096. +// +// UPSTREAM'S INTERPOLATION PIPELINE: the caller pins keyframes and the model +// generates the motion between them. Structurally it is `ti2vid_two_stage` — +// same parser, same two stages, same Euler stepper, same frozen stage-2 sigma +// set, same unadapted guided stage 1 — and it differs in exactly TWO fields. +// Both differences render either way, which is why each one is commented at +// length where it is set. +// +// ─── DIFFERENCE 1: THE IMAGE-CONDITIONING BUILDER ──────────────────────────── +// `:211` and `:260` call `image_conditionings_by_adding_guiding_latent` +// (helpers.py:343-367). `ti2vid_two_stages.py:231` and `:276` call +// `combined_image_conditionings` (:272-308). The two functions differ by ONE +// branch: the second sends `frame_idx == 0` to `VideoConditionByLatentIndex`, +// which REPLACES latent frame 0's clean tokens; the first has no branch at all +// and sends every image, frame 0 included, to `VideoConditionByKeyframeIndex`, +// which APPENDS. See `Ltx2ImageConditioningBuilder`. +// +// That is the pipeline's NAME: the first image is a keyframe to interpolate +// FROM, not a frame to overwrite. +// +// ─── DIFFERENCE 2: STAGE 2'S SOUNDTRACK IS THE ONE THAT LEAVES ─────────────── +// `:271` binds `video_state, audio_state = self.stage_2(...)` and `:293` decodes +// that name. `ti2vid_two_stages.py:289` binds `video_state, _` under its own +// comment at `:287-288` — "Stage 2 refines video only; discard its audio" — and +// `:311` decodes the name `:247` bound instead. There is no comment either way +// in this file; the BINDING is the statement, and it is the opposite one. +// +// ─── WHAT IS THE SAME, AND IS STILL DERIVED RATHER THAN COPIED ─────────────── +// The stepper is Euler because neither `self.stage_1(...)` (:231-252) nor +// `self.stage_2(...)` (:271-290) passes `stepper` or `loop`, so +// `DiffusionStage.__call__`'s defaults apply (utils/blocks.py:524-527). +// `distilled.py:76-84` selects the ANCESTRAL stepper on generation 2.5 and +// reaches this pipeline through nothing. +// +// ─── `FactoryGuidedDenoiser` IS `GuidedDenoiser` ON THE DEFAULT PATH ───────── +// `:232` builds a `FactoryGuidedDenoiser`, which resolves a guider per step from +// sigma (utils/denoisers.py:332-343). On this pipeline's default path it +// resolves ONE: `main()` passes plain `MultiModalGuiderParams` (:325-340), so +// `create_multimodal_guider_factory` takes `MultiModalGuiderFactory.constant` +// (guiders.py:360), which builds a single `(inf, params)` bin (:312-315) and +// returns the same guider at every sigma. `_guided_denoise` +// (utils/denoisers.py:61-211) is then what runs, and that is `Ltx2GuidedDenoise`. +// #1096 named the per-sigma denoiser as a blocker on this row without deriving +// that; the sigma-BINNED arm — `MultiModalGuiderFactory.from_dict` +// (guiders.py:317-330), which no CLI flag builds — is owed under #1187. +Ltx2PipelineRecipe KeyframeInterpolationRecipe(const Ltx2PipelineParams& params, + const std::string& negative_prompt) { + Ltx2PipelineRecipe recipe; + + Ltx2PhaseRecipe stage1; + // Upstream's own attribute names (`:100`, `:112`), which is what a refusal + // quotes back to a caller. + stage1.name = "stage_1"; + // `width // 2, height // 2` (`:203-209`). This is also what makes + // `assert_resolution(is_two_stage=True)` (`:170`) the 64-divisor arm here: + // `max_spatial_downscale()` derives the divisor from this field. + stage1.spatial_downscale = 2; + // EMPTY on purpose: `self._scheduler.execute(steps=num_inference_steps)` + // (`:199-200`) is resolved at run time from the request's step count. + stage1.sigmas = {}; + stage1.use_official_sigma_schedule = true; + // ...AND THAT SAME CALL PASSES NO LATENT, which `schedulers.py:31` reads as + // `default_number_of_tokens` = 4096 rather than as the target grid. The ONE + // upstream site that goes the other way is `ti2vid_two_stages_hq.py:267`, i.e. + // `Res2sTwoStageRecipe`, which leaves this at the default. That three shipped + // arms still take the target grid is #1150, not this line. + stage1.schedule_tokens = Ltx2PhaseScheduleTokens::kSchedulerDefault; + // `ModalitySpec.noise_scale` defaults to 1.0 (utils/types.py:110) and + // `:244-247` sets none, so stage 1 starts from pure noise. #1013 is the defect + // this line exists to not repeat: at 0.0 the state stays as + // `create_initial_state` wrote it, which with no initial latent is all zeros, + // and a zero-initialized denoise still returns a finite clip. + stage1.noise_scale = 1.0; + // `create_multimodal_guider_factory(params=video_guider_params, ...)` + // (`:222-225`), whose six fields are the params table's video row through the + // CLI defaults (`:325-332`, utils/args.py:947-1006). + stage1.video_guidance = params.video_guider; + // THE PARAMS TABLE'S AUDIO ROW, as on `Ti2VidTwoStageRecipe` and NOT as on + // `A2VidTwoStageRecipe`. All three come off the same parser; the difference is + // that A2Vid's audio stream is the caller's FROZEN take, so it builds a + // default `MultiModalGuiderParams()` (a2vid_two_stage.py:237-239). This + // pipeline GENERATES its soundtrack, and `:226-229` hands the audio guider + // factory the real params, filled from six `--audio-*` / + // `--v2a-guidance-scale` flags at `:333-340`. + stage1.audio_guidance = params.audio_guider; + // `FactoryGuidedDenoiser(...)` at `:232-237` — see the header comment for why + // that is `Ltx2GuidedDenoise` here. + stage1.denoiser = Ltx2PhaseDenoiser::kGuided; + stage1.allow_guidance_override = true; + // `loras=tuple(loras)` (`:104`) — stage 1 runs WITHOUT the distilled adapter, + // against stage 2's `stage_2_loras = (*tuple(loras), *tuple(distilled_lora))` + // (`:111`, taken at `:116`). Upstream's own pipeline table says the same: + // `2 | Full + distilled LoRA | Euler | Keyframe interpolation` + // (packages/ltx-pipelines/CLAUDE.md:24), and its LoRA-conventions section + // scopes the adapter to stage 2 for TI2Vid, A2Vid and Keyframe alike. + // + // Deleting this line reverts to one fused weight set for both stages, which is + // #1118 exactly, and it renders: stage 1's guided schedule would run against + // base + distilled where upstream runs it against the base alone. + stage1.loras = Ltx2PhaseLoraScope::kNoAdapters; + stage1.stepper = Ltx2StepperKind::kEuler; + + Ltx2PhaseRecipe stage2; + stage2.name = "stage_2"; + stage2.spatial_downscale = 1; + // `stage_2_sigmas: torch.Tensor = STAGE_2_DISTILLED_SIGMAS` (`:166`) — a + // DEFAULT ARGUMENT, so this phase's schedule is frozen even though stage 1's + // is not (utils/constants.py:19-23). + stage2.sigmas = Stage2DistilledSigmas(); + stage2.use_official_sigma_schedule = false; + // `noise_scale=stage_2_sigmas[0].item()` on BOTH modality specs (`:282`, + // `:287`) — the upsampled latent is only valid at the noise level this stage + // starts from. + stage2.noise_scale = Stage2DistilledSigmas().front(); + // `self.upsampler(video_state.latent[:1])` (`:255`). + stage2.input_transform = Ltx2PhaseInputTransform::kSpatialUpsample; + // `SimpleDenoiser(v_context_p, a_context_p)` (`:272`) takes no params at all, + // so both guider fields stay at the positive-only defaults. + stage2.denoiser = Ltx2PhaseDenoiser::kSimple; + // TRUE, and this is the pair `allow_guidance_override` alone cannot express. + // The flags DO exist on this pipeline's parser (`:301` selects + // `default_2_stage_arg_parser`), so a request carrying one is legal — it + // reaches stage 1's guider and nothing else. Refusing would reject a request + // upstream accepts; applying would switch on guidance upstream's stage 2 does + // not have, and `kSimple` above is what stops that. + stage2.allow_guidance_override = true; + // Left at `kAllAdapters`: `stage_2_loras` (`:111`) is every adapter this + // engine holds. + stage2.stepper = Ltx2StepperKind::kEuler; + + recipe.phases = {stage1, stage2}; + // `default_2_stage_arg_parser` sets the request geometry to the FINAL output + // (utils/args.py:1128); stage 1 runs at half through `spatial_downscale`. + recipe.height = params.stage_2_height(); + recipe.width = params.stage_2_width(); + recipe.num_frames = params.num_frames; + recipe.frame_rate = params.frame_rate; + recipe.num_inference_steps = params.num_inference_steps; + recipe.default_image_crf = params.default_image_crf; + // `:178-186` encodes `[prompt, negative_prompt]` and reads `ctx_n` into BOTH + // guider factories' `negative_context` (`:224`, `:228`). + recipe.negative_prompt = negative_prompt; + recipe.video_output_phase = 1; + // ONE, AND `Ti2VidTwoStageRecipe` ABOVE CARRIES ZERO. `:271` is `video_state, + // audio_state = self.stage_2(...)` — the audio name is REBOUND, not discarded + // — and `:293` is `self.audio_decoder(audio_state.latent)`, which therefore + // decodes stage 2's take. `ti2vid_two_stages.py:289` binds `video_state, _` + // under its own comment at `:287-288` and decodes stage 1's. + // + // COPYING THE TI2VID LINE HERE WOULD RENDER: a soundtrack that is finite, the + // right length, at the right sample rate, and one refinement stage stale. + recipe.audio_output_phase = 1; + // Stage 1's schedule IS the step count (`:200`), so a `steps` override is + // upstream's `--num-inference-steps` and is honoured. Stage 2 carries its own + // explicit `sigmas`, which the engine reads before it consults this flag, so + // the override cannot reach the distilled refinement. + recipe.allow_request_sigmas = true; + recipe.fixed_num_inference_steps = false; + // `__call__` (`:147-168`) has no initial-latent parameter at all: stage 1's + // video spec carries none (`:244-247`) and stage 2's is the upsampler's output + // (`:283`). + recipe.allow_request_latents = false; + recipe.allow_negative_prompt = true; + // `distilled_lora` is a POSITIONAL, non-defaulted parameter (`:68`) — this + // pipeline cannot even be CONSTRUCTED without one — and `--distilled-lora` is + // `required=True` (utils/args.py:1140-1155) on the parser `:301` selects. + recipe.requires_distilled_lora = true; + // ...and NOT `requires_audio_input`: there is no `--audio-path` here, the + // soundtrack is GENERATED, and `__call__` takes `images` rather than a + // waveform. A recipe written by copying `A2VidTwoStageRecipe` would inherit + // that flag and refuse every render. + // + // THE FIELD THIS ROW ADDS, and the one that makes this recipe the pipeline it + // is rather than a second `ti2vid_two_stage`. `:211` and `:260` both call + // `image_conditionings_by_adding_guiding_latent` (helpers.py:343-367), so + // frame 0 APPENDS as a keyframe instead of REPLACING latent frame 0. See the + // header comment on `Ltx2ImageConditioningBuilder`, and the phase loop's + // first-frame arm, which is the one reader. + recipe.image_conditioning = Ltx2ImageConditioningBuilder::kAddGuidingLatent; + return recipe; +} + } // namespace Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, @@ -1866,6 +2050,32 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, if (model_version == "2.5") { return Ti2VidTwoStageRecipe(Ltx2DetectPipelineParams("2.5"), LightricksNegativePrompt()); } + } else if (pipeline_kind == "keyframe_interpolation") { + // All four generations, mirroring the `ti2vid_two_stage` rows above line for + // line: `main()` calls `resolve_cli_params()` (keyframe_interpolation.py:300) + // and hands the result to `default_2_stage_arg_parser(params=params)` + // (`:301`), so the generation comes off the CHECKPOINT. Restricting these + // rows would be a local invention. + // + // THE KIND IS UPSTREAM'S FILE AND CLASS NAME, not a `*_two_stage` shorthand, + // and that is deliberate rather than an inconsistency: `keyframe_two_stage` + // would describe the STRUCTURE this recipe shares with three others and hide + // the two fields that separate it from them. Every plausible misspelling is + // refused by name rather than defaulted. + if (model_version == "2") { + return KeyframeInterpolationRecipe(Ltx2Params20(), kOmniNegativePrompt); + } + if (model_version == "2.3") { + return KeyframeInterpolationRecipe(Ltx2Params23(), kOmniNegativePrompt); + } + if (model_version == "2.4") { + return KeyframeInterpolationRecipe(Ltx2DetectPipelineParams("2.4"), + LightricksNegativePrompt()); + } + if (model_version == "2.5") { + return KeyframeInterpolationRecipe(Ltx2DetectPipelineParams("2.5"), + LightricksNegativePrompt()); + } } else if (pipeline_kind == "t2a_one_stage") { if (model_version == "2") return T2aOneStageRecipe(Ltx2Params20(), kOmniNegativePrompt); if (model_version == "2.3") return T2aOneStageRecipe(Ltx2Params23(), kOmniNegativePrompt); diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index e3020ecda..56d8ef2d2 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -3229,11 +3229,62 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { if (wants_first_frame) { const Ltx2LatentVolume encoded = encode_conditioning_image("first_frame", image_bytes); - // `frame_idx == 0` -> `VideoConditionByLatentIndex` (helpers.py:295-300). - // It REPLACES tokens that already exist and the token count never changes. + // WHICH ITEM FRAME 0 TAKES IS THE RECIPE'S TO SAY, and it is the ONE line + // on which upstream's two image-conditioning builders differ (row + // LTX25-KEYFRAME-INTERP, #1096): + // + // combined_image_conditionings (helpers.py:295-300) + // frame_idx == 0 -> VideoConditionByLatentIndex. REPLACES tokens + // that already exist; the token count never changes. + // image_conditionings_by_adding_guiding_latent (helpers.py:343-367) + // NO branch. Frame 0 takes VideoConditionByKeyframeIndex like every + // other frame, and APPENDS a latent frame of tokens. + // + // A `pipeline_kind` string test here would be one more place for the next + // recipe on the second builder to be missed, so the flag rides the recipe + // — see `Ltx2ImageConditioningBuilder`. + // + // NOTHING ABOUT THE RENDER'S SHAPE CAN SEE THIS. Both arms condition on + // the same image and both return a clip of the right size, the right frame + // count and the right sample rate; the only observable is the sequence + // LENGTH the DiT ran over, which is `im.trace.video_tokens`. + const bool frame0_appends = + recipe.image_conditioning == Ltx2ImageConditioningBuilder::kAddGuidingLatent; + // The sequence length BEFORE this item, so the digest below can be taken + // from the tokens the append actually grew. On the replace arm it is also + // the length after, which the assertion under `frame0_appends` states. + const int64_t before_first_frame = video.tokens; + Ltx2LatentState state = ToLatentState(video, /*pos_dims=*/3); - Ltx2ConditionVideoByLatentIndex(&state, vshape, /*patch_size=*/1, encoded, image_strength, - /*latent_idx=*/0); + if (frame0_appends) { + // `frame_idx = 0` is not a formality: `VideoConditionByKeyframeIndex` + // offsets its positions by `frame_idx` (keyframe_cond.py:52), so this + // argument alone decides WHERE IN TIME the opening keyframe lands. It is + // gated below, on a temporal window recomputed from `fps`, because + // nothing else can see it — the sibling arm found the same hole with + // mutation M10 and carries the same shape of check. + // + // `causal_fix = true` is upstream's value for this item and it is INERT + // at this call site. MEASURED, so the next reader does not go looking + // for the gate that cannot exist: `Ltx2ConditionVideoByKeyframe` gates + // the fix on `frame_idx == 0` (keyframe_cond.py:49, + // `latent_tools.causal_fix if self.frame_idx == 0 else False`), so the + // gate is OPEN here and the fix is applied — and then the + // `num_pixel_frames == 1` narrow at `:56-57` overwrites the temporal END + // the fix moved, while the temporal START clamps to 0 either way for a + // keyframe whose latent depth is 1 (`max(0 + 1 - time, 0)`, and + // `encoded.frames != 1` is refused above). Flipping this argument moves + // 0 of 48 position values on a probe of the same shapes; the fix becomes + // observable only at `num_pixel_frames != 1`, which no arm here passes. + // The `frame_idx == 0` gate itself is what carries the risk, and it is + // gated on the seam in `test_ltx2_vae` at a shape where it shows. + Ltx2ConditionVideoByKeyframe(&state, encoded, /*patch_size=*/1, factors, fps, + /*frame_idx=*/0, image_strength, + /*num_pixel_frames=*/1, /*causal_fix=*/true); + } else { + Ltx2ConditionVideoByLatentIndex(&state, vshape, /*patch_size=*/1, encoded, image_strength, + /*latent_idx=*/0); + } FromLatentState(state, &video); // The witness, taken from the TOKENS THAT WERE WRITTEN rather than from @@ -3254,12 +3305,92 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { const int64_t placed = Ltx2VideoTokenCount({1, vshape.channels, 1, vshape.height, vshape.width}, 1); + // AND WHERE THEY LANDED DEPENDS ON THE ARM. The replace arm writes latent + // frame 0, which is the FRONT of the clean tensor; the append arm writes + // the tokens it just added, which are the TAIL. Reading the front on the + // append arm would digest unconditioned tokens and report a healthy + // conditioning for a state the keyframe never reached — exactly the + // failure the paragraph above says this field exists to prevent. + const ptrdiff_t first = frame0_appends + ? static_cast(before_first_frame * video.width) + : 0; const std::vector written( - video.clean.begin(), - video.clean.begin() + static_cast(placed * video.width)); + video.clean.begin() + first, + video.clean.begin() + first + static_cast(placed * video.width)); + // AND THE SLICE IS THE ONE THE IMAGE LANDED IN. `first` is the only thing + // that decides which tokens the three fields below describe, and it was + // ungated: forcing it to 0 on the append arm made the digest describe the + // untouched FRONT of the sequence, and both binaries stayed green — which + // is the exact failure the paragraph above says these fields exist to + // prevent, arriving through the slice rather than through the conditioning. + // + // Recomputed from `encoded`, not from `first`, so the two are independent + // expressions. Both conditioning items write the SAME bytes — the + // patchified conditioning volume — the replace arm at latent frame 0 + // (latent_cond.py:38-39) and the append arm at the tail + // (keyframe_cond.py:82) — so one expectation serves both, and a slice that + // names any other window reds by value rather than by shape. + Ltx2VideoLatentShape placed_shape = vshape; + placed_shape.frames = encoded.frames; + const std::vector expected = + Ltx2VideoPatchify(encoded.data.data(), placed_shape, /*patch_size=*/1); + VT_CHECK(written == expected, + "ltx2 video: `image_digest` and `image_absmax` must describe the tokens the " + "conditioning item WROTE. The slice this arm read back does not hold the " + "patchified conditioning volume, so the trace reports a healthy conditioning for " + "a window the image never reached"); im.trace.image_tokens = placed; im.trace.image_digest = DigestF32(written); im.trace.image_absmax = AbsMax(written); + + // THE ARM DID WHAT ITS NAME SAYS. Asserted rather than assumed for the + // reason the last-frame arm below asserts the same thing: a build whose + // append did not grow the sequence leaves buffers longer than the count + // that describes them, and the DiT then reads a prefix, renders a + // plausible clip and never mentions the keyframe. The replace polarity is + // asserted too, because the two arms share one digest slice above and that + // slice is only correct while each arm moves the count the way this says. + if (frame0_appends) { + VT_CHECK(video.tokens == before_first_frame + placed, + "ltx2 video: this recipe takes `image_conditionings_by_adding_guiding_latent` " + "(ltx-pipelines/utils/helpers.py:343-367), whose frame-0 item APPENDS " + "(keyframe_cond.py:79-82), and the sequence did not grow by one latent frame " + "of tokens"); + + // ...AND IT LANDED ON PIXEL FRAME 0, which is the whole content of + // `frame_idx` and which nothing above can see. MEASURED, on this arm, + // exactly as the sibling last-frame arm measured it: changing + // `frame_idx` from 0 to 3 left both binaries GREEN, because an opening + // keyframe pinned to the wrong pixel frame appends the same number of + // tokens carrying the same content and only sits somewhere else in time. + // + // The window is recomputed from `fps` alone and never read back from the + // argument, so the two are independent expressions. Upstream offsets in + // integer PIXEL space and then divides the temporal axis by fps + // (keyframe_cond.py:52-59), and `num_pixel_frames = 1` narrows the end to + // `start + 1` BEFORE that division (`:56-57`) — so the first appended + // token spans exactly `[0, 1/fps)`. Asserting the END as well as the + // START is what keeps this check reading the composition rather than the + // offset alone: it is the one place `num_pixel_frames` is observable. + const double want_t0 = 0.0; + const double want_t1 = static_cast(static_cast(1.0 / fps)); + const double got_t0 = video.positions[static_cast(before_first_frame * 2)]; + const double got_t1 = video.positions[static_cast(before_first_frame * 2 + 1)]; + VT_CHECK(std::abs(got_t0 - want_t0) <= 1e-5 && + std::abs(got_t1 - want_t1) <= 1e-5 * std::max(1.0, std::abs(want_t1)), + "ltx2 video: the first-frame keyframe's appended tokens must span pixel frame 0, " + "i.e. [" + + std::to_string(want_t0) + ", " + std::to_string(want_t1) + + "), and the first appended token spans [" + std::to_string(got_t0) + ", " + + std::to_string(got_t1) + + "). A keyframe that appends the right number of tokens at the wrong TIME " + "renders a clip of the right length that pins the image to the wrong end"); + } else { + VT_CHECK(video.tokens == before_first_frame, + "ltx2 video: `combined_image_conditionings` sends frame 0 to " + "`VideoConditionByLatentIndex` (helpers.py:295-300), which REPLACES tokens that " + "already exist (latent_cond.py:38-39), so the sequence length must not move"); + } } // ── the LAST-frame keyframe (row LTX25-TOKEN-APPEND, issue #930) ──────── @@ -3282,6 +3413,18 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { if (wants_last_frame) { const Ltx2LatentVolume encoded = encode_conditioning_image("last_frame", last_frame_bytes); + // THE SEQUENCE LENGTH BEFORE THIS ITEM, and it is not `target_tokens`. + // Every assertion below locates this item's tokens, and the only thing + // that knows where they start is the count at the moment of the append — + // NOT the target grid, which is what stands before the FIRST append and + // nothing after it. Reading the target here was correct only while this + // arm was the first append, and row LTX25-KEYFRAME-INTERP put the + // first-frame arm's own append in front of it on the + // `image_conditionings_by_adding_guiding_latent` recipes: with both ends + // pinned, `positions[target_tokens * 2]` named the FIRST frame's keyframe + // at temporal 0 and the check below aborted the render (#1219). + const int64_t before_last_frame = video.tokens; + Ltx2LatentState state = ToLatentState(video, /*pos_dims=*/3); Ltx2ConditionVideoByKeyframe(&state, encoded, /*patch_size=*/1, factors, fps, /*frame_idx=*/frames - 1, image_strength, @@ -3293,7 +3436,7 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // that dropped the grown count would leave a state whose buffers are // longer than the count that describes them — and the DiT would then read // a prefix, render a plausible clip, and never mention the keyframe. - VT_CHECK(video.tokens > target_tokens, + VT_CHECK(video.tokens > before_last_frame, "ltx2 video: a keyframe conditioning must APPEND tokens " "(keyframe_cond.py:79-82) and this one left the sequence length unchanged"); VT_CHECK(static_cast(video.latent.size()) == video.tokens * video.width && @@ -3320,13 +3463,15 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // expressions and a mutation of the argument alone moves one and not the // other. `Ltx2ConditionVideoByKeyframe` offsets the item's temporal // coordinates by `frame_idx` in integer PIXEL space and then divides the - // temporal axis by fps (keyframe_cond.py:52-58), so the first appended + // temporal axis by fps (keyframe_cond.py:52-59), so the first appended // token's temporal START is `frame_idx / fps`. Positions are // [pos_dims, tokens, 2] concatenated PER DIMENSION, so the temporal axis - // is dimension 0 and the first appended token sits at `target_tokens * 2`. + // is dimension 0 and this item's first token sits at + // `before_last_frame * 2` — the count at the moment of ITS append, which + // is the target grid only when nothing appended before it. const double want_t0 = static_cast(static_cast( static_cast(frames - 1) / fps)); - const double got_t0 = video.positions[static_cast(target_tokens * 2)]; + const double got_t0 = video.positions[static_cast(before_last_frame * 2)]; VT_CHECK(std::abs(got_t0 - want_t0) <= 1e-5 * std::max(1.0, std::abs(want_t0)), "ltx2 video: the last-frame keyframe's appended tokens must carry the temporal " "position of pixel frame `frames - 1` (" + @@ -3356,6 +3501,11 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // stage's composition, which renders a clip whose keyframes simply disagree // with the video around them. if (!slot_positions.empty()) { + // AGAIN THE COUNT AT THE APPEND, not the target grid, and for the reason + // the last-frame arm states: two supplied keyframes can already stand in + // front of this item, so `target_tokens` is the length before the FIRST + // append and describes nothing here (#1219). + const int64_t before_slots = video.tokens; Ltx2LatentState state = ToLatentState(video, /*pos_dims=*/3); Ltx2ConditionVideoByGeneratedKeyframeSlots( &state, vshape, /*patch_size=*/1, factors, fps, slot_positions, @@ -3364,11 +3514,20 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { FromLatentState(state, &video); VT_CHECK(video.tokens == - target_tokens + static_cast(slot_positions.size()) * - slot_layout.tokens_per_keyframe, + before_slots + static_cast(slot_positions.size()) * + slot_layout.tokens_per_keyframe, "ltx2 video: the generated keyframe slots must APPEND one latent frame of tokens " "per slot (keyframe_slots.py:83-84, :136-140), and the sequence did not grow by " "that many"); + // ...AND THE RECORDED LAYOUT DESCRIBES THOSE TOKENS. `first_token` is the + // state's own pre-append count (keyframe_slots.py:143-147), and the marked + // walk below starts there; a layout that named some other token would + // count the wrong window and still report a healthy total. + VT_CHECK(slot_layout.first_token == before_slots, + "ltx2 video: the generated keyframe layout must start at the token the slots were " + "appended at (keyframe_slots.py:143-147), and it names " + + std::to_string(slot_layout.first_token) + " against " + + std::to_string(before_slots)); // THE MARKER REACHED THE NEW TOKENS. This is the one thing the slot arm // buys over an ordinary append, it is what `marked=true` exists for // (keyframe_slots.py:121), and nothing about the render's shape, its token diff --git a/tests/vllm/models/test_ltx2_pipeline.cpp b/tests/vllm/models/test_ltx2_pipeline.cpp index db48bf4e7..0c08e92be 100644 --- a/tests/vllm/models/test_ltx2_pipeline.cpp +++ b/tests/vllm/models/test_ltx2_pipeline.cpp @@ -3761,3 +3761,274 @@ TEST_CASE("ltx2 ti2vid: all four generations resolve and nothing else does") { CHECK_THROWS((void)vllm::ResolveLtx2PipelineRecipe("ti2vid", "2.5")); CHECK_THROWS((void)vllm::ResolveLtx2PipelineRecipe("ti2vid_two_stage_hq", "2.5")); } + +// ─── LTX25-KEYFRAME-INTERP (#1096) ─────────────────────────────────────────── + +TEST_CASE("ltx2 keyframe: the recipe is the INTERPOLATION pipeline, not the plain two-stage one") { + // `KeyframeInterpolationPipeline` (keyframe_interpolation.py:55 @ fd4ded7f). + // Row LTX25-KEYFRAME-INTERP, issue #1096. + // + // THIS PIPELINE AND `ti2vid_two_stage` SHARE A PARSER, A STAGE LAYOUT, A + // STEPPER AND A SIGMA SET, and they disagree in exactly two fields. Both + // disagreements render either way: an interpolation that overwrites its first + // keyframe, and a soundtrack that is finite, the right length, at the right + // sample rate and the wrong take. So the controls below are drawn from THAT + // recipe above all, and no assertion can pass by two values coinciding. + const vllm::Ltx2PipelineRecipe kf = + vllm::ResolveLtx2PipelineRecipe("keyframe_interpolation", "2.5"); + const vllm::Ltx2PipelineRecipe ti2v = + vllm::ResolveLtx2PipelineRecipe("ti2vid_two_stage", "2.5"); + const vllm::Ltx2PipelineRecipe res2s = + vllm::ResolveLtx2PipelineRecipe("res2s_two_stage", "2.5"); + const vllm::Ltx2PipelineRecipe a2v = + vllm::ResolveLtx2PipelineRecipe("a2vid_two_stage", "2.5"); + const vllm::Ltx2PipelineRecipe distilled = + vllm::ResolveLtx2PipelineRecipe("distilled_two_stage", "2.5"); + const vllm::Ltx2PipelineRecipe one = vllm::ResolveLtx2PipelineRecipe("one_stage", "2.5"); + REQUIRE(kf.phases.size() == 2u); + + // ── THE FIELD THAT IS THIS PIPELINE'S NAME ──────────────────────────────── + // + // `:211` and `:260` call `image_conditionings_by_adding_guiding_latent` + // (helpers.py:343-367), which has NO branch: every image, `frame_idx == 0` + // included, becomes a `VideoConditionByKeyframeIndex` that APPENDS. Every + // other pipeline calls `combined_image_conditionings` (:272-308), whose + // `frame_idx == 0` arm is a `VideoConditionByLatentIndex` that REPLACES. + // + // FIVE CONTROLS, because the default is `kCombined` and an assertion that only + // read this recipe would pass on a build where the field exists and nothing + // sets it — and a single control would pass on a build where the default had + // been flipped the other way instead. + CHECK(kf.image_conditioning == vllm::Ltx2ImageConditioningBuilder::kAddGuidingLatent); + CHECK(ti2v.image_conditioning == vllm::Ltx2ImageConditioningBuilder::kCombined); + CHECK(res2s.image_conditioning == vllm::Ltx2ImageConditioningBuilder::kCombined); + CHECK(a2v.image_conditioning == vllm::Ltx2ImageConditioningBuilder::kCombined); + CHECK(distilled.image_conditioning == vllm::Ltx2ImageConditioningBuilder::kCombined); + CHECK(one.image_conditioning == vllm::Ltx2ImageConditioningBuilder::kCombined); + + // ── stage 1 (keyframe_interpolation.py:198-252) ─────────────────────────── + const vllm::Ltx2PhaseRecipe& s1 = kf.phases[0]; + CHECK(s1.name == "stage_1"); + // `width // 2, height // 2` (:203-209), which is also what makes + // `assert_resolution(is_two_stage=True)` (:170) the 64-divisor arm here. + CHECK(s1.spatial_downscale == 2); + CHECK(kf.max_spatial_downscale() == 2); + CHECK(s1.input_transform == vllm::Ltx2PhaseInputTransform::kInitial); + // `self._scheduler.execute(steps=num_inference_steps)` (:199-200): DERIVED at + // run time, against `distilled_two_stage`'s frozen 9-sigma stage 1. + CHECK(s1.sigmas.empty()); + CHECK(s1.use_official_sigma_schedule); + CHECK_FALSE(distilled.phases[0].sigmas.empty()); // the control + + // THE SCHEDULE ANCHOR. That same `execute` call passes NO latent, so + // `schedulers.py:31` resolves `tokens` to `default_number_of_tokens` = 4096 + // rather than to the target grid. `ti2vid_two_stages_hq.py:267` — the res_2s + // recipe — is the ONE upstream site that passes `latent=empty_latent`, and it + // is the control here precisely because it is the exception. That three other + // shipped arms still derive from the target grid is #1150, not this row. + CHECK(s1.schedule_tokens == vllm::Ltx2PhaseScheduleTokens::kSchedulerDefault); + CHECK(ti2v.phases[0].schedule_tokens == vllm::Ltx2PhaseScheduleTokens::kSchedulerDefault); + CHECK(res2s.phases[0].schedule_tokens == vllm::Ltx2PhaseScheduleTokens::kTargetLatent); + // Nothing that predates this row moved. + CHECK(one.phases[0].schedule_tokens == vllm::Ltx2PhaseScheduleTokens::kTargetLatent); + CHECK(a2v.phases[0].schedule_tokens == vllm::Ltx2PhaseScheduleTokens::kTargetLatent); + + // `ModalitySpec.noise_scale` defaults to 1.0 (utils/types.py:110) and :244-247 + // sets none, so stage 1 starts from pure noise. #1013: at 0.0 the state stays + // as `create_initial_state` wrote it, which with no initial latent is all + // zeros, and a zero-initialised denoise still returns a finite clip. + CHECK(s1.noise_scale == 1.0); + // NEITHER `self.stage_1(...)` (:231-252) NOR `self.stage_2(...)` (:271-290) + // passes `stepper` or `loop`, so `DiffusionStage.__call__`'s own defaults + // apply — `euler_denoising_loop` and `EulerDiffusionStep()` + // (utils/blocks.py:524-527). Two steppers reach this arm through nothing and + // both are asserted beside it: `distilled.py:76-84` selects the ANCESTRAL one + // on this very generation, and the HQ pipeline hands BOTH its stages + // `Res2sDiffusionStep()` (ti2vid_two_stages_hq.py:258). + CHECK(s1.stepper == vllm::Ltx2StepperKind::kEuler); + CHECK(distilled.phases[0].stepper == vllm::Ltx2StepperKind::kEulerAncestral); + CHECK(res2s.phases[0].stepper == vllm::Ltx2StepperKind::kRes2s); + CHECK(s1.stepper_eta == 0.0); + CHECK(s1.noise_seed_offset == 0); + // `loras=tuple(loras)` (:104) against stage 2's `stage_2_loras` (:111, passed + // at :116) — stage 1 runs the UNADAPTED model, which upstream's own pipeline + // table calls `Full + distilled LoRA` (packages/ltx-pipelines/CLAUDE.md:24) + // and which is the reason #1118 blocked this row. The mirror-image control is + // `res2s_two_stage`, which upstream fuses on BOTH stages. + CHECK(s1.loras == vllm::Ltx2PhaseLoraScope::kNoAdapters); + CHECK(res2s.phases[0].loras == vllm::Ltx2PhaseLoraScope::kAllAdapters); + // `FactoryGuidedDenoiser(...)` (:232-237). On this pipeline's DEFAULT path it + // resolves ONE guider for the whole schedule: `main()` passes plain + // `MultiModalGuiderParams` (:325-340), so `create_multimodal_guider_factory` + // takes `MultiModalGuiderFactory.constant` (guiders.py:360), which builds a + // single `(inf, params)` bin (:312-315). #1096 named a per-sigma denoiser as a + // blocker without deriving that; #1187 owns the sigma-BINNED arm. + CHECK(s1.denoiser == vllm::Ltx2PhaseDenoiser::kGuided); + // `:301` selects `default_2_stage_arg_parser`, which carries the six video + // guider flags (utils/args.py:947-1006), so an override is legal. + CHECK(s1.allow_guidance_override); + // The video guider is the params table's row, shared with `one_stage` so a + // change to the table moves both... + CHECK(s1.video_guidance.cfg_scale == one.phases[0].video_guidance.cfg_scale); + CHECK(s1.video_guidance.stg_scale == one.phases[0].video_guidance.stg_scale); + CHECK(s1.video_guidance.rescale_scale == one.phases[0].video_guidance.rescale_scale); + CHECK(s1.video_guidance.modality_scale == one.phases[0].video_guidance.modality_scale); + CHECK(s1.video_guidance.stg_blocks == one.phases[0].video_guidance.stg_blocks); + // ...and the values themselves, so this case still says which arm it is on if + // both were changed together. `rescale_scale = 0.7` is what makes the x0-space + // question live on the DEFAULT path (guiders.py:268-271, #1039/#1092). + CHECK(s1.video_guidance.cfg_scale == 3.0); + CHECK(s1.video_guidance.stg_scale == 1.0); + CHECK(s1.video_guidance.rescale_scale == 0.7); + CHECK(s1.video_guidance.modality_scale == 3.0); + + // THE AUDIO GUIDER IS THE TABLE'S ROW, as on `ti2vid_two_stage` and NOT as on + // `a2vid_two_stage`. This pipeline GENERATES its soundtrack and :226-229 hands + // the audio guider factory the real params, filled from six `--audio-*` / + // `--v2a-guidance-scale` flags at :333-340. A2Vid's audio stream is the + // caller's FROZEN take, so it builds a default `MultiModalGuiderParams()` + // (a2vid_two_stage.py:237-239) and its cfg 7.0 would buy an unconditional + // forward for a delta multiplied into a latent the sampler cannot move. + CHECK(s1.audio_guidance.cfg_scale == one.phases[0].audio_guidance.cfg_scale); + CHECK(s1.audio_guidance.cfg_scale == 7.0); + CHECK(s1.audio_guidance.DoUnconditionalGeneration()); + CHECK(a2v.phases[0].audio_guidance.cfg_scale == 1.0); // the control + CHECK_FALSE(a2v.phases[0].audio_guidance.DoUnconditionalGeneration()); + + // ── stage 2 (keyframe_interpolation.py:254-290) ─────────────────────────── + const vllm::Ltx2PhaseRecipe& s2 = kf.phases[1]; + CHECK(s2.name == "stage_2"); + CHECK(s2.spatial_downscale == 1); + // `self.upsampler(video_state.latent[:1])` (:255). + CHECK(s2.input_transform == vllm::Ltx2PhaseInputTransform::kSpatialUpsample); + // `stage_2_sigmas: torch.Tensor = STAGE_2_DISTILLED_SIGMAS` (:166) — a DEFAULT + // ARGUMENT, so the schedule is frozen for this phase even though stage 1's is + // not. Byte for byte the distilled recipe's stage 2 (utils/constants.py:19-23). + CHECK(s2.sigmas == distilled.phases[1].sigmas); + CHECK_FALSE(s2.use_official_sigma_schedule); + REQUIRE(s2.sigmas.size() == 4u); + // `noise_scale=stage_2_sigmas[0].item()` on BOTH modality specs (:282, :287) — + // the upsampled latent is only valid at the noise level this stage starts from. + CHECK(s2.noise_scale == s2.sigmas.front()); + CHECK(s2.stepper == vllm::Ltx2StepperKind::kEuler); + CHECK(res2s.phases[1].stepper == vllm::Ltx2StepperKind::kRes2s); // the control + // `SimpleDenoiser(v_context_p, a_context_p)` (:272) takes no params at all. + CHECK(s2.denoiser == vllm::Ltx2PhaseDenoiser::kSimple); + // ...AND the override is still ALLOWED, which is the pair `allow_guidance_ + // override` alone cannot express: the flags DO exist on this pipeline's + // parser, so a request carrying one is legal — it reaches stage 1's guider and + // nothing else. `kSimple` above is what makes it inert here. The control on + // the other polarity is `distilled_two_stage`, whose parser never adds the + // flags (utils/args.py:1188) so both of its phases REFUSE. + CHECK(s2.allow_guidance_override); + CHECK_FALSE(distilled.phases[1].allow_guidance_override); + // Left at the default: `stage_2_loras` (:111) is every adapter this engine + // holds, and :116 is where stage 2 takes it. + CHECK(s2.loras == vllm::Ltx2PhaseLoraScope::kAllAdapters); + CHECK_FALSE(s2.video_guidance.DoUnconditionalGeneration()); + CHECK_FALSE(s2.video_guidance.DoPerturbedGeneration()); + + // ── the recipe (keyframe_interpolation.py:147-168, :297-358) ────────────── + // `default_2_stage_arg_parser` sets the request geometry to the FINAL output + // (utils/args.py:1128); stage 1 runs at half through `spatial_downscale`. + CHECK(kf.height == ti2v.height); + CHECK(kf.width == ti2v.width); + CHECK(kf.num_frames == ti2v.num_frames); + CHECK(kf.frame_rate == ti2v.frame_rate); + CHECK(kf.num_inference_steps == one.num_inference_steps); + CHECK(kf.default_image_crf == one.default_image_crf); + CHECK(kf.video_output_phase == 1); + + // THE SECOND FIELD THAT SEPARATES THIS FROM `ti2vid_two_stage`, and it is the + // one most likely to be copied from it. `:271` binds `video_state, + // audio_state = self.stage_2(...)` and `:293` decodes that name, so stage 2's + // soundtrack is what LEAVES. `ti2vid_two_stages.py:289` binds `video_state, _` + // under its own comment at `:287-288` — "Stage 2 refines video only; discard + // its audio" — and decodes the name `:247` bound instead. There is no comment + // either way in this file; the BINDING is the statement. + // + // Writing 0 here would decode stage 1's take: finite, the right length, at the + // right sample rate, and the wrong one. The controls are two-against-two, so + // this cannot pass because every two-stage recipe happens to agree. + CHECK(kf.audio_output_phase == 1); + CHECK(a2v.audio_output_phase == 1); + CHECK(ti2v.audio_output_phase == 0); + CHECK(res2s.audio_output_phase == 0); + CHECK_FALSE(kf.audio_only); + + // Stage 1's schedule IS the step count (:200), so `--num-inference-steps` is + // honoured; stage 2 carries its own explicit sigmas and is unaffected either + // way, exactly as upstream's two parameters are. + CHECK(kf.allow_request_sigmas); + CHECK_FALSE(kf.fixed_num_inference_steps); + CHECK_FALSE(distilled.allow_request_sigmas); // the control + // `:150` takes a negative prompt and :178-186 encodes `[prompt, + // negative_prompt]` into the two guider factories' `negative_context`. + CHECK(kf.allow_negative_prompt); + CHECK(kf.negative_prompt == one.negative_prompt); + CHECK_FALSE(kf.negative_prompt.empty()); + CHECK_FALSE(distilled.allow_negative_prompt); // the control + // No `__call__` parameter carries an initial latent (:147-168): stage 1's + // video spec has none and stage 2's is the upsampler's output. + CHECK_FALSE(kf.allow_request_latents); + // `distilled_lora` is POSITIONAL and non-defaulted (:68), and + // `--distilled-lora` is `required=True` on the parser :301 selects + // (utils/args.py:1140-1155). + CHECK(kf.requires_distilled_lora); + // ...but there is NO `--audio-path` on this pipeline: the soundtrack is + // generated, not supplied. This is the field that separates the recipe from + // `a2vid_two_stage`, which sets both flags, so asserting only the first would + // pass on a copy of that recipe. + CHECK_FALSE(kf.requires_audio_input); + CHECK(a2v.requires_audio_input); // the control + CHECK(a2v.requires_distilled_lora); + CHECK(ti2v.requires_distilled_lora); + CHECK_FALSE(one.requires_distilled_lora); + CHECK_FALSE(distilled.requires_distilled_lora); +} + +TEST_CASE("ltx2 keyframe: all four generations resolve and nothing else does") { + // FOUR ROWS, mirroring `ti2vid_two_stage` and `a2vid_two_stage` line for line + // and for the same reason: `main()` calls `resolve_cli_params()` (:300) and + // hands the result to `default_2_stage_arg_parser(params=params)` (:301), so + // the generation comes off the checkpoint. There is no "which generations + // support this pipeline" question upstream, so restricting these rows would be + // a local invention. + for (const char* version : {"2", "2.3", "2.4", "2.5"}) { + INFO("version = ", std::string(version)); + CHECK_NOTHROW((void)vllm::ResolveLtx2PipelineRecipe("keyframe_interpolation", version)); + const vllm::Ltx2PipelineRecipe r = + vllm::ResolveLtx2PipelineRecipe("keyframe_interpolation", version); + REQUIRE(r.phases.size() == 2u); + CHECK(r.image_conditioning == vllm::Ltx2ImageConditioningBuilder::kAddGuidingLatent); + CHECK(r.requires_distilled_lora); + CHECK_FALSE(r.requires_audio_input); + CHECK(r.phases[0].spatial_downscale == 2); + CHECK(r.phases[0].stepper == vllm::Ltx2StepperKind::kEuler); + CHECK(r.phases[0].loras == vllm::Ltx2PhaseLoraScope::kNoAdapters); + CHECK(r.phases[0].schedule_tokens == vllm::Ltx2PhaseScheduleTokens::kSchedulerDefault); + CHECK(r.audio_output_phase == 1); + } + // The 2.4 and 2.5 rows take Lightricks' negative prompt and the older two take + // vLLM-Omni's, which is the split every four-key row makes: the negative + // prompt travels with the GENERATION, not with the pipeline. + CHECK(vllm::ResolveLtx2PipelineRecipe("keyframe_interpolation", "2.5").negative_prompt == + vllm::ResolveLtx2PipelineRecipe("one_stage", "2.5").negative_prompt); + CHECK(vllm::ResolveLtx2PipelineRecipe("keyframe_interpolation", "2").negative_prompt == + vllm::ResolveLtx2PipelineRecipe("one_stage", "2").negative_prompt); + CHECK(vllm::ResolveLtx2PipelineRecipe("keyframe_interpolation", "2.5").negative_prompt != + vllm::ResolveLtx2PipelineRecipe("keyframe_interpolation", "2").negative_prompt); + + // A version the table does not carry is REFUSED by name, never defaulted onto + // a neighbouring generation's guidance scales. + CHECK_THROWS((void)vllm::ResolveLtx2PipelineRecipe("keyframe_interpolation", "2.6")); + CHECK_THROWS((void)vllm::ResolveLtx2PipelineRecipe("keyframe_interpolation", "")); + // ...and so are the near-miss spellings. This kind is upstream's FILE and + // CLASS name, unlike every `*_two_stage` kind in the table, so the plausible + // wrong spellings are the ones that assume the table's convention rather than + // upstream's. + CHECK_THROWS((void)vllm::ResolveLtx2PipelineRecipe("keyframe_interpolation_two_stage", "2.5")); + CHECK_THROWS((void)vllm::ResolveLtx2PipelineRecipe("keyframe_two_stage", "2.5")); + CHECK_THROWS((void)vllm::ResolveLtx2PipelineRecipe("keyframe", "2.5")); + CHECK_THROWS((void)vllm::ResolveLtx2PipelineRecipe("keyframe_interp", "2.5")); +} diff --git a/tests/vllm/models/test_ltx2_vae.cpp b/tests/vllm/models/test_ltx2_vae.cpp index a4c39eb76..8414778ce 100644 --- a/tests/vllm/models/test_ltx2_vae.cpp +++ b/tests/vllm/models/test_ltx2_vae.cpp @@ -2717,6 +2717,77 @@ TEST_CASE("ltx2 conditioning: a KEYFRAME appends tokens at its own pixel frame") CHECK(no_fix.positions == state.positions); } +TEST_CASE("ltx2 conditioning: the keyframe causal fix is gated on frame_idx == 0, and shows") { + // THE GATE `keyframe_cond.py:49` MIRRORS — `latent_tools.causal_fix if + // self.frame_idx == 0 else False` — and the reason it needs its own case is + // that the sibling case above cannot see it. That one passes + // `num_pixel_frames = 1`, and at that value the fix is INERT AT EVERY + // `frame_idx`: `get_pixel_coords` rewrites the temporal axis to + // `max(value + 1 - time, 0)` (patchifiers.py:166-169), which leaves a + // one-latent-frame keyframe's START at 0 either way, and then the + // `num_pixel_frames == 1` narrow at `keyframe_cond.py:56-57` overwrites the END + // the fix had moved. So `no_fix.positions == state.positions` there holds + // whether the gate is wired forwards, backwards, or not at all. + // + // MEASURED, on a probe of `Ltx2ConditionVideoByKeyframe` at these shapes: + // `num_pixel_frames = 1` gives 0 of 48 differing position values at + // `frame_idx` 0 and at `frame_idx` 8 alike; `num_pixel_frames != 1` gives 4 of + // 48 at `frame_idx` 0 and 0 of 48 at `frame_idx` 8. This case is that probe. + // The production first-frame arm (`ltx2_video.cpp`) passes + // `num_pixel_frames = 1`, so its `causal_fix = true` is unobservable BY + // CONSTRUCTION and no call-site check can gate it — this is where the gate + // lives instead. + const vllm::Ltx2VideoLatentShape target = CondVideoTarget(); + const vllm::Ltx2ScaleFactors factors; + const vllm::Ltx2LatentVolume keyframe = CondVolume("ltx2.cond.keyframe", 4, 1, 2, 2); + + // `num_pixel_frames` anything but 1, so the temporal END survives to be read. + constexpr int64_t kWideFrames = 2; + + auto positions_at = [&](int64_t frame_idx, bool causal_fix) { + vllm::Ltx2LatentState state = + vllm::Ltx2CreateVideoLatentState(target, kCondPatch, factors, kCondFps, true); + vllm::Ltx2ConditionVideoByKeyframe(&state, keyframe, kCondPatch, factors, kCondFps, frame_idx, + /*strength=*/0.6, kWideFrames, causal_fix); + return state.positions; + }; + + // ── frame 0: the gate is OPEN, so the argument reaches `get_pixel_coords` ── + const std::vector at0_fix = positions_at(0, true); + const std::vector at0_no = positions_at(0, false); + REQUIRE(at0_fix.size() == at0_no.size()); + CHECK_MESSAGE(at0_fix != at0_no, + "at frame_idx 0 `causal_fix` is passed through unchanged " + "(keyframe_cond.py:49), so flipping it must move the temporal positions. Equal " + "here means the argument never reaches `get_pixel_coords` and the production " + "call sites are choosing a value nothing consumes"); + + // ...and the DIRECTION is upstream's, not merely different. The fix shortens + // the first latent frame's pixel span from `[0, time)` to `[0, 1)`, because + // the VAE's stride for the very first frame is 1 (patchifiers.py:166-169), so + // the fixed temporal END is the SMALLER of the two. A flag wired backwards + // moves the positions and fails only this half. + const size_t tokens_before = static_cast( + vllm::Ltx2VideoTokenCount(target, kCondPatch)); + const size_t first_end = tokens_before * 2 + 1; // dimension 0, first appended token, [1] + REQUIRE(at0_fix.size() > first_end); + CHECK_MESSAGE(at0_fix[first_end] < at0_no[first_end], + "the causal fix SHORTENS the first frame's temporal span " + "(patchifiers.py:166-169); fixed end " + << at0_fix[first_end] << " against unfixed " << at0_no[first_end]); + + // ── any other frame: the gate is CLOSED and the argument is discarded ────── + const std::vector at5_fix = + positions_at(vllm_test::kLtx2CondKeyframeFrameIdx, true); + const std::vector at5_no = + positions_at(vllm_test::kLtx2CondKeyframeFrameIdx, false); + CHECK_MESSAGE(at5_fix == at5_no, + "a keyframe that is not at pixel frame 0 has no first frame to correct, so " + "`keyframe_cond.py:49` forces False and the argument must change nothing. This " + "is the half the sibling case states, and at `num_pixel_frames = 1` it holds " + "vacuously"); +} + TEST_CASE("ltx2 conditioning: a REFERENCE VIDEO is translated into the target's frame") { const vllm::Ltx2VideoLatentShape target = CondVideoTarget(); const vllm::Ltx2ScaleFactors factors; diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index d9356af4c..1367fdbf5 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -1838,6 +1838,73 @@ TEST_CASE("ltx2 video: GENERATED keyframe slots are SERVED, and the marker reach CHECK(trace.slot_tokens_extracted == 0); } + SUBCASE("slots applied AFTER a supplied keyframe still describe their own tokens") { + // THE ORDERING HALF, and it had no case until + // [#1219](https://github.com/mudler/vllm.cpp/issues/1219). Upstream applies + // conditioning items in list order and DFR appends the slot item LAST + // (dfr_pipeline.py:320-330, :353-365), so a supplied keyframe stands in front + // of the slots whenever the caller pins one. Every slot assertion the engine + // makes has to locate the slot tokens, and this is the arrangement in which + // "the tokens after the target grid" and "the tokens this item appended" are + // DIFFERENT windows. + // + // A closing keyframe is the reachable way to get an append in front of the + // slots on this kind: `wants_last_frame` takes `VideoConditionByKeyframeIndex` + // and appends (keyframe_cond.py:79-82) on every pipeline, where the first + // frame only appends on the `image_conditionings_by_adding_guiding_latent` + // recipes. The two features are unrelated and compose, which is the point. + const std::string closing = ws.root + "/gk_closing.ppm"; + WriteBytes(closing, ConditioningPpm(20, 28, 51)); + + auto request = [&](const char* dir, bool slots, bool keyframe) { + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/" + dir); + if (slots) gen.extras[vllm::multimodal::kLtx2GeneratedKeyframesExtra] = "2"; + if (keyframe) { + gen.last_frame_path = closing; + gen.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "0"; + } + const vllm::multimodal::VideoResult result = engine->Generate(gen); + CHECK(result.frame_count == 9); + return ltx2->last_conditioning(); + }; + + // Four renders, so the count below is a RELATION between measured lengths + // rather than a literal: each item's own cost is measured on its own, and + // the claim is that together they cost the sum. `image_tokens` cannot serve + // here — the engine fills it on the FIRST-frame arm only, and this closing + // keyframe leaves it 0. + const vllm::multimodal::Ltx2ConditioningTrace bare = request("gk_bare2", false, false); + const vllm::multimodal::Ltx2ConditioningTrace slots_only = request("gk_alone", true, false); + const vllm::multimodal::Ltx2ConditioningTrace kf_only = request("gk_kf_only", false, true); + const vllm::multimodal::Ltx2ConditioningTrace both = request("gk_after_kf", true, true); + + REQUIRE(slots_only.slot_positions.size() == 2); + REQUIRE(slots_only.slot_marked_tokens > 0); + REQUIRE(slots_only.video_tokens > bare.video_tokens); + REQUIRE(kf_only.video_tokens > bare.video_tokens); + + MESSAGE("video_tokens bare " << bare.video_tokens << " slots " << slots_only.video_tokens + << " keyframe " << kf_only.video_tokens << " both " + << both.video_tokens); + + // The slots are placed and MARKED exactly as they are alone: a keyframe in + // front changes WHERE they sit in the sequence and nothing about what they + // are. This is the assertion the engine's own slot checks stand behind, and + // the arrangement that makes "past the target grid" and "what this item + // appended" two different windows. + CHECK(both.slot_positions == slots_only.slot_positions); + CHECK_MESSAGE(both.slot_marked_tokens == slots_only.slot_marked_tokens, + "a supplied keyframe in front of the slots changed how many slot tokens " + "carry the trained marker — expected " << slots_only.slot_marked_tokens + << ", got " << both.slot_marked_tokens); + CHECK_MESSAGE(both.video_tokens == slots_only.video_tokens + + (kf_only.video_tokens - bare.video_tokens), + "the two appending items must cost what each costs alone: " + << slots_only.video_tokens << " + " + << kf_only.video_tokens - bare.video_tokens << " against " + << both.video_tokens); + } + SUBCASE("a negative count gets upstream's OWN reason") { // `evenly_spaced_keyframe_positions` raises "num_keyframes must be // non-negative" (utils/helpers.py:372-373) before anything looks at the @@ -8025,3 +8092,581 @@ TEST_CASE("ltx2 ti2vid: the distilled-LoRA requirement refuses BY WHAT IS MISSIN vllm::multimodal::LoadVideoEngine(Ti2VidParams(ws.paths, lora)); CHECK_NOTHROW((void)engine->Generate(Ti2VidGen(ws.root + "/no_take"))); } + +// ─── LTX25-KEYFRAME-INTERP (#1096) ─────────────────────────────────────────── + +namespace { + +// A `keyframe_interpolation` engine on the shipped fixture. The same two +// load-side requirements `ti2vid_two_stage` has: the spatial upsampler stage 2 +// needs, which `ConditioningParams` supplies, and the distilled adapter, which +// upstream makes a POSITIONAL non-defaulted parameter +// (keyframe_interpolation.py:68) as well as `--distilled-lora required=True` +// (utils/args.py:1140-1155). +// +// NO `audio_path`: this pipeline generates its soundtrack, and unlike +// `ti2vid_two_stage` the take that leaves is STAGE 2's (`:271`, `:293`). +vllm::multimodal::VideoModelParams KeyframeParams(const ltx2_fixture::Paths& paths, + const std::string& lora) { + vllm::multimodal::VideoModelParams mp = ConditioningParams(paths); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = "keyframe_interpolation"; + mp.extras[vllm::multimodal::kLtx2LoraPathExtra] = lora; + return mp; +} + +// `steps = 2`, because stage 1's schedule is DERIVED from the step count +// (keyframe_interpolation.py:199-200) and two sigma intervals exercise the loop. +// The case that gates the schedule ANCHOR raises this to 3 and says why. The STG +// block list is the fixture's, not this row's: the reduced DiT has TWO blocks +// and the params row this recipe resolves names block 28. +vllm::multimodal::VideoGenParams KeyframeGen(const std::string& out_dir, int64_t size = 64) { + vllm::multimodal::VideoGenParams gen = FixtureGen(out_dir); + gen.steps = 2; + gen.height = size; + gen.width = size; + OneStageFixtureGuidance(&gen); + return gen; +} + +} // namespace + +TEST_CASE("ltx2 keyframe: the first frame is a KEYFRAME that APPENDS, not a latent that replaces") { + // THE ROW'S IDENTITY, and the one thing that separates + // `image_conditionings_by_adding_guiding_latent` (helpers.py:343-367) from + // `combined_image_conditionings` (:272-308): the second sends `frame_idx == 0` + // to `VideoConditionByLatentIndex`, which REPLACES latent frame 0's clean + // tokens and never changes the token count (latent_cond.py:38-39); the first + // has NO branch and sends it to `VideoConditionByKeyframeIndex`, which APPENDS + // a latent frame of tokens (keyframe_cond.py:79-82). + // + // GATED THROUGH `LoadVideoEngine` + `Generate`, not on the recipe struct: the + // recipe case in test_ltx2_pipeline proves the field is SET, and this one + // proves it is CONSUMED (#1013). + // + // AND NO PIXEL COMPARISON CAN SEE IT. Both arms condition on the same image, + // both return a clip of the right size, the right frame count and the right + // sample rate. The only observable is the sequence LENGTH the DiT ran over, + // which is `Ltx2ConditioningTrace::video_tokens`, recorded before the trim. + Workspace ws; + const std::string lora = + WriteFixtureLora(ws.root + "/distilled.safetensors", kFixtureLoraTarget, 1.0F); + + struct Conditioned { + int64_t video_tokens; + int64_t image_tokens; + std::string bytes; + }; + + // ONE image, ONE geometry, ONE seed. The only thing that differs between the + // two loads is `pipeline_kind`, so nothing but the builder can move the + // numbers below. + auto render = [&](const char* kind, bool with_image, const std::string& tag) -> Conditioned { + vllm::multimodal::VideoModelParams mp = KeyframeParams(ws.paths, lora); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = kind; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + vllm::multimodal::VideoGenParams gen = KeyframeGen(ws.root + "/" + tag); + if (with_image) { + gen.first_frame_ppm = ConditioningPpm(20, 28, 1); + // CRF 0 EXPLICITLY, because an LTX-2.5 checkpoint RESOLVES 18 when the + // caller leaves it unset (`ImageConditioner.resolve_crf`, blocks.py:977-983) + // and the H.264 round trip that 18 needs is refused by name in this tree. + // Every image-conditioning case here asks for 0 for the same reason; it is + // orthogonal to which builder places the image. + gen.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "0"; + } + const vllm::multimodal::VideoResult result = engine->Generate(gen); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + std::string all; + for (int64_t f = 0; f < result.frame_count; ++f) { + char name[64]; + std::snprintf(name, sizeof(name), "/frame_%06lld.ppm", static_cast(f)); + all += ReadAll(gen.output_dir + name); + } + return Conditioned{t.video_tokens, t.image_tokens, all}; + }; + + const Conditioned kf = render("keyframe_interpolation", /*with_image=*/true, "kf"); + const Conditioned ti = render("ti2vid_two_stage", /*with_image=*/true, "ti"); + // THE THIRD RENDER, and without it the equality below passes on a tree where + // BOTH arms append: it pins what the bare target grid is on this geometry, so + // "the plain arm placed its image without growing the sequence" is a claim + // about the grid rather than a comparison of two unknowns. + const Conditioned bare = render("ti2vid_two_stage", /*with_image=*/false, "bare"); + const Conditioned kf_bare = render("keyframe_interpolation", /*with_image=*/false, "kf_bare"); + + MESSAGE("video_tokens keyframe " << kf.video_tokens << " ti2vid " << ti.video_tokens + << " bare " << bare.video_tokens); + + // NEITHER ARM PLACED NOTHING. A build that encoded the image and never + // conditioned on it would satisfy every count comparison below by making them + // all equal, so this is a REQUIRE. + REQUIRE(kf.image_tokens > 0); + REQUIRE(ti.image_tokens > 0); + // The same image at the same geometry encodes to the same number of tokens + // under both builders — only WHERE they go differs. + CHECK(kf.image_tokens == ti.image_tokens); + + // ── the plain arm REPLACED: its sequence is the bare target grid ─────────── + CHECK_MESSAGE(ti.video_tokens == bare.video_tokens, + "`combined_image_conditionings` sends frame 0 to " + "`VideoConditionByLatentIndex`, which replaces tokens that already exist " + "(latent_cond.py:38-39), so the count must not move — it went from " + << bare.video_tokens << " to " << ti.video_tokens); + CHECK(kf_bare.video_tokens == bare.video_tokens); + + // ── and this arm APPENDED, by exactly one image's worth of tokens ────────── + CHECK_MESSAGE(kf.video_tokens == bare.video_tokens + kf.image_tokens, + "`image_conditionings_by_adding_guiding_latent` has no frame-0 branch " + "(helpers.py:343-367), so frame 0 takes `VideoConditionByKeyframeIndex` and " + "APPENDS (keyframe_cond.py:79-82). Expected " + << bare.video_tokens + kf.image_tokens << " tokens, got " << kf.video_tokens); + CHECK(kf.video_tokens > ti.video_tokens); + + // ── and the render CHANGED, which is what a token count alone cannot say ─── + // + // Against the SAME kind with no image, deliberately, and not against the plain + // arm: the two arms place the same content, so a pixel difference between THEM + // says only that they placed it differently, which the counts above already + // say with a reason. What this adds is that the conditioning reached the + // pixels at all on this arm. + CHECK_MESSAGE(kf.bytes != kf_bare.bytes, + "the appended keyframe left the rendered clip unchanged, so it was placed into " + "a state the decode never read"); + CHECK_FALSE(kf.bytes.empty()); +} + +TEST_CASE("ltx2 keyframe: BOTH ends pin at once - two appends, and each is located at its own") { + // THE PIPELINE'S HEADLINE REQUEST, and it aborted the render + // ([#1219](https://github.com/mudler/vllm.cpp/issues/1219)). + // `KeyframeInterpolationPipeline` exists to generate the motion BETWEEN + // pinned keyframes; `docs/USAGE.md`'s worked example for this kind passes + // `--first-frame open.ppm --last-frame close.ppm`, and `ltx2-gen --help` says + // to use the two together. So this is not an edge case, it is the case. + // + // WHAT BROKE, and it is a shape worth keeping in view: the last-frame arm + // located its own appended tokens at `positions[target_tokens * 2]`, which is + // the first token PAST THE TARGET GRID — correct only while that arm owned the + // first append. Row LTX25-KEYFRAME-INTERP put a second appending item in front + // of it on the `image_conditionings_by_adding_guiding_latent` recipes, so the + // index then named the FIRST frame's keyframe at temporal 0, and the arm's own + // positional assertion threw. A derived index that was right by ORDER rather + // than by construction, and the item that changed the order was two hundred + // lines above it. + // + // THE CONTROL IS THE OTHER BUILDER, and it is what makes this case say + // "ordering" rather than "keyframes". `ti2vid_two_stage` takes the same two + // images through `combined_image_conditionings`, whose frame-0 item REPLACES + // (helpers.py:295-300), so it has exactly ONE append and never reached the + // defect. It stayed green throughout and must stay green here. + Workspace ws; + const std::string lora = + WriteFixtureLora(ws.root + "/both.safetensors", kFixtureLoraTarget, 1.0F); + + const std::string closing = ws.root + "/closing.ppm"; + WriteBytes(closing, ConditioningPpm(20, 28, 41)); + + struct Pinned { + int64_t video_tokens; + int64_t image_tokens; + }; + + // `ends` selects which of the two slots are filled, so one lambda produces the + // bare grid, each single end, and both — and the counts below are then + // relations between renders of one geometry rather than literals. + auto render = [&](const char* kind, bool first, bool last, const std::string& tag) -> Pinned { + vllm::multimodal::VideoModelParams mp = KeyframeParams(ws.paths, lora); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = kind; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + vllm::multimodal::VideoGenParams gen = KeyframeGen(ws.root + "/" + tag); + if (first) gen.first_frame_ppm = ConditioningPpm(20, 28, 40); + if (last) gen.last_frame_path = closing; + // One CRF for both slots, which is the surface: upstream resolves the CRF + // once for the whole `images` list (blocks.py:966-983), and the H.264 round + // trip an LTX-2.5 checkpoint would otherwise resolve is refused by name here. + if (first || last) gen.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "0"; + const vllm::multimodal::VideoResult result = engine->Generate(gen); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + // The artifact, not only the trace: the render has to come back at the + // requested length, because the appended tokens are trimmed back off before + // the unpatchify and a clip of the wrong length is the visible half of a + // trim that read the wrong count. + CHECK(result.frame_count == 9); + return Pinned{t.video_tokens, t.image_tokens}; + }; + + // ── THE CONTROL, FIRST, so a red below is attributable ───────────────────── + // + // Replace + one append. This is the arm that was already green, and running it + // first means a failure here says "both arms are broken" rather than letting + // the new arm's red stand for the pair. + const Pinned ti_both = render("ti2vid_two_stage", true, true, "ti_both"); + const Pinned ti_bare = render("ti2vid_two_stage", false, false, "ti_bare"); + REQUIRE(ti_both.image_tokens > 0); + CHECK_MESSAGE(ti_both.video_tokens == ti_bare.video_tokens + ti_both.image_tokens, + "`combined_image_conditionings` REPLACES at frame 0 and APPENDS at the closing " + "frame, so pinning both ends grows the sequence by exactly ONE image — expected " + << ti_bare.video_tokens + ti_both.image_tokens << ", got " + << ti_both.video_tokens); + + // ── AND THE ARM THIS ROW ADDS: two appends, both located ─────────────────── + const Pinned kf_both = render("keyframe_interpolation", true, true, "kf_both"); + const Pinned kf_bare = render("keyframe_interpolation", false, false, "kf_bare"); + const Pinned kf_first = render("keyframe_interpolation", true, false, "kf_first"); + const Pinned kf_last = render("keyframe_interpolation", false, true, "kf_last"); + + MESSAGE("video_tokens bare " << kf_bare.video_tokens << " first " << kf_first.video_tokens + << " last " << kf_last.video_tokens << " both " + << kf_both.video_tokens << " image_tokens " + << kf_both.image_tokens); + + REQUIRE(kf_both.image_tokens > 0); + CHECK(kf_bare.video_tokens == ti_bare.video_tokens); + // Each end alone appends one image's worth... + CHECK(kf_first.video_tokens == kf_bare.video_tokens + kf_both.image_tokens); + CHECK(kf_last.video_tokens == kf_bare.video_tokens + kf_both.image_tokens); + // ...and the two together append BOTH, which is the count the defect could + // never produce because the render never finished. + CHECK_MESSAGE(kf_both.video_tokens == kf_bare.video_tokens + 2 * kf_both.image_tokens, + "`image_conditionings_by_adding_guiding_latent` has no frame-0 branch " + "(helpers.py:343-367), so BOTH pinned ends take `VideoConditionByKeyframeIndex` " + "and append (keyframe_cond.py:79-82) — expected " + << kf_bare.video_tokens + 2 * kf_both.image_tokens << ", got " + << kf_both.video_tokens); + // ...and it is strictly more than the builder that replaces at frame 0, which + // is what the two arms differ by. + CHECK(kf_both.video_tokens > ti_both.video_tokens); +} + +TEST_CASE("ltx2 keyframe: the pipeline renders through vllm.h, guided on the UNADAPTED stage 1") { + // THE REACHABILITY CLAIM, and it is the point of this case rather than a note + // beside it. Entry point: `LoadVideoEngine` with a documented value of the + // documented `pipeline_kind` LOAD extra plus `lora_path`, then `Generate`. + // Nothing here constructs a recipe, a guider, a phase or a modality by hand. + // Deleting the `keyframe_interpolation` dispatch row in + // `ResolveLtx2PipelineRecipe` REDs this case at the LOAD, which is what + // separates measuring a capability from measuring a class + // (.agents/reachability.md). + // + // `ltx2-gen --pipeline-kind keyframe_interpolation --lora-path ... + // --upsampler-path ...` is the same two calls through the ABI, as a thin + // client that includes no internal header. + // + // AND #928 DOES NOT EXCLUDE THE HTTP ROUTE HERE, unlike on `a2vid_two_stage`. + // That recipe needs `audio_path`, a PER-GENERATION extra, and + // `VideoGenParamsFromRequest` writes none. All three knobs THIS recipe needs — + // `pipeline_kind`, `lora_path`, `upsampler_path` — are LOAD extras, which a + // server supplies through `--video-extra KEY=VALUE`, and `requires_audio_input` + // is false, which the recipe case gates. That is a statement about the REQUEST + // SURFACE and not a second reach claim: no case here drives the HTTP route end + // to end, so it is not measured and is not asserted. + Workspace ws; + const std::string lora = + WriteFixtureLora(ws.root + "/distilled.safetensors", kFixtureLoraTarget, 1.0F); + + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(KeyframeParams(ws.paths, lora)); + REQUIRE(engine != nullptr); + auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + CHECK(ltx->pipeline_kind() == "keyframe_interpolation"); + + const vllm::multimodal::VideoResult result = engine->Generate(KeyframeGen(ws.root + "/kf")); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + CHECK(result.frame_count == 9); + // The soundtrack is GENERATED and decoded through the vocoder, so it comes + // back at the BWE arm's rate rather than at the audio VAE's. + CHECK(result.sample_rate > 0); + + // ── stage 1 ran upstream's GUIDED denoiser, on all four arms ─────────────── + // + // AN EVALUATION COUNT CANNOT SEE THIS. A denoiser call is ONE evaluation + // whether guidance ran or not, so only the FORWARD counters — actual + // `Ltx2DitForward` calls — distinguish a guided stage 1 from a + // `SimpleDenoiser` one. The trace's guided fields are recorded at step 0 of + // phase 0, which is this recipe's stage 1, and its guider is the params + // table's video row (cfg 3.0, stg 1.0, rescale 0.7, modality 3.0), so all four + // passes run and the rescale branch — the one term that is NOT invariant + // between x0 and velocity space — is live on the DEFAULT path. + REQUIRE_MESSAGE(t.video_guided, "stage 1 did not go through the guided seam at all"); + CHECK(t.video_cond_forwards == 1); + CHECK(t.video_uncond_forwards == 1); + CHECK(t.video_perturbed_forwards == 1); + CHECK(t.video_modality_forwards == 1); + CHECK(t.video_guidance_cfg_scale == 3.0); + CHECK(t.video_guidance_stg_scale == 1.0); + CHECK(t.video_guidance_rescale_scale == 0.7); + CHECK(t.video_guidance_modality_scale == 3.0); + CHECK(t.dit_forwards > t.dit_evaluations); + + // ── the four arms are combined in X0 SPACE, not velocity space ──────────── + // + // `_guided_denoise` converts each pass to x0 BEFORE it combines them + // (ltx-core utils.py:39-52, `sample - velocity * sigma`). Every LINEAR term is + // invariant under that change of variable, so cfg, stg and modality cannot see + // the difference; the RESCALE branch is not invariant, and `rescale_scale` + // defaults to 0.7 here, so a space error lands on the default path + // (#1039, #1092). + // + // A MAGNITUDE ASSERTION CANNOT GATE IT. On a reduced fixture + // `std(cond)/std(pred)` is 1.0 to 1e-5 in BOTH spaces. `x0 == latent - + // sigma*velocity` is an equation between three RECORDED tensors instead: exact + // in x0 space, and in velocity space `x0` IS the velocity, so the residual + // becomes the whole sample and `|x0 - velocity|` collapses to exactly 0 — + // which is what the RED prints. + const size_t n = t.video_first_latent.size(); + REQUIRE(n > 0); + const size_t tokens = t.video_first_timesteps.size(); + REQUIRE(tokens > 0); + const size_t width = n / tokens; + REQUIRE(width * tokens == n); + // THE FIXTURE CAN DECIDE THIS AT ALL: the two candidate tensors coincide when + // the sample is zero. A REQUIRE, because nothing below discriminates once it + // fails. + double latent_span = 0.0; + for (const float x : t.video_first_latent) { + latent_span = std::max(latent_span, std::abs(static_cast(x))); + } + REQUIRE_MESSAGE(latent_span > 1e-3, "the step-0 sample is zero, so the two candidate tensors " + "coincide and nothing below discriminates"); + + struct KeyframeArm { + const char* name; + const std::vector& velocity; + const std::vector& x0; + }; + const KeyframeArm arms[] = { + {"cond", t.video_first_cond_velocity, t.video_first_cond}, + {"uncond", t.video_first_uncond_velocity, t.video_first_uncond}, + {"perturbed", t.video_first_perturbed_velocity, t.video_first_perturbed}, + {"modality", t.video_first_modality_velocity, t.video_first_modality}, + }; + for (const KeyframeArm& arm : arms) { + INFO("arm = " << std::string(arm.name)); + REQUIRE(arm.velocity.size() == n); + REQUIRE(arm.x0.size() == n); + // PER ARM, because a zeroed velocity makes `to_denoised` the identity on + // THIS arm alone and would satisfy the equation while proving nothing. + double velocity_span = 0.0; + for (const float x : arm.velocity) { + velocity_span = std::max(velocity_span, std::abs(static_cast(x))); + } + REQUIRE_MESSAGE(velocity_span > 1e-6, "this arm's velocity is zero, so the equation below " + "holds for a reason that is not the one it tests"); + // `x0 = latent - sigma_token * velocity` (model.py:590-604), with the + // PER-TOKEN timestep and not the schedule scalar. + double residual = 0.0; + double against_velocity = 0.0; + for (size_t i = 0; i < n; ++i) { + const double sigma = static_cast(t.video_first_timesteps[i / width]); + const double expected = static_cast(t.video_first_latent[i]) - + sigma * static_cast(arm.velocity[i]); + residual = std::max(residual, std::abs(static_cast(arm.x0[i]) - expected)); + against_velocity = std::max( + against_velocity, + std::abs(static_cast(arm.x0[i]) - static_cast(arm.velocity[i]))); + } + INFO("max|x0 - (latent - sigma*v)| = " << residual); + INFO("max|x0 - velocity| = " << against_velocity); + CHECK(residual < 1e-4); + CHECK(against_velocity > 1e-6); + } +} + +TEST_CASE("ltx2 keyframe: stage 1's sigma shift takes the 4096 anchor, not the target grid") { + // `LTX2Scheduler.execute` takes an OPTIONAL latent and `schedulers.py:31` is + // `tokens = math.prod(latent.shape[2:]) if latent is not None else + // default_number_of_tokens`, with `default_number_of_tokens` = MAX_SHIFT_ANCHOR + // = 4096 (`:11`, `:29`). `keyframe_interpolation.py:199-200` passes NO latent; + // `ti2vid_two_stages_hq.py:267` — our `res2s_two_stage` — passes + // `latent=empty_latent` and is the ONE upstream site that does. + // + // A 2x2 OVER (recipe, geometry), because neither half alone is load-bearing. + // The equalities alone pass on a build that hard-codes 4096 for everything; + // the inequalities alone pass on a `target_tokens`-everywhere tree. Only the + // pair says that the anchor is per-phase AND selected correctly. + Workspace ws; + const std::string lora = + WriteFixtureLora(ws.root + "/distilled.safetensors", kFixtureLoraTarget, 1.0F); + + // Two geometries whose stage-1 target grids differ. Both divide 64, which is + // `Ltx2AssertResolution`'s divisor on a `spatial_downscale = 2` recipe. + const int64_t kSmall = 64; + const int64_t kLarge = 128; + + // TWO NUMBERS OUT OF EACH RENDER, AND THE SECOND ONE IS WHAT KEEPS THE + // TRAJECTORY HALF OF THIS CASE HONEST. The recomputations at the end run at + // the step count the RENDER used. Restating that count as a literal down there + // decouples it from `gen.steps`: lowering the render to 2 steps would then + // leave every assertion in this case green while making the trajectory claim + // vacuous. That is not hypothetical — `ltx25-ti2vid-recipe.md` shipped this + // case with a literal and a fresh review found the mutation undetected. + struct Rendered { + int64_t schedule_tokens; + int64_t steps; + }; + + auto rendered_for = [&](const char* kind, int64_t size, const std::string& tag) -> Rendered { + vllm::multimodal::VideoModelParams mp = KeyframeParams(ws.paths, lora); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = kind; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + vllm::multimodal::VideoGenParams gen = KeyframeGen(ws.root + "/" + tag, size); + // THREE STEPS, NOT THE FIXTURE'S TWO, and the reason is asserted below: the + // scheduler's `stretch` pins sigma[0] at 1.0 and the LAST non-zero sigma at + // `terminal` = 0.1 (schedulers.py:48-55), so a 2-step schedule is + // {1, 0.1, 0} for EVERY token count and the anchor cannot reach the + // trajectory at all. Three steps is the shortest schedule with an interior + // sigma for the shift to move. Lowering this number is REFUSED rather than + // deprecated: it comes back out of this lambda and the trajectory assertions + // recompute at it, so 2 here reds them by name. + gen.steps = 3; + // `keyframe_interpolation` resolves `stg_blocks = [28]` (constants.py:86-87) + // and this fixture's DiT has TWO blocks, so its perturbed pass is refused by + // name unless the request names a block that exists. The HQ preset ships + // `stg_blocks = []` beside `stg_scale = 0.0` (constants.py:105, :113), asks + // for no perturbed pass at all, and FIXES its stage-2 guidance — so giving + // it the same override is refused outright. `KeyframeGen` carries the + // override, so the control has to take it back off. + if (std::string(kind) != "keyframe_interpolation") { + gen.extras.erase(vllm::multimodal::kLtx2VideoStgBlocksExtra); + gen.extras.erase(vllm::multimodal::kLtx2AudioStgBlocksExtra); + } + (void)engine->Generate(gen); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + // Written only on the branch that CALLS `Ltx2SigmaSchedule`, so a phase + // carrying frozen sigmas leaves it 0. Stage 2 of both recipes does, which is + // why what lands here is stage 1's. + REQUIRE(t.schedule_tokens > 0); + return Rendered{t.schedule_tokens, gen.steps}; + }; + + const Rendered kf_small_r = rendered_for("keyframe_interpolation", kSmall, "kf_small"); + const Rendered kf_large_r = rendered_for("keyframe_interpolation", kLarge, "kf_large"); + const Rendered hq_small_r = rendered_for("res2s_two_stage", kSmall, "hq_small"); + const Rendered hq_large_r = rendered_for("res2s_two_stage", kLarge, "hq_large"); + + const int64_t kf_small = kf_small_r.schedule_tokens; + const int64_t kf_large = kf_large_r.schedule_tokens; + const int64_t hq_small = hq_small_r.schedule_tokens; + const int64_t hq_large = hq_large_r.schedule_tokens; + + // The step count the four renders ACTUALLY ran at, read back out of them + // rather than restated below. All four have to agree, or "the step count" is + // not one number and nothing below can be recomputed at it. + const int64_t rendered_steps = kf_small_r.steps; + REQUIRE(kf_large_r.steps == rendered_steps); + REQUIRE(hq_small_r.steps == rendered_steps); + REQUIRE(hq_large_r.steps == rendered_steps); + + MESSAGE("keyframe: " << kf_small << " / " << kf_large << " res2s: " << hq_small << " / " + << hq_large); + + // ── this arm's schedule is RESOLUTION-INDEPENDENT, at upstream's constant ── + const int64_t anchor = vllm::Ltx2SchedulerParams{}.default_number_of_tokens; + CHECK(anchor == 4096); // schedulers.py:11 — pinned, not read back from the build + CHECK_MESSAGE(kf_small == anchor, + "stage 1's sigma shift was fitted on " << kf_small << " tokens, but upstream " + "passes no latent (keyframe_interpolation.py:199-200) and gets " << anchor); + CHECK(kf_large == anchor); + + // ── and the HQ arm's is NOT, which is what stops the above being a constant ─ + CHECK_MESSAGE(hq_small != hq_large, + "the res_2s arm reported the same anchor at two resolutions, so this fixture " + "cannot tell the two branches apart and the equalities above prove nothing"); + CHECK(hq_small != anchor); + CHECK(hq_large != anchor); + + // ── and the two anchors really do produce DIFFERENT sigmas ──────────────── + // + // The strongest half: a claim about the trajectory rather than about the + // counter that reports it. `sigma_shift = tokens*mm + b` (schedulers.py:35-39), + // so two token counts give two schedules — unless the shift arithmetic has + // been flattened, in which case selecting the anchor would be inert and every + // assertion above would still pass. + // AT `rendered_steps`, NOT AT A LITERAL. + const std::vector at_anchor = vllm::Ltx2SigmaSchedule(rendered_steps, anchor); + const std::vector at_target = vllm::Ltx2SigmaSchedule(rendered_steps, hq_small); + REQUIRE(at_anchor.size() == at_target.size()); + CHECK_MESSAGE(at_anchor != at_target, + "the 4096 anchor and the target grid produce the SAME schedule on this fixture, " + "so nothing above measures which one was taken"); + + // AND THE RENDER'S STEP COUNT IS LOAD-BEARING, WHICH IS WORTH AN ASSERTION + // RATHER THAN A COMMENT. TWO assertions, because they fail for different + // reasons. The first names the render's own step count, so lowering + // `gen.steps` to make this case faster fails HERE, by name, rather than + // silently turning the comparison above into a value against itself. The + // second pins the degeneracy itself, so it fails if a scheduler change ever + // makes a 2-step schedule token-dependent. + CHECK_MESSAGE(rendered_steps > 2, + "the renders above ran at " << rendered_steps << " steps, and a schedule that " + "short is {1, 0.1, 0} for EVERY token count (schedulers.py:48-55), so the " + "trajectory comparison above compares a value with itself"); + CHECK_MESSAGE(vllm::Ltx2SigmaSchedule(/*steps=*/2, anchor) == + vllm::Ltx2SigmaSchedule(/*steps=*/2, hq_small), + "a 2-step schedule now DOES depend on the token count, so the stretch no longer " + "pins both of its non-zero sigmas and the comment above is wrong"); + CHECK(vllm::Ltx2SigmaSchedule(/*steps=*/2, anchor).size() == 3u); +} + +TEST_CASE("ltx2 keyframe: the distilled-LoRA requirement refuses BY WHAT IS MISSING") { + // `distilled_lora` is a POSITIONAL, non-defaulted parameter + // (keyframe_interpolation.py:68) and `--distilled-lora` is `required=True` + // (utils/args.py:1140-1155) on the parser `:301` selects. Stage 2's three-sigma + // refinement (`:166`) is what that adapter was trained for. Without it the + // render FINISHES: a clip of the right size, frame count and sample rate, with + // a distilled schedule run on undistilled weights. + Workspace ws; + const std::string lora = + WriteFixtureLora(ws.root + "/distilled.safetensors", kFixtureLoraTarget, 1.0F); + + vllm::multimodal::VideoModelParams mp = KeyframeParams(ws.paths, lora); + mp.extras.erase(vllm::multimodal::kLtx2LoraPathExtra); + try { + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + FAIL_CHECK("a keyframe_interpolation load with no distilled LoRA must be refused"); + } catch (const std::exception& e) { + const std::string message = e.what(); + INFO("message = " << message); + // It names the PIPELINE that was asked for, which is what makes the refusal + // actionable rather than generic. This recipe is the THIRD user of + // `requires_distilled_lora`, and #1151 is the record of what the message + // said when it had only one: the previous pipeline's own line numbers. + CHECK(message.find("keyframe_interpolation") != std::string::npos); + CHECK(message.find("distilled LoRA") != std::string::npos); + CHECK(message.find("lora_path") != std::string::npos); + CHECK(message.find("default_2_stage_arg_parser") != std::string::npos); + CHECK(message.find("a2vid_two_stage.py") == std::string::npos); + CHECK(message.find("ti2vid_two_stages.py") == std::string::npos); + } + + // THE CONTROL: the same load WITH the adapter renders, so the case is about + // the requirement and not about any load failure. + CHECK_NOTHROW((void)vllm::multimodal::LoadVideoEngine(KeyframeParams(ws.paths, lora))); + // THE SECOND CONTROL: the DEFAULT kind is fine without an adapter, so this is + // this recipe's requirement and not a new global one. + CHECK_NOTHROW((void)vllm::multimodal::LoadVideoEngine(ConditioningParams(ws.paths))); + + // ── and NO audio take is demanded, which `a2vid_two_stage` does ──────────── + // + // The two recipes come off the same parser and share `requires_distilled_lora`, + // so a recipe written by copying that one would inherit `requires_audio_input` + // and refuse every render. There is no `--audio-path` on this pipeline: + // keyframe_interpolation.py:147-168 takes `images`, not a waveform. + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(KeyframeParams(ws.paths, lora)); + CHECK_NOTHROW((void)engine->Generate(KeyframeGen(ws.root + "/no_take"))); +}