feat(LTX25-KEYFRAME-INTERP): the interpolation pipeline, on the conditioning builder that has no frame-0 branch (#1096)

.agents/specs/ltx-2-5.md

@@ -971,13 +971,20 @@ each saying what is absent, what a future row starts from, and what blocks it.
   negative RoPE shift (`dubit.py:351-353`), which our one ported shift structurally cannot
   produce because it clamps at zero (`ltx2_conditioning.cpp:596-601`), and on the Dub-It
   IC-LoRA.
-- [#1096](https://github.com/mudler/vllm.cpp/issues/1096) —
-  `KeyframeInterpolationPipeline` (`keyframe_interpolation.py`). `Ltx2ConditionVideoByKeyframe`
-  IS served and mutation-proven reachable (`ltx25-token-append.md:270`). Blocked on a
-  multi-keyframe request surface — the ABI carries two scalar slots
-  (`include/vllm.h:934-935`) against upstream's repeatable `--image PATH FRAME_IDX STRENGTH`
-  — and on a per-sigma guided denoiser, ours being fixed per phase
-  (`ltx2_pipeline.cpp:1069-1070`) and audio-only.
+- ~~[#1096](https://github.com/mudler/vllm.cpp/issues/1096) —
+  `KeyframeInterpolationPipeline` (`keyframe_interpolation.py`).~~ LANDED as row
+  `LTX25-KEYFRAME-INTERP` ([`ltx25-keyframe-interp.md`](ltx25-keyframe-interp.md)),
+  and two of the three blockers recorded here were stale by the time it was picked
+  up. The per-sigma denoiser resolves ONE guider on this pipeline's default path —
+  `main()` passes plain `MultiModalGuiderParams`, so
+  `create_multimodal_guider_factory` takes `constant()` and builds a single
+  `(inf, params)` bin (`guiders.py:312-315`) — and both checkpoints are on the NAS
+  with #1148 closed at `40a796aa9`. The multi-keyframe surface is real, is NOT
+  what makes this pipeline different, and is now
+  [#1187](https://github.com/mudler/vllm.cpp/issues/1187). What WAS different, and
+  is named in none of the above, is the conditioning BUILDER: `:211` and `:260`
+  call `image_conditionings_by_adding_guiding_latent` (`helpers.py:343-367`), so
+  frame 0 is a keyframe that APPENDS rather than a latent that REPLACES.
 - [#1097](https://github.com/mudler/vllm.cpp/issues/1097) — `ltx2-gen` silently discards a
   second `--lora`, and `kKnownLoadExtras`' own comment still says "nine of these ten" over a
   twelve-entry array. Product code, so filed rather than fixed in this records change.

.agents/specs/ltx25-ti2vid-recipe.md

@@ -164,7 +164,7 @@ inverts how #1093 and `ltx25-res2s-loop.md:80-88` both framed it.
 | `retake.py:287` | no | `retake`, non-distilled arm | **no** |
 | `a2vid_two_stage.py:226` | no | `a2vid_two_stage` stage 1 | **no** |
 | `ti2vid_two_stages.py:244` | no | **this row** | this row makes it yes |
-| `keyframe_interpolation.py:200` | no | unported (#1096) | n/a |
+| `keyframe_interpolation.py:200` | no | `keyframe_interpolation` stage 1 | yes, since row LTX25-KEYFRAME-INTERP (#1096) |
 | `ti2vid_two_stages_hq.py:267` | **yes** | `res2s_two_stage` stage 1 | yes |
 
 So the engine mirrors the exception and diverges from the rule. That is #1150,
@@ -538,7 +538,13 @@ RED before the recipe landed, captured on the same binaries: `test_ltx2_pipeline
   (`:151`'s `*tuple(loras)`). This engine's one adapter slot is upstream's
   `distilled_lora`; the second list has no spelling here until the adapter arity
   refusal lifts (`ltx2_lora.h:167-172`).
-- **`keyframe_interpolation`** (#1096), the fourth pipeline on this parser.
+- ~~**`keyframe_interpolation`** (#1096), the fourth pipeline on this parser.~~
+  Landed as row `LTX25-KEYFRAME-INTERP`, which took this row as its template and
+  set `schedule_tokens = kSchedulerDefault` on its own stage 1. That moved the
+  keyframe line of the table above from `n/a` to `yes` and left the count owed to
+  #1150 at **three**, where it already stood: the six-to-one split is the
+  `latent` column, which no port can change, and the keyframe arm was never one
+  of the divergent three because it was unported rather than wrong.
 - **`allow_request_latents` has no reader in `src/`**, on any recipe, owned by
   [#1152](https://github.com/mudler/vllm.cpp/issues/1152). Four assignments on
   `origin/main` and five with this row's, zero readers in either count, against

docs/FEATURES.md

@@ -181,6 +181,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change.
 | LTX-2.5 VIDEO guidance | LTX-2.5 DiT, joint video+audio | `test_ltx2_video`'s `ltx2 one_stage:` cases through `Generate`; all FOUR arms carry the x0 invariant and the guider output replays EXACTLY | `--pipeline-kind one_stage` runs `_guided_denoise`: 4 forwards/step, combined per modality in **x0**. Was ONE unguided forward, every `video_guidance` field dead ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) |
 | LTX-2.5 cross-attention perturbations | LTX-2.5 DiT | `test_ltx2_video` gates each direction ALONE, on a forward where one stream is PRESENT but DISABLED so only that one runs: the flag moves the stream it writes, the other leaves it bit-identical. Swapping the two is RED | `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN` ported, which `modality_scale = 3.0` selects on every video row. The DEVICE forward takes no perturbations, so that pass is refused there by name |
 | LTX-2.5 audio-to-video (`A2VidPipelineTwoStage`) | LTX-2.5 DiT + audio VAE encoder + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video`'s `ltx2 a2vid:` cases through `LoadVideoEngine`+`Generate`; the take's latent is bit-identical across SEEDS and moves with the WINDOW | `--pipeline-kind a2vid_two_stage`. Guided half-res stage 1, DERIVED schedule, plain Euler; distilled 3-sigma stage 2. `--audio-path` and `--lora` REQUIRED; the distilled adapter rides stage 2 ALONE (#1118) |
+| LTX-2.5 keyframe interpolation (`KeyframeInterpolationPipeline`) | LTX-2.5 DiT + spatial upsampler | `ltx2 keyframe:` cases in `test_ltx2_pipeline` / `test_ltx2_video` via `LoadVideoEngine`+`Generate`: frame 0 APPENDS against a `ti2vid_two_stage` control, the x0 invariant on four arms, the 4096 anchor | `--pipeline-kind keyframe_interpolation`. No frame-0 special case, so `--first-frame` is guidance to interpolate FROM; stage 2's audio leaves. `--lora` REQUIRED. `--last-frame` new (#1191). CPU fixtures |
 | LTX-2.5 two-stage text/image-to-video (`TI2VidTwoStagesPipeline`) | LTX-2.5 DiT + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video` `ltx2 ti2vid:` cases through `LoadVideoEngine`+`Generate`; the x0 invariant on all FOUR arms, and the 4096 anchor read at two geometries against a res_2s control that moves | `--pipeline-kind ti2vid_two_stage`. Guided half-res stage 1 on the UNADAPTED model, plain Euler; distilled 3-sigma stage 2. `--lora` REQUIRED, no `--audio-path`; stage 1's audio leaves. CPU fixtures, Full-model run owed |
 | LTX-2.5 guidance knobs | LTX-2.5 request surface | `test_ltx2_video` renders with an override and refuses one on a fixed recipe | Seven video/audio guider extras mirroring `default_1_stage_arg_parser`, plus a negative embeds pair for a tower-less engine. Refused whole on `distilled_two_stage` and `retake`, whose guidance is distilled in |
 | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) |

docs/USAGE.md

@@ -3438,7 +3438,7 @@ CHECKPOINT_ROOT=... VLLM_CPP_LTX2_TOWER_E2E=1 \
 
 Recipes resolve on an EXACT `(pipeline_kind, model_version)` pair and refuse
 anything else by name rather than defaulting, because a plausible but wrong sigma
-schedule or guidance scale renders a video instead of failing. **Twenty-four**
+schedule or guidance scale renders a video instead of failing. **Twenty-eight**
 pairs resolve, derived from `ResolveLtx2PipelineRecipe`:
 
 | `pipeline_kind` | resolving `model_version` | what it also needs |
@@ -3452,6 +3452,7 @@ pairs resolve, derived from `ResolveLtx2PipelineRecipe`:
 | `t2a_one_stage` | 2, 2.3, 2.4, 2.5 | a text tower; no video VAE is asked for |
 | `a2vid_two_stage` | 2, 2.3, 2.4, 2.5 | `upsampler_path`, `lora_path`, and an `audio_path` on every request |
 | `ti2vid_two_stage` | 2, 2.3, 2.4, 2.5 | `upsampler_path` and `lora_path` |
+| `keyframe_interpolation` | 2, 2.3, 2.4, 2.5 | `upsampler_path` and `lora_path` |
 
 This list ran to ten until 2026-08-17, omitting `dfr` entirely and all four
 `t2a_one_stage` rows. **`dfr` at 2 is refused deliberately, not by oversight**:
@@ -3659,6 +3660,82 @@ of `/v1/videos`. That is a statement about the request surface: the gated path
 is `vllm_video_engine_load` plus `vllm_video_generate`, which is what `ltx2-gen`
 drives, and no test here exercises the HTTP route end to end.
 
+### `keyframe_interpolation`: generating the motion between pinned frames
+
+`KeyframeInterpolationPipeline` — you supply the keyframes, the model generates
+what happens between them. Its two stages are `ti2vid_two_stage`'s: a guided
+half-resolution stage 1 on the **unadapted** model, then a 2x latent upsample and
+a distilled three-sigma refinement. It needs the same `--lora-path` and
+`--upsampler-path`, for the same reasons.
+
+```sh
+ltx2-gen \
+  --pipeline-kind keyframe_interpolation \
+  --checkpoint "$CHECKPOINT_ROOT/ltx-2.5/..." \
+  --upsampler-path "$CHECKPOINT_ROOT/ltx-2.5/.../spatial-upsampler.safetensors" \
+  --lora-path "$CHECKPOINT_ROOT/ltx-2.5/.../ltx-2.5-22b-distilled-lora-450-bf16.safetensors" \
+  --prompt 'the balloon drifts from the left ridge to the right one' \
+  --first-frame open.ppm --last-frame close.ppm --image-crf 0 \
+  --height 704 --width 1216 --num-frames 121 --steps 30 \
+  --output-dir out/
+```
+
+**Two fields separate it from `ti2vid_two_stage`, and both render either way.**
+
+**The first frame is a KEYFRAME, not a replacement.** Every other pipeline maps a
+conditioning image at frame 0 onto a latent-index item, which overwrites the
+tokens of latent frame 0 in place. This one drops that special case: the image is
+appended as keyframe guidance the model interpolates *from*, and the sequence the
+transformer runs over grows by one latent frame. Nothing about a rendered clip
+shows which mapping was used — both return the right size, the right frame count
+and the right sample rate with the image visibly present — so the difference is
+gated on the token count the transformer actually ran over.
+
+**The soundtrack that leaves is stage 2's**, where `ti2vid_two_stage` keeps stage
+1's and discards its refinement stage's audio. Upstream says so by what it binds
+rather than in a comment, and the two pipelines bind opposite ways.
+
+Everything else is shared. `--lora-path` is **required** and the load is refused
+without it: upstream makes the distilled adapter a positional, non-defaulted
+constructor argument as well as a required flag, and the adapter rides **stage 2
+alone** while stage 1 runs the base weights. There is no `--audio-path`; the
+soundtrack is generated. Height and width describe the FINAL output and must
+divide 64, because stage 1 halves them. Its stage-1 sigma shift is fitted on the
+scheduler's fixed 4096-token anchor rather than on the target latent grid, which
+is what upstream's `execute(steps=...)` with no latent resolves to.
+
+**`--last-frame` is new with this kind** and works on every pipeline that takes
+images: the ABI and the engine have served a closing keyframe since
+[#930](https://github.com/mudler/vllm.cpp/issues/930), and `ltx2-gen` had never
+read the field ([#1191](https://github.com/mudler/vllm.cpp/issues/1191)). Both
+image slots share one `--image-crf` and one strength, and a keyframe at an
+**interior** frame is not requestable — upstream's `--image PATH FRAME_IDX
+STRENGTH [CRF]` is repeatable and this request surface carries two fixed slots
+([#1187](https://github.com/mudler/vllm.cpp/issues/1187)).
+
+**Which weights this was gated against: reduced CPU fixtures, and nothing else.**
+Upstream runs this pipeline on the FULL model
+(`ltx-2.5-22b-dev-transformer-bf16.safetensors`, 42,018,190,584 bytes, 4349
+tensors, 21.004 B parameters, pure BF16, `model_version` `2.5.0`), which is on
+the NAS and header-verified, and which `LTX25-BF16-DIT`
+([#1148](https://github.com/mudler/vllm.cpp/issues/1148)) made loadable. **What
+is owed is the run**: a comparison against upstream's own render on the same
+checkpoint, prompt and seed. Do **not** substitute a distilled transformer to try
+the arm out — the distilled scales are trained into those weights, so a
+CFG-guided stage 1 on top samples a trajectory they were never trained for and
+renders a plausible clip with nothing in its size, frame count, sample rate or
+errors to show it ([#1137](https://github.com/mudler/vllm.cpp/issues/1137)).
+
+All three knobs this arm needs are LOAD extras, so a server supplies them with
+`--video-extra pipeline_kind=keyframe_interpolation` and the same for
+`lora_path` and `upsampler_path`. Like `ti2vid_two_stage` and unlike
+`a2vid_two_stage` it needs no per-generation extra, so
+[#928](https://github.com/mudler/vllm.cpp/issues/928) does not stand in the way
+of `/v1/videos` — though `/v1/videos` forwards no image either, so a server
+render is unconditioned. That is a statement about the request surface: the gated
+path is `vllm_video_engine_load` plus `vllm_video_generate`, which is what
+`ltx2-gen` drives, and no test here exercises the HTTP route end to end.
+
 ### Retake: regenerating a time window of an existing clip
 
 `retake` is `RetakePipeline`: it keeps the source clip outside a window and

examples/ltx2_gen/main.cpp

@@ -101,7 +101,8 @@ const char* Need(int argc, char** argv, int i, const char* flag) {
       "                [--lora <ic-lora.safetensors> [STRENGTH]]  fused at load; 1.0\n"
       "                [--prompt-valid-rows N]   how many embed rows are real tokens\n"
       "                [--frames N] [--width N] [--height N] [--seed N]\n"
-      "                [--first-frame <image.ppm>] [--image-crf 0]\n"
+      "                [--first-frame <image.ppm>] [--last-frame <image.ppm>]\n"
+      "                [--image-crf 0]\n"
       "                [--audio-path <in.wav>] [--audio-start-time S]\n"
       "                [--audio-max-duration S]\n"
       "                [--device cpu|cuda]\n\n"
@@ -135,7 +136,12 @@ const char* Need(int argc, char** argv, int i, const char* flag) {
       "round trip needs libx264 and none is vendored here, so leaving --image-crf out\n"
       "resolves 18 and REFUSES by name. --image-crf 0 is upstream-legal and OUT OF\n"
       "DISTRIBUTION: the model sees pixels it was not trained on. That is a quality\n"
-      "cost, and this tool states it rather than turning it on quietly.\n\n"
+      "cost, and this tool states it rather than turning it on quietly.\n"
+      "--last-frame takes a second PPM and pins the CLOSING frame. It is a KEYFRAME\n"
+      "rather than a replacement: its tokens are APPENDED to the sequence, carrying\n"
+      "the temporal position of pixel frame N-1, and are trimmed off again before the\n"
+      "clip is decoded. Both slots share one --image-crf and one strength, and a\n"
+      "keyframe at an INTERIOR frame is not requestable (#1187).\n\n"
       "AUDIO-TO-VIDEO. --audio-path takes a 16-bit PCM WAV and CONDITIONS the render\n"
       "on it: the take is decoded, encoded through the audio VAE\'s encoder, truncated\n"
       "to the clip\'s duration, and then held FROZEN through every denoise phase, so\n"
@@ -221,7 +227,19 @@ const char* Need(int argc, char** argv, int i, const char* flag) {
       "divide 64, since stage 1 halves them. Upstream runs this on the FULL\n"
       "(non-distilled) transformer; pointing it at a distilled checkpoint renders a\n"
       "plausible clip on a trajectory those weights were never trained for, and\n"
-      "nothing in the output says so.\n");
+      "nothing in the output says so.\n\n"
+      "KEYFRAME INTERPOLATION generates the motion BETWEEN keyframes you pin.\n"
+      "--pipeline-kind keyframe_interpolation selects it. Its two stages are the\n"
+      "ti2vid_two_stage ones -- guided half-res stage 1 on the UNADAPTED model, then a\n"
+      "distilled three-sigma refinement -- and it needs the same --lora and\n"
+      "--upsampler for the same reasons. TWO fields differ and both of them render\n"
+      "either way. First, --first-frame is a KEYFRAME here rather than a frame that\n"
+      "overwrites the opening latent: upstream drops the frame-0 special case, so the\n"
+      "image is appended as guidance the model interpolates FROM instead of replacing\n"
+      "what it would otherwise generate. Second, the audio.wav you get back is STAGE\n"
+      "2\'s, not stage 1\'s as on ti2vid_two_stage. Use --first-frame and --last-frame\n"
+      "together to pin both ends of the clip; a keyframe at an INTERIOR frame is not\n"
+      "requestable yet.\n");
   std::exit(code);
 }
 
@@ -233,7 +251,7 @@ int main(int argc, char** argv) {
   std::string workdir = "/tmp/ltx2_gen", out_path, ffmpeg = "ffmpeg", device = "cuda";
   // BORROWED by `vllm_video_generate`, like the extras below, so it is owned
   // here and pointed at only after parsing.
-  std::string prompt, first_frame, image_crf;
+  std::string prompt, first_frame, last_frame, image_crf;
   std::string audio_path, audio_start_time, audio_max_duration;
   // RETAKE (row LTX25-RETAKE, #924): a source clip DIRECTORY and the window to
   // regenerate. `--ref-video` is a directory of frame_%06d.ppm, not a container.
@@ -319,6 +337,15 @@ int main(int argc, char** argv) {
     // own 18 and refuse, which is the point: this CLI must not be the thing that
     // quietly turns an out-of-distribution render on.
     else if (f == "--first-frame") first_frame = Need(argc, argv, ++i, "--first-frame");
+    // `--last-frame` pins the CLOSING keyframe, at pixel frame `frames - 1`. The
+    // ABI has carried `last_frame` and the engine has served it since row
+    // LTX25-TOKEN-APPEND (#930); this CLI simply never read the field, which
+    // #1191 records. It matters from `keyframe_interpolation` on, because a
+    // pipeline whose whole job is the motion BETWEEN two pinned frames could
+    // otherwise only be asked for one of them. Same `--image-crf` and the same
+    // strength as the first frame, because the request surface carries one of
+    // each (#1187).
+    else if (f == "--last-frame") last_frame = Need(argc, argv, ++i, "--last-frame");
     else if (f == "--image-crf") image_crf = Need(argc, argv, ++i, "--image-crf");
     // AUDIO-TO-VIDEO (#922). Upstream's `--audio-path` is REQUIRED because that
     // CLI drives the A2V pipeline and nothing else (a2vid_two_stage.py:312-317);
@@ -399,6 +426,7 @@ int main(int argc, char** argv) {
   vp.output_dir = workdir.c_str();
   if (!prompt.empty()) vp.prompt = prompt.c_str();
   if (!first_frame.empty()) vp.first_frame = first_frame.c_str();
+  if (!last_frame.empty()) vp.last_frame = last_frame.c_str();
   if (!ref_video.empty()) vp.ref_video = ref_video.c_str();
 
   // The PER-GENERATION extras are a SEPARATE array from the load-time ones, and