record(LTX25-DECODE-SPEED): three scalar single-threaded loops in the LTX CPU path, and the number ltx2_video.cpp said was owed

.agents/issue-index.md

@@ -379,6 +379,9 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1189](https://github.com/mudler/vllm.cpp/issues/1189) | `VT-QUANT-FP8-GROUP` | Block-wise (128x128) FP8 so `Qwen/Qwen3.8-27B-FP8` runs instead of being refused. `weight_block_size` appears nowhere in `src/` or `include/`, so `469f38395` refuses the arm by name (#1166). Six independently landable milestones; M1 lands here. M1 is `vt::QuantFp8Group`, the dynamic per-token per-group activation quant, CPU and CUDA. The numerics mirror the kernel that ACTUALLY EXECUTES on a CUDA-alike platform with a contiguous input, which is the C++ custom op at `csrc/libtorch_stable/quantization/w8a8/fp8/per_token_group_quant.cu:42-96` and NOT the Triton kernel at `fp8_utils.py:95-150`: `fp8_utils.py:635-650` calls the former and returns before the latter. The two arms differ, and the difference is measurable rather than cosmetic. The CUDA kernel divides twice, `local_absmax / max_8bit` at `:68` and `static_cast<float>(src) / y_s` at `:85`; the Triton kernel multiplies by `(1.0 / fp8_max)` at `fp8_utils.py:145` under a comment that names the 1-ULP gap. Upstream's own test tolerates the gap with `rtol=0.15` (`test_block_fp8.py:112-114`), so a value comparison cannot tell the two apart and only a byte comparison against a spelled-out reference can. `eps` is the reduction's INITIAL value (`:47`), not a post-clamp, which is what keeps an all-zero group from dividing by zero. Scope refused here and owed to later milestones: the block-scaled GEMM (M2), `Fp8BlockWeight` and the loader (M3), `Fp8BlockLinearMethod` and the Qwen3.5 wiring (M4), the mainloop-scaled CUTLASS kernel and the column-major/TMA-aligned scale layouts (M5), merged `gate_up`/QKV (M6). M1 lands UNREACHED: no production entry point dispatches `vt::QuantFp8Group` at its merge commit, M4 owns the wiring, and `.agents/specs/vt-quant-fp8-group.md` lists it under `## Owed`. The CUDA arm compiles and its on-hardware leg is owed too, because the row took no GPU lease by design: the CPU arm is the gateable one | feature |
 | [#1190](https://github.com/mudler/vllm.cpp/issues/1190) | `ENV-AGNOSTIC-CAMPAIGN` | One operator's hosts, share paths and addresses are written into 227 tracked files where a `.env` placeholder belongs, so a second developer who follows the protocol documents is told to reach a box on one home network. Re-derived at `fd64c76ee` with `git grep -cIE 'dgx\.casa\|nas_share\|192\.168\.\|thor:gpu0'`: `dgx.casa` 203 files, `nas_share` 32, `192.168.` 24, `thor:gpu0` 11. The mechanism was already there and almost unused: `.env.example` is tracked, `.env` and `.agents/developer-preferences.md` are ignored, `scripts/agent-onboard.py --env-set` already records one answered value and refuses an undeclared key, and `${VLLM_SOURCE}` and `${GPU_LOCK}` already resolve in 59 and 50 files while `${DEVICE_ARCH}`, `${DEVICE_TOOLKIT_ROOT}` and `${DEVICE_COMPILER}` resolve in none. This SCOPING row lands the rule and the mechanism, not the sweep. The rule is a substitution test: replace the literal with a second developer's value and if the sentence stays true it is CONFIGURATION and becomes `${KEY}`, and if it becomes false it is PROVENANCE and stays literal. That reverses the issue's own ranking, because all 34 hits in the densest guide file `.agents/environment.md` are provenance or named-profile definition and none is configuration, so density does not predict the defect and a blind `sed` would falsify records. Landed: three new keys `GATE_CHECKOUT`, `SHARED_STORAGE_ROOT` and `GATE_DEVICE` derived from the literals that recur and map to no existing key; the create-on-first-use route in `scripts/agent-start.py`, which printed `environment: missing` as a status label and then listed next actions that never mentioned it, so the fallback in practice was a host name copied from a document; the matching obligation in `AGENTS.md`; and one worked example. The worked example `scripts/dgx-bringup.sh` found a live defect rather than a cosmetic one: it defaulted `CUTLASS_DIR` to `$HOME/cutlass_probe` while `.agents/environment.md:389` records `$HOME/cutlass-4.5.0` as mandatory on the same box, and a configure that misses CUTLASS silently drops the sm120a NVFP4 GEMM and FlashAttention-2, which that file measures as moving the SACRED `test_qwen27_paged_engine` from 235/235 to 234/235 with the source untouched, so a stale hard-coded default is a false green. Its new test also caught that `set -a; . ./.env; set +a` over an `.env.example` copy blanks a value the caller exported, so the process-environment contract is now executable. Waves `ENV-AGNOSTIC-W1-TOOLING` through `ENV-AGNOSTIC-W5-LEDGERS` own the sweep, partition all 227 files, and open their own issues. Spec [`env-agnostic.md`](specs/env-agnostic.md) | bug |
 | [#1193](https://github.com/mudler/vllm.cpp/issues/1193) | `SPEC-DSPARK-QWEN3-ROUTING` | A Qwen3 DSpark draft declaring `architectures=["DSparkDraftModel"]` with `model_type` `qwen3` has no route. The pin forces every DSpark draft that is not `Qwen3DSparkModel` or `Gemma4DSparkModel` onto `model_type` `deepseek_v4` (`vllm/config/speculative.py:934-944` @ `555967922`), and vLLM PR 52197 (merged 2026-08-17 at `7075ddac`) replaced that with a leading branch normalizing the pair to `Qwen3DSparkModel`. We diverge from BOTH: the forced rewrite was never ported, so nothing in `src/vllm/entrypoints/model_loader.cpp` reads a draft config's `architectures` key at all, and `SpeculativeConfig::IsDsparkDraft` (`include/vllm/config/speculative.h:120-136`) has no production caller — every reference outside its header is in `tests/vllm/config/test_speculative_dspark.cpp:132-140`, and `ResolveSpecConfig` branches on `cli.method` alone. The checkpoint is real and gateable here: `RadixArk/Qwen3.8-27B-DSpark` at revision `85ef153be924f17ce4bf62726954eeaa4a73e854` carries exactly that config shape in one 2718576122-byte shard, drafting five layers for a 64-layer Qwen3.8-27B target | bug |
+| [#1202](https://github.com/mudler/vllm.cpp/issues/1202) | — | `Ltx2FuseLoraIntoTensor` (`src/vllm/model_executor/models/ltx2_lora.cpp:321-334`) computes the `(B * strength) @ A` LoRA product with a scalar single-threaded triple loop: one thread, no blocking, no SIMD, a non-inlined `vt::BF16ToF32` per multiply, and an inner operand `pair->a[k * cols + i]` striding by `cols` so every load in the innermost loop is its own cache line. Measured on `dgx` (GB10, 20 cores) loading the full/dev transformer (21,004,025,600 params) with the shipped 8.9 GB distilled adapter: three `gdb` stacks all reading `vt::BF16ToF32` <- `Ltx2FuseLoraIntoTensor` <- `Ltx2LoadDitFromSafetensors` <- `Ltx2VideoEngine::Load`, one thread at 99.9% of one core with 19 idle, and an f32 working set growing 9.432 -> 10.235 GiB over 300-629 s = **2.3% of one pass in 10.4 minutes**, cross-checked against the sum of `out*in*rank` over the 1660 targeted modules = 8.53e12 MAC, consistent with ~0.53 GFLOP/s. The operation is a rank-`r` GEMM and belongs on the `vt::` GEMM seam like every other projection in the tree; the arithmetic is already a correct mirror of `fuse_loras.py:103-116` (`B * strength` rounds to bf16 BEFORE the product, f32 accumulation, bf16 store) and only the execution strategy is wrong, so a replacement has a bit-exact oracle rather than a tolerance. Blocks every LoRA-bearing pipeline kind on the full model; `one_stage` is unaffected because upstream marks it `Full` with no adapter, which is why it is the only full-model arm that currently reaches generation. Owed by [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) `## Owed`, whose §5 already frames "why the decode is single-threaded and on the host" | perf |
+| [#1208](https://github.com/mudler/vllm.cpp/issues/1208) | — | `Linear` in the LTX-2.5 text tower (`src/vllm/model_executor/models/ltx2_text_encoder.cpp:60-71`) is a scalar single-threaded triple loop with a `double` accumulator and both operands widened per multiply, so it cannot reach the f32 FMA path even if the compiler vectorised it. On the full model's caption projection (in_features on the order of 1.9e5) cost is `rows * out_features * in_features` and it dominates the pre-generation wall: measured on `dgx`, resident memory went flat at t≈150 s and stayed **byte-identical for 1073 s** with stacks in the text feature extractor. Because `ltx2-gen` prints nothing between load and completion ([#1010](https://github.com/mudler/vllm.cpp/issues/1010) owns that), this presents to a user as a hang rather than as slow arithmetic, and it is why an earlier probe's "reached Generate" claim was retracted — the trace does not support it. TWO separate defects: the execution strategy, which belongs on the `vt::` GEMM seam; and the `double` accumulator, which is NOT a mirror — `torch.nn.functional.linear` on f32 inputs accumulates in f32, and the comment directly above this function cites `F.linear` as its reference, so the widening diverges from the oracle it names, cannot be bit-compared against upstream, and hides reduction-order differences an f32 accumulator would expose. Per the dtype-polarity rule an f64 accumulator kept deliberately needs a one-line reason beside it. Every LTX-2.5 pipeline kind goes through the text tower, so this is on the critical path of all of them including `one_stage`. Owed by [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) `## Owed` | perf |
+| [#1210](https://github.com/mudler/vllm.cpp/issues/1210) | — | The two-stage LoRA rebind cost that `src/vllm/multimodal/ltx2_video.cpp:2843-2851` records as "UNMEASURED on real weights", with a later perf row owning the number, is now measured. A two-stage recipe loads FUSED, phase 0 asks `Ltx2PhaseLoraScope::kNoAdapters` and `Ltx2RebindDitLoras` un-fuses, phase 1 asks `kAllAdapters` and re-fuses — so the load-time fusion is **provably wasted**, undone before any denoise step runs, and the DiT is left fused so the next render pays the same two again. At [#1202](https://github.com/mudler/vllm.cpp/issues/1202)'s measured ~0.53 GFLOP/s each pass is hours, and a two-stage full-model render pays three of them before the first step. TWO independent fixes, not one change: making the fusion fast (#1202) shrinks the constant but leaves the wasted round trip; separately, `Ltx2PipelineRecipe::phases` is available before `Load` runs, so the load could honour phase 0's scope and skip the fuse/un-fuse entirely. The terminal fused state is chosen rather than forced and is worth revisiting in the same change. Affects `a2vid_two_stage` and `ti2vid_two_stage`; `one_stage` pays nothing. NOT CLAIMED: the wall-clock figures are a rate measured over a 10.4-minute window and extrapolated, not a completed pass — no two-stage full-model render has completed, so the end-to-end number stays open; the rate, thread count and stack attribution are measured. Owed by [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) `## Owed` | perf |
 | [#1219](https://github.com/mudler/vllm.cpp/issues/1219) | `LTX25-KEYFRAME-INTERP` | Pinning BOTH keyframes aborts the render on every recipe whose frame-0 item APPENDS, which is `keyframe_interpolation` -- the one pipeline that exists to interpolate between two pinned ends, whose `docs/USAGE.md` worked example passes `--first-frame open.ppm --last-frame close.ppm` and whose `ltx2-gen --help` says to use the two together. Reachable from the shipped CLI and from `vllm_video_engine_load` / `Generate`. The defect is a derived index that was correct only by APPEND ORDER: the last-frame arm of `src/vllm/multimodal/ltx2_video.cpp` located its own appended tokens at `video.positions[target_tokens * 2]`, and `target_tokens` is the phase's fixed target grid, so that index names the first token PAST THE GRID -- the first appended token in the sequence, whichever item appended it. Row `LTX25-KEYFRAME-INTERP` ([#1096](https://github.com/mudler/vllm.cpp/issues/1096)) put a second appending item in front of it, because `image_conditionings_by_adding_guiding_latent` sends frame 0 to `VideoConditionByKeyframeIndex`, which APPENDS (`keyframe_cond.py:79-82`), rather than to `VideoConditionByLatentIndex`, which replaces (`latent_cond.py:40-41`). With both ends pinned the index named the FIRST frame's keyframe at temporal 0 and the arm's own positional assertion threw `must carry the temporal position of pixel frame frames - 1 (0.333333), but the first appended token starts at 0.000000`. The `ti2vid_two_stage` control -- replace at frame 0, one append -- never reached it and stayed green, which is what says ordering rather than keyframes. The generated-keyframe-slot arm carried the same shape in its growth check. Neither assertion is weakened: fixed in flow by capturing the sequence length at the moment of each append and locating each item's tokens from that, plus a new assertion that the recorded `generated_keyframe_layout.first_token` equals that pre-append count. Gated by a permanent `test_ltx2_video` case pinning both ends on both builders with the bare grid and each single end as controls, captured RED at the pre-fix source with that exact message. Found by the fresh review of #1096 | bug |
 | [#1220](https://github.com/mudler/vllm.cpp/issues/1220) | — | The two LTX schedule-anchor cases in `tests/vllm/multimodal/test_ltx2_video.cpp` return the REQUEST step count from their render lambda under a comment saying the count is read back out of the render, so the four `REQUIRE(x.steps == rendered_steps)` lines compare four copies of one request field rather than four renders. WEAKENED, not vacuous, and that is why it is filed rather than repaired in flow: the lambda pins the step count at 3 with a stated reason -- a 2-step schedule is `{1, 0.1, 0}` for every token count, so the 4096 anchor cannot reach the trajectory -- and the assertions below recompute `Ltx2SigmaSchedule` at it, so lowering that literal still reds them by name. What it cannot see is an engine that ignored the request and ran a different number of steps. `Ltx2ConditioningTrace::dit_evaluations` (`include/vllm/multimodal/ltx2_video.h:757-774`) already carries the observation, incremented at one site inside the shared `Evaluate` lambda, and is `steps` per phase on the Euler and ancestral arms against `2 * steps + 1` on res_2s. Not repaired during the review repair of [#1096](https://github.com/mudler/vllm.cpp/issues/1096) because it changes what a LANDED case measures on both pipelines and on the res_2s control, so it owes its own red-first evidence that the derived count reproduces the current literal on every arm rather than a quiet re-derivation of a passing assertion. Listed under `## Owed` in [`ltx25-keyframe-interp.md`](specs/ltx25-keyframe-interp.md) | bug |
 | [#1230](https://github.com/mudler/vllm.cpp/issues/1230) | — | Twenty-two LTX-2.5 source citations name two upstream anchors short, and the SAME FILES already carry the corrected form beside the stale one, so a reader who greps either string finds both and has nothing that says which is right. Re-derived at the LTX-2 pin `fd4ded7f` by reading the pinned files: `latent_cond.py:38` is `latent_state = latent_state.clone()` and `:39` is blank, so the two writes every citation means are `:40-41`; `schedulers.py:31` is the return annotation `) -> torch.FloatTensor:`, so the `tokens = math.prod(latent.shape[2:])` read is `:32`. Eleven citations of each stale form, across `src/vllm/multimodal/ltx2_video.cpp`, `src/vllm/model_executor/models/ltx2_pipeline.cpp`, `include/vllm/model_executor/models/ltx2_pipeline.h`, `include/vllm/model_executor/models/ltx2_conditioning.h`, four test files and two specs; the corrected form is already landed in eight more, including `include/vllm/multimodal/ltx2_video.h:625` and `include/vllm/model_executor/models/ltx2_samplers.h:32`. The disagreement is INSIDE single files: `ltx2_video.cpp` cites `schedulers.py:31` at `:3618` and `:32` at `:3017` and `:3600`. **A partial correction is strictly worse than none**, which is why this is a row rather than an in-flow fix and why PR #1209's review-repair commit had its seven anchor corrections REVERTED rather than extended. Those seven left `ltx2_video.cpp` reading both forms a hundred lines apart with nothing recording which to believe -- correctable by a grep while uniform, not correctable while mixed -- and one of them lived in `include/vllm/`, a `USER_USAGE_PREFIXES` path in `scripts/check-doc-checkpoint.py:99`, which is a pure path match with no content analysis, so a comment-only anchor edit in a public header demanded a `docs/USAGE.md` edit the change did not owe. Both consequences are properties of doing it piecemeal and neither arises from one sweep over all 22. Filed by the review-repair flow of `LTX25-KEYFRAME-INTERP` ([#1096](https://github.com/mudler/vllm.cpp/issues/1096)), which owes it under `## Owed` rather than fixing it in flow: the sweep opens five files that row does not otherwise touch and a 22-site mechanical edit owes its own reviewer | bug |

.agents/specs/ltx25-decode-speed.md

@@ -1418,6 +1418,9 @@ authority and no fresh review (§8).
 | [#1021](https://github.com/mudler/vllm.cpp/issues/1021) | 5c — DiT staging is 7.5 min at 70.5 MiB/s, against 127.8 on rung 2; GPU idle, 0.15 cores | owed |
 | [#1024](https://github.com/mudler/vllm.cpp/issues/1024) | 0 — the GPU is idle for the WHOLE post-load render, not only the decode | owed; carries the owed `utilization.gpu` positive control |
 | [#1040](https://github.com/mudler/vllm.cpp/issues/1040) | none — the evidence for rungs 1 and 2 and for §1.4 is on an unreachable host, and neither rung's sampler cadence closes | owed |
+| [#1202](https://github.com/mudler/vllm.cpp/issues/1202) | 7 — `Ltx2FuseLoraIntoTensor` is a scalar single-threaded loop, ~0.53 GFLOP/s | owed, MEASURED on the full model: 2.3% of one pass in 10.4 min |
+| [#1208](https://github.com/mudler/vllm.cpp/issues/1208) | 8 — the text tower's `Linear` is scalar, single-threaded and `double`-accumulating | owed, MEASURED at 1073 s of byte-identical RSS; also a dtype-polarity divergence from `F.linear` |
+| [#1210](https://github.com/mudler/vllm.cpp/issues/1210) | 9 — the two-stage rebind fuses, un-fuses and re-fuses; the load-time pass is provably wasted | owed; supplies the number `ltx2_video.cpp:2849` records as UNMEASURED |
 
 **[#1008](https://github.com/mudler/vllm.cpp/issues/1008) is NOT owed here. It
 landed.** It was filed by this row as lever 2 and taken by `LTX25-DECODE-DTYPE`,