From e5956e60a8d91809db8fec3e519ad50433f4e9b4 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 08:52:52 +0000 Subject: [PATCH] record(LTX25-DECODE-SPEED): three scalar single-threaded loops in the LTX CPU path, and the number ltx2_video.cpp said was owed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three defects found while attempting the first full-model LTX-2.5 render are filed and now have an owner. None is fixed here; this change is the record. They are one story. The LTX prep path hand-rolls scalar arithmetic where the rest of the tree routes through the `vt::` GEMM seam, and it does so in three places that all sit between `Load` and the first denoise step: - #1202 `Ltx2FuseLoraIntoTensor` computes the LoRA product with a triple scalar loop at a measured ~0.53 GFLOP/s, one thread of twenty. - #1208 the text tower's `Linear` is scalar, single-threaded and accumulates in `double`, which also diverges from the `F.linear` it cites as reference. - #1210 the two-stage rebind fuses at load, un-fuses for phase 0 and re-fuses for phase 1, so the load-time pass is provably wasted. #1210 supplies a number the tree explicitly asked for. `ltx2_video.cpp:2849` records the two-rebind cost as "UNMEASURED on real weights" and says a later perf row owns it. This is that measurement. The owner is `ltx25-decode-speed.md`, whose §5 already frames "why the decode is single-threaded and on the host". These extend that finding from the decode to the load and prep path, so they belong to its `## Owed` table rather than to a new row. Scope is deliberately narrow. The rates are measured over a 10.4-minute window and extrapolated; no full-model pass has completed, so no end-to-end figure is claimed. What is measured is stated as measured and what is projected is labelled. `test_cpu_x86_llamacpp_floor` fails in the local preflight with exit 4 (`NO_QUIET_WINDOW`) instead of 2 at loadavg 68.9-79.9, which is #618 exactly. This diff touches two files under `.agents/` and cannot reach a CPU benchmark harness. `check-agent-record.py`, `check-public-doc-tables.py`, doc-checkpoint and now-current all pass. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 3 +++ .agents/specs/ltx25-decode-speed.md | 3 +++ 2 files changed, 6 insertions(+) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index ec4e81558..b7543617d 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -368,3 +368,6 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1185](https://github.com/mudler/vllm.cpp/issues/1185) | `ENV-ORACLE-WHEEL-IN-LEASE` | The pinned vLLM oracle BUILDS, installs, imports and sees the GPU inside an `rc` lease on `dgx:gpu0`, measured 2026-08-18, which falsifies the `nvcc` clause four records carried. [`lease-runtime-staging.md`](specs/lease-runtime-staging.md) said the oracle "needs `nvcc`, which the worker still lacks", and `.agents/environment.md`, [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) and [`gpu-lease-methodology.md`](specs/gpu-lease-methodology.md) each derived a blocker from it. The build job (`buildvllm.sh`, staged sha256 prefix `15e140d41f44e7c2`) asserted the checkout against the pin BEFORE compiling, printing `PIN CONFIRMED` at `5559679229bc961848b121ccdeaa8fa5d79bec98` and aborting otherwise, took `nvcc` from the toolkit row `MODEL-NEMOTRON-H-ABI-A3-E2E` staged (`NVCC_RC=0`, CUDA `release 13.3, V13.3.73`) and produced `WHEEL_RC=0`, `PERSIST_RC=0` and a 434 MiB `vllm-0.1.dev1+g555967922.cu133-cp312-cp312-linux_aarch64.whl`, sha256 `7c58b339741a288fbb313f4f5196c9c92a9e3b3c3ebe2ea970b0ff50bb9bcba4`. The identity job (`oracleenv.sh`, prefix `6119f5223f5d818c`) asserted from `cd /`, outside any source tree: `vllm.__version__ = 0.1.dev1+g555967922`, `IDENTITY_RC=0`, `cuda True NVIDIA GB10`, `CUDA_RC=0`. SCOPE, and it carries the same weight as the result: RUNNING A MODEL IS UNTESTED. Only build, install, import and `torch.cuda.is_available()` are measured, and [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) records that the last time an oracle reached this far it consumed the host in the step AFTER `torch.compile` and REBOOTED the box, at `gpu_memory_utilization` 0.75 and again at 0.30, so the fraction is not the lever. The version string is an OPEN discrepancy: `.agents/upstream-sync.md` records `vllm_runtime_version = 0.23.1rc1.dev1511+g555967922`, the commit segment matches and satisfies the pin's binding `+g` rule, and the prefix differs because a shallow fetch stops `setuptools_scm` counting commits since the last tag, so a full-string gate needs a deeper fetch or a recorded pretend-version. The venv is NOT staged, because that job was killed at a 90-minute ceiling mid-copy and its partial tree was removed, so only the WHEEL is durable. Four staging walls, all artifacts of the NAS rather than of CUDA: `cp -a` preserves `file_mode=0664` so `nvcc` exited 126. CIFS `nounix` stores no symlink so `include` and `lib64` vanished and CMake reported `Could NOT find CUDA (missing: CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) (found version "13.3")`, naming the version and denying the toolkit in one line. 32 library links `libfoo.so` and `libfoo.so.MAJOR` had to be rebuilt because only the `libfoo.so.X.Y.Z` real files survived. And `markupsafe` existed as a dist-info with NO package files from a `pip --target` killed at a 35-minute ceiling, so Marlin codegen died on `ModuleNotFoundError`. The `rc` worker container is REUSED between jobs, so a repair inside a staging branch is skipped on the next run (`nvcc already in place`) and an environment repair must be unconditional and assert its postcondition. CONSEQUENCE for the rows #1129 blocked, [#1003](https://github.com/mudler/vllm.cpp/issues/1003), [#915](https://github.com/mudler/vllm.cpp/issues/915), [#821](https://github.com/mudler/vllm.cpp/issues/821) and [#81](https://github.com/mudler/vllm.cpp/issues/81): UNBLOCKED FOR THE BUILD STEP and STILL BLOCKED FOR A MODEL RUN. None can take a measurement until a model run is demonstrated. Job details, walls and non-claims in [`oracle-wheel-in-lease.md`](specs/oracle-wheel-in-lease.md) | verification | | [#1190](https://github.com/mudler/vllm.cpp/issues/1190) | `ENV-AGNOSTIC-CAMPAIGN` | One operator's hosts, share paths and addresses are written into 227 tracked files where a `.env` placeholder belongs, so a second developer who follows the protocol documents is told to reach a box on one home network. Re-derived at `fd64c76ee` with `git grep -cIE 'dgx\.casa\|nas_share\|192\.168\.\|thor:gpu0'`: `dgx.casa` 203 files, `nas_share` 32, `192.168.` 24, `thor:gpu0` 11. The mechanism was already there and almost unused: `.env.example` is tracked, `.env` and `.agents/developer-preferences.md` are ignored, `scripts/agent-onboard.py --env-set` already records one answered value and refuses an undeclared key, and `${VLLM_SOURCE}` and `${GPU_LOCK}` already resolve in 59 and 50 files while `${DEVICE_ARCH}`, `${DEVICE_TOOLKIT_ROOT}` and `${DEVICE_COMPILER}` resolve in none. This SCOPING row lands the rule and the mechanism, not the sweep. The rule is a substitution test: replace the literal with a second developer's value and if the sentence stays true it is CONFIGURATION and becomes `${KEY}`, and if it becomes false it is PROVENANCE and stays literal. That reverses the issue's own ranking, because all 34 hits in the densest guide file `.agents/environment.md` are provenance or named-profile definition and none is configuration, so density does not predict the defect and a blind `sed` would falsify records. Landed: three new keys `GATE_CHECKOUT`, `SHARED_STORAGE_ROOT` and `GATE_DEVICE` derived from the literals that recur and map to no existing key; the create-on-first-use route in `scripts/agent-start.py`, which printed `environment: missing` as a status label and then listed next actions that never mentioned it, so the fallback in practice was a host name copied from a document; the matching obligation in `AGENTS.md`; and one worked example. The worked example `scripts/dgx-bringup.sh` found a live defect rather than a cosmetic one: it defaulted `CUTLASS_DIR` to `$HOME/cutlass_probe` while `.agents/environment.md:389` records `$HOME/cutlass-4.5.0` as mandatory on the same box, and a configure that misses CUTLASS silently drops the sm120a NVFP4 GEMM and FlashAttention-2, which that file measures as moving the SACRED `test_qwen27_paged_engine` from 235/235 to 234/235 with the source untouched, so a stale hard-coded default is a false green. Its new test also caught that `set -a; . ./.env; set +a` over an `.env.example` copy blanks a value the caller exported, so the process-environment contract is now executable. Waves `ENV-AGNOSTIC-W1-TOOLING` through `ENV-AGNOSTIC-W5-LEDGERS` own the sweep, partition all 227 files, and open their own issues. Spec [`env-agnostic.md`](specs/env-agnostic.md) | bug | | [#1193](https://github.com/mudler/vllm.cpp/issues/1193) | `SPEC-DSPARK-QWEN3-ROUTING` | A Qwen3 DSpark draft declaring `architectures=["DSparkDraftModel"]` with `model_type` `qwen3` has no route. The pin forces every DSpark draft that is not `Qwen3DSparkModel` or `Gemma4DSparkModel` onto `model_type` `deepseek_v4` (`vllm/config/speculative.py:934-944` @ `555967922`), and vLLM PR 52197 (merged 2026-08-17 at `7075ddac`) replaced that with a leading branch normalizing the pair to `Qwen3DSparkModel`. We diverge from BOTH: the forced rewrite was never ported, so nothing in `src/vllm/entrypoints/model_loader.cpp` reads a draft config's `architectures` key at all, and `SpeculativeConfig::IsDsparkDraft` (`include/vllm/config/speculative.h:120-136`) has no production caller — every reference outside its header is in `tests/vllm/config/test_speculative_dspark.cpp:132-140`, and `ResolveSpecConfig` branches on `cli.method` alone. The checkpoint is real and gateable here: `RadixArk/Qwen3.8-27B-DSpark` at revision `85ef153be924f17ce4bf62726954eeaa4a73e854` carries exactly that config shape in one 2718576122-byte shard, drafting five layers for a 64-layer Qwen3.8-27B target | bug | +| [#1202](https://github.com/mudler/vllm.cpp/issues/1202) | — | `Ltx2FuseLoraIntoTensor` (`src/vllm/model_executor/models/ltx2_lora.cpp:321-334`) computes the `(B * strength) @ A` LoRA product with a scalar single-threaded triple loop: one thread, no blocking, no SIMD, a non-inlined `vt::BF16ToF32` per multiply, and an inner operand `pair->a[k * cols + i]` striding by `cols` so every load in the innermost loop is its own cache line. Measured on `dgx` (GB10, 20 cores) loading the full/dev transformer (21,004,025,600 params) with the shipped 8.9 GB distilled adapter: three `gdb` stacks all reading `vt::BF16ToF32` <- `Ltx2FuseLoraIntoTensor` <- `Ltx2LoadDitFromSafetensors` <- `Ltx2VideoEngine::Load`, one thread at 99.9% of one core with 19 idle, and an f32 working set growing 9.432 -> 10.235 GiB over 300-629 s = **2.3% of one pass in 10.4 minutes**, cross-checked against the sum of `out*in*rank` over the 1660 targeted modules = 8.53e12 MAC, consistent with ~0.53 GFLOP/s. The operation is a rank-`r` GEMM and belongs on the `vt::` GEMM seam like every other projection in the tree; the arithmetic is already a correct mirror of `fuse_loras.py:103-116` (`B * strength` rounds to bf16 BEFORE the product, f32 accumulation, bf16 store) and only the execution strategy is wrong, so a replacement has a bit-exact oracle rather than a tolerance. Blocks every LoRA-bearing pipeline kind on the full model; `one_stage` is unaffected because upstream marks it `Full` with no adapter, which is why it is the only full-model arm that currently reaches generation. Owed by [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) `## Owed`, whose §5 already frames "why the decode is single-threaded and on the host" | perf | +| [#1208](https://github.com/mudler/vllm.cpp/issues/1208) | — | `Linear` in the LTX-2.5 text tower (`src/vllm/model_executor/models/ltx2_text_encoder.cpp:60-71`) is a scalar single-threaded triple loop with a `double` accumulator and both operands widened per multiply, so it cannot reach the f32 FMA path even if the compiler vectorised it. On the full model's caption projection (in_features on the order of 1.9e5) cost is `rows * out_features * in_features` and it dominates the pre-generation wall: measured on `dgx`, resident memory went flat at t≈150 s and stayed **byte-identical for 1073 s** with stacks in the text feature extractor. Because `ltx2-gen` prints nothing between load and completion ([#1010](https://github.com/mudler/vllm.cpp/issues/1010) owns that), this presents to a user as a hang rather than as slow arithmetic, and it is why an earlier probe's "reached Generate" claim was retracted — the trace does not support it. TWO separate defects: the execution strategy, which belongs on the `vt::` GEMM seam; and the `double` accumulator, which is NOT a mirror — `torch.nn.functional.linear` on f32 inputs accumulates in f32, and the comment directly above this function cites `F.linear` as its reference, so the widening diverges from the oracle it names, cannot be bit-compared against upstream, and hides reduction-order differences an f32 accumulator would expose. Per the dtype-polarity rule an f64 accumulator kept deliberately needs a one-line reason beside it. Every LTX-2.5 pipeline kind goes through the text tower, so this is on the critical path of all of them including `one_stage`. Owed by [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) `## Owed` | perf | +| [#1210](https://github.com/mudler/vllm.cpp/issues/1210) | — | The two-stage LoRA rebind cost that `src/vllm/multimodal/ltx2_video.cpp:2843-2851` records as "UNMEASURED on real weights", with a later perf row owning the number, is now measured. A two-stage recipe loads FUSED, phase 0 asks `Ltx2PhaseLoraScope::kNoAdapters` and `Ltx2RebindDitLoras` un-fuses, phase 1 asks `kAllAdapters` and re-fuses — so the load-time fusion is **provably wasted**, undone before any denoise step runs, and the DiT is left fused so the next render pays the same two again. At [#1202](https://github.com/mudler/vllm.cpp/issues/1202)'s measured ~0.53 GFLOP/s each pass is hours, and a two-stage full-model render pays three of them before the first step. TWO independent fixes, not one change: making the fusion fast (#1202) shrinks the constant but leaves the wasted round trip; separately, `Ltx2PipelineRecipe::phases` is available before `Load` runs, so the load could honour phase 0's scope and skip the fuse/un-fuse entirely. The terminal fused state is chosen rather than forced and is worth revisiting in the same change. Affects `a2vid_two_stage` and `ti2vid_two_stage`; `one_stage` pays nothing. NOT CLAIMED: the wall-clock figures are a rate measured over a 10.4-minute window and extrapolated, not a completed pass — no two-stage full-model render has completed, so the end-to-end number stays open; the rate, thread count and stack attribution are measured. Owed by [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) `## Owed` | perf | diff --git a/.agents/specs/ltx25-decode-speed.md b/.agents/specs/ltx25-decode-speed.md index 87f3bd1c9..d8000c1c6 100644 --- a/.agents/specs/ltx25-decode-speed.md +++ b/.agents/specs/ltx25-decode-speed.md @@ -1418,6 +1418,9 @@ authority and no fresh review (§8). | [#1021](https://github.com/mudler/vllm.cpp/issues/1021) | 5c — DiT staging is 7.5 min at 70.5 MiB/s, against 127.8 on rung 2; GPU idle, 0.15 cores | owed | | [#1024](https://github.com/mudler/vllm.cpp/issues/1024) | 0 — the GPU is idle for the WHOLE post-load render, not only the decode | owed; carries the owed `utilization.gpu` positive control | | [#1040](https://github.com/mudler/vllm.cpp/issues/1040) | none — the evidence for rungs 1 and 2 and for §1.4 is on an unreachable host, and neither rung's sampler cadence closes | owed | +| [#1202](https://github.com/mudler/vllm.cpp/issues/1202) | 7 — `Ltx2FuseLoraIntoTensor` is a scalar single-threaded loop, ~0.53 GFLOP/s | owed, MEASURED on the full model: 2.3% of one pass in 10.4 min | +| [#1208](https://github.com/mudler/vllm.cpp/issues/1208) | 8 — the text tower's `Linear` is scalar, single-threaded and `double`-accumulating | owed, MEASURED at 1073 s of byte-identical RSS; also a dtype-polarity divergence from `F.linear` | +| [#1210](https://github.com/mudler/vllm.cpp/issues/1210) | 9 — the two-stage rebind fuses, un-fuses and re-fuses; the load-time pass is provably wasted | owed; supplies the number `ltx2_video.cpp:2849` records as UNMEASURED | **[#1008](https://github.com/mudler/vllm.cpp/issues/1008) is NOT owed here. It landed.** It was filed by this row as lever 2 and taken by `LTX25-DECODE-DTYPE`,