LTX-2.5 text-tower Linear is a scalar single-threaded double-accumulating loop; dominates pre-generation wall and reads as a hang

[localai-bot]

Ltx2TextFeatureExtractorForward's Linear is a scalar, single-threaded, double-accumulating triple loop. On the full LTX-2.5 text tower's caption projection it dominates wall time before generation starts, and it is the reason a full-model render appears to hang between "engine loaded" and the first denoise step.

Where

src/vllm/model_executor/models/ltx2_text_encoder.cpp:60-71:

for (int64_t r = 0; r < rows; ++r) {
  const float* xr = x.data() + r * w.in_features;
  for (int64_t o = 0; o < w.out_features; ++o) {
    const float* wr = w.weight.data() + o * w.in_features;
    double acc = has_bias ? static_cast<double>(w.bias[o]) : 0.0;
    for (int64_t i = 0; i < w.in_features; ++i)
      acc += static_cast<double>(xr[i]) * static_cast<double>(wr[i]);
    out[r * w.out_features + o] = static_cast<float>(acc);
  }
}

One thread, no blocking, no SIMD. The accumulator is double and both operands are widened per multiply, so it cannot vectorise into the f32 FMA path even if the compiler tried.

Observed

On dgx (GB10, 20 cores) running one_stage on the full/dev transformer: after "engine loaded", resident memory went flat at t≈150 s and stayed byte-identical for 1073 s with no progress output, in a phase whose stacks land in the text feature extractor. ltx2-gen prints nothing between load and completion, so this presents to a user as a hang rather than as slow arithmetic.

The caption projection is the large one (in_features on the order of 1.9e5), and cost here is rows * out_features * in_features — the shape where a scalar loop is least affordable.

Two separate problems

1. Execution strategy. This is a plain GEMM. It should route through the existing vt:: GEMM seam like every other projection in the tree, not be hand-rolled. Same defect class as #1202 (Ltx2FuseLoraIntoTensor), and the two together are most of the pre-generation wall on a full-model render.

2. The double accumulator is not a mirror of upstream. torch.nn.functional.linear on f32 inputs accumulates in f32, not f64. The comment directly above this function cites F.linear as the reference, so widening the accumulator diverges from the oracle it names. It is not obviously wrong — wider is usually closer to exact — but it means this path cannot be bit-compared against upstream, and it hides reduction-order differences that a f32 accumulator would expose. Whichever way it is resolved, it should be a stated decision with a reason beside it, per the dtype-polarity rule in AGENTS.md.

Suggested resolution

Route through the vt:: GEMM seam, and decide the accumulator question explicitly against the pinned oracle. If f64 is kept deliberately, annotate it with the reason; if it moves to f32, the existing goldens will say whether anything shifts.

Blast radius

Every LTX-2.5 pipeline kind goes through the text tower, so this is on the critical path of all of them, including the only full-model arm that currently runs (one_stage).

[localai-bot]

Measured on the full 21.004 B model, GB10, 2026-08-18

First run in which the full/dev transformer was actually loaded (evidence: ltx2-gen: family=ltx-2.5 dit=…dev-transformer-bf16.safetensors, examples/ltx2_gen/main.cpp:491, reachable only past the null check at :486).

Configuration: one_stage --device cuda, 1024x576/25f, 42,018,190,584-byte dev transformer from local NVMe, --video-stg-guidance-scale 0 --audio-stg-guidance-scale 0 --a2v-guidance-scale 1.0 --v2a-guidance-scale 1.0.

The shape of the wall

Sampling at 1 Hz for 6482.6 s (108 min), CPU% of a 20-core box:

window	max cpu	reading
t≈115 s	1991%	~20 cores — weight load / device staging
t≈300-1800 s	101%	one core, ~25 min
t≈2100 s	1995%	~20 cores, brief
t≈2100-3600 s	101%	one core, ~25 min
t≈3900 s	1995%	~20 cores, brief
t≈3900-6483 s	101%	one core, 43 min and still running at kill

Three short multi-core bursts separated by long single-core stretches. Roughly 93 of the 108 minutes — about 86% of wall — ran on one core of twenty, with RSS frozen at 30.993 GiB throughout each stretch.

Peak system memory 79.662 GiB at t=3858.7 s; MemAvailable never fell below ~46.8 GiB against a 12.0 GiB floor, so this is not memory pressure. Load completed in ~144 s, and system peak during load was 74.59 GiB, reproducing an earlier probe's 74.39 GiB within 0.2 GiB.

No output was produced in 108 minutes. No video, no audio, no gov.txt.

What this does NOT establish

The run reported dit_runs=0 and gpu_edges=0 throughout, and both counters were broken, so neither is evidence:

• eu-stack attached and listed TIDs but unwound zero frames, so every phase label degraded to OTHER and no stack-based forward count was possible.
• gpu_pct read nan on every sample. Cause: the probe queried utilization.gpu,memory.used in one call; GB10 reports memory.used as [N/A] under unified memory, the parse threw, and the handler discarded both values — an available counter destroyed by an unavailable one sharing its parse.

So I cannot say whether a DiT forward ran. The three multi-core bursts are consistent with phase boundaries but are not attributed to a symbol. Claiming "never reached a denoise step" would be reading a verdict out of a broken instrument.

What it does establish

The single-threaded stretches match this issue's signature exactly — one core pinned, RSS flat, no progress output — and they dominate the wall on the full model. Whatever occupies them, an 86% single-threaded fraction on a 20-core host is the controlling cost of a full-model render, and it is resolution-independent, so rendering smaller does not shrink it.

Practical consequence: a full-model LTX-2.5 render is not currently reachable in a bounded GPU lease. Four attempts have now failed to produce a frame; this one had correct staging, a verified binary, ample memory and a loaded model, and still produced nothing in 108 minutes.

Owed next

The instrument fix is already applied on the share (the nvidia-smi queries are split so one [N/A] cannot discard the other). The attribution gap needs a working unwinder or an in-process phase marker — #1010 owns per-render progress output, and its absence is why this presents as a hang rather than as slow arithmetic.

[localai-bot]

Corrections and a sharper result

Three corrections to my comment above, plus the finding that supersedes its framing.

1. The run was not ended by a timeout, the guard, or a render failure. rc-worker-4b8lj shows RESTARTS: 1. The worker pod restarted and took the job with it; the guard wrote neither # ABORT nor # exit= because it was SIGKILLed with the container, and the client logged connection refused. The third single-core stretch was still running. So 6482.6 s is a lower bound, not a duration.

2. MemAvailable's true minimum is 39.97 GiB at t=3859 s, not the ~46.8 GiB I quoted — that was the steady-state tail, read off the end of the log rather than minimised over it. Still far above the 12.0 floor, so the conclusion is unchanged, but the number above was wrong.

3. Single-core fraction is 95% (6175 s of 6478 samples), not the ~86% I estimated from 300 s buckets.

The structure, and why it points somewhere specific

window	duration	cpu	rss
t=1-114	103 s	one core	0.63 → 69.38 GiB (load)
t=114-142	28 s	multi, peak 1995%	→ 30.31 (device staging)
t=142-1882	1740 s	one core	32.78 → 41.28 GiB
t=1882-2043	161 s	multi	41.87 → 31.13
t=2043-3781	1738 s	one core	32.58 → 41.14 GiB
t=3781-3894	113 s	multi	42.14 → 41.79
t=3894-6483+	2589 s+	one core	flat 30.99 (still running)

Two single-core stretches of 1740 s and 1738 s — 0.11% apart. #1087 records a "~1731 s resolution-independent single-threaded phase"; both match it within 0.5%. That phase is paid at least twice per render, and this is the first trace showing it.

The third stretch has a different signature — rss flat at 31 GiB where the paired ones climb ~8.5 GiB (32→41) — so it is plausibly a different operation rather than a third repeat. That rss delta is the discriminator for identifying it.

This weakens the attribution to this issue

Labelled as hypothesis, not fact: two stretches of equal cost fit the guided render's two text-conditioning passes (positive and unconditional — cfg_scale 3.0 ≠ 1.0 forces the unconditional branch at ltx2_pipeline.cpp:521-523) better than they fit this issue's caption projection. The projection's two project() calls have a 2:1 size ratio (4096 vs 2048 out_features) and so should not produce equal-duration stretches.

So the 95% single-threaded fraction is established, and its attribution to this function is not. Whoever takes this row should confirm attribution by measurement before assuming the fix lands here. eu-stack attaches and lists TIDs but resolves zero frames in this container, so stack sampling is not available — an in-process phase marker is likely needed, which is #1010's territory.

Arithmetic for whoever fixes it

From the checkpoint configs: flat = 49 × 3840 = 188,160; per-row cost = (4096 + 2048) × 188,160 = 1,156,055,040 MACs. Sequence length is unknown, so the achieved rate inverts to 0.34 GMAC/s (B=1, T=512) up to 5.4 GMAC/s (B=2, T=4096) over 1738 s. The low end sits beside Ltx2FuseLoraIntoTensor's measured ~0.53 GFLOP/s (#1202) — consistent with a scalar double-accumulating loop that cannot reach the f32 FMA path.