feat(BACKEND-ROCM): register a ROCm attention backend for kROCM - #1056
Merged
localai-bot merged 45 commits intoAug 18, 2026
Conversation
The engine-level attention registry had no kROCM entry, so SelectAttentionBackendName threw on ROCm while the runtime kernel path worked. Add a name-only RocmAttentionBackend (ROCM_ATTN) registered for kROCM, mirroring the Metal/Vulkan/Tenstorrent rows, and give the ROCm platform the upstream rocm.py priority lists so dense requests resolve to ROCM_ATTN. The backend reports the NHD KV layout this runtime allocates and the stride-driven kernel reads, deliberately not upstream's (2, num_blocks, ...) shape, with the deviation documented in the header. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4 [Freebuff]
…d called it success (mudler#1029) (mudler#1035) mudler#967 taught `IsCudaKeepQuantSupported` to return true for `kIQ1_S` and `kIQ1_XXXS`. Three dispatch switches consume that predicate, and mudler#967 extended one of them. `MatmulBTQuantGroupedKernelCuda` used the same predicate to skip its CPU fallback, then dispatched through a `switch (w)` with no case for either dtype and no `default:`. It quantized the activation, launched nothing, and returned. `CheckCuda(cudaGetLastError())` reported success, because a launch that never happened cannot fail, so the output tensor kept whatever it already held. An independent review measured that on GB10 through a poisoned output buffer: both IQ1 encodings left -12345 in place, at NMSE 4.58e6 and 9.96e6 against the CPU oracle, while the `iq2_s` control passed. That path is the default routed-expert path of Qwen3.8-2.4T, and the two encodings are 96.92 % of it, so mudler#967 turned correct-but-slow into silently wrong. The fused MoE SwiGLU seam had the same hole, where a NAMED refusal became silence. Both grouped switches gain the two arms. No kernel code was needed: the grouped templates are generic in `W` and depend only on `DotSuperblock<W>` and `FinalFactor<W>`, which the dense path already specializes. The `default:` that now throws on all three is the more useful half. Past that predicate there is no fallback left, so any future missing case is a silent no-op unless something refuses out loud. It landed green because the grouped dispatch had no test at all: two files in `tests/` mention `MatmulBTQuantGrouped` and neither mentions `kCUDA`. Both grouped seams are now driven over the dense gate's case table, against the CPU grouped golden, through a poisoned output buffer. `cuda_quant_iq_tables.cuh` claimed from the day it landed that a runtime test memcmps the device tables against the CPU ones. No such test existed, and nothing read `vt::cuda::d_iq1s_grid` at all: the CPU tests digest the host symbols, which are different objects in a different address space. Replaying the CUDA gate's own mt19937(0x5EED) stream, 266 of 2048 `d_iq1s_grid` entries (13.0 %) are never addressed, so drifting entry 0 is caught and drifting entry 3 is green at 150032/150032. That figure came from the review and was re-derived here rather than quoted: the dense gate's widest weight is 128 blocks and 4096 grid draws, giving 1782 distinct entries and 266 untouched. The seal now exists and covers all eight device codebooks byte for byte. Both false comments are corrected. The same replay measured one thing in the other direction: the new grouped gate builds a 64-row weight, 512 blocks and 16384 draws, and that reaches all 2048 entries. Recorded as a fact rather than as a closure. It is coverage by accident of shape, one edit away from shrinking again, and it says nothing about the seven other tables. ## The poison earns its place It is not redundant with the value comparison. Mutating `MatmulBTQuantKernel` to write nothing leaves the golden AND the independent reconstruction both at the poison value, so `memcmp` passes with ZERO failures and only the poison assertion fires. | mutation | applied | compiled | result | |---|---|---|---| | baseline (unmutated) | n/a | yes | 1 case, 112 assertions, 0 failed, SUCCESS | | M1 grouped golden writes nothing | 2 lines | yes | FAILURE, 110 failed assertions | | M2 grouped golden ignores `eids[p]` | 2 lines | yes | FAILURE, 11 failed assertions | | M3 grouped golden ignores the broadcast row | 2 lines | yes | FAILURE, 11 failed assertions | | M4 `MatmulBTQuantKernel` writes nothing (golden AND reference) | 1 line | yes | FAILURE, exit 1, **memcmp failures 0**, poison failures 55 | Each mutation was restored byte-exact (`cmp -s` against a pre-mutation snapshot), and the tree is clean against `HEAD` afterwards. `applied` is measured against that snapshot rather than against `HEAD`, because the tree already carries the fix. A CPU arm of the same probe runs on every host, because both CUDA cases return early without a device and doctest prints SUCCESS for a gate that asserted nothing. It takes this box from 0 assertions to 112. ## What did NOT run, and why **The CUDA arms of the new gate have not run on a device.** `dgx.casa` was unreachable for the whole of this work (`ssh: connect to host dgx.casa port 22: No route to host`; `ping` 100 % packet loss, retried), this box has neither a CUDA toolkit nor an NVIDIA device (`nvcc not found`, `nvidia-smi` absent, no `/dev/nvidia*`), and there is no second CUDA host. So: - The two new CUDA test cases and the codebook seal are UNVERIFIED at runtime. - The confirmation that removing either new `case` turns the grouped gate red is UNVERIFIED at runtime. It follows statically from the `default:` that throws, and from the poison assertion for the pre-fix shape, but it has not been run. - The `cuda-fat-build` CI job compiles `cuda_quant_dot.cu` with nvcc over ten SM targets, so the production edit gets a compile check here. It configures with `-DVLLM_CPP_BUILD_TESTS=OFF`, so the CUDA-only test block does not. Issue mudler#1029 stays OPEN for the GB10 run rather than being closed by this PR. ## Not done: `-Wswitch` for CUDA The issue proposed it. It would not have caught this defect. `-Wswitch` is silent whenever a `default:` label exists, and a `default:` is exactly what this change adds. The flag that fires is `-Wswitch-enum`, which warns on every enum switch in the tree that omits an enumerator even with a default. That is a tree-wide change with no measurement behind it, on a compile lane this box cannot run. Recorded here and in the spec rather than done. ## Gate - Full CPU build, CI configuration (`-DVLLM_CPP_BUILD_TESTS=ON`, no build type, so asserts are live): exit 0, zero `error:` lines. - Full `ctest`: **100 % tests passed, 0 failed out of 492**, exit 0, 710.36 s. Two did not run and both are pre-existing environment skips (`test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`). - Focused: `test_cuda_quant_dot` 10 cases, 112 assertions, 0 failed. The CUDA cases return early on this host, which is why the CPU arm exists: without it the same binary reports SUCCESS on 0 assertions. - This branch replaces mudler#1034, whose history carried a `git merge` commit with git's default message and therefore no trailers. Rewriting it would have meant a force-push, so the work was cherry-picked onto `origin/main` instead. The resulting tree is byte-identical: `git rev-parse HEAD^{tree}` on both heads is `dbc2bb620d5a1b6da356a25218f1c850ea6fbbb1`, so the `ctest` run above is a run of exactly this tree. - `scripts/agent-preflight.sh`: **All gates green**, including `check-agent-record`, `commit-trailers` and `commit-style`. - New: `src/vt/cuda/cuda_iq_table_seal.h` declares the seal's copy. It is a gate instrument, called only from `tests/vt/test_cuda_quant_dot.cpp`, and it exists because a host translation unit cannot take the address of a `__device__` array. Named here rather than left to be discovered. Refs mudler#1029, mudler#912. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…ain out of red (mudler#1057) `scripts/check-public-doc-tables.py` exits 1 at `origin/main`: docs/BENCHMARKS.md carries 36 prose paragraphs against a 35 budget and docs/FEATURES.md 22 against 21. The checker runs in the `pre-push` hook and at `.github/workflows/ci.yml:160`, so the red is not confined to one branch. Every push in the repository is refused, including pushes that touch neither page. The two paragraphs arrived at `e34d71379` (mudler#1054), a two-character Apple Clang capture fix that also wrote one narrative paragraph into each page. Nothing about that fix was wrong. The pages simply had no paragraph left to spend. Each paragraph moves into the keyed row its content belongs to, which is what the checker's own message prescribes: "content belongs in table ROWS and prose only explains them". The Apple Clang build disposition becomes a `Darwin Qwen3.5 build repair` row in the docs/BENCHMARKS.md `Open gaps` table, beside the rows that already record NOT APPLICABLE and no-number-owed dispositions. The Apple Clang platform fact folds into the `Metal (Apple Silicon)` row of the docs/FEATURES.md backend table, which is where a macOS toolchain fact is keyed. Nothing is deleted, and the `max_prose_paragraphs` constants are untouched, because raising one is a checker semantic change that owes its own row, spec and red-before test. That redesign is real and it is somebody's. A whole-page paragraph count on a shared file is exactly the shape AGENTS.md Records rejects ("Limit an entry, not a shared file"), and `ENG-RECORD-CONFLICT-SURFACES` already scopes the removal of the doc-gating global counters. This change does not attempt it. It restores the gate and leaves the argument where its spec holds it. ## The information survived docs/BENCHMARKS.md, before: > **Darwin Qwen3.5 build repair (2026-08-16).** Benchmarking is NOT APPLICABLE. The change removes a redundant namespace-scope lambda capture that Apple Clang rejects under `-Werror`; it does not change generated refusal text, model math, or any runtime path. The binding gate is the Apple Clang build. after, as a row of the `Open gaps` table: | Track | Status | Next gate | |---|---|---| | Darwin Qwen3.5 build repair (mudler#1054, 2026-08-16) | **NOT APPLICABLE.** Removing a redundant namespace-scope lambda capture that Apple Clang rejects under `-Werror` changes no generated refusal text, no model math and no runtime path | None. The binding gate is the Apple Clang build | docs/FEATURES.md, before: > The Qwen3.5 MoE loader also builds under Apple Clang with project warnings promoted to errors. Its layout-refusal path uses the same messages and behavior on every platform. after, folded into the existing `Metal (Apple Silicon)` row of the backend table, whose `vllm.cpp` cell was the bare `✅`: | Backend | vllm.cpp | vLLM | SGLang | llama.cpp | |---|---|---|---|---| | Metal (Apple Silicon) | ✅ builds under Apple Clang with project warnings promoted to errors, the Qwen3.5 MoE loader included; its layout-refusal path uses the same messages and behavior on every platform (mudler#1054) | ☐ | ☐ | ✅ | ## Evidence Every exit status below was captured directly, never after a pipe. | Step | Result | |---|---| | Red before, at `0f8580e269ceac5f4174c92cfdf79b386980b26b` | exit 1, naming BENCHMARKS 36/35 and FEATURES 22/21 | | Green after, at `1ab285acb` | exit 0 | | Armed, docs/BENCHMARKS.md +1 prose paragraph | exit 1, "36 prose paragraphs, over the 35 budget" | | Armed, docs/FEATURES.md +1 prose paragraph | exit 1, "22 prose paragraphs, over the 21 budget" | | Restore after each mutation | `sha256sum -c` OK on both files, checker back to exit 0 | | `pre-push` hook on this commit | exit 0, pushed without `--no-verify` | | `pre-push` hook on red `main`, hand-fed the same stdin | exit 1, `check-public-doc-tables.py FAILED on 0f8580e` | Both pages now sit at exactly their budget, 35 of 35 and 21 of 21, which is where they sat before mudler#1054 and is the standing cost this checker's docstring already records. Keyed records, this branch against `origin/main`: docs/BENCHMARKS.md keeps 188 of 188 keys byte-identical and adds one, `Darwin Qwen3.5 build repair (mudler#1054, 2026-08-16)`; docs/FEATURES.md keeps 193 of 194 byte-identical, changes only `Metal (Apple Silicon)`, and adds and removes none. `.agents/issue-index.md` was rebuilt from `origin/main`'s file with one row appended: `origin/main`'s bytes are a byte-identical prefix, 275 rows and 275 unique ids. Widths against the entry caps: the new cells measure 181 and 187 characters against `MAX_CELL_CHARS = 220`, and the rows 285 and 227 against `MAX_ROW_CHARS = 600`. The 220-character cell that both pages already carry is untouched. Checkers, each exit captured directly: `check-public-doc-tables.py` 0, `check-agent-record.py` 0, `check-issue-index-append-only.py` 0, `check-now-current.py` 0, `check-commit-style.py --range origin/main..HEAD` 0, `check-commit-trailers.py --range origin/main..HEAD` 0, `check-doc-checkpoint.py --commit 1ab285a` 0. That last one is armed too: `--commit b5618b3` exits 1 on the known USAGE.md miss. No build was run. This change edits two markdown pages and an append-only index, so a compile would prove nothing about a paragraph count, and this is stated rather than left for the reader to assume. Closes mudler#1055. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…d one (mudler#1009) (mudler#1041) Lever 3 of the `LTX25-DECODE-SPEED` investigation (mudler#1006, PR mudler#1038 — mudler#1018 was the earlier pull request and is now closed). Closes mudler#1009. `ParallelForRows` (`src/vt/cpu/cpu_threadpool.cpp:413`) is synchronous and 10+ CPU kernels in this tree dispatch through it. Zero of them were in the LTX-2.5 conv video VAE decode, whose 42 convolutions carry ~7.25 TFLOP at 448x256/25f and ran on one core of twenty. Three sites now dispatch: `CausalConv3d`'s output nest, its padding gather, and `Linear3d`. ## The axis is the whole risk, so it is argued at the site The sibling dtype row (mudler#1008, `d1b0ea3a8`) had to change this convolution's summation **order** to a blocked one to stay inside a 5e-06 tolerance. Parallelism is the second thing that can change a summation order. The partition is the output line `(oc, ti, hi)`, `out.w` contiguous elements. `Volume::At(oc, ti, hi, wi)` is `((oc*t + ti)*h + hi)*w + wi`, so row `r` is exactly `[r*out.w, (r+1)*out.w)` of `out.data` — no element is written twice — and the entire `ci * kernel^3` reduction stays inside one output element's body in the blocked order mudler#1008 shipped. A worker therefore executes the serial arm's instruction sequence, in the serial arm's order, on the serial arm's values, for every element it owns. The result cannot depend on the worker count **or on which worker stole which chunk**, and the second half matters: `ParallelForRows` steals through an atomic cursor, so the row-to-thread assignment is genuinely non-deterministic run to run. Splitting the reduction axis `ic` into per-thread partials would also be a legal convolution. It is rejected in the comment at the site, because it would make the summation order a function of the thread count. ## The numerics did not move at all Both suites were rebuilt with `kLtx2GoldenTol` set to `0.0` before the change and again after, so every golden reports its `max|diff|` rather than its verdict. **All 34 recorded margins — 23 in `test_ltx2_vae`, 11 in `test_ltx2_tiling` — came back byte-for-byte identical**, compared by diffing the two sorted lists rather than by eye: `VAE_MARGINS_IDENTICAL (23 values)`, `TILING_MARGINS_IDENTICAL (11 values)`. | golden arm | before (serial) | after (20-thread global pool) | tol | |---|---|---|---| | Conv video decoder | 1.72853e-06 | 1.72853e-06 | 5e-06 | | non-causal Conv video decoder | 2.08616e-06 | 2.08616e-06 | 5e-06 | | norm_eps-binding video decoder | 1.54972e-06 | 1.54972e-06 | 5e-06 | | tiled decode, untiled control A | 2.74181e-06 | 2.74181e-06 | 5e-06 | | tiled decode, untiled control B | 2.80142e-06 | 2.80142e-06 | 5e-06 | | every other arm in both suites | unchanged | unchanged | — | Those before-values are also the ones `ltx25-decode-dtype.md` §8.1 recorded on its own host, which is an independent check that this box reproduces the sibling row. No tolerance was touched. That table is itself a threading gate: the suite runs on the global pool, `hardware_concurrency` wide, so every LTX-2.5 video golden after this change executes on 20 workers, and the "Conv video decoder" fixture carries a `res_x_y` block so `Linear3d` and `conv_shortcut` are on that path too. ## Two cases, because one of them measures nothing on its own A thread-count A/B is green on a serial implementation. Shipping only that would have been a test that passes while measuring nothing. * **"the decode DISPATCHES its convolutions to the CPU threadpool"** reads the pool's public work-stealing cursor through `ChunkAdd(0)`, which is a non-mutating read. A fresh pool reads 0; a pool that has run a partitioned dispatch reads at least `nth`. Asserting 0 *before* the decode is the instrument's own positive control. **Before this change it fails `CHECK( 0 > 0 )`.** * **"the decode is BIT-IDENTICAL across thread counts"** decodes the same latent at 1, 2, 3, 5 and 8 workers and `memcmp`s every arm against the 1-worker one, which short-circuits to the pre-change serial path. **3 and 5 are there because `nchunk` derives from `nth * 4`, not from `nth`** — 45/30/18/12 are the chunk strides at this fixture's 360 `conv_in` output lines, four different partitions of the same output. It is NOT that 3 and 5 fail to divide the row counts: 360 and 15 are both divisible by each. That was the comment's original claim and it was false; see the repairs below. Both enter through `Ltx2VideoDecodeStreaming` — what the render path calls at `src/vllm/multimodal/ltx2_video.cpp:3258` — and both assert an analytically derived value of exactly **7**, not a recorded one, because mudler#1008 recorded that a zero-filled stub satisfies an expectation of zero. ## The CPU A/B Same binary, `VLLM_CPP_CPU_THREADS` the only variable, one decode through `Ltx2VideoDecodeStreaming`, 14 runs per count across an ascending and a descending sweep so an ordering drift shows as spread rather than hiding in a mean. | threads | runs | min s | median s | max s | spread | speedup | efficiency | |---|---|---|---|---|---|---|---| | 1 | 14 | 1.9859 | **2.0418** | 2.1172 | 6.4% | 1.00x | 100% | | 2 | 14 | 1.0266 | **1.0552** | 1.0843 | 5.5% | **1.93x** | 96.7% | | 4 | 14 | 0.5464 | **0.5555** | 0.5674 | 3.8% | **3.68x** | 91.9% | | 8 | 14 | 0.2920 | **0.3013** | 0.3072 | 5.0% | **6.78x** | 84.7% | | 16 | 14 | 0.2129 | **0.2232** | 0.2597 | 21.0% | **9.15x** | 57.2% | | 20 | 14 | 0.2024 | **0.2234** | 0.2534 | 22.8% | **9.14x** | 45.7% | A second shape at the checkpoint's real `base_channels` of 128: 5.1015 s at one thread against 0.5276 s at twenty, **9.67x**. That is the weakest number here — `n = 3` against the table's 14, no min/median/max, same contended box — so it corroborates the table's shape at a second channel width and is not independently a three-significant-figure result. The public records carry `~9x at 16-20 workers` with the conditions rather than either decimal. **The load it was taken at.** One-minute load average 4.03 to 6.77 on a box whose one-minute average had been between 2 and 94 the same day, with one non-agent process holding ~1.07 cores throughout. That process is part of why 16 and 20 spread 21-23% where everything at or below 8 spreads under 7%. **No ceiling is declared.** The implied serial fraction at 9.14x on 20 workers is 6.3%, which is the right order for `PixelNorm`, `Silu`, `ApplyAdaLn`, the residual add and `expand` — every one still serial, every one listed under `## Owed`. Memory bandwidth is the second candidate and is not separated here. **Determinism, proven a second time:** the output checksum was bit-identical across all **84** A/B decodes — six worker counts, two sweep directions, two shapes — on pseudo-random weights rather than the engineered fixture. ## What is NOT claimed No end-to-end render speedup, no ratio against any oracle, no composition figure with mudler#1008. There is no GPU here, `dgx.casa` was unreachable throughout, and `ltx_core` is not installed. The harness shape is synthetic and says so; what generalises from it is the scaling, not the absolute wall. ## ThreadSanitizer, with the instrument controlled first `RelWithDebInfo` + `VLLM_CPP_SANITIZE=thread`: `test_ltx2_vae` 42/42, `test_ltx2_tiling` 10/10, `test_ltx2_video` 57/57, all `EXIT=0`, zero `WARNING: ThreadSanitizer`. Two instrument problems had to be settled before that meant anything. The binaries would not start at all — `FATAL: ThreadSanitizer: unexpected memory mapping`, `EXIT=66`, an ASLR-against-shadow-layout failure that a `&&` chain would have read as a race; `setarch x86_64 -R` fixes it. And a sanitizer that reports nothing is indistinguishable from one that is not instrumenting, so a deliberate unsynchronised write was compiled into `CausalConv3d`'s parallel body in the same lane: **87** `WARNING: ThreadSanitizer: data race`, `EXIT=66`, then reverted, rebuilt, and back to 0 and `EXIT=0`. ## Mutations, three facts each | mutation | numstat | built | exit | detected by | |---|---|---|---|---| | **T0** — all three dispatches reverted | 13/8 | yes, 0 errors | **1** | the dispatch case, `CHECK( 0 > 0 )` | | T1 — `CausalConv3d`'s output loop alone | 3/2 | yes, 0 errors | **0** | **nothing. 42/42 and 10/10 pass** | | T2 — the padding gather alone | 5/3 | yes, 0 errors | **0** | **nothing. 42/42 and 10/10 pass** | | T3 — `Linear3d` alone | 5/3 | yes, 0 errors | **0** | **nothing. 42/42 and 10/10 pass** | | D1 — chunk-dependent value, visible at 1 worker | 1/0 | yes, 0 errors | **1** | 10 cases + 2 tiling cases | | **D2** — the same defect INVISIBLE at 1 worker | 1/0 | yes, 0 errors | **1** | the bit-identity `memcmp`, on all four non-base arms | | **R** — production call site deleted | 17/2 | yes, 0 errors | **1** | the dispatch case on the cursor AND the value; the identity case's non-degeneracy `REQUIRE` | | T1, first attempt | 4/2 | **NO, 45 errors** | — | **nothing — a mutation that does not build establishes nothing** | **T1's failed first attempt is in the table on purpose.** One unbalanced brace closed the anonymous namespace early and produced 45 `-Werror` errors that read as unrelated `unused-function` complaints hundreds of lines away. The runner refused to draw a verdict rather than running a stale binary and printing a plausible 42/42. **T1, T2 and T3 are an honest gap and it is owed.** One work-stealing cursor is shared, so reverting any single site leaves the other two dispatching and the case reads non-zero. It gates *"at least one of the three sites dispatches"*, and T0 is what holds the conjunction. T3 additionally cannot be seen by that fixture at all, since `Linear3d` is only reached through a `res_x_y` block. What does bound each site is the golden table above — the "Conv video decoder" arm reaches all three at 20 workers and did not move — and the wall-clock, which is what a serial convolution would actually cost. **D1 is beside D2 because it is the weaker of the two.** D1 perturbs the first row of every chunk including the first, so the 1-worker arm moves too and the case fails on its value assertion before reaching the `memcmp`. D2 perturbs only chunks that do not start at row 0, which is invisible at one worker, so the `memcmp` across worker counts is the only thing that can report it. It does, on all four. ## Gate `cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF`, `-j6`, `ctest -j4`. Run twice, the second at this branch head so a green gate chains to the push. `CONFIGURE_EXIT=0`, `BUILD_EXIT=0`, `: error:` count **0**, `ctest -N` **492**, `CTEST_EXIT=0`, **100% tests passed, 0 tests failed out of 492** in both runs (308.99 s and 316.87 s). Two pre-existing skips, `test_modelopt_mixed_precision_checkpoint` and `test_voxtral_e2e`. `No space left` **0** and `BFD` internal-error/assertion **0** across every log, both greps positive-controlled against a synthetic file carrying the real message forms — 1 and 2 hits there, 0 in the real logs. `check-doc-checkpoint --commit` green on each of the three commits and armed (`b5618b305` exits 1). Load average 32-52 on the first run and **82-94** on the head run, on a shared 20-core box; none of the load-dependent suites flaked in either. Free disk 21-30 GiB of 447 GB; the 834 MiB sanitizer tree was removed after use. ## One deliberate omission **No `.agents/issue-index.md` row is appended for mudler#1009.** That row already exists at `.agents/issue-index.md:279` on PR mudler#1038, branch `row/LTX25-DECODE-SPEED-R2`, which filed the issue and is unmerged. mudler#1018 was the pull request that carried it first; mudler#1018 is closed and mudler#1038 supersedes it. `.gitattributes` sets `merge=union` on that file and `scripts/check-agent-record.py` refuses a duplicate issue number, so a second copy here would turn `main` red for every branch the moment mudler#1038 merges — which is what a duplicate mudler#995 row did on 2026-08-16. The sibling dtype row made the same call for mudler#1008. The link lives in the spec and in this body; the index link arrives with mudler#1038. One index row IS appended by this branch, for the new issue mudler#1044 below, and mudler#1044 is not among the ids mudler#1038 appends. ## The review, and the five findings repaired after it A fresh reviewer returned **PASS with no blocking findings**: the gate reran at 492/492, reduction safety was verified from `Volume::At`'s index arithmetic, determinism was proven by a mutation caught at all four worker counts, ThreadSanitizer was clean against an 84-race positive control, the keyed records were proven key by key, all nine anchors landed exactly, and a correctly-shaped zero-filled buffer fails both new cases, so the zero-stub trap is closed. Five non-blocking findings followed. Each was re-verified before it was repaired, because a finding is a hypothesis; **none of the five was rejected**. | # | Finding | Verified how | Repair | |---|---|---|---| | F1 | The T1/T2/T3 gap was argued in §8.6 prose with no issue and no `## Owed` entry | §7's table has no per-site row; `ParallelForRows` seeds ONE pool cursor (`cpu_threadpool.cpp:438`, advanced `:455`), so two surviving sites keep it non-zero | Filed [mudler#1044](mudler#1044), added it to §7 `## Owed` and to the issue index, owned by `LTX25-DECODE-THREADS` | | F2 | `FEATURES.md` and `USAGE.md` carried `9.14x`/`9.67x` bare, against a 21-23% spread | Cell widths measured with `check-public-doc-tables.py`'s own parser | `~9x at 16-20 workers` with the conditions; the bare `9.67x at c=128` is gone from `FEATURES.md` | | F3 | The mudler#1009 index row was cited at `:275` on PR mudler#1018 | mudler#1018 is **CLOSED**; the row is at line **279** of the index on `row/LTX25-DECODE-SPEED-R2` (PR mudler#1038) | Citation corrected here and in the spec, and the two other spec pointers at the closed pull request with it. The DECISION is unchanged | | F4 | §8 cited evidence at `dac85969c`, which does not resolve on the branch | `git merge-base --is-ancestor dac85969c HEAD` exits **1** | §8 cites `d653f7319` and states why the measurement transfers | | F5 | The determinism case's stated reason was arithmetically false | 360 and 15 are both divisible by 3 and by 5; the strides are 45/30/18/12 | The comment now states the real mechanism and says outright not to "fix" the row counts | **F1 is owed, not implemented.** mudler#1044 carries the closing test the reviewer supplied — a per-dispatch `Threadpool::RunCount()` bumped in `Run()` and an EXACT expected count rather than `> 0`, plus a fixture carrying a `res_x_y` block because `Linear3d` is unreachable with `decoder_blocks` empty. A new gate needs its own red-before evidence and its own fresh review, so it is a row. **The index row names its owner rather than leaning on `## Owed`, deliberately.** `owed_issues()` in `scripts/check-agent-record.py` splits on a bare `\n## Owed` and this spec's heading is `## 7. Owed`, so nothing listed there is visible to the unowned ratchet. Measured: the unowned count is **33 before and after**, against `UNOWNED_HIGH_WATER = 33`. Four other specs have the same numbered heading (`ltx25-decode-dtype`, `ltx25-token-append`, `nemotron-h-a2q1-fp8-mamba`, `nemotron-h-a2q2-nvfp4-moe-lmhead`); that is a record observation this row does not repair. **Cell widths, since `MAX_CELL_CHARS = 220` binds and the `BENCHMARKS.md` cell sat at exactly 220.** `BENCHMARKS.md` LTX-2.5 axes 220 to **212**; `FEATURES.md` decode-threading 210 to **204**. Both measured with the checker's own `_table_rows`, and `check-public-doc-tables.py` exits 0. **Keyed records, proven key by key against the merge base.** `BENCHMARKS.md`: 179 unrelated keys byte-identical, only `LTX-2.5 axes` changed, none added or removed. `FEATURES.md`: 194 unrelated keys byte-identical. `USAGE.md`: 200 unrelated keys byte-identical, no table row touched. `.agents/issue-index.md`: `origin/main`'s version is a **byte-identical prefix**, exactly one appended line, and it is mudler#1044. **Nothing was re-measured.** `dgx.casa` is down and the A/B harness is deliberately not in the tree, so every wall-clock figure above stands as the implementer recorded it. No end-to-end render speedup is claimed here either. **Gate after the repairs.** `CONFIGURE_EXIT=0`, `BUILD_EXIT=0`, `: error:` count **0** on a full 1449-target build, `ctest -N` **492**, `CTEST_EXIT=0`, **100% tests passed, 0 tests failed out of 492** in 165.47 s, the same two pre-existing skips. `No space left` **0** and `BFD`/internal-error **0** across both logs, each grep positive-controlled against a synthetic file carrying the real message forms (2, 1 and 2 hits there, 0 in the real logs). `check-doc-checkpoint --commit` green on all four branch commits and armed: `b5618b305` exits **1**. One-minute load 10 to 26; free disk 40 GiB falling to 21 GiB. ## Pushed with `--no-verify`, and why The `pre-push` hook refuses this branch on `check-public-doc-tables.py`: docs/BENCHMARKS.md has 36 prose paragraphs, over the 35 budget docs/FEATURES.md has 22 prose paragraphs, over the 21 budget **This branch did not cause it.** Matched-arm check: `origin/main` alone, in a detached worktree with no branch content, fails with the *identical* numbers. Bisected to `e34d71379` (mudler#1054, an AppleClang capture fix that also added +5 lines to BENCHMARKS and +4 to FEATURES); `283c7e492` immediately before it exits 0. Filed as mudler#1055. AGENTS.md: "Hooks are bypassable convenience, not evidence", and "a commit that needs an exception argues for it in its own message". This is that argument. The same checker run against this branch's own edits is clean -- it changes exactly one key in BENCHMARKS (`LTX-2.5 axes`) and adds exactly one in FEATURES (`LTX-2.5 Conv VAE decode threading`), with every unrelated key proven byte-identical to `origin/main`. ## Gate at the merged tree `CONFIGURE_EXIT=0`, `BUILD_EXIT=0`, `: error:` 0, 493 targets linked, `ctest -N` 495, **494 of 495 passed**. The one failure is `test_serve_low_tools`, which is **mudler#428** ("the concurrency-cap assertion races the server-side counter and reads 3 under load") -- it passes 3/3 when re-run alone, and this branch touches no serve or tools file. Attribution verified against the issue that names the test, not assumed from a family. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…t kernels parallelised, and the same song comes out (mudler#672) (mudler#1061) FOLLOWING_AGENTS_PROTOCOL Spec §11.4 named this as owed and said exactly why it was not in the device arm's change: *"a bit-identity claim needs its own measurement"*. This is that change. It is **arm-independent** — it helps every user, with or without an accelerator, and it is the only one of §11.4's three items that helps a user who has no GPU at all. ## What moved Three host-reference kernels now partition their OUTPUT elements across the one threadpool `vt::cpu` already owns (`src/vt/cpu/cpu_threadpool.h`, the 1:1 ggml port). Nothing is rerouted, no dtype narrows, no tolerance is touched. | kernel | share of its half | partitions by | |---|---|---| | `vocoder1d::ConvTranspose1d` | **88.5 % of the acoustic half** | output channel | | `vocoder1d::Conv1d` | 7.7 % of the acoustic half | output channel | | `music3::LinearNoBias` | **42-57 % of the AR half** | flat (row, out) element | `ConvTranspose1d` and `Conv1d` are **shared** — MiniMax-H3's audio VAE, LTX-2's audio VAE, BigVGAN and IndexTTS-2.5 all call them — so this is not a Music3 private path, and their numerics had to stay byte-for-byte where they were. ## The correctness claim, in its strong form **A whole generated song is BYTE-IDENTICAL.** `minimax-music3-gen` against the real 28.5 GB checkpoint, `--duration 0.1 --steps 2 --seed 7 --device 0`, identical lyrics and description, built once from `d9441ef3` and once from this tree: ``` base-0.1.wav 12332 bytes sha256 12452152876072b280a7a2551dd182731a8475decc625758de28c345f194de9d new-0.1.wav 12332 bytes sha256 12452152876072b280a7a2551dd182731a8475decc625758de28c345f194de9d cmp: no difference ``` Not "the tolerances still pass", not "the RMS agrees to five digits" — the same bytes, through five stages and three touched kernels. Every gated Music3 number was taken on this path, so a path that emits identical bytes has not moved one. **And the four full-scale real-weight gates reproduce their recorded counts value for value**: loader 21/1413, `acoustic_real` 6/76, `llm_real` 4/220, `ar_real` 4/894 — exactly what spec §11.5 recorded before this change. ## Why no reduction order can move `LinearNoBias` and `Conv1d` were ALREADY indexed by their output: each element owned one sequential `double` accumulator, and it still does, walked in the same ascending order. `ConvTranspose1d` needed the real argument, because it was a SCATTER — but a destination accumulator is only ever reached from its own group's inputs, so pivoting to `(dst_c, ic, t, k)` leaves the sequence of additions into any one accumulator exactly as it was, and the `value == 0.0` skip is a property of `(ic, t)` that moves with them. `-ffp-contract=off` is untouched. A size guard keeps the AR half from getting slower: below 2^16 scalar multiply-accumulates the body runs inline, because that half already spends ~25 % of its wall clock inside `Threadpool::Barrier` and a sub-microsecond dispatch is not worth a kick. It moves WHERE a body runs, never what it computes. ## The gate, and the finding a first draft of it would have missed `test_host_parallel` — **7 cases / 861 assertions** — compares each shipped kernel against a VERBATIM copy of its own pre-parallel loop, carried in the test file, at five thread counts, with BITWISE equality. The oracle is the old code, not the new code at another thread count: a consistently reassociated sum is still consistent. **THE FINDING: a `double` accumulator stored through a `float` cannot see a reduction-order change, so the obvious version of this gate is green under the exact defect it exists to catch.** Splitting `LinearNoBias`'s dot into two interleaved accumulators — the textbook reassociation — left every ordinary-shape assertion GREEN, and so did reversing `Conv1d`'s input-channel walk. A reassociated sum of well-scaled terms differs by ~2^-53 relative while the store rounds at 2^-24; the narrowing swallows it. Same class as the recorded "bf16 store absorbs reduction-order defects", one dtype up. Two cases restore the teeth, each added because a mutation stayed green: `LinearNoBias` with `+2^30`/`-2^30` at taps 0 and 1, and `Conv1d` with a `-2^40` bias against a `+2^40` first tap. The serial walk cancels immediately and accumulates the remainder exactly; any other walk carries the big magnitude through it. A second leg exists because bit-identity alone is satisfied by never parallelising at all: the guard case asserts the body actually ran on more than one thread, deterministically (`ParallelForRows` seeds worker `ith` with chunk `ith` and the grid is 4x-oversubscribed, so for every thread count here `nchunk > nth`). ### Mutations: 8 applied, 7 RED, 1 unmoved and explained | # | mutation | result | |---|---|---| | M1 | `LinearNoBias` dot split into two interleaved accumulators | **RED** (5) — only once the cancellation case existed | | M1b | `LinearNoBias` drops the first term of every dot | **RED** (15) | | M2 | `ConvTranspose1d` walks its group's `ic` descending | **RED** (5) | | M3 | `ConvTranspose1d` walks `k` descending | **GREEN, correctly** — those taps land in DIFFERENT accumulators, so the order between them is not a reduction order. Recorded rather than counted, because it says what the gate does not claim | | M4 | size guard hard-wired to run inline | **RED** (4) on the thread-distinctness leg only, which is why that leg exists | | M5 | guard drops the last row of every range | **RED** (114) | | M6 | `ConvTranspose1d`'s reused per-thread scratch not cleared | **RED** (24) | | M7 | `Conv1d` walks `ic` descending | **RED** (5) — again only once its cancellation case existed | M4 was **invalid as first written**: deleting the guard's use of `work_per_row` tripped `-Werror=unused-parameter`, so the compiler refused it and the gate never got to speak. A build failure is not a red gate; it was re-run keeping the parameter used. That is spec §9.4's trap, hit again. Sources restored and verified `sha256`-identical after every mutation, final rebuild green. ## Speed — PENDING, and said so rather than fudged The wall-clock pair is **not reported here**. The two runs that exist are not comparable: the checkpoint is mmap'd from a CIFS mount, so the first run of a series pays a 27 GB fault-in no later run pays (`d9441ef3` 369.5 s **cold** vs this tree 311.8 s warm), and a second series had another session's full `ctest` land on the box mid-run — 1-minute load average **76.6 on 20 cores** — which voided its `--duration 0.4` pair too. A contention-guarded re-measurement (waits for two consecutive quiet samples with no foreign compiler or test binary, then alternates arms and takes two samples of each, `uptime` on both sides) is running and lands in its own commit together with `.agents/benchmark-record.md`, `docs/BENCHMARKS.md` and `docs/STATUS.md`. **The correctness axis is CLOSED; the speed axis is PENDING.** Spec §12.4. ## Gates — all green, with CASE and assertion counts `assertions: 0` is a skip wearing a pass, so the checkpoint-gated suites are listed twice: without the checkpoint (skip) and with it. ``` test_host_parallel 7 / 861 test_minimax_music3_loader 21 / 1413 * test_vocoder1d 10 / 58 test_minimax_music3_ar 25 / 338 test_bigvgan 6 / 65 test_minimax_music3_acoustic 27 / 265 test_minimax_h3 79 / 57395 test_minimax_music3_quant 29 / 125 test_indextts2_family 7 / 22 test_minimax_music3_speech 9 / 223 test_indextts2_render 3 / 14 test_minimax_music3_ar_real 4 / 894 * test_indextts2_pipeline 8 / 433 test_minimax_music3_acoustic_real 6 / 76 * test_ltx2_vae 40 / 3097 test_minimax_music3_quant_real 6 / 319 test_speech_engine 11 / 38 test_minimax_music3_llm_real 4 / 220 * test_speech_api 6 / 67 test_wavenet 3 / 133 test_openai_api_server 62 / 727 test_codec_encoder 6 / 176 test_capi 65 / 653 ``` `*` = run with `VLLM_CPP_MUSIC3_CHECKPOINT` / `CHECKPOINT_ROOT` set; without them `ar_real` and `llm_real` report `4 / 0` and `test_indextts2_e2e` reports `1 / 0`, which are skips and are named as such rather than counted as passes. x86-64 CPU Release build, `-DVLLM_CPP_BUILD_TESTS=ON -DVLLM_CPP_SERVER=ON -DVLLM_CPP_TRITON=OFF`. ## Known-red, checked against a matched arm rather than assumed * `check-public-doc-tables` reds on the BENCHMARKS/FEATURES **prose-paragraph budgets**. Reproduced byte-for-byte on pristine `origin/main` `0f8580e26`, and already filed as [mudler#1055](mudler#1055) (`mudler#1054` pushed both pages over). The pre-push hook refuses on it, so this branch was pushed with `--no-verify` — deliberately, over a budget this change did not move, exactly as the hook's own message offers. * `windows-msvc-*` is [mudler#968](mudler#968), an LTX-2.5 `C4244`, untouched here. * `agent-record` via `check-env-doc` on `VT_MOE_EXPERT_STREAM_*` is [mudler#995](mudler#995) / [mudler#1000](mudler#1000). `scripts/agent-preflight.sh --staged --no-require-role` is otherwise clean: `doc-checkpoint`, `now-current`, `commit-trailers` and `commit-style` all OK. ## What remains OWED (spec §11.4, unchanged by this PR) * `vt::ConvTranspose1d` as a real op with **CPU and CUDA providers**, and the vocoder routed through it. `vt` has no transposed 1-D convolution of any kind, and `vt::Conv2d` / `vt::DepthwiseConv1d` are CPU-only, so the stage that is 88.5 % of the acoustic half still has no CUDA kernel behind any op it could route through. Three consumers, so it is its own change. * The RVQ depth decoder and the 2.4B fp32 DiT onto `vt::MatmulBT` with device-resident weights. Issue: mudler#672 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
… the guided denoiser nothing had (mudler#1005, mudler#1013) (mudler#1032) `T2AOneStagePipeline` (`t2a_one_stage.py:43`, `__call__` at `:109` @ `fd4ded7f`) renders a soundtrack and no picture. This is the first path here that returns a `VideoResult` with zero frames, and the first that runs the DiT with `video = nullptr`. Issue [mudler#1005](mudler#1005). Spec [`.agents/specs/ltx25-t2a-one-stage.md`](.agents/specs/ltx25-t2a-one-stage.md). Also files and fixes [mudler#1013](mudler#1013) and [mudler#1039](mudler#1039) in the same flow, and files [mudler#1031](mudler#1031), which is CLOSED as a duplicate of [mudler#1022](mudler#1022) and whose index row is corrected here — see *The mudler#1031 row was stale before it landed* below. Four more are FILED AND NOT FIXED here, each because fixing it needs something this branch does not have, and each therefore naming its owner: [mudler#1048](mudler#1048) (the LTX-2.5 checkpoint pin, which needs a GPU and a real checkpoint), [mudler#1049](mudler#1049) (`Ltx2Guidance` dead in production, pre-existing from mudler#641), [mudler#1050](mudler#1050) (the guider rescale's `std` comment, same provenance) and [mudler#1052](mudler#1052) (`test_engine_core_proc`'s load-dependent shutdown case, unrelated engine code). All eleven are linked from `.agents/issue-index.md` and from this body; mudler#1005, mudler#1013 and mudler#1039 are in the spec's scope and the other four are under its `## Owed`. ## What changed since the first review Two things, both from the fresh review of `3d9d9c9bb`. **[mudler#1039](mudler#1039): the guidance was combined in VELOCITY space, and upstream combines x0.** This was a defect on the DEFAULT arm, in code that had not landed. It is fixed here, with the RED captured, and it is the subject of the two new sections below. **The mudler#1031 index row was stale.** It said `check-agent-record` is RED on `origin/main`; that was repaired by `ff264cb82` (PR mudler#1025) before this branch merged it. The row is corrected in place, which is possible only because it has not landed yet. ## What changed since the SECOND review The fresh review of `c1fe35592` passed on the correctness of the mudler#1039 fix and returned one blocking finding, one record obligation and four prose items. All are addressed below. It also measured two pre-existing defects that this branch deliberately does NOT fix; both are filed and owned. **BLOCKING: the mudler#1039 gate covered ONE of the three guidance arms.** `ltx2_t2a.cpp:41-43` says `to_denoised` is applied to EVERY PASS. The gate held that claim for the CONDITIONAL pass only: it recorded `first_step_velocity` and `first_step_cond` for that arm, nothing observed the unconditional or perturbed forwards, and nothing pinned what `Ltx2EulerStep` consumed. The default T2A arm runs three forwards per step, so a build that converts `cond` correctly and leaves either other arm in velocity space renders a different waveform through a guider whose `cond` term is impeccable, with a healthy forward count and nothing else to see it by. That is mudler#1039 again, one arm over. Reproduced at `c1fe35592` before the repair, on the same comma-free filter as the green run (`--test-case=ltx2 t2a*`, 10 cases / 526 assertions / exit 0). Each mutation applied to ONE file, `git diff --stat` taken against the PRE-MUTATION working tree rather than against `HEAD` (the repair is uncommitted while the harness runs, so a diff against `HEAD` would report it too and the stat would stop being the mutation's own), rebuilt with the `: error:` count printed beside the verdict, exit code captured DIRECTLY, and restored from a content SNAPSHOT with `os.utime(now)` and a sha256 compare. | Mutation | `git diff --stat` | BUILT | before | after | |---|---|---|---|---| | A1 the PERTURBED (STG) pass alone left in velocity space | `ltx2_t2a.cpp \| 4 ++--` | YES (0 errors) | **SURVIVED** exit 0, 10 cases / 526 | DETECTED exit 1, 10 / 548 | | A2 the UNCONDITIONAL pass alone left in velocity space | `ltx2_t2a.cpp \| 4 ++--` | YES (0 errors) | **SURVIVED** exit 0, 10 / 526 | DETECTED exit 1, 10 / 548 | | A3b `ToDenoised` applied twice, BELOW the step-0 record (the reviewer's R1b) | `ltx2_t2a.cpp \| 2 +-` | YES (0 errors) | **SURVIVED** exit 0, 10 / 526 | DETECTED exit 1, 10 / 548 | | A3c `ToDenoised` applied twice, ABOVE the step-0 record | `ltx2_t2a.cpp \| 1 +` | YES (0 errors) | **SURVIVED** exit 0, 10 / 526 | DETECTED exit 1, 10 / 548 | | A4 the perturbed arm's recorded velocity ZEROED (the guard, not a defect) | `ltx2_t2a.cpp \| 1 +` | YES (0 errors) | the field did not exist | DETECTED exit 1, 10 / 538 | | N1 the original mudler#1039 shape, restored in full | `ltx2_t2a.cpp \| 5 ++---` | YES (0 errors) | DETECTED | DETECTED exit 1, 10 / 548 | A3c is not from the review. It was found while closing A3b: the reviewer's placement sits between the step-0 record and the Euler step, so recovering the Euler input sees it, and moving the same edit one statement earlier does not. Closing both needs two independent checks rather than one. N1's first draft dropped `ToDenoised`'s only call site and failed to build on `-Werror=unused-function`, at 1 compile error. **A mutation that does not build reads as a passing test**, so it is rewritten as two edits that keep the function used. The reviewer's own R1' hit the same trap and therefore proved nothing; that is why every row above prints BUILT and the error count. **The repair is observability plus three checks, not a change to the fix.** `Ltx2T2aResult` and `Ltx2ConditioningTrace` gain a (raw velocity, x0 prediction) pair for the unconditional and perturbed arms, and the latent the Euler step wrote. The uncond and perturbed vectors stay EMPTY when the guider does not ask for that arm, because the forward did not run; a zero-filled one of the right length would be indistinguishable from a forward that returned zeros. Then, all inside the existing end-to-end case through `LoadVideoEngine` and `VideoEngine::Generate`: - the SAME equation `x0 == latent - sigma*velocity` on every arm the render ran, exact in x0 space and off by the whole sample in velocity space, with `t2a_uncond_forwards > 0` and `t2a_perturbed_forwards > 0` asserted first so a silently skipped arm cannot vacate its own check; - the guider's output REPLAYED through the shipped `Ltx2MultiModalGuidance` over the three recorded arms, required bit-equal to `t2a_first_denoised`. This does not gate the guider's arithmetic, which the control case below already does; it gates that the pipeline handed it these tensors and passed its result on UNTOUCHED, which is what A3c moves and no per-arm check can see; - `t2a_first_next_latent` recovered from `t2a_first_denoised` through `x + (x - denoised)/sigma * (sigma_next - sigma)`, the schedule re-derived from `Ltx2SigmaSchedule` and tied to the render by the sigma it recorded. That is what A3b moves. **Non-vacuity, per arm rather than once.** `latent_span > 1e-3` stays shared, since a zero sample makes the two candidate tensors coincide on every arm. Its partner `sigma * velocity_span > 1e-6` moves INSIDE the per-arm loop, because a zero velocity makes `to_denoised` the identity for that arm alone, and "expected zero, and a stub also produces zero" is the trap this campaign has already hit twice. A4 is the mutation that proves the guard is armed rather than decorative: zeroing one arm's recorded velocity takes the case red through the `REQUIRE`, at 538 assertions rather than 548 because the `REQUIRE` aborts the case. The replay check carries its own control (`t2a_first_denoised != t2a_first_cond`, so the guider MOVED what it was handed) and the Euler check carries two (`|dt| > 1e-3`, so the step is not the identity, and `scale > 1e-3`, so the residual bounds something). **The rescale's numeric difference is still NOT asserted, and the reason was re-measured rather than inherited.** `std(cond)/std(pred)` is 1 to printed precision on this fixture, so `factor = 0.7*1 + 0.3` is exactly 1, the rescale is a no-op in BOTH spaces, and the difference term `(factor - 1) * latent` is identically zero. Owed against the real-checkpoint render, unchanged. **RECORD OBLIGATION: the LTX-2.5 checkpoint pin.** [mudler#1048](mudler#1048). `docs/USAGE.md` names six LTX-2.5 artifacts by bare file name with no HuggingFace repo, no revision and no sha256, at `:663-670` and `:2183-2188` on `origin/main` plus the text-to-audio recipe at `:853-857`, where AGENTS.md § *Say which weights, and from where* requires all three per arm. Campaign-wide and pre-existing rather than introduced here, verified rather than asserted: `grep -n sha256 docs/USAGE.md` returns two checkpoint hashes and BOTH belong to MiniMax-Music3 (`:3127`, `:3269`), while MiniMax-H3 (`:1950-1993`) and MiniMax-Music3 (`:3123-3149`) each carry a full table and LTX-2.5 carries none anywhere. **Recorded and deliberately not fabricated**: this row claims no render on real weights, so there is no checkpoint it was gated against to pin. One `## Owed` bullet, one index row, one issue. The recipe's `--audio-vae` is also corrected to `ltx-2.5-audio-vae-bf16.safetensors`, which is what the other two LTX-2.5 recipes on the page name. **Two pre-existing defects the review measured, filed and NOT fixed here.** [mudler#1049](mudler#1049): `Ltx2Guidance` is dead in production and is the only path to `Ltx2CfgDelta` and `Ltx2StgDelta`; `Ltx2BatchedPerturbationConfig` is constructed only in tests. All four landed with mudler#641. [mudler#1050](mudler#1050): the guider rescale's `std` comment claims the biased estimator "would be a small, everywhere, resolution-dependent gain error", and `factor = std(cond)/std(pred)` divides two `std`s over the same count, so the `(n-1)` cancels exactly. The review's biased-versus-unbiased mutation survived because it is an IDENTITY, not because the gate is blind. The code is right; the comment is the defect. **Four prose fixes.** 1. **Spec 6b overclaimed** that this row ends a test-only driver for four symbols. Only `Ltx2MultiModalGuidance` gains a production call site; 6b now carries the measured table and names mudler#1049. 2. **The test comment** at `test_ltx2_video.cpp` said "NO extra is touched either". `T2aGen` sets two extras and `audio_stg_blocks` IS a guider field. Narrowed to the claim that is true and separately pinned: `rescale_scale` is the recipe's own 0.7. The same false claim in this body is corrected below. 3. **The READER ANCHORS relocation reason was false**, corrected here and in the spec's Risks section. 4. **`docs/FEATURES.md`'s mutation figure** moves from "13 mutations, 12 DETECTED" to "18 mutations, 17 DETECTED", the 18th still the `sigmas[0]` identity. ## The merge of `origin/main` `fa3723b85` `origin/main` advanced mid-repair. PR mudler#1038 is records-only: a new spec and thirteen appended index rows (mudler#1006-mudler#1012, mudler#1014-mudler#1016, mudler#1021, mudler#1024, mudler#1040), none colliding with the seven this branch appends (mudler#1005, mudler#1013, mudler#1031, mudler#1039, mudler#1048, mudler#1049, mudler#1050) or with mudler#1052 below. **The union driver's clean result on the index was rejected, and it was wrong rather than merely suspect.** `git merge` reported `Auto-merging .agents/issue-index.md` with no conflict, and the file it produced INTERLEAVES this branch's rows among main's newly appended ones: the first difference is at byte 122253, where main has mudler#1006 and the union result has mudler#1005. So `origin/main`'s file is not a byte-identical prefix of it, and an index that is not a prefix of main's is one a later union merge can duplicate or silently reinstate a row into. Taken instead as main's file WHOLESALE plus this branch's own suffix, with three checks rather than an assurance, and re-verified on the COMMITTED blobs because `check-issue-index-append-only.py` reads committed state only: 1. PREFIX: `HEAD:.agents/issue-index.md`'s first **144213** bytes are byte-identical to `origin/main:.agents/issue-index.md`. The `cmp` itself is armed: flipping one byte inside that prefix reports a difference. 2. SUFFIX: the remaining **11285** bytes are byte-identical to the branch's own append at `22267d794`. 3. COUNT: **277 rows, 277 unique issue ids**. ## mudler#1039 — the guider combines x0, and this port combined velocities Upstream never hands the denoiser the raw velocity model. `DiffusionStage` builds `X0Model(self._prepared_builder().build(device=target, **kwargs))` (ltx-pipelines `utils/blocks.py:480-482`), and `X0Model.forward` returns `to_denoised(audio.latent, ax, audio.timesteps)` (ltx-core `model/transformer/model.py:590-604`), which is `sample - velocity * sigma` (ltx-core `utils.py:39-52`). So `_guided_denoise`'s `all_v, all_a = transformer(...)` (`utils/denoisers.py:188`) already carries DENOISED tensors, and `audio_guider.calculate(cond_a, uncond_a, ptb_a, mod_a)` at `:203` combines those. `Ltx2T2aGenerate` took `Ltx2DitForward`'s velocities straight into `Ltx2MultiModalGuidance` and applied `ToDenoised` once to the result. That is the same function only while `rescale_scale == 0`. `calculate`'s linear terms (`guiders.py:261-266`) are invariant under `x0 = latent - sigma*v`; the rescale at `:268-271` is not. Upstream's `factor` is `std(x0_cond)/std(x0_pred)` and it scales the whole x0, giving `factor*(latent - sigma*v)`, where scaling the velocity gives `latent - sigma*factor*v`. The two differ by `(factor - 1) * latent`, non-zero wherever the latent is — and on this path the state IS the unit-variance noise, so everywhere. `rescale_scale = 0.7` is the shipped T2A default (`utils/constants.py:63`, `utils/args.py:1101-1106`), so **every default render took the divergent branch**. Nothing already gated could see it. The three forward counters, `t2a_video_stream_present`, `t2a_perturbed_blocks`, the latent absmax and the waveform's length, channel count and sample rate are identical between the two forms. **Fixed by moving the conversion, not by moving the rescale**, and that choice is the structural mirror rather than the shorter diff. The per-pass `x0_model` lambda IS `X0Model`: it applies `ToDenoised` on the way out of every forward, so the guider combines x0 and `Ltx2MultiModalGuidance` stays a faithful port of `calculate` over whatever the model returned. Reaching the same numbers by moving the rescale into the guidance seam would put `to_denoised` inside `calculate`, where upstream does not have it, and would leave the seam correct only for this one composition. **The VIDEO arm is unaffected, checked rather than assumed.** `git grep -n Ltx2MultiModalGuidance -- src include` returns exactly ONE production call site, `ltx2_t2a.cpp`. `Ltx2PipelineParams::video_guider` and `Ltx2PhaseRecipe::video_guidance` are recipe fields that nothing reads: the joint driver runs one UNGUIDED forward per step and applies `ToDenoised` to that single velocity (`ltx2_video.cpp:3034-3036`), which is the same tensor in either space because there is no combination to be invariant under. There is no second instance to fix, and there will be one the moment a guided video denoiser is wired. ## mudler#1039 — the test, and what the fixture cannot decide **The reduced fixture CANNOT resolve the rescale's numeric consequence.** That is measured, not assumed. Its DiT responds to the conditioning at ~1e-5 of its own output, so `std(cond)/std(pred)` is 1.0 to 1e-5 in BOTH spaces, both factors land within 1e-5 of 1.0, and the two candidate step-0 predictions sit **7.6e-07 apart against a span of 3.41**. The first draft of the test asserted exactly that difference; its own separation guard refused it. That case would have been GREEN either way, which is the failure this campaign keeps paying for. So the defect is gated at two places: **1. End to end, through the production entry point.** `ltx2 t2a: the guider is handed x0 predictions and not raw velocities` loads through `LoadVideoEngine` and renders through `VideoEngine::Generate`. An earlier revision of this body said "no extra touched", and that is FALSE: `T2aGen` sets `audio_stg_blocks` and a negative prompt, and `audio_stg_blocks` is a guider field. The claim that matters is narrower and true — `rescale_scale` is the recipe's own 0.7, pinned in the case before anything is read off a render, and `audio_stg_blocks` selects which block the perturbed forward skips rather than how the arms are combined. The case pins the EQUATION ``` cond == latent - sigma * velocity ``` between three recorded step-0 tensors. Exact in x0 space; off by the whole sample in velocity space. No fixture scale meets it by accident: a zero sample or a zero velocity makes the two candidate tensors coincide and fails the two `REQUIRE`s that precede it rather than passing it. **2. At the seam, for the numeric consequence.** `ltx2 t2a: rescale_scale 0 is the control because both spaces agree there` runs the real `Ltx2MultiModalGuidance` over both spaces with a non-zero, non-constant latent. MEASURED: relative disagreement **1.50e-07 at `rescale_scale = 0.0`** and **0.352 at the shipped 0.7**. That is what makes 0.0 the control rather than the assertion site. **RED before, from mutation N1 (revert to velocity space):** ``` test_ltx2_video.cpp:5371: ERROR: CHECK( err_x0 <= 1e-5 * latent_span ) is NOT correct! values: CHECK( 3.43642 <= 3.38677e-05 ) logged: sigma = 1 max|latent| = 3.38677 max|velocity| = 0.415609 |cond - (latent - sigma*velocity)| = 3.43642 |cond - velocity| = 0 elements = 3328 test_ltx2_video.cpp:5378: ERROR: CHECK( err_v > 1e-2 * latent_span ) is NOT correct! values: CHECK( 0 > 0.0338677 ) [doctest] test cases: 1 | 0 passed | 1 failed | 66 skipped [doctest] assertions: 16 | 14 passed | 2 failed | [doctest] Status: FAILURE! exit 1 ``` `|cond - velocity| = 0` **exactly** is the finding. GREEN after, same comma-free filter: 1 case, 16 assertions, 0 failed, **exit 0**. **New mutations**, each on ONE file, rebuilt, run, restored in a `finally` with the restore verified by sha256, and `git diff --stat` scoped to the mutated file so the number is the mutation's own: | Mutation | `git diff --stat` | BUILT | exit | verdict | |---|---|---|---|---| | N1 revert to velocity-space guidance | `ltx2_t2a.cpp \| 4 ++--` | YES (0 errors) | 1 | DETECTED | | N2 delete the production call site | `ltx2_video.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED, 2 cases red | | N3 take x0 against a ZERO sample | `ltx2_t2a.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED | | N4 drop the rescale branch entirely | `ltx2_pipeline.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED | N2 is the REACHABILITY mutation: replacing `const Ltx2T2aResult rendered = Ltx2T2aGenerate(req);` with a default-constructed result turns both the new case and the existing render case RED. N4 is why the seam case is not decorative — it is the only one of the four the end-to-end case does not see. Observability added for this: four step-0 tensors and step 0's sigma on `Ltx2T2aResult` and the trace — the sample, the conditional pass's RAW velocity, the tensor handed to the guider, and the guider's result. `first_step_cond` is upstream's own `DenoisedLatentResult.cond` (`utils/denoisers.py:206`). **No GPU result is claimed.** `dgx.casa` is down, so there is no render on real weights, and the 18.17 % figure in mudler#1039 is synthetic-tensor algebra rather than a measurement. The rescale's end-to-end consequence is listed under `## Owed` in the spec, against the real-checkpoint render already owed there. ## The mudler#1031 row was stale before it landed As appended, the row said `check-agent-record` and `test_check_agent_record` are RED on `origin/main` because `.agents/issue-index.md` lists issue mudler#995 twice, and that the repair needs a contract decision plus a checker-semantics spec. It does not. [mudler#1022](mudler#1022) had already read both mudler#995 rows and found neither well-formed, and `ff264cb82` (PR [mudler#1025](mudler#1025)) landed that repair on `main` before this branch merged it at `3d9d9c9bb`. Measured here rather than inferred: `python3 scripts/check-agent-record.py` prints `agent record OK: ENGINE=156 MODEL=377 QUANT=82 KERNEL=51 BACKEND=83` and exits 0. mudler#1031 is closed as a duplicate of mudler#1022. **Corrected in place, and that is a narrow exception argued here rather than a licence to edit rows.** `.agents/issue-index.md` carries `merge=union`: once the row lands it can never be corrected, because an edit to a landed row is duplicated rather than merged. It has not landed. This branch added it, so the net diff against `origin/main` is still additions only, which is what `scripts/check-issue-index-append-only.py --base origin/main` checks. **No row already on `main` is touched.** `check-issue-index-append-only.py --base origin/main --head HEAD` exits **0** on this branch. Its POSITIVE CONTROL — a commit deleting the `mudler#168` row, which is on `main` — exits **1** with `removed: | [mudler#168]...`, so the instrument is armed and not merely quiet. One note on that instrument, because it presents as a verdict about the tree and is not: it diffs `merge-base..HEAD`, so it reads COMMITTED state and is blind to the working tree. Deleting a row of `main`'s in the working tree leaves it printing `OK: issue index append-only` and exiting 0. It has to be run after the commit, and it was. ## The audio-only shape FITS the engine The dispatch that opened this row expected a possible `NEEDS_DECISION` on the entry point. It is not needed, and the reason is upstream's own shape rather than a convenience here. T2A expresses its duration through a placeholder `VideoPixelShape` at 512x512 whose height and width it documents as unused (`t2a_one_stage.py:37-40`), then calls the SAME `DiffusionStage.__call__` every video pipeline calls. So the request shape T2A needs is the request shape `VideoGenParams` already carries. `VideoResult` carries `frame_count` and `audio_path` as independent fields, so an audio-only result is `frame_count = 0`, an empty `frame_dir`, and an EMPTY `mux_argv`: composing an ffmpeg argv over a frame pattern matching no file would hand the caller a command that cannot run. The numerics live in a new translation unit mirroring upstream's own file, reached from `Generate` before any video geometry is resolved. Threading an `is_t2a` flag through the joint driver would put nine new branches inside a function that already runs 1900 lines, and a third of it builds a video stream this pipeline has no counterpart for. ## Three things that fail silently if guessed Two of them were refusals whose stated reasons do not describe this case, and both were re-derived at `332aed738` rather than inherited. **1. `Ltx2DitForward` demanded BOTH streams and blamed the AudioOnly weight contract.** That is a claim about the CHECKPOINT, and T2A never loads one: upstream reads the ordinary AudioVideo FILE through `LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP` (`model_configurator.py:228-239`) and builds an AudioOnly MODULE from the subset. Every line below that guard was already written against `video != nullptr` (`ltx2_dit.cpp:786-869 @ 332aed7`), so lifting it reaches a path the file already had. What remains true, that a checkpoint saved with only the audio subset cannot be materialized, moves to where it is true: the loader, about the file. **2. `enabled = false` is NOT the same shape.** The same message advised it as the substitute. Upstream's predicate is `run_v2a = run_ax and (video is not None and vx.numel() > 0)` (`transformer.py:269`): it tests PRESENCE. A disabled-but-present video stream still feeds video-to-audio cross attention from a latent T2A never meant to exist, and still returns a playable waveform of exactly the right length, channel count and sample rate. Our port mirrors that polarity at `ltx2_dit.cpp:251 @ 332aed7`, so the trap was live here too. **3. The engine had no guided denoiser at all.** One forward per step, no guider parameter read anywhere. Correct for `distilled_two_stage`, which builds a `SimpleDenoiser` upstream too; wrong for T2A, whose CLI defaults are `cfg_scale = 7.0` and `stg_scale = 1.0` (`utils/constants.py:58-66` through `:118`), so `do_unconditional_generation` and `do_perturbed_generation` are both true (`guiders.py:275-281`) and the default path is THREE forwards per step. `Ltx2MultiModalGuidance` was ported, gated, and reached by nothing but its own tests until now. Its three neighbours are NOT ended by this row and an earlier revision of the spec claimed they were: `Ltx2CfgDelta` and `Ltx2StgDelta` are reachable solely through `Ltx2Guidance`, whose only caller is `tests/vllm/models/test_ltx2_pipeline.cpp:710`, and `Ltx2BatchedPerturbationConfig` is constructed nowhere outside that same file ([mudler#1049](mudler#1049)). STG is the one genuinely new numeric: `all_perturbed` on `Ltx2AttentionArgs` is upstream's `use_attention = not all_perturbed` (`attention.py:557`), which replaces the attention output with the raw value projection before `to_out`. `Ltx2DitForward` gains a `perturbations` argument, which is upstream's own parameter on `LTXModel.forward` (`model.py:492`), so this mirrors a signature rather than inventing a seam. `nullptr` is `perturbations=None` and every existing caller is byte-identical. ## The bug this found and fixed in flow (mudler#1013) `OneStagePhase` left `Ltx2PhaseRecipe::noise_scale` at the struct default of **0.0**, and 0.0 is not "no extra noise": `Ltx2GaussianNoise` is `latent + noise_scale * (noise - latent)`, so the state stayed exactly as `create_initial_state` wrote it, which with no initial latent is **all zeros**. A `one_stage` render denoised a zero tensor on both streams. Upstream's `ModalitySpec.noise_scale` defaults to 1.0 (`utils/types.py:110`) and `TI2VidOneStagePipeline.__call__` constructs both specs without it (`ti2vid_one_stage.py:233-239`). The two neighbouring recipes set it explicitly, which is what made the omission legible. No gate saw it because every end-to-end test loads `distilled_two_stage`, and a zero-initialized denoise still returns a finite clip of the right size, frame count and sample rate. Fixed here because the `t2a_one_stage` rows are built FROM `OneStageRecipe` and would have inherited it. **`dmd2` leaves the same field at 0.0 and is NOT corrected by analogy**: its source is vLLM-Omni's `LTX_POSITIVE_ONLY_RECIPE`, which is not checked out here, and a recipe whose upstream nobody read is exactly where a plausible fix lands wrong. Listed under `## Owed`. ## An existing assertion is REPLACED, not widened `tests/vllm/models/test_ltx2.cpp`'s "a single-stream model type is REFUSED" pinned the old refusal's message. The new form pins upstream's actual contract, `transformer.py:259-260` ("At least one of video or audio must be provided"), and is strictly stronger: it also asserts what a one-stream call RETURNS, that the other stream's output vector is EMPTY, and that the audio-only forward is NOT equal to the joint one with the video ignored. The old assertion could not tell a served one-stream forward from a broken one, because both threw. ## Reachability **A production entry point reaches this, and the test enters through it.** ``` include/vllm.h vllm_video_generate -> src/capi/vllm_c.cpp engine->Generate(gen) -> vllm::multimodal::VideoEngine::Generate -> Ltx2VideoEngine::Generate (the audio_only branch) -> Ltx2VideoEngine::GenerateAudioOnly -> Ltx2T2aGenerate -> Ltx2DitForward(..., /*video=*/nullptr, &ain, ...) -> Ltx2MultiModalGuidance -> Ltx2EulerStep -> Ltx2AudioDecoderForward -> Ltx2VocoderWithBweForward -> audio.wav ``` The command-line arm is the same call: `ltx2-gen --pipeline-kind t2a_one_stage`, as a thin ABI client including no internal header. **M1 is the reachability mutation.** Deleting the production call site turns the focused gate RED (exit 1, 4 of 8 cases failed), so the gate measures a capability rather than a class. **What is NOT reachable, stated rather than left to be found.** `pipeline_kind` is a LOAD knob and `--video-extra KEY=VALUE` reaches `VideoModelParams::extras` at `server_main.cpp:492`, so a server started with `--video-extra pipeline_kind=t2a_one_stage` reaches this by static chain. **That chain was read, not exercised** — no test drives a T2A render through `/v1/videos`, and it is reported as unverified rather than claimed. The six per-generation guider extras do NOT reach that route at all, because `VideoGenParamsFromRequest` never forwards `VideoRequest::metadata` to `VideoGenParams::extras` ([mudler#928](mudler#928)). A T2A render over the route therefore takes the recipe's own guider defaults. ## Mutations (M1-M9, the original wave) The four mudler#1039 mutations are in their own section above; these nine are the row's original wave, re-stated unchanged. Focused gate `./build/tests/test_ltx2_video "--test-case=*t2a*"`. Each mutation applied to ONE file, rebuilt, run, restored in a `finally` and the restore verified by **sha256**; the harness rebuilds the restored tree before anything else measures it. Exit codes captured directly, never through a pipe. Filters are comma-free. | Mutation | `git diff --stat` | BUILT | exit | verdict | |---|---|---|---|---| | M1 delete the production call site (reachability) | `ltx2_video.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED, 4 of 8 cases red | | M2 hand the forward a present-but-DISABLED video stream | `ltx2_t2a.cpp` (see note) | YES (0 errors) | 1 | DETECTED | | M3 never run the unconditional forward | `ltx2_t2a.cpp` (see note) | YES (0 errors) | 1 | DETECTED | | M4 ignore `stg_blocks` and perturb EVERY block | `ltx2_t2a.cpp` (see note) | YES (0 errors) | 1 | DETECTED | | M5 `all_perturbed` falls through to ordinary attention | `ltx2.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED | | M6 revert the `one_stage` `noise_scale` (mudler#1013) | `ltx2_pipeline.cpp \| 2 +-` | YES (0 errors) | 1 | DETECTED | | M7 scale the initial latent by `sigmas[0]` | `ltx2_t2a.cpp` (see note) | YES (0 errors) | 0 | **SURVIVED** | | M8 write a frame on the audio-only path | `ltx2_video.cpp \| 1 +` | YES (0 errors) | 1 | DETECTED | | M9 a skipped step RECOMPUTES the conditional forward instead of reusing | `ltx2_t2a.cpp \| 39 +++---` | YES (0 errors) | 1 | DETECTED | **A note on the first fact for four rows, because it reported something misleading and that is worth writing down rather than tidying away.** `git diff --stat` measures against `HEAD`, not against the pre-mutation working tree, so on a run where `ltx2_t2a.cpp` also carried an uncommitted change the stat reported 45-47 lines rather than the mutation's own 1-3. The number is therefore not a measurement of the mutation on those rows. It is kept, with this note, rather than replaced by a prettier one: the fact the protocol asks for is what the command printed. M1, M5, M6, M8 and M9 were measured against a clean file and their stats are the mutations'. **M9 is a mutation for a defect this port ACTUALLY SHIPPED in its first draft**, not an invented one. `should_skip_step` does not mean "skip the guidance and keep the conditional prediction": upstream returns `DenoisedLatentResult.result_or_none(denoised=last_denoised_audio)` (`utils/denoisers.py:85-91`) BEFORE it assembles any pass, so a skipped step runs NO forward and reuses the previous denoised prediction. The first draft ran the conditional forward and used it, which is a whole extra forward per skipped step on a different trajectory, producing a waveform of exactly the right length. Only the forward count separates them, and it is what the new case asserts, with an unskipped control. **M7 survived, and the resolution is the useful part.** It is the mutation a reader coming from another flow-matching sampler expects to be REQUIRED, and it changed nothing. Not a blind gate: an identity. `LTX2Scheduler` starts at `linspace(1, 0, steps + 1)[0] == 1`; the shift map sends 1 to exactly 1 (`schedulers.py:41-45`); the stretch sends it to `1 - (1 - 1)/scale_factor`, again exactly 1 (`:47-55`). `sigmas[0]` is 1.0 for every step count. The identity is now GATED rather than recorded as a survival, and a pin on an identity cannot turn the arm red, so M7 stays survived by construction. **And that gate found a second thing.** `steps = 1` returns `-nan`, on both sides: `one_minus_z` is `[0.0]`, `scale_factor` is 0, and the stretch computes `1 - 0/0` (`schedulers.py:49-54`). Upstream's own arithmetic, excluded from the pin with the reason beside it, and named under `## Owed`. **Two harness notes, because both would otherwise read as verdicts about the code.** A `.pyc` for `scripts/agent-start.py` was truncated to exactly 4096 bytes on this shared box and `agent-preflight.sh` reported `FAIL test_agent_start` with `EOFError: marshal data too short`; removing the file made it pass 20/20. And M4's first form asserted the STG perturbation on a latent filled with a constant: self-attention over identical rows returns a weighted average of identical values, which IS the value projection, so the perturbation was a numeric no-op and the case reported "the perturbation changed nothing" about a correct build. ## Arms | Arm | Disposition | |---|---| | bf16 / f32 safetensors | **ported**, and what the gate runs on | | NVFP4, FP8 (the DiT tower) | **ported by inheritance**, and that is a claim about the LOAD rather than about a render: this path consumes whatever arm `Ltx2LoadDitFromSafetensors` materialized, adds no GEMM and selects no arm. UNMEASURED on real quantized weights, because the GPU was out of bounds | | GGUF k-quants | **not applicable**, and not merely undone. `quantization_factory.py:23-26` enumerates upstream's inference kinds exhaustively as fp8-cast, fp8-scaled-mm, nvfp4-cast and nvfp4-prequant, with `assert_never` at `:50`. No upstream behaviour to mirror, and llama.cpp does not carry this architecture | | int8-convrot | out of scope, unchanged, already refused by name | ## Refused by name, and owed - **The DEVICE arm.** `Ltx2DitForwardDevice` takes both streams by reference throughout, so a one-stream device forward is a rewrite of that function rather than the lifted check the host forward needed. `device != 0` is REFUSED rather than served the host forward behind a device handle. - **Isolated-modality guidance** — there is no second modality to run it over, which is upstream's own reason for pinning `modality_scale` to 1.0. - **The sigma-binned guider factory**, **AUTO duration**, **`max_batch_size`**, the prompt enhancer, a **one-step schedule**, the **`dmd2` `noise_scale`**, and a **real-checkpoint T2A render** (fixtures only; the GPU was out of bounds). - **Value goldens from executed upstream for the T2A COMPOSITION.** The bricks either side have them; the chain does not. - **The rescale's numeric consequence END TO END** (mudler#1039). Gated at the seam (0.352 relative at the shipped 0.7) and at the space (exactly, through the engine, now on all three arms), and NOT on a render, because the reduced fixture's guidance deltas are ~1e-5 of the prediction and both rescale factors land within 1e-5 of 1.0. The real-checkpoint render above is what closes it. - **The LTX-2.5 CHECKPOINT PIN** (mudler#1048), campaign-wide and pre-existing. No repo, no revision, no sha256 for any LTX-2.5 artifact anywhere in `docs/USAGE.md`. Recorded and not fabricated: there is no render on real weights to pin against. - **`Ltx2Guidance`, `Ltx2CfgDelta`, `Ltx2StgDelta` and `Ltx2BatchedPerturbationConfig` are dead in production** (mudler#1049), pre-existing from mudler#641. This row ends only `Ltx2MultiModalGuidance`'s test-only-driver state. - **The guider rescale's `std` comment states an impossible consequence** (mudler#1050). The code is right; the comment is the defect. - **`test_engine_core_proc`'s immediate-shutdown case is load-dependent** (mudler#1052), and until now no issue named it. ## Gate Clean `build/` on the merged tree. ``` cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF cmake --build build -j6 ctest --test-dir build -j4 --output-on-failure ``` Re-run on the tree AFTER the `fa3723b85` merge, from a deleted `build/`: `CONFIGURE_EXIT=0`, `BUILD_EXIT=0`, `: error:` count **0** (the grep armed by a seeded control that returns 1), `ctest -N` **492**. Three `ctest -j4` runs of the full suite, same binary throughout, each `99% tests passed, 1 tests failed out of 492` in ~164 s with `CTEST_EXIT=8`, plus the usual 2 skipped (`test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`). Box load 2.6 to 7.8 across the runs; free disk 21 G at the end, 31 G before the build. **The identity of the failing test rotates**, which is the strongest single fact about it: run 1 `test_engine_core_proc`, run 2 `test_cpu_threadpool`, run 3 `test_engine_core_proc`. Both are on the declared load-dependent list and both pass alone with exit 0 (`Passed 0.03 sec` and `Passed 0.18 sec`). `test_engine_core_proc` was NOT dismissed on an inherited excuse. Measured: **2 failures in 3 `ctest -j4` runs**, **0 in 25 solo runs** on an idle box at load 3.34, **0 in 25 solo runs against 20 spinning processes**, and 0 in two `ctest -R` runs. CPU pressure alone does not reproduce it, so the regime is the `-j4` harness rather than load as such. The failing assertion is `CHECK( abort_seen )` at `tests/vllm/v1/test_engine_core_proc.cpp:481`, which searches for the abort frame over a FIXED budget of 1000 dequeues while a `max_tokens=100000` request keeps the busy loop emitting token deltas — nothing bounds how many frames precede the abort. This branch touches no file under `tests/vllm/v1/` or `src/vllm/v1/`. **No issue named that test, and the earlier revision of this body blamed the wrong one.** mudler#294 is "test_async_llm: reusing an aborted request id races the core abort" — a different defect in a different test. Filed as [mudler#1052](mudler#1052) with the measurements above, indexed, and listed under `## Owed`. A misattributed flake is worse than an untracked one, because the next reader checks the citation, finds an open issue about something else, and stops looking. **No render on real weights is claimed anywhere in this body.** `dgx.casa` is down. `No space left` and `BFD` are both **0** in the build and ctest logs, and each grep has a POSITIVE CONTROL that returns 1 on a seeded file in the same session — the first `BFD` pattern tried returned 0 on the control too, which is a wrong pattern rather than an absence, and it was widened until the control fired. **The `READER ANCHORS` list DID move, and an earlier revision of this body gave a false reason for it.** It said the change only appends at `~3665`, below the last anchored line. Two hunks sit ABOVE it: the `ltx2_t2a.h` include at `@@ -36,6 +36,7`, which shifts every anchor by one, and the audio-only video-VAE exception at `@@ -974,8 +975,21`, which adds thirteen more and moves the last four by fourteen (`@@ -1018,7 +1032,7` is above 1231 too and is net zero). That is exactly why the list reads `782 792 793 855 951 967 969 1060 1085 1190 1231` here against `781 791 792 854 950 966 968 1046 1071 1176 1217` on `origin/main`. The anchors were correctly RE-DERIVED with the test's own walk and `test_ltx2_video` passes **23/23**, so the outcome is right; only the stated reason was wrong, and a false reason is what makes the next reader skip the re-derivation. `check-doc-checkpoint --commit` run on **all 13** commits of this branch, merges included (mudler#573), all exit 0, with the armed control `--commit b5618b3` exiting 1. `check-issue-index-append-only.py --base origin/main --head HEAD` exits 0 on the COMMITTED head, and its control — a real commit deleting the on-`main` `mudler#168` row, built with `git commit-tree` so the worktree never moved — exits **1** with `removed: | [mudler#168]...`. `scripts/agent-preflight.sh` is **All gates green**, including `check-agent-record` (`ENGINE=156 MODEL=377 QUANT=82 KERNEL=51 BACKEND=83`), which the earlier revision of this body reported as known-red — see the mudler#1031 section above for why that is no longer true. **Known-red, each proven pre-existing rather than asserted.** `test_cpu_x86_llamacpp_floor` exits 4 (`NO_QUIET_WINDOW`) under load, which is [mudler#618](mudler#618) rather than a result. `windows-msvc-*` has no `main` baseline ([mudler#584](mudler#584)). `test_ltx2_video` carries a pre-existing LeakSanitizer leak under the `address,undefined` lane ([mudler#1037](mudler#1037), a Gemma-4 rope cache via `DevicePool`), which this change neither introduces nor touches. **One instrument failure, recorded rather than tidied away.** A `.pyc` for `scripts/agent-start.py` was truncated to exactly 4096 bytes on this shared box, and `agent-preflight.sh` reported `FAIL test_agent_start` with `EOFError: marshal data too short`. Removing the file made it pass 20/20. A corrupt byte-cache presenting as a failing gate is the shape where an infrastructure fault arrives as a verdict about the code. ## Operator gate at the final merged tree Re-run by the operator on `3dd490a94` (this branch merged with `b493f4981`), not inherited from the implementer: CONFIGURE_EXIT=0 BUILD_EXIT=0 ": error:" 0 493 targets linked ctest -N 495 CTEST_EXIT=0 100% tests passed, 0 failed out of 495 The merge was gated rather than assumed because both sides touch `CMakeLists.txt`: a clean textual merge of a build file is not a build file that works. It merged to one added line and still carries exactly one `ltx2_t2a` reference, so the new translation unit is registered once. The mudler#1039 guidance gate was verified independently by mutating the perturbed arm back into velocity space: BUILD_EXIT=0 with 0 compile errors, run exit 1, failing on exactly the two per-arm equation checks. A first attempt referenced a lambda the repair had renamed, failed to build with 1 error, and is recorded as establishing nothing. Pushed with `--no-verify`: the pre-push hook refuses every branch because `origin/main` itself fails `check-public-doc-tables` (mudler#1055, caused by mudler#1054 and fixed by mudler#1057). Matched-arm evidence is in mudler#1055 -- `origin/main` alone fails with identical numbers. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…ional CUDA TU (mudler#960) (mudler#991) Closes mudler#960. Closes mudler#989. Spec: [`.agents/specs/vt-fp8-quant-arch-gate.md`](https://github.com/mudler/vllm.cpp/blob/row/VT-FP8-QUANT-ARCH-GATE-960-V2/.agents/specs/vt-fp8-quant-arch-gate.md). Owning row: `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` (mudler#517), whose A2-Q1 unit (mudler#810) is the caller this blocks. Base: mudler#940. ## What was wrong `vt::QuantFp8Static`'s **only** CUDA registration was `src/vt/cuda/cuda_matmul_fp8_cutlass.cu:376`, and `CMakeLists.txt:1668` compiles that translation unit **only when `VT_CUTLASS_FP8_ARCHS` is non-empty**. The kernel body has **no cutlass dependency of any kind** — zero `cutlass`/`CUTLASS` tokens in `QuantFp8StaticKernelCuda` (`:353-370`). It is `out[i] = e4m3_rne_sat(x[i] * (1/input_scale))`, a grid-stride elementwise loop over a hardware convert intrinsic. It shared a TU with the cutlass sm120 fp8 GEMM and inherited that GEMM's arch set. On sm_110 (Thor) that intersection is empty. `cutlass-fp8: DISABLED (no requested arch in [110] provides it)` is the arch's **documented normal profile**, not a misconfiguration. So `OpId::kQuantFp8Static` was not registered for `DeviceType::kCUDA` at all, on that arch and on **every** CUDA arch outside the cutlass-fp8 cell. **Nothing refused first**, which is what made it expensive. The op's GEMM partner `kMatmulFp8CublasLt` **is** registered unconditionally (`src/vt/cuda/cuda_matmul.cu:920`), so `MatmulFp8CutlassD`'s guard — which keys on it — passed. The missing quant then resolved through `src/vt/op_provider.cpp:501` to the portable CPU reference tier, eligible because `CudaBackend::UnifiedMemory()` is true, which dereferenced **device** pointers on the host: ``` [vt reference-tier] op=QuantFp8Static device=cuda has NO native kernel; running the PORTABLE CPU fallback (correct but slow) SIGSEGV ``` Nothing silently dequantized. Nothing refused either. It crashed one call later, under a banner claiming "correct but slow". ## The change | | | |---|---| | `src/vt/cuda/cuda_quant_fp8.cu` | new, unconditionally compiled TU; kernel + helpers + registration moved verbatim | | `CMakeLists.txt:1586` | added to `target_sources(vllm PRIVATE ...)` directly inside `if(VLLM_CPP_CUDA)` | | `src/vt/cuda/cuda_matmul_fp8_cutlass.cu:322` | kernel and registration removed; a comment records why, so the next reader does not move it back | The only edit to the moved code is the local `Check()` message prefix, `matmul_fp8_cutlass` → `quant_fp8`, which stopped being true the moment the code moved. No arithmetic, no dtype dispatch, no dispatch condition is touched. Because the new TU is not in `_VT_CUDA_FEATURE_SOURCES`, `CMakeLists.txt:2231-2236` gives it the full `${VLLM_CPP_CUDA_ARCHITECTURES}` gencode list, which is the point. **Why a new file, not `cuda_matmul.cu` or `cuda_ops.cu`.** Both were real candidates and both were rejected in the spec. `cuda_matmul.cu` already hosts the unconditional fp8 GEMM registrations — the strongest argument for it — but it is the cuBLAS/cuBLASLt GEMM wrapper TU and this is not a GEMM. `cuda_ops.cu` is unconditional, already includes `<cuda_fp8.h>`, and hosts `RmsNormQuantFp8`, literally the fused arm of this same math. The defect, though, is precisely that a kernel's compilation was governed by a feature it does not use, and both alternatives re-create a weaker form of that by binding it to an unrelated file's includes and requirements. A file named for the op makes the invariant readable in the CMake diff, is what the structural checker can assert without inference, and is the natural home as the fp8 activation-quant family grows. This also **restores upstream's own partition**: vLLM builds `static_scaled_fp8_quant` from `csrc/quantization/w8a8/fp8/common.cu` in the unconditional `VLLM_EXT_SRC` list, and gates only its cutlass `scaled_mm` sources behind `CUDA_ARCHS` intersections. ## Two instruments, because neither covers the other **G4, `tests/vt/test_ops_fp8_cpu.cpp`** — the runtime pin, and the stronger claim: it observes the property that matters, `OpRegistered(kQuantFp8Static, kCUDA)`, rather than a proxy. It needs no CUDA *device* (registration is a table fill before `main`), because "which build" is the axis the defect lived on. Its second assertion requires `kMatmulFp8Cutlass` to track `VT_CUTLASS_FP8` instead, so the case proves the two are **independent** rather than merely that one is present. **`scripts/check-cuda-op-arch-gate.py`** — the tripwire. G4 can only speak on a CUDA build **without** cutlass-fp8, and **no CI job produces one**: the GB10 gate host resolves `cutlass-fp8: ENABLED`, where the defect is unreachable by construction, and every other job is CPU-only. G4 would not have caught mudler#960 before it landed; it caught it here only because someone carried a binary to Thor. The checker reads the build description, runs in the ordinary checker lane on every host, and fails at PR time. Four clauses per entry, no inference about what a kernel "needs": HOME (the TU is in the unconditional list), REGISTERED (exactly one live registration), UNGUARDED (preprocessor depth 0, so an `#ifdef VT_CUTLASS_FP8` wrapper is not a pass), EXCLUSIVE (no second `kCUDA` registration elsewhere). C++ is read through `checker_text.normalize_source`, so a commented-out or `#if 0`-ed registration reads as absent. ## Evidence ### Thor, sm_110, no cutlass — the red-before `CUDA target architectures: 110`, `cutlass-fp8: DISABLED`, `CUTLASS not found` on every build. `BUILD_EXIT=0`, `warnings: 0`, `enospc: 0`. Disk 319 G → 318 G. | tree | binary sha256 | `test_ops_fp8_cpu` | |---|---|---| | base `0e1bee42f` | `6b4d4df071a6…` | `test cases: 2 \| 1 passed \| 1 failed \| 2 skipped` · `assertions: 43 \| 43 passed \| 0 failed` · `Status: FAILURE!` · **exit 139 (SIGSEGV)** | | base + G4 only, RED-first | `63b7940e8609…` | G4 isolated: `test cases: 1 \| 0 passed \| 1 failed \| 4 skipped` · `assertions: 2 \| 1 passed \| 1 failed` · `Status: FAILURE!` · exit 1 | | base + G4 + fix | `690bf71448ea…` | `test cases: 5 \| 5 passed \| 0 failed \| 0 skipped` · `assertions: 62 \| 62 passed \| 0 failed` · `Status: SUCCESS!` · exit 0 | | HEAD, fresh tree, clean build | `dc83e683f4fd…` | identical: `5 \| 5 passed \| 0 failed` · `62 \| 62 passed \| 0 failed` · `SUCCESS!` · exit 0 | The base row reproduces mudler#960 verbatim, **including the trap mudler#844 names**: `assertions: 43 | 43 passed | 0 failed` printed beside `Status: FAILURE!`, so anything grepping the assertions line alone reads a crash as green. The RED-first row is the one that matters. G4's first assertion failed — `CHECK( vt::OpRegistered(vt::OpId::kQuantFp8Static, DeviceType::kCUDA) )` → `values: CHECK( false )` — while its second **passed**, `CHECK_FALSE( false )` on `kMatmulFp8Cutlass`, so the case was not vacuous and the arch genuinely lacks the cutlass GEMM. Non-zero case count in both directions. The case name carries no comma: `-tc` splits on commas, and a comma would have selected nothing and reported `SUCCESS!` with exit 0. In the green rows, **G2** — the CPU-vs-CUDA byte comparison, zero tolerance, five scales × 4096 elements — executes on sm_110 for the first time and passes, `bad == 0` throughout, with no `[vt reference-tier]` banner printed at all. That closes the arm `vt-fp8-w8a8-cpu-arm.md` (mudler#468) recorded as owed on a non-cutlass CUDA arch. The last row is a fresh `git archive` into a new directory with no build state, because three incremental rebuilds of one tree do not prove the committed tree builds. ### GB10, sm_121a — unchanged, which is the claim Both builds assert `CUDA target architectures: 121a` **and `CUDA feature cutlass-fp8: ENABLED for [121a]`** in the configure log; a `DISABLED` line would void the result. `BUILD_EXIT=0`, `warnings: 0`, `enospc: 0`, `-j 4`, 2.7 T free. Every binary sha differs between columns, so neither is a stale artifact. | suite | BEFORE (`0e1bee42f`) | AFTER | |---|---|---| | `test_ops_fp8_cpu` | `4 \| 4 passed \| 0 failed` · `60 \| 60 passed \| 0 failed` · `SUCCESS!` | `5 \| 5 passed \| 0 failed` · `62 \| 62 passed \| 0 failed` · `SUCCESS!` | | `test_ops_fp8_cutlass` | `8 \| 8 passed \| 0 failed` · `86 \| 86 passed \| 0 failed` · `SUCCESS!` | identical | | `test_linear_method` | `10 \| 9 passed \| 1 failed` · `97 \| 95 passed \| 2 failed` · `FAILURE!` | identical | | `test_ops_fused_chain` | `10 \| 10 passed \| 0 failed` · `583 \| 583 passed \| 0 failed` · `SUCCESS!` | identical | The only delta is `test_ops_fp8_cpu` gaining exactly G4: +1 case, +2 assertions. The four pre-existing cases and their 60 assertions are untouched. `test_linear_method` fails **identically in both columns**, which is how this run proves it is not ours rather than assuming it: `linear_method: MXFP4 fused gate_up ~= split (numerically) + fused path ran`, `test_linear_method.cpp:247`, `CHECK( after == before + 1 )`, twice — a Marlin dispatch counter on a suite with no fp8 arm. That is mudler#907's `test_linear_method` row at the cutlass-enabled build's shape. **One trap worth reading, because it nearly produced a false result.** The first AFTER build reported 4 cases / 60 assertions — G4 missing from a binary whose sha *had* changed. `tar` restored the test file with its local mtime (06:46 UTC), older than the object compiled during the base build (07:10 UTC), so ninja skipped the compile and relinked only because `libvllm.a` had changed. The suite reported on a binary containing the fix but not the test. The table above is from a rebuild after `touch`. **Verify the case count, not the sha.** ### CPU-only, local Release-equivalent build, `BUILD_EXIT=0`, **0 warnings**. `ctest` **489/489 passed, 0 failed** (2 skipped for absent checkpoints). `test_ops_fp8_cpu` reads 4 / 56 here: G4 compiles out on a non-CUDA build, which is correct and is why it is `#if defined(VLLM_CPP_CUDA)`. `scripts/agent-preflight.sh`: every gate green except `test_cpu_x86_llamacpp_floor`, which exited `NO_QUIET_WINDOW` (4) at loadavg 77.49 while this box was building — the harness refusing to measure under contention, mudler#618, inherited. ### The checker's mutation evidence, executed rather than described `scripts/check-pr-size.py --base <merge-base> --head HEAD` **passes**, which is not a formality: its evidence contract checks the branch into a scratch worktree, runs `tests/scripts/test_check_cuda_op_arch_gate.py` at HEAD (must pass, non-zero case count), then overwrites the checker with a disabled stub and re-runs the same module (must fail, non-zero case count). Both halves are machine-verified. Locally the suite is 14 cases: one per clause, the two "text the compiler never sees" disguises, a live-tree case, and `test_empty_source_list_is_not_a_pass` — because a checker reporting OK on a parse that matched nothing is a green light attached to no measurement. That contract also caught a defect in the first version of the suite: it bound the checker's functions at import time, so the stub produced an ImportError rather than failing cases and the contract reported `semantic evidence did not execute tests` — neither red nor green, the instrument declining to say. Fixed before this branch was pushed. ## mudler#844 is not fixed here This removes one live **instance** and does not address the class. The portable reference tier still accepts `DeviceType::kCUDA` tensors, still dereferences them, and still calls itself "correct but slow" while doing it. The next feature-gated op to lose its native kernel reproduces mudler#960 exactly. That repair is a larger change to the tier and wants its own spec; mudler#844 stays open. ## mudler#989 fixed in flow, because it had to be Registering the new checker's creation mutation means editing `scripts/check-pr-size.py`, and that file's own evidence contract requires its suite green at HEAD. `tests/scripts/test_check_pr_size.py` has been red on `main` since mudler#888 added `.agents/reachability.md` with no path class — and red **silently**: it runs in no CI job and no preflight suite, so the only thing that ever loads it is the checker-evidence contract, which fires only when someone edits a checker. Third instance of the class after mudler#856 and mudler#668, fixed the same way both were: one entry in `PROCEDURE_FILES`, next to its sibling guides, with a comment naming the issue. Wiring that suite into CI is deliberately **not** done here. ## Why this replaces mudler#990 mudler#990 was the same change on `row/VT-FP8-QUANT-ARCH-GATE-960` and is closed, not abandoned. `documentation-checkpoint` is a PER-COMMIT gate and it was right: the commit that edits `CMakeLists.txt` owes `docs/USAGE.md` in that same commit, and mine did not. There is an honest thing to say there, so it is said — a `DISABLED` CUDA feature removes its kernels, not the ops that do not need them, and a `[vt reference-tier]` banner naming an op on a `cuda` device is a defect to report rather than a slow path to accept. A later commit cannot satisfy a per-commit gate, so the history had to be rebuilt. `main` is never force-pushed and neither is a row branch here, so this is a new branch with the corrected history rather than a rewrite of the pushed one. The trees differ by exactly `docs/USAGE.md`, +25 lines (`git diff --name-only` between the two heads). Every measurement above was taken against that tree and none of it is restated from memory. ## Not in scope The kernel's arithmetic, option parsing and dispatch conditions (all moved byte-for-byte); `MatmulFp8Cutlass`, whose arch gate is **correct** because it really is a cutlass sm120 kernel; and wiring NemotronH to anything (mudler#810 / mudler#517 A2-Q1), which this unblocks but does not do. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
… the fingerprints are identical (mudler#672) (mudler#1067) Records the measurement for the parallelisation that landed in mudler#1061 (`591d24ff`). The code was already merged; this is the number and its provenance, which arrived after that squash and would otherwise have been lost with the branch. ## The measurement The vocoder convolution chain, at its real geometry, on the 20-core x86 box — min of 5 interleaved rounds: | | | |---|---| | chain | **13.36 s → 1.25 s = 10.7×** | | `ConvTranspose1d` stages | 1.97×, 1.92×, **19.86×**, **15.98×** | | `Conv1d` | **12.03×** | | `LinearNoBias` | **10.88×** | `uptime` 3.36 before, 12.64 after. ## Why kernel-level and not end-to-end Both e2e attempts are **void**, and this says which rather than quietly reporting the better one: - the base arm ate a cold 27 GB CIFS fault-in — **369.5 s cold vs 311.8 s warm** - the second pair ran while a foreign `ctest` had the box at 1-minute **load 76.6** A contention-guarded series is still running. Its first pair is baseA 464.8 s vs newA 351.8 s and is **not** reported here, because one pair is not a series. ## Bitwise equality, three independent legs No tolerance was set or widened — the claim is **bitwise**, so no tolerance applies. 1. A whole generated song is byte-identical: `base-0.1.wav` / `new-0.1.wav`, 12332 bytes, both sha256 `1245215287607…4de9d`, `cmp` clean, on the real 28.5 GB checkpoint. 2. All six kernels' output **FNV-1a fingerprints match between arms in every round**, at production geometry. 3. The four real-weight gates reproduce their recorded counts value for value: loader 21/1413, `acoustic_real` 6/76, `llm_real` 4/220, `ar_real` 4/894. ## Newly recorded as owed Stages 0 and 1 gain only ~2× because the pivot trades **weight** locality for accumulator locality — fixable by a weight pre-transpose **without touching any reduction order**. That is now written down rather than left as an unexplained asymmetry in the per-stage table. Still owed and unchanged: `vt::ConvTranspose1d` as a real op with CPU **and CUDA** providers plus routing — the 88.5% stage has no CUDA kernel to route to — and the depth decoder plus 2.4B fp32 DiT onto `vt::MatmulBT`. Gate: `check-agent-record`, `check-public-doc-tables`, `check-doc-checkpoint`, `check-commit-trailers` all OK. FOLLOWING_AGENTS_PROTOCOL Issue: mudler#672 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…umber measured a dead cache (mudler#912, mudler#1066) (mudler#1076) The wiring review of `ENG-EXPERT-STREAM` returned FAIL on eleven findings. Four of them were one defect at different depths: the lane was not connected to anything, and nothing could have told you. ## F1, and the number it voided `Qwen35ExpertStream::EndStep()` had no caller anywhere in `src/` or `include/`. Deleting its definition still compiled. `ExpertSlotCache::Acquire` marks every entry it serves `protected_this_step` and only `EndStep` clears that mark, so protection was permanent: once the cache filled, `ColdestEvictable` returned -1, `Acquire` returned slot -1, `Slice` returned nullptr, and `KqExpertSlice` fell back to the mmap path. The step clock never advanced, so the hotness decay, the LFU score, the LRU tiebreak and eviction never ran in production at all. **The published run's own numbers say when it died.** 8000 slots against 2790 slices per token is 2.87 tokens of capacity. Prefill is ONE forward and therefore one step, its working set fit, and its 4.5x TTFT is real and stands. Decode is one forward per token, so from partway through token 3 every slice was refused and served from the mapping, which IS the baseline. The "streaming ON: steady decode unchanged" row measured the lane being off, and it is retracted. The causal claim recorded beside it -- that `EnsureSpan` memcpy's from the mapping and so inherits the fault path -- is a true statement about the code that this measurement did not establish, because the run never exercised the fill path it blames. It is now recorded as plausible and unmeasured. The step boundary lives in `ForwardLayers`, which every MoE entry point funnels through exactly once per forward, as an RAII guard so a step that ends by throwing still ends. `qwen3_moe.cpp` composes the same MoE block from another translation unit and marks its own step for the same reason. ## The gate found something nobody was looking for Closing F3 meant entering through `Qwen3_5Model::Forward` rather than constructing the cache by hand. Its byte-identity case failed on all 160 logits while each arm was internally deterministic, so the two arms genuinely disagreed. `Qwen35ExpertStream` is a process-lifetime singleton and keyed its cache on the tower's host buffer ADDRESS. Its comment stated the premise and drew the wrong conclusion: a tower's base pointer IS stable for the model's life, but the CACHE is not scoped to one model's life. Free a model, load another, and the allocator hands the new towers addresses the old ones held, so the new model's expert resolved to an entry filled from a different checkpoint -- as a HIT, which by contract moves no bytes, so nothing downstream had anything to observe. Instrumenting `KqExpertSlice` to memcmp each slot against the slice it claims to be: **24 towers occupied 21 distinct addresses, and 20 of 222 slices returned another tower's bytes.** Filed as mudler#1066 and fixed with `OwnedTensor::TowerUid()`, a process-unique counter that cannot collide because it never goes backwards. ## The rest **F2.** `EnsureFile` acquires before it preads, because the read needs a destination, so a throw left the key RESIDENT over a slot holding a prefix of the right bytes and a tail of the previous expert. The retry was an ordinary HIT and a hit moves no bytes, so the GEMM multiplied half of one expert spliced onto half of another. The commit that added the pread claimed the opposite. `ExpertSlotCache::Invalidate` undoes the acquisition and returns the slot. **F4.** `GgufFile::SourceOfSpan` had zero coverage and it is the one place in the chain that chooses a FILE. Deleting the sibling walk and forcing `offset = 0` both survived the full gate; both are "wrong shard at a plausible offset", where the pread succeeds and returns exactly the bytes asked for. **F5.** The `MADV_WILLNEED` hint was inert: madvise needs a page-aligned address, GGUF data is aligned to `general.alignment` (default 32), and the return value was discarded. Now aligned, extended and counted. **No speedup is claimed.** **F7.** The singleton used `static T* inst = nullptr; if (inst == nullptr) ...`, not the magic-static idiom fourteen lines above, and double-constructs an ~18 GiB store under two concurrent first calls. Separately the store was sized from whichever slice arrived first, so a UD quant that keeps `down_proj` at higher precision refused its own first down slice mid-decode. **F6, F8-F11.** The spec gains `## Gates Full `ctest`: **490/490 passed, 0 failed**, exit 0. Full `scripts/agent-preflight.sh`: **all gates green**, zero FAIL and zero SKIP. Release build, 825 targets, 0 errors. ## Owed`. `Contains` was byte-identical to `IsResident` and its comment had orphaned `capacity_exhausted()`'s. The size-before-acquire ordering is pinned on a FULL cache, where it is observable. `ReleaseHost`/`AdoptDeviceBytesAsHost` left `mmap_fd` set after clearing `mmap_src`. The `expert >= 0` ternary had an unreachable arm. ## Observability, because none of this was visible One stderr line, every `VT_MOE_EXPERT_STREAM_STATS_EVERY` steps: ```text [expert-stream] steps=N hits=H misses=M evictions=E fills=F bytes=B exhausted=X advised=A ``` `steps == 0` and `exhausted > 0` are exactly the F1 signature. The run that produced the void number printed one line at startup and none afterwards. Taken from the new gate, which is the whole repair in four lines: ```text [expert-stream] steps=1 hits=0 misses=48 evictions=0 fills=48 bytes=52224 exhausted=0 advised=48 [expert-stream] steps=2 hits=48 misses=48 evictions=0 fills=48 bytes=52224 exhausted=0 advised=48 [expert-stream] steps=3 hits=96 misses=48 evictions=0 fills=48 bytes=52224 exhausted=0 advised=48 [expert-stream] steps=4 hits=96 misses=90 evictions=26 fills=90 bytes=97920 exhausted=0 advised=90 ``` `steps` advances, so the step clock has a caller. `hits` accumulates across steps, so entries survive and the eviction policy is running. `evictions` becomes nonzero once the budget binds. `exhausted` stays 0, so nothing fell back to the mapping. `advised` tracks the fills, so the readahead hint is accepted rather than rejected for an unaligned address. Under the defect this repairs the first line would read `steps=0 ... exhausted=32`, and there would be no second. ## Mutations Every one reports applied (non-empty `git diff --stat`), compiled (a non-building mutation is INVALID, not a pass), a non-zero case count, and a byte-for-byte restore. | Mutation | Result | |---|---| | M12a delete the `ForwardLayers` step guard | CAUGHT | | M12b `EndStepIfActive` to a no-op | CAUGHT | | M13 disable the production call site | CAUGHT | | revert `TowerUid` to the buffer address | CAUGHT | | let a COPY inherit a tower identity | CAUGHT | | drop `Invalidate` from the failed fill | CAUGHT | | keep `mmap_fd` after `ReleaseHost` | CAUGHT | | delete `SourceOfSpan`'s sibling walk | CAUGHT | | force `SourceOfSpan` offset to 0 | CAUGHT | | unaligned `madvise` address | CAUGHT | | drop the largest-slice reservation | CAUGHT | An earlier pass of this table is discarded rather than reported: the harness restored with `git checkout -- <file>`, which reverted uncommitted work, so three runs reported on a tree missing a repair. The harness now refuses to run on a dirty worktree, and every mutation above was taken on a clean one. ## Owed No performance number is claimed here. Re-measuring decode on a live cache needs `dgx.casa` and the 370 GiB checkpoint; the box was unreachable throughout and this host has neither a CUDA device nor room for the model. The `pread` path and the readahead lever are unmeasured on the model for the same reason, and Windows has no streaming arm (`EnsureFile` refuses by name, no `pread(2)`). All of it is listed under `## Owed` in the spec. Closes mudler#1066. Repairs the review of mudler#912. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
MSVC refuses the implicit narrowing in the two `StreamState` <-> `Ltx2LatentState` position copies under `/WX`, so the native Windows build cannot compile `ltx2_video.cpp` at all. The range-assign is replaced by an explicit `static_cast` loop in each direction. The cast is not a claim that precision does not matter here; it is a claim that none is lost, and the file already carries the argument. `ltx2_video.cpp:193-196` records that the positions round trip exactly because every value in the `double` buffer was widened from a `float` in the first place -- the `double` exists only because the DiT surface takes one. Both writers agree: the video axis is `static_cast<double>(float_expr)` at `:2463-2466`, and the audio axis comes from `Ltx2AudioPatchTimings`, which returns `std::vector<float>`, at `:2535`. So `float -> double -> float` reproduces the bits, and the explicit cast performs exactly the conversion the implicit assign already performed. This closes the last MSVC compile blocker in the mudler#503 class. The lane still fails afterwards, later and for a different reason -- `test_openai_api_server.exe` exits `0xC0000409` with no doctest summary, a runtime defect the build never reached before. That is tracked separately and is not this change. A second commit refreshes the READER ANCHORS comment. The casts add nine net lines above `kKnownLoadExtras`, shifting every recorded anchor by nine, and `test_ltx2_video` derives and compares that list on every run so the drift is caught rather than silently rotting. The replacement list was derived against main at `0f8580e26` with this branch merged, because main moved this file twice since the branch base (`332aed738`, `3ce1cf7c7`). Fixes mudler#968 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode] Signed-off-by: harleywilsoneng <harleywilsoneng@users.noreply.github.com>
…ee counts that did not re-derive (mudler#857) (mudler#1075) FOLLOWING_AGENTS_PROTOCOL Closes the two GPU-free `## Owed` items from the llama.cpp repin (mudler#857, `283c7e492`) and files one issue. Records only: the diff is `.agents/specs/oracle-llamacpp-repin-stock.md` and one appended `.agents/issue-index.md` row. ## The leading word boundary is restored Stage 2 was rebuilt to compile from the same `FAV` dict as stage 1, which fixed a real disagreement, but the retired pipe carried a leading `\b` that `FAV` did not. Restoring it, measured on a clean detached worktree at `0f8580e269` with the instrument extracted from the spec's own fence: | ratio branches | files | stage 1 | files hit | stage 2 | |---|---:|---:|---:|---:| | as they were | 4525 | 1278 | 146 | 1042 | | leading `\b` restored | 4525 | 1260 | 145 | 1024 | **18 removals, zero insertions.** All 18 adjudicated by shape and site: 14 `C2x`/`C3x`, 2 `sm8x`/`sm_12x`, 2 `Q8_0`, every one glued text with no verdict among them. The safety claim is the one that mattered, since trading a false positive for a false negative is the unsafe direction. Verified structurally rather than by sampling: each of the 18 carried **exactly one** stage-1 alternative, the loose ratio, and **zero** other comparison tokens, with **zero** tight-ratio matches. No verdict was lost. It also closes a question the spec had left open. Of the 38 candidates that match a ratio and are invisible to the retired pipe, the boundary drops 34 and **keeps 4**, and the four it keeps are exactly the fractional-leading-zero spellings. So the boundary removes glued text and keeps every real ratio, including every one below 1.0. ## The instrument stays in this document, with the condition that would reverse it The open question was whether to keep growing per-shape controls inside a markdown fence or move the sweep to `tools/` with a suite under `tests/tools/`. **A caution I had raised turned out to be wrong, and was re-derived rather than inherited.** `check-pr-size.py:510` attaches the mutation-evidence contract to class `governance_checker`, defined at `:162` as `scripts/check-*.py|sh`; `tools/*.py` and `tests/tools/test_*.py` both fall through to `:448` as `product`. The contract would not have attached. What refuses the move is what CI would enforce instead. `tests/CMakeLists.txt:12` auto-discovers `tests/tools/test_*.py`, so the suite would run every build, and what it would schedule green is a self-test that proves **liveness and not shape coverage**. Scheduling that green raises its authority without changing its content, beside a paragraph saying the green is no evidence about coverage. Secondarily, `check-role-discipline.py:63-71` makes `tools/` a feature prefix, so every later edit needs a reviewed `row/*` PR where `.agents/` is integration, and the sweep has no consumer after mudler#1003. The decisive argument is the policy one: **nothing lands dead.** After mudler#1003 the instrument is reachable only from its own test, which proves the class works and never that anything reaches it. **The condition that reverses this is recorded**, because a refusal without one reads as permanent: if mudler#1003 ever gains a second consumer, the decisive argument evaporates and the move to `tools/` should be taken again. Also recorded is the weakest of the three reasons, and that a scheduled self-test genuinely would catch a recurrence of the sixth hole. Per-shape controls on every alternative are refused separately: a control is a string its author writes, so seventeen invented shapes would read as coverage while establishing nothing. ## A third option killed itself under measurement A per-alternative census was proposed, on the theory that an alternative with near-empty reach is the signature of the `×` defect. It was measured against the very defect it was meant to catch: the historical dead branch reaches **5312** lines and its dead `×\b` half still reaches **470**, against the repaired branch's **2121**. It would have printed 470, not something near 1, giving no threshold a reader could act on. Rejected, with its numbers recorded in place of the feature. Measuring a proposed instrument against the bug it is meant to catch is cheaper than shipping it and discovering the same thing later. ## Counts that did not survive re-derivation Three small counts in this document were wrong, in a document whose own thesis is that a count without a re-derivable enumeration is untrustworthy. They were repaired **by re-deriving**, and the section now carries a second fence, extracted verbatim and run, that prints every corrected number: ``` stop 35 leave 18 survive 17 blind 38 dropped 34 ``` - The four kept ratios are four **lines** and three spellings, `3+1` rather than `2+1+1`, since one line carries two of them. - **35** candidate lines stop matching a ratio, 18 leave and 17 survive. The 34/18/16 reading is true only when scoped to the 38 lines invisible to the retired pipe, and that scope is now stated, with the differing line named. - Two antecedent slips corrected: three obligations rather than "all three" after naming two, and one anchor moved rather than two. The same class caught the row again while it was being repaired: `origin/main` landed `check-cuda-op-arch-gate` mid-flight, moving the recorded gate blocks from 26/45/77 to 27/46/79. Both moves are now named, with the reader told to re-derive the pair per run rather than trust either. Also recorded: the adjudication table assigns one shape per line, so `sm_12x` appears on five lines with four filed under `C2x`/`C3x`; all five are glued text either way, so the adjudication does not move. And the recipe's single-fence filter raises `ValueError` if a second matching python fence is ever added to this spec, which fails loud rather than selecting the wrong block, and which keeping the instrument in a document inherits. ## Filed [mudler#1058](mudler#1058) — `e34d71379e70` carries none of the required trailer lines, verified by re-running the checker rather than by taking the report. Owned by `GATE-SQUASH-TRAILERS`, since mudler#870's CI `--filled` PR-body guard is the missing enforcement. The commit is not repaired, because history is not rewritable here. ## Gates `scripts/agent-preflight.sh`: **79 ok, 0 FAIL, 0 SKIP**, exit 0, gated against `origin/main 1000264`, named in both range headings, with the trailer block executed rather than skipped. The instrument's own mutation pass was rebuilt independently in review: 43 red, 3 green, 0 defects, every red row reporting `applied=True` with a non-zero line delta, `compiles=True`, `rc=1`. The two dict-regeneration controls apply cleanly and stay green, which is what shows the rewrite mechanism is not what turns the other rows red. ## Owed The remaining `## Owed` items on this row all need hardware and are unchanged: the twelve re-takes under mudler#1003, the keep-f16 default's decision, and a chosen revision for Laguna's Poolside figure before it can be re-taken at all. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
`scripts/check-supported-models.py` gates the FEATURES row list against `REGISTER_VLLM_MODEL` in `src/`, but it does not gate the TOTALS in the prose around it, so the counts drifted while the list stayed correct. The registry has 40 architectures today; the comparison table, the "cannot drift" sentence, and the text-generation breakdown all still said 38. The breakdown was also arithmetically broken: it claimed 32 text-generation architectures out of a 38 total that also covers 3 Parakeet ASR entry points and the LlamaModel embedding arch, which leaves 34, not 32. With the real total of 40 the same subtraction gives 36, and the figures close. Taken from mudler#977, which carried these four edits plus seven README count updates. The README half is not included: `check-doc-checkpoint.py` refuses a README change that touches no landing source, on the ground that routine count refreshes do not justify landing-page churn, and that gate is right here. mudler#976 stays open for it. The README hunks also carried ABI v20/45, where `include/vllm.h` says v21 and exports 46, so none of that lands. Re-applied onto current main rather than merged: main had 11 further commits to this file since mudler#977 was written, and taking the branch's version wholesale would have reverted them. Refs mudler#976 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
…e the page said could not (mudler#1088) (mudler#1090) Closes mudler#1088. `docs/USAGE.md` published 448x256 at 25 frames as "Measured NOT to complete", with the reason that its decode loses about 59 GB in 24 s. Two renders on `dgx.casa` on 16 to 17 August 2026 against `main` `0b0b8900f` completed that geometry in 3085 s and completed 704x448 at 25 frames in 4231 s. The page now records the newer envelope, and `docs/BENCHMARKS.md` no longer says the opposite of it in a cell a reader meets first. ## What was measured Container `vllmcpp-build:gb10`, `Release`, `VLLM_CPP_CUDA=ON`, arch `121a`, `TRITON=ON`, CUTLASS absent so FlashAttention-2 was not built, which is like for like with the earlier renders. `VLLM_CPP_CPU_THREADS=20`, NVFP4 transformer, no `--allow-unported`. `0b0b8900f` carries mudler#1041 threaded decode, mudler#1032 T2A and mudler#1036 f32 decode accumulators. | Geometry | Result | Wall | |---|---|---| | 448x256 / 25 frames | completed | 3085 s | | 704x448 / 25 frames | completed | 4231 s | | 1024x576 / 25 frames | not attempted to completion, another session claimed the box | n/a | The ~59 GiB cliff did not recur under a 2 s memory guard that would have seen it: `MemAvailable` floors of 38.96 GiB over 1289 samples at 448x256 and 38.89 GiB over 1743 samples at 704x448, zero samples under 34 GiB on either, peak use of 80 of 119 GiB, and no reboot. The 704x448 artifact was verified rather than inferred from an exit code: 25/25 distinct frame md5s, 0 near-uniform and 0 near-black frames, adjacent-frame mean absolute difference 4.381 against a uniform-noise reference of 85.3 on the same shape, 0/24 zero-motion pairs, and audio at 48 kHz stereo, 1.010 s, -37.29 dBFS, 20/20 windows above threshold. The mp4 lives at `benchmarks/media/ltx25-704x448-25f-audio.mp4` on the render host and is gitignored by `.gitignore:35`, so it is not committed here. ## What is not claimed One run per geometry on a contended shared box with no oracle on either side. Two points establish no scaling law. 704x448 is not published as a ceiling: the next rung up stopped because another session claimed the box, not because of memory or an envelope. The page says all of this in its own words. ## The 59 GB is kept, not deleted It is the reason the old row gave, so deleting it would remove the evidence the newer result is measured against. It stays attributed to its own run, which is rung F1 in `.agents/benchmark-record.md`: a prompt-embeds render with no text tower that an armed watchdog ended at 13.77 GiB against an 18 GiB floor, rather than the engine failing. Attributing the fall is still mudler#1014, and this change does not close it. ## The dominant cost moved off the decode `docs/USAGE.md` said most of a 320x192/25f render is spent in the host VAE decode. After mudler#1041 threaded that decode, the dominant cost is a resolution-independent phase of about 1731 s, measured at 1731 s and 1732 s across two rungs whose voxel counts differ 2.75x, which is 57 to 66% of wall. That is mudler#1087, which owns naming the phase. The sampler classified by CPU-time rate rather than by symbol, so what is measured is a duration and a scaling law and not a function, and the page says so. ## Files | Record | Edit | |---|---| | `docs/USAGE.md` | envelope table rows, the paragraph under it, the bounded-by paragraph, and the mudler#1009 paragraph's stale "has not been re-measured" clause | | `docs/BENCHMARKS.md` | the `LTX-2.5 axes` row, edited in place as two table cells, 208 and 214 characters against `MAX_CELL_CHARS = 220`, so no prose paragraph is added to a page sitting at 35 of 35 | | `.agents/specs/ltx25-resolution-envelope.md` | new section 4.1 recording what superseded section 4, and the `## Owed` bullet that section 4 wrote | | `.agents/issue-index.md` | one row appended for mudler#1088, zero rows edited, zero removed | ## Evidence Records only. No `src/`, `include/` or `tests/` change, so no build was run and none is claimed. Key-by-key proof, taking `HEAD`'s version of each file and reapplying the scoped edit: | Record | Keys in base | Keys now | Unrelated keys byte-identical | Changed | Added | Removed | |---|---|---|---|---|---|---| | `docs/USAGE.md` | 205 | 206 | 203 of 203 | `**Measured to complete on one GB10**` | `Largest size tried`, `Superseded, kept for the record` | `Measured NOT to complete` | | `docs/BENCHMARKS.md` | 190 | 190 | 189 of 189 | `LTX-2.5 axes` | none | none | Issue index, the three verifications the append-only rule needs: the base file is a byte-identical prefix of the new one, the addition is exactly one line whose sha256 is `65933626d961a41b…`, and the file has 290 rows against 290 unique issue ids. The union driver was never allowed to resolve anything: the file was rebuilt as base bytes plus the row. Checkers, each with a red control observed on the same tree before the green was believed: | Checker | Result | Armed control | |---|---|---| | `check-doc-checkpoint.py --staged` and `--commit cedb85e` | 0 | `--commit b5618b3` exits 1, "changed user_usage but did not update docs/USAGE.md" | | `check-public-doc-tables.py` | 0 | padding the new cell past 220 characters exits 1 at line 487, "table cell of 333 chars exceeds 220" | | `check-issue-index-append-only.py --base origin/main` | 0 | committing a deletion of the `mudler#168` row exits 1, "this range removes or edits lines" | | `check-agent-record.py` | 0 | replacing the new row's owning row with a dash exits 1, "34 rows name no owner, above the recorded 33" | | `check-commit-style.py --range origin/main..HEAD` | 0 | an empty commit whose subject ends in a period exits 1 | | `check-commit-trailers.py --range origin/main..HEAD` | 0 | an empty commit with no trailer block exits 1 on three lines | | `check-pr-size.py --base origin/main --head HEAD` | 0 | n/a, no control run | Every tree mutation was restored and the restored file re-hashed to the pre-mutation sha256 before the next step. The key proof itself was seen red first, on an expectation that omitted the one key the change does edit in place, so its green is not a tautology. `scripts/agent-preflight.sh --staged` and `scripts/agent-ready.py` both report `All gates green` on `21544efd9`. `agent-ready` then exits 1 only on `expected exactly one live PR for row/LTX25-ENVELOPE-RECORD; found 0`, which this pull request is. `origin/main` advanced twice during this work, to `e9dfa6319` and then `9143196c7`. Both were merged in and every checker re-run afterwards; the second merge is the merge commit on this branch, and its message carries the trailer block because the range gate caught that it did not. ## What could not be verified The first `scripts/agent-preflight.sh` run exited 1 on `test_cpu_x86_llamacpp_floor`, on the unmodified tree before any edit in this branch. Its own output names the cause: `load=120.50`, so the harness discarded the contended leg and returned `NO_QUIET_WINDOW` (4) where the case expects `GIVING_UP` (2). That is mudler#618. It passed on the later runs once the box quieted, so this branch has no evidence of that case being sound, only of it being load-dependent as mudler#618 already says. The renders themselves were performed by another session and are reported here from its results. This branch did not run them, holds no GPU, and did not rebuild anything. `.agents/specs/ltx25-decode-speed.md` and `.agents/benchmark-record.md` also discuss the 448x256 rung. Neither is edited here: the decode-speed spec already records that the "inside the decode" half of the old sentence is unsupported, and the benchmark record is an append-only log of what each run observed, which stays true of the run it describes. Reconciling the investigation spec against the new rungs belongs to mudler#1087, which owns the phase. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
… so adjacent emphasis re-paired ACROSS spans (mudler#1083, mudler#672) (mudler#1089) Fixes mudler#1083. Found by the mudler#672 oracle sweep. ## The defect `CleanCaption` emulated upstream's `(?!\*)` with a **captured** `($|[^*])` (`src/vllm/model_executor/models/minimax_music3_ar.cpp:85,114`), and a captured group is not a zero-width assertion. Consuming the character after the closing `*` advanced `regex_replace` past it, so an emphasis span opening within **one character** of the previous close was never examined — and the surviving asterisks then re-paired **across** the intended spans. | in | upstream | before this PR | |---|---|---| | `a *b* *c* d` | `a b c d` | `a b *c* d` | | `*dreamy* *ambient* pads` | `dreamy ambient pads` | `dreamy *ambient* pads` | | `Warm *lo-fi* *jazzy* keys with a *soft* *brushed* snare` | `Warm lo-fi jazzy keys with a soft brushed snare` | `Warm lo-fi *jazzy keys with a soft brushed* snare` | | `*a* *b* *c*` | `a b c` | `a *b c*` | Row three is the one worth reading twice: it is a **re-association**, not a leftover marker — the third and sixth asterisks got paired, so the caption handed to the tokenizer is a string `_clean_caption` (`encoders.py:72` @ diffusers `c6da9936`) would never emit. `encoders.py`'s own header states that whitespace-level prompt changes change the generated audio, which is what makes this a checkpoint-contract break rather than cosmetics. ## The fix, and which construct it uses `std::regex`'s ECMAScript grammar has negative **lookahead** but no lookbehind, so the two sides are spelled differently on purpose: the trailing side is ported literally, and only the leading `(?<!\*)` stays emulated. ```cpp static const std::regex kItalic(R"((^|[^*])\*([^*\n]+)\*(?!\*))"); line = std::regex_replace(line, kItalic, "$1$2"); ``` Consuming on the **leading** side is safe by the same argument that condemns it on the trailing side: that character sits *before* the span rather than between this span and the next one, so no scan position is skipped. The asymmetry is the fix, not an oversight, and the call site says so. ## Evidence **RED first**, on the tree as it stood — 2 cases / 6 assertions failing on exactly the divergences the issue measured: ``` values: CHECK( a b *c* d == a b c d ) values: CHECK( dreamy *ambient* pads == dreamy ambient pads ) values: CHECK( Warm lo-fi *jazzy keys with a soft brushed* snare == Warm lo-fi jazzy keys with a soft brushed snare ) values: CHECK( a *b c* == a b c ) [doctest] test cases: 2 | 0 passed | 2 failed | 24 skipped [doctest] assertions: 21 | 15 passed | 6 failed | ``` **GREEN after**: `test_minimax_music3_ar` 26 cases / 352 assertions (from 25 / 338). Why the gate could not see it before: `markdown_and_tags`, the only prompt golden with italics, carries one span per line (` * *hazy*`) and this defect needs adjacency. Two new prompt goldens — `adjacent_emphasis` and `unbalanced_emphasis` — were produced by **executing** the pinned `_clean_caption`, not by reading the regex. The generator's own emission path reproduces the committed block byte-for-byte both before and after, so `gen-minimax-music3-ar-goldens.py` and the `.inc` cannot disagree. `unbalanced_emphasis` is the negative side (`a **b* c` and `a *b** c` survive untouched), which is what keeps the leading guard from being dropped once the trailing one is a true lookahead. **Three mutations, all RED**, restored `sha256`-identical and rebuilt green: | mutation | result | |---|---| | consume the trailing neighbour again (the defect) | RED, 2 cases / 6 assertions | | drop the leading `(^\|[^*])` guard | RED, 2 cases / 3 assertions | | delete the production call site | RED, 2 cases / 12 assertions | **Differentially, against the pinned oracle.** 16 012 inputs — 12 realistic markdown descriptions plus 16 000 from a markdown-flavoured alphabet — through the built `CleanCaption` and through `_clean_caption` at the pin: ``` inputs=16012 pre=147 post=85 fixed=62 NEWLY_BROKEN=0 post is a subset of pre: True ``` All 85 residual mismatches are **one separate degenerate class**, recorded as owed in the spec's §10.7 and deliberately not chased here: a caption that is entirely a horizontal rule, where Python's `re.MULTILINE` lets `\s` span a newline and collapse several lines at once while we apply the rule per line (73 of 85 match `^\s*[-*_]{3,}\s*$` directly; the other 12 are the same mechanic after tag rewriting). Owed beside it, already known: `std::tolower` is per byte, and we split lines on `\n` only where `splitlines()` also covers `\v`, `\f`, bare `\r`, U+2028 and U+2029. ## No golden's provenance is affected The oracle capture's prompt (`tests/parity/goldens/minimax_music3_oracle/manifest.json`) contains **no asterisk at all**, so no committed waveform, latent or code golden was conditioned on a caption this change would rewrite. `markdown_and_tags` does carry one italic span, but its expectation came from executing upstream and it keeps that expectation byte-for-byte. ## Gates run CPU-only Release, 20 cores, load 21-48 throughout (a music generation was running on the box). | suite | result | |---|---| | `test_minimax_music3_ar` | 26 / 352 | | `test_minimax_music3_ar_real` (`CHECKPOINT_ROOT` set) | 4 / 894 | | `test_minimax_music3_llm_real` (`CHECKPOINT_ROOT` set) | 4 / 220 | | `test_minimax_music3_acoustic` | 27 / 265 | | `test_minimax_music3_acoustic_real` | 6 / 64 | | `test_minimax_music3_e2e_real` | 9 / 37 | | `test_minimax_music3_loader` | 21 / 1393 | | `test_minimax_music3_quant` | 29 / 125 | | `test_minimax_music3_quant_real` | 6 / 319 | | `test_minimax_music3_speech` | 9 / 223 | | `test_speech_api` | 6 / 67 | | `test_speech_engine` | 11 / 38 | | `test_openai_api_server` | 62 / 733 | | `test_capi` | 65 / 653 | | `test_minimax_h3` | 79 / 57395 | | `test_indextts2_family` | 7 / 22 | `scripts/agent-preflight.sh --no-require-role`: **1 gate failed, `test_cpu_x86_llamacpp_floor`** — the load-dependent mudler#618 class (`NO_QUIET_WINDOW` at loadavg 24-48). Proved not mine: it fails identically in a worktree of pristine `origin/main` at the same load (3 failures there, 2 here — the count moves with load, which is the mudler#618 signature), and neither that test nor its subject appears in this diff. Every other gate `ok`. FOLLOWING_AGENTS_PROTOCOL Issue: mudler#1083, mudler#672 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…n an NVFP4 Marlin arena (mudler#1085) NemotronH's 23 MoE blocks reach the device on an NVFP4 Marlin arena, gated on a synthetic fixture A2-Q2 was split. This unit lands the MoE half; `lm_head` and the real-checkpoint per-block gate are A2-Q2b (`.agents/specs/nemotron-h-a2q2b-realckpt-lmhead.md`, committed here so the deferral points at a document rather than a promise). Keeping `lm_head` on the host preserves A2-R's attributability: both arms still end in the identical host projection, so a token difference stays attributable to the MoE arm. ## LANDS UNREACHED, and this is the disclosure `NemotronHMoeBlockDevice` has **no production caller and is reachable by no configuration**. Not "not wired yet" -- G-SAFE refuses first: `MakeNemotronHKVCache` builds an attention group over the 6 GQA layers (`nemotron_h_registry.cpp:232`), the runner fills `attn_kv_` from those buffers (`runner.cpp:906-916`) and passes `.attn_kv = attn_kv_` on every forward (`runner.cpp:1371`), while the interlock requires `attn_kv.empty()` (`nemotron_h_registry.cpp:162`). `vllm_engine_load` is the only text entry point, so no ABI path bypasses the runner: a NemotronH engine refuses before emitting a token. Wiring owner: **A2-P**. Tracking issue: **mudler#810**. Listed under the spec's `## Owed`. **Nothing from this unit executes in production** -- `PrepareNemotronHForCausalLM` is deliberately left a no-op so the lazy 16.5 GB repack cannot fire on a load whose forward will refuse. ## The gate, exactly as measured (GB10 sm_121a) ``` HEADROOM_GB=115 BUILD_RC=0 TEST_RC=0 error: 0 device-vs-host worst relative deviation: 0 over 512 elements separation of a routed-scale defect: 0.6 over 512 elements accepting at band 0.3 [doctest] test cases: 2 | 2 passed | 0 failed | 0 skipped [doctest] assertions: 10 | 10 passed | 0 failed | [doctest] Status: SUCCESS! ``` 512 = T*H, so this is not a mute instrument. The feature lines are cited from `configure.log` in the same tree, **not** from this run: the incremental build did not re-run configure, so `run2.log` does not reprint them. `cutlass-nvfp4`, `cutlass-fp8`, `marlin-nvfp4` and `fa2` all read `ENABLED for [121a]`, with `CUTLASS found at /cutlass` and `Triton AOT ... sm_121a`. A `DISABLED` line or `[121]` would have voided the result. ## Mutations: Q2-M1 and Q2-M2 RUN and RED | Arm | diff_stat | compile_exit | compile_err | bin_sha16 | test cases | assertions | Status | deviation | |---|---|---|---|---|---|---|---|---| | BASELINE | 0 | 0 | 0 | `fff47ae76914ac85` | 2 \| 2 passed | 10 \| 10 passed | SUCCESS! | 0 over 512 | | Q2-M1 nibble order flipped | 2 | 0 | 0 | `8d4d6f33b04fa542` | 2 \| 1 failed | 10 \| 1 failed | FAILURE! | 1.29663 over 512 | | Q2-M2 `weight_scale_2` ignored | 2 | 0 | 0 | `9298af5ef070d021` | 2 \| 1 failed | 10 \| 1 failed | FAILURE! | 15 over 512 | | RESTORED_CONTROL | 0 | 0 | 0 | `fff47ae76914ac85` | 2 \| 2 passed | 10 \| 10 passed | SUCCESS! | 0 over 512 | Each patch asserts its anchor appears **exactly once** before applying, so a mutation that never applied cannot read as a passing test. Every binary sha is distinct from baseline, and `RESTORED_CONTROL` reproduces the baseline sha `fff47ae76914ac85` exactly -- the control that catches an mtime-preserving restore letting ninja skip the rebuild. **Q2-M3 through Q2-M7 are OWED to the fresh reviewer**, with the same harness (`mut.sh` on the gate host). Q2-M3 matters most: `src/vt/ops.cpp:874-895` validates no extent of `b_q_weight` and **nothing at all** about `b_scales`, so an expert-stride defect is silent at the op boundary and only the numeric gate can see it. ## The synthetic bit-exactness is a BOUNDED claim (spec 13.6.1) The fixture's output is bf16, its contraction is K=128, its E2M1 codes are exactly representable and its group scales are powers of two -- precisely the conditions under which a bf16 store absorbs genuine reduction-order differences. It says nothing about the real geometry (H=2688, I=1856, E=128, top_k=6, a 21x longer contraction), the real `weight_scale_2` and group-scale values (neither powers of two nor uniform), or the Marlin thread configs the fixture does not select. **A red on A2-Q2b's real-checkpoint gate after this is predicted, not a contradiction.** ## Design decisions, argued **Spec 4.3 answered by a fourth route.** The shared expert runs as an E=1 slice of the same arena -- the documented dense mechanism (`dense_nvfp4_gemm.h:41`), which is how vLLM reaches the same csrc kernel. So **neither** function named `MarlinDenseResidentFor` is reachable from this model, and mudler#984's address-keyed cache cannot bite this row however mudler#984 is resolved. Provable by absence, and it needs no `Nvfp4Weight` copy of the 23 shared pairs, so `rep.host_bytes` stays at its pinned `18888922112`. **The repack is LAZY and explicitly transitional, naming A2-P.** Spec 4.2 puts it in `Prepare`, but that reason is forward-looking and false today: nothing captures `NemotronHDeviceForward`, while `ModelRegistry::Prepare` is called unconditionally from both `GPUModelRunner` constructors. Building there now would make every production engine load pay 16.5 GB for a path nothing reaches. "Nothing lands dead" covers an unreached forward, which costs nothing; it does not cover an unreached allocation inside a reached hook. **Spec 3's freeing premise was wrong and is corrected in 13.1.** Neither existing instance frees per expert, so a literal reading peaks near 32 GB. Each expert here streams through one reused 2.8 MB staging pair: no raw tower to accumulate, nothing to tail-free, nothing to get wrong. Two instrument defects were found and fixed rather than worked around: the band `sqrt(agreed*separation)` collapsed to 0 on exact agreement and failed on the best possible outcome, and `deviation: 0` could not be distinguished from a loop that examined nothing. Both are recorded in 13.6.1. G-SAFE's three clauses are byte-unchanged. `host_bytes` is unchanged. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
… CHECKPOINT_ROOT (mudler#1084) `/mnt/nas_share` no longer exists on the gate box, so every tracked default built on it named a path that cannot resolve. The DGX profile in `.agents/environment.md` named no NAS location at all, which is why the correction went into the untracked `.env` alone and the tree kept the dead path. The profile now gives the live location and the reason it cannot move back. `/mnt` is the ephemeral root overlay of an immutable Kairos OS and loses its contents at every reboot; `/usr/local` is `COS_PERSISTENT` and survives. On 2026-08-16, after an 8 h 19 min outage, the mount came back because the `/oem` boot-stage unit worked and `/mnt/nas_share` did not. A bare path correction would invite the next reader to restore a convenience symlink that the following reboot deletes, so the profile carries the reason and not only the value. The seven live defaults now derive from `CHECKPOINT_ROOT`, which four sibling scripts already did (`scripts/measure-ltx2-keyframes-meta.py:30`, `scripts/gen-ltx2-prompt-tokens-goldens.py:28`, `scripts/measure-ltx2-prompt-adaln.py:27`, `tools/oracle/music3_oracle.py:31`). One untracked `.env` line then moves the location, instead of another sweep. `test_minimax_music3_quant_real` loses its literal fallback: an undeclared root skips and names the two variables, rather than probing a path nobody declared. ## What was classified as a record, and kept All 41 `/mnt/nas_share` hits were classified before any edit. `.agents/benchmark-record.md`, the LTX-2.5 and Nemotron-H specs, `.agents/model-matrix.md`, the captured goldens under `tests/parity/goldens/` and the generated `.inc` headers state where a past measurement read its bytes. That is provenance, and AGENTS.md is explicit that rewriting an existing file to satisfy a rule is out of scope unless a row asks for the rewrite. Two live-looking hits are deliberately left, and §4 of the spec says why. `tests/vllm/multimodal/test_qwen3_5_moe_vl_hw.cpp:67` is the second entry of a probe list whose first entry is already `/usr/local`, so a fresh run resolves, and the measured fact covers `dgx.casa` rather than the cluster nodes. `tests/vllm/test_pretokenizer.cpp:377` states that the GPT-4o regex was transcribed from that path into `tools/gen_pretok_goldens.py`; it is the same sentence shape as two comments this change does update, mudler#1073 lists those two and not this one, and an implementer should not decide that by guessing. ## Evidence | Claim | Result | |---|---| | `scripts/agent-preflight.sh` | PASS before the edits, and PASS on the merged head except `test_cpu_x86_llamacpp_floor` | | that one failure | [mudler#618](mudler#618), the load-dependent contended-leg case: `AssertionError: 4 != 2`, `NO_QUIET_WINDOW after 30s (busy=118% load=29.11)`, so the harness exits 4 instead of the 2 it asserts. 9 of its 10 cases pass. Attributed rather than assumed: a foreign `minimax-music3-` job held 1345% CPU throughout, and `git diff origin/main HEAD -- tests/scripts/ tests/tools/` is EMPTY, so the failing gate runs main's own code | | `check-doc-checkpoint.py --base origin/main --head HEAD` | `OK: public documents match the claims this change makes.` | | `check-commit-trailers.py` and `check-commit-style.py` | ok for all three commits | | the changed gate builds and runs | warning-free on the merged tree; with the root unset, 6 cases print `SKIP music3 q4_k artifact identity: VLLM_CPP_MUSIC3_GGUF and CHECKPOINT_ROOT are both unset`; with `CHECKPOINT_ROOT` set they name the composed `…/minimax-music3-gguf/rvq_depth_decoder_q4_k.gguf` | | the tokenizer tool refuses cleanly | with the root unset and no arguments, `error: the following arguments are required: --tokenizer-json`; with it set, `--help` prints the `$CHECKPOINT_ROOT/…` default | Rewriting those skip messages surfaced [mudler#1079](mudler#1079), fixed in the same flow. All four streamed the case name as a `const char*`, which doctest 2.5.2 stringifies through its bool overload, so each printed `SKIP 1` and named no case. That binary reports 6 passed with 0 assertions when the checkpoint is absent, so the message is what separates a skipped run from a gated one. Measured scope before the fix: 4 hits, all in this one file. [mudler#1077](mudler#1077) was also found and is NOT fixed here. `.env.example:37`, `.agents/environment.md:29` and `tests/vllm/multimodal/test_ltx2_video.cpp:2128-2132` each state that nothing in the tree reads `CHECKPOINT_ROOT`, and six gates read it. It is left because `test_ltx2_video.cpp` reasons FROM that claim when it chooses a separate variable, and reversing a design decision needs its own review rather than a path substitution. It is listed under `## Owed` in the spec. `windows-msvc-cpu` and `windows-msvc-vulkan` are red on every pull request ([mudler#584](mudler#584), [mudler#968](mudler#968)) and are inherited, not caused here. `origin/main` moved to `a332fb98d` during the work and appended an issue-index row. The merge is local, and the index was rebuilt by hand rather than left to the union driver, which had placed main's row after this branch's three. Main's file is now a byte-exact prefix of the result, 306 of 309 lines, checked with `diff`. GitHub does not run that driver, so the file would otherwise present as CONFLICTING there. The merged tree was rebuilt and rerun, not only merged clean. Spec: [`.agents/specs/nas-mount-path.md`](.agents/specs/nas-mount-path.md). Closes mudler#1073. Closes mudler#1079. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…rning spec got wrong about it (mudler#517) (mudler#1082) A2-P is the last unit blocking mudler#810 and every Nemotron benchmark. It is the one that NARROWS G-SAFE for the first time, that removes or narrows `scripts/runner-routing-allowlist.txt:26`, and that gives A2-R's and A2-Q's device arms their first production caller. **Records only: no product code, no lifecycle change.** Per the governing spec's §1.4 the implementation is a separate PR by a different agent and the reviewer is a third; `row/MODEL-NEMOTRON-H-ABI-A2Q2A` is also in flight on the same files, so a spec-first landing avoids a conflict. ## What the unit is `ForwardNemotronHForCausalLM` still returns `HostLogits(NemotronHForward(...))` (`nemotron_h_registry.cpp:185-187`). That host reference recomputes Q/K/V over the whole sequence every call (`nemotron_h.cpp:657-659`), runs one dense causal `vt::Attention` over the whole `[T,.]` (`:671`), and carries no recurrent state between steps. A2-P replaces it, for the single-request case, with a forward that reads and writes the runner's paged KV and its persistent conv/SSM pages. The interlock at `nemotron_h_registry.cpp:161-170` is narrowed exactly as its own `:158-160` comment pre-committed: the `attn_kv` and `gdn_state` clauses are CONSUMED, `num_reqs <= 1` STAYS until A2-B. A reviewer who finds the whole `VT_CHECK` gone returns FAIL, and so does one who finds the surviving clause weakened to a warning. ## Two corrections to the governing spec, both measured **Its §2 seam claim is wrong for this architecture.** `dense_attn::AttnBlock` applies `vt::RopeNeox` unconditionally (`dense_attn_block.h:497`) and reads a `cfg.rms_norm_eps` that `hf_config.cpp:551` defaults to `0.0` for a checkpoint shipping `layer_norm_epsilon`. Nemotron-H has no RoPE at all: a case-insensitive grep for rotary/rope/q_norm/k_norm over `nemotron_h.py` at the pin returns zero hits, and its four-line forward (`:474-483`) sends q and k straight into `self.attn`. There is no rope-free entry point, and `rotary_dim == 0` ABORTS at `ops.cpp:1427-1429` rather than bypassing. Filed as mudler#941; A2-R already took the model-local road and A2-P extends that block. **Its §2.7 R4 named unplanned kernel work that already existed.** `vt::CausalConv1dFwd` admits a bf16 conv state wherever the backend answers `SupportsCompressedConvState()` (`ops.cpp:1644-1650`) -- CUDA, Vulkan and ROCm all do. It landed at `908bad0ac` on 2026-08-09, SIX DAYS before the spec that called it a risk. The decision is unchanged (bf16 page, never widened to f32); only its cost was mis-stated. Re-measure a stated blocker at your own base. Three in-tree anchors are also measured stale and recorded so they are fixed rather than copied forward: `dense_attn_block.h:496` (actual `:497`), `nemotron_h.cpp:585-630` for `NemotronHAttentionMixer` (actual `:631`), and `nemotron_h.cpp:822-825` for the CPU-queue check (actual `:868-871`, with a SECOND one at `:1031-1034` that A2-P must preserve). ## What the spec decides so implementation does not - The RED-first test enters through `ModelRegistry::Forward`, never through `NemotronHDeviceForward` -- which today has exactly ONE non-declaration call site in the tree, `test_nemotron_h_forward.cpp:1805`, the test-only-driver shape `reachability.md` names. The red is already available: that call refuses at `nemotron_h_registry.cpp:161`. - The paged branch passes `input` WHOLE, and its predicate carries the residency clause, because `kimi_linear_registry.cpp:99-100` has THREE clauses and the fragment quoted in our own comment has two. - The allowlist entry is REMOVED only if A2-Q2's `lm_head` arm has landed; otherwise it is NARROWED. Deleting it while still returning `HostLogits` reds `check-runner-routing-consistency.py`, and widening the allowlist to satisfy the checker is the defect it exists to stop. - The fresh-request state zeroing (`gdn_attn.h:126-139`) is a caller obligation the kernels do not gate; a stale mamba block is fluent, plausible and wrong. Mutation P-M4 is the instrument, and a survivor there is a coverage hole rather than a pass. - P-M6 widens the conv page to f32 and the token gate must NOT red. That asymmetry IS the demonstration that a token gate cannot see a too-wide dtype, and it is reported as a pair. R3 of the governing spec is CLEARED and re-verified rather than inherited: mudler#496 W2 landed at `43a6c5518`, and `kMamba2ChunkScan`, `kMamba2StateUpdate`, `kCausalConv1dFwd` and `kRmsNormGatedGroup` all register from an unconditionally compiled CUDA TU. ## Gate `scripts/agent-preflight.sh` on this branch: every record gate green except `test_cpu_x86_llamacpp_floor`, which exits `NO_QUIET_WINDOW` at loadavg 22 -- the harness refusing to measure, mudler#618, inherited and reproduced at the base with zero modifications. The mudler#873 gates are FIXED; a red on those would be the implementer's. ## The issue index Appends ONE row, for **mudler#941**, naming the owning row. It does NOT append a second mudler#810 row: mudler#810 already occupies `.agents/issue-index.md:254`, and `scripts/check-agent-record.py:1437-1442` refuses a duplicate because under `merge=union` a duplicate is what two branches appending the same issue look like -- the mudler#995 shape that reds `main`. The A2-Q spec split (`4496ef196`) declined for the same reason and said so. mudler#941 was unindexed, is open, is consumed here in flow, and its still-open item 3 is listed under the spec's `## Owed`. Row written with no raw pipe in any cell, per mudler#1033. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…ten coordinates that pointed at the wrong lines (mudler#1099) Four LTX-2 pipelines had no recipe row, no refusal, no `Ltx2UnportedPipelineFeature` marker and no issue. AGENTS.md draws the line at exactly that: a refusal naming a missing part is documented debt, and silence is not. `.agents/specs/ltx-2-5.md` §2's "Out" list named none of them. Filed, each with what is absent, what a future row starts from, and what blocks it, owned by `ROAD-V1-LTX25` in the index and listed under the campaign spec's new `## Owed`: - mudler#1093 `TI2VidTwoStagesPipeline` — CFG-guided on the FULL model in stage 1, the distilled LoRA on stage 2 alone. Not our `distilled_two_stage`, and not mudler#921's HQ variant. - mudler#1094 `HDRICLoraPipeline` — needs a LogC3 decode tail this tree has zero of. - mudler#1095 `DubItPipeline` — needs a negative RoPE shift our one ported shift structurally cannot produce, because it clamps at zero. - mudler#1096 `KeyframeInterpolationPipeline` — the conditioning block IS served; the multi-keyframe request surface and a per-sigma guided denoiser are not. mudler#1093 was not silent: `ltx25-resolution-envelope.md` already carried it under `## Owed` as "not separately filed, because mudler#644 already owns 'close every refused arm'". That bullet now points at the issue, and says why an umbrella row could not carry what this arm is blocked on. Two findings from the audit that produced this are REJECTED on evidence rather than recorded. `Ltx2AudioPatchify` was reported ported-but-undriven; it runs at `ltx2_video.cpp:2595` on every render, and it is its OTHER call site that is undriven. `KeyframeInterpolationPipeline` was reported as pure porting; it needs two checkpoints absent from the NAS. ## The page said things that were not true `docs/USAGE.md` published a `t2a_one_stage` command using `--steps 30` on a binary with no `--steps` flag, where an unknown argument exits 2. The published command could not run. Also corrected: a second `--lora` does not refuse (mudler#1097 — and the refusal turns out to be unreachable from EVERY production path, not only the CLI); ten resolving `(kind, version)` pairs where there are fifteen; `VLLM_ABI_VERSION 19` and 36 exports where the header says 21 and 46; "four markers" beside three; ten load extras beside twelve; "CLI and ABI only" for two knobs with no CLI flag; and a refusal reason ("only the decoder is ported") that phase L11 removed. Ten `file:line` coordinates into `ltx2_video.cpp` and `ltx2_loader.cpp` were stale by 40 to 2200 lines, at eleven citation sites — the family registry cited at `:1529`, living at `:3723`. Every symbol existed, which is why they read as plausible. Re-derived from the sentence making each claim, never from the cited span, and the first anchor in each affected passage is pinned at `b5756ea8c`, because no gate here checks a documentation anchor (mudler#632, mudler#911). The second commit on this branch corrects that paragraph's own count, which said nine and was itself never derived. A paragraph about stale citations carrying an underived number is the same defect one level up, so it is a commit rather than a quiet fix. ## README is NOT touched, and that is a finding rather than a scope cut It should have been. It carries zero `LTX` mentions against a `minimax` control of seven, and says "37 registered architectures" four times where `docs/FEATURES.md` says 40 — corrected two commits ago in `9143196c7`, which left the README's four copies alone. Two gates refuse the fix independently, and both were run rather than assumed. `MAX_README_CHARS = 30000` against a measured 29,989 means the LTX-2.5 matrix row could only land by deleting another architecture's row. And `check-doc-checkpoint.py:346-354` refuses any README change not accompanied by a landing-source edit, per commit (mudler#573), so splitting it out does not help either. A two-family paragraph was written, measured to fit with four characters to spare, and then reverted when the second gate fired; it is preserved verbatim in mudler#1098 rather than lost. Neither gate is touched here. Making a red gate green by widening its assertion is the one repair AGENTS.md rules out, and the honest alternative — an unrelated edit to `CMakeLists.txt` to unlock the checkpoint gate — is the fake-update pattern that same checker's history records six escape hatches' worth of. mudler#1098 asks for the two decisions. ## Issues Filed by this change and linked in `.agents/issue-index.md`, the campaign spec's `## Owed`, and here: mudler#1093, mudler#1094, mudler#1095, mudler#1096, mudler#1097, mudler#1098. Every index row names `ROAD-V1-LTX25` as its owner, so the unowned count stays at exactly the recorded `UNOWNED_HIGH_WATER = 33` — which is a two-sided ratchet and reds either way. Each is additionally listed under `## Owed`, so ownership survives both known defects in `owed_issues()` (mudler#1042). ## Gates, each with an armed control No build was run and none was needed: this change touches only `.agents/` records and `docs/USAGE.md`. Nothing under `src/`, `include/`, `tests/`, `examples/` or `CMakeLists.txt` is modified. | gate | result | armed control | |---|---|---| | `scripts/agent-preflight.sh` | **All gates green** (27 record gates, 44 mutation suites, committed range and trailers vs `0bac476b7`) | see rows below | | `check-agent-record.py` | exit 0 | appending an unowned row makes it exit 1: *"34 rows name no owner, above the recorded 33"*; restored byte-for-byte, exit 0 again | | `check-issue-index-append-only.py --base origin/main --head HEAD` | exit 0 | a REAL COMMIT editing `mudler#168`'s row makes it exit 1: *"removes or edits lines"*; `git reset --hard` back to the good SHA, exit 0 again. A working-tree control would have returned 0 and measured nothing | | `check-doc-checkpoint.py --base origin/main --head HEAD` | exit 0 | `--commit b5618b3` exits 1 (*"changed user_usage but did not update docs/USAGE.md"*) | | `check-readme-structure.py` | exit 0 | appending 42 characters exits 1: *"README is 30038 chars, over the 30000-char landing-page budget"* | | `check-public-doc-tables.py` | exit 0 | `docs/BENCHMARKS.md` and `docs/FEATURES.md` are untouched, so their 35/35 and 21/21 paragraph budgets are unchanged | | `check-pr-size.py --base origin/main --head HEAD` | exit 0 | | Three index verifications, run against `git show origin/main:.agents/issue-index.md` rather than against the working tree: byte-identical prefix **true**; all 291 pre-existing rows byte-identical and in order **true**; all 297 issue ids unique, **no duplicates**. The union driver was never allowed to resolve anything. ## Verified, not taken on trust Every claim corrected here was re-measured against the tree, and three audit findings were **rejected on evidence**: - `Ltx2AudioPatchify` is not undriven. `ltx2_video.cpp:2595` calls it in the production phase loop on every render. - `KeyframeInterpolationPipeline` is not pure porting. `--distilled-lora` is `required=True` on its parser and stage 1 needs the full `-dev-` transformer; neither file is on the NAS, where `find -iname '*lora*'` returns nothing against an 8-file `*.safetensors` control. - The resolving-pair count is fifteen, not the thirteen reported; the page said ten, not the nine reported. Both sides of that finding were wrong. `TI2VidTwoStagesPipeline` was reported as unrecorded debt and was not: `ltx25-resolution-envelope.md` already carried it under `## Owed` with a stated reason for not filing. It is filed anyway, and that bullet now says why the reason changed rather than being overwritten. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…whether a file did `FEATURE_SURFACE_PREFIXES` covered all of `src/vllm/model_executor/models/`, so any edit to any model translation unit classified as `feature_surface` and owed `docs/FEATURES.md`. That is the classify-by-directory trigger this file's own header says the 2026-08-11 rewrite removed, still standing for model files: > A one-line compile fix owed three public-doc edits, so this gate produced 16 > of the last 20 red CI runs, and it had accreted SIX hardcoded exact-path-set > escape hatches -- one per legitimate change it had blocked. > [...] Editing src/ alone owes nothing. The cost stopped being hypothetical on 2026-08-16. `e34d71379` (mudler#1054) is a one-line lambda-capture change to `models/qwen3_5_weights.cpp` that alters no capability. The gate demanded the surface; the commit answered with prose in BENCHMARKS, FEATURES and STATUS; that prose crossed the `check-public-doc-tables` paragraph budgets; and because that checker also runs in the pre-push hook, every branch in the repository was blocked from pushing. That is mudler#1055, re-filed by a second agent as mudler#1062 with mudler#1064 as a duplicate fix PR, alongside mudler#1058 which is still open. Two shared-file gates in series, each individually defensible. What the project supports is what the registry registers, so `feature_surface` now keys off a change to the set of `REGISTER_VLLM_MODEL(...)` entries in the touched file, read through `blob()` like the existing `measurement_changes`. `scripts/check-supported-models.py` already gates `docs/FEATURES.md` against that same set, so the signal is authoritative rather than newly invented. This NARROWS a gate, which is the risky direction, so the polarity is pinned by test rather than asserted: adding an architecture, adding a whole new registered file, and removing an architecture each still owe `docs/FEATURES.md`, and the four `.agents/*-matrix.md` records keep their path trigger because editing one IS the claim. Red before, green after, and mutation-proven. `test_doc_checkpoint.py` goes 27 -> 32 cases. Before the implementation, `test_editing_a_registered_model_owes_nothing` failed with the FEATURES demand. After it, 32 pass. Because the three still-owes cases were green under the OLD path trigger too, they could not by themselves show the new trigger works, so `registration_changes` was mutated to return `[]`: that fails exactly those three and no others, and restoring returns 32 green. Does not close mudler#595. A genuine new architecture still writes the shared `docs/FEATURES.md` table, so the lock that issue names survives; this removes the contention for every fix, refactor and port phase that changes no registration. mudler#515 is the identical shape for `CMakeLists.txt` -> `docs/USAGE.md` and is untouched. Both are recorded under `## Owed` in the spec, together with the class this narrowing gives up: a capability change inside an already-registered model now goes undemanded. Refs mudler#595 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
…MSVC says so `e34d71379` removed `[&kRequired]` from the `refuse` lambda in `CheckMoeQuantLayoutSupported` as "the redundant namespace-scope capture". Two different variables carry that name, and only one of them is namespace-scope: `kMoeExpertLayoutHelp` (`:894`) is a `static const std::string` and needs no capture, while `kRequired` (`:929`) is a function-local `const std::string&` bound to it, is odr-used in the lambda body, and therefore must be captured. MSVC rejects the result with `error C3493` and `main` has not compiled under it since. Naming the namespace-scope constant inside the lambda satisfies both compilers at once: MSVC has nothing left to capture, and the AppleClang diagnostic mudler#1054 set out to remove stays removed. Reverting to `[&kRequired]` would fix MSVC and reintroduce that diagnostic. Behavior is unchanged because `kRequired` IS `kMoeExpertLayoutHelp`; the reference is still used by the other refusal at `:947`, so it does not become dead. Verified by CI rather than locally for the compiler that matters: this box has no MSVC, so `windows-msvc-cpu` and `windows-msvc-vulkan` on this PR are the gate. `g++ -std=c++20 -fsyntax-only` on the TU returns 0. Worth recording why this landed green. The guarding gate is a source-TEXT assertion -- it "rejects `const auto refuse = [&kRequired]` and finds `const auto refuse = []`" -- which passes whether or not the translation unit compiles, so it cannot fail on the defect it was written to guard. mudler#1054 also records that its host had neither CMake nor Clang, so the change was compiled by neither compiler it concerns. And `windows-msvc-*` are skipped on `main` (mudler#503), so no baseline existed for the break to regress from and it first surfaced on an unrelated PR (mudler#983). An instance of mudler#503, not a new report of it. Fixes mudler#1068 `documentation-checkpoint` refused an earlier revision of this commit, because `FEATURE_SURFACE_PREFIXES` covered all of `src/vllm/model_executor/models/` and so any edit there owed `docs/FEATURES.md` -- for a change that adds no capability and alters no behavior. That demand is exactly what mudler#1054 answered with prose, and that prose is what crossed the `check-public-doc-tables` budgets and blocked every push in the repository (mudler#1055). Rather than feed it again, the trigger was repaired: `8fa405bb7` (mudler#1086, issue mudler#595) now keys `feature_surface` off a change to the set of `REGISTER_VLLM_MODEL(...)` registrations. This commit changes none, so the gate passes it on its own terms and no exception is claimed. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
…e docs told an operator to read, and then its comments outran the code twice (mudler#912, mudler#1091, mudler#1106, mudler#1108) (mudler#1100) Repairs the six findings of [mudler#1091](mudler#1091), a fresh review of the [mudler#912](mudler#912) wiring repair that landed as `a332fb98d` (mudler#1076), and then the four findings of [mudler#1106](mudler#1106), a fresh review of this pull request. Row `ENG-EXPERT-STREAM`, spec [`expert-streaming.md`](https://github.com/mudler/vllm.cpp/blob/main/.agents/specs/expert-streaming.md). None of the ten findings was a red test. Every one was a gap in what the gate could see, which is the same class as the defect the first repair had just fixed — and mudler#1106 finding 1 is that class reintroduced, one file away, by the change that fixes it. ## The liveness line could not print the zero the docs told an operator to read `ReportStats` had exactly one caller, `EndStep`, and it returned early on `steps == 0`. So the run where the step boundary is never reached — F1, the defect the line exists to reveal — printed nothing at all. Measured on one binary with `VT_MOE_EXPERT_STREAM_STATS_EVERY=1`: healthy, 8 lines; F1 reinjected, 0 lines and only the startup banner. `stats_every_` also defaults to 16, so a short healthy run prints nothing either. A benchmark that reads absence as failure therefore calls a working lane VOID, which is what happened to the streaming benchmark and why it had to be restarted. One final line is now printed from the store's own destructor, once per process, crossing both early returns. Measured red-then-green on the same production binary: ``` before: ./build/tests/test_expert_stream_wiring 2>&1 | grep -c 'steps=' -> 0 after: ./build/tests/test_expert_stream_wiring 2>&1 | grep -c 'steps=' -> 1 [expert-stream] steps=8 hits=96 misses=177 evictions=113 fills=177 bytes=192576 exhausted=0 advised=177 ``` **Not a second teardown hook registered when streaming is REQUESTED**, which was the first shape tried and would have covered the "asked for it, never built a store" run as well. On a CPU-only host that hook's only unique job is unreachable from any test, because `Reserve` and `Get` sit in one call chain and a device platform is what separates them. It would have been an untestable branch added to fix an untestable-branch problem. What replaces it is a protocol the docs now state: the `[expert-stream] ON ...` banner means a store was built, the final line says what it did, and each present/absent combination means exactly one thing. The residual is recorded under the spec's `## Owed` with its reason. `docs/ENVIRONMENT.md:82` and `docs/USAGE.md:3105` both instructed the operator to read `steps == 0` off the unreachable line; both now describe the final line and the banner pair. ## `CHECK(s.advised > 0)` could not fail for the defect it named Reinjecting the pre-fix unaligned `madvise` address exits 0 in 40 of 40 runs. The measured reason is that `> 0` over 48 calls is satisfied whenever heap layout happens to page-align a single slice, and one did: `advised=1` against `fills=48`. `advised == fills` is the true healthy invariant on that arm — madvise is issued on the mapping-copy path only, and only when the key is not already resident, which is exactly the condition under which `EnsureSpan` goes on to fill, with `exhausted == 0` asserted beside it as the premise. It was verified stable over 50 consecutive runs **before** being asserted rather than after. Two residuals it rests on are now stated beside it rather than left to be rediscovered: `madvise` rounds the range's END up past a heap allocation and would return `ENOMEM` on an unmapped trailing page, so the equality holds because the allocator's arena page is mapped and not because the arithmetic guarantees it; and the counters are cumulative, so the equality is a statement about everything that ran before it — the pread case at the end of the file fills without advising. That ordering is not implicit: the `CHECK_FALSE(s0.active)` opening the case fails loudly if anything ran first. ## "Every MoE entry point funnels through here exactly once per forward" was false `Qwen3_5Model::ForwardDense`, `Qwen3_5MTPModel::Forward`, `Qwen3_5MTPModel::ForwardPaged` and `Qwen3_5ReplayLayer` all reach `ExpertMlpKq -> KqExpertSlice` and marked no step. ONE of them, `Qwen3_5MTPModel::ForwardPaged`, is the production spec-decode DRAFT forward, so a draft's acquisitions stayed `protected_this_step` across the following target forward — F1 at draft scale. This paragraph said "the MTP pair" through two revisions; the table further down is the measured version, and the last section of this body is the repair. **One forward is one step, and that is the call the draft forced.** A draft is a complete forward whose slices are finished with when it returns; folding it into the target's step would pin them across a second forward for no benefit, so a spec-decode iteration advances the clock once per draft plus once for the target. The opposite mistake is the one adding guards invites — a guard nested inside another ends the step twice, which decays every resident entry an extra tick for a step that never happened — so the guard REFUSES to nest, stated as a precondition in the same idiom `MatmulF32Slice` uses for `expert >= 0` rather than handled. `RunMoeBlock` stays deliberately unguarded: it is one block, not a forward, and `qwen3_moe.cpp` owns the boundary for the model that composes it. That exemption is what makes the `steps == 0` case constructible without breaking anything. ## Three smaller repairs, all the same class `ExpertStreamer::EnsureFile` is the arm every real GGUF-mmap checkpoint takes and no test reached it, so the `file_offset + offset` composition was unverified. A CPU-local case now drives it through the production seam from a temp file at a deliberately awkward offset (4109 bytes: past a page, not on a page, not on a 34-byte Q8_0 block), and it PROVES the arm rather than assuming it — `advised` stays flat while `fills` grows, which is the one number that separates a pread from an `EnsureSpan`. No box and no 370 GiB checkpoint needed. `OwnedTensor::TowerUid`'s comment promised an identity for "this tensor's CURRENT bytes" while the code keys on `bytes.data()`. The comment now states where the guarantee stops, and a borrowed-buffer case pins both halves — same address with different contents keeps the uid, a different address does not — because [mudler#1066](mudler#1066) was that same overclaim on that same field. The smaller correct change: the comment, not the code. `SetForceFallback` has no production caller and was incrementing the operator-facing `exhausted_`, so a gate asking for the unstreamed arm told an operator to raise a budget that was never the reason. Measured: `exhausted=42` from the switch alone. It has its own counter now, kept off the stderr line because in a production process it is always zero. ## Then the review of that repair (mudler#1106) found three claims that outran the code The six repairs above are correct, and a fresh reviewer reproduced all thirteen of their mutation claims independently. What failed is what was said about them. **The teardown comment named a hook that was never built.** `qwen3_5_internal.h` said the final line is reached at teardown by "a static registered the first time streaming is requested, plus the store's own destructor, whichever runs first". There is no such static — `grep -rn 'atexit\|quick_exit'` over `qwen3_5.cpp` returns nothing, and the section above says in this same body that the hook was deliberately not built. It also promised "exactly one line per process, even on a run with zero steps" without either qualifier `docs/USAGE.md` carries. This is mudler#1091 finding 5 — a comment promising more than the code — reintroduced one file away in the change that fixes it, which is the strongest argument on record that the class is a habit and not an accident. `~Qwen35ExpertStream` is now named as the only production path to the LINE, with both qualifiers (a store must have been BUILT; the process must RUN its static destructors), and with what the exposed seam costs a caller: it takes the once-flag, so calling it suppresses the teardown line for the rest of the process. That is a fourth banner/final-line shape, and `docs/USAGE.md` now tabulates it rather than leaving a gate's own output looking like a crashed process. `docs/FEATURES.md` carried the same overclaim in one line and now says IF a store existed. **"Nothing lands dead" was claimed for four step guards and holds for one.** The claim in this body's previous revision — "every added path is reached from a production entry point at this commit" — was true of one guard in four: | Guard | Production caller | |---|---| | `Qwen3_5MTPModel::ForwardPaged` | YES — `runner.cpp:2183` -> `spec_decode/mtp/speculator.cpp:107,262` | | `Qwen3_5MTPModel::Forward` | no — reached only through `ForwardLogitsHost`, a "standalone parity convenience" (`qwen3_5_mtp.h:135`) with no caller outside `tests/` | | `Qwen3_5Model::ForwardDense` | no — the parity reference by `qwen3_5.h:234`; callers are `test_op_parity.cpp:1107`, `test_runner.cpp:1278`, `test_qwen35_paged_forward.cpp:293,320,385,403` | | `Qwen3_5ReplayLayer` | no — per-layer parity replay by `qwen3_5.h:322`; only caller `test_op_parity.cpp:1050` | Per [`.agents/reachability.md`](https://github.com/mudler/vllm.cpp/blob/main/.agents/reachability.md), a call site inside a test is not reach. **No guard is deleted.** They are correct where they sit, they cost nothing, and they become live the moment any of those entry points gains a production caller — and adding the guard later, together with the caller, is precisely how this row lost its step boundary in the first place. What changes is the record: the three are named as a staged slice that lands unreached, in the commit body, here, and under the spec's `## Owed`, tracked as [mudler#1108](mudler#1108). **The nesting refusal was asserted everywhere and pinned nowhere.** The source, the spec and this body all stated that the guard refuses to nest. Deleting its `VT_CHECK` left both focused binaries fully green — `test_expert_stream_steps` 6/6 rc 0, `test_expert_stream_wiring` 4/4 rc 0 — and it appeared in none of the thirteen mutations. No legitimate call graph can nest a step, because every forward that takes expert slices is a complete forward that no other one contains, so a gate cannot reach the refusal through production code, and a gate that reimplemented the flag would prove its own copy. `Qwen35ExpertStreamStep` therefore names its `Begin`/`End`, `detail::ExpertStreamStepScope` forwards to them, and the new case asserts the refusal twice: a second scope throws, AND a real `ForwardDense` entered while the scope is held throws too. The second is the load-bearing one — it is what shows the two share a boundary rather than agreeing by coincidence, and mutation `MN3`, which gives the scope a parallel flag, kills exactly that pair and nothing else. That refusal stays **UNGATED on `Qwen35ExpertStreamRequested()`**, deliberately. "One forward is one step" is a property of the call graph, not of the streaming lane, so a nest is a defect whether or not a store exists. Arming it only under streaming — the rare configuration — would let the default path establish a nest that nobody sees until someone switches streaming on, which is this row's recurring shape. The cost is that a forced nest reds every Qwen3.5 forward and not merely the streamed ones, and that breadth is the point. **The MSVC repair was incomplete.** `::setenv` sat at namespace scope in *both* new gates with no `_WIN32` guard, so the previous revision's claim that "only the two questions about the statistics line need POSIX" was false and so was the file comment saying the step-clock cases are "built everywhere". `setenv(3)` is POSIX, MSVC's CRT has only `_putenv_s`, `tests/CMakeLists.txt:1087` adds the target unconditionally and `scripts/build-windows-release.ps1` configures `VLLM_CPP_BUILD_TESTS=ON` — neither translation unit compiled there. Both now use `vllm_test::SetEnv` from `tests/support/test_env.h`, which is where that branch has lived since [mudler#603](mudler#603) and which a new env-flipping test is supposed to use; the one `overwrite=0` call keeps its semantics as an explicit `getenv` test, because the shim is deliberately two-argument. CI could not have reported it. The Windows lanes fail earlier, inside the product library, on the pre-existing [mudler#1068](mudler#1068) (verified: `git diff origin/main...HEAD` on `qwen3_5_weights.cpp` is empty), and a lane that never reaches a test TU cannot fail in one. The static checker that could have is blind twice over — `scripts/check-windows-portability.py` reads only the sources reachable from the shipped SERVER target, so no test TU at all, and its `POSIX_PATTERNS` name neither `setenv` nor `unsetenv`. Measured `Windows portability contract OK`, rc 0, on the unrepaired tree. Filed as [mudler#1107](mudler#1107) against `ENG-RELEASE-WINDOWS` and NOT fixed here: changing a checker's semantics needs its own spec and red-before evidence, and widening the scan to `tests/` has to separate a guarded POSIX call from an unguarded one across a large surface. ## And then the review of THAT repair: the code was right and the sentence about it was not, for the fourth time A fresh scoped review of `4ada1fb8d` returned FAIL on one blocking finding and three advisory ones. It reproduced every mutation independently and ran a 500/500 gate, so nothing here changes behaviour. What it found is that two source comments asserted the reachability claim **this same delta's records refute**: `qwen3_5.cpp` called "the MTP pair" the production spec-decode draft path five lines above the guard, and `test_expert_stream_steps.cpp` said the same, while the spec's `## Owed`, the `mudler#1108` index row and this body's own table all said one guard in four. The tree stated two contradictory things about one fact, and the false half sat where a reader hits it first. Its closing warning is the instruction this round actually followed: *"this is the fourth consecutive review in which the code was right and the sentence about it was not, and a targeted patch on two known lines is how the fifth one gets set up."* So rather than patch the two cited lines, every claim in this row's delta about reachability, production paths and guarantees was audited against the code. **51 claims examined, 12 wrong, 12 repaired.** Search terms: `production`, `reached`/`reaches`, `only caller`, `no caller`, `every`, `always`, `exactly`, `guarantee`, `never`, plus call-graph greps for each named symbol (`ExpertMlpKq`, `KqExpertSlice`, `ForwardLayers`, `ForwardDense`, `ForwardPaged`, `ForwardLogitsHost`, `Qwen3_5ReplayLayer`, `ExpertStreamSetForceFallback`, `ExpertStreamFlushStats`, `FlushFinalStats`, `ExpertStreamStepScope`, `RunMoeBlock`, `ReportStats`). The other 39 hold, including every one the reviewer had already reproduced. Five of the twelve were cited by the review. **Seven were not**, and they are the reason the audit was worth doing rather than the patch: | # | Where | The claim | Why it is wrong | |---|---|---|---| | 1 | `qwen3_5.cpp:5359` | "TEARDOWN IS THE REAL CALLER, in `~Qwen35ExpertStream` below" heading `FlushFinalStats` | The destructor deliberately does NOT route through `FlushFinalStats` — its own comment three lines below says so. Identical in shape to the advisory finding about the header, and it had gone unreported | | 2 | `qwen3_5.cpp:8176` | `Qwen3_5MTPModel::Forward`'s guard: "a spec-decode iteration therefore advances the clock once per draft plus once for the target" | True of the pair, but written inside the overload that never runs it. `ForwardPaged` does | | 3 | `.agents/specs/expert-streaming.md:1233` | "The MTP pair is the production spec-decode DRAFT path" | Same false claim, in the record that is supposed to be the correction | | 4 | `.agents/specs/expert-streaming.md:1291` | "named as the only production caller" | Caller vs path — the same conflation as the header | | 5 | `.agents/issue-index.md:310` (`mudler#1091` row) | "The MTP pair is the production spec-decode DRAFT path" | In an APPEND-ONLY record. Corrected in its only open window: the row was added by this pull request and does not exist at the merge base, so the diff carries no removal line and a union merge cannot duplicate it. `check-issue-index-append-only.py` re-run green against `22056e238` | | 6 | this body | "The MTP pair is the production spec-decode DRAFT path" | Contradicted by this body's own table, 40 lines later | | 7 | this body | "named as the only production caller" | Same conflation | The three advisory findings are repaired as asked. **`docs/FEATURES.md`** promised the exit line under one of its two qualifiers; the cell measured 220 of 220 against `MAX_CELL_CHARS`, so the fix is a trade and the trade is stated: `(mudler#1106)` is spent to buy "on a clean exit", which is worth more to an operator than a cross-reference the spec, both docs and `git log --grep` already carry. Back to exactly 220. **`docs/USAGE.md`**'s fourth shape was keyed on an observable that does not discriminate — `PrintStatsLine` makes the periodic and final lines byte-identical, so a run of ≥16 steps that then crashes matches it as well as row 3 — and is now keyed on its cause, a call to `ExpertStreamFlushStats`, with a paragraph saying outright that stderr cannot separate the two. **`qwen3_5_internal.h:422`** now leads with the sharper true statement: `ExpertStreamFlushStats` has ZERO production callers and exists for the gate, while the only production path to the LINE is the destructor, which does not call it. **On the judgement call the review left open:** `ForwardPaged`'s caller is itself "UNREACHABLE unless a speculator is configured" (`runner.cpp:2120`), and `.agents/reachability.md` defines a production entry point as a server or command-line path **on its default configuration**. "Has a production caller" is true; "reached on the default configuration" would not be. That nuance is recorded on `mudler#1108`, which already owns this debt — but the qualifier is also carried in the three sentences being rewritten here anyway, because leaving a knowingly imprecise clause standing immediately after an audit for imprecise clauses is the failure this round exists to stop. No guard changed, and no claim structure was re-litigated beyond that clause. Nothing executable changed, which is exactly why only a reader catches this class: the focused gate cannot regress on any of it. The evidence is the audit — the counts and greps above — plus an unchanged gate. ## Evidence Red first for every fix. Findings 1, 3 and 6 went red on the unmodified tree (0 statistics lines; `Steps() - before == 1` failing `0 == 1` at all four entry points; `off.exhausted == 0` failing `42 == 0`). Finding 2's red is the reinjected F5 defect against the tightened assertion (`1 == 48`). Findings 4 and 5 are reachability, so their red is the mutation. The mudler#1106 repairs are comments, records and one pinned guarantee, so the guarantee's red is its mutation and the rest have nothing executable to redden. **16 mutations, 16 caught.** Every row records a non-empty `git diff --stat` or a changed sha256, a compile status, and the doctest case count, because a mutation that does not build reads as a passing test and a filter matching nothing prints SUCCESS. Two first attempts were INVALID rather than passing and are recorded as such: `M9` did not build (`-Werror` on an unused `file_offset`), and `M1`/`M2` first reported the CHILD process's doctest summary, because a failing case dumps the child's output into the parent's log and the first `test cases:` match therefore belongs to the child. Every row below takes the LAST match. | id | finding | target | applied | compiled | run | doctest cases (parent) | verdict | |---|---|---|---|---|---|---|---| | M1 | 1 teardown flush | `test_expert_stream_steps` | 153 ins / 23 del | rc=0 | rc=1 | 6 run / 5 passed / 1 failed | CAUGHT | | M2 | 1 `final` bypasses both early returns | `test_expert_stream_steps` | 156 ins / 24 del | rc=0 | rc=1 | 6 run / 4 passed / 2 failed | CAUGHT | | M3 | 2 unaligned madvise (the pre-fix F5 defect) | `test_expert_stream_wiring` | 157 ins / 25 del | rc=0 | rc=1 | 4 run / 3 passed / 1 failed | CAUGHT | | M4 | 3 MTP `Forward` guard | `test_expert_stream_steps` | 153 ins / 23 del | rc=0 | rc=1 | 6 run / 5 passed / 1 failed | CAUGHT | | M5 | 3 MTP `ForwardPaged` guard | `test_expert_stream_steps` | 153 ins / 23 del | rc=0 | rc=1 | 6 run / 5 passed / 1 failed | CAUGHT | | M6 | 3 `Qwen3_5Model::ForwardDense` guard | `test_expert_stream_steps` | 153 ins / 23 del | rc=0 | rc=1 | 6 run / 5 passed / 1 failed | CAUGHT | | M7 | 3 `Qwen3_5ReplayLayer` guard | `test_expert_stream_steps` | 153 ins / 23 del | rc=0 | rc=1 | 6 run / 5 passed / 1 failed | CAUGHT | | M8 | 3 regression: the `ForwardLayers` guard itself | `test_expert_stream_wiring` | 154 ins / 24 del | rc=0 | rc=1 | 4 run / 1 passed / 3 failed | CAUGHT | | M9 | 4 pread drops `file_offset` | `test_expert_stream_wiring` | 155 ins / 24 del | rc=0 | rc=1 | 4 run / 3 passed / 1 failed | CAUGHT | | M10 | 4 pread drops the slice offset | `test_expert_stream_wiring` | 155 ins / 24 del | rc=0 | rc=1 | 4 run / 3 passed / 1 failed | CAUGHT | | M11 | 5 `TowerUid` stops re-stamping a moved buffer | `test_qwen36_weights` | 1 ins / 1 del | rc=0 | rc=1 | 10 run / 8 passed / 2 failed | CAUGHT | | M12 | 6 forced fallback charged back to `exhausted` | `test_expert_stream_wiring` | 153 ins / 22 del | rc=0 | rc=1 | 4 run / 3 passed / 1 failed | CAUGHT | | M13 | reachability: the slice seam itself | `test_expert_stream_wiring` | 155 ins / 24 del | rc=0 | rc=1 | 4 run / 1 passed / 3 failed | CAUGHT | | MN1 | mudler#1106.3 delete the nesting `VT_CHECK` | `test_expert_stream_steps` | 2 lines, sha `9ca33ee207a5`→`52b389634b48` | rc=0 | rc=1 | 7 run / 6 passed / 1 failed | CAUGHT | | MN2 | mudler#1106.3 `End` never clears the flag | `test_expert_stream_steps` | 1 line, sha `9ca33ee207a5`→`f3f740d98573` | rc=0 | rc=1 | 7 run / 3 passed / 4 failed | CAUGHT | | MN3 | mudler#1106.3 the scope gets a PARALLEL flag | `test_expert_stream_steps` | 13 lines, sha `9ca33ee207a5`→`b0d983980f3b` | rc=0 | rc=1 | 7 run / 6 passed / 1 failed | CAUGHT | `MN1` reds all six assertions of the new case and reports `Steps() - before` as 3 where 1 is correct, which is the double-count the guard exists to stop. `MN3` reds exactly two — `forward_threw` and its message — which is the pair that proves the scope and the production guard share a boundary; the "a second scope throws" half survives a parallel flag by construction, and that is why it is not asserted alone. Each mutation was restored from a byte copy (never `git checkout --`, which would have restored the index over uncommitted work) and the file's sha256 re-checked against the pre-mutation value before the next one ran. **Not mutation-proven: the Windows repair.** No MSVC is reachable from this host, the Windows CI lanes cannot report a test TU while mudler#1068 stands, and the static checker that would have caught it is mudler#1107. The spec's `## Owed` says so rather than leaving it to be assumed. ## Gate Merged `origin/main` at `22056e238` first. The branch was behind it, so `agent-preflight.sh` had been SKIPPING its `commit-trailers` and `commit-style` range gates, and a conflict-free `git merge-tree` says nothing about whether the merged tree compiles. `cmake --build build -j 12 && ctest -j 6` on the MERGED tree, CPU-only Release, 20 cores: ``` build_rc=0 100% tests passed, 0 tests failed out of 500 ctest_rc=0 ``` 500 rather than 498: `test_expert_stream_steps` is new here, and `test_nemotron_h_moe_device` arrived with the merge. `scripts/agent-preflight.sh` reports **All gates green**, with `doc-checkpoint`, `issue-index append-only`, `commit-trailers` and `commit-style` all RUN over `22056e238..HEAD` rather than skipped. The claims-accuracy repair on top (`3ef9d023c`) reruns the same gate from a fresh build tree, and it has to be UNCHANGED because nothing executable moved: ``` build_rc=0 100% tests passed, 0 tests failed out of 500 ctest_rc=0 ``` `scripts/agent-preflight.sh` **All gates green**, rc 0, both `--staged` and over the committed range. Two notes on instrument hygiene, since this round is entirely about claims that were not checked. The header edits invalidate dependents, so the first `cmake --build` was followed by a second that recompiled **506 targets** — running `ctest` after the first alone would have measured stale objects; a third build reports `ninja: no work to do`, which is what says the tested binaries are the committed tree. And `issue-index append-only` passing is not by itself evidence, because a vacuous range also prints OK: the range it actually examined is **5 added lines and 0 removal lines** over `22056e238..HEAD`, which is why editing the `mudler#1091` row this pull request appends cannot union-duplicate anything. `windows-msvc-cpu` and `windows-msvc-vulkan` are red, as they are on `main` and on every recent pull request: `qwen3_5_weights.cpp` does not compile under MSVC since mudler#1054, filed as [mudler#1068](mudler#1068) and untouched here. Two `docs/FEATURES.md` trades are recorded rather than hidden, because that cell sits against the 220-character keyed-table limit and every addition to it evicts something. The first cost "LFU + LRU tiebreak" to buy "IF a store existed": an eviction policy is implementation and is stated twice in the spec, while which guarantee an operator actually gets is not. The second cost `(mudler#1106)` to buy "on a clean exit", the qualifier the first trade had left out — a cross-reference the spec, `docs/ENVIRONMENT.md`, `docs/USAGE.md` and `git log --grep` all still carry, spent on the half of the guarantee an operator cannot recover from anywhere else. The cell measures 220 of 220 both before and after. Not run: anything on `dgx.casa`. A benchmark held the host mutex and every repair here is CPU-local. The decode re-measure on a live cache stays owed for that host, unchanged by this change. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
… in ltx2.h that was wrong (mudler#1092) (mudler#1102) The LTX-2.5 video denoise loop ran ONE unguided forward per step. Every recipe resolved a `video_guidance` that nothing read, so a `pipeline_kind = one_stage` render ignored `cfg_scale = 3.0`, `stg_scale = 1.0`, `rescale_scale = 0.7` and `modality_scale = 3.0` and denoised along a different trajectory than `ti2vid_one_stage.py:221-226` @ `fd4ded7f`, which builds a `FactoryGuidedDenoiser` from exactly those. Issue [mudler#1092](mudler#1092). Spec [`.agents/specs/ltx25-guided-video.md`](.agents/specs/ltx25-guided-video.md), committed at `36510ec2d`, before any code. `_guided_denoise` (`ltx-pipelines utils/denoisers.py:61-211`) is now ported, in its own translation unit because upstream has its own file. It assembles the four passes -- cond, uncond, perturbed, isolated-modality -- from the UNION of what the two guiders want, runs each through the caller's `X0Model`, and combines each modality with its own guider over the same splits. Four unported pipelines (`a2vid_two_stage.py:230`, `ti2vid_two_stages.py:248`, `ti2vid_two_stages_hq.py:271`, `keyframe_interpolation.py:232`) were each blocked on this one piece; none is any more. ## The conversion lives in the caller's lambda, not in the seam `DiffusionStage` never hands the loop a velocity model: it hands `X0Model(builder.build(...))` (`utils/blocks.py:480-482`, forward at `model.py:590-604`). The seam therefore takes a callable and can only ever receive denoised tensors. Combining velocities and converting once afterwards is a different function wherever `rescale_scale != 0`, which is 0.7 on every video row of the params table. That defect shipped on the audio arm of this tree and is [mudler#1039](mudler#1039). `post_process_latent` stays OUT of the denoiser. Upstream applies it in the LOOP, to the guider's result (`utils/samplers.py:35`, `:484`), and the difference is not cosmetic: the rescale is a scalar over the whole tensor, so it multiplies the conditioned tokens too, and the after-guider application is what pins them back to `clean`. ## The NOT-PORTED note in ltx2.h was wrong, not stale It refused `SKIP_A2V_CROSS_ATTN` and `SKIP_V2A_CROSS_ATTN` because "nothing upstream that this port serves constructs them -- STG is built from `stg_blocks` and reaches the SELF-attention types alone". STG does. The isolated-modality pass does not: `_guided_denoise` builds BOTH cross types with `blocks=None` whenever either guider has `modality_scale != 1.0` (`denoisers.py:125-138`), and every VIDEO row of the params table sets it to 3.0. The sentence was true of text-to-audio, which pins the field to 1.0 (`t2a_one_stage.py:202`), and it was written while text-to-audio was the only guided path here. Both directions are now `cross_attn_skip_all` booleans on `Ltx2DitPerturbation`, gating the A2V and V2A branches exactly as `transformer.py:335` and `:367` do. The same shape appears once more and is corrected the same way. `negative_prompt` and the five `audio_*` guider knobs were refused on every non-t2a engine, on the same reading of upstream. `default_1_stage_arg_parser` carries the whole audio guider row beside the video one (`utils/args.py:947-1066`, the audio row opening at `:1008`) and `TI2VidOneStagePipeline` consumes both (`ti2vid_one_stage.py:211-218`). That refusal is gone; the direction that survives refuses a knob describing a PICTURE on a pipeline that renders none. The case that asserted the old behaviour is rewritten rather than deleted, because "this used to be refused" is what a later reader needs. ## What a caller gains, and what is refused Seven per-generation extras mirror `default_1_stage_arg_parser`, reaching `ltx2-gen` and the C ABI: `--video-cfg-guidance-scale`, `--video-stg-guidance-scale`, `--video-rescale-scale`, `--video-skip-step`, `--video-stg-blocks`, `--a2v-guidance-scale`, `--v2a-guidance-scale`. Every one is refused whole on a phase whose recipe sets `allow_guidance_override = false` -- the distilled and retake recipes, whose guidance is distilled into the weights. That field had never been read. The unconditional forward needs a negative conditioning. With a tower it is the second half of the encode `GenerateAudioOnly` already performed and discarded; without one, `negative_prompt_embeds_path` and `negative_audio_prompt_embeds_path` are the negative half of the existing embeds fallback. That pair is a LOCAL ADAPTATION and is recorded as one in the spec: upstream has no embeds surface at all. With neither, a `cfg_scale` other than 1.0 is refused by name rather than served the positive context twice, which would leave the whole classifier-free term at exactly zero. Two further refusals exist because the alternative renders. An `stg_blocks` list naming no block this checkpoint has would perturb nothing under upstream's membership test, leaving `stg_scale * (cond - perturbed)` at zero. And the perturbed or isolated-modality pass on the device arm cannot run at all: `Ltx2DitForwardDevice` takes no `perturbations` argument, so it is refused by name rather than served an unperturbed forward. That arm is OWED and the spec lists it under `## Owed`. ## What is unchanged `distilled_two_stage`, `dfr`, `retake` and `dmd2` keep the guider they always had, which is `Ltx2MultiModalGuiderParams`'s own default construction and is upstream's `_POSITIVE_ONLY_GUIDER` (`denoisers.py:25-28`). They issue one forward per step through the new seam, and no golden in the suite moved. ## Reachability Entry point: `vllm_video_generate` -> `VideoEngine::Generate` on an engine loaded with `pipeline_kind = one_stage`, a documented value of a documented load extra that needs no other flag. Every gate case enters there; nothing constructs a guider, a DiT, a modality or a perturbation by hand. Deleting the production call site -- the `Ltx2GuidedDenoise` line in the phase loop, replaced by the single unguided forward this change removes -- turns the suite RED at 3 failed cases and 13 failed assertions (M11 below). ## The gate `test_ltx2_video`: 71 cases, 2145 assertions, exit 0. It asserts `x0 == latent - sigma*velocity` per token on ALL FOUR arms, with the PER-TOKEN sigma rather than the schedule scalar: exact in x0 space, off by the whole sample in velocity space. Non-vacuity is stated twice, once for a zero sample and once per arm for a zero velocity. Each arm is also asserted to DIFFER from the conditional arm, so a pass whose context or perturbation never reached the forward fails rather than passing as a perfectly converted copy. It then replays `Ltx2MultiModalGuidance` over the recorded arms, and again over arms REBUILT FROM THE RAW VELOCITIES, and recovers the Euler step's input from the latent the sampler wrote. A seam-level control puts the two spaces apart only at a non-zero rescale, with the modality term present, which the T2A control could not carry. A second case runs the same guided configuration WITH an image conditioning, because `post_process_latent` is a literal no-op without one. ## Mutations Twelve, each reporting three facts, because two of them are how a mutation lies: that it applied (`git diff --stat`), that it BUILT (compile-error count), and the exit code captured directly rather than after a pipe. | # | Mutation | Applied | Built | Exit | Result | |---|---|---|---|---|---| | M1 | cond pass left in VELOCITY space | 1 file, +1 | yes, 0 errors | 1 | RED | | M2 | uncond pass left in VELOCITY space | 1 file, +1 | yes, 0 errors | 1 | RED | | M3 | perturbed pass left in VELOCITY space | 1 file, +1 | yes, 0 errors | 1 | RED | | M4 | modality pass left in VELOCITY space | 1 file, +1 | yes, 0 errors | 1 | RED | | M5 | second `ToDenoised` BELOW the step-0 record | 1 file, +3/-1 | yes, 0 errors | 1 | RED | | M6 | second `ToDenoised` ABOVE the step-0 record | 1 file, +3/-1 | yes, 0 errors | 1 | RED | | M7 | uncond pass given the POSITIVE context | 1 file, +1/-3 | yes, 0 errors | 1 | RED | | M8 | modality pass given NO cross-attn perturbation | 1 file, +2/-2 | yes, 0 errors | 1 | RED | | M9 | video self-attn perturbation dropped in the DiT | 1 file, +1/-1 | yes, 0 errors | 1 | RED | | M10a | `post_process_latent` ADDED per arm | 1 file, +3/-2 | yes, 0 errors | 0 | GREEN, an IDENTITY -- see below | | M10b | `post_process_latent` MOVED per arm | 1 file, +4/-3 | yes, 0 errors | 1 | RED | | M11 | REACHABILITY: the `Ltx2GuidedDenoise` call site deleted | 1 file, +12/-1 | yes, 0 errors | 1 | RED (3 cases, 13 assertions) | Every restore was verified with a scoped `git diff --stat` reporting clean, and every restored file had its mtime bumped before the rebuild, because a restored file with an older mtime lets ninja skip and the NEXT measurement runs the PREVIOUS mutation's binary. **M10a is a no-op, and saying so took a measurement.** `post_process_latent` is `x*mask + clean*(1-mask)`, so it can only touch a mask-0 token; such a token's per-token sigma is 0, so `X0Model` returns `latent - 0*v`, which is `latent`; and a conditioned token's `latent` IS its clean value. Every arm already equals what post-processing would write. The first response to the green was to strengthen the gate -- that is where the rebuilt-from-velocities replay came from -- and when that did not move it either, the conditioned case was written to state the identity in an assertion. M10b, the placement that actually changes the render, is RED against it. ## Gate numbers ``` cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF cmake --build build -j6 && ctest --test-dir build -j4 --output-on-failure ``` | | | |---|---| | head | `420f6b4741ef8b5faae0874ccfb2d622d7c4a7d6`, remote-verified with `git ls-remote` | | `CONFIGURE_EXIT` | 0 | | `BUILD_EXIT` | 0, `: error:` count 0 | | `ctest -N` | Total Tests: 498 | | `CTEST_EXIT` | 0 | | result | 100% tests passed, 0 failed out of 498; 2 skipped (`test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`) | | `No space left` / `BFD` | 0 in both logs; positive control on the same file, `Passed` matches 496 lines | | disk | 13 G free, 98% used | | load | 1.63 at start, 38.57 at end (another agent building) | ## What this does NOT claim No number here comes from a running oracle. vLLM-Omni is UNPINNED ([mudler#633](mudler#633)) and carries no LTX-2.5 recipe; no LTX-2.5 checkpoint on this host has a recorded sha256 ([mudler#1048](mudler#1048)). The guidance is gated against upstream SOURCE at `fd4ded7f`, not upstream OUTPUT, and the spec's honesty statement says so. [mudler#1049](mudler#1049) is PARTLY retired: `Ltx2BatchedPerturbationConfig` now has a product caller. `Ltx2Guidance`, `Ltx2CfgDelta` and `Ltx2StgDelta` stay dead, and the spec argues why routing through them would mean inventing a kind dispatch upstream does not have rather than closing a wiring gap. # The fresh review, and what it moved The review returned CHANGES REQUESTED on `420f6b474`: one blocking finding and ten non-blocking. It reproduced the core exactly -- the seam, the x0 space, all four arms, the reachability, the gate -- and independently confirmed the live defect: at the merge base `video_guidance` had exactly two hits, a declaration and a write with no reader, against a positive control where `audio_guidance` finds its T2A consumer. None of that is revisited below. ## B1, the blocking one: the two cross booleans were gated together `Ltx2DitPerturbation`'s two flags reached `Ltx2BlockArgs` and the gate could not tell which of them the DiT applied. Three mutations were **GREEN** over the whole binary: the DiT ignoring `video_cross_attn_skip_all` (M12), ignoring `audio_cross_attn_skip_all` (M13), and **swapping** which flag gates which direction (M15). Only ignoring both (M14) was caught. A build that applies one direction, or applies both to the wrong ones, renders -- on the DEFAULT video arm, whose `modality_scale` is 3.0 -- with the isolated-modality term half wrong. Two things made the shipped-path case blind to it. The end-to-end `MaxAbsDiffOf(video_first_modality, video_first_cond)` still fires with one direction applied, because the modality pass still differs from `cond`. And `Ltx2ConditioningTrace::video_modality_skipped_{a2v,v2a}` is assigned from the perturbation struct **the seam built** (`ltx2_denoisers.cpp:315-316`), so it records what was handed over and nothing about what the DiT did with it -- while its message claimed the latter. That message is corrected rather than left to overstate what it measures. **The repair is test-only, and the separation comes from upstream's own predicates.** `run_a2v` needs the VIDEO stream enabled and the audio stream merely PRESENT; `run_v2a` needs the reverse (`transformer.py:265-269`). So a forward with `audio->enabled = false` runs A2V alone, and one with `video->enabled = false` runs V2A alone -- the configuration `ltx2.h` already documents as rendering rather than failing. Each row asserts BOTH halves: the flag for that direction MOVES the stream it writes, and the flag for the other direction leaves it BIT-IDENTICAL. The second half is what detects the swap. Doing this on a both-enabled forward is impossible on this fixture, because block 1's V2A reads what block 0's A2V wrote. `ltx2.h`'s disclosure that called M15 untestable is **retired**. Its premise was true -- both directions are off together on the shipped path -- and its conclusion did not follow, because nothing obliges the separating test to use the shipped combination. | # | Mutation of `ltx2_dit.cpp` | `git diff --stat` | Built | Exit | Result | Fails at | |---|---|---|---|---|---|---| | M12 | DiT ignores `video_cross_attn_skip_all` | 1 file, +1/-1 | yes, 0 errors | 1 | **RED** | `moved(a2v_off.video, base.video)` | | M13 | DiT ignores `audio_cross_attn_skip_all` | 1 file, +1/-1 | yes, 0 errors | 1 | **RED** | `moved(v2a_off.audio, base.audio)` | | M14 | DiT ignores BOTH | 1 file, +2/-2 | yes, 0 errors | 1 | **RED** | both of the above, the both-flags subcase, and the shipped-path `mod != cond` | | M15 | DiT SWAPS the two flags | 1 file, +2/-2 | yes, 0 errors | 1 | **RED** | all four directional checks, including both "must NOT move" ones | Every run was the WHOLE binary rather than a `--test-case` filter, because several case names here contain commas and doctest splits `-tc` on them; a truncated filter matches zero cases and prints `SUCCESS!` with exit 0. Each run is recorded with its case and assertion counts so a zero-count run cannot pass for a green one: baseline **72 cases / 2182 assertions / exit 0**, and each mutation ran the same 72 and 2182 with 1, 1, 2 and 1 cases failing respectively. Every restore was byte-verified and mtime-bumped before the rebuild. ## B3, the other finding with product code behind it The empty-`stg_blocks` refusal is a real divergence and the recorded reason was wrong. Measured at `fd4ded7f`: | Evidence | Where | |---|---| | "Set to `[]` to disable STG", in the same table and idiom as `stg_scale` -> 0.0 and `cfg_scale` -> 1.0 | `ltx-pipelines/docs/multimodal-guidance.md:13` | | `MultiModalGuiderParams.stg_blocks` DEFAULTS to `[]` | `guiders.py:204` | | the flags are `nargs="*"`, so `[]` has a CLI spelling; `nargs="+"` was the one-character way to forbid it | `args.py:979-985`, `:1039-1045`, `:1107-1113` | | `LTX_2_3_HQ_PARAMS` SHIPS `stg_blocks=[]` on both modalities | `constants.py:105`, `:113` | | no validation of `stg_blocks` anywhere in that tree | whole-tree search, null results recorded | `blocks=None` means EVERY block and `blocks=[]` means NO block (`perturbations.py:26-33`), and `ApplyStgBlocksExtra` exists to keep PRESENT-and-empty distinct from ABSENT -- which the refusal then made unreachable. Dropped in `ApplyGuidanceOverrides`, exempted in `check_reaches_a_block`. **The out-of-range refusal stays**, because that is a request disagreeing with the CHECKPOINT rather than a caller asking for nothing, and upstream never meets it (48-block checkpoints only). Upstream does not skip the pass either: `do_perturbed_generation` reads `stg_scale` alone (`guiders.py:279-281`). The new case asserts the pass RAN, perturbed no block, and returned `cond` **bit for bit**, with a named-block control beside it so the case is about emptiness rather than about the extra being read at all. **One sub-claim is REJECTED on evidence.** The finding argued that `audio_stg_blocks=""` is still accepted on `t2a_one_stage` because that path returns before `ApplyGuidanceOverrides`. It does return there, and the request is still refused -- by `ltx2_t2a.cpp:203-214`, which builds the block mask and fails when no bit is set, and which `git log -S` puts on `main` at `0b0b8900f` with [mudler#1032](mudler#1032), not on this branch. So there is no asymmetry today: both arms refuse and both diverge from upstream. Fixing the video half creates one, which is why [mudler#1111](mudler#1111) is filed, indexed, and listed under `## Owed`. It is not fixed in flow because it changes a landed row's gated behaviour and one of its cases. ## B6, the anchors Re-derived against `fd4ded7f` from the sentence making each claim, never by reading text out of the cited span. `_guided_denoise` is **61-211**, not 62-207. `enabled = not skip` is at **158, 168**; 151 and 161 are the `= None` initializers. The V2A guard is **367**; 366 is blank. The batched config is built at **182-187**; 172-176 is a comment plus the per-sample replication at `:175`. The partial blend is **572-573**. The one `PromptEncoder` call is **166-174**. `default_1_stage_arg_parser` is **930-1067** with its guider flags at **947-1066**. The two `--*-stg-blocks` flags open at **979-985** and **1039-1045**. `cross_attn_skip_all` is DECLARED at `transformer_args.py:70`; 118 is a call site. `modality_scale = 3.0` is at `constants.py:54, :64` and `_PARAMS_SINCE_VERSION` at **130-133**, so the cited 40-80 covered neither. `CFGGuider` and `STGGuider` are **11-27** and **56-74**. The `perturbations` ARGUMENT is `model.py:493`; 492 is the `def`. No gate protects a spec anchor ([mudler#632](mudler#632)), so the 43 replacements were applied by a script that asserts the expected hit count per edit and refuses the whole run on a mismatch. Two were caught that way and re-derived. ## The rest | Finding | Disposition | |---|---| | B2 | `docs/FEATURES.md` still called T2A "the only GUIDED arm", made false by this PR's own row two lines below. Corrected **inside the existing cell** at 202 of 220 chars; the page's prose-paragraph count is unchanged at **21 of 21**, because adding a paragraph there re-reds `main` for the whole repo ([mudler#1055](mudler#1055)). `check-public-doc-tables.py` green. | | B4 | `INFO("arm = " << arm.name)` printed `arm = 1`, doctest stringifying a `const char*` through its bool overload, so M1-M4 produced byte-identical failure context. Wrapped in `std::string` at all three sites in the file. | | B5 | The rescale control's modality claim is structurally true and numerically inert. The case now MEASURES it: `4.054e-01` at `modality_scale` 3.0 against `4.118e-01` at 1.0, asserted to agree within a factor of two. Restated in the case title, the comment and spec 7.2, so a later reader cannot lean on this control for modality coverage -- the modality arm's gate is the per-arm invariant, which M4 turns red. | | B7 | "the seam cannot be handed a velocity" is caller discipline, not a type guarantee: `Ltx2X0Outputs` carries the velocity beside the prediction, so a lambda that swaps them compiles and renders. The **claim** is restated and the code left alone, because dropping the velocity would delete what the invariant is checked against. M1-M4 are the real gate. | | B8 | `origin/main` merged in twice (it moved during the repair) and the gate rerun on the merged tree. Spec 1 now carries mudler#1093-mudler#1096 from `281e6a120`. `.agents/issue-index.md` was rebuilt both times by taking `origin/main` wholesale and re-appending this branch's rows: 320-line prefix byte-identical, 304 rows, 304 distinct ids. | | B10 | The new `docs/USAGE.md` section gains the `/v1/videos` caveat its two siblings carry, a flag-to-extra table with the raw key spellings an ABI caller needs, the audio row's spellings, the load-extra status of the two negative-embeds keys, and the empty-list behaviour B3 decided. The rows are placed in that section rather than in the retake-scoped table at `:3128`, where they would be filed under the wrong pipeline. | | B9, B11 | recorded by the reviewer as not this repair's. | ## Gate numbers, on the merged head ``` cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF cmake --build build -j6 && ctest --test-dir build -j4 --output-on-failure ``` | | | |---|---| | head | `8e0f19650dffa5b4daff6ec16ca9e27a49dc8508`, remote-verified with `git ls-remote` | | `CONFIGURE_EXIT` | 0 | | `BUILD_EXIT` | 0, `: error:` count 0 | | `ctest -N` | Total Tests: 499 | | `CTEST_EXIT` | 0 | | result | **100% tests passed, 0 failed out of 499**; 2 skipped (`test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`) | | an earlier run of the same gate | at `09aee8cc2`, under load 134-168 from other agents on the box, `test_serve_low_tools` ([mudler#428](mudler#428)) and `test_engine_core_proc` ([mudler#1052](mudler#1052)) failed and both **Passed** on a serial re-run at load 71, exit 0. Both are green in the run above | | `test_ltx2_video` | Passed, 204.30 s; standalone 72 cases / 2182 assertions / exit 0 | | `No space left` / `BFD` | 0 in both logs, with positive controls on the same files: 495 `Linking` lines in the build log, 497 `Passed` lines in the ctest log | | disk | 68 G free, 85% used at the gate; it reached **23 M free / 100%** earlier in this session and `check-test-registration` failed with "Cannot open file for write" plus CMake's misreported "Inappropriate ioctl for device" -- an ENOSPC wearing a verdict about the code, confirmed by `dd` writing 22 of 64 MB and by the same gate passing once space returned | | load | 9.58 at the gate's start, 58.34 at the end (other agents on the box) | FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…o step 2 carries state (mudler#517) (mudler#1113) `ForwardNemotronHForCausalLM` returned `HostLogits(NemotronHForward(...))`: a host reference that recomputes Q/K/V over the whole sequence on every call, pages nothing, and rebuilds the recurrent state from scratch each step. A scheduler step reaching it would have decoded step 2 onward with FRESH mamba state and NO KV -- fluent output, wrong tokens, no error -- which is why the G-SAFE interlock refused a paged step outright rather than letting it run. This lands the forward that interlock was waiting for. `NemotronHPagedForward` writes each step's K/V into `input.attn_kv` at `attn_meta.slot_mapping` and reads attention back out of those pages, and it gathers the conv and SSM rows out of `input.gdn_state` at the metadata's state indices, runs the mixer over them, and scatters the updated rows back. G-SAFE therefore loses its `attn_kv` and `gdn_state` clauses in this same change and KEEPS `num_reqs <= 1`, which A2-B removes. ## The chain, entry point down `include/vllm.h::vllm_complete_tokens` -> the engine -> `GPUModelRunner` -> `runner.cpp:1465 ModelRegistry::Forward` -> `ForwardNemotronHForCausalLM` -> the paged branch -> `NemotronHAttnBlockPaged` and the recurrent gather/scatter. Mutation P-M7 is the proof rather than this paragraph: deleting the paged branch reds 6 of the 12 cases. ## Four decisions the spec made and this change did not re-open - The state slot is INDEXED even at one request. A forward that hardcodes slot 0 passes every gate this unit owns and then fails silently under batching, so the indexing machinery lands here and only the count is one. - The persistent conv page stays **bf16**. The bf16 conv-state kernel arm the governing spec called unplanned work had already landed at `908bad0ac`, so the f32 the conv kernel wants is the transient working row `vt::GdnStateGather` produces. No page is widened and no f32 escape is taken. - Attention does NOT route through `dense_attn::AttnBlock`, which applies `vt::RopeNeox` unconditionally and reads an eps this checkpoint does not ship, while NemotronH has no positional embedding at all (mudler#941). The model-local block A2-R established is extended instead. - The runner-routing allowlist entry is **NARROWED, not removed**. `lm_head` is NVFP4 and refuses on a non-CPU queue until A2-Q2b lands; deleting the entry while the forward still returns host logits would red `check-runner-routing-consistency.py`, and widening the allowlist to satisfy that checker is the defect it exists to stop. ## The red, and what it was On the base commit every runner-driven case refuses at `nemotron_h_registry.cpp:161` with "the PAGED/BATCHED decode path is not ported", because the runner hands it non-empty `attn_kv` and `gdn_state`: `7 failed | 2 passed`, 1503 assertions, exit 1. After the change: **12 cases / 3256 assertions, exit 0**. ## Two cases exist because the token arms could not see the carry With BOTH recurrent states zeroed on every step the paged decode still emitted `26,17,4,20,2,23`, byte-identical to the reference, and mutations P-M1, P-M2, P-M4 and P-M9 all survived the first pass: at this geometry the MoE block's `routed_scaling_factor` dominates the residual and the argmax over 32 vocabulary entries does not move. The spec's stop condition names that a coverage hole owing a direct assertion rather than a pass, so a decode step's per-layer output is now compared against the reference's last row (144 elements over 5 layers, worst relative 1.20e-3 against a measured band of 2e-3) and a fresh prefill over a DIRTY state slot against a fresh reference (1728 elements, worst relative 0). With those, all nine mutations RED. ## Mutations Every arm reports its `git diff --stat` (the edit applied), compile exit and error count, a binary sha distinct from baseline, a non-zero case count, and a byte-identical restore afterwards. | # | Mutation | Result | |---|---|---| | P-M1 | carried SSM state zeroed every step | RED 1/12 | | P-M2 | carried CONV state zeroed every step | RED 1/12 | | P-M3 | K/V paged but attention read from a fresh dense K/V | RED 4/12 | | P-M4 | fresh-request zeroing dropped | RED 1/12 | | P-M5 | state slot replaced by a literal 0 | RED 1/12 -- **only** the indexed-slot case; the token arm stays GREEN, which is the pair | | P-M6 | conv page widened to f32 | RED 1/12 -- **only** the memory-format assertion; the token arms stay GREEN, which is the demonstration that a token gate cannot see a too-wide dtype | | P-M7 | production call site deleted | RED 6/12 | | P-M8 | narrowed G-SAFE replaced by a fall-through | RED 1/12 | | P-M9 | decode/prefill classification inverted | RED 1/12 | ## What is NOT gated **The A3 end-to-end token gate against the released checkpoint has not run, and the cause is measured contention rather than a fault.** `dgx.casa` answered at 07:21 with 74 of 119 GB available, then stopped answering SSH for an hour; when it answered again its uptime was 11h28m -- the same boot, so no reboot -- at **loadavg 211.44 with 3 GB of 119 available**. A 20.1 GiB checkpoint cannot load into 3 GB. The spec's §8.1 makes `NEEDS_DECISION` the answer only when the gate cannot run for a reason OTHER than contention, so this is recorded as the other result the rules allow, pending a named external resource, written into `docs/BENCHMARKS.md` rather than left as silence. Owed with it, and listed under the spec's `## Owed`: `examples/nemotron_h_gen` and its `docs/USAGE.md` weights block. The ABI surface that example would exercise is the same surface the A3 gate drives, so shipping the client before the gate can run would ship a client for a path nobody has watched produce a token. Also still refusing by name, each naming its owner: the device NVFP4 `lm_head` (A2-Q2b), the FP8 mamba projections (A2-Q1, mudler#940 -- A2-P carries their STATE and changes no projection), MTP (W5) and GGUF (W7). ## Inherited red, subtracted with evidence `test_cpu_x86_llamacpp_floor` fails as `NO_QUIET_WINDOW after 30s (busy=113% load=60.73)` -- the harness REFUSING to measure under load, which is mudler#618. Verified rather than assumed: the diff touches none of that test's inputs, and the local box was at loadavg 60 from other agents' builds. Every other preflight gate is green. Closes nothing: mudler#810 stays open until the A3 gate is green and A2-B lands. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
## Row `DOCS-FEATURES-REASONING-COUNT` - documentation-only alignment for mudler#1103. ## Before starting - Issue/PR search and existing claim: filed mudler#1103; PR mudler#977 covers other landing-page counts but not this reasoning-parser total. - Pull request shape selected at row claim: one documentation-only commit from an isolated worktree. - Roadmap or matrix row, plus `scripts/ready-for-helper.py` result when applicable: not applicable; no lifecycle row moves. - Exact current-code and test/evidence anchors inspected: `reasoning_parser_names()` in `src/vllm/entrypoints/openai/reasoning_parsers/abstract.cpp` contains 12 names; `docs/FEATURES.md` reported 10. ## What changed Update the public feature table from 10 to 12 reasoning-content parser names. Closes mudler#1103. ## Evidence - [ ] `scripts/agent-preflight.sh` passes (not run; this task used the explicit user-provided CPU-only scope and fresh upstream worktree). - [x] `python3 scripts/check-readme-structure.py` - OK. - [x] `python3 tests/scripts/test_check_readme_structure.py` - 21 tests passed. - [x] `python3 scripts/check-supported-models.py` - exactly 40 architectures. - [x] `python3 scripts/check-agent-record.py` and `python3 tests/scripts/test_agent_record.py` - checker OK, 77 tests passed. - [x] `python3 scripts/check-model-checklist.py` and its test - checker OK, 10 tests passed. - [x] `python3 scripts/check-doc-checkpoint.py --commit HEAD` and `python3 tests/scripts/test_doc_checkpoint.py` - checker OK, 27 tests passed. - [x] `python3 scripts/check-commit-trailers.py --range upstream/main..HEAD --filled` - OK. - [x] `python3 scripts/check-role-discipline.py --base upstream/main --head HEAD --pending-pr-head 716a78d` - OK. - [x] Same-change doc obligations: only `docs/FEATURES.md` changes; no lifecycle or benchmark claim moves. ## Speed claims - [x] This PR makes NO speed claim. ## Honest gaps No GPU tests ran because the host has no usable GPU and this is a source-backed documentation correction. README drift is already covered by open PR mudler#977, so this PR does not duplicate it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex]
Collaborator
|
Thanks for this, and for splitting it from #1065 — I reviewed both together. What's genuinely good. You found a real seam gap. Heads-up first: no CI has run on either PR. Fork PRs here need a maintainer Three things, then I think we can land it.
One question I want your view on rather than dictating. Upstream's Process, and none of it is your fault. Our protocol wants a committed Your commits pass Reachability, for the record: this one is not reached at its own merge |
…, and the oracle cannot load a 27B on dgx (mudler#81) (mudler#1116) FOLLOWING_AGENTS_PROTOCOL Runs the owed DGX arms for MTP speculation depth (mudler#81), pays the padded control on real weights, and localizes a blocker that outranks this row. Records, a committed adjudication instrument, and four repaired defects. ## The padded control is PAID, on real weights, in one clean window This is the assertion the row was built around, and it is the one that could not be forged. Margin fixed at **0.10 absolute before the run**: | depth | real acceptance | padded control | margin | |---|---|---|---| | 1 | 0.731 / 0.683 / 0.750 | 0.000 | +0.58 to +0.75 | | 2 | 0.539 / 0.618 | 0.000 | +0.44 to +0.52 | | 3 | 0.507 | 0.000 | +0.41 | All six clear by **+0.51 to +0.75**. The control's **depth-0** rate matches the real arm (0.892-0.925 against 0.868-0.878), which is exactly what a control that touches only columns >= 1 must look like: it is not a dead run, it is a run whose later columns carry no information. Why this mattered enough to specify: a naive `accepted_by_depth()[1] > 0` assertion **passes on a padded drafter**, because a padded row is `[t0, t0, ...]` and acceptance at column 1 needs only the target's greedy continuation to repeat a token, which real text does routinely. Rates against a control, not counts. All seven arms ran to exit 0 in ONE window at loadavg 0.16 to 1.86, all three guards passing, clocks pinned. That **lifts the earlier VOID** on the `padded_k3`/`padded_k4` throughput figures, which existed only because those arms had run at loadavg 10.77 and 20.41 against real arms at 1.5 to 2.9. ## The divergence is deterministic, narrowed to three forwards, and NOT adjudicated `our-ON` is not token-identical to `our-OFF`. Reproduced exactly on independently generated streams: **1718 divergent positions, 18 adjudicable, 3 distinct probe points.** Only the first divergence per arm and prompt is adjudicable, since after it the arms carry different prefixes and any later comparison measures two different conditionings. | prompt | pos | OFF | ON | |---|---|---|---| | 0 prose | 12 | 79733 | 279, all six ON arms | | 1 prose | 1 | 25 | 7318 at k=2 and k=3, **198 at k=4** | | 2 code | 69 | 15336 | 1727, all six | | 3 code | - | - | matches everywhere | Prompt 1 position 1 carries the argument: the **same position resolves to three different tokens under three values of k**. A depth defect cannot produce that, because the accept walk is a prefix walk and `k` never enters the emitted value. A flat distribution decided by the last bits of the logits produces exactly that. Mechanism located in the tree rather than guessed: `include/vllm/v1/spec_decode/rejection_sampler.h:36-41` states accept-iff-equal guarantees every emitted token is one the non-speculative run would have emitted - a property conditional on something the comment never names, that the verify logits equal the decode logits. `src/vllm/v1/worker/gpu/cudagraph_dispatch.h:13-16` records that they are not the same forward: verify runs at query length `1+k` **eager**, plain decode at `query_len == 1`. **This is written down as a hypothesis and nothing more.** It is not reported as a defect, because adjudicating it needs oracle logprobs and the oracle would not load. `scripts/mtp-k-gt-1-neartie-gap.py` is committed for whoever gets a working oracle: it teacher-forces the pinned oracle on the shared prefix, decides each gap against `kNearTieMnats = 500`, asserts oracle identity and prompt-tokenization equality first, and aborts by name on an empty distribution rather than letting a broken probe read as a verdict about the code. ## The finding that outranks this row **No vLLM leg of any row can currently run on `dgx.casa`.** The pinned oracle cannot load a 27B there, and the box reboots trying. The first failure was an instrument failure wearing a code verdict: the reimaged host has **no C compiler at all** - no gcc, cc, clang, ninja or nvcc, no `/usr/include/stdio.h`, no python headers - so Triton's JIT died after the weights had loaded and vLLM reported `Engine core initialization failed`. Four `ORACLE_EXIT=1` legs would have read as "the oracle cannot run this configuration". `enforce_eager` was not reached for. That was repaired with a toolchain container and the repair proved out: the engine reached `torch.compile` for the first time across three passes. Then it consumed the host. **The obvious explanation was then tested and refuted.** Re-running the byte-identical instrument at a lower memory fraction: | `gpu_memory_utilization` | collapse | box | |---|---|---| | 0.75 | yes | thrashed 42 minutes, survived | | 0.30 | **yes** | **REBOOTED** (`boot_id` changed, `journalctl` gap 09:10:15Z to 09:13:55Z) | So the memory fraction is **not** the lever and a lower value is **not** a safety margin. What the A/B did buy: weight loading finished with 66 GiB free and compilation with 88 GiB free, so the collapse is the step **after** `torch.compile`, most likely the profiling forward or graph capture. That is a located hypothesis, recorded as one. The next attempt should vary `max_num_batched_tokens` and `cudagraph_capture_sizes` one at a time with a `MemAvailable` sampler, and should not assume the memory fraction is the lever, because this pass believed that and the A/B said otherwise. ## Four defects found and repaired 1. `.agents/environment.md` prescribed `CC=/usr/bin/gcc`, a path that has not existed since the host was reimaged. Corrected with the working container recipe. 2. The driver called the OFF oracle leg with an empty `k` and died on `int('')`. Fixed at the caller so the staged script stayed byte-identical to what runs. 3. The foreign-container guard excluded our own containers on the grounds that "our arms are UNNAMED". They are not - ours ran as `wizardly_allen`. Repaired with an explicit prefix. 4. `trap cleanup EXIT INT TERM` never exits, so a SIGTERM reset the clocks and then let the driver **continue to its next leg** on a box with no memory left. The third guard those arms run behind is worth keeping for its own reason: at an earlier acquire, `gpu_apps_at_acquire` was empty and `mem_avail_GiB` was 52 against a 45 floor - both original guards passed - while a `--device cpu` container held **53.54 GiB**. GB10 reboots rather than OOM-killing, and `gpu_memory_utilization` does not bound host RAM here. ## Evidence `scripts/agent-preflight.sh`: **79 ok, 0 FAIL, 0 SKIP, 0 `--`**, exit 0, gated against `origin/main 268da6b`, named in both range headings, both range blocks executed rather than skipped. The count is over all **four** markers: `--` is a fourth state that an `ok|FAIL|SKIP` grep silently drops. `test_cpu_x86_llamacpp_floor` red twice during the work and is proven inherited: a pristine `origin/main` worktree fails the same file with `NO_QUIET_WINDOW` while a neighbouring `ctest -j 4` drove load to 127. ## Owed - **The adjudication**, needing an oracle that loads. Three probe points, one command, and the instrument is committed. - **The vLLM leg**, so no token gate is claimed here. - **The c>1 A/B at matched k, and the 35B lane.** - **A loadable 27B oracle on `dgx.casa`**, which blocks every oracle-dependent row and not only this one. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…vider that is BYTE-IDENTICAL to the host (mudler#1115) `vt` had **no transposed 1-D convolution of any kind, on any device.** The two 1-D convolutions it carried are `vt::CausalConv1dFwd` (causal, stateful, SiLU-folded) and `vt::DepthwiseConv1d` (centre-padded, depthwise), and neither can express a scatter that GROWS the time axis. `vt::Conv2d` and `vt::DepthwiseConv1d` are moreover registered CPU-only. So the stage that is **88.5 % of MiniMax-Music3's acoustic-half profile** had no device op to route to at all, and hand-rolling a kernel outside the shared seam is what `AGENTS.md` forbids. This adds `vt::Conv1d` and `vt::ConvTranspose1d` with a **CPU provider and a CUDA provider**, and routes the shared 1-D BigVGAN core through them. ## What reaches the op Verified by call-site survey, not assumed. `vocoder1d` is the shared core, so routing it routes everything that decodes through it: | file | `Conv1d` sites | `ConvTranspose1d` | reaches | |---|---|---|---| | `minimax_music3_acoustic.cpp` | :154, :710, :720, :785, :794, :812 | :741 | yes | | `minimax_h3_audio_vae.cpp` | :111, :133, :181, :196, :232, :305, :671 | :146 | yes | | `ltx2_audio_vae.cpp` | :582, :643, :702, :794 | :654, :838 | yes | | `bigvgan.cpp` | :25 | :60 | yes | | `minimax_music3_ar.cpp` | :502 | — | yes | | `indextts2_pipeline.cpp` | :164 | — | yes | | `bigvgan_loader.cpp`, `minimax_music3_loader.cpp` | — | — | n/a: load-time weight-norm folding, no convolution | A whole-`src/` sweep found no other caller. One gap is **named rather than left to be discovered**: `ltx2_audio_vae.cpp:75` carries its own 2-D host convolution loop behind no op at all; out of scope here and filed with the rest of the unrouted surface as mudler#1114. ## The CPU path did not move, and it is proved The CPU kernels are the `vocoder1d` host loops as they stood at `8fa405bb7`, carried into `src/vt/cpu/cpu_conv1d_general.cpp` statement for statement — same f64 accumulator, same visit order, same bias seeding, same `value == 0.0` skip, same output-channel partition over the same threadpool, and the tensors are VIEWS over the caller's own `std::vector` rather than copies. `test_host_parallel` compares the shipped function against a **verbatim copy of the pre-change loop** at five thread counts, bitwise. This PR also adds the transposed op's missing catastrophic-cancellation case, which §12 had for `LinearNoBias` and `Conv1d` but not for `ConvTranspose1d`. ## The CUDA provider is BYTE-IDENTICAL — no tolerance is claimed The row was scoped expecting a tolerance to justify against a measured control. None is needed. Both providers are one f64 accumulator per output element. For `Conv1d` that is trivial — the host loop is already a gather. For `ConvTranspose1d` it is the whole design: the host loop is a SCATTER, so fix a destination cell `p` and ask which additions land in it and in what order. The answer is `ic` ascending, then input position `t` ascending, and for each `t` at most ONE tap `k` (the one with `t*stride + k*dilation == p`). A thread that owns `p` and sweeps in that order performs the **identical sequence of f64 additions into the identical accumulator**. Two details are load-bearing rather than cosmetic: - the `value == 0.0` skip is reproduced exactly, because dropping it changes the **sign** of a zero output cell — `(-0.0) + (+0.0) == +0.0` while `-0.0` alone stays `-0.0`; - the bias is added LAST for the transposed op and FIRST for the forward one, matching each host loop respectively. That leaves one way the arms could disagree: FMA contraction. **Both sides are pinned.** The host by the project-wide `-ffp-contract=off` (`CMakeLists.txt:40-56`, added for exactly this class of bug); the device kernel locally by `__dmul_rn` / `__dadd_rn`, because nvcc's flags are separate and its `-fmad` default is on. So the gate asserts `memcmp` equality. ## The f64 accumulator is a deliberate divergence from torch torch accumulates an f32 conv in f32. These do not, because f64 is what the host reference used and therefore what every committed golden for all four consumers was taken with. Named here rather than inherited silently, per `.agents/porting.md` "Mirror the memory format" — a WIDER accumulator is exactly the class of divergence a token gate cannot see. It costs **nothing in bytes moved**: activations and weights stay f32 in memory and only the register width differs. `vt::DepthwiseConv1d` accumulates in f32 and its own byte-exactness gate pins that width, so these are **siblings** and it is untouched — the same call that op made against `vt::CausalConv1dFwd`. ## The gate had to earn its teeth, twice **An f64 accumulator stored through an f32 cannot see a reduction-order change** on well-scaled data. That is measured, not supposed. So every equality claim is also exercised on engineered catastrophic cancellation (`+2^40` / `-2^40` taps through a shared weight row). And the cancellation case asserts its **own** teeth: reversing the input-channel sweep must change the answer, and the check reports how many cells move. A weaker mutation was tried first and **correctly read 0** — swapping which channel carries the positive tap leaves the partial sums at the same magnitude at the same step, so it is not an order change at all. Recorded in the test so it is not re-derived. ## Gates Assertion counts included because `assertions: 0` is a skip wearing a pass. The last four need `CHECKPOINT_ROOT=/mnt/nas_share/checkpoints` or they silently skip. | suite | cases | assertions | |---|---|---| | `test_ops_conv1d_general` (new) | 8 | 347 (x86) / **385 (Thor sm_110)** | | `test_host_parallel` | 8 | 877 | | `test_vocoder1d` | 10 | 58 | | `test_bigvgan` | 6 | 65 | | `test_minimax_h3` | 79 | 57,395 | | `test_ltx2_vae` | 42 | 3,120 | | `test_indextts2_family` | 7 | 22 | | `test_op_provider` | 12 | 412 | ### The stronger leg, which was not planned The consumer gates were run **twice on Thor, once on each arm**, and they are identical: | suite | `VLLM_CPP_VOCODER_DEVICE=cpu` | `=cuda` | |---|---|---| | `test_host_parallel` | 8 / 877 | **8 / 877** | | `test_vocoder1d` | 10 / 58 | **10 / 58** | | `test_bigvgan` | 6 / 65 | **6 / 65** | `test_host_parallel` is not an ordinary suite to pass on a device arm: its oracle is a **verbatim in-test copy of the pre-op host loop** and its comparison is bitwise. A green there with the device selected says the CUDA kernel is byte-identical to the pre-change scalar host loop **end to end through the consumers' own entry point**, at every shape it carries including the engineered cancellation cases — not merely at the op boundary. Two things it does not say, so the leg is not over-read: its thread-count sweep is redundant on the device arm (the host pool is unused there), and these are three suites, not the four consumers' full golden sets. `test_minimax_h3` (79 / 57,395) and `test_ltx2_vae` (42 / 3,120) were run on the CPU arm only, and re-gating them with the device selected is exactly the work the default flip waits on. Measurements taken on `b25a7ebf6`; `git diff` against HEAD is **empty** for both kernels, `ops.cpp`, `include/vt/ops.h` and both gate files. A same-SHA re-run is queued. The 347 → 385 difference **IS** the CUDA-vs-CPU `memcmp` arm, which is how the Thor run proves it executed rather than skipped (both `[SKIP]` lines are absent from its output). Measured on Jetson Thor `kairos-4db2`, aarch64, sm_110, driver 595.78, in `vllmcpp-thor:cuda13.0.1` (nvcc 13.0.88), built `-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=110 -DVLLM_CPP_TRITON=OFF`. Full local build exit 0; `scripts/agent-preflight.sh --no-require-role` reports **All gates green**. ## One checker changed, and it got stronger `tests/scripts/test_vocoder1d_single_home.py` went red with `Conv1d has 2 definitions`. It guards a FORK of the vocoder core — a failure no numeric test can see, because a fresh copy agrees on the day it is made — using a line-anchored TEXT match, which cannot see a namespace and so read `vt::Conv1d`'s definition in `src/vt/ops.cpp` as a second copy of `vocoder1d::Conv1d`. It is the opposite of a second copy: it is the op the core now delegates to. Excluding `src/vt/` was **priced**, not simply taken, because excluding a tree from a guard is how guards die. Two assertions were added beside it: 1. the core must still **call** `vt::Conv1d` / `vt::ConvTranspose1d` — without it the count reads a happy `1` while `vocoder1d.cpp` quietly re-grows its own loops and all six consumers leave the shared seam, with every numeric gate still green; 2. the walk must report **how many files it examined** — without it a scan narrowed to `vocoder1d.cpp` alone reports every count as 1 and passes while seeing none of the tree. Mutated, not argued (scratch copy, restored byte-for-byte): | | result | |---|---| | RED-BEFORE: unrepaired checker, this tree | `AssertionError: 2 != 1 : Conv1d has 2 definitions` — the exact CI failure | | GREEN-AFTER | 6 tests, OK | | M1: delegation removed | **FAILS** on the new assertion — while the count still read 1 | | M2: genuine fork added to `bigvgan.cpp` | **FAILS** — the original invariant survives | | M3: walk narrowed to one file | **FAILS** on the new count-of-files assertion | The first mutation attempt was itself broken — no `.git` in the scratch copy, so `git ls-files` failed and the suite reported ERRORS rather than a failure, an infra fault presenting as a code verdict. That is why a control run of the unmutated copy is part of the evidence. Recorded in the spec at §13.9. ## Staged, and said so What is **not** reached is the DEFAULT. The device arm ships opt-in behind `VLLM_CPP_VOCODER_DEVICE=cuda`; `cpu` remains the default, so every consumer above is byte-for-byte where it was. Flipping four shipped audio models onto a device arm needs its own re-gate against each one's goldens, and that is not a default the row that ADDED the arm is entitled to set. Per `.agents/reachability.md` "Landing a slice that is not reached yet", the three things it asks for: what is not reached is `ResolveConvDevice()`'s default resolution; the row that owns the wiring is `MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation`; the issue is mudler#672. Listed in the spec's owed table. Also owed and named rather than hidden: the device arm allocates, uploads, downloads and frees **per call**, with a queue per call. That is deliberately literal — `cuda` means cuda, with no size threshold quietly sending small shapes back to the host, because a threshold would make the consumer gates report on a state they were not given. Device-resident weights, one persistent queue, and a chain that stays on the device between stages are all left on the table. ## Speed: VOID — the timings were taken outside the fleet lease **Every timing here is void, and the reason is more useful than the numbers were.** The GPU fleet is scheduled by `rc`. These runs went in by `ssh` + `docker run` directly on the box, serialised by `flock ~/gpu.lock` — the *old* mutex — while the concurrent MiniMax-Music3 DiT session held the same box through `rc`. Two different mutexes, neither excluding the other: verbatim the failure `.agents/environment.md` already records for a `GPU_LOCK` naming the wrong path ("`flock` succeeds on it, so the run is unserialised and only looks like someone else misbehaving. That cost a whole Marlin series, mudler#777"). That is almost certainly the 3x swing below — a defect in how the samples were taken, not a fact about the kernel. **It was not simply re-run under a lease** because `rc run` executes inside the worker's container, and thor's worker has no toolchain at all (`no gcc / g++ / cmake / ninja / nvcc / make`, probed) and does not mount the box's `$HOME` where the build tree lives; its `/workspace` is the shared NAS. A valid re-measurement needs a worker image carrying the CUDA devel toolchain, or this build placed on `/workspace` by something that has one. Until then the speed axis has **no instrument**, and that is step zero of the open gap, not a caveat. `dgx:gpu0` is also UP and schedulable (GB10, unified memory) — the "dgx.casa is down" note this row was briefed with was stale — so a second, materially different device is available for the re-measurement. ### The void numbers, retained rather than deleted Two things, and they must not be collapsed. Measured on Thor sm_110, idle box (`uptime` 4.54 before / 4.57 after, 0 other users), same binary, `VLLM_CPP_VOCODER_DEVICE` the only variable, best-of-3, three interleaved reps, via the new `vocoder-conv-ab`: | stage | shape | CPU (14 cores) | CUDA | | |---|---|---|---|---| | up0 | 1536→768, L=96, stride 8, K=16 | 0.0596 s | 0.1538 s | 0.39x | | up1 | 768→384, L=96 | 0.0142 s | 0.0388 s | 0.37x | | up2 | 384→192, L=96 | 0.0022 s | 0.0056 s | 0.39x | | up3 | 192→96, L=96 | 0.0004 s | 0.0010 s | 0.40x | | **chain** | | **0.0765 s** | **0.2000 s** | **0.38x** | **And that A/B is not accepted**, because a sweep taken minutes later on the same box and binary put the CPU chain at frames=96 at **0.2280 s** against 0.0765 s — 3x, same arm, same workload — while CUDA read 0.2000 s in both: | frames | CPU | CUDA | |---|---|---| | 96 | 0.2280 s (0.0765 s in the run above) | 0.2000 s | | 384 | 0.7950 s | 0.7757 s | | 1536 | 2.5908 s | 3.0677 s | The device arm is stable to four digits across runs; the HOST arm is the untrustworthy instrument here. So no ratio is claimed. What survives is the weaker and defensible statement: **at no measured size did the device arm win**, and at the largest, most compute-dominated point it was 1.18x slower. **A hypothesis, labelled as one.** The per-stage ratios are flat (0.37–0.40x across a 150x span of work), which is the signature of a compute-RATE difference rather than per-call staging, since fixed overhead would punish the smallest stage most. The candidate is the f64 accumulator — consumer/Jetson Blackwell runs fp64 at a fraction of its fp32 rate — and f64 is not optional here: it is what makes the arms byte-identical and what four models' goldens were taken with. Nothing has read a counter: `nsys` in that image is 2024.2.3 and cannot trace CUDA on that box. **Open gap, not a ceiling.** Next steps, in order: **take the lease** (nothing above is admissible until the arms are measured under `rc`); get an instrument (newer `nsys`/`ncu` on Thor); remove the per-call staging (§13.6's owed list — the flat ratio argues it is *not* the dominant term, which is why it should be measured rather than assumed); an f32-accumulate device variant, which is the lever if the fp64 hypothesis holds and which is expensive in the right way because it is **not** byte-identical and cannot inherit this row's `memcmp`; and a GPU whose fp64 is not 1/64 (Thor is the only one here, dgx.casa is down). This does not touch the correctness result, which is what the PR turns on. The device arm shipping OFF by default was already right for numerics reasons; this says it would also have been right for speed. ## The `Conv2d` / `DepthwiseConv1d` device arms — assessed, and declined here The obvious follow-on was to extend the same machinery to the two existing CPU-only conv ops. **The survey says do not, and the reason is not kernel difficulty: a CUDA provider for them would be dead on arrival.** Of the seven models named as stuck behind them, exactly ONE (`parakeet_encoder.cpp:166,194`) calls either; the other six run their own host loops, and three of those are 3-D convolutions the 2-D op cannot express. `muse_glimmer_vision.cpp` has no convolution at all — it is already on `vt::MatmulBT`. No caller passes device tensors either, so a provider landed today would be reached by its own test and nothing else. Filed with the full evidence, including why the 27 gated dtype combinations and the f32 accumulator make the kernel bodies non-shared: **mudler#1114**. Issue: mudler#672 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…he guidance the naive merge would have dropped (mudler#921) (mudler#1125) `pipeline_kind=res2s_two_stage` serves upstream's `TI2VidTwoStagesHQPipeline` -- the high-quality arm. New TU `ltx2_samplers.{h,cpp}`, mirroring upstream's own `utils/` partition: a stepper advances a substep, a sampler decides how many there are. Ported at `fd4ded7f`: `Ltx2Phi` (`utils/res2s.py:4-22`), `Ltx2GetRes2sCoefficients` + `Ltx2PhiCache` (`:25-62`), `Ltx2Res2sNormalizeNoise` (`utils/samplers.py:160-170`), `Ltx2Res2sDenoisingLoop` (`:208-447`), and `Res2sTwoStageRecipe` (`ti2vid_two_stages_hq.py:59-340` + `utils/constants.py:95-115`). THE SAMPLER IS THE HQ VARIANT. Serving the HQ preset on the Euler loop renders a plausible clip that is quietly not HQ, at roughly half the model evaluations the preset was tuned for, and no shape or token gate can see it. The gate is therefore an exact DiT-evaluation count -- `2n+1` when the schedule ends at 0, else `2n` -- asserted with the eval-sigma sequence beside it, so two forwards at the same sigma also fails. End to end: 7 and 11 evaluations at 3 and 5 steps on `res2s_two_stage`, against 3 and 5 on `one_stage`. `phi` is a CANCELLATION CLIFF, not a series expansion. Upstream guards only `abs(z) < 1e-10` and otherwise evaluates the quotient directly, so its own `phi(2,-1e-10)` is 0.0 and `phi(2,-1e-8)` is 1.1102230246251563. A Taylor expansion near zero -- the numerically BETTER port -- returns 0.5 and diverges from what the model actually ran. Pinned with `==`. GUIDANCE, which the merge nearly dropped. `daeff67f2` (mudler#1092) landed the guided video denoiser into the same phase loop. A textual resolution keeping `Evaluate` as a bare `Ltx2DitForward` would have made the HQ preset the only unguided video arm in the tree -- upstream's stage 1 runs a `GuidedDenoiser` at cfg 3.0/7.0 and rescale 0.45 (`ti2vid_two_stages_hq.py:271-281`) -- and the evaluation count cannot see it, because a denoiser call is one evaluation guided or not. So `Evaluate` builds the `Ltx2X0Model` lambda and calls `Ltx2GuidedDenoise`, and a SECOND counter was added: `dit_forwards` counts actual `Ltx2DitForward` calls and is `3 * (2n+1)` on HQ stage 1 (cond + uncond + modality), asserted exactly at two step counts with `forwards != evaluations` as its own assertion. Stripping guidance from the res_2s arm alone is RED. `step_index` mirrors upstream literally: `step_idx` at the first evaluation (`samplers.py:301`), a literal 0 at the substep beside its one-element schedule (`:385`), `n_full_steps` at the terminal one (`:437`). Since `should_skip_step` is `step % (skip_step + 1) != 0`, the literal 0 makes the substep unskippable at any `skip_step` -- inert on the HQ preset's own `skip_step = 0`, live for a request override. A mutation survivor the review did not list: the substep's x0 conversion must use the latent THAT EVALUATION was handed, because the substep runs over `x_mid`. Reading the stream latent moves the whole substep prediction and nothing could see it, since the loop's arithmetic is gated with a fixture denoiser that performs no conversion. Now gated and RED. The engine's `VT_CHECK` was a tautology -- both operands came from the same `stats` object and `2n+1 > n` holds for every n -- and is now the trace delta against `stats.evaluations`. The argument is executable: the same under-counting defect beside the restored old check is GREEN. Both generator scripts are committed rather than described. `scripts/gen-ltx2-res2s-goldens.py` imports upstream's own `phi`, `get_res2s_coefficients`, `Res2sDiffusionStep`, `post_process_latent`, `_channelwise_normalize` and `res2s_audio_video_denoising_loop` at the pin and reproduces `ltx2_res2s_goldens.inc` byte for byte -- which is the evidence the goldens are upstream's and not this port's. Supersedes mudler#1101, whose branch carried a merge commit with a bare subject and no trailer block. `check-commit-trailers.py` walks merges, that commit was a first-parent ancestor of every candidate head, and repairing it would have needed a force-push. Same tree, one commit, block intact. Owed, not claimed: no render on real weights, and no oracle-run comparison -- everything is gated against upstream SOURCE at `fd4ded7f`. The HQ preset is host-only here, because its `modality_scale = 3.0` asks for the isolated-modality pass and `Ltx2DitForwardDevice` takes no perturbations; that is mudler#1092's owed device work, not newly incurred. Closes mudler#921. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Fixes mudler#837. ## Problem On a peer-MoE device hop, the single per-thread (TLS) hipBLAS handle is destroyed on the wrong device, corrupting the `GetBlas` accessor during multi-GPU Gemma-4 MoE serving. ## What changed - `GetBlas` **dual-slot** TLS keyed by device, plus a host-lifetime seam, so the handle is no longer destroyed across the device hop. The production `GetBlas` in `src/vt/rocm/rocm_matmul_hipblaslt.hip` is edited in place and routes through the new engine, so the change is reached rather than merely present. - Product-call seam and an exit-77 HIP probe; `StreamIsCapturing` is made load-bearing on the product path. - Spec and identity probe (0→1→0→1) land in the same PR, spec first. ## Verification Built and smoked on gfx1201, shipping ROCm 7.2.4 (clang `f58b06d`), as the combined mudler#837+mudler#838+mudler#839 stack on `d1b0ea3a`: gfx1201 build `BUILD_RC=0`, all 30 layers resident and bound, ready in ~20 s; batched-MoE prefill deadlock-free (T=2012 done, `PEER_ACT` active, no hang, no HIP error); Paris/63 quality PASS. Stated plainly, because it bounds what this PR alone demonstrates: that smoke covers the three-PR stack, not this branch in isolation. What *is* isolated here is the host-side test evidence — 12 cases with real RED mutants (swapped selector, destroy-on-hop, missing `SetStream`, capture-time `SetDevice`, forwarded device 0, forwarded null stream), plus a source-invariant case that reds if the production call site is deleted. ## Maintainer changes on top Two repairs were applied while landing, neither touching behavior: - The donor evidence moved from `.agents/evidence/` — a path `scripts/check-pr-size.py` cannot classify, so the gate refused the change outright — to `.agents/specs/rocm-gemma4-getblas-dualslot-donor.{md,log}`, which match `SPEC` and `SPEC_EVIDENCE` at `check-pr-size.py:188-189`. - `docs/FEATURES.md` is left as `main` has it. The branch had rewritten the Gemma4 row, dropping the `VT_GEMMA4_*`/`VT_ATTN_*` env pointer, the `test_gemma4_rocm_fp8_seams` seam name and the spec link. Restoring those and adding the mudler#837 sentence does not fit — `check-public-doc-tables` caps a cell at 220 characters and main's is already 219 — and `check-doc-checkpoint` does not ask for a FEATURES edit here, since `src/vt/rocm/` is not a feature surface and no `REGISTER_VLLM_MODEL` set changed (mudler#595, mudler#1086). The mudler#837 detail stays where this branch already put it in full: `docs/USAGE.md` and the spec. The branch was rebuilt by rebase rather than merge so it carries no untrailered merge commit; all six original commits are preserved with their authorship. ## Known-unrelated CI `windows-msvc-cpu` and `windows-msvc-vulkan` were red on every open PR from a break predating this branch (mudler#503). That queue has since been cleared — mudler#968 landed as mudler#983 and mudler#1068 as mudler#1069 — and only mudler#584's runtime crash remains. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
…nce (mudler#672) (mudler#1121) feat(MODEL-MUSIC-MUSIC3): the 2.4B fp32 DiT onto the device, staged once (mudler#672) FOLLOWING_AGENTS_PROTOCOL Spec §11.4 recorded three device rows as owed. §12 closed the arm-independent one. This closes the **DiT**, which is the one that mattered: at a real duration it is not one stage among six, it is the request. A 45 s clip at the shipped defaults runs `DitForward` **660 times** (30 steps x 2 CFG branches x 11 windows) for roughly **634 TFLOP against ~29 TFLOP for the entire autoregressive half** — about 20x everything else in the model put together. That is why `d9441ef3`'s device arm reached only 0.946x: it moved the 8.6B language model and left the stage twenty times larger on the host. ## What moved, and onto what **No new kernel.** Every op already existed with a CUDA provider; this adds a forward that composes them. | reference helper | shared op | |---|---| | `Linear` | `vt::MatmulBT` (+ `vt::Add` for the rank-1 bias) | | `LayerNorm` | `vt::LayerNorm` | | `ApplyPartialRotary` | `vt::RopeFromCache` | | `Attention` (NON-causal) | `vt::AttentionCross`, bias `nullptr` | | `value * silu(gate)` | `vt::SiluAndMul`, over a stage-time half swap | | `PointwiseConv` (both 1x1) | `vt::MatmulBT` on the transposed activation | **The 1x1 convolutions are GEMMs, and that is what unblocked the row.** `vt` has no CUDA 1-D convolution provider — the finding §11.4 recorded against the vocoder applies to the DiT's `preprocess_conv` and `postprocess_conv` too. But `conv(x)^T[t][co] = SUM_ci x^T[t][ci] * W[co][ci] = MatmulBT(x^T, W)`, so the forward works FRAME-MAJOR throughout and transposes once on the host at each end, where the tensors are `[128, length]`. Nothing is hand-rolled outside the seam. **fp32 stays fp32.** Spec §2.1: the acoustic half is float32 by upstream's choice. Every staged weight and device activation is `kF32`. ## Correctness — same goldens, same bounds, nothing widened **The CPU arm is bit-identical structurally, not by measurement.** `minimax_music3_acoustic.cpp`, `minimax_music3_ar.cpp`, `minimax_music3_llm.cpp` and `vocoder1d.cpp` have a **ZERO DIFF**. `--speech-device 0` takes the same `DitForward`, source byte for source byte, so there is no number to move. The device forward is an additional entry point in a new file, the shape `minimax_h3_device.cpp` already uses. Reduced dimensions, vs upstream's own goldens, at the EXISTING bound (`kRelTol` 1e-5 / `kAbsFloor` 1e-6), each case reporting BOTH arms' distance to the golden: | arm | worst \|arm - upstream\| | |---|---| | host `DitForward` (the accepted control) | 1.565e-07 | | device forward, CPU backend | 1.192e-07 | | device forward, CUDA sm_110 | 2.980e-07 | FULL SCALE, the real 2.4B checkpoint vs the committed oracle capture on `thor:gpu0`, 11 008 values/step, bounds unchanged (1e-4 / 5e-5 / 5e-6): | arm | step | bit-identical | mean\|d\| | max\|d\| | outside | |---|---|---|---|---|---| | Thor CPU | first | 423 (3.843%) | 1.71434e-06 | 2.38419e-05 | 0 | | Thor CPU | last | 235 (2.135%) | 2.22396e-06 | 2.83718e-05 | 0 | | Thor CUDA | first | 473 (4.297%) | 1.64344e-06 | 2.47955e-05 | 0 | | Thor CUDA | last | 222 (2.017%) | 2.44677e-06 | 2.59876e-05 | 0 | | CONTROL torch-vs-torch | first | 15.416% | 7.526e-07 | 7.153e-06 | — | | CONTROL torch-vs-torch | last | 5.596% | 1.424e-06 | 1.335e-05 | — | The device arm sits ON TOP of the host arm — better on two of four figures, marginally worse on the other two — and both sit at the same multiple of the recorded torch-vs-torch control. **The Thor CPU arm reproduces the x86-64 numbers this spec already recorded VALUE FOR VALUE**, so the CPU path is unchanged across two architectures. **Two mutations, because a bound nothing violates has not been shown to discriminate.** Pre-swapping the `ff_in` halves makes the stage-time swap undo the test's, so the forward computes `silu(value) * gate`: **20 of 20 values outside the bound, worst |diff| 1.538e-03**, four orders above the noise. And the conditional/unconditional branches must differ: 20 of 20 do, on both backends. ## Speed — `thor:gpu0` (NVIDIA Thor, sm_110), per DiT forward Named because a number without its device is meaningless across this fleet; nothing here is compared to a `dgx:gpu0` or `orin:gpu0` number. `VLLM_CPP_MUSIC3_DIT_REPEAT=R` times ONLY the forward loop — the 9.7 GB load, the golden reads and the staging are outside it. | arm | repeats | fwd | loop | per forward | staging | load | |---|---|---|---|---|---|---| | CPU | 1 | 4 | 819.818584 s | **204.954646 s** | no-op | 3.42 | | CPU | 1 | 4 | 819.992 s | **204.998 s** | no-op | 10.37 | | CUDA | 1 | 4 | 0.749077 s | **0.187269 s** | 0.603561 s | 4.79 | | CUDA | 3 | 12 | 2.110301 s | **0.175858 s** | 0.660600 s | 5.32 | | CUDA | 1 | 4 | 0.743367 s | **0.185842 s** | 0.609463 s | 5.1 | | CUDA | 1 | 4 | 0.743881 s | **0.185970 s** | 0.612787 s | 4.44 | Fit: **slope 0.170607 s/forward, intercept 0.063012 s**. The device R=1 point was taken THREE times across two sessions, bracketing R=3, at 0.749077 / 0.743367 / 0.743881 s — 0.77 % spread. **204.955 s host vs 0.1706-0.1873 s device: 1102x on the matched pair, 1201x on the slope.** **The contention asymmetry was measured away, not argued away.** The first CPU point sat at load 10.37 against the device arm's 4.4-5.3, which would have inflated the ratio if it mattered. Re-taken on an idle box (load 3.42) with the fixed instrument it reads **204.954646 s against 204.998 s, 0.021 %**: the host DiT forward is single-threaded on 14 cores, so a load of 10 still leaves it a core. Both points are reported. **The weights are staged ONCE, as a measurement.** One staging costs 0.60-0.66 s; the entire FOUR-forward loop costs 0.745 s and the TWELVE- forward loop 2.110 s, where twelve stagings would be 7.35 s alone. The loop's intercept is a tenth of one staging. A per-forward upload is arithmetically excluded. **Whole-process, which is lower and is the honest ceiling on what a user sees today:** 1054-1071 s vs 238-298 s (**3.5-4.5x**) for the same binary including the identical NAS load, the spread being NAS cache state rather than compute; the full two-arm correctness series, 49 min 17 s vs 15 min 49 s (**3.12x**). The distance between 1100x on the DiT and 4x on the process IS the owed list. **No e2e song pair**, and that is a limit not an omission: at 30 steps the host DiT alone extrapolates to ~37.6 h, and at a setting short enough to run, the pair would be measuring the vocoder. **No parity claim** — SGLang-Omni is `gateable = no` and every reference axis stays `PENDING`. ## One instrument defect, found inside this change The first timing line printed `DIT_TIMING arm=1` on the CPU run: a `const char*` in a doctest `MESSAGE` chain takes the **bool** conversion. That is mudler#672's OWN §11.5 defect reappearing in a new line — the lesson was written down and a fresh `<<` chain reintroduced it. Both lines are now assembled as one `std::string`. EVERY number was then re-taken with the fixed instrument, and the CPU arm's pre-fix point is kept BESIDE its post-fix twin rather than replaced by it, because the pair is what proves the label defect never touched the values: 819.992 s against 819.818584 s. ## What is still OWED, and a correction to §11.4 §11.4 said the depth decoder was blocked on "nothing but the work". **That is wrong, and this row found it out.** The depth decoder and the condition mix run at `ArCompute::kBFloat16`, which rounds the RESULT of every op to bf16 (`minimax_music3_ar.cpp:36-40`). Routing them through an f32 `vt::MatmulBT` would silently drop that rounding — a change to the numbers wearing a refactor's clothes. Mirroring them needs bf16 STORAGE, a dtype decision with its own evidence. They are also ~15 TFLOP against the DiT's 634. The vocoder row is unchanged: `vt` still has no `ConvTranspose1d`, and that op has three consumers. `minimax_music3_device` is added to the merged-GEMM allowlist with a reason rather than folded: `ff.net.0.proj` is ALREADY one merged `nn.Linear` and ALREADY one `MatmulBT`, so a merged-GEMM seam has nothing left to merge, and `layers::UnquantizedMlpGateUpMethod` is bf16-only, bias-free and `OwnedTensor`-resident — three shared-layer changes, not a model fold. ## Lease discipline, recorded because it was imperfect The CUDA build, the correctness series and the first timing runs were driven over `ssh` under `flock $HOME/gpu.lock` — this row's brief, since superseded by `rc`. During that window the fleet reported `thor:gpu0` FREE while it was in use. The first device series ran under a real `rc hold` (`a91d21dc`, 10:32Z-10:54Z) that carried no `--reason` and a 75 m TTL for ~22 m of work; both are errors. The matched pair above ran under `8aa5cd6d` with a reason string and a 40 m TTL, released early at 23 m on completion. `rc run` cannot serve this particular job because the build lives on the device host's filesystem and `rc run`'s container sees only `/workspace`; a shell is the sanctioned case for a `hold`. Thor's uptime is unbroken across every arm (`system boot Jun 5 15:36`; `up 2 days, 14:37` at the first arm through `16:56` at the last), so no pair straddles a restart, and no number here is compared to one from `dgx:gpu0` (GB10) or `orin:gpu0` (Orin). ## Gates Local x86-64, non-zero assertion counts: `test_minimax_music3_acoustic` 32/283 (was 27/265), `test_minimax_music3_speech` 9/223, `test_minimax_music3_ar` 26/352, `test_minimax_music3_loader` 21/1393, `test_minimax_music3_quant` 29/125, `test_speech_engine` 11/38, `test_capi` 65/653, `test_speech_api` 6/67, `test_openai_api_server` 62/733. With `CHECKPOINT_ROOT` set: `test_minimax_music3_acoustic_real` 6/76, `test_minimax_music3_ar_real` 4/894, `test_minimax_music3_quant_real` 6/319, `test_minimax_music3_llm_real` 4/220 — every one matching its recorded count. Thor CUDA build: `test_minimax_music3_acoustic` 32/291 (8 more than x86 — the CUDA device case RUNS instead of skipping), `test_minimax_music3_acoustic_real` 6/985 (`device 0`) and 6/988 (`device 1`), `test_minimax_music3_speech` 9/223, `test_speech_engine` 11/37 (one fewer BY DESIGN on a CUDA build). Issue: mudler#672 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…eld a boolean could not express (mudler#1117) (mudler#1120) `A2VidPipelineTwoStage` (`a2vid_two_stage.py:53` @ `fd4ded7f`) had no recipe row, so `pipeline_kind = a2vid_two_stage` got the generic table refusal naming the pair rather than the missing machinery. [mudler#922](mudler#922) is CLOSED and closed the audio **conditioning**, not the recipe: a supplied take rode `distilled_two_stage`, which [`ltx25-a2v-audio-input.md`](.agents/specs/ltx25-a2v-audio-input.md):438-446 already records as a different trajectory. A closed issue is not evidence. Issue [mudler#1117](mudler#1117). Spec [`.agents/specs/ltx25-a2vid-recipe.md`](.agents/specs/ltx25-a2vid-recipe.md), committed at `d38ab8c86`, before any code. ## What the recipe is, field by field Four fields differ from the recipe a take rides today, and **each of them renders** — a finished clip at the right size, frame count and sample rate. | Field | a2vid | `distilled_two_stage` | Upstream | |---|---|---|---| | stage 1 sigmas | DERIVED from the step count | the frozen 9-value distilled list | `self._scheduler.execute(steps=num_inference_steps)`, `:225-227` | | stage 1 guidance | the params table's video row, caller-overridable | fixed, override refused | `MultiModalGuider(params=video_guider_params, ...)` `:230-240`, fed from `utils/args.py:947-1006` | | stage 1 stepper | plain Euler | `kEulerAncestral` on 2.5 | `:229-258` passes no `stepper`, so `EulerDiffusionStep()` applies, `utils/blocks.py:526-527` | | stage 1 AUDIO guider | the DEFAULT positive-only params | n/a | `MultiModalGuiderParams()` at `:237-239`, `ltx-core components/guiders.py:200-210` | The audio guider is the field a reader is most likely to "fix" by symmetry with `OneStagePhase`, which takes the table's row and is right to (`ti2vid_one_stage.py:215-218`). A2Vid does not: the stream it would guide is frozen, so the table's `cfg_scale = 7.0` would buy an unconditional forward and a negative text encode for a delta multiplied into a latent the sampler cannot move. Stage 2 is upstream's `STAGE_2_DISTILLED_SIGMAS` (`:164`), the spatial upsample (`:261`), `noise_scale = stage_2_sigmas[0]` (`:288`) and `SimpleDenoiser` (`:278`). Four rows — 2, 2.3, 2.4, 2.5 — mirroring the `t2a_one_stage` rows one for one and for the same reason: the pipeline takes whatever `resolve_cli_params()` read off the checkpoint (`:311`, against `t2a_one_stage.py:178-179`). ## Two things upstream makes REQUIRED, refused by name - **`--audio-path`** is `required=True` (`:312-317`). Checked at generate time, because `pipeline_kind` is a load knob and `audio_path` is a per-generation extra, so the question is only decidable once a request exists. Without the take the soundtrack is GENERATED and the clip looks finished. - **`--distilled-lora`** is `required=True` (`utils/args.py:1140-1153`). Checked at load. Without the adapter, stage 2's three-sigma refinement runs on weights that were never distilled for it. Both are keyed on a recipe flag rather than a `pipeline_kind` string compare, for the reason `audio_only` already gives in the header. The second flag has a second user waiting: `ti2vid_two_stages` ([mudler#1093](mudler#1093)) and `keyframe_interpolation` ([mudler#1096](mudler#1096)) select the same parser. ## The phase field a boolean could not express `Ltx2PhaseDenoiser { kGuided, kSimple }` is new, and it exists because `allow_guidance_override` cannot describe a2vid's stage 2. That boolean answers "does this pipeline's CLI carry the guider flags at all". `distilled.py` selects `default_2_stage_distilled_arg_parser` (`utils/args.py:1188`), which never adds them, so an override there names a knob the pipeline has no surface for and the engine refuses it — correctly, and that refusal is landed and gated. `a2vid_two_stage.py:311` selects `default_2_stage_arg_parser`, which DOES carry them, and they reach stage 1's guider alone (`:233-236`) because stage 2 constructs `SimpleDenoiser` and takes no params. Neither value of the boolean says that. `false` rejects a request upstream accepts. `true` applies the override to stage 2's positive-only params and switches on a guidance pass upstream does not run — invisibly, since an extra forward changes no output shape, frame count or sample rate. So a2vid's stage 2 is `allow_guidance_override = true` **and** `kSimple`, and the skip is tested AFTER the refusal: every recipe that refuses today is `kSimple` too, so the other order would turn three landed refusals into silent ignores. ## Reachability Entry point: `LoadVideoEngine` with `pipeline_kind = a2vid_two_stage` — a documented value of a documented load extra — then `Generate` with the `audio_path` extra. The chain is `include/vllm.h` -> `src/capi/vllm_c.cpp` -> `Ltx2VideoEngine::Generate` -> the a2vid dispatch row -> the phase loop -> `Ltx2GuidedDenoise`. `ltx2-gen --pipeline-kind a2vid_two_stage --audio-path` is the same two calls through the ABI. No test constructs a recipe, a guider, a phase or a modality by hand. **M1 is the reachability mutation**: deleting the dispatch row REDs the whole case at the load. **What does NOT reach it, stated rather than left to be found.** `/v1/videos` cannot drive this. `VideoGenParamsFromRequest` (`video_engine.cpp:349-384`) never writes `gen.extras`, so no per-generation extra reaches any engine over HTTP ([mudler#928](mudler#928)). `pipeline_kind` IS a load extra and reaches a server through `--video-extra`, and every request to such a server is then refused for the missing take. Reachable from `include/vllm.h` and from `ltx2-gen`; not over HTTP. ## RED before green Both binaries, with the dispatch row absent and everything else in place: ``` test_ltx2_pipeline 44 cases | 42 passed | 2 failed; 2498 assertions | 1 failed; exit 1 test_ltx2_video 74 cases | 72 passed | 2 failed; 2191 assertions | 4 failed; exit 1 ERROR: test case THREW exception: Unsupported LTX pipeline kind/version: 'a2vid_two_stage'/'2.5'. Recipes are resolved from an EXACT (kind, version) table ... and never defaulted: a plausible-but-wrong sigma schedule or guidance scale renders a video rather than failing. ``` Green on the same tree with the row present: `test_ltx2_pipeline` 44/44, 2598 assertions, exit 0; `test_ltx2_video` 74/74, 2275 assertions, exit 0. ## The take is CONSUMED, not carried A recipe assertion proves `frozen = True` and `noise_scale = 0.0` are SET. These say the DiT saw the consequence, read off the LAST phase — so stage 2's own `noise_scale` of 0.909375, which the loop applies to both streams, is inside what they measure: - `audio_frozen`, derived from the denoise mask the loop uses and read AFTER the noiser; - `audio_sigma_max == 0.0`, the scalar `Modality.sigma` half of upstream's `frozen` (`utils/types.py:104-106`), which the mask cannot reach; - the latent's digest is **bit-identical across seeds** — it is the encoded file and not a sample — and **moves with the window**, so the first control cannot be passing on a constant. ## The guidance arms, in x0 space, on all four Stage 1's guider is cfg 3.0 / stg 1.0 / rescale 0.7 / modality 3.0, so all four passes run and the rescale branch — the one term that is not invariant between the two spaces — is live. `x0 == latent - sigma*velocity` is asserted per arm. M11's RED, verbatim: ``` arm = cond max|x0 - (latent - sigma*v)| = 1.64497 max|x0 - velocity| = 0 arm = uncond max|x0 - (latent - sigma*v)| = 1.64082 max|x0 - velocity| = 0 arm = perturbed max|x0 - (latent - sigma*v)| = 1.64939 max|x0 - velocity| = 0 2 cases | 1 passed | 1 failed; 94 assertions | 8 failed; exit 1 ``` The stage-2 skip has no trace field to read, so it is measured on artifact bytes: two renders whose only difference is `video_stg_scale = 1.0`, which is ALREADY stage 1's own value (`utils/constants.py:52`) and is 0.0 on stage 2. Equal bytes mean the override stopped at stage 1. Both recipe values are `REQUIRE`d first, so a table change turns the comparison into a failure rather than a tautology. ## Mutations Focused gate `./build/tests/test_ltx2_video --test-case='ltx2 a2vid:*'` — the filter is admissible only because the harness asserts it matched exactly 2 cases and a non-zero assertion count; neither case name contains a comma. Recipe-shape mutations also run the whole `test_ltx2_pipeline` binary. | # | Mutation | `git diff --stat` | BUILT | `: error:` | exit | counts | verdict | |---|---|---|---|---|---|---|---| | M1 | the a2vid dispatch row deleted (reachability) | `ltx2_pipeline.cpp \| 2 +-` | YES | 0 | 1 | 2c/9a | DETECTED | | M2 | stage 1 `spatial_downscale` 2 -> 1 | `ltx2_pipeline.cpp \| 2 +-` | YES | 0 | 1 | 2c/19a | DETECTED | | M3 | stage 1 given the frozen distilled sigmas | `ltx2_pipeline.cpp \| 2 +-` | YES | 0 | 1 | 2c/94a | DETECTED | | M4 | stage 1 `allow_guidance_override` -> false | `ltx2_pipeline.cpp \| 2 +-` | YES | 0 | 1 | 2c/19a | DETECTED | | M5 | stage 1 audio guider taken from the params table | `ltx2_pipeline.cpp \| 2 +-` | YES | 0 | 1 | 2c/94a | DETECTED | | M6 | stage 1 stepper -> `kEulerAncestral` | `ltx2_pipeline.cpp \| 2 +-` | YES | 0 | 0 video / **1 pipeline** | 2c/94a; 44c/2598a | DETECTED by the RECIPE case only | | M7 | the required-take check never fires | `ltx2_video.cpp \| 2 +-` | YES | 0 | 1 | 2c/92a | DETECTED | | M8 | the required-adapter check never fires | `ltx2_video.cpp \| 2 +-` | YES | 0 | 1 | 2c/91a | DETECTED | | M9 | the frozen stream's denoise mask left at 1 | `ltx2_video.cpp \| 2 +-` | YES | 0 | 1 | 2c/94a | DETECTED | | M10 | the frozen scalar sigma left at the schedule's | `ltx2_video.cpp \| 2 +-` | YES | 0 | 1 | 2c/94a | DETECTED | | M11 | every guidance arm left in VELOCITY space | `ltx2_video.cpp \| 3 +--` | YES | 0 | 1 | 2c/94a, 8 failed | DETECTED | | M12 | the `kSimple` skip deleted, so the override reaches stage 2 | `ltx2_video.cpp \| 2 +-` | YES | 0 | 1 | 2c/94a | DETECTED | **M6 is stated as it measured, not as it would read better.** The stepper is a recipe field and only the recipe case sees it; the end-to-end case has no baseline to compare a trajectory against, and this recipe's `noise_seed_offset` is 0 so the ancestral arm changes no digest the trace carries. **One harness defect, found and fixed in flow.** The stage-2 artifact comparison was `CHECK(a == b)` over PPM pixels and a WAV. A failing one dumps raw bytes into the report, and that killed the mutation harness with a `UnicodeDecodeError` between applying M12 and restoring it — the exact shape mudler#922's spec records. The `finally` restored the tree, the comparison is now a differing-byte COUNT (`99c4cf811`), and the harness decodes with `errors="replace"`. ## Gate ``` cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF cmake --build build -j6 && ctest --test-dir build -j4 --output-on-failure ``` On the merged tree at `85e562d1d`, `origin/main = 589abad`, CLEAN rebuild (`rm -rf build` first, because `589abad12` adds headers and an incremental build masks `-Werror`): | | | |---|---| | `CONFIGURE_EXIT` | 0 | | `BUILD_EXIT` | 0 | | `: error:` in the build log | 0 | | `ctest -N` | Total Tests: 502 | | `CTEST_EXIT` | 0 | | pass/fail | `100% tests passed, 0 tests failed out of 502` (2 skipped: `test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`) | | `No space left` / `BFD` in the build log | 0 and 0 over 1475 lines, against a positive control file where the same two greps return 1 and 1 | | load average | 99.37 at configure, 41.20 entering ctest, 83.82 after | | free disk | 30G of 447G before, 20G after; the build tree is removed on completion | 502, not 501: `589abad12` adds `test_ops_conv1d_general`. **Nothing known-red fired** — `test_async_llm` (mudler#294), `test_engine_core_proc` (mudler#1052), `test_serve_low_tools` (mudler#428) and `test_cpu_x86_llamacpp_floor` (mudler#618) are all load-dependent and all passed; `windows-msvc-*` (mudler#584) has no `main` baseline and does not run here. Both ltx2 binaries were also run WHOLE, with no `--test-case` filter, and their case and assertion counts asserted non-zero rather than read as "no failures": `test_ltx2_video` 74 cases / 2276 assertions / `Status: SUCCESS!` / exit 0, and `test_ltx2_pipeline` 44 / 2598 / `SUCCESS!` / exit 0. `Status:` is grepped beside the counts because a thrown doctest case prints `0 failed` next to `FAILURE!`, and the exit code is taken directly rather than after a pipe. ## What is NOT claimed **No render on real weights, and no oracle-run comparison.** The GPU is out of bounds for this row and `dgx.casa` is network-alive with an sshd that will not handshake. `vllm-omni` is UNPINNED ([mudler#633](mudler#633)) and carries no LTX-2.5 recipe; no LTX-2.5 checkpoint here has a recorded sha256 ([mudler#1048](mudler#1048)). Every value is read off upstream SOURCE at `fd4ded7f` and every number is measured on this tree's own reduced fixture. **The adapter placement diverges and cannot be repaired from the request.** `stage_2_loras = (*loras, *distilled_lora)` (`:114`) puts the distilled adapter on stage 2 ALONE, against `loras=tuple(loras)` for stage 1 (`:107`). This engine fuses at load into ONE weight set — `ltx2_video.cpp:816-820` is the only `dit_options.loras.push_back` in the tree — so stage 1 sees it too. That changes the trajectory, so it changes the PIXELS: the frames this renders are not the ones upstream renders for the same checkpoint, take and seed. What it leaves untouched is the frame count, the shapes, the sample rate and the errors — nothing in the SHAPE of the result says anything is wrong. It is therefore [mudler#1118](mudler#1118) and `## Owed` rather than a comment. It bounds mudler#1093 and mudler#921, which need the same seam. **It is not undetectable, and an earlier draft of this paragraph said it was.** "Changes nothing a caller can read" is false — a caller reads pixels — and it is false in the direction that hides work, because it implies no instrument could ever see this. The instrument that WOULD see it is the real-weights comparison against upstream's own render that the section above records this row as not having: same checkpoint, same take, same seed, upstream's stage 1 on the base weights against ours on base + distilled. That comparison is owed, not impossible, and mudler#1118 owns it. `docs/USAGE.md` tells a reader of the a2vid command the same thing in the same change. **Two anchors in the dispatch that started this row were wrong** and are corrected rather than propagated. The a2v guidance default is **3.0**, at `utils/args.py:986-995` through `utils/constants.py:54`, not 0.0 at `a2vid_two_stage.py:318-323` — which is `--audio-start-time`. And the `AudioConditioner` is built at `:96-102` and called at `:200`, not `:53`/`:143`. **Six anchors of this row's own, corrected after review.** No gate protects an anchor here ([mudler#632](mudler#632)), so each was re-derived from the sentence making the claim rather than by reading the cited span: | Was | Is | How it was checked | |---|---|---| | `--a2v-guidance-scale` at `utils/args.py:987-996` | `:986-995` | `parser.add_argument(` opens at `:986`, the name is at `:987`, the default at `:989`, the closer at `:995`. The old span started at the name and ran one line past | | `default_2_stage_distilled_arg_parser` at `utils/args.py:1187` | `:1188` | `:1187` is blank | | the six video-guider flags at `utils/args.py:947-1006` | `:947-1006` | `--video-cfg-guidance-scale` at `:948` through `--video-skip-step` at `:997-1006`; the audio group starts at `:1007`. The old span stopped at the last flag's opening line | | "stage 1's **40**-step schedule" | 30 on this row's 2.5 | `LTX_2_3_PARAMS` sets `num_inference_steps=30` (`utils/constants.py:85`), `LTX_2_4_PARAMS` inherits it (`:124`), and 2.5 resolves onto that row. 40 is 2.0's own default (`:47`) | | "the params row names block **29**" | 28 | `LTX_2_3_PARAMS` overrides 2.0's `[29]` to `[28]` (`:86`), inherited the same way | | "all four the params table **distinguishes**" | four keys of OUR table | `_PARAMS_SINCE_VERSION` (`:130-133`) has two rows, `(2,4)` and `(2,3)`, falling through to `LTX_2_PARAMS` at `:179`. `Ltx2DetectPipelineParams` mirrors that and already said so | Neither number was load-bearing — the code reads `params.num_inference_steps` and the tests assert relatively — which is exactly why nothing caught them. The §2 table also listed `LightricksNegativePrompt()` flat where the dispatch uses `kOmniNegativePrompt` on `2` and `2.3`; that split is which reference owns the row, and `t2a_one_stage` splits identically at the same four versions. `include/vllm/multimodal/ltx2_video.h:473` carries the same `987-996` span, but it is on `main` and predates this row, so it is reported rather than rewritten here. **This row's own insertions staled anchors elsewhere, and that is reported rather than swept.** Two are repaired here because this row owns them, and both now name the construct instead of a line range: the spec pointed at `ltx2_pipeline.cpp:1464-1471` for the `t2a_one_stage` rows, which the F4 comment above them moved in this very commit, and `docs/USAGE.md` pointed at `ltx2_pipeline.cpp:1306-1313` for the DFR refusal, which this row's 145 inserted lines moved to `:1434-1441`. FIVE more `docs/USAGE.md` coordinates moved the same way, in sections this row does not own. Each was re-derived by matching `main`'s content at the cited line against this tree: | Cited at | Was | Is now | |---|---|---| | `USAGE.md:918` | `ltx2_video.cpp:2900` | `:2968` | | `USAGE.md:1170` | `ltx2_video.cpp:1007-1012` | `:1040-1045` | | `USAGE.md:1174` | `ltx2_video.cpp:1955-1990` | `:1998-2033` | | `USAGE.md:1176` | `ltx2_video.cpp:1991-2004` | `:2034-2047` | | `USAGE.md:2672` | `ltx2_pipeline.h:768-803` | `:848-883` | They are NOT rewritten here. That page states its own convention — "read an unpinned coordinate as unverified" — and its three `ltx2_video.cpp` coordinates that sit beside a `@ b5756ea` pin are implicitly pinned by it and are correctly unchanged. Its last re-derivation was a scoped unit of work with its own review, not a rider on someone else's row, and this one should be too. Two things checked and found NOT to be this row's doing: `USAGE.md:2644` cites `ltx2_video.cpp:377-383` for `kKnownLoadExtras`, which is at `:378-385` on `main` and on this branch alike, so that one is pre-existing (the same passage already names [mudler#1097](mudler#1097) for stale text beside that array). **Merge hazard.** The `READER ANCHORS` list in `ltx2_video.cpp` is re-derived in this change (`798 808 809 871 967 983 1018 1109 1134 1239 1280 1322 1324`). A clean `git merge` that inserts a line above it will not warn, and `test_ltx2_video` will. It was re-derived once more after merging `589abad12`, with the same walk the test uses, and is unchanged — that commit adds `vt::Conv1d` and touches no LTX-2.5 file. The deriving instrument was armed before it was believed: one inserted line above the anchors moves all thirteen and it reports `MISMATCH`, exit 1, where the real tree reports `MATCH`, exit 0. A checker that reads its own expectation out of the file it checks is a tautology unless you make it fail ([mudler#911](mudler#911)). `.agents/issue-index.md` was taken deliberately rather than by the union driver: it is untouched by `589abad12`, and it is verified against that revision anyway — byte-identical prefix, no base row changed or dropped, every id unique, this branch appending exactly `mudler#1117` and `mudler#1118`. That check was armed too: editing a base row trips the prefix and row halves, and appending a duplicate id trips the third. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…ondition is the device (mudler#1129, mudler#1130) (mudler#1141) FOLLOWING_AGENTS_PROTOCOL Makes a GPU lease the required path in the root policy, keyed on the device rather than on the shell. Issues [mudler#1129](mudler#1129) and [mudler#1130](mudler#1130). Records and policy only. Seven files, all Markdown. No checker touched. ## The defect was an absence, not a stale instruction `AGENTS.md` matched `flock`, `gpu.lock`, `GPU_LOCK`, `mutex` and `ssh` **zero** times. It named no GPU access method at all. Since the root file states that it holds the complete policy and that files under `.agents/` "cannot add or weaken a rule in this file", the lease requirement living only in `.agents/environment.md` was **guidance rather than a rule**. That is what this change fixes. `CLAUDE.md` is a symlink to `AGENTS.md`, so the two cannot drift. ## The rule, and why it keys on the device Work on a fleet GPU goes through a [resource-controller](https://github.com/mudler/resource-controller) lease. `dgx:gpu0`, `thor:gpu0` and `orin:gpu0` are named, and the list is stated as a **lower bound**: a device `rc devices` reports is a fleet device even when this file has not caught up, so a growing fleet widens the rule rather than exempting the new box. The first draft keyed the condition on whether the local shell had `rc`. That was wrong in a way worth recording, because it reopened the exact failure this change exists to close: **`rc devices` reports your access, not the device's membership.** A host without the client that can still `ssh` to dgx would have been routed to the file mutex alone. So a missing client, an unanswering controller and a refused authentication are each enumerated by name, and each means get the client or report the controller down. **None of them turns a fleet device into a box reachable by `ssh` plus `flock`**, because the fleet cannot see that mutex. The unreachable-controller case is not hypothetical: `thor:gpu0` was observed reading `unknown (no contact 1m0s)` during review. A GPU that is genuinely not a fleet device keeps the existing `${GPU_LOCK:-$HOME/gpu.lock}` mutex, so no reader is stranded. Where both exist, the mutex runs **inside** the lease and never instead of one. ## The cost of not having this rule is already measured `.agents/specs/minimax-music3.md` §13.10 retains a whole speed axis as **VOID** because one session took the mutex over `ssh` while another held the same box through `rc`. Two mutexes that cannot see each other, and a likely 3x swing. That is the argument, and it is a recorded loss rather than a projected risk. ## What a lease can and cannot carry, measured `/workspace` in the worker is `//192.168.68.102/Data`, the same NAS mounted locally at `/mnt/nas_share/rc/`. It round-trips both ways and is writable, 7.3T with about 4.0T free. Direct execution from `/workspace` is refused, exit 126, for a script **and** a copied ELF, and `chmod +x` from the worker returns `Operation not permitted`. The mount carries `file_mode=0664` with `forceuid`/`forcegid`/`nounix` and **no** `noexec`. The same bytes read `-rwxr-xr-x` locally, where the share mounts `file_mode=0755`, so **the exec bit is a per-mount presentation rather than stored state**. An earlier draft concluded from that "a runtime cannot be staged". **Measurement refuted it.** Two routes ran staged content, each exit 0: - `/lib/ld-linux-aarch64.so.1 /workspace/<elf>`, since the loader takes the program as an argument exactly as `sh` does; - `cp` to `/tmp`, `chmod +x`, run, for both a script and an ELF. `/tmp`, `/var/tmp` and `/home/rc` are writable with real exec bits on a 3.6T overlay at 2.5T free. So the exec bit is not the wall. What remains is that nothing inside a lease can produce or fetch the first copy, the pinned oracle venv sits on the dgx host where no lease can see it, and **whether a relocated CUDA venv starts is UNMEASURED** and recorded as such. ## The blocker, narrowed rather than closed mudler#1129 records that no vLLM leg of any row can currently run on `dgx.casa` by a lease-compliant path, and the reason is now stated correctly: **because nothing has staged a runtime on the NAS**, not because the worker lacks a toolchain. That distinction decides what the next person tries. `.agents/specs/mtp-k-gt-1.md` is corrected at both sites. It had carried "it cannot reach the oracle venv **and could not start it if it could**", which the staging measurement falsifies. That clause is now retracted by name, both routes are quoted with their exit codes, and the row's owner is told **staging is worth attempting rather than futile**. mudler#1129 names `SPEC-MTP-K-GT-1` as its owning row, so that spec is exactly where the blocked owner looks, and the correction had reached `environment.md` without reaching it. The `.agents/issue-index.md` row still carries the falsified clause, correctly: the index is append-only and a landed row is never edited, so the correction lives in the spec the row points to. ## Also filed [mudler#1130](mudler#1130), fixed in flow in `.agents/workflow.md` `## Isolation`: **a merged PR is necessary but not sufficient before removing a worktree.** A worktree for merged PR mudler#1035 was reaped while carrying commits that four instruments described three different ways, and `rescue/es-cuda-grouped-unpushed` preserves the orphaned SHAs. Squash merging makes ancestry useless for this check, so the rule is to verify `@{u}..HEAD` is empty **and** that the content reached `main`. Checking either alone gets it wrong in one direction or the other. ## Gates `scripts/agent-preflight.sh`: **79 ok, 0 FAIL, 0 SKIP, 0 `--`**, exit 0, gated against `origin/main d1e5e9b`, named in both range headings, both range blocks executed rather than reporting that neither gate examined the tree. `test_gpu_lock_one_truth` stays 5/5 with exactly one `**GPU mutex:**` bullet, and that green is not vacuous: inserting a second bullet drives it red with `AssertionError: 2 != 1`, verified and restored by hash. The guarded bullet was **strengthened** rather than weakened, gaining "this runs INSIDE an `rc` lease, never instead of one" while its `${GPU_LOCK}` text stayed verbatim. Each spec commit precedes the rule edit it justifies, verified in history by timestamp and topological order rather than from the commit messages. `test_cpu_x86_llamacpp_floor` red during review at loadavg 83 and reproduced **worse** on a pristine `origin/main` worktree at loadavg 82 to 96, same `NO_QUIET_WINDOW` signature. It is green here on a quiet box. This branch is Markdown only and that harness reads none of it. ## Owed - **mudler#1129**: stage a runtime on the NAS and measure whether a relocated CUDA venv starts. That is the one test that would reopen the oracle path, and it is untried rather than impossible. - The three fleet-side fixes are named without choosing between them, since none is ours to make: the worker image gains a toolchain, `rc run` gains an `--image` flag, or `/workspace` is mounted allowing execution. The third removes only the copy step. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…efault OFF Implements [`.agents/specs/rocm-decode-attn-d128.md`](https://github.com/mudler/vllm.cpp/blob/main/.agents/specs/rocm-decode-attn-d128.md), which landed ahead of this change as mudler#564. The ROCm half of mudler#382. The CUDA half merged as mudler#425 (`66399617`); this mirrors it, adopting that arm's flag, default and stated reason rather than inventing new ones. ## What changes `d == 128` — the Qwen3-dense / Llama / Mistral head size — reaches the fast decode kernels instead of falling through to the generic `PagedAttnOnline`. `LoadRowEplBf16`/`StoreRowEplBf16` gain an `EPL=4` (`uint2`) case beside the existing `EPL=8`/`16`; the dispatch gates and the two launch switches gain a `d == 128` arm. No new kernel and no new algorithm — the kernel bodies were already generic over `EPL`. **Default OFF, opt in with `VT_ATTN_DECODE_D128=1`** — the same env var, default and reason as the merged CUDA arm. The arm is correctness-complete but not byte-exact against the kernel it replaces: warp-strided online softmax reduces the KV sequence in a different **order** than `PagedAttnOnline`'s per-tile loop, so a greedy anchor can move at an exact bf16 tie. Shipping OFF keeps every existing golden byte-identical. The flip owes the near-tie razor, a distributional gate and regen under the ratified-tie rule, and per the spec must be argued **per backend** — see the reversal below. That is what keeps mudler#382 open. ## Reviewer note Spec §4 item 3 writes the gate as `(d == 128 && (decode_d128 || decode_wmma))`. This commit implements it **without** the `decode_wmma` disjunct, which is what the same item's "Forward reference" paragraph instructs: `VT_ATTN_DECODE_WMMA` does not exist in the tree, and the flag lands with the rocWMMA arm on its own branch. The difference is intentional; it is visible in the diff before the note explaining it is. ## Evidence gfx1200 (RX 9060 XT, RDNA4, 32 CU), ROCm 7.2.3, `$GPU_LOCK` held. All figures are a same-binary flag A/B — no rebuild between arms — at 1024-token synthetic prompt, 128 generated, greedy, seed 0, 2 reps per cell agreeing within ~1%. | Model | head_dim | decode path | TPOT OFF | TPOT ON | speedup | |---|---|---|---|---|---| | Qwen3-0.6B | 128 | `qg=2` fused | 42.53 ms | 11.78 ms | **3.61x** | | Qwen3-1.7B | 128 | `qg=2` fused | 52.85 ms | 21.93 ms | **2.41x** | | Qwen3-4B | 128 | `qg=4` per-head | 81.89 ms | 39.22 ms | **2.09x** | | Qwen3.5-0.8B | 256 | — (control) | 23.76 ms | 23.55 ms | 1.01x | Qwen3-4B has no GQA fusion at any head_dim, so its 2.09x isolates the `EPL` widening from the fusion. **Qwen3.5-0.8B is the negative control and it earned its keep.** Its `head_dim` is 256, so the `d == 128` gate provably cannot reach it. Its first OFF rep came in a 33% outlier at 31.14 ms, which a blind 2-rep average would have reported as a ~1.2x "win" for a model the flag cannot affect. Re-run three times: 23.86 / 23.75 / 23.68 against ON's 23.52 / 23.57. End-to-end output throughput rises less than TPOT on the same runs (0.6B 2.48x, 1.7B 2.05x, 4B 2.02x) because they carry a 1024-token prefill the flag does not touch. TPOT isolates decode; throughput dilutes it. ### Concurrency — the advantage grows, it does not compress Qwen3-1.7B, `--num-prompts` = 2x concurrency: | Conc | tok/s OFF | tok/s ON | ratio | TPOT ratio | |---|---|---|---|---| | 1 | 12.89 | 24.66 | 1.91x | 2.40x | | 2 | 23.27 | 47.45 | 2.04x | 2.45x | | 4 | 39.10 | 86.86 | 2.22x | 2.46x | | 8 | 58.97 | 147.35 | **2.50x** | **2.77x** | | 16 | 78.43 | 227.08 | **2.90x** | **3.18x** | This refuted the prediction made before the run, which reasoned that a tiny grid at concurrency 1 flatters the fast kernel. The dominant effect is the reverse: from c8 to c16 the fallback scales only **1.33x** against the arm's **1.54x**, and scaling efficiency at c16 relative to perfect-linear-from-c1 is **38% OFF against 58% ON**. `PagedAttnOnline` is the batch-scaling bottleneck, not merely slow per call, so the win is largest in the regime a server actually runs in. The c1 row reproduces an independently-run four-model sweep to within ~1% (52.85/21.93 there vs 53.40/22.26 here). ### Correctness - `ctest -R 'rocm|cross_device'` **5/5**, chained directly to the exact-SHA push. - New case: "paged attention at Qwen3 geometry (bf16, GQA 2, head_dim 128) matches the CPU oracle", looped over `RegisteredDevices()`, NMSE <= 5e-4 vs the CPU oracle plus `OpProviderStats::declines == 0`. Genuinely new coverage — the existing generic cross-device test runs at `d=8, f32` and never reached any bf16 `EPL`-templated kernel, so none of them had bf16 coverage in this suite. (The merged CUDA arm shipped with no test at all.) - Because the arm ships OFF **and** its flag is read into a `static const bool` — once per process — the default registration can only ever gate the fallback. `tests/CMakeLists.txt` adds a second invocation with the flag set, same shape as the existing `test_dense_gateup_fused_marlin_off_*` pair. Verified non-vacuous against the mudler#463 trap: 1 case, 6 assertions, not zero. - Full `ctest` 448/455. The 7 failures are **proven** pre-existing, not asserted: a clean `main` `2784dd7b` worktree built from source with none of this code fails the identical set (only `test_op_parity`'s index shifts 403 -> 404, from the added registration). They are a missing `shellcheck`, an mmap-RSS assertion, a JSON type error, and the `SharedExpertGate` ROCm registration gap owed to unmerged mudler#509. - `agent-preflight` fails 9, a strict **subset** of that same baseline's 10 (differing only by `role-undeclared`). `check-commit-trailers` and `check-doc-checkpoint` both pass against this base. ## Carried finding mudler#382 measured this same `EPL=4` arm **1.6x slower** on sm_110 / Jetson AGX Thor, where gfx1200 measures it 2-3.6x faster. Recorded, not reconciled — different kernels, different fallbacks, different memory systems. It is why the default-ON flip must be argued per backend rather than once, and it is preserved in the spec rather than averaged away. ## Against the pinned oracle: 6.35x to 1.75x slower on per-token decode Measured after the tables above, with **both sides in the same container**, oracle = vLLM `555967922` in its production configuration via `vllm bench serve`. Qwen3-0.6B, 1024 in / 128 out, concurrency 1, **8 prompts**, warmup discarded, **3 reps**: | | TPOT reps | mean | vs oracle | |---|---|---|---| | ours, flag unset | 42.54 / 42.46 / 42.19 | 42.40 ms | 6.35x slower | | ours, `VT_ATTN_DECODE_D128=1` | 11.97 / 11.38 / 11.66 | **11.67 ms** | **1.75x slower** | | vLLM `555967922` | 6.57 / 6.90 / 6.58 | 6.68 ms | — | Running our binary against the container's ROCm rather than the host's is a substitution, so it was proved inert first: in-container matches native at 42.79 vs 42.53 ms unset, and 12.03 vs 11.78 ms with the flag. The prompt count is load-bearing. At `--num-prompts 2` the oracle returned TPOT **6.96 ms and 13.45 ms on consecutive reps**, a ~2x spread averaging to a plausible-looking and entirely fictional number. At 8 prompts with a discarded warmup both sides hold to ~±0.3 ms. This number lived only in a PR comment, which a squash merge does not carry into the tree. It is now in the spec's §5 and appended to `.agents/benchmark-record.md`, with its caveats attached rather than trailing. ## The container/glibc blocker was RETRACTED An earlier revision of this body, and the spec's §6, said the oracle re-measure was blocked on a Nix-glibc vs container-glibc ABI mismatch. **That diagnosis was wrong and is retracted.** Our binary runs inside the pinned oracle container; the earlier failures were self-inflicted (`LD_LIBRARY_PATH` exported container-wide, which breaks the container's own tools, plus a bind mount that silently yielded nothing and looked exactly like a missing ELF interpreter). §6 now reads "not run — **not blocked**", and the WMMA-spec cross-reference is gone. A false blocker in the record is worse than no record, because it stops the next person from trying. ## Not claimed **This does not close mudler#488.** That issue asks for a **per-call** kernel comparison and explicitly asserts no cause. The number above is **per-token latency** with asymmetric harnesses — the oracle over HTTP via `vllm bench serve`, ours in-process — so TPOT is the only comparable axis, and TTFT, E2EL and end-to-end throughput carry the oracle's HTTP and tokenizer overhead and are directional only. It is not the same-tool per-call trace `AGENTS.md` wants before a throughput claim. `rocprofv3` is present in the container and our binary traces under it; what is still owed is decode-phase windowing on the oracle side, or the trace compares our decode against vLLM's model load and graph capture. One board, one shape. **`docs/BENCHMARKS.md`'s ROCm axis stays PENDING**, and the row this PR adds is marked DIRECTIONAL and sits beside the existing row rather than overwriting it. **The flag-ON arm still has no proof it REACHES the new kernel, now filed as mudler#1134.** `RegisteredDevices()` (`tests/vt/test_backend_cross_device.cpp:84-96`) enumerates `{kCUDA, kMETAL, kVULKAN, kXPU, kROCM}` and excludes `kCPU`, so on a CPU-only runner — which is what CI has — the new case reports 1 test case, 0 assertions, exit 0, for **both** registrations. On ROCm the case's only backend assertion is `OpProviderStats::declines == 0`, counted at **provider** granularity, so it is identical with the flag set and unset. §9 stop condition 2 is left OPEN. The spec disclosed this honestly; what was missing is the issue `AGENTS.md` requires for a known gap not fixed in flow. Searched before filing: not a duplicate of mudler#463 (the unset-weights-env-var shape, which does not describe the `declines` half), mudler#785 (a kernel that never LAUNCHES behind a dead `#if`) or mudler#900 (same family, LTX-2.5 subject). Also out of scope and named in the spec's new `## Owed` section: the dtype gap (ROCm's decode-opt is bf16-only at every head_dim, so 4 of 5 dtype combinations still fall to `PagedAttnOnline` at `d=128` — pre-existing, inherited, not introduced), `qg=4`/`qg=8` fusion, `d=128` prefill, and the rocWMMA arm. ## Record repairs carried in the final commit `docs(BACKEND-ROCM): retract the blocker, keep the oracle number, and file the gap`, on top of joral's commits, which are untouched. It carries the retraction above; the oracle number into §5 and `.agents/benchmark-record.md`; `docs/BENCHMARKS.md` and `docs/STATUS.md` reconciled; §7's stale "**two** flag-on ctest registrations" corrected to one, matching what `8aedd780` already fixed in §4 item 3 and the Test-coverage section; a literal `## Owed` heading over the owed list; and mudler#1134 filed and appended to `.agents/issue-index.md`. Two comment-only edits at `rocm_paged_attn.hip:330` and `:455`, which still enumerated "EPL=8 → d=256, EPL=16 → d=512" without the new `EPL=4` case although the top-of-file comment at `:264` had been updated. The branch is **rebased** onto `origin/main` `d1e5e9bc` — it was 39 commits behind, and the rebase drops the earlier `merge: upstream/main` commit. Gates rerun from the worktree with explicit SHAs: `check-commit-trailers`, `check-commit-style`, `check-doc-checkpoint`, `check-public-doc-tables`, `check-agent-record` and `check-pr-size`, all OK. Not rebuilt and not re-run on hardware: every gfx1200 figure here is joral's, unchanged. Row: BACKEND-ROCM Issue: mudler#382 Issue: mudler#1134 Spec: mudler#564 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
…coped to what the developer said (mudler#1003) (mudler#1145) FOLLOWING_AGENTS_PROTOCOL Records the `VT_GGUF_KEEP_F16` default as a settled product decision, and scopes that record to what the developer actually said. Issue [mudler#1003](mudler#1003). Records and documents only. Six files, all Markdown. No product code, no checker, no measurement taken here. ## The default was owed a decision, not a re-measurement `gguf_keep_quant.cpp:233` ships `VT_GGUF_KEEP_F16` default ON, and the recorded reason the prefill loss was acceptable was stated in the competitor's terms: 204 t/s called "comfortably above the competitor floor" of `pp128 173.2`. That floor was measured against a local-only llama.cpp fork `237ad9b96` that exists on no remote. A re-taken stock `pp128` landing above 204 t/s would therefore have removed the only recorded justification for a shipped default. The developer settled it instead. The default stays ON, and the trade is now stated over our own same-binary arms, so **the default no longer depends on how the llama.cpp re-take turns out.** mudler#1003 still owes that re-take, and it still governs every published ratio on this path. What it can no longer do is invalidate the default. ## The decision and the reasoning are separated on purpose The first commit recorded the decision together with an argument for it. The second commit splits them, because these specs are read as authoritative by later sessions and the two carry different weight. `## Decision (2026-08-17)` is now three headed subsections: - **What the developer decided.** Two things, and nothing else: the default keeps its value, and it gets documented. Their instruction is quoted verbatim. The section states that they gave no rationale, that a product call does not owe one, and that only the quoted instruction carries their authority. - **Why this record holds the trade to be the right one.** The binding L7 A/B, then the argument that a gigabyte of resident weight decides whether a model fits a fixed unified pool at all, marked as this record's argument. - **What the decision changes for mudler#1003.** Opens by saying the consequence reading is this record's too. The same split is applied at four sites in `oracle-llamacpp-repin-stock.md`, and in `quantization-matrix.md`, where "the developer settled that trade" became "kept that default, with no rationale attached", naming the tie-break as the spec's reasoning. `docs/ENVIRONMENT.md` carried "the project's judgment is that a gigabyte decides", which is the same borrowed authority in a public surface. It now reads "the recorded reason for keeping it on is". Same argument, no borrowed voice. ## The trade, unchanged and unhidden Read straight off the L7 A/B. Two of the three axes regress: | axis | keep-f16 OFF | keep-f16 ON (the default) | move | |---|---|---|---| | peak RSS | 3.885 GiB | 2.832 GiB | 1.053 GiB better | | TTFT, 3 reps | 570/571/574 ms | 628/625/625 ms | about 9% worse | | TPOT | 40.4 ms | 40.95 ms | about 1.4% worse | Tokens are identical across the base arm, the default arm and the `VT_CPU_REF=1` oracle, md5 `809f2d0d6aac93a11faecd68df8a131f`. A deployment that values prefill over resident bytes sets `VT_GGUF_KEEP_F16=0`, the same-binary opt-out that reproduces the 3.885 GiB base. `docs/BENCHMARKS.md` row 4 previously said "about 10% prefill". The A/B says 224 to 204 t/s, which is 8.9%, so the cell and both restatements now say about 9%. ## Evidence `scripts/agent-preflight.sh` at `d6d9d2386`, gated against `origin/main a7583ac`, named in both range headings with both range blocks executed. Real counts by block: session role 1, record gates 27, mutation suites 46, committed range 3, commit trailers 2. **79 ok, 0 FAIL, 0 SKIP**, exit 0. Re-run independently by the operator at the same SHA with the same result rather than taken from the implementer's report. `git diff origin/main -- src/ include/ tests/` is 0 lines, and `gguf_keep_quant.cpp:233` is byte-for-byte unchanged, which is the point: this lands a record, not a behaviour change. `test_cpu_x86_llamacpp_floor` (mudler#618) passed on every run, so no pristine-worktree reproduction was needed. Disk was checked before each verdict, 21G to 32G free, so no verdict was read under ENOSPC. `docs/FEATURES.md` is the doc-checkpoint pairing the `quantization-matrix.md` edit owes, caught by a staged run failing on `feature_surface`. Its cell measures 218 characters against the 220 cap, paid for by shortening its wording rather than by touching any budget. ## Owed The comments at `gguf_keep_quant.cpp:173-228` and `test_gguf_keep_quant.cpp:478-494` still cite the contaminated denominators as the default's justification. Re-anchoring both rides mudler#1003's re-take, listed under `## Owed` in `oracle-llamacpp-repin-stock.md`. After this decision that is a wording repair rather than a decision. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…r the row that proves nothing did (mudler#1033) (mudler#1078) `check_table_shapes` already carried the right measurement -- unescaped pipes per table line, `re.findall(r"(?<!\\)\|", line)` at `scripts/check-agent-record.py:1292` -- and its call site handed it the roadmap, coordination, every matrix and every live spec. It did not hand it `.agents/issue-index.md`, and nothing else in the tree counts that file's cells. So the one record surface every change must write, whose rows are prose long enough to hide a stray pipe inside a code span, was the only markdown table in the record set with no shape gate at all. The function is untouched. Only its argument list was short, and widening the function to absorb a red would be the move AGENTS.md forbids. ## RED, then GREEN Path added, `mudler#1003` unrepaired, run bare: ``` $ python3 scripts/check-agent-record.py ERROR: .agents/issue-index.md:279: table has 9 pipes; expected 5 RC=1 ``` Exactly one row, which is the stop condition this spec set: a second red would have meant another defect landed while this was in flight and would have needed its own issue rather than a silent repair here. After escaping the four pipes, the same command bare: ``` $ python3 scripts/check-agent-record.py agent record OK: ENGINE=157 MODEL=377 QUANT=82 KERNEL=51 BACKEND=83 RC=0 ``` `tests/scripts/test_agent_record.py` gains three cases and is red on both counts before the call-site change (`FAILED (failures=2)`, exit 1), green after (80 tests, exit 0). The first captures the paths `main()` really hands the gate, from the real call rather than from the source text, so a commented-out line or an unused constant cannot satisfy it. The second runs the gate on the shipped file. The third mutates a copy -- strips the trailing pipe from the last row -- and asserts on the error text, which cannot be produced unless the mutation was both written and read back. ## The row repair, and the exception it needs Line 279 is the `mudler#1003` `ORACLE-LLAMACPP-REPIN-STOCK` row, which arrived with `283c7e492` (mudler#1051) carrying four unescaped pipes inside code spans, at columns 2705, 3106, 3115 and 3338. Only the escaping changes: the repaired line is byte-identical to the original once the four added backslashes are removed, it is four bytes longer, and it is the only line of the file that differs from `main`. Every other row is byte-identical and in the same order, the key list is a strict prefix plus `mudler#1033`, and a pipe histogram over all 290 table lines now reads `{5: 290}`. That edit is what `scripts/check-issue-index-append-only.py` forbids, and that gate is RED on this branch. It is not weakened, and there is no waiver registry, so the argument lives here with the diff. The append-only contract cannot repair a malformed row: appending a corrected copy leaves the broken one in place and adds a duplicate key, which makes `check-agent-record` angrier, and the file only becomes well-formed by editing the row where it sits. The two gates are in genuine contradiction on this tree, and that contradiction is the defect. This is the same argument `ff264cb82` (mudler#1025) made for the duplicate `mudler#995` repair, and the same containment: the gate is preflight-only, `.github/workflows/ci.yml` runs the record gate at `:121` and runs no append-only job, so this costs no CI red, and once merged a later branch diffs a `main` whose row is already repaired and sees no removal in its range. ## Two premises of the report measured false Recorded rather than quietly dropped, because a wrong premise that survives costs the next reader an investigation. The checker does NOT stop at the first finding. mudler#1033 attributes part of the two-day concealment to that shape. `main()` threads one `errors` list through every check and prints all of it at the end; the `if not errors:` guard above the block covers only the missing-canonical-record case. A scratch index carrying a duplicate key, a short row and the unrepaired `mudler#1003` row reports all three in one run and exits 1 once. So there was nothing to contain and nothing is changed there -- the concealment was real, and its only cause was that `check_table_shapes` never saw this path. The four pipes are not in a `git diff` piped into `grep`. That span does not exist in the row: it carries one `git diff` and three `git grep`, none piped. They were located by re-running the checker's own regex over the line, which is the only reason the discrepancy is visible. ## Gates `scripts/agent-preflight.sh --quiet` at `271fff52c`, exit 1, **one** failure and **zero** skips: ``` FAIL issue-index append-only FAIL: .agents/issue-index.md is append-only, and this range removes or edits lines. 1 gate(s) failed: issue-index append-only ``` That is the exception argued above, and it is the only red. Every other record gate, mutation suite, committed-range gate and trailer gate is `ok`. CI's `windows-msvc-cpu` and `windows-msvc-vulkan` are red on every pull request (mudler#584, mudler#968) with no `main` baseline. Not this change: the diff is one checker, one test file, one spec and one record, and carries no C++. Spec: [`.agents/specs/gate-issue-index-table-shape.md`](.agents/specs/gate-issue-index-table-shape.md). The spec commits first (`aed1424aa`), the implementation second (`271fff52c`), so the commit order proves spec-before-code inside the one pull request. ### The index will go CONFLICTING here `.gitattributes` gives `.agents/issue-index.md` `merge=union`, and GitHub does not run that driver, so this file conflicts on the web the moment `main` appends a row. Do not take the auto-merge. Merge `origin/main` locally, discard the merged file (`git checkout <main-sha> -- .agents/issue-index.md`), re-apply the four `mudler#1003` escapes and re-append the `mudler#1033` row, then assert by hand that `main`'s file is a strict prefix apart from line 279 and that every other row is byte-identical and in the same order. Closes mudler#1033. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…side a lease on thor:gpu0 (mudler#1146) (mudler#1147) `mudler#1129` is closed and records that a leased worker "cannot start Python", with `python3`, `pip`, `gcc`, `curl` and `git` all ABSENT. Three records in this tree carry that reasoning forward as the live cause of the oracle blocker. A reader who follows them concludes that staging a runtime is futile. It has been measured working. On `thor:gpu0` on 2026-08-17, through five `rc run` jobs, the worker runs as `uid=0(root)` with `/usr/bin/gcc`, `/usr/bin/python3` and a working `apt-get`, and a relocated CUDA runtime staged on `/workspace` imports torch 2.13.0+cu130, reports `cuda available = True` on `NVIDIA Thor` capability (11,0), runs a bf16 matmul, and compiles and executes a Triton kernel (`TRITON_JIT_OK = 4096.0 PASS`, `PROBE5_RC=0`). mudler#1129's Direction 2 probe was overtaken by a fleet change rather than wrong when it was taken. This pull request lands the record, not a new measurement. `.agents/specs/lease-runtime-staging.md` is new. It carries the four walls between a staged runtime and a running one, the working recipe, and the evidence: five `rc` job IDs with the sha256 of each staged script, taken over the files at `/mnt/nas_share/rc/oracle-probe/`, which is the folder the worker reads as `/workspace/oracle-probe/`. The four walls, in the order they appear. The `pip --target` must run FROM the worker, because the submitting host is `x86_64` and the workers are `aarch64`. `Python.h` is absent until `apt-get install -y python3-dev`, which succeeds as root. The NAS mount presents `file_mode=0664`, so Triton cannot execute its own `ptxas-blackwell`, and `TRITON_PTXAS_PATH` does not fix it because it redirects only the plain `ptxas`. So `PYTHONPATH` is ordered, with an exec-capable `/tmp` copy of `triton` ahead of the NAS tree. The scope limits are stated in the spec's own section, in each corrected record, and here, because the point of this row is that the result must not be over-read. It is `thor:gpu0` at capability (11,0) ONLY, and the GB10 is `sm_121a` and UNMEASURED. Only `torch`, `triton` and `numpy` are staged, so the pinned vLLM oracle is still not shown to run and mudler#1129's consequence for the oracle-dependent rows is narrowed rather than closed. The CUDA skew between the `+cu130` wheel and the `release 12.8, V12.8.93` ptxas is recorded as observed and not adjudicated. Two measurements landed while this branch was open, and both are folded in. Job `fd5654c0` staged `numpy` into the same tree (`NUMPY_RC=0`, `numpy 2.5.2`, `NUMPY_IMPORT_RC=0`), so the earlier "numpy is absent" line is replaced and the spec notes that the five earlier logs carry torch's `Failed to initialize NumPy` warning because they predate it. The same job closed the prebuilt-wheel question for our pin, which matters because the spec says the oracle needs `nvcc`. An aarch64 vLLM wheel exists in general: `pip download --no-deps vllm` fetched a 307,180,998-byte `manylinux_2_28_aarch64` wheel, `VLLM_DL_RC=0`. Our pin is not reachable that way, because `https://wheels.vllm.ai/nightly/vllm/` lists four wheels for exactly one commit and is a moving pointer rather than an archive, and because `0.23.1rc1.dev1511+g555967922` is a development version that is not on PyPI. The four 404s under a per-commit URL scheme are recorded as carrying no weight, since that scheme was never confirmed against a known-good case and the host's own root 404s while `/nightly` returns 200. The consequence is narrow: reproducing the pinned oracle needs a source build or a deliberate pin advance. Nobody established that vLLM never retains per-commit wheels, and the record does not say so. `.agents/environment.md` scopes its ABSENT list to the box and the day that produced it, keeps the `dgx.casa` conclusion with its narrower reason, and gains "A relocated CUDA runtime starts on `thor:gpu0`". It adds no second `**GPU mutex:**` bullet, so `test_gpu_lock_one_truth` (mudler#777) stays green. `.agents/specs/gpu-lease-methodology.md` corrects "it has no compiler, no downloader and no Python", and its `## Owed` and `## Now` no longer call the relocation UNMEASURED. `.agents/specs/mtp-k-gt-1.md` is where the blocked row's owner looks, because mudler#1129's index row names `SPEC-MTP-K-GT-1`. Its `## Owed` cell and its `## Now` now say that staging works on `thor:gpu0` and that this row still cannot resume, because the staged tree holds `torch` and `triton` and not the pinned oracle. `.agents/issue-index.md` gains one appended row for mudler#1146 naming `ENV-LEASE-RUNTIME-STAGING` as its owner, and exactly one. mudler#1129's existing row is untouched, because the file is append-only and GitHub holds its closed state. The mudler#1146 row's scope clause was corrected in place when the two measurements landed, rather than appended twice: the row has never been on `main`, so no union merge can see two versions of it, and a second row for the same issue is what `check_issue_index` refuses as a duplicate. Every correction names the binaries it probed rather than counting them, and keeps the two boxes apart. `python3` and `gcc` were probed on `thor:gpu0`, and `pip` now with them (job `fd5654c0`). On `dgx:gpu0`, job `609c4944` invoked `python3 -m pip install --target` and then hit `max_runtime exceeded (35m0s)`, so it proves `pip` starts there and not that its install finished. `curl` and `git` were probed on neither. A comment on mudler#1129 records the same thing, and says its Direction 2 measurement was overtaken by a fleet change rather than wrong when it was taken. Records and documents only. `git diff origin/main -- src/ include/ tests/` is 0 lines. No checker was weakened and no budget was raised. Gate: `scripts/agent-preflight.sh` at `ae9ead8ba`, all gates green, exit 0. Closes nothing. mudler#1146 stays open for the `dgx:gpu0` probe at `sm_121a`, for staging the pinned oracle, which needs `nvcc` first, and for confirming the `wheels.vllm.ai` per-commit URL scheme against a known-good case before anyone reads those 404s as evidence. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…iT moves between them (mudler#1118) (mudler#1140) Closes mudler#1118. `Ltx2PhaseRecipe::loras` carries upstream's per-stage adapter set, and `A2VidTwoStageRecipe` gives stage 1 `kNoAdapters` — `loras=tuple(loras)` at `a2vid_two_stage.py:107` against `(*tuple(loras), *tuple(distilled_lora))` at `:114`, read at Lightricks/LTX-2 `fd4ded7f`. Until now `dit_options.loras` was pushed once at load and every phase of every recipe ran the same fused weights, so stage 1's guided schedule ran against base + distilled where upstream runs it against the base alone. That rendered: the frames differed while the frame count, the shapes and the sample rate did not, which is why nothing caught it. ## Upstream holds ONE transformer, not two The premise that upstream pays two resident weight sets is false, and correcting it is what settled the design. Both `DiffusionStage.from_checkpoint` calls name the same `model_paths.transformer()` — `a2vid_two_stage.py:104` and `:116`, `ti2vid_two_stages.py:137` and `:148` — and differ only in the adapter tuple. So upstream pays a second MATERIALIZATION, not a second model. `Ltx2RebindDitLoras` is exactly that. It re-materializes from the pristine file the tensors an adapter targets, fuses the phase's set back in, and writes into the buffer the view already points at, so no pointer moves, `Ltx2DitWeights` stays valid without re-binding, and no second weight set ever exists. Peak residency rises by one tensor plus the adapter's own A/B factors. Three shapes were costed and the third chosen: | Shape | Resident memory | Exactness | Verdict | |---|---|---|---| | A second resident `Ltx2DitWeights` | doubles the DiT (18.7 GB nvfp4 / 21.0 GB fp8 / ~39 GB bf16) | exact | rejected — heavier than the reference, and one GB10 has 119 GB with no swap | | Unfused runtime LoRA | + the adapter | `Wx + s*B(Ax)` vs `round_bf16(W + s*BA)x` | rejected — a rounding divergence AND a different GEMM path | | Re-materialize the targeted tensors at the phase boundary | none | exact | CHOSEN — it is what upstream does | The base is reconstructed by RE-READING it, never by subtracting the delta: `round_bf16(round_bf16(W + d) - d)` is not `W`. ## The field is a SET, not a boolean Upstream needs two placements: stage 2 only for TI2Vid, A2Vid and Keyframe (`ltx-pipelines/CLAUDE.md:48`), both stages for HQ and DFR (`:49`, `:50-51`). Stage 1 `kNoAdapters` with stage 2 defaulted gives the first; both defaulted gives the second. Two enumerators are the COMPLETE space while the adapter arity is capped at one by a gated refusal (`ltx2_lora.h:167-172`). `kAllAdapters` is the default, so `distilled_two_stage`, `dfr`, `retake`, `one_stage`, `res2s` and `t2a_one_stage` keep the behaviour they were gated with — upstream-correct, since `distilled.py:131` builds one stage set. Per-phase STRENGTH is deliberately absent: `ti2vid_two_stages_hq.py:92-101` needs one, no recipe here would set it, and landing a branch nothing can select is what `ltx2_lora.h:41-44` already argues against. Owed to mudler#921. ## The gate that distinguishes per-phase from load-time fusion `ltx2 a2vid: the distilled adapter rides stage 2 ALONE`, driven entirely through `LoadVideoEngine` + `Generate` with the documented `pipeline_kind`, `lora_path`, `lora_strength` and `max_phase` load extras: - stage 1 alone (`max_phase=0`), strength 1.0 vs 0.0: **0 of 63809** artifact bytes move. The adapter is not on stage 1. - both stages, the same two strengths: **11 of 146753** bytes move. The adapter reaches stage 2. Both halves are load-bearing. The first REDs on today's load-time fusion; the second REDs on an engine that simply stopped fusing. A gate asserting only "a LoRA was applied" passes on the defect. Strength 0 is the control rather than "no adapter", because `requires_distilled_lora` refuses an a2vid load carrying no `lora_path`. The loader side gates exactness byte-for-byte against a fresh load in both directions, including the widened-to-f32 host arm the CPU parity forward runs. ## Mutations Every row prints four facts, because each has produced a false green here before. | Mutation | diff | BUILT | compile_err | exit | verdict | |---|---|---|---|---|---| | M1 drop stage 1's `kNoAdapters` | 1 file, -1 | yes | 0 | 1 | RED | | M2 delete the rebind call from the phase loop | 1 file, +1/-3 | yes | 0 | 1 | RED | | M3 delete the load-time fuse site (standing reachability mutation) | 1 file, -1 | yes | 0 | 1 | RED | | M4 rebind that does not restore the base | 1 file, +1/-1 | yes | 0 | 1 | RED | | M5 make the phase scope field inert | 1 file, +1/-1 | yes | 0 | 1 | RED | M4's first anchor was not unique and silently mutated `Ltx2StreamDitToDevice` instead, presenting as a compile error about the code under test; the harness now asserts each anchor occurs exactly once. ## Reachability M3 is the proof: deleting the production call site REDs `test_ltx2_video` (6 cases). The path is `include/vllm.h` -> `LoadVideoEngine` -> `Generate`, and `ltx2-gen --pipeline-kind a2vid_two_stage --lora ... --audio-path ...` is the same two calls through the ABI. `/v1/videos` cannot drive it, because `VideoGenParamsFromRequest` writes no `gen.extras` (mudler#928) — stated so the reach claim excludes it. ## The IC-LoRA refusal is NARROWED, not retired Its reason 2 — "this engine holds one DiT, fused at load, that every phase runs" — is now false, and both the comment and the `Fail` text say so and name what closed it. Reason 1, the reference clip's pixel path (`iclora_utils.py:112-117`, `:87-89`, `:144-148`), is untouched by this row, so the refusal stands. Retiring it here, as the dispatch proposed, would have shipped an arm whose geometry nothing supplies. ## Gate `cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF` CONFIGURE_EXIT 0, BUILD_EXIT 0, `: error:` count 0, `ctest -N` 503, CTEST_EXIT 8, `99% tests passed, 1 tests failed out of 503`. The one failure is `test_serve_low_tools` (mudler#428, load-dependent); it passes in isolation, measured rather than assumed. `No space left` 0 and `BFD` 0, each with a positive control in the same log (505 of 506 build lines matched `Building|Linking`; 500 `Passed` lines in the ctest log). Load average 73.8 at build, 47.1 at rerun; 13 GB free. No GPU and no real-weights result is claimed: this is a weight-lifetime seam and the CPU goldens are its correct gate. A real-weights comparison against upstream's own render is still owed. It is no longer blocked on an artifact: the distilled adapter and the full `-dev-` transformer have both since landed on the NAS and their headers were read (see the review repair below). What is owed is the RUN. Spec: `.agents/specs/ltx25-phase-lora.md`, committed in `86422587f` before any product code. ## Review repair (2026-08-17) A fresh review returned PASS with three non-blocking findings, all records. They are repaired in `17b211384`, `10d9d010e` and `08ea64b15`. No behaviour changed: a spec `## Outcome`, two header comments, one source comment, one `docs/USAGE.md` section and one new issue. **THE HEADLINE GATE'S MARGIN IS ONE BYTE.** "ltx2 a2vid: the distilled adapter rides stage 2 ALONE" has two halves, and the review's mutations measured both. At head, unmutated: stage 1 alone moves **0 of 63809** artifact bytes, both stages move **11 of 146753**, `SUCCESS!` over 2420 assertions. Deleting `stage1.loras = kNoAdapters` moves stage 1 to **1 of 63809** and reds `CHECK(s1_differing == 0)`. So the assertion that catches this row's own defect catches it by 0.0016% of the artifact. It cannot go falsely RED, because the render is deterministic. It can go falsely GREEN: a fixture change to the block count, the 2-step schedule, the sigma table or the PPM's 8-bit quantization could take that 1 to 0. The other half carries 11 and reds under "stopped fusing altogether", so the pair does not fail open together. Both margins and all three mutation counts are now in the spec's `## Outcome`, which the row also owed for reaching `DONE`. **PER-PHASE STRENGTH NEEDS MORE THAN A FIELD, and mudler#921 is CLOSED.** `Ltx2RebindDitLoras` early-returns on `currently_fused == fuse`, and its header calls that a no-op it detects itself. The state is a BOOLEAN, so it means "already fused" and never "already fused at this strength". HQ is stage 1 at 0.25 and stage 2 at 0.5 (`ti2vid_two_stages_hq.py:92-101`, `:154`, `:165`; defaults at `utils/args.py:1174-1184`) with BOTH stages fused, so that early return would no-op the transition and stage 2 would silently render at stage 1's strength. The trap is now written beside the early return. The review named mudler#921 as the inheriting owner; mudler#921 was closed as completed the same day by `LTX25-RES2S-LOOP` (`4d7748646`, PR mudler#1125), which correctly scoped the distilled LoRA out but did not list it under its own `## Owed` — so the debt outlived its owner. **mudler#1144 is filed for it**, the spec and both headers point there, and a forwarding comment is left on mudler#921. Recorded with it: `res2s_two_stage` already runs both stages at strength 1.0 where upstream runs 0.25/0.5, pre-existing and unstated until now, which is why mudler#1144 is a `bug`. **THE ARTIFACTS ARRIVED.** The `## Owed` bullet said the real-weights comparison needs an adapter `find /mnt/nas_share/checkpoints -iname '*lora*'` returns nothing for. That control now returns two. Headers read on the files, not copied from a model card: `ltx-2.5-22b-distilled-lora-450-bf16.safetensors`, 8,899,889,568 bytes, 3320 BF16 tensors = 1660 `lora_A`/`lora_B` pairs, rank and alpha 450, `model_version` 2.5.0, data end == file size; and `ltx-2.5-22b-dev-transformer-bf16.safetensors`, 42,018,190,584 bytes, 4349 tensors, 21.004 B params, BF16 4059 / F32 290, `model_version` 2.5.0, `keyframes_abs_pos_embedding` present, data end == file size. Also folded in, so nobody derives them twice: the f32-widen branch's direct write is unreachable with a device queue; contract drift between load and rebind cannot occur; after a `max_phase = 0` render the reference refusal prints "no adapter was supplied" because `lora_fused_tensors` doubles as the state bit, which is message-only on a path that refuses anyway. The phase-loop comment claimed a two-stage render pays one rebind; it pays two, and `docs/USAGE.md` now says so and says the cost is UNMEASURED. **One gate is RED and it is not hidden.** `check-doc-checkpoint.py` classifies any `include/vllm/` path as `user_usage` and demands `docs/USAGE.md` in the SAME commit, by path, without reading content. `17b211384` and `10d9d010e` changed only COMMENTS in two headers and carried no `docs/USAGE.md`, so both fail that gate per-commit. That is a real rule, deliberately per-commit because a diff-scoped range is never re-covered. Repairing it in place needs a rewrite of two pushed commits and this session may not force-push, so it is reported rather than hidden. `08ea64b15` carries the owed `docs/USAGE.md` section, and the squash-merge commit was simulated with `git commit-tree` and checked: it exits 0. The red is on the pull-request lane only. No build was run for the repair: the change is comments and prose, and no build can move a doctest `MESSAGE` count. `READER ANCHORS` was re-derived with a faithful port of the test's own walk and is unchanged at `823 833 834 896 992 1008 1043 1134 1159 1264 1305 1347 1349`; the port was armed first against a one-line insertion and reported MISMATCH at exit 1. `check-public-doc-tables.py`, `check-issue-index-append-only.py`, `check-agent-record.py`, `check-commit-style.py` and `check-commit-trailers.py` all exit 0, and the first two were each armed and refused at exit 1 before the tree was restored byte-for-byte. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…s gate has not run gets corrected (mudler#1128) The A2-P spec's `## Owed` lists two things this row carries and one it cannot: the thin ABI driver, the `docs/USAGE.md` weights block that rides with it, and the A3 end-to-end token gate. The first two land here. **The third did not run, and the reason on record for why it has not run turned out to be wrong.** Issue: [mudler#810](mudler#810). ## `examples/nemotron_h_gen` Modelled on `examples/kimi_linear_gen` exactly as the governing spec §6.1 requires: one project include (`vllm.h`), linked against `vllm::shared`, and no `scripts/example-abi-allowlist.txt` row, because it reaches no internal header. `check-surface-coverage.py` counts 18 example units with the same 8 reaching internal headers as before. `deepseek_v4_gen` and `laguna_gen` were deliberately not copied; both drive a bespoke forward through internal headers and are the transition state that allowlist exists to retire. The golden is JSON rather than the `.npy` plus raw-i32 pair Kimi's battery uses, so the driver carries a small scanner for that one committed shape. It refuses a shape it does not recognise instead of returning an empty vector, because an empty vector downstream is a comparison over zero elements and a comparison over zero elements reports a perfect score. **The guards are proven armed against a real engine, not argued for.** The driver is model-agnostic, so it was exercised on a small local checkpoint: | arm | result | |---|---| | golden width 8, all 8 match | `STRICT PASS`, exit 0 | | golden width 8, 0 of 8 match | `DIVERGENCE`, exit 1 | | `--steps 4` against golden width 8, **4 of 4 matched** | `SHORT`, exit 4 | | 5 malformed goldens | parse refusal, exit 2 each | The third row is the one that matters. It matched every token it looked at and still refused, because it had looked at half the golden. ## The A3 gate has not run, and its cause was wrong TWICE *First cause, dead:* contention. The A2-P spec records `dgx.casa` at loadavg 211 with 3 of 119 GB. Re-measured under a lease, the box is idle at loadavg 0.36 with 115 of 119 GB free and the GPU at 0%, checkpoint present and revision-verified. *Second cause, also dead, and it was mine:* this branch briefly claimed no CUDA binary could be built at all. **That was a HOST measurement reported as a CONTAINER measurement, and for this purpose they are different machines.** Measured inside `rc run`, the worker container is Ubuntu 24.04 running as uid 0 with `gcc`, `g++`, `cmake`, `ninja`, `make`, `python3`, `git` and `apt` present, the GB10 visible to `nvidia-smi`, and working DNS. **Only `nvcc` is absent**, and apt's `nvidia-cuda-toolkit` 12.0.140 is too old for sm_121a, so CUDA 13.x installs from the NVIDIA repo. Neither `docker` nor `sudo` is involved, and the earlier ask for them is withdrawn. The host toolchain finding ([mudler#1019](mudler#1019)) is real but gates nothing, because the host is not where work runs. The checkpoint sentences were wrong the same way, and are corrected against `findmnt` rather than paraphrase: ``` /usr/local/nas_share //192.168.68.102/Data cifs rw,relatime,vers=3.1.1,... ``` The checkpoint resolves under BOTH `/usr/local/nas_share/checkpoints/...` and `~/ckpt/...`, so `.env`'s `CHECKPOINT_ROOT` is correct and was not "fixed". **Corrected in place, with the wrong claim kept beside the correction.** A known-false line left inside a landed record is worse than the original error, because the next reader cannot tell it apart from the parts that were right, and the two-machine confusion is the part worth keeping. *Genuinely outstanding:* install `nvcc` in the build container, and whether that container can see `$CHECKPOINT_ROOT`, which is **OPEN** and claimed by nobody. `docs/BENCHMARKS.md` records the gate as pending a named resource, never as a pass. Arm 2 of the gate (three prompts concurrent and interleaved) stays blocked by design: G-SAFE refuses `num_reqs > 1` and A2-B owns that clause. ## `scripts/runner-routing-allowlist.txt:26` STAYS, decided by mutation `nemotron_h.cpp:1031-1034` still refuses the NVFP4 `lm_head` on a non-CPU queue, so the forward returns `HostLogits`. Deleting the entry in a scratch copy takes `check-runner-routing-consistency.py` from `OK` to `ERROR` naming `ForwardNemotronHForCausalLM returns HostLogits`, exit 1; the tree was restored byte-for-byte. A2-Q2b removes it. The allowlist was not widened. ## Records `docs/USAGE.md` gains the weights block AGENTS.md requires in the change that makes the capability reachable: repo, revision, staged path, byte total, the verified sha256 of the first shard, and every arm named, including the four refused ones. Its stale "refuses to run" row is corrected, and it says plainly that no token gate result exists yet. No lifecycle state changed, so no `docs/STATUS.md` write is owed. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
… the dense list verbatim Maintainer review of mudler#1056 found the upstream citations did not resolve at the pin: get_kv_cache_shape is rocm_attn.py:247-256 (not :188-194), _get_backend_priorities runs to rocm.py:441, its AITER gates are is_mha_enabled()/:is_aiter_found_and_supported() (:434,:436) with no on_gfx9(), and rocm.py:661-665 is the separate get_vit_attn_backend. Verified each against the files at pin 555967922 and corrected every citation. Per the review, drop the gfx9 inference entirely and mirror upstream's dense list verbatim: ROCM_ATTN, ROCM_AITER_FA, ROCM_AITER_UNIFIED_ATTN, TRITON_ATTN, TURBOQUANT (unregistered names are skipped by the walk, so this costs nothing). Record two review decisions in backend.h + the new rocm-attn-backend.md spec: (a) upstream's use_kv_connector gate does not apply because our registered shape is the shared symmetric NHD layout, not the asymmetric K/V-outermost views that guard protects; (b) the NHD-vs-KV-outermost layout deviation is kept as ONE exact tracked exception rather than flipping to FLASH_ATTN. Also refresh the lifecycle docs the row owes (backend-matrix BACKEND-ROCM row, docs/FEATURES.md, docs/ROCM.md M3). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4 [Freebuff]
…, by name (mudler#1123) (mudler#1132) A 370 GiB checkpoint loaded for 26 minutes on `--device cuda`, reported ready, and then died on the first request. It now refuses at load, by name, and says what is missing. `Qwen3.8-2.4T-A95B UD-Q1_0` serves correctly on `--device cpu` on a DGX Spark: TTFT 667.0 s, steady decode 44.2 s/token, coherent output. On `--device cuda`, same box and same binary, it reached a serving state after 26 minutes and then died inside the EngineCore busy loop with `vt cuda: cudaMalloc: out of memory` (mudler#1123). Loading for 26 minutes and dying mid-stream is the worst of the three available behaviours, and `AGENTS.md` already says which one is right: refuse an unimplemented arm at load with a message that names the missing part. ## The allocation, named and sized The log line could not say which allocation failed or how big it was. `CudaBackend::Alloc` is `Check(cudaMalloc(&p, bytes), "cudaMalloc")` (`src/vt/cuda/cuda_backend.cu:77-81`) and `Check` composes `"vt cuda: " + what + ": " + cudaGetErrorString(err)` (`:48-52`), where `what` is a compile-time literal. `bytes` is in scope and discarded. So the answer came from the code and the checkpoint rather than from the message. **It is `d.b.Alloc(nb)` in `ResidentWeight`, `qwen3_5.cpp:1010-1011`**, with `nb = w.bytes.size()` — for a routed-expert weight, the whole STACKED `[E*N,K]` keep-quant tower, every expert of one matrix of one layer, in one contiguous `cudaMalloc`. Both switch positions of the keep-quant MoE path reach that same line, which is why no knob avoids it. Every `f:N` below is a CALL SITE — the line inside `f` that invokes the next hop — never a definition line: | Configuration | Path | |---|---| | default (grouping on) | `MoeBlock:6615,6616,6620` → `KqGrouped:5694` → `ResidentWeight` | | `VT_MOE_EXPERT_STREAM=1`, which DISABLES grouping (`:5670-5676`) | `ExpertMlpKq:5651,5652` → `MatmulF32Slice:5611` → `KqExpertSlice:5595` → `KqResidentSlice:5114` → `ResidentWeight` | `KqExpertSlice`'s slot arm is guarded by `is_cpu()` (`:5578`), so on a device platform it falls through before the store is even constructed. Ruled out by reading rather than by assumption: `cuda_moe.cu` and `cuda_glue.cu` contain no allocation at all, and `BuildMoeMarlinResident` (`:6010-6215`, whose per-expert allocations are `:6049-6064` plus two repack temporaries at `:6094-6095`) is not on this path, because `MoeBlock` takes the fp4/Marlin arm at `:6555` under `const bool fp4 = !w.expert_gate_fp4.empty()` (`:6548`), and a GGUF populates `expert_*_kq`. The size came from re-censusing both GGUF tensor tables at revision `567d3e6ac26c5474b18311e619c04350fb9a5556` over all ten shards by HTTP range request, no tensor data downloaded: **1702 records parsed against the 1702 declared in `split.tensors.count`**, which is the coverage claim. | Tower | Bytes | | Count | |---|---|---|---| | IQ1_XXXS `ffn_{gate,up,down}_exps` | **1,275,068,416** | 1.1875 GiB | 276 | | Q2_K, block 92 (the `nextn` MTP block) | **2,818,572,288** | 2.6250 GiB | 3 | | all `*_exps` | **360,374,599,680** | **335.62 GiB** | 279 | Cross-check that the arithmetic and the running lane agree on the same weight: `1,275,068,416 / 512 = 2,490,368`, exactly the `slot_bytes=2490368` the row's W4 banner printed. **The budget needed the right instrument.** `nvidia-smi --query-gpu=memory.total,memory.free,memory.used` answers `[N/A], [N/A], [N/A]` on a GB10, and the `rc` fleet label records `vram=[N/A]M`. `cudaMemGetInfo` answers honestly — measured on `dgx:gpu0` under an `rc` hold, through `libcudart.so.13`: ``` cudaMemGetInfo rc = 0 free = 122059919360 (113.677 GiB) total = 128452956160 (119.631 GiB) attr Integrated rc=0 value=1 attr UnifiedAddressing rc=0 value=1 ``` `total` is EXACTLY `/proc/meminfo MemTotal` (125442340 kB) times 1024. So 335.62 GiB of tower staging is 2.8x the whole machine; the load survives only because a borrowed tower costs zero anonymous bytes, and staging exhausts the pool after roughly `(119.631 - 62) / 1.1875 = 48` towers, partway through layer 16 of 93. This corrects the hypothesis in the issue body, which had the mechanism right and the quantity wrong: it is not ~6.5 GB of per-token staging that fails, it is the whole tower set being made device-resident once. Per-token slicing never runs. ## The refusal Keyed on the measured condition, never on "CUDA + GGUF" and never on an architecture name: refuse <=> needs_weight_staging AND budget != 0 AND needed > budget so a GGUF that fits the pool still loads on `--device cuda`, and every `--device cpu` load is byte-identical because a non-staging platform returns before a footprint is even computed. Three polarities are deliberate. **The footprint is a PER-TENSOR lower bound, and the sum is wrong in BOTH directions.** Per tensor it is `min(gguf_bytes, elems * model_dtype_bytes)`: a kept-quantized weight is staged verbatim, an expanded one at the model dtype, and which happens is a per-tensor loader policy this module does not try to predict. Each term is therefore a true lower bound on that tensor's staged size. **The sum is not a lower bound on the load**, and the first version of this pull request claimed it was ("so the refusal can never over-refuse"). It can over-refuse: a tensor counted and never staged is a positive over-count, and one is present on every default load — the MTP / `nextn` block, block 92, 20 tensors, 8,940,488,704 of 397,245,341,184 bytes, **8.33 GiB, 2.2506 %**. A budget in `[what a default load stages, what the bound counts)` rejects a weight set that fits. It also under-counts, by everything that is not a weight, which is larger. The two errors are on DIFFERENT quantities and never cancel, so "the under-count dominates" is an argument about a number the refusal does not compare. Both directions are named in `gguf_device_fit.h`, the over-count is now pinned executably rather than described, and both remainders are owed — the over-count to mudler#1136, the under-count to `KV-WARMUP-PROFILE`. **The budget is the pool TOTAL, not the free bytes**, so the verdict does not move with contention. **A budget of 0 means UNKNOWN, which is not a verdict.** A caller that cannot learn the budget declines to decide, because refusing a load on an unknown budget would break every device whose budget nothing probes. That is the opposite polarity from `gemma4_moe.cpp:506`, which refuses a device allocation on unknown, and the difference is deliberate: there a hung `hipMalloc` is worse than a host fallback, here a false refusal is worse than a late failure. **Which platforms this covers**, stated precisely because the first version got it wrong in both halves. The two predicates coincide on exactly one platform: `needs_weight_staging()` is true only on `CudaPlatform` (`platforms/cuda.cpp:71`) and only `CudaPlatform` probes a budget. So EVERY NVIDIA GPU this build runs on — discrete or GB10 — gets both the probe and the refusal; a discrete card is `CudaPlatform` too, not a separate case. ROCm, Vulkan and Metal answer `needs_weight_staging() == false` (ROCm says so explicitly, `platforms/rocm.cpp:74`): they read the mapping where it lies, so there is no staging allocation to fail and the refusal is inapplicable rather than owed. What is owed on ROCm is the separate `Backend::DeviceMemoryInfo` capability (mudler#1126). The budget arrives as NEW data on `ResidencyPolicy`, probed once by `CudaPlatform`'s registrar. It is deliberately **not** `Backend::DeviceMemoryInfo`, whose comment claimed "ROCm/CUDA override with hipMemGetInfo/cudaMemGetInfo" while only ROCm does (`src/vt/rocm/rocm_backend.hip:338-345`) — so `Gemma4MoE`'s device-expert LRU is dead on every CUDA device today, and waking it is a behaviour change with its own measurement. That comment is corrected here, in **both** places that carried it: `include/vt/backend.h:78-93` and `gemma4_moe.cpp:440-448`. The capability is mudler#1126. `ResolveModelDeviceType` is extracted from `SelectQueueForModel` because the check must know the target device before any weight I/O while the load's queue is not created until after the weights load. One description, not two — and on the AUTO arm that description CREATES A QUEUE, for the reason in the next section. Reachable from two production entry points, both of which funnel through `LoadedEngine::FromModelDir`: the server binary, on its embedding lane (`server_main.cpp:959`) and its generative lane (`:1123`), and the C ABI (`vllm_c.cpp:766`). `examples/bench/bench_core.h:586` also calls it and is an example, which `AGENTS.md` says is not a production entry point. ## Round two: the review repairs (mudler#1136) A fresh review at `e7d0a1f7c` returned FAIL on one gate and five claims. One of those claims guarded a real defect. **`ResolveModelDeviceType` could remove a working load.** The header claimed it picks what `SelectQueueForModel` picks, so "the two cannot drift". On the AUTO arm they could. `SelectQueueForModel` wraps `CreateQueue()` in a try/catch and falls back to CPU, and its own comment says why: "a platform can be registered while CreateQueue still fails, and CPU must remain reachable". The resolver asked `CurrentPlatform()` and stopped, so on such a box it answered `kCUDA` while the load ran on CPU — and the refusal then rejected a checkpoint by naming `'cuda'` for a model that previously loaded and served on CPU. Whether `CreateQueue()` fails is knowable only by calling it, so both callers now share one `ResolveAutoDevice` that creates the queue and hands it to whichever caller wants one; `ResolveModelDeviceType` destroys the queue it does not use, because `vt::Queue` is a non-owning handle with no destructor and dropping it would leak the stream. The cost is one extra stream created and destroyed per auto-arm GGUF load, stated in the header rather than hidden. **`CudaPlatform`'s policy assembly is now pinnable without a CUDA build.** mudler#1123 recorded "delete the `device_memory_total_bytes` assignment" as an owed mutation because `platforms/cuda.cpp` compiles only in a CUDA build. The four-line assembly moved to `CudaResidencyPolicy` in `vllm/platforms/interface.h`, which compiles everywhere, and `test_platform.cpp` pins every field. `cuda.cpp` keeps only the `cudaMemGetInfo` probe it alone can make; that call and the constructor threading are still unproven and still said to be. Its one-line edit was checked with `g++ -fsyntax-only` against a stub `cuda_runtime.h`, and the instrument was itself verified: a deliberate typo in the same expression returns rc 1 naming it. **Fourteen stale or wrong anchors and names**, each re-derived against the final tree rather than re-quoted (the two this change authored itself are counted in the next paragraph, not here): `cuda_backend.cu:75-81` is `:77-81`; `BuildMoeMarlinResident` is `:6010-6215`, not `:6010-6062`; `gemma4_moe.cpp:439-447` and `:494-506` are `:449-455` and `:506`; `CheckGgufDeviceFit` never existed, the function is `CheckDeviceWeightFit`; the call-chain table mixed call sites with definition lines and now declares one convention and keeps it; and the header's upstream anchor `vllm/v1/worker/gpu/model_runner.py:504,647` pointed at a `DraftModelSpeculator` `set_attn` call and a `torch.zeros`. The startup memory profile is `GPUWorker.determine_available_memory` (`gpu_worker.py:451-495`) around `profile_run` (`gpu/model_runner.py:682`), and it takes the weight bytes as an INPUT recorded after the load completes (`gpu/model_runner.py:315`) — a stronger statement of why upstream has no counterpart than the one that was there. That `:504,647` pair was copied from `.agents/engine-matrix.md`'s `KV-WARMUP-PROFILE` row, which still carries it plus a third stale anchor; filed as **mudler#1139** and NOT fixed here, because the fix is one cell in `engine-matrix.md` that PR mudler#1119 is concurrently bumping alongside the hardcoded count in `check-agent-record.py`. **The anchors this change itself moved.** Inserting ~45 lines near line 100 of `model_loader.cpp` shifts every absolute line citation below it. Measured between `e7d0a1f7c` and this head by comparing the TEXT at each cited line: **203 moved line references over 109 citing sites in 45 files**, 10 unmoved. Two were authored by this change (`model_loader.cpp:135-141` in the reachability gate, `:1411-1412` in the arithmetic gate) and both are corrected and added to this round's anchor verifier — the trap the round before hit with `platforms/cuda.cpp:67`. The other 107 are not swept, and not for effort: several were already stale at `e7d0a1f7c` (`model-matrix.md:197` cites `:184-223` as the "live loader" while line 184 there is `static const bool once = [] {`), so rewriting them from the current tree would launder pre-existing debt into a clean-looking record. Filed as **mudler#1143** with the measurement and three candidate fixes. **One contradiction was self-inflicted and caught before it shipped.** `gguf_device_fit.h` opened by calling the footprint "WRONG LOW rather than wrong high" and closed twelve lines later by saying it can over-refuse. The fix is naming the scope: the sum is a lower bound on staging EVERY TENSOR IN THIS FILE, which is not a lower bound on what one load stages, since a load may stage a subset. The per-tensor term is exact-or-low; the SET is exact-or-high. That is where the over-count lives, and it is why the field keeps the name `lower_bound_bytes`. **The false comment had a second copy.** The correction to `include/vt/backend.h` is the finding; the audit for it found the same "(ROCm/CUDA)" claim at the seam's only call site in `gemma4_moe.cpp`. Correcting the header alone would have left the falsehood in the tree. **`docs/USAGE.md` and `docs/ENVIRONMENT.md`** now state the platform coverage correctly and warn an operator about the over-count window. **The "Release (what CI uses)" claim was false.** `build-test-cpu` (`ci.yml:903-911`) passes no `CMAKE_BUILD_TYPE`, so CI's test lane defines no `NDEBUG` and has asserts live. This round was configured and gated in exactly that configuration. ## Gates Configured as CI's `build-test-cpu` does — `-DVLLM_CPP_BUILD_TESTS=ON -DCMAKE_EXPORT_COMPILE_COMMANDS=ON`, no `CMAKE_BUILD_TYPE`, so `assert` is live. `origin/main` advanced to `a7583ac75` during the round and is merged in with an AUTHORED merge commit, because `agent-preflight.sh` SKIPS both trailer gates when `origin/main` is not an ancestor of HEAD — and a skip still exits 0, so a reader who trusts the return code reads two unexecuted gates as green. That is documented at `agent-preflight.sh:35-42` and `--fail-on-skip` is the opt-in for it; every preflight figure below is from a post-merge run with that flag. Build and gate were re-run in full after the merge. **That merge silently corrupted a record, and a gate caught it.** Both sides had appended after the same `mudler#837 / mudler#1117 / mudler#1118` tail of `.agents/issue-index.md` — this branch `mudler#1136 / mudler#1139 / mudler#1143`, main `mudler#1129 / mudler#1130 / mudler#1134` — and the `merge=union` driver emitted the shared three-row block TWICE, once as each side's context. `git merge` exited 0 and reported no conflict. `check-agent-record.py` and `test_agent_record` both failed with "issue mudler#837 listed twice. Under `merge=union` a duplicate is what two branches appending the same issue look like". Deleting the second copy was not enough, and a SECOND gate said why: `issue-index append-only` stayed red, reporting main's `mudler#1129` and `mudler#1130` as REMOVED, because the two files disagree on where the shared block sits. Main's tail runs `mudler#1129, mudler#1130, mudler#837, mudler#1117, mudler#1118, mudler#1134`; this branch's ran `mudler#1127, mudler#837, mudler#1117, mudler#1118, mudler#1136, ...`. Any resolution that keeps this branch's ordering reads as a reorder, and a reorder of an append-only file is a deletion plus an insertion however the rows are spelled. Resolved the way `AGENTS.md` prescribes for a keyed record — take the COMPLETE target-branch version, then apply the scoped edit again: `origin/main`'s file verbatim with the seven rows only this branch has appended at the end. Verified rather than assumed: every line of main's version is present, `git diff --numstat origin/main` reports **`7 0`** (seven insertions, zero deletions), 322 rows, no duplicate id. `merge=union` makes concurrent appends merge without a conflict; it does not make a RELOCATED append safe, and the duplicate is what a relocation looks like on the way out. Two gates were needed to see the whole of it, which is the argument for both. | Gate | Result | |---|---| | configure into an empty build dir, then full build (re-run after the merge) | rc 0, 508 `Built target` lines, **0 ENOSPC**, 0 `error:`, 0 `warning:` | | `test_gguf_device_fit` | 8 cases, 61 assertions, 0 failed, rc 0 (was 7/53) | | `test_gguf_device_fit_reach` | 5 cases, 23 assertions, 0 failed, rc 0 (was 3/14) | | `test_platform` | 12 cases, 95 assertions, 0 failed, rc 0 (was 11/86) | | `test_device_selection` (regression on the resolver) | 2 cases, 11 assertions, 0 failed, rc 0 | | `ctest --test-dir build` (serial, exactly CI's command) | **506 tests, 100 % passed, 0 failed, rc 0**, 293 s, on the merged tree; 2 expected skips (`test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`). 506 rather than 505 because the merge brings `test_backend_cross_device`, so the count moving is itself the proof the merged test is registered and ran. The gate was run THREE times over the round and every run was 100 %, at host load averages of 16-134, with no starvation flakes | | `scripts/agent-preflight.sh --staged --fail-on-skip` | rc 1 — **80 of 81 gates ok, ZERO skipped, one failure, and it is not this change**: see below | The single preflight failure is `commit-trailers`, on the two unauthored MERGE commits already on the task branch (`058f5f31a`, `e7d0a1f7c`): both carry the default `git merge` subject and an empty body, and `scripts/check-commit-trailers.py:331,336` walks the merge-base range with no merge exclusion. This change's own commit passes the contract on its own range (rc 0), and `main` is squash-only so neither merge commit ever reaches it. Repairing them means rewriting history that is already pushed, which `AGENTS.md` forbids doing by force, so it is a branch-history decision rather than a code one. Every other gate — all 80 — is `ok`, including `check-agent-record` (`ENGINE=157`, no collision with mudler#1119), `issue-index append-only`, `check-env-doc`, `check-public-doc-tables` and `commit-style`. **RED captured first, on a compiling stub**, because a red that fails to BUILD reads as a pass: | Red | Result | |---|---| | the two AUTO-arm cases, against the unfixed resolver | 5 cases, **2 failed, 4 assertions red**, rc 1 — one because the refusal fired for a load that runs on CPU, one because `queues_created` stayed 0, which is the divergence itself | | the `CudaResidencyPolicy` case, against a stub returning a default policy | 12 cases, **1 failed, 7 assertions red**, rc 1, and the case count MOVED 11 → 12 | The case count is asserted to move because a test class that never compiles into the binary prints a clean pass: the first run of `test_platform` here read 11 cases and 86 assertions — the pre-change numbers — because the build had already compiled that file before the case was added. The `test_gguf_device_fit` over-count case has **no red-first**, and that is stated rather than papered over: it characterises behaviour the tree already had, so its only evidence is mutation M4 below. ### Mutations Seven, each printing its `applied` proof (sha256 before/after plus a non-empty `git diff --stat`), its build rc, and a NON-ZERO doctest case count taken from the LAST `test cases:` line. A mutation that did not apply, or did not build, is reported INVALID and never as a pass. Every file was restored from a byte backup — never `git checkout --`, which would discard an uncommitted repair — and the restore verified by sha256. Every literal was asserted to match EXACTLY once before it was applied, so a mutation cannot silently hit the wrong site or no site. | # | Mutation | applied | compile | Result | |---|---|---|---|---| | M1 | the resolver goes back to the bare `CurrentPlatform()` query (the pre-fix code) | `16389220`→`8396036f`, 20 lines | rc 0 | reach: 5 cases, **2 failed** — CAUGHT. `test_device_selection` 2/2 green, which is correct: it covers only the explicitly-named arm | | M2 | the probe queue is dropped instead of destroyed | `16389220`→`54e7ace6`, 1 line | rc 0 | reach: 5 cases, **1 failed** — CAUGHT | | M3 | `CudaResidencyPolicy` stops carrying the probed budget | `5e51dfc0`→`483449d7`, 1 line | rc 0 | platform: 12 cases, **1 failed** — CAUGHT | | M4 | the footprint EXCLUDES `*.nextn.*`, i.e. the over-count is removed | `4edabac3`→`fcbcf384`, 1 line | rc 0 | fit: 8 cases, **1 failed** — CAUGHT. This is the over-count case's only evidence, because it has no red-first | | M5 | delete the production call site (`if (fit.refuse) throw ...`) | `16389220`→`22879f14`, 1 line | rc 0 | reach: 5 cases, **2 failed** — CAUGHT | | M6 | strict `>` becomes `>=` | `4edabac3`→`8b409c96`, 1 line | rc 0 | fit: 8 cases, 2 failed; reach: 5 cases, 1 failed — CAUGHT | | M7 | drop the non-staging early return | `4edabac3`→`035a886e`, 1 line | rc 0 | fit: 8 cases, 1 failed; reach: 5 cases, 2 failed — CAUGHT | Every row above was produced in ONE pass against the head being merged, after the last repair, so none of it is evidence about an earlier tree. `git status` is empty afterwards and each file's sha256 matches its pre-mutation value byte for byte. **Still owed and still said to be**: the `cudaMemGetInfo` call in `platforms/cuda.cpp` and the constructor that threads its value are not mutation-proven, because no CUDA toolkit is reachable from this host. mudler#1123 owed the whole policy assembly; this round moved the assembly out and pinned it, so what remains owed is the probe alone. Its file is syntax-checked, with a verified positive control. ## Owed, filed rather than folded in - **mudler#1124** — the device-side expert slot store this refusal stands in for. Four concrete pieces, sized at 2790 slices per token times 2,490,368 bytes = 6.95 GB per token. Not started here because W7 owns the pluggable backing store and the CPU arm's own decode bandwidth is still VOID. - **mudler#1126** — `CudaBackend::DeviceMemoryInfo`, and the Gemma4 measurement that has to come with it. Both false comments about it are corrected here; the capability is not built. - **mudler#1127** — moving `VT_DEVICE_WEIGHT_BUDGET_BYTES` into the `vllm_cpp` config namespace once mudler#1119 lands. - **mudler#1136** — the bound's over-count direction. Not closed because closing it means the bound taking a per-tensor staging POLICY as input, which is the caller's knowledge and not the file's, and an exclusion that is wrong under-counts toward zero, restoring the exact failure this row removed — on a device nobody on this fleet has to measure the change against. - **mudler#1139** — `KV-WARMUP-PROFILE`'s three stale upstream anchors, blocked on the `engine-matrix.md` / `check-agent-record.py` record lock that mudler#1119 holds. - **mudler#1143** — `model_loader.cpp`'s 109 line-number citations across 45 files, and the three candidate fixes for the class. Needs a row of its own. All six are listed under `## Owed` in [`expert-streaming.md`](.agents/specs/expert-streaming.md). Coordination note for mudler#1119: the two changes overlap on `src/vllm/entrypoints/model_loader.cpp`, `include/vllm/entrypoints/model_loader.h`, `docs/USAGE.md`, `docs/ENVIRONMENT.md` and `.agents/issue-index.md`. This one adds no roadmap row and touches neither `.agents/engine-matrix.md` nor `scripts/check-agent-record.py`, specifically so it cannot collide with mudler#1119's `ENGINE_ROWS` bump; `check-agent-record.py` still reports `ENGINE=157` here. The issue-index rows are appends. Closes mudler#1123. Closes mudler#1136. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…he one `if` that made it unreadable (mudler#1154) Closes mudler#1148. `PlanDit` refused any DiT checkpoint carrying neither `U8` nor `F8_E4M3`, so `ltx-2.5-22b-dev-transformer-bf16.safetensors` — the FULL model — could not be read at all. Upstream's table (`packages/ltx-pipelines/CLAUDE.md:17-30` @ `fd4ded7f`) marks `Full` or `Full + distilled LoRA` for six pipelines, four of which are landed here as `one_stage`, `t2a_one_stage`, `res2s_two_stage` and `a2vid_two_stage`, so all four could only ever run against a distilled checkpoint. That is a different sampling regime, it renders plausibly, and nothing validates the class (mudler#1137). `Ltx2DitQuant` grows `kNone`. It is not a third scheme: `_DTYPE_CASTABLE` (`single_gpu_model_builder.py:51-57` @ `fd4ded7f`) is float32/float64/float16/ bfloat16 and calls uint8-NVFP4 and float8 "quantized payloads", so unquantized is the BASELINE the two quantized arms are exceptions to. ## The scope was read before it was fixed, and it held Nothing past `PlanDit` had ever been exercised on a pure-bf16 file, so the spec enumerates nine callees with the answer for each. `MaterializeDitTensor` already carries a `BF16` branch at `:454-462` — the branch every bias on the FP8 arm already takes — `ParseLtx2DitParamsFromManifest` reads shapes and never doubles an unpacked width, `FuseLorasInto` and `Ltx2RebindDitLoras` key on the dtype the materializer returned, and no consumer in `src/` branches on `Ltx2DitCheckpoint::quant` at all. So this adds no materialization code. Had it needed any, the dispatch asked for `NEEDS_DECISION` rather than a half path. ## The refusal that survives says something a reader can act on The old message said "use the L2 path", and `Ltx2LoadDitFromSafetensors` IS the L2 path — it calls `PlanDit` on its first line, as do `Ltx2ParseDitParamsFromCheckpoint`, `Ltx2StreamDitToDevice` and `Ltx2RebindDitLoras`. It sent every reader in a circle. What replaces it fires when the file carries no dtype this loader reads, and names both the dtypes the file holds and the four encodings the loader materializes. F16 is the live case: upstream's castable set lists `torch.float16` and this port has no F16 path. ## The gate proves numbers, not loading `test_ltx2_loader` compares every contract weight BIT-FOR-BIT against the bf16 the fixture wrote, and checks the expectation itself for the two shapes a stub hits by accident — `want_nonzero == want_total` (a zero fill scores 0) and `distinct > 1000` (a constant fill scores 1) — plus `rank2 == unquantized_weight_tensors` so the comparison provably covered the 90 tensors, 1,257,728 elements, the quantized arms take a different branch for. `REQUIRE(dtype == kBF16)` holds the memory format, which a value comparison cannot see. `test_ltx2_video` owns reachability: `LoadVideoEngine` on its default configuration loads an unquantized fixture DiT and renders phase 0, and the file is asserted quantized-free before the engine opens it. ## Eight mutations, and one of them changed a test M5 put "use the L2 path" back into the surviving refusal and was NOT DETECTED: 37 of 37 passed at exit 0. The F16 case had retyped only the `BF16` entries, leaving the F32 `scale_shift_table` families, so `PlanDit` still saw a readable dtype and the refusal came from `MaterializeDitTensor` instead — a message that satisfies the same two assertions. The case passed while the branch it exists for went unexercised. Repaired to retype every entry (halving the F32 payloads, because `safetensors_reader.cpp:165-173` cross-checks `numel * dtype_size == nbytes`) and to assert the message is `PlanDit`'s before asserting its content. M5 and M8 both DETECT after it. The full table with all four facts per mutation is in the spec §9; M1 reds `test_ltx2_video` and M7 (deleting the production call site) reds 61 of 80 cases. ## The hub's content hash for this repo is a fabrication that type-checks `Lightricks/LTX-2.5` is gated, and its tree API answers an unauthenticated caller with an `lfs.oid` that is one character repeated 64 times — right length, lowercase hex, `len == 64` passes — identically for all 14 LFS files. The first draft of the `docs/USAGE.md` pin carried five sha256 columns from that field. The assertion that caught it was written for another reason entirely: the dev and distilled bf16 transformers are the same SIZE (42,018,190,584 bytes each), so the row wanted to say a size check cannot separate them, and `dev_oid != distilled_oid` fired. `docs/USAGE.md` now publishes one sha256, the local copy's over all 42,018,190,584 bytes, labelled as such, and says "not obtainable here" for the other four with mudler#1048 owning them. Spec §8. ## Owed, and named A real-weights MATERIALIZATION. The gated case parses the shipped dev file's 677,616-byte header and stops, because a full load materializes ~42 GB of host bf16 that the CPU gate cannot hold; `dgx:gpu0` is unhealthy and neither `thor:gpu0` nor `orin:gpu0` can hold it. Run locally against the real file, it resolves `kNone`, recovers 48 layers / 4096 / 2048 / 128 / 128, reads `keyframes_abs_pos_embedding` as TRAINED, counts 0 U8, 0 F8_E4M3, 0 sidecars, 4059 BF16 and 290 F32, and adopts its declared config onto the identical weight contract: 1 case, 18 assertions, exit 0. Also owed: the checkpoint CLASS check (mudler#1137) and the memory envelope, both listed under the spec's `## Owed`. ## Gate `cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF` | | pre-merge (`c04f49f5f`) | merged (`652c02749`) | |---|---|---| | `CONFIGURE_EXIT` | 0 | 0 | | `BUILD_EXIT` | 0 | 0 | | `: error:` | 0 | 0 | | `ctest -N` | 504 | 506 | | `CTEST_EXIT` | 0 | 8 | | pass/fail | 100% passed, 0 failed of 504 | 99% passed, 1 failed of 506 | | wall clock | 55.49 s | 221.16 s | | load average | 25.9 | 37.5 -> 49.9 | The one red is `test_engine_core_proc`, and it is [mudler#1052](mudler#1052) by name: the failing assertion is `CHECK(abort_seen)` at `tests/vllm/v1/test_engine_core_proc.cpp:481` in "EngineCoreProc: abort-mode shutdown aborts in-flight requests", which is the fixed 1000-frame budget racing an unbounded producer that issue describes. It passed in both pre-merge runs at load 22 and 25, failed in both merged runs at load 37 and 49, and passes ALONE on the merged tree (1 of 1 matched, 0 failed, exit 0). This row touches no file within reach of it. `No space left` and `BFD` are 0 in both the build and ctest logs, against positive controls that read 1 for the same greps. Disk went 49 GB -> 9.7 GB -> 28 GB free during the run as sibling agents built; the build tree is 11 GB and is removed with the worktree. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…t grades (mudler#503) Closes the structural half of mudler#503. `windows-msvc-cpu` and `windows-msvc-vulkan` now run on the lane that grades `main`, and the published verdict grades them. `scripts/main-baseline.py` already refuses three shapes of not-green: a job that ran and failed, a job still running, and an expected job the payload never mentioned. It added the third precisely because absence could otherwise wear green's face. It could not see this instance. Both MSVC jobs carried `if: github.event_name == 'pull_request'`, so they were not DEFINED for any event the `schedule`/`workflow_dispatch` lane fires on, and `EXPECTED_JOBS` did not name them — so they were absent from `failing`, from `pending`, from `not_run`, from `missing` and from `covered` alike. Nothing printed them, and the verdict read `NEWEST BASELINE: GREEN` while `main` did not compile under MSVC. Measured on the deliberate `workflow_dispatch` baseline 32044993401: `conclusion=success`, both jobs `skipped`. That is not a bookkeeping complaint. mudler#503, mudler#603, mudler#729, mudler#965, mudler#968 and mudler#1068 each landed on `main` unseen and then reddened an unrelated contributor's pull request, presenting to that author as a defect their own diff had caused. Five of the six were found by someone who first had to prove the red was not theirs. ## What changes The condition becomes `pull_request || schedule || workflow_dispatch`, both jobs join `baseline-summary`'s `needs:`, and both join `EXPECTED_JOBS`. Running on the lane and being graded by it are separate properties, and having only the first would rebuild mudler#274's defect by omission: the job would fail, and the published verdict would not move. `push` stays excluded, as a decision rather than an omission. That lane cannot answer "is main green" by construction — every expensive job there groups on `github.ref`, constant for every push, so consecutive pushes cancel each other (26 of 40 runs in the window `ci.yml` records) — and two `windows-2022` runners on 55 pushes a day buys a bill, not a baseline. The condition stays a byte-exact literal in `check-release-workflow.py::validate_pr_ci` rather than becoming a predicate over conditions. That function compares each Windows job's WHOLE mapping against a literal dict, which is how the lane proves it holds no release, upload, write-token or OIDC authority (mudler#117), and widening the pin would trade a property this change does not own. `always()`, `github.event_name != 'push'`, an added `push` arm and a reverted bare `pull_request` are each still rejected. ## Red-before Each mutation was applied to the head tree, run, and reverted, with sha256 verifying the restore on both mutated files: | Mutation | `check-release-workflow.py` | `test_main_baseline.py` | |---|---|---| | revert both `if:` to `pull_request` only | rc=1 | 4 failures | | drop both from `baseline-summary` `needs:` | rc=0 | 3 failures | | widen both `if:` to `always()` | rc=1 | 8 failures | | add `|| github.event_name == 'push'` | rc=1 | 2 failures | | narrow `EXPECTED_JOBS` back to nine | rc=0 | 4 failures | Four new cases in `tests/scripts/test_main_baseline.py` and four new mutations in `tests/scripts/test_release_pipeline.py`. The `if:` is resolved to a boolean per event through the existing `resolve_boolean`, not grepped, for the reason that helper exists: `always()` and `!= 'push'` both admit the baseline lane and both are wrong, and only the `push` half separates them from the intent. Every file under `.github/workflows/` loads under a duplicate-key-rejecting loader, and `ci.yml` carries **16 jobs before and 16 after** with an identical job-name set. PyYAML accepts a duplicate key that GitHub rejects, and a rejected workflow runs zero jobs. The two mutation anchors in `test_release_pipeline.py` that carried the job header and its three comment lines now anchor on the `if:` line, whose two occurrences and CPU-first order the test asserts rather than assumes. The old anchors coupled a mutation suite to prose, so editing a comment reddened it. ## The first baseline after this lands is RED On mudler#584, and that is the correct first verdict — the same shape `baseline-summary`'s own comment already records for the six sanitizer failures of mudler#274. It read GREEN before only because it never ran these jobs. Nothing here blocks a pull request: `baseline-summary` runs only on `schedule`/`workflow_dispatch` and can never fire on a contributor's branch. ## mudler#584 is narrowed, not fixed Recorded under `## Owed` in `.agents/specs/windows-baseline-coverage.md` and in the issue index, because it is a C++ edit in a file this session could not build. `0xC0000409` is **not** evidence of a stack buffer overrun. It is the status `__fastfail` raises for every fail-fast code, so `abort()` — and therefore `std::terminate()` — and the CRT invalid-parameter handler both surface as it, and `__fastfail` bypasses SEH, which is why doctest's Windows handler cannot report it and why the whole doctest output is the version banner. The crash is localised from the log. `LogHttpIngress` (`src/vllm/entrypoints/openai/api_server.cpp:223`) fires for `/v1/chat/completions` and no other route, over `std::cerr` (`request_logger.cpp:26`, unit-buffered, so absence is evidence rather than buffering). `tests/vllm/entrypoints/openai/test_api_server.cpp` reaches that route at exactly eight sites in file order, and the eighth and last logged request — `body_bytes=92 stream=0 max_tokens=4 prompt_chars=5` — is byte-for-byte the 92-byte body posted at `:1291-1294` inside the socket smoke case. The server answered it, then produced nothing for 0.78 s and fast-failed, which places the fault between `:1294` and the `REQUIRE` at `:1325` and excludes the 1.0 s poll loop at `:1323-1324`. Why nobody could read it: fifteen cases hold a joinable `std::thread` across throwing assertions, so any throw in between destroys a joinable thread, `std::terminate()` runs, `abort()` runs, and on MSVC that is `__fastfail` — a named assertion failure becomes an opaque `0xC0000409` with no reporter output. That half is provable by inspection; whether it is what fired here is not, because no output survives to say so. The next step is that RAII change plus a rerun, which either resolves mudler#584 or hands the next session the exact line. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
… graph and advance cur_pos on device (mudler#1105) feat(BACKEND-TENSTORRENT-HOST-FREE-FORWARD): capture the Qwen3 decode graph and advance cur_pos on device (mudler#1105) The Tenstorrent decode path hung after about 38 replays because each step copied cur_pos and update_idxs into the live trace. This change captures the Qwen3 dense decode graph and advances cur_pos on-device with ttnn::plus_one, matching the upstream executor.py pattern. A P150 run of Qwen3-0.6B "Hello" at 80 tokens completed 79 replays with no hang, 5.8x vs eager, and 22/22 argmax vs the per-step-copy baseline. The path is inert unless VT_TT_HOST_FREE_DECODE is set. A new batch size after the first capture is refused rather than freezing cur_pos. CPU and Vulkan builds link via header no-ops. MSVC accepts the MoE refuse path. VT_TT_DUMP_KV is paid only on kTENSTORRENT. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:grok-4.6 [Grok]
…ule anchor upstream actually uses (mudler#1093) (mudler#1156) Closes mudler#1093. Closes mudler#1151. `pipeline_kind = ti2vid_two_stage` resolves `TI2VidTwoStagesPipeline` (`ti2vid_two_stages.py:61` @ `fd4ded7f`) on all four generations the table keys. Stage 1 generates at half resolution under full CFG on the UNADAPTED model, on a schedule derived from the step count; stage 2 upsamples the latent 2x and refines it with the distilled adapter on the frozen three-sigma schedule and no guider. Asking for it used to get the generic table refusal. It is not `distilled_two_stage`, which builds one stage set (`distilled.py:131`), freezes stage 1's sigmas and gives 2.5 the ancestral stepper; and not `res2s_two_stage`, which fuses the adapter on both stages at 0.25/0.5 and runs the second-order sampler. It is three fields from `a2vid_two_stage`: no take is required, the audio guider is the parameter table's row rather than the positive-only default because this pipeline SAMPLES its soundtrack (`:255-258` against `a2vid_two_stage.py:237-239`), and `audio_output_phase` is 0 because upstream discards stage 2's audio in its own words at `:287-289`. `stepper = kEuler` is derived, not assumed: neither `self.stage_1(...)` (`:247`) nor `self.stage_2(...)` (`:289`) passes `stepper` or `loop`, so `DiffusionStage.__call__`'s defaults apply (`utils/blocks.py:524-527`). ## The schedule anchor, and a six-to-one split nobody had counted `LTX2Scheduler.execute` takes an optional latent, and `schedulers.py:31` reads its absence as `default_number_of_tokens` = `MAX_SHIFT_ANCHOR` = 4096. Upstream calls it in seven places and **six pass no latent**; `ti2vid_two_stages_hq.py:267` is the only one that passes one. This engine has always derived from `target_tokens`, so it mirrors the exception and diverges from the rule — on `one_stage` (four version keys), `a2vid_two_stage` stage 1 and `retake` as well as on the arm being added. `Ltx2PhaseRecipe::schedule_tokens` carries the choice. **Its default is today's behaviour, so no landed arm moves.** mudler#1150 owns the flip, because correcting the other three re-samples five shipped, gated arms and rewrites their goldens, which needs its own spec and fresh review. This row sets `kSchedulerDefault` on the one phase it ships. `.agents/specs/ltx25-res2s-loop.md:80-88` saw the HQ/plain split and concluded the divergence was "on the PLAIN two-stage arm and not this row's to move". That is right about HQ and wrong about the blast radius. ## The gate found that two steps cannot see the anchor at all The 2x2 over (recipe, geometry) reports, from the case's own `MESSAGE`: ```text ti2vid: 4096 / 4096 res2s: 2 / 8 ``` So this arm's anchor is resolution-independent and the HQ arm's is not. But the trajectory half of the same case went **RED with those counters already correct**: `stretch` pins sigma[0] at 1.0 and the last non-zero sigma at `terminal` = 0.1 (`schedulers.py:48-55`), so a 2-step schedule is `{1, 0.1, 0}` for **every** token count. The fixture's usual step count is 2. A version of this case asserting only the counters would have been green, correct, and unable to see whether the anchor reached anything. It now renders at 3 steps. This also bounds mudler#1150: a short schedule cannot see the anchor either, which is part of why three shipped arms carried the divergence with every gate green. **The first fix for that was itself vacuous, and the fresh review caught it.** The case pinned the degeneracy by asserting that a 2-step schedule is token-independent — true, load-bearing against a future scheduler change, and silent about this case's own step count, because the trajectory half recomputed at the LITERAL 3 while the render read `gen.steps`. Mutating `gen.steps` 3 -> 2 was NOT DETECTED: 83 cases, 2577 assertions, exit 0. Nothing the case asserted depended on the step count it rendered at, and the only render-derived observable in it is `schedule_tokens`, a COUNT that does not move with steps. The body of this pull request claimed the opposite, which under `squash_merge_commit_message = PR_BODY` would have become the permanent commit message of the row whose headline finding is that a gate can be green, correct and blind. The step count now comes back OUT of the render lambda, in a `Rendered{tokens, steps}` pair, and both recomputations run at it; `rendered_steps > 2` is asserted directly beside the 2-step pin, because the two fail for different reasons. M7 below is that mutation, re-run. ## A refusal that still advertised a closed issue `requires_distilled_lora`'s message told callers this engine fuses once at load so stage 1 sees the adapter. That closed at `4ae0f54ab`, and `LTX25-PHASE-LORA` repaired the reference-conditioning refusal carrying the same claim ~1100 lines away without finding this one or the `ltx2-gen --help` text. The message also hard-coded `a2vid_two_stage.py`'s line numbers while being deliberately keyed on the flag so this row would inherit it, so the first arm to arrive would be named in one sentence and cited to another pipeline's source in the next. It now cites `default_2_stage_arg_parser` (`utils/args.py:1123`, `:1140-1155`), which all of these pipelines select. The a2vid test asserted the stale string was PRESENT and now asserts it is absent. ## Tests Every assertion carries a control drawn from the recipe it would be confused with, so none can pass by two values coinciding. - **The recipe, field by field** — `audio_output_phase` 0 against `a2vid`'s 1, `kEuler` against `distilled`'s ancestral and `res2s`'s `kRes2s`, the audio guider at 7.0 against `a2vid`'s 1.0, `requires_audio_input` FALSE against `a2vid`'s true, plus the four version keys and the plural near-miss. - **The anchor is CONSUMED**, through `LoadVideoEngine` + `Generate`, not read off the struct (mudler#1013). - **The x0 invariant on all four arms** — cond, uncond, perturbed, modality. Guidance combines x0, not velocity; linear terms are invariant and the rescale branch is not, and `rescale_scale` is 0.7 here, so a space error lands on the default path (mudler#1039/mudler#1092). A magnitude assertion cannot gate it; the equation `x0 == latent - sigma*velocity` can, and its RED prints `|x0 - velocity| = 0`. - **`dit_forwards`, not an evaluation count** — a denoiser call is one evaluation either way. - **The `requires_distilled_lora` refusal**, with a control that renders. ### Mutations, four facts each | Mutation | diff --stat | BUILT | compile_err | exit | verdict | |---|---|---|---|---|---| | M1 `schedule_tokens` deleted | 1 file, 1+/1- | yes | 0 | `test_ltx2_video` 1 | DETECTED | | M2 `stage1.loras` deleted | 1 file, 1+/1- | yes | 0 | `test_ltx2_pipeline` 1 | DETECTED | | M3 `audio_output_phase` 0 -> 1 | 1 file, 1+/1- | yes | 0 | `test_ltx2_pipeline` 1 | DETECTED | | M4 `requires_distilled_lora` deleted | 1 file, 1+/1- | yes | 0 | both 1 | DETECTED | | M5 dispatch row deleted (reachability) | 1 file, 1+/1- | yes | 0 | both 1 | DETECTED | | M6 guidance left in VELOCITY space | 1 file, 1+/1- | yes | 0 | `test_ltx2_video` 1, 31 assertions | DETECTED | | M7 test 2's `gen.steps` 3 -> 2 | 1 file, 1+/1- | yes | 0 | `test_ltx2_video` 1, 2 assertions | DETECTED | M7 was NOT DETECTED at `15f0a0638` and is what the repair commit closes. It now reds on both `CHECK( at_anchor != at_target )` — `{1, 0.1, 0} != {1, 0.1, 0}` — and `CHECK( rendered_steps > 2 )` — `2 > 2`, exit 1 captured directly, 83 cases / 82 passed / 1 failed, 2581 assertions / 2 failed. The pre-edit line content was asserted before the mutation and the tree was restored byte-for-byte, with `os.utime` and a rebuild that recompiled the translation unit rather than skipping it. Anchor uniqueness was asserted before each edit, whole binaries were run (a zero-match `-tc` filter prints `SUCCESS!` at exit 0), case and assertion counts were non-zero throughout, and the tree was byte-restored after each. ## Gate ```text CONFIGURE_EXIT=0 BUILD_EXIT=0 ": error:" count=0 ctest -N: 506 CTEST_EXIT=0 100% tests passed, 0 failed out of 506 test_ltx2_pipeline 54 cases / 3182 assertions / exit 0 test_ltx2_video 83 cases / 2581 assertions / exit 0 "No space left"=0, "BFD"=0, both greps proven live against an injected control ``` RED before the recipe landed, on the same binaries: `test_ltx2_pipeline` 54 / 3063, `Status: FAILURE!`, exit 1, both new cases throwing `Unsupported LTX pipeline kind/version: 'ti2vid_two_stage'/'2.5'`; `test_ltx2_video` 82 / 2432, exit 1. ## What is NOT claimed **No real-weights render.** mudler#1148 closed at `40a796aa9` while this row was in flight and this branch merged it, so the bf16 load path is no longer the obstacle and the spec's blocker paragraph was corrected rather than shipped stale. What is owed is a GPU lease and two renders on the same checkpoint, prompt and seed. Substituting a distilled checkpoint would sample a trajectory those weights were never trained for and render plausibly with nothing in the output to show it (mudler#1137), so it was not done. ## What the fresh review corrected The implementation passed: 10/10 anchors exact at the pin, 506/506, six of seven mutations detecting. The seventh is M7 above. Four record corrections rode with it, each verified before it was applied: - The spec's `## Outcome` gate table recorded `test_ltx2_video` at 82 / 2496, wrong on both numbers at every head this row pushed. - Spec Tests §2 said the trajectory half would "require it equal to what the render sampled". `Ltx2ConditioningTrace` exposes no sampled sigmas, so no such comparison exists; it compares two independently recomputed schedules and requires them to DIFFER, and the render-to-trajectory link runs through the counter alone. - `utils/args.py:1140-1153` survived at eight sites against eight carrying `:1140-1155`. Read at the pin `fd4ded7`, the `--distilled-lora` `add_argument` block runs 1140 to 1155 and `:1153` is a help-string line, so every live site is unified on `:1140-1155`. `.agents/issue-index.md` keeps its two, because that file is append-only and is never edited in place. - `## Owed` said "Five assignments" of `allow_request_latents` and also "adds a sixth write". `origin/main` carries four and this branch five. Filed and owed rather than fixed here: **mudler#1150** (the three remaining divergent arms) and **mudler#1152** (`allow_request_latents` is written by every recipe and read by nothing, against a positive control that has a real reader). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Brings the branch onto current main so the ROCm attention registration lands on today's tree. A cherry-pick of the branch tip conflicts in four C++ files because the branch predates main by enough that replaying the diff loses its merge-base context; the merge itself is clean. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
…TTN is for Two landing repairs on the ROCm attention registration. `docs/FEATURES.md` line 266 was 362 characters in one table cell against `check-public-doc-tables`'s 220 limit, which refuses the change outright. The cell keeps what a reader of that table needs -- the M3 claim, both PR numbers and a spec link -- at 208 characters; the long form already lives in `.agents/backend-matrix.md`, which is where that checker's message points. `docs/USAGE.md` gains the paragraph the change actually owes a user. Until this PR the ROCm kernels were registered (`kPagedAttention`, `kReshapeAndCache`) while `RocmPlatform::get_attn_backend_priority` returned an empty list, so `SelectAttentionBackendName` had nothing to resolve for `kROCM` -- the only platform in that state. It now returns upstream's dense order verbatim. The paragraph also says the thing a user most needs to know: nothing routes to the name until the runner asks for it (mudler#1065), and no flag changes, because this is what the engine picks rather than something anyone selects. EXCEPTION, argued rather than waived: `documentation-checkpoint` still refuses commit `3604c0e06` -- the contributor's -- because it changes `include/vllm/` (`USER_USAGE_PREFIXES`) and `.agents/backend-matrix.md` (`FEATURE_SURFACE_FILES`) without touching `docs/USAGE.md` in that same commit. The checker walks commits individually, so a `docs/USAGE.md` edit in a later commit cannot satisfy an earlier one, and the only way to clear it would be to rewrite a contributor's commit content. The repository squash-merges with `squash_merge_commit_message = PR_BODY`, so what lands is one commit carrying both the code and this documentation -- the state the checker is asking for. The per-commit walk is measuring an intermediate that never reaches `main`. This is the `USER_USAGE_FILES` half of the shape recorded as mudler#515; mudler#1086 narrowed the sibling `feature_surface` trigger to a registration-set change, and the `user_usage` path prefix still keys off the path. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
localai-bot
pushed a commit
to tbrasser/vllm.cpp
that referenced
this pull request
Aug 17, 2026
Brings the branch onto current main. Merged rather than cherry-picked for the same reason as mudler#1056: the branch predates main by enough that replaying its diff loses merge-base context, while the merge is clean. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
localai-bot
pushed a commit
to tbrasser/vllm.cpp
that referenced
this pull request
Aug 18, 2026
Picks up mudler#1056, which this PR requires: the runner resolves a backend name, and before mudler#1056 ROCm had none registered. The only conflict is docs/USAGE.md, where both changes add a paragraph — mudler#1056's on what ROCM_ATTN is, this one's on the block-size contract. Both are kept. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
kPagedAttentionandkReshapeAndCachehave been registered forkROCMsincethe kernel fan-out, with an 80 KB
src/vt/rocm/rocm_paged_attn.hipbehind them —but
RocmPlatform::get_attn_backend_priorityreturned an empty list, soSelectAttentionBackendNamehad nothing to resolve. ROCm was the only platformin that state; Metal, Vulkan and Tenstorrent all return
{"FLASH_ATTN"}.This registers
RocmAttentionBackendunder the nameROCM_ATTNforkROCM,following the Metal/Vulkan/Tenstorrent registration idiom, and fills in the
dense, MLA and sparse priority lists from upstream
rocm.pyat the pinnedrevision
555967922. The dense list mirrors upstream verbatim —{ROCM_ATTN, ROCM_AITER_FA, ROCM_AITER_UNIFIED_ATTN, TRITON_ATTN, TURBOQUANT}—because the selection walk skips names that are not registered, so carrying the
AITER entries costs nothing and avoids an inference about which boards gate them.
The KV-layout deviation, recorded rather than glossed
Upstream's
ROCM_ATTNis the K/V-outermost layout:rocm.py:521-522says sooutright. This tree uses NHD, so registering that name against NHD inverts the
name's defining property upstream. The alternative — registering
FLASH_ATTNforkROCM, as Metal, Vulkan and Tenstorrent do — was considered and rejected,because it would leave
ROCM_ATTNpermanently unregistered in the priority list,which is a false claim that ROCm attention is unsupported.
The deviation is therefore carried as one exact tracked exception, recorded in
include/vllm/v1/attention/backend.h, in.agents/specs/rocm-attn-backend.md§3,and referenced from
docs/ROCM.md. It has an explicit expiry: if a realupstream-layout ROCm kernel lands, the registration changes shape and this stops
being an exception.
Upstream also appends
ROCM_ATTNonlyif not use_kv_connector(
rocm.py:429-433), guarding an asymmetric native K/V cache view. That premisedoes not exist here — this registration uses the symmetric NHD layout — and §4 of
the spec records both the reasoning and the condition under which it would stop
holding.
Reachability, stated plainly
At this commit nothing routes to the registered name:
SelectAttentionBackendNamehas no production caller on
main. That arrives with #1065, which is why thesewere split and why #1056 must land first — #1065 alone calls the selector
unconditionally, and on ROCm's empty list
initialize_kv_cachewould throw..agents/specs/rocm-attn-backend.md§7 carries this under## Owedand §8records the reachability state.
Maintainer changes on top
docs/FEATURES.mdline 266 was 362 characters in one cell againstcheck-public-doc-tables's 220 limit, which refuses the change outright. Now208, keeping the M3 claim, both PR numbers and a spec link; the long form
already lives in
.agents/backend-matrix.md.docs/USAGE.mdgains the paragraph this change owes a reader — what wasmissing, what is registered now, and that no flag changes because the engine
picks this rather than the user.
mainrather than cherry-picked: the branch predates mainby enough that replaying its diff conflicts in four C++ files, while the merge
itself is clean.
Issue: #41
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]