diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 234ccfd2d..39e2db87f 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -276,7 +276,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1020](https://github.com/mudler/vllm.cpp/issues/1020) | `SPEC-MTP-K-GT-1` | TWO coupled defects in the Qwen3.5 spec-decode graph layer, which must be repaired in ONE change. **(a)** A speculative verify step whose ACTUAL draft depth differs from the CONFIGURED `num_speculative_tokens` silently falls out of the CUDA graph to eager, with no log and no counter. `src/vllm/v1/worker/gpu/runner.cpp:1383 @ 332aed738` passes `num_spec()` (the configured constant) into the spec-graph gates at `src/vllm/model_executor/models/qwen3_5_moe.cpp:143-148` and `qwen3_5_dense.cpp:172-177`, which admit a batch only when its uniform query length equals `1 + num_spec()` EXACTLY (`cudagraph_dispatch.h:45-46,76-82`). `src/vllm/v1/core/sched/scheduler.cpp:616-622` truncates a request's drafts to the step's token budget, so at k>1 a clamped step can hand every request the same shorter prefix, produce a perfectly uniform batch at a query length no graph is admitted for, and run the verify eager. At k=1 the same truncation is all-or-nothing and the batch merely stops being uniform, so this is reachable in a NEW shape once depth is configurable. The cost is measured, not hypothetical: capturing the T=1+k verify moved the 35B cells 0.870x to 0.986x in `c5615cfe0`. **(b)** The graph slot ring is keyed on `S` ALONE (`qwen3_5.cpp:9276`, dense sibling `:9698`, with `S = B` on a spec step at `:9253`), and the predicate that reaches it is `uniform_decode = input.pure_decode \|\| (spec_graph && ...)`, so BOTH branches share one map and two uniform query lengths already collide on one key: at k=1, 8 requests pure-decode and 4 requests spec both give `S = 8`; at k=3, 2 requests spec gives `S = 8` too. `SizeSlot` invalidates on `fa_cols` and `aux_taps` (`:9280-9330`) and on nothing keyed to spec-versus-pure. This is PRE-EXISTING since SPEC-DSPARK W8 ([#442](https://github.com/mudler/vllm.cpp/issues/442)) made a uniform spec batch capturable at all, and MTP depth does NOT widen it, because `num_spec()` is one value per engine. A first revision of [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) section 4.2a claimed the key does not collide at all; a fresh review disproved it and the claim is withdrawn there. (a) and (b) are one repair because a predicate widened to the step's ACTUAL query length is only unambiguous once the ring distinguishes those query lengths. Also owed: a measured before-and-after on the capture-set size and persistent logits memory, and a counter or log for the eager fallback so it can never again be invisible. NOT fixed in flow: widening the capture set is a persistent-memory decision that has to be measured on a GPU, which was held by another session for the whole flow, and which no CPU gate can observe at all. Found while porting MTP k>1 ([#81](https://github.com/mudler/vllm.cpp/issues/81) M1); listed under `## Owed` in [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) | bug | | [#1027](https://github.com/mudler/vllm.cpp/issues/1027) | `SPEC-MTP-K-GT-1` | DUPLICATE of [#1022](https://github.com/mudler/vllm.cpp/issues/1022), and FIXED on `origin/main` by [#1025](https://github.com/mudler/vllm.cpp/pull/1025) before this row landed. Filed while merging `origin/main` into `SPEC-MTP-K-GT-1` ([#81](https://github.com/mudler/vllm.cpp/issues/81)): `.agents/issue-index.md` listed [#995](https://github.com/mudler/vllm.cpp/issues/995) TWICE on `origin/main` @ `45b022cdc`, because `332aed738` (#996) and `45b022cdc` (#997) each appended a row for the same issue without seeing the other and the path carries `merge=union`, so `check-agent-record` and `test_agent_record` were RED there and on every branch merging it. The filing claimed the file "cannot be made green by any edit to it" because `check-agent-record` refuses the duplicate while `check-issue-index-append-only` refuses the deletion that resolves it. **That premise was WRONG**, and the reason is worth keeping: `check-issue-index-append-only.py:47-56` diffs `merge-base(origin/main, HEAD)..HEAD`, not the file's own history, so the two checkers conflict only while the duplicate sits in the MERGE BASE. #1025 merged the two rows by key ON MAIN, which moved the merge base, and a branch that then takes main's version and appends removes nothing at all. Both checkers are green on one tree with neither weakened. The union driver still produces the WRONG result automatically here: `git merge-tree` against `origin/main` re-added the deleted row and yielded TWO `#995` rows again, which is why the keyed-record rule (take the complete target-branch version, re-apply the scoped edit) applies rather than the driver's output. Owed only a duplicate-close on GitHub, which this flow had no authority for; listed under `## Owed` in [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) | bug | | [#857](https://github.com/mudler/vllm.cpp/issues/857) | `ORACLE-LLAMACPP-REPIN-STOCK` | The recorded llama.cpp pin `237ad9b96` names an object no remote carries: `git branch -r --contains 237ad9b96` is empty and it lives only on local branch `localai-paged` in the developer's checkout. `git describe --tags` returns `b9827-65-g237ad9b96`, so it is 65 of OUR OWN performance commits past upstream tag `b9827`, and the recorded `pin_label = b9892` came from `git rev-list --count`, which returns 9892. That label is not merely derived, it COLLIDES: stock reached tag `b9892` exactly 65 commits after the same base `b9827`, so upstream `b9892` also counts 9892 and resolves to `ee445f93d`, `git merge-base 237ad9b96 b9892` is `b9827`, and neither is an ancestor of the other. A reader checking out `b9892` to reproduce a number silently gets stock. That already happened here: `rpi5-a76-llamacpp-20260806.md` substituted stock `b9892` after finding the pin unobtainable while `cpu-x86-llamacpp-20260811.md` built the fork under the same label. The working tree at the pin also carried 27 uncommitted entries at +2279/-762, so the binaries came from a tree in no repository. `ORACLE-LLAMACPP-REPIN-STOCK` moves the record to stock `b10451` (`10bf611e5`) and drops `gateable` to `no` naming THIS issue, which still owes the build-and-run on dgx.casa that would make it `yes`. Spec [`oracle-llamacpp-repin-stock.md`](specs/oracle-llamacpp-repin-stock.md) | bug | -| [#1003](https://github.com/mudler/vllm.cpp/issues/1003) | `ORACLE-LLAMACPP-REPIN-STOCK` | Every recorded llama.cpp floor number was measured against the local-only fork `237ad9b96` and is owed a re-take against stock `b10451`. The contamination is NOT confined to CUDA, which is what makes it a CPU-oracle defect rather than a filing error: six of the 65 fork commits touch `ggml/src/ggml-cpu/`, and `git diff --numstat b9827 237ad9b96 -- ggml/src/ggml-cpu/` returns `ops.cpp` at +318/-13 and `ggml-cpu.c` at +2/-1, so 320 inserted lines across the two files and +305 net in `ops.cpp` alone, in `ssm_conv_f32`, `gated_delta_net_*`, `flash_attn_ext_f16*` and `scale`, and `570aadd7a` states in its own body that the fused GDN op and the discriminated SSM_CONV decode op are emitted DEFAULT-ON and implemented for the CUDA family "and the CPU reference ONLY". `git grep ssm_conv_update origin/master` and `git grep gated_delta_net_inplace origin/master` both return nothing, and the CPU floor arm built that fork with `GGML_CUDA=OFF` on a `qwen35` model whose CPU graph reaches exactly those ops. THIRTEEN measurements are enumerated with a re-take verdict each in the spec, and the enumeration is the output of a recorded three-stage sweep rather than a hand list, because successive fresh reviews each found a verdict a hand list had missed: the GB10 20-core floor (prefill 1.18x PASS, decode 0.97x, RSS 1.01x), the 2026-07-22 remeasure, the G4/G7 refresh, GDN-orientation, elementwise-GEMM, the threadpool context, Vulkan `BENCH-VK-LLAMA`, the x86 RSS 1.0022x, the Pi 5 arm, the Muse Glimmer and #391 arm, `KERNEL-GEMM-CPU-TILED`, and keep-f16's L7 "RSS gap CLOSED to 1.01x llama.cpp". **The sweep's OWN path set was the fourth miss.** It read `git ls-files docs .agents benchmarks`, so `README.md` at the repository root sat outside the instrument built to stop hand enumeration, and the front page carried the CPU comparison table, the "1.18x llama.cpp's prefill" headline and the "matches llama.cpp" 4.36 vs 4.35 Vulkan claim unmarked, that last one being the most fragile verdict in the whole set. The fix is not to add one path, which is the same hand list one level up: stage 0 is now `git ls-files` with NO path arguments, so a file added anywhere is in scope the day it is committed. 651 files become 4514, candidates 961 become 1008 over 117 files rather than 93, and stage-2 favourable 719 becomes 752, so scanning everything costs 47 lines of adjudication and no exclusion list. A FIFTH hole surfaced with it: `CMP` carried no token for the word this project uses in its own headline, so `README.md:310` "decode **matches llama.cpp Vulkan**" was invisible to both path sets. Adding `match(es|ed)` takes candidates to 1095 over 141 files and favourable to 842. **A SIXTH hole was different in kind: a token that was PRESENT and DEAD.** `CMP` read `[0-9]+(?:\.[0-9]+)? *[x×]\b`, and since U+00D7 is not a `\w` character the `\b` after it can only be satisfied by a FOLLOWING word character, so the `×` alternative fired on `3.9×decode` and on nothing else this tree writes: `1.18× llama.cpp`, `| 1.023× |` and `2× over llama.cpp` all MISSED. The stage-2 filter spells the same idea without the trailing `\b` and was live the whole time, so the two expressions disagreed and only the narrower one gated. Repairing it to `(?:x\b|×)` moves stage 1 from 1118 to 1225 candidates over 141 to 144 files at `85a9a7ae7`, a larger correction than the `match` token. The five earlier holes were all things ABSENT from a list, which a careful reader can find; a dead branch cannot be found by reading, because the instrument prints the same clean output whether the branch is unreachable or the tree is clean. The sweep therefore now SELF-TESTS against known-positive and known-negative strings before it scans, and that assertion is armed rather than decorative: run against the defective expression it replaces, it fails on exactly those three shapes. The twelfth measurement is the one that matters most, because it is the only place the contamination reaches shipped behaviour rather than a document: keep-f16's llama.cpp denominators (pp128 173.2, peak RSS 2.798 GiB) are quoted in `src/vllm/model_executor/model_loader/gguf_keep_quant.cpp:173-228` as the justification for `VT_GGUF_KEEP_F16` shipping DEFAULT ON, and restated beside a `CHECK` in `tests/vllm/test_gguf_keep_quant.cpp:478-494`. **That default is owed a DECISION, not only a re-wording, and an earlier pass of this row recorded the opposite.** It said the default stands because its acceptance is a same-binary ours-versus-ours A/B (3.885 to 2.832 GiB, tokens byte-identical) that no denominator move touches. That reads one row of a three-row table. The binding L7 A/B (`gguf-keep-quant-loader.md:587-590`) has three axes and TWO regress: prefill about 10% worse (224 to 204 t/s, TTFT 571 to 625 ms) and decode about 1.4% worse (TPOT 40.4 to 40.95 ms), bought for 1.05 GiB of peak RSS. The recorded reason the PREFILL loss is acceptable is stated at `:595` in the contaminated denominator's own terms: 204 t/s is "about 9% under the keep-f16-off default's 224 t/s but comfortably above the competitor floor", and that floor IS the fork's pp128 173.2. Since `b10451` is 624 commits past `b9827` and carries its own `fused_gdn`, the direction is NOT established, so if a re-taken stock pp128 lands above 204 t/s the "comfortably above" clause fails and this default's only recorded justification for its prefill regression is gone. The RSS leg would still stand alone and may well suffice, but it would then be a live call rather than a settled one. Re-taking row 12 therefore owes a keep-f16 verdict to `QUANT-GGUF-KEEPQ-LOADER`, and the repin row deliberately does NOT change the default, having measured nothing. The product comment additionally quoted "205 t/s = 1.16x AHEAD of pp128 176.6", and NO recorded run produces either operand: `git grep 176.6` returns only an unrelated MoE microbenchmark mean in MICROSECONDS, while the binding record says 204 against 173.2 = 1.18x. That pair is left unreconciled rather than guessed, because picking one would assert an attribution nobody measured, and #1003 owes one re-measured pair to replace both. FIVE llama.cpp revisions are in play, not two and not the four an earlier pass recorded. They are the fork `237ad9b96`, stock `b9892` on the Pi 5 arm, stock `704485942` at `b10362-5` on Muse Glimmer and #391 (which appeared nowhere in `docs/` or the spec before the sweep), stock `030ebb5` at `b10358` carrying one explicitly NON-BINDING datapoint, and **a Poolside FORK on branch `laguna` with NO commit SHA recorded anywhere in this tree** (`github.com/poolsideai/llama.cpp@laguna`, named at `laguna-s21-w4-2026-07-31.md:65`). That fifth one is measurement THIRTEEN, the Laguna-S-2.1 `27.8 tok/s` on the identical UD-Q4_K_XL GGUF, and the `15x` warm / `18x` cold gap plus the `18x` to `4.7x` and `18x` to `3.6x` W8/W9 claims all derive from it. It was hidden behind the dead `×` branch for five drafts. A branch name is a moving reference, not a revision, so unlike the other four it was never reproducible and cannot simply be re-run: re-taking it means CHOOSING a revision first (stock `b10451` if it now carries `laguna`, or a named Poolside commit) and recording it as a sixth entry. No verdict there is favourable to us, so the count of seven favourable verdicts is unchanged, but it is the target the whole Laguna speed campaign ranks its levers against. Direction: against `b9827` the fork can only be at or above stock, so our recorded deficits are upper bounds and our wins are lower bounds. Against `b10451` the direction is NOT established, because 624 stock commits landed after `b9827` and upstream now carries its own `fused_gdn`. That window is 624 rather than the 122 an earlier draft recorded: the developer's llama.cpp clone is SHALLOW, grafted at `687e77892`, so every distance and ancestry answer crossing the graft is truncated and looks correct. That graft also invalidated this row's own headline evidence. "The pin is on no remote" rested on `git branch -r --contains 237ad9b96` returning empty, and a control disproves the instrument rather than merely doubting it: `git branch -r --contains b9827`, for an upstream release tag that is beyond argument an ancestor of `origin/master`, lists 70 remote branches today and omits `origin/master` (that total reads a clone outside our control and was 68 when first written; the OMISSION is the load-bearing half and reproduces exactly). The finding survives on evidence a graft cannot corrupt: `gh api repos/ggml-org/llama.cpp/commits/237ad9b96` and the same call against `mudler/llama.cpp` both return HTTP 422 "No commit found for SHA", with `gh api repos/ggml-org/llama.cpp/commits/10bf611e5` resolving on the identical call shape as the positive control, and `gh api repos/ggml-org/llama.cpp/git/ref/tags/b10451` binding the new label to the new commit ON the remote, which is the check the old `pin_label = b9892` would have failed. SEVEN recorded verdicts can flip unfavourably, five of them on the public page and one of them holding up a shipped default, and the GB10 prefill 1.18x is NOT the most exposed: ordered by fragility they are the Vulkan `BENCH-VK-LLAMA` decode 4.36 vs 4.35 `MET` (a 0.23% margin inside its own 0.69% 7-leg spread, whose source calls it "a narrow pass, not a comfortable one"), the Muse Glimmer in128 prefill 1.023x (a 2.3% margin inside our own arm's 4.5% leg spread over n=4), the GB10 peak RSS 1.01x PARITY and decode 0.97x tie (ties by declaration, so any denominator movement makes them gaps), keep-f16's "RSS gap CLOSED to 1.01x" with "prefill 1.18x AHEAD" (the same 2.798 GiB denominator, and the one verdict a user's bytes depend on), `KERNEL-GEMM-CPU-TILED` "at parity with ggml's stock kernel, ahead on 4 of 6 shapes" (overlapping bands, one shape already behind), the GB10 prefill 1.18x PASS, then the Pi 5 peak RSS 2.841 vs 3.747 GiB at 0.758x, whose denominator was already stock `b9892` so its only exposure is `b9892` to `b10451` drift. Re-take all seven, not prefill alone. Also owes the re-anchoring of the source citations. The spec records the three commands that count them rather than a bare number, because the number moved inside this pull request already. At `bf621287a` they return 112 files mentioning the SHA, 69 attributing a path or a tree to it in the `@ ` form, and 52 carrying a `path.ext:LINE` anchor within three lines. The first is now 113 because `README.md` finally names the fork its front-page numbers were measured against. No reviewer can fetch that object. Listed under `## Owed` in [`oracle-llamacpp-repin-stock.md`](specs/oracle-llamacpp-repin-stock.md) | perf | +| [#1003](https://github.com/mudler/vllm.cpp/issues/1003) | `ORACLE-LLAMACPP-REPIN-STOCK` | Every recorded llama.cpp floor number was measured against the local-only fork `237ad9b96` and is owed a re-take against stock `b10451`. The contamination is NOT confined to CUDA, which is what makes it a CPU-oracle defect rather than a filing error: six of the 65 fork commits touch `ggml/src/ggml-cpu/`, and `git diff --numstat b9827 237ad9b96 -- ggml/src/ggml-cpu/` returns `ops.cpp` at +318/-13 and `ggml-cpu.c` at +2/-1, so 320 inserted lines across the two files and +305 net in `ops.cpp` alone, in `ssm_conv_f32`, `gated_delta_net_*`, `flash_attn_ext_f16*` and `scale`, and `570aadd7a` states in its own body that the fused GDN op and the discriminated SSM_CONV decode op are emitted DEFAULT-ON and implemented for the CUDA family "and the CPU reference ONLY". `git grep ssm_conv_update origin/master` and `git grep gated_delta_net_inplace origin/master` both return nothing, and the CPU floor arm built that fork with `GGML_CUDA=OFF` on a `qwen35` model whose CPU graph reaches exactly those ops. THIRTEEN measurements are enumerated with a re-take verdict each in the spec, and the enumeration is the output of a recorded three-stage sweep rather than a hand list, because successive fresh reviews each found a verdict a hand list had missed: the GB10 20-core floor (prefill 1.18x PASS, decode 0.97x, RSS 1.01x), the 2026-07-22 remeasure, the G4/G7 refresh, GDN-orientation, elementwise-GEMM, the threadpool context, Vulkan `BENCH-VK-LLAMA`, the x86 RSS 1.0022x, the Pi 5 arm, the Muse Glimmer and #391 arm, `KERNEL-GEMM-CPU-TILED`, and keep-f16's L7 "RSS gap CLOSED to 1.01x llama.cpp". **The sweep's OWN path set was the fourth miss.** It read `git ls-files docs .agents benchmarks`, so `README.md` at the repository root sat outside the instrument built to stop hand enumeration, and the front page carried the CPU comparison table, the "1.18x llama.cpp's prefill" headline and the "matches llama.cpp" 4.36 vs 4.35 Vulkan claim unmarked, that last one being the most fragile verdict in the whole set. The fix is not to add one path, which is the same hand list one level up: stage 0 is now `git ls-files` with NO path arguments, so a file added anywhere is in scope the day it is committed. 651 files become 4514, candidates 961 become 1008 over 117 files rather than 93, and stage-2 favourable 719 becomes 752, so scanning everything costs 47 lines of adjudication and no exclusion list. A FIFTH hole surfaced with it: `CMP` carried no token for the word this project uses in its own headline, so `README.md:310` "decode **matches llama.cpp Vulkan**" was invisible to both path sets. Adding `match(es\|ed)` takes candidates to 1095 over 141 files and favourable to 842. **A SIXTH hole was different in kind: a token that was PRESENT and DEAD.** `CMP` read `[0-9]+(?:\.[0-9]+)? *[x×]\b`, and since U+00D7 is not a `\w` character the `\b` after it can only be satisfied by a FOLLOWING word character, so the `×` alternative fired on `3.9×decode` and on nothing else this tree writes: `1.18× llama.cpp`, `\| 1.023× \|` and `2× over llama.cpp` all MISSED. The stage-2 filter spells the same idea without the trailing `\b` and was live the whole time, so the two expressions disagreed and only the narrower one gated. Repairing it to `(?:x\b\|×)` moves stage 1 from 1118 to 1225 candidates over 141 to 144 files at `85a9a7ae7`, a larger correction than the `match` token. The five earlier holes were all things ABSENT from a list, which a careful reader can find; a dead branch cannot be found by reading, because the instrument prints the same clean output whether the branch is unreachable or the tree is clean. The sweep therefore now SELF-TESTS against known-positive and known-negative strings before it scans, and that assertion is armed rather than decorative: run against the defective expression it replaces, it fails on exactly those three shapes. The twelfth measurement is the one that matters most, because it is the only place the contamination reaches shipped behaviour rather than a document: keep-f16's llama.cpp denominators (pp128 173.2, peak RSS 2.798 GiB) are quoted in `src/vllm/model_executor/model_loader/gguf_keep_quant.cpp:173-228` as the justification for `VT_GGUF_KEEP_F16` shipping DEFAULT ON, and restated beside a `CHECK` in `tests/vllm/test_gguf_keep_quant.cpp:478-494`. **That default is owed a DECISION, not only a re-wording, and an earlier pass of this row recorded the opposite.** It said the default stands because its acceptance is a same-binary ours-versus-ours A/B (3.885 to 2.832 GiB, tokens byte-identical) that no denominator move touches. That reads one row of a three-row table. The binding L7 A/B (`gguf-keep-quant-loader.md:587-590`) has three axes and TWO regress: prefill about 10% worse (224 to 204 t/s, TTFT 571 to 625 ms) and decode about 1.4% worse (TPOT 40.4 to 40.95 ms), bought for 1.05 GiB of peak RSS. The recorded reason the PREFILL loss is acceptable is stated at `:595` in the contaminated denominator's own terms: 204 t/s is "about 9% under the keep-f16-off default's 224 t/s but comfortably above the competitor floor", and that floor IS the fork's pp128 173.2. Since `b10451` is 624 commits past `b9827` and carries its own `fused_gdn`, the direction is NOT established, so if a re-taken stock pp128 lands above 204 t/s the "comfortably above" clause fails and this default's only recorded justification for its prefill regression is gone. The RSS leg would still stand alone and may well suffice, but it would then be a live call rather than a settled one. Re-taking row 12 therefore owes a keep-f16 verdict to `QUANT-GGUF-KEEPQ-LOADER`, and the repin row deliberately does NOT change the default, having measured nothing. The product comment additionally quoted "205 t/s = 1.16x AHEAD of pp128 176.6", and NO recorded run produces either operand: `git grep 176.6` returns only an unrelated MoE microbenchmark mean in MICROSECONDS, while the binding record says 204 against 173.2 = 1.18x. That pair is left unreconciled rather than guessed, because picking one would assert an attribution nobody measured, and #1003 owes one re-measured pair to replace both. FIVE llama.cpp revisions are in play, not two and not the four an earlier pass recorded. They are the fork `237ad9b96`, stock `b9892` on the Pi 5 arm, stock `704485942` at `b10362-5` on Muse Glimmer and #391 (which appeared nowhere in `docs/` or the spec before the sweep), stock `030ebb5` at `b10358` carrying one explicitly NON-BINDING datapoint, and **a Poolside FORK on branch `laguna` with NO commit SHA recorded anywhere in this tree** (`github.com/poolsideai/llama.cpp@laguna`, named at `laguna-s21-w4-2026-07-31.md:65`). That fifth one is measurement THIRTEEN, the Laguna-S-2.1 `27.8 tok/s` on the identical UD-Q4_K_XL GGUF, and the `15x` warm / `18x` cold gap plus the `18x` to `4.7x` and `18x` to `3.6x` W8/W9 claims all derive from it. It was hidden behind the dead `×` branch for five drafts. A branch name is a moving reference, not a revision, so unlike the other four it was never reproducible and cannot simply be re-run: re-taking it means CHOOSING a revision first (stock `b10451` if it now carries `laguna`, or a named Poolside commit) and recording it as a sixth entry. No verdict there is favourable to us, so the count of seven favourable verdicts is unchanged, but it is the target the whole Laguna speed campaign ranks its levers against. Direction: against `b9827` the fork can only be at or above stock, so our recorded deficits are upper bounds and our wins are lower bounds. Against `b10451` the direction is NOT established, because 624 stock commits landed after `b9827` and upstream now carries its own `fused_gdn`. That window is 624 rather than the 122 an earlier draft recorded: the developer's llama.cpp clone is SHALLOW, grafted at `687e77892`, so every distance and ancestry answer crossing the graft is truncated and looks correct. That graft also invalidated this row's own headline evidence. "The pin is on no remote" rested on `git branch -r --contains 237ad9b96` returning empty, and a control disproves the instrument rather than merely doubting it: `git branch -r --contains b9827`, for an upstream release tag that is beyond argument an ancestor of `origin/master`, lists 70 remote branches today and omits `origin/master` (that total reads a clone outside our control and was 68 when first written; the OMISSION is the load-bearing half and reproduces exactly). The finding survives on evidence a graft cannot corrupt: `gh api repos/ggml-org/llama.cpp/commits/237ad9b96` and the same call against `mudler/llama.cpp` both return HTTP 422 "No commit found for SHA", with `gh api repos/ggml-org/llama.cpp/commits/10bf611e5` resolving on the identical call shape as the positive control, and `gh api repos/ggml-org/llama.cpp/git/ref/tags/b10451` binding the new label to the new commit ON the remote, which is the check the old `pin_label = b9892` would have failed. SEVEN recorded verdicts can flip unfavourably, five of them on the public page and one of them holding up a shipped default, and the GB10 prefill 1.18x is NOT the most exposed: ordered by fragility they are the Vulkan `BENCH-VK-LLAMA` decode 4.36 vs 4.35 `MET` (a 0.23% margin inside its own 0.69% 7-leg spread, whose source calls it "a narrow pass, not a comfortable one"), the Muse Glimmer in128 prefill 1.023x (a 2.3% margin inside our own arm's 4.5% leg spread over n=4), the GB10 peak RSS 1.01x PARITY and decode 0.97x tie (ties by declaration, so any denominator movement makes them gaps), keep-f16's "RSS gap CLOSED to 1.01x" with "prefill 1.18x AHEAD" (the same 2.798 GiB denominator, and the one verdict a user's bytes depend on), `KERNEL-GEMM-CPU-TILED` "at parity with ggml's stock kernel, ahead on 4 of 6 shapes" (overlapping bands, one shape already behind), the GB10 prefill 1.18x PASS, then the Pi 5 peak RSS 2.841 vs 3.747 GiB at 0.758x, whose denominator was already stock `b9892` so its only exposure is `b9892` to `b10451` drift. Re-take all seven, not prefill alone. Also owes the re-anchoring of the source citations. The spec records the three commands that count them rather than a bare number, because the number moved inside this pull request already. At `bf621287a` they return 112 files mentioning the SHA, 69 attributing a path or a tree to it in the `@ ` form, and 52 carrying a `path.ext:LINE` anchor within three lines. The first is now 113 because `README.md` finally names the fork its front-page numbers were measured against. No reviewer can fetch that object. Listed under `## Owed` in [`oracle-llamacpp-repin-stock.md`](specs/oracle-llamacpp-repin-stock.md) | perf | | [#1006](https://github.com/mudler/vllm.cpp/issues/1006) | `LTX25-DECODE-SPEED` | LTX-2.5 render speed has never been attributed on any axis: `docs/BENCHMARKS.md` carries one LTX line, under `## Open gaps`. The shipped video VAE decode is the **CPU reference arm** and production executes it — `src/vllm/multimodal/ltx2_video.cpp:3258 @ 332aed738` calls `Ltx2VideoDecodeStreaming`, reaching `Ltx2ConvVideoDecode` via `ltx2_video_vae_tiled.cpp:113,369 @ 332aed738` — while the file itself says `src/vllm/model_executor/models/ltx2_video_vae.cpp:46-49 @ 332aed738` "no memory or throughput number should be taken from it". One 448x256/25f decode is ~7.25 TFLOP over 42 convs (COMPUTED from the LTX-2.5 conv VAE config in the checkpoint header) against a measured wall of 2681.02 s — ~2.7 GFLOP/s. **That wall is ONE sample on a CONTENDED box** and carries its conditions: its own source records a full 405-target build and a 416-test `ctest -j 6` running in the same window ([`ltx25-tiled-decode.md`](specs/ltx25-tiled-decode.md):377-380) at `uptime` load average 30.2 (`:454`), with no idle-host same-binary repeat. It is also pre-[#1008](https://github.com/mudler/vllm.cpp/issues/1008): the f64 accumulation it measured landed out at `d1b0ea3a8`, and the f32 arm has never been timed. Owning row for the ranked levers in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | feature | | [#1007](https://github.com/mudler/vllm.cpp/issues/1007) | — | The LTX-2.5 video VAE decode has **no device arm at all**: `vt::OpId::kLtx2` (`include/vllm/model_executor/models/ltx2_kernels.h @ 332aed738`, `src/vt/cuda/cuda_ltx2.cu @ 332aed738`) is the DiT device-forward glue — seven ops, no convolution — and nothing the decode reaches. Not an unwired path; the arm does not exist. Every oracle runs this decode GPU-resident and decides placement at build time: Lightricks `packages/ltx-pipelines/src/ltx_pipelines/utils/blocks.py:1139` + `packages/ltx-core/src/ltx_core/loader/single_gpu_model_builder.py:273 @ fd4ded7f2`, SGLang `python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/ltx_2/decoding_av.py:71 @ f63458b5b`, vLLM-Omni `vllm_omni/diffusion/models/interface.py:92 @ a4ea67a21` ("VAE(s) (always on GPU)"), diffusers `src/diffusers/models/autoencoders/ltx2_diffusion_decoder.py:208-209 @ c6da9936e` ("No CPU path", which is scoped to ONE of that file's two processors, `LTX2VideoVaeNeighborhoodNattenProcessor` at `:203`, in the DIFFUSION decoder this port refuses by name; the sibling `LTX2VideoVaeNeighborhoodAttnProcessor` at `:153` is "Portable ... Runs anywhere the flex attention path runs", so this citation narrows to NATTEN and the other three carry the claim) — cited at the revision [`oracles/diffusers.md`](oracles/diffusers.md) PINS, not at the SHA the local checkout sits on. Lever 1, ranked first on magnitude and last on cost. Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | feature | | [#1008](https://github.com/mudler/vllm.cpp/issues/1008) | `LTX25-DECODE-DTYPE` | **FIXED, and this row records it as fixed rather than as owed.** The LTX-2.5 conv video VAE decode accumulated in **f64** at 8 sites (`src/vllm/model_executor/models/ltx2_video_vae.cpp:165,201,303,312,546,570,579,916 @ 332aed738`, with `static_cast` on **29 lines / 30 occurrences** — `:287` carries two, so the line count and the occurrence count are different numbers and both are stated) and stored **f32 NCDHW** (`Volume::At`, `:73-75 @ 332aed738`). No oracle accumulates in f64 anywhere on this path — Lightricks `model/video_vae/conv_video_decoder.py:282-284` and `model/common/normalization.py:32-40 @ fd4ded7f2`, diffusers `src/diffusers/models/autoencoders/autoencoder_kl_ltx2.py:50-59 @ c6da9936e`, SGLang `python/sglang/multimodal_gen/configs/pipeline_configs/ltx_2.py:189 @ f63458b5b` (bf16, deliberately overriding an fp32 base default) — and upstream's default-on fast path is `channels_last_3d` for weights AND activations (`memory_efficient_decode.py:617-627,655-656 @ fd4ded7f2`). No EXISTING gate could see it: the golden generator casts every upstream parameter to f32 (`scripts/gen-ltx2-vae-goldens.py:223`), so the oracle itself ran f32 (`ltx2_video_vae.cpp:41-44 @ 332aed738`). **Closed by `LTX25-DECODE-DTYPE` ([spec](specs/ltx25-decode-dtype.md)), merged as `d1b0ea3a8` via [PR #1036](https://github.com/mudler/vllm.cpp/pull/1036)**, which took the DTYPE half only: at that commit `double acc` is **0** and `static_cast` is **6 lines / 7 occurrences**, all annotated scalar-constant exceptions. It shipped its own instrument — a separable-reduction width case entering through the production `Ltx2VideoDecodeStreaming` (`tests/vllm/models/test_ltx2_vae.cpp`) — and it had to change its DESIGN to keep the numerics: naive serial f32 pushed `test_ltx2_tiling`'s non-causal untiled control to 5.00679e-06 against a 5e-06 tolerance, repaired by per-input-channel BLOCKED summation, which is what torch's f32 convolution does, with no tolerance widened. **Still owed there, not here:** NDHWC / `channels_last_3d` (that spec §5 — a shared helper `minimax_h3.h:756` hard-codes NCDHW across three models), bf16 storage, and a width gate for nine narrowed sites including `Linear3d`, which was widened back to `double` with 40/40 cases still passing. The speed magnitude of the change is **UNMEASURED**: `dgx.casa` was unreachable for that row's whole duration. Filed by [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) as lever 2 and deliberately NOT listed under its `## Owed`, because a closed issue owed by a row that did not close it misattributes the work | feature | @@ -305,20 +305,20 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#960](https://github.com/mudler/vllm.cpp/issues/960) | `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | `vt::QuantFp8Static`'s ONLY CUDA registration lived at `src/vt/cuda/cuda_matmul_fp8_cutlass.cu:376` (@ `0e1bee42f`), and `CMakeLists.txt:1668` compiles that translation unit only when `VT_CUTLASS_FP8_ARCHS` is non-empty — yet the kernel body has ZERO cutlass tokens (`:353-370`): it is `out[i] = e4m3(x[i] * (1/input_scale))`, a grid-stride elementwise convert. So on every CUDA arch outside the cutlass-fp8 cell — sm_110/Thor is the measured one, and `cutlass-fp8: DISABLED for [110]` is that arch's DOCUMENTED NORMAL PROFILE, not a misconfiguration — `OpId::kQuantFp8Static` was not registered for `DeviceType::kCUDA` at all. Nothing refused first: the GEMM partner `kMatmulFp8CublasLt` IS registered unconditionally (`src/vt/cuda/cuda_matmul.cu:920`), so `MatmulFp8CutlassD`'s guard passed, and the missing quant then resolved through `src/vt/op_provider.cpp:501` to the portable CPU reference tier — eligible because `CudaBackend::UnifiedMemory()` is true — which dereferenced DEVICE pointers on the host and SIGSEGV'd one call later under a banner reading "correct but slow". Fixed by relocating the registration to a new unconditionally-compiled TU `src/vt/cuda/cuda_quant_fp8.cu`, which restores upstream's own partition (vLLM builds `static_scaled_fp8_quant` from the unconditional `VLLM_EXT_SRC` list and gates only its cutlass `scaled_mm` sources). This removes one live INSTANCE of [#844](https://github.com/mudler/vllm.cpp/issues/844) and does not address its class, which stays open. Unblocks the FP8 W8A8 arm on every non-cutlass CUDA arch — the base [#810](https://github.com/mudler/vllm.cpp/issues/810)/[#517](https://github.com/mudler/vllm.cpp/issues/517) A2-Q1 needs, where 46 FP8 mamba projections are 36.6% of decode bytes. Spec [`vt-fp8-quant-arch-gate.md`](specs/vt-fp8-quant-arch-gate.md) | bug | | [#989](https://github.com/mudler/vllm.cpp/issues/989) | `VT-FP8-QUANT-ARCH-GATE` | `scripts/check-pr-size.py`'s `classify_path` has no entry for `.agents/reachability.md` (added by `POLICY-NOTHING-LANDS-DEAD`, [#888](https://github.com/mudler/vllm.cpp/issues/888) @ `8f49ac3be`), and it FAILS CLOSED, so `pr-size` — a REQUIRED check — refuses every pull request that touches that guide, and `tests/scripts/test_check_pr_size.py` has been red on `main` ever since. Red SILENTLY: that suite is wired into no CI job and is not in `agent-preflight.sh`'s `SUITES`, so the only thing that ever loads it is `check-pr-size`'s own executable-evidence contract, which fires only when a PR edits a checker — the red is reachable exclusively by the next person who must touch that file, and presents to them as their own breakage (the [#584](https://github.com/mudler/vllm.cpp/issues/584)/[#965](https://github.com/mudler/vllm.cpp/issues/965) shape). Third instance of the class after [#856](https://github.com/mudler/vllm.cpp/issues/856) (`issue-index.md` + the style guides) and [#668](https://github.com/mudler/vllm.cpp/issues/668) (`.agents/oracles/*`), both fixed in flow by the row that tripped over them. FIXED IN FLOW while landing [#960](https://github.com/mudler/vllm.cpp/issues/960), which could not register its new checker's creation mutation without touching `check-pr-size.py` at all. NOT fixed: wiring that suite into CI, which is its own change and would red `main` until this landed | bug | | [#1066](https://github.com/mudler/vllm.cpp/issues/1066) | `ENG-EXPERT-STREAM` | `Qwen35ExpertStream` (`src/vllm/model_executor/models/qwen3_5.cpp`) is a **process-lifetime singleton** and keyed its slot cache on `(TowerId(base), expert)`, where `base` is the expert tower's host buffer **ADDRESS**. Its own comment stated the premise and drew the wrong conclusion: "A tower's identity is its base pointer, which is stable for the model's life". The premise is true; the conclusion does not follow, because the CACHE is not scoped to one model's life. Free a model, load another, and the allocator hands the new towers addresses the old ones held, so the new model's expert resolves to an entry filled from a DIFFERENT checkpoint — returned as a HIT, which by contract moves no bytes, so no counter moves and nothing downstream has anything to observe. MEASURED on two synthetic 4-layer/4-expert MoE models in one process, instrumenting `KqExpertSlice` to `memcmp` each returned slot against the tower slice it claims to be: **24 towers occupied 21 distinct addresses, and 20 of 222 slices returned another tower's bytes**; end to end the two arms disagreed on all 160 logits while each arm was internally deterministic (0 differing values on a repeat), which rules out nondeterminism. Invisible to every existing test of this row by construction, because all of them build the cache, store and streamer by hand and none runs two models through the production seam. Reachable by any process that loads a model, releases it, and loads another. Fixed by `OwnedTensor::TowerUid()`, a lazily assigned process-unique counter stamped on the tensor and re-stamped when `bytes` moves (so a copy cannot inherit an identity along with a different buffer); a counter cannot collide because it never goes backwards. Found and fixed while repairing the F1-F11 wiring review for [#912](https://github.com/mudler/vllm.cpp/issues/912). Spec [`expert-streaming.md`](specs/expert-streaming.md) | bug | -| [#1083](https://github.com/mudler/vllm.cpp/issues/1083) | `MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation` | `CleanCaption`'s italic unwrap emulated upstream's `(?!\*)` with a CAPTURED `($|[^*])` (`src/vllm/model_executor/models/minimax_music3_ar.cpp:85,114` @ `a332fb98d`), and a captured group is not a zero-width assertion: consuming the trailing neighbour advanced `regex_replace` past it, so an emphasis span opening within ONE character of the previous close was never examined and the surviving asterisks re-paired ACROSS the intended spans. `*a* *b* *c*` -> `a *b c*` where `_clean_caption` (`encoders.py:72` @ diffusers `c6da9936`) gives `a b c`, and `Warm *lo-fi* *jazzy* keys with a *soft* *brushed* snare` -> `Warm lo-fi *jazzy keys with a soft brushed* snare` — a re-association, not a leftover marker, so the caption handed to the tokenizer is a string upstream would never emit. `encoders.py`'s own header states that whitespace-level prompt changes change the generated audio, so this is a checkpoint-contract break. Invisible to the gate because `markdown_and_tags`, the only golden with italics, carries ONE span per line and the defect needs adjacency. FIXED by porting the trailing side LITERALLY as `(?!\*)` — std::regex's ECMAScript grammar has negative lookahead though not lookbehind, so only the leading `(? `a *b c*` where `_clean_caption` (`encoders.py:72` @ diffusers `c6da9936`) gives `a b c`, and `Warm *lo-fi* *jazzy* keys with a *soft* *brushed* snare` -> `Warm lo-fi *jazzy keys with a soft brushed* snare` — a re-association, not a leftover marker, so the caption handed to the tokenizer is a string upstream would never emit. `encoders.py`'s own header states that whitespace-level prompt changes change the generated audio, so this is a checkpoint-contract break. Invisible to the gate because `markdown_and_tags`, the only golden with italics, carries ONE span per line and the defect needs adjacency. FIXED by porting the trailing side LITERALLY as `(?!\*)` — std::regex's ECMAScript grammar has negative lookahead though not lookbehind, so only the leading `(? 0)` in `tests/vllm/model_executor/test_expert_stream_wiring.cpp` was not load-bearing: reinjecting the pre-fix unaligned `madvise` address exits 0 in 40 of 40 runs, because `> 0` over 48 calls is satisfied whenever heap layout page-aligns a single slice — measured `advised=1` against `fills=48`. Tightened to `advised == fills`, which is the true healthy invariant on this arm (madvise runs on the mapping-copy path only, and only on a non-resident key, which is exactly when `EnsureSpan` goes on to fill) and which was verified stable over 50 consecutive runs. **(3)** "Every MoE entry point funnels through here exactly once per forward" was false: `Qwen3_5Model::ForwardDense`, `Qwen3_5MTPModel::Forward`, `Qwen3_5MTPModel::ForwardPaged` and `Qwen3_5ReplayLayer` all reach `ExpertMlpKq -> KqExpertSlice` and marked no step. ONE of them, `Qwen3_5MTPModel::ForwardPaged`, is the production spec-decode DRAFT forward, so draft acquisitions stayed `protected_this_step` across the following target forward; the other three are parity-only entry points and their guards land unreached, which is [#1108](https://github.com/mudler/vllm.cpp/issues/1108). All four now carry the guard — one forward is one step, including a draft, because a draft is a complete forward whose slices are finished with when it returns — and the guard refuses to NEST, so adding one cannot double-advance the hotness clock. `RunMoeBlock` stays deliberately unguarded: it is one block, and qwen3_moe.cpp owns the boundary for the model that composes it. **(4)** `ExpertStreamer::EnsureFile`, the arm every real GGUF-mmap checkpoint takes, was reached by no test, so the `file_offset + offset` composition was unverified; a CPU-local case now drives it from a temp file at a deliberately awkward offset and proves the arm by `advised` staying flat while `fills` grows. **(5)** `OwnedTensor::TowerUid`'s comment claimed identity for "this tensor's CURRENT bytes" while the code keys on `bytes.data()`; the comment now states the address limit and a borrowed-buffer case pins both halves, because [#1066](https://github.com/mudler/vllm.cpp/issues/1066) was that same overclaim on that same field. **(6)** `SetForceFallback` has no production caller and incremented the operator-facing `exhausted_`, telling an operator to raise a budget that was never the reason (measured `exhausted=42` from the test switch alone); it now has its own counter. All six fixed in flow, each mutation-proven: 13 mutations, 13 caught, every one with a non-empty `git diff --stat`, a zero compile status and a non-zero doctest case count. Spec [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1073](https://github.com/mudler/vllm.cpp/issues/1073) | `FIX-NAS-PATH-1073` | The NAS moved to `/usr/local/nas_share` and `/mnt/nas_share` is gone, so every tracked default built on `/mnt` broke. `/mnt` is the EPHEMERAL root overlay of the gate box's immutable Kairos OS and does not survive a reboot; `/usr/local` is `COS_PERSISTENT` and does. Observed 2026-08-16 after an 8 h 19 min outage: the mount came back because the `/oem` boot-stage unit worked, `/mnt/nas_share` did not, and the untracked `.env` still declared `CHECKPOINT_ROOT=/mnt/nas_share/checkpoints` — a gate that reads a path `.env` does not declare is not the gate its spec names. `.agents/environment.md` documented NO NAS location at all (measured: the file held no `/mnt` string), so the repair adds the path AND the `COS_PERSISTENT` reason, because a bare path correction invites the next reader to restore the dead location as a symlink. The seven live defaults now derive from `CHECKPOINT_ROOT`, which four sibling scripts already did: `scripts/gen-minimax-music3-manifest.py:17`, `scripts/gen-ltx2-quant-goldens.py:48`, `tools/parity/dump_tokenizer_gpt4o.py:36,39,57`, `tools/gen_pretok_goldens.py:57`, `src/vllm/tokenizer/pretokenizer.cpp:319`, `tests/parity/test_minimax_music3_quant_real.cpp:133,144` and `docs/USAGE.md:3069,3453`. The 41 hits were classified before any edit and the records that cite the old path KEEP it: `.agents/benchmark-record.md`, the LTX-2.5/Nemotron-H specs, `.agents/model-matrix.md`, the captured goldens and the generated `.inc` headers state where a past measurement read its bytes, which is provenance. Spec [`nas-mount-path.md`](specs/nas-mount-path.md) | bug | | [#1077](https://github.com/mudler/vllm.cpp/issues/1077) | — | `.env.example:37`, `.agents/environment.md:29` and `tests/vllm/multimodal/test_ltx2_video.cpp:2128-2132` each state that nothing in the tree reads `CHECKPOINT_ROOT`, and six gates read it: `tests/parity/test_minimax_music3_ar_real.cpp:162`, `_e2e_real.cpp:170`, `_llm_real.cpp:137`, `_quant_real.cpp:130,140`, `tests/vllm/models/test_ltx2_text_encoder.cpp:2299`, and `test_nemotron_h_loader.cpp:161` tells the reader to export it. No product code under `src/` or `include/` reads it, so the accurate statement is that the LIBRARY never reads it while several gates do. It costs more than tidiness: `test_ltx2_video.cpp` reasons FROM the claim when it chooses a separate `LTX2_CHECKPOINT_ROOT` ("this would be its first reader"), and that reasoning is void. Found while repairing [#1073](https://github.com/mudler/vllm.cpp/issues/1073) and NOT fixed there, because reversing a design decision needs its own review rather than a path substitution. Listed under `## Owed` in [`nas-mount-path.md`](specs/nas-mount-path.md) | bug | | [#1079](https://github.com/mudler/vllm.cpp/issues/1079) | `FIX-NAS-PATH-1073` | All four skip messages in `tests/parity/test_minimax_music3_quant_real.cpp` streamed the case name as a `const char*`, and doctest 2.5.2 stringifies that through its bool overload, so every one printed `SKIP 1` and named no case. The comment above the helpers states the obligation the messages then failed: a gate that silently passes when its asset is absent has not reported. It matters here because the binary reports `6 passed` with `assertions: 0` when the checkpoint is absent, so the message text is all that separates a skipped run from a gated one. Pre-existing on `main` at `100026481`. FIXED IN FLOW while landing [#1073](https://github.com/mudler/vllm.cpp/issues/1073), which rewrote those exact messages and would have carried the defect forward under a changed line; the fix streams `std::string(what)`. Scope measured before fixing: 4 hits, all in this one file | bug | | [#1106](https://github.com/mudler/vllm.cpp/issues/1106) | `ENG-EXPERT-STREAM` | A fresh review of the [#1091](https://github.com/mudler/vllm.cpp/issues/1091) repair ([#1100](https://github.com/mudler/vllm.cpp/pull/1100) @ `3da7b4ca2`) returned FAIL on four findings. The six functional repairs are correct and all 13 mutation claims reproduce independently; what failed is what was SAID about them. **(1)** `qwen3_5_internal.h:421-424` stated that the final statistics line is reached "at process teardown: a static registered the first time streaming is requested, plus the store's own destructor" — there is no such static, `grep -rn 'atexit\|quick_exit'` over `qwen3_5.cpp` returns nothing, and #1100's own body says the hook was deliberately not built; the comment also dropped both qualifiers `docs/USAGE.md` carries (a store must have been BUILT, and static destructors must RUN). Verbatim the defect #1091 finding 5 reports about `TowerUid`, reintroduced one file away in the change that fixes it. **(2)** "Nothing lands dead" was claimed for four step guards and holds for ONE: only `Qwen3_5MTPModel::ForwardPaged` has a production caller (`runner.cpp:2183` -> `spec_decode/mtp/speculator.cpp:107,262`); `Qwen3_5MTPModel::Forward`, `Qwen3_5Model::ForwardDense` and `Qwen3_5ReplayLayer` are parity-only entry points whose every caller is under `tests/`, and per `.agents/reachability.md` a call site inside a test is not reach. The guards are correct where they sit, so the code is unchanged and the CLAIM is; the residual is [#1108](https://github.com/mudler/vllm.cpp/issues/1108). **(3)** The nesting refusal (`qwen3_5.cpp:5517`) was asserted in the source, the spec and the pull request body and pinned by nothing: deleting its `VT_CHECK` left both focused binaries fully GREEN (6/6 and 4/4, rc 0) and it appeared in none of the 13 mutations. Unreachable through production code by construction — every forward that takes expert slices is a complete forward that no other one contains — so `detail::ExpertStreamStepScope` exposes the guard's own `Begin`/`End` and a case asserts the refusal twice: a second scope throws, AND a real `ForwardDense` entered while the scope is held throws too, which is what proves the two share a boundary rather than agreeing by coincidence. Kept UNGATED on `Qwen35ExpertStreamRequested()` on purpose: one forward is one step is a property of the call graph, not of the streaming lane, and arming it only under the rare configuration would let the default path establish a nest nobody sees until streaming is switched on. **(4)** `::setenv` sat at namespace scope in both new gates with no `_WIN32` guard; it is POSIX, MSVC's CRT has only `_putenv_s`, `tests/CMakeLists.txt:1087` adds the target unconditionally and `scripts/build-windows-release.ps1` configures `VLLM_CPP_BUILD_TESTS=ON`, so neither translation unit compiled there and the file comment claiming the step-clock cases are "built everywhere" was false. Both now use `vllm_test::SetEnv` from `tests/support/test_env.h`, the shim [#603](https://github.com/mudler/vllm.cpp/issues/603) landed for exactly this. CI could not report it: the Windows lanes fail earlier, inside the product library, on [#1068](https://github.com/mudler/vllm.cpp/issues/1068), and a lane that never reaches a test TU cannot fail in one. All four fixed in flow; three mutations on the added guarantee, three caught, each with a changed sha256, a zero compile status and a non-zero doctest case count. Spec [`expert-streaming.md`](specs/expert-streaming.md) | bug | -| [#1107](https://github.com/mudler/vllm.cpp/issues/1107) | `ENG-RELEASE-WINDOWS` | `scripts/check-windows-portability.py` exits 0 on a tree carrying an unguarded `::setenv` in a test translation unit — measured `Windows portability contract OK`, rc 0, on the unrepaired `test_expert_stream_steps.cpp`. It misses the class twice over, and either miss alone is enough. SCOPE: `check()` builds its `texts` map from `shipped_server_sources(...)`, the sources reachable from the shipped SERVER target, so no file under `tests/` is read for ANY of its rules. VOCABULARY: `POSIX_PATTERNS` names `fork|execvp|waitpid|pipe|read|write|open|close|fsync|pread|pwrite|getpid|stat` plus the `unistd.h`-family includes, and neither `setenv` nor `unsetenv` appears — so even inside the scanned set the call would pass. The second miss is the one that generalises: the three private `_putenv_s` copies [#603](https://github.com/mudler/vllm.cpp/issues/603) records, and the shim it landed, all exist because this is the recurring call, and the checker holding the Windows contract does not know its name. Invisible in CI because `windows-msvc-*` are PR-only with no `main` baseline ([#584](https://github.com/mudler/vllm.cpp/issues/584)) and are currently red inside the product library on [#1068](https://github.com/mudler/vllm.cpp/issues/1068), so a lane failing before it reaches a test TU cannot report a new test-TU failure; the static checker was the only instrument that could have. Found while repairing [#1106](https://github.com/mudler/vllm.cpp/issues/1106) finding 4 and NOT fixed there: a checker semantic change needs its own spec, red-before test and green-after evidence, and widening the scan to `tests/` has to separate a guarded POSIX call from an unguarded one across a large surface, which wants measurement rather than a guess | bug | +| [#1107](https://github.com/mudler/vllm.cpp/issues/1107) | `ENG-RELEASE-WINDOWS` | `scripts/check-windows-portability.py` exits 0 on a tree carrying an unguarded `::setenv` in a test translation unit — measured `Windows portability contract OK`, rc 0, on the unrepaired `test_expert_stream_steps.cpp`. It misses the class twice over, and either miss alone is enough. SCOPE: `check()` builds its `texts` map from `shipped_server_sources(...)`, the sources reachable from the shipped SERVER target, so no file under `tests/` is read for ANY of its rules. VOCABULARY: `POSIX_PATTERNS` names `fork\|execvp\|waitpid\|pipe\|read\|write\|open\|close\|fsync\|pread\|pwrite\|getpid\|stat` plus the `unistd.h`-family includes, and neither `setenv` nor `unsetenv` appears — so even inside the scanned set the call would pass. The second miss is the one that generalises: the three private `_putenv_s` copies [#603](https://github.com/mudler/vllm.cpp/issues/603) records, and the shim it landed, all exist because this is the recurring call, and the checker holding the Windows contract does not know its name. Invisible in CI because `windows-msvc-*` are PR-only with no `main` baseline ([#584](https://github.com/mudler/vllm.cpp/issues/584)) and are currently red inside the product library on [#1068](https://github.com/mudler/vllm.cpp/issues/1068), so a lane failing before it reaches a test TU cannot report a new test-TU failure; the static checker was the only instrument that could have. Found while repairing [#1106](https://github.com/mudler/vllm.cpp/issues/1106) finding 4 and NOT fixed there: a checker semantic change needs its own spec, red-before test and green-after evidence, and widening the scan to `tests/` has to separate a guarded POSIX call from an unguarded one across a large surface, which wants measurement rather than a guess | bug | | [#1108](https://github.com/mudler/vllm.cpp/issues/1108) | `ENG-EXPERT-STREAM` | Three of the four `Qwen35ExpertStreamStep` guards [#1091](https://github.com/mudler/vllm.cpp/issues/1091) finding 3 added land UNREACHED. Only `Qwen3_5MTPModel::ForwardPaged` is reachable from a production entry point (`src/vllm/v1/worker/gpu/runner.cpp:2183` -> `src/vllm/v1/worker/gpu/spec_decode/mtp/speculator.cpp:107,262`). `Qwen3_5MTPModel::Forward` is reached only through `ForwardLogitsHost`, which `qwen3_5_mtp.h:135` documents as "standalone parity convenience" and which itself has no caller outside `tests/`; `Qwen3_5Model::ForwardDense` is the parity reference by `qwen3_5.h:234` (callers `tests/parity/test_op_parity.cpp:1107`, `tests/vllm/v1/worker/test_runner.cpp:1278`, `tests/vllm/models/test_qwen35_paged_forward.cpp:293,320,385,403`); `Qwen3_5ReplayLayer` is per-layer parity replay by `qwen3_5.h:322` (only caller `tests/parity/test_op_parity.cpp:1050`). Per `.agents/reachability.md` a call site inside a test is not reach, so "every added path is reached from a production entry point at this commit" was true of one guard in four. NOTHING IS DELETED: the guards are correct where they sit, cost nothing, and become live the moment any of those entry points gains a production caller — and adding the guard later WITH the caller is precisely how this row lost its step boundary in the first place. What is owed is the record, so this is named as a staged slice that lands unreached rather than claimed as reached. Closes when one of those entry points gains a production caller, or when they are retired as parity references; neither is scheduled. Split out of [#1106](https://github.com/mudler/vllm.cpp/issues/1106) finding 2 so the debt stays open after that repair closes. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#941](https://github.com/mudler/vllm.cpp/issues/941) | `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | `.agents/specs/nemotron-h-abi-e2e.md` §2 named `dense_attn::AttnBlock` as NemotronH's device attention seam, and that block cannot serve this architecture: it applies `vt::RopeNeox` unconditionally (`include/vllm/model_executor/models/dense_attn_block.h:497` @ `10002648199` — the `:496` cited in-tree at `nemotron_h_device.cpp:56` is a comment line, so the anchor is stale as well as the claim), while Nemotron-H has NO RoPE at all: a case-insensitive grep for `rotary`, `rope`, `q_norm`, `k_norm` over `vllm/model_executor/models/nemotron_h.py` at the pinned oracle `5559679229bc` returns ZERO hits, and its four-line attention forward (`:474-483`) sends `q` and `k` straight into `self.attn` with no positional transform. It also reads `cfg.rms_norm_eps`, which `src/vllm/transformers_utils/hf_config.cpp:551` defaults to `0.0` for a checkpoint that ships `layer_norm_epsilon` and `norm_eps` and no `rms_norm_eps`. There is NO rope-free entry point: the header's whole public surface is enumerated in the A2-P spec §2.3, `:490-493` selects only WHICH rope implementation, and `rotary_dim == 0` ABORTS at `src/vt/ops.cpp:1427-1429` rather than bypassing — so neither failure mode announces itself as one. Same shape as [#810](https://github.com/mudler/vllm.cpp/issues/810): a shared function reconstructing behaviour from HF-config fields the model does not ship, and two of its three failure modes are silent. PARTLY FIXED IN FLOW by the A2-P spec: item 1 (correct the seam claim so no later implementer is sent that way) is answered by [`specs/nemotron-h-a2p-paged-forward.md`](specs/nemotron-h-a2p-paged-forward.md) §2.3 plus the correction block appended to the governing spec's §1; item 2 (a model-local block in the `granite.cpp:84` / `gemma4.cpp:206` idiom) was already answered by A2-R at `598226e96` and is extended by A2-P. STILL OPEN and deliberately not fixed here: item 3, whether defaulting `rms_norm_eps` to `0.0` rather than refusing is right in general — that is tree-wide, needs its own red-before, and is listed under `## Owed` in the A2-P spec | bug | | [#1093](https://github.com/mudler/vllm.cpp/issues/1093) | `ROAD-V1-LTX25` | `TI2VidTwoStagesPipeline` (`ti2vid_two_stages.py:61` @ `fd4ded7f`) has no recipe row, no refusal and no `Ltx2UnportedPipelineFeature` marker, so asking for it gets the generic table refusal (`ltx2_pipeline.cpp:1328-1332`) naming the pair rather than the missing machinery. It is NOT the `distilled_two_stage` we ship: stage 1 is CFG-guided on the FULL model (`ti2vid_two_stages.py:247-259`) where `distilled.py:265-266` uses `SimpleDenoiser`; the distilled LoRA rides stage 2 ALONE (`:151`); stage-1 sigmas are scheduler-derived (`:243-245`) against our fixed `DistilledSigmas()` (`ltx2_pipeline.cpp:1163`); and `distilled.py:94-107` has no `distilled_lora` parameter at all. It is also NOT `TI2VidTwoStagesHQPipeline` (`ti2vid_two_stages_hq.py:59`), which [#921](https://github.com/mudler/vllm.cpp/issues/921) owns and which puts the LoRA on BOTH stages at separate strengths. Blocked on (a) a guided VIDEO denoise loop: the guidance arithmetic IS ported and generic (`ltx2_pipeline.cpp:439-524`) but its only consumer is the audio-only T2A loop (`ltx2_t2a.cpp:367`, `video=nullptr` at `:332-334`), and the joint loop is single-forward (`ltx2_video.cpp:3036-3040`); (b) TWO checkpoints absent from the NAS, `ltx-2.5-22b-distilled-lora-450-bf16.safetensors` (`--distilled-lora` is `required=True`, `utils/args.py:1146`) and the full `ltx-2.5-22b-dev-transformer-bf16.safetensors` stage 1 runs. Control: `find /mnt/nas_share/checkpoints -iname '*lora*'` returns nothing while `-name '*.safetensors'` returns the 8 LTX-2.5 files we hold, every transformer among them a `-distilled-` build. Side finding: `one_stage` writes `video_guidance` at `ltx2_pipeline.cpp:1069` and nothing reads it. Already under `## Owed` in [`ltx25-resolution-envelope.md`](specs/ltx25-resolution-envelope.md) as "not separately filed"; filed now because an umbrella row cannot say what THIS arm is blocked on | feature | -| [#1094](https://github.com/mudler/vllm.cpp/issues/1094) | `ROAD-V1-LTX25` | `HDRICLoraPipeline` (`hdr_ic_lora.py:229` @ `fd4ded7f`) is absent with no refusal and no `Ltx2UnportedPipelineFeature` marker. It is the only upstream pipeline returning LINEAR HDR rather than display-referred pixels, so the gap is colour science and not only scheduling. Five citations exist outside `.agents/` and none is an implementation: comments at `ltx2_lora.h:168-169`, `ltx2_pipeline.h:588`, `test_ltx2_lora.cpp:481`, `test_ltx2_video.cpp:542`, plus one string literal inside a `Fail(...)` argument at `ltx2_lora.cpp:246`; `git grep -i HDRICLora` returns zero hits tree-wide. Blocked on (a) a LogC3 / ACEScct decode tail: upstream `ltx-core/hdr.py:37-43` (ARRI EI-800 constants), `:53-65` (compress/decompress), `:82-84` (`HDRTransfer`), `:140-172` (`to_linear`, `to_hdr_linear`), applied at `hdr_ic_lora.py:624` and selected from the adapter's own safetensors metadata (`:178-209`). `git grep -i "logc3|HDRTransfer|to_hdr_linear|acescct"` returns zero product-code hits here, with `git grep -i yuv` as the control (live ffmpeg argv at `minimax_h3_mux.cpp:80`), and the exclusion is already deliberate: [`ltx25-retire-dead-arms.md`](specs/ltx25-retire-dead-arms.md):167 classifies scene-linear HDR colour as "no - colour science". (b) TWO artifacts not on the NAS: `--hdr-lora` (`required=True`, `:833`) from repo `Lightricks/LTX-2.3-22b-IC-LoRA-HDR`, which upstream names only by repo, and `--text-embeddings` (`:834`), since this pipeline loads no text encoder at all (`:275-281`). (c) Per-phase stage-2 tiling and IC-LoRA toggles (`:102-104`, consumed `:485-500`), which our fixed two-phase recipe shape cannot express | feature | +| [#1094](https://github.com/mudler/vllm.cpp/issues/1094) | `ROAD-V1-LTX25` | `HDRICLoraPipeline` (`hdr_ic_lora.py:229` @ `fd4ded7f`) is absent with no refusal and no `Ltx2UnportedPipelineFeature` marker. It is the only upstream pipeline returning LINEAR HDR rather than display-referred pixels, so the gap is colour science and not only scheduling. Five citations exist outside `.agents/` and none is an implementation: comments at `ltx2_lora.h:168-169`, `ltx2_pipeline.h:588`, `test_ltx2_lora.cpp:481`, `test_ltx2_video.cpp:542`, plus one string literal inside a `Fail(...)` argument at `ltx2_lora.cpp:246`; `git grep -i HDRICLora` returns zero hits tree-wide. Blocked on (a) a LogC3 / ACEScct decode tail: upstream `ltx-core/hdr.py:37-43` (ARRI EI-800 constants), `:53-65` (compress/decompress), `:82-84` (`HDRTransfer`), `:140-172` (`to_linear`, `to_hdr_linear`), applied at `hdr_ic_lora.py:624` and selected from the adapter's own safetensors metadata (`:178-209`). `git grep -i "logc3\|HDRTransfer\|to_hdr_linear\|acescct"` returns zero product-code hits here, with `git grep -i yuv` as the control (live ffmpeg argv at `minimax_h3_mux.cpp:80`), and the exclusion is already deliberate: [`ltx25-retire-dead-arms.md`](specs/ltx25-retire-dead-arms.md):167 classifies scene-linear HDR colour as "no - colour science". (b) TWO artifacts not on the NAS: `--hdr-lora` (`required=True`, `:833`) from repo `Lightricks/LTX-2.3-22b-IC-LoRA-HDR`, which upstream names only by repo, and `--text-embeddings` (`:834`), since this pipeline loads no text encoder at all (`:275-281`). (c) Per-phase stage-2 tiling and IC-LoRA toggles (`:102-104`, consumed `:485-500`), which our fixed two-phase recipe shape cannot express | feature | | [#1095](https://github.com/mudler/vllm.cpp/issues/1095) | `ROAD-V1-LTX25` | `DubItPipeline` (`dubit.py` @ `fd4ded7f`) is absent with no `Ltx2UnportedPipelineFeature` marker, and its gap is narrower than the silence suggests. `Ltx2ConditionAudioByReference` is ported (`ltx2_conditioning.h:308`, `ltx2_conditioning.cpp:615-622`), gated (`test_ltx2_vae.cpp:2926`, call at `:2948`) and UNDRIVEN: zero call sites in `src/` or `examples/`, its only other `src/` appearance being inside the refusal string at `ltx2_video.cpp:1993,:1997`. Control: the siblings `Ltx2ConditionVideoByLatentIndex` (`ltx2_video.cpp:2698`) and `Ltx2ConditionVideoByKeyframe` (`:2749`) DO have production call sites, so the grep finds both polarities. Reference audio is already refused by name (`ltx2_video.cpp:1991-2004`, reached from `vllm_c.cpp:1643` and `video_engine.cpp:377-380`). Blocked on (a) the negative RoPE shift, `positions = positions - aud_dur - 0.04` (`dubit.py:351-353`, applied for both stages from `:266-272`): our `Ltx2ConditionAudioByReference` applies no shift, and the ONE ported temporal shift (`ltx2_conditioning.cpp:596-601`) clamps `std::max(0.0, ...)` so it is structurally incapable of producing a negative position; (b) the Dub-It IC-LoRA, `ltx-2.3-22b-ic-lora-dubit-0.9.safetensors` from `Lightricks/LTX-2.3-22b-IC-LoRA-DubIt` (`MODELS-LTX-2.3.md:44`), required exactly once at `dubit.py:364-365` and absent from the NAS; (c) reference-audio ingestion end to end. REJECTS an audit claim in the same breath: `Ltx2AudioPatchify` is NOT undriven - it runs at `ltx2_video.cpp:2595` on every render; its `Ltx2CreateAudioLatentState` call site (`ltx2_conditioning.cpp:484`) is the undriven one. Record drift found while measuring: [`ltx25-a2v-audio-input.md`](specs/ltx25-a2v-audio-input.md):466-472 cites the gate at `test_ltx2_vae.cpp:2412,:2431` where it now sits at `:2926,:2948`, and [`ltx25-ic-lora.md`](specs/ltx25-ic-lora.md):339 says the audio VAE encoder has no load path, which #922 made false | feature | -| [#1096](https://github.com/mudler/vllm.cpp/issues/1096) | `ROAD-V1-LTX25` | `KeyframeInterpolationPipeline` (`keyframe_interpolation.py` @ `fd4ded7f`) is absent with no `Ltx2UnportedPipelineFeature` marker, and its conditioning building block IS served: `Ltx2ConditionVideoByKeyframe` (`ltx2_conditioning.h:172`, `.cpp:530`) is reached from `include/vllm.h:935` through `vllm_c.cpp:1635,:1646` to `ltx2_video.cpp:2745-2751`, mutation-proven at [`ltx25-token-append.md`](specs/ltx25-token-append.md):270. Blocked on (a) no multi-keyframe request surface: the ABI carries two scalar slots, `first_frame` and `last_frame` (`include/vllm.h:934-935`), the engine request two paths and one blob (`video_engine.h:89-93`), and the indices are hard-coded (`latent_idx=0` at `ltx2_video.cpp:2699`, `frame_idx=frames-1` at `:2750`); the CLI exposes only `--first-frame` (`examples/ltx2_gen/main.cpp:269`) and the server only a first frame (`video_engine.cpp:365-372`), against upstream's repeatable `--image PATH FRAME_IDX STRENGTH [CRF]` (`utils/args.py:805-817`, expanded per keyframe at `utils/helpers.py:343-367`). `num_generated_keyframes` is a DIFFERENT feature (model-invented interior slots, `ltx2_video.cpp:1328-1354`) and must not be mistaken for it. (b) A per-sigma guided denoiser: ours is one struct per PHASE (`ltx2_pipeline.h:526-527`, assigned `ltx2_pipeline.cpp:1069-1070`) and audio-only, with `git grep "build_from_sigma|GuiderFactory|per_sigma"` returning 0 against a `sigma` control of ~10 lines in the same header; upstream resolves guiders per step from sigma (`utils/denoisers.py:304-361`, `ltx-core/components/guiders.py:294-342`). REJECTS the audit's "pure porting, no missing checkpoint": `--distilled-lora` is `required=True` on the parser this pipeline uses (`utils/args.py:1146`, selected at `keyframe_interpolation.py:301`, consumed `:111-122`) and stage 1 runs the full `-dev-` transformer; neither file is on the NAS. It needs no IC-LoRA, which is the half of that framing that holds | feature | +| [#1096](https://github.com/mudler/vllm.cpp/issues/1096) | `ROAD-V1-LTX25` | `KeyframeInterpolationPipeline` (`keyframe_interpolation.py` @ `fd4ded7f`) is absent with no `Ltx2UnportedPipelineFeature` marker, and its conditioning building block IS served: `Ltx2ConditionVideoByKeyframe` (`ltx2_conditioning.h:172`, `.cpp:530`) is reached from `include/vllm.h:935` through `vllm_c.cpp:1635,:1646` to `ltx2_video.cpp:2745-2751`, mutation-proven at [`ltx25-token-append.md`](specs/ltx25-token-append.md):270. Blocked on (a) no multi-keyframe request surface: the ABI carries two scalar slots, `first_frame` and `last_frame` (`include/vllm.h:934-935`), the engine request two paths and one blob (`video_engine.h:89-93`), and the indices are hard-coded (`latent_idx=0` at `ltx2_video.cpp:2699`, `frame_idx=frames-1` at `:2750`); the CLI exposes only `--first-frame` (`examples/ltx2_gen/main.cpp:269`) and the server only a first frame (`video_engine.cpp:365-372`), against upstream's repeatable `--image PATH FRAME_IDX STRENGTH [CRF]` (`utils/args.py:805-817`, expanded per keyframe at `utils/helpers.py:343-367`). `num_generated_keyframes` is a DIFFERENT feature (model-invented interior slots, `ltx2_video.cpp:1328-1354`) and must not be mistaken for it. (b) A per-sigma guided denoiser: ours is one struct per PHASE (`ltx2_pipeline.h:526-527`, assigned `ltx2_pipeline.cpp:1069-1070`) and audio-only, with `git grep "build_from_sigma\|GuiderFactory\|per_sigma"` returning 0 against a `sigma` control of ~10 lines in the same header; upstream resolves guiders per step from sigma (`utils/denoisers.py:304-361`, `ltx-core/components/guiders.py:294-342`). REJECTS the audit's "pure porting, no missing checkpoint": `--distilled-lora` is `required=True` on the parser this pipeline uses (`utils/args.py:1146`, selected at `keyframe_interpolation.py:301`, consumed `:111-122`) and stage 1 runs the full `-dev-` transformer; neither file is on the NAS. It needs no IC-LoRA, which is the half of that framing that holds | feature | | [#1097](https://github.com/mudler/vllm.cpp/issues/1097) | `ROAD-V1-LTX25` | `ltx2-gen --lora a --lora b` fuses `b`, DISCARDS `a` and exits 0. `SetExtra` (`examples/ltx2_gen/main.cpp:212-221`) overwrites an existing key in place, so N invocations of `--lora` (`:255-262`) leave exactly one `lora_path` extra. `docs/USAGE.md:809-813` published the opposite - "a second `--lora`" as one of three things that refuse by name - and is corrected in the change that filed this. The wider half, measured after filing: the refusal is unreachable from EVERY production entry point, not only the CLI. `ltx2_video.cpp:813` is the only `dit_options.loras.push_back` in the tree and runs at most once under `if (!lora_path.empty())`, so `options.loras.size()` is 0 or 1 for the CLI, for `vllm_video_engine_load` and for the server alike; `Ltx2ResolveLoraReferenceFactors`'s `> 1` branch (`ltx2_lora.cpp:243-248`) is reached only by `test_ltx2_lora.cpp:384,:492`. So it is correct code guarding a state nothing can construct yet, and the state it guards is what N-adapter fusion ([#932](https://github.com/mudler/vllm.cpp/issues/932)) introduces - which is where the reachability half belongs. Two fix shapes, neither chosen here: refuse the second `--lora` in the CLI, or accumulate and let the library refusal fire. Second defect in the same area and from the same landing: `ltx2_video.cpp:362-363` says "nine of these ten reach a reader" about `kKnownLoadExtras`, which now holds TWELVE entries (`:377-383`) after `lora_path` and `lora_strength` landed with #923; eleven of twelve reach a reader and `duration_head_path` is still the one that does not | bug | | [#1098](https://github.com/mudler/vllm.cpp/issues/1098) | `ROAD-V1-LTX25` | `README.md` cannot be corrected, and TWO gates each refuse the fix independently. The claims that are wrong right now: **"37 registered architectures"** four times (`README.md:11,:80,:253,:296`) where `docs/FEATURES.md` says **40** in three places, corrected two commits earlier in `9143196c7`; and **ZERO `LTX` occurrences** against a `minimax` control of 7, so the video-generation announcement names one of the two shipped video families. Blocker 1: `README.md` measures **29,989 chars against `MAX_README_CHARS = 30000`** (`scripts/check-readme-structure.py:47`), so the `LTX-2.5` matrix row (~130 chars) could only land by DELETING another architecture's row - the shared-file lock AGENTS.md forbids in its own words ("Limit an entry, not a shared file"), and the third instance after the two whole-file budgets [#364](https://github.com/mudler/vllm.cpp/issues/364) retired on that argument, `MAX_CHARS` in `check-now-current.py` and the `chars` key in `check-public-doc-tables.py`. The per-ENTRY caps beside it (`MAX_CELL_CHARS = 220`, `MAX_PARAGRAPH_CHARS = 900`) are what actually stop a landing page decaying into a status log. Blocker 2, which is the decisive one: `check-doc-checkpoint.py:346-354` refuses ANY README change that does not also touch a LANDING SOURCE (`:104-113` - `.agents/mission.md`, `CMakeLists.txt`, three `benchmarks/demo/*.json`, `examples/{cli,server}/main.cpp`), evaluated PER COMMIT ([#573](https://github.com/mudler/vllm.cpp/issues/573)) so splitting the edit out does not help, and its own comment calls the rule deliberate and directly tested. It has no arm for a README CORRECTION as against README CHURN, and a correction has a natural witness: the value disagrees with the projection that owns it. So a two-family paragraph was written, MEASURED to fit at 445 chars against the old 438 with 4 to spare, and then REVERTED unlanded; it is preserved verbatim in the issue thread rather than lost. Asked: whether the checkpoint gate should distinguish correction from churn, and whether the whole-file cap should exist at all. NOT asked: raise the constant, which is widening an assertion to keep a gate green | bug | | [#595](https://github.com/mudler/vllm.cpp/issues/595) | — | `check-doc-checkpoint` keys `feature_surface` off the PATH `src/vllm/model_executor/models/`, so every edit to any model TU owes `docs/FEATURES.md` — the same classify-by-directory defect the file's own header says its rewrite removed for `src/`, `include/` and `tests/`. Measured cost: `e34d71379` (#1054) is a one-line lambda-capture change that alters no capability; the gate demanded the surface, the commit answered with prose, the prose crossed the `check-public-doc-tables` paragraph budgets, and because that checker also runs in the pre-push hook it blocked EVERY branch in the repository from pushing ([#1055](https://github.com/mudler/vllm.cpp/issues/1055), re-filed as [#1062](https://github.com/mudler/vllm.cpp/issues/1062) with a duplicate fix PR, plus [#1058](https://github.com/mudler/vllm.cpp/issues/1058) still open). The repair for the MSVC break the same commit caused ([#1068](https://github.com/mudler/vllm.cpp/issues/1068)) hit the identical demand. Narrowed here to a change in the set of `REGISTER_VLLM_MODEL(...)` registrations, which is what `check-supported-models.py` already gates the table against; adding, removing or renaming an architecture still owes the surface. The LOCK this issue names is NOT closed by that — a genuine new architecture still writes the shared table — so #595 stays open, listed under `## Owed` in [`doc-checkpoint-feature-trigger.md`](specs/doc-checkpoint-feature-trigger.md). Sibling shape for `CMakeLists.txt` -> `docs/USAGE.md` is [#515](https://github.com/mudler/vllm.cpp/issues/515) | bug | @@ -331,3 +331,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1117](https://github.com/mudler/vllm.cpp/issues/1117) | `LTX25-A2VID-RECIPE` | `A2VidPipelineTwoStage` (`a2vid_two_stage.py:53` @ `fd4ded7f`) has no recipe row, so `pipeline_kind = a2vid_two_stage` gets the generic table refusal (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1328-1332`) naming the pair rather than the missing machinery. [#922](https://github.com/mudler/vllm.cpp/issues/922) is CLOSED and closed the audio CONDITIONING, not the recipe: a supplied take rides `distilled_two_stage`, which [`ltx25-a2v-audio-input.md`](specs/ltx25-a2v-audio-input.md):438-446 already records as a different trajectory. Measured at `daeff67f2`: `git grep -n '"a2vid' -- src include tests docs examples` returns TWO hits, both upstream anchors inside `Fail`-message assertions (`tests/vllm/multimodal/test_ltx2_video.cpp:4363,:4427`), against a control of 4 for `"one_stage"` in `include/` alone. Four differences from the recipe it rides, each read at the pin: stage 1 is CFG/STG/modality-guided and caller-configured (`:230-240`, fed from `utils/args.py:947-1006`, `--a2v-guidance-scale` defaulting to `video_guider.modality_scale` = 3.0 at `utils/constants.py:54,:64`) where `distilled_two_stage` fixes `allow_guidance_override = false`; stage 1's schedule is scheduler-derived (`:225-227`) against our fixed `DistilledSigmas()`; stage 1 is plain Euler (`:229-258` passes no `stepper`, `utils/blocks.py:526-527`) against our `kEulerAncestral` on 2.5; and the AUDIO guider is the DEFAULT positive-only one (`:237-239`, `ltx-core components/guiders.py:200-210`) rather than the params table's cfg-7.0 row. Two non-schedule facts that must not be guessed: `--audio-path` is `required=True` (`:312-317`), and the distilled LoRA rides stage 2 ALONE (`:114` against `:107`) with `--distilled-lora` `required=True` (`utils/args.py:1140-1153`). Unblocked by `Ltx2GuidedDenoise` landing at `daeff67f2` (#1092/#1102), which [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) `## Owed` names this arm against. Spec [`ltx25-a2vid-recipe.md`](specs/ltx25-a2vid-recipe.md) | enhancement | | [#1118](https://github.com/mudler/vllm.cpp/issues/1118) | `LTX25-A2VID-RECIPE` | LoRA adapters fuse ONCE, at load, into ONE weight set: `src/vllm/multimodal/ltx2_video.cpp:816-820` is the only `dit_options.loras.push_back` in the tree and it runs under `if (!lora_path.empty())`, so every phase of every recipe sees the same fused `im.dit.weights`. Three upstream pipelines build TWO `DiffusionStage`s from the same checkpoint with DIFFERENT adapter sets at `fd4ded7f`: `a2vid_two_stage.py:107` against `:114-119`, `ti2vid_two_stages.py:140` against `:151`, and `ti2vid_two_stages_hq.py:154,:165` at two separate strengths (`:92-101`). `distilled.py:131` builds ONE stage set, which is why `distilled_two_stage`, `dfr` and `retake` have never needed this and no gate has ever asked. Consequence for the arm landing with #1117: that row mirrors `--distilled-lora required=True` (`utils/args.py:1140-1153`) by refusing an `a2vid_two_stage` load with no `lora_path`, so the 3-step stage-2 schedule cannot run on a checkpoint with no distilled adapter — but it CANNOT mirror upstream leaving stage 1 WITHOUT that adapter, so stage 1's guided schedule runs against base + distilled LoRA where upstream runs it against the base alone. That divergence RENDERS, and the PIXELS it renders are not upstream's: it moves the trajectory, so the frames themselves differ, while the frame count, the shapes, the sample rate and the errors are all exactly what they were — nothing in the SHAPE of the result says anything is wrong, which is why it is filed rather than left in a comment. It is not undetectable, and saying so would be the more damaging error: the instrument that WOULD see it is a real-weights comparison against upstream's own render on the same checkpoint, take and seed, upstream's stage 1 on the base weights against ours on base + distilled. Two fix shapes, neither chosen: a second `Ltx2DitWeights` per adapter set (what upstream pays, two `from_checkpoint` calls at `a2vid_two_stage.py:103,:115`) or unfused runtime LoRA selectable per phase; the first doubles resident DiT weights and the second changes the GEMM path. Bounds #1093 and #921, which need the same seam. Listed under `## Owed` in [`ltx25-a2vid-recipe.md`](specs/ltx25-a2vid-recipe.md) | enhancement | | [#1134](https://github.com/mudler/vllm.cpp/issues/1134) | `BACKEND-ROCM` | The `VT_ATTN_DECODE_D128=1` ctest registration added by #767 cannot show the ROCm `d=128` decode arm REACHED the new kernel, and is empty off ROCm. `RegisteredDevices()` (`tests/vt/test_backend_cross_device.cpp:84-96`) enumerates `{kCUDA, kMETAL, kVULKAN, kXPU, kROCM}` and excludes `kCPU`, so on a CPU-only runner — which is what CI has — the new "Qwen3 geometry (bf16, GQA 2, head_dim 128)" case reports 1 test case, 0 assertions, exit 0, for BOTH registrations. On ROCm hardware the case's only backend assertion is `OpProviderStats::declines == 0`, and `OpProviderStats` counts at PROVIDER granularity, so it is identical with the flag set and unset; the NMSE bound passes on either kernel because the arm is correctness-complete. The two compose: there is no machine in this project on which the flag-ON registration distinguishes itself from the flag-OFF one. Disclosed in [`specs/rocm-decode-attn-d128.md`](specs/rocm-decode-attn-d128.md) §4, its `## Owed` section and its result banner, and §9 stop condition 2 is left OPEN rather than claimed discharged. Closing it needs a kernel-selection counter in `src/vt/rocm/rocm_paged_attn.hip` asserted to DIFFER between the two registrations; the CPU-runner half wants `kCPU` in `RegisteredDevices()` or a non-zero-assertion floor per [#463](https://github.com/mudler/vllm.cpp/issues/463). Not a duplicate of #463 (that is the unset-weights-env-var shape and does not describe the `declines` granularity half), #785 (a kernel that never LAUNCHES behind a dead `#if`, a code defect not a coverage one) or #900 (same family, LTX-2.5 subject) | bug | +| [#1033](https://github.com/mudler/vllm.cpp/issues/1033) | `GATE-ISSUE-INDEX-TABLE-SHAPE` | `check_table_shapes` (`scripts/check-agent-record.py:1292`) never ran on `.agents/issue-index.md`, so a malformed row there was invisible to every gate. The function already counted unescaped pipes per table line with exactly the right regex; its call site (`:1527-1530`) passed `roadmap_v1.md`, `coordination.md`, `*MATRIX_PATHS` and `*spec_paths`, and simply did not pass this path. Nothing else in the tree counts this file's cells, which made the index the ONLY markdown table in the record set with no shape gate. It is also the one record surface every change must write, with rows long enough to hide a stray pipe in a code span. Arming it reds exactly one row on `origin/main` at `100026481`: a pipe histogram over the index's 289 table lines reads `{5: 288, 9: 1}`, and the outlier is line 279, the [#1003](https://github.com/mudler/vllm.cpp/issues/1003) `ORACLE-LLAMACPP-REPIN-STOCK` row that arrived with `283c7e492` ([#1051](https://github.com/mudler/vllm.cpp/pull/1051)), carrying four unescaped pipes inside code spans at columns 2705, 3106, 3115 and 3338. FIXED IN FLOW: the path is added, the four pipes are escaped, and three cases in `tests/scripts/test_agent_record.py` hold it — one capturing the paths `main()` really hands the gate, one running it on the shipped file, one mutating a copy so the instrument is proven to fire. Repairing the row EDITS an append-only file, so `check-issue-index-append-only.py` is red on the branch and the exception is argued in the commit body, as `ff264cb82` ([#1025](https://github.com/mudler/vllm.cpp/pull/1025)) argued the same one: appending a corrected copy would leave the broken row in place and add a duplicate key, so the file only becomes well-formed by editing it where it sits. TWO PREMISES OF THE REPORT MEASURED FALSE and are recorded in the spec rather than quietly dropped: the checker does NOT stop at the first finding (one `errors` list, three findings in one run, exit 1 once), and the four pipes are NOT in a `git diff` piped into `grep` — that span does not exist in the row. Spec [`gate-issue-index-table-shape.md`](specs/gate-issue-index-table-shape.md) | bug | diff --git a/.agents/specs/gate-issue-index-table-shape.md b/.agents/specs/gate-issue-index-table-shape.md new file mode 100644 index 000000000..7ab75350f --- /dev/null +++ b/.agents/specs/gate-issue-index-table-shape.md @@ -0,0 +1,222 @@ +# Spec — the issue index is a table nothing measured + +Issue: [#1033](https://github.com/mudler/vllm.cpp/issues/1033) +Row: `GATE-ISSUE-INDEX-TABLE-SHAPE` (unplaced record/gate defect; the index is a +record surface, not a matrix row) +State: `ACTIVE` + +## Scope + +`check_table_shapes` (`scripts/check-agent-record.py:1292`) counts the +unescaped pipes on every table line of every path it is handed and reports any +line whose count differs from the first line of that table: + +```python +pipes = len(re.findall(r"(? int: check_row_contracts(rows, by_id, errors) check_model_invariants(errors) spec_paths = [path for row in rows if row.state in READY_STATES for path in local_spec_paths(row)] + # ISSUE_INDEX is here for the same reason every other record table is: + # nothing else counts its cells. It was the ONE record surface every + # change must write and the only markdown table in the set with no shape + # gate, so a row that lost its trailing pipe, or carried an unescaped one + # inside a code span, mis-rendered on GitHub while every gate stayed + # green (#1033). The constant is reused rather than respelled so this + # gate and check_issue_index cannot drift onto different files. check_table_shapes( - [AGENTS / "roadmap_v1.md", AGENTS / "coordination.md", *MATRIX_PATHS, *spec_paths], + [ + AGENTS / "roadmap_v1.md", + AGENTS / "coordination.md", + ISSUE_INDEX, + *MATRIX_PATHS, + *spec_paths, + ], errors, ) check_spec_location(errors) diff --git a/tests/scripts/test_agent_record.py b/tests/scripts/test_agent_record.py index b0bccdd98..11385d176 100644 --- a/tests/scripts/test_agent_record.py +++ b/tests/scripts/test_agent_record.py @@ -4,6 +4,7 @@ from __future__ import annotations import importlib.util +import io import re import sys import tempfile @@ -1336,5 +1337,91 @@ def test_owed_issues_reads_specs_with_a_glob(self) -> None: self.assertIsInstance(agent_record.owed_issues(), set) +class IssueIndexTableShape(unittest.TestCase): + """The index is a TABLE, and until #1033 nothing counted its cells. + + `check_issue_index` reads the index by regex, row by row, and answers about + KEYS: is the number well-formed, does it link to itself, is it listed twice, + does it name an owner. None of that is the table's SHAPE. A row that lost + its trailing pipe still matches `ISSUE_ROW`, and a row carrying an unescaped + pipe inside a code span matches it too -- both mis-render on GitHub while + every gate in the tree stays green. + + `check_table_shapes` is the function that measures shape, it already carried + the right regex, and its call site simply did not name this path. + """ + + def paths_main_hands_the_shape_gate(self) -> list: + """The paths the REAL call site passes, captured from the real call. + + Read from the call rather than from the source text on purpose. A test + that greps `check-agent-record.py` for the string `issue-index` passes + on a line that is commented out, on a second call site that is never + reached, and on a constant that is defined and never used. + """ + + captured: list = [] + + def capture(paths, errors) -> None: + captured.extend(paths) + + with mock.patch.object(agent_record, "check_table_shapes", capture): + with mock.patch.object(sys, "stdout", io.StringIO()): + with mock.patch.object(sys, "stderr", io.StringIO()): + agent_record.main() + return captured + + def test_check_table_shapes_covers_the_issue_index(self) -> None: + paths = self.paths_main_hands_the_shape_gate() + # A run that handed the gate NOTHING would satisfy any assertNotIn and + # would satisfy an assertIn only by accident, so the count is asserted + # first. It is the same "how many things did you examine" question the + # index itself went two days without an answer to. + self.assertGreater( + len(paths), 1, "main() handed check_table_shapes no paths at all" + ) + # assertTrue rather than assertIn: the path list runs to ~180 entries + # and assertIn prints all of them, which buries the sentence that says + # what is wrong under the evidence that it is. + self.assertTrue( + agent_record.ISSUE_INDEX in paths, + f"{agent_record.ISSUE_INDEX.name} is not among the {len(paths)} " + "paths main() hands check_table_shapes, so nothing counts the " + "cells of the one record surface every change must write (#1033)", + ) + + def test_the_shipped_issue_index_is_a_well_formed_table(self) -> None: + # The case that would have fired in the offending PR's own preflight. + errors: list[str] = [] + agent_record.check_table_shapes([agent_record.ISSUE_INDEX], errors) + self.assertEqual(errors, []) + + def test_a_malformed_index_row_is_caught(self) -> None: + """The mutation. Without it the two cases above prove only that a list + contains a path and that a file happens to be clean today. + + The copy lives under ROOT because `check_table_shapes` reports through + `relative_to(ROOT)`; a path outside the tree would raise instead of + reporting, and an exception in the harness is not the gate firing. + """ + + text = agent_record.ISSUE_INDEX.read_text(encoding="utf-8") + rows = text.rstrip("\n").split("\n") + self.assertTrue(rows[-1].endswith("|"), "the last index row is not a row") + rows[-1] = rows[-1][:-1] + + with tempfile.TemporaryDirectory(dir=ROOT) as tmp: + mutated = Path(tmp) / "issue-index.md" + mutated.write_text("\n".join(rows) + "\n", encoding="utf-8") + # The mutation APPLIED: one byte shorter, one pipe fewer. + self.assertEqual( + len(mutated.read_text(encoding="utf-8")), len(text) - 1 + ) + errors: list[str] = [] + agent_record.check_table_shapes([mutated], errors) + + require(errors, rf"issue-index\.md:{len(rows)}: table has 4 pipes; expected 5") + + if __name__ == "__main__": unittest.main()