Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agents/issue-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -291,3 +291,4 @@ rather than merged. `scripts/check-agent-record.py` gates both.
| [#1024](https://github.com/mudler/vllm.cpp/issues/1024) | — | An LTX-2.5 render invoked with `--device cuda` **stages 35.54 GiB onto the GPU and then never uses it**. MEASURED on `dgx.casa` (GB10, boot id `03717c9d-63c8-4652-a8fe-a63d012c5718`, build `0e1bee42f`), 320x192/25f, per-PID over **347 samples**: GPU utilization never exceeded 2%, was **exactly 0 in 321/347**, and **every one of the 26 non-zero samples falls inside the DiT staging window** (t<=251 s) — so **every post-staging sample reads 0**, while the compute-app footprint sits flat at 36396 MiB and the process holds **exactly 1.00 core of 20** across the two `utime` windows this figure covers (+116.7 s over 117 s, +142.2 s over 142 s = **259 s of measurement**; §1.3 of the spec records a third window in the same regime, +648.9 s over 649 s, so 259 s is what these two cover and NOT the sampler's total, which the record cannot settle — [#1040](https://github.com/mudler/vllm.cpp/issues/1040)); 0 frames written. **This row states sample counts and states NO minute figure, because no consistent one exists.** Drafts variously said "17+ minutes" here and "over 15 minutes" over "the first 1192 s" in the spec; 347 samples at a 2 s cadence span 694 s, so neither fits, and no dropped-sample rate was ever recorded. The GPU-zero half is over EVERY sample; the core-count half rests on 259 s. Staging took the device path (`Anonymous` stayed at 0.01 GiB, so no f32 widening — `src/vllm/multimodal/ltx2_video.cpp:786 @ 332aed738`). Leaves a disjunction this issue cannot yet split, because nothing timestamps a phase boundary ([#1010](https://github.com/mudler/vllm.cpp/issues/1010)): either the denoise is not taking `Ltx2DitForwardDevice` (`ltx2_video.cpp:2946` vs `:2948 @ 332aed738`), or it is and something before it burns the rest of the run single-threaded. Broader than [#1007](https://github.com/mudler/vllm.cpp/issues/1007): the 0% GPU is a property of the whole post-load render, not of the decode phase, so `docs/USAGE.md:873-874 @ 332aed738` understates it. **Owed: a positive control that `utilization.gpu` reads high for a real kernel on GB10** — this box already returns `[N/A]` for `--query-gpu=memory.used`. Sampler CSV not retrievable ([#1040](https://github.com/mudler/vllm.cpp/issues/1040)). Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | bug |
| [#1040](https://github.com/mudler/vllm.cpp/issues/1040) | — | `LTX25-DECODE-SPEED` ([#1006](https://github.com/mudler/vllm.cpp/issues/1006)) records three sets of PASSING measurements and **none has a retrievable evidence artifact**, which `.agents/verification.md` requires alongside the SHA, command, environment and exit status. Rung 1's 248 per-PID sampler rows, rung 2's 1082 rows and 347 side-car per-PID samples, rung 2's `run.log`, and §1.4's `~/work/ltx25-e2e/render8-console.log` all live only on `dgx.casa` — `ping -c 2 -W 3 dgx.casa` exits 1 with 100% packet loss and `Destination Host Unreachable` (checked 2026-08-16), this box's documented unified-memory OOM-reboot mode, which needs a physical power cycle. The spec's `REMOTE_UNVERIFIED` mark was correctly scoped to rung 2's EXIT REASON; this issue extends the same honesty to the passing numbers. **Second and more corrosive: neither rung's sampler cadence closes.** Rung 1 states 248 samples at 2 s over a 701 s run split 192+56 across 450 s and 164 s windows — at 2 s those windows hold 225 and 82, and 248 samples cover 496 s of 701. A draft of rung 2's §5 stated 347 per-PID samples at 2 s "over the first 1192 s" — at 2 s that window holds 596. No dropped-sample rate is recorded anywhere, so the wall each sample set covers is NOT derivable, and the spec now states sample counts and fractions rather than minute figures. The raw CSVs settle both in one pass. Filed while repairing the fresh review of [PR #1038](https://github.com/mudler/vllm.cpp/pull/1038); not fixable in that flow, because no edit to the tree produces a file on a host that does not answer. Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | bug |
| [#1029](https://github.com/mudler/vllm.cpp/issues/1029) | `ENG-EXPERT-STREAM` | `IsCudaKeepQuantSupported` (`src/vt/cuda/cuda_quant_dot.cu`) returns true for `kIQ1_S` and `kIQ1_XXXS` since [#967](https://github.com/mudler/vllm.cpp/pull/967), and THREE dispatch switches consume that predicate while #967 extended only the dense one. `MatmulBTQuantGroupedKernelCuda` uses the predicate to SKIP its CPU fallback and then dispatches through a `switch (w)` with no case for either dtype and no `default:`: it quantizes the activation, launches NOTHING, returns, and `CheckCuda(cudaGetLastError())` reports success because there was no launch to fail, so the output tensor is never written. Measured on GB10 by an independent review through a poisoned output buffer: both IQ1 encodings leave `-12345` in place at NMSE `4.58e6` / `9.96e6` against the CPU oracle, with `iq2_s` passing as the control. Reachable by DEFAULT (`qwen3_5_gguf_weights.cpp` -> `qwen3_5.cpp` `KqGrouped` -> `vt::MatmulBTQuantGrouped`, `VT_QWEN35_GROUPED_MOE` on) on the 96.92 % of `Qwen3.8-2.4T` those two encodings make up, so #967 converted correct-but-slow into silently wrong. Same omission in the fused `MoeGateUpSwiGLUGroupedCuda` seam, where it turned a NAMED refusal into silence. It landed green because the CUDA grouped dispatch had NO test: `grep -rl MatmulBTQuantGrouped tests/` found two files and neither mentioned `kCUDA`. Repaired by adding both arms to both grouped switches, a `default:` that THROWS and names the dtype on all three (the general fix: past that predicate there is no fallback left), and a grouped + fused CUDA gate over the dense gate's case table driven through a POISONED output buffer. Also seals the device codebooks against the CPU tables, which `cuda_quant_iq_tables.cuh` claimed from the day it landed and no test did: 266 of 2048 `d_iq1s_grid` entries (13.0 %) are never addressed by the gate's own `mt19937(0x5EED)` stream. Device evidence is OWED and the issue stays open for it: `dgx.casa` was unreachable throughout and this box has no CUDA device, so only the CPU arm ran. Spec [`expert-streaming.md`](specs/expert-streaming.md) | bug |
| [#1055](https://github.com/mudler/vllm.cpp/issues/1055) | `ENG-RECORD-CONFLICT-SURFACES` | `main` went RED on `check-public-doc-tables.py` at `e34d71379`: two prose paragraphs added beside a two-character source fix took docs/BENCHMARKS.md to 36 paragraphs of 35 and docs/FEATURES.md to 22 of 21, and the checker runs both in the `pre-push` hook and at `.github/workflows/ci.yml:160`, so every branch in the repository inherited a red it did not cause. Fixed in flow by folding each paragraph into the keyed row its content belongs to, which is what the checker's own message prescribes: the Apple Clang build disposition into the docs/BENCHMARKS.md `Open gaps` table, the Apple Clang platform fact into the docs/FEATURES.md backend table. No paragraph was deleted and no budget was raised. The deeper defect is the budget itself, a whole-page count on a shared file, which AGENTS.md Records names as the anti-pattern (`Limit an entry, not a shared file`); redesigning it is `ENG-RECORD-CONFLICT-SURFACES` scope, whose spec already carries the obligation | bug |
6 changes: 1 addition & 5 deletions docs/BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -456,11 +456,6 @@ built on it rather than keeping the flattering one.

**CPU elementwise GEMM, transpose-free `[K,N]` path (2026-08-07).** On dgx aarch64 the `[K,N]` path beats `[N,K]` by 1.16x to 1.30x, byte-identically. The x86 arm is INDICATIVE ONLY, not binding: that box is VOID for timing per `CLAIM-KERNEL-CPU-ELEM-GEMM-1`. `VT_CPU_MATMUL_STEAL` ships default OFF and is NOT measured; it must justify itself by measurement and may measure neutral.

**Darwin Qwen3.5 build repair (2026-08-16).** Benchmarking is NOT APPLICABLE.
The change removes a redundant namespace-scope lambda capture that Apple Clang
rejects under `-Werror`; it does not change generated refusal text, model math,
or any runtime path. The binding gate is the Apple Clang build.

## Open gaps

| Track | Status | Next gate |
Expand All @@ -474,6 +469,7 @@ or any runtime path. The binding gate is the Apple Clang build.
| Accepted-and-inert serve args (`SERVE-RECIPE-ARGS`, #606) | **No number owed**: argument parsing only, so nothing to time and no oracle leg. Correctness gate 4 cases / 58 asserts GREEN, RED-first, mutation-proven | None. A speed axis would be fabricated; closes on review plus the operator gate rerun |
| DeepSeek-V2-Lite MLA | Attributed miss, `ACTIVE` | Throughput at every concurrency |
| Qwen3.5 text-only arms (#490) | **No number; run gates OWED**, both `PARTIAL`. The loader half is CLOSED (#740, #864 `DONE`), so what blocks these is hardware, not a refusal | No fitting ckpt for either causal-LM arm: no denominator. `Qwen3.8-2.4T-A95B` is ~4.8 TB vs 128 GB; its load plan resolves, which is not a token |
| Darwin Qwen3.5 build repair (#1054, 2026-08-16) | **NOT APPLICABLE.** Removing a redundant namespace-scope lambda capture that Apple Clang rejects under `-Werror` changes no generated refusal text, no model math and no runtime path | None. The binding gate is the Apple Clang build |
| Qwen3.6-35B-A3B published BF16 (#740, #864) | **No number, and none was owed: the 2026-08-15 gate measured TOKENS.** Correctness MET vs the pinned oracle: 6/7 prompts STRICT 16/16, the 7th an exact tie (#910); SACRED 3/3 byte-identical | A throughput / latency / memory grid on this checkpoint. Nothing is measured, so nothing is claimed |
| MoE vision tower image + video (#891) | **NOT gated, no number.** The 333 `model.visual.*` tensors load and the tower computes, on sm_110 FALLBACK attention, not the shipped GB10 path. The token-exact mm gates never ran | Both modality gates on GB10 through the shipped fast path, then a per-modality speed grid |
| Dense image/video after the #891 merge (#908) | **UNVERIFIED, network-blocked.** Dense TEXT is 235/235 at `2f2bce926`, a true before/after (binary md5 `db889909d4…` vs `49ded1ece8…`, 500 TUs recompiled). The modality arms were not re-run | Re-run the dense image and video gates once the fixtures are reachable |
Expand Down
6 changes: 1 addition & 5 deletions docs/FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,10 +148,6 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks.
| `CohereForCausalLM` | Command-R / Cohere (and Cohere2) | scaffold: W0 tiny-random oracle run-verified; real-checkpoint gate blocked | no run |
<!-- supported-arch-table:end -->

The Qwen3.5 MoE loader also builds under Apple Clang with project warnings
promoted to errors. Its layout-refusal path uses the same messages and behavior
on every platform.

### Standalone and non-registered lanes

These run through dedicated forwards, not the `REGISTER_VLLM_MODEL` registry, so
Expand Down Expand Up @@ -253,7 +249,7 @@ both refuse, naming what is missing.
|---|---|---|---|---|
| CUDA | ✅ sm_80 to sm_121a | ✅ | ✅ | ✅ |
| CPU (x86, Arm i8mm; A76 assembly correct/default, llama speed gate open, and the closed 20-core floor ran a SUPERSEDED fork denominator rather than the stock `b10451` pin, re-take owed #1003) | ✅ | ◐ | ☐ | ✅ |
| Metal (Apple Silicon) | ✅ | ☐ | ☐ | ✅ |
| Metal (Apple Silicon) | ✅ builds under Apple Clang with project warnings promoted to errors, the Qwen3.5 MoE loader included; its layout-refusal path uses the same messages and behavior on every platform (#1054) | ☐ | ☐ | ✅ |
| Vulkan | ◐ | ☐ | ☐ | ✅ |
| ROCm | W0 verified on 5 gfx archs; dense and GDN models run all-native. Strict CPU parity is open in the measured near-tie regime (#269) | 44 registered ops including full GDN; ctest-green gfx1151/1103/1100/1201/1200 ([#41](https://github.com/mudler/vllm.cpp/issues/41)). APU managed allocation is unverified. [ROCM.md](ROCM.md) | ✅ | ✅ |
| XPU / TPU | ☐ | ✅ | ◐ | ☐ |
Expand Down
Loading