diff --git a/.agents/environment.md b/.agents/environment.md index 8b0e45889..2f4dd52ec 100644 --- a/.agents/environment.md +++ b/.agents/environment.md @@ -72,11 +72,14 @@ labels at all. `class=train` and `gpu_model=GB10` do match. `rc run` has no `--idle-timeout`, `--max-runtime`, `--no-wait`, `--priority`, `--select` and `--timeout`. -### What a leased worker can and cannot do, measured 2026-08-17 +### What the `dgx:gpu0` leased worker can and cannot do, measured 2026-08-17 Probed with one `rc run -d dgx:gpu0 --max-runtime 2m` job (`ff28ada1-0cd3-4867-bf9b-f67050d0608b`). Verify this again before you plan work -around it, because the worker image can change under you. +around it, because the worker image can change under you. **It did change.** The +`thor:gpu0` worker measured later the same day carries `python3` and `gcc`, which +this list calls absent, so read this section as one box on one day. The `thor` +reading is in "A relocated CUDA runtime starts on `thor:gpu0`" further down. - The command runs as user `rc` in a **k3s pod**, hostname `rc-worker-`. `/.dockerenv` is absent and 8 `KUBERNETES_*` variables are set, so it is a pod @@ -86,8 +89,11 @@ around it, because the worker image can change under you. `flock`, and **`/workspace`**. - **Absent: `gcc`, `cc`, `clang`, `nvcc`, `ninja`, `cmake`, `make`, `python3`, `python`, `pip`, `docker`, `sudo`, `git`, `ssh`, `curl`, - `/usr/include/stdio.h`, and any `/usr/local/cuda*` toolkit.** A worker cannot - compile, cannot start Python, and cannot install anything. + `/usr/include/stdio.h`, and any `/usr/local/cuda*` toolkit.** This `dgx:gpu0` + worker cannot compile, cannot start Python, and cannot install anything. + **Do not carry that clause to another device.** On `thor:gpu0` the same day the + worker ran as `uid=0(root)` with `/usr/bin/gcc`, `/usr/bin/python3` and a + working `apt-get` ([#1146](https://github.com/mudler/vllm.cpp/issues/1146)). - **The host filesystem is not visible.** `/home/mudler` does not exist inside the worker. - `/workspace` is the house NAS, measured as `//192.168.68.102/Data 7.3T total, @@ -109,8 +115,12 @@ lease carries bytes, and the exec bit is a mount option" below measures staged content starting under the dynamic loader and after a copy to `/tmp`, so what blocks the oracle is that nothing has put a runtime where a lease can see it. That is why recent GPU work reached for `ssh`, and the bypass is a symptom of -this gap rather than a discipline problem. Do not design the migration here. The -row that takes #1129 owns it. +this gap rather than a discipline problem. Do not design the migration here. +`ENV-LEASE-RUNTIME-STAGING` owns the design, and +[`lease-runtime-staging.md`](specs/lease-runtime-staging.md) holds the working +recipe. That recipe stages `torch` and `triton`, not the pinned oracle, and it +ran on `thor:gpu0` and not here, so the sentence above still stands for +`dgx.casa` today. **This confirms and extends a finding that already landed, rather than making a new one.** `.agents/specs/minimax-music3.md` §13.10 probed `thor`'s worker on @@ -158,26 +168,79 @@ job's working directory `/` is not writable. **So the lease carries bytes, and bytes are enough to run.** A runtime staged on `/workspace` can start under the dynamic loader, or after a copy to `/tmp`. What -the worker cannot do is produce or fetch that runtime, because it has no `curl`, -`wget`, `git`, `gcc`, `nvcc`, `cmake` or `python3`. Present and useful for -staging: `cp`, `cat`, `tar`, `chmod`, `perl`, `flock` and `nvidia-smi`. +this `dgx:gpu0` worker cannot do is produce or fetch that runtime, because it has +no `curl`, `wget`, `git`, `gcc`, `nvcc`, `cmake` or `python3`. Present and useful +for staging: `cp`, `cat`, `tar`, `chmod`, `perl`, `flock` and `nvidia-smi`. **The +`thor:gpu0` worker does produce one**, because it is root and carries `apt-get` +and `gcc`. That is the section below. **This narrows [#1129](https://github.com/mudler/vllm.cpp/issues/1129) and does not close it.** The pinned oracle stays unreachable because its virtual environment lives at `~/venvs/vllm-oracle-pin-555967922` on the dgx host, which no lease can see, and only a host-side actor reached over `ssh` can place a copy -on the NAS. Whether that copy then starts is UNMEASURED. A CUDA virtual -environment holds absolute paths in its shebangs and its `RECORD` files, so -treat the relocation as an open question rather than a solved step. +on the NAS. **Whether a relocated CUDA runtime then starts is no longer +UNMEASURED. It starts, on `thor:gpu0`.** The section below has the reading. A +CUDA virtual environment still holds absolute paths in its shebangs and its +`RECORD` files, so a `pip install --target` tree is the shape that was measured +and a copied venv is not. **Three fleet-side changes would each remove the staging problem, and none of -them is ours to make.** Whoever owns the fleet picks one. +them is ours to make.** Whoever owns the fleet picks one. **A fourth route was +then measured, and it needs nobody's permission:** the `thor:gpu0` worker runs +as root with a working `apt-get`, so a job provisions its own container. 1. The worker image gains a toolchain and a Python interpreter. 2. `rc run` gains an `--image` flag, so a job selects an image that has them. 3. `/workspace` is mounted so that a file there can carry an exec bit. This one removes the copy step only, because the two routes above already execute. +### A relocated CUDA runtime starts on `thor:gpu0`, measured 2026-08-17 + +Probed with six `rc run` jobs on `thor:gpu0`: `6f4bdb03`, `9c0ebeac`, `8beba132`, +`f60d945f`, `63c60a90` and `fd5654c0`. A `torch`, `triton` and `numpy` tree +staged on `/workspace` imports, initializes CUDA, runs a bf16 matmul, and +compiles and executes a Triton kernel. The job IDs in full, the staged-script +sha256 values, the four walls and the working recipe are in +[`lease-runtime-staging.md`](specs/lease-runtime-staging.md) +([#1146](https://github.com/mudler/vllm.cpp/issues/1146)). + +``` +torch.__version__= 2.13.0+cu130 cuda available = True +device 0 = NVIDIA Thor capability = (11, 0) +triton.__version__ = 3.7.1 TRITON_JIT_OK = 4096.0 PASS +``` + +The recipe, once per worker container: + +```sh +apt-get update -qq && apt-get install -y -qq python3-dev +mkdir -p /tmp/tp && cp -a /workspace/oracle-probe/site/triton /tmp/tp/ +chmod -R +x /tmp/tp/triton/backends/nvidia/bin/ + +export PYTHONPATH=/tmp/tp:/workspace/oracle-probe/site +export CPATH=/workspace/oracle-probe/pyhdr/python3.12:${CPATH:-} +``` + +**Read the scope before you quote it.** This is `thor:gpu0` at capability (11,0) +and nothing else. The GB10 is `sm_121a` and is UNMEASURED, so nothing here +licenses a claim about the Spark. Only `torch`, `triton` and `numpy` are staged, +so the pinned vLLM oracle is still not shown to run: it is a source build with +compiled extensions and it needs `nvcc`, which the worker lacks. The torch wheel +is `+cu130` while the staged `ptxas` reports `release 12.8, V12.8.93`, and that +skew is recorded as observed rather than adjudicated. + +**A prebuilt wheel does not remove the `nvcc` requirement, and that is measured.** +An aarch64 vLLM wheel exists in general: `pip download --no-deps vllm` on the +worker fetched `vllm-0.27.1-cp38-abi3-manylinux_2_28_aarch64.whl`, 307,180,998 +bytes. Our pin is not reachable that way, because +`https://wheels.vllm.ai/nightly/vllm/` lists wheels for exactly ONE commit and is +a moving pointer rather than an archive, and because the pin is a development +version that is not on PyPI. Four 404s under a per-commit URL scheme were also +seen, and they prove nothing, because that scheme was never confirmed against a +known-good case. So reproducing the pinned oracle needs a source build or a +deliberate pin advance. Nobody established that vLLM never retains per-commit +wheels. + ### The `flock` orphan hazard that motivated the replacement The harness family in this repository puts the `flock` handle on a **subshell**, diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 39e2db87f..fe5027de2 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -331,4 +331,5 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1117](https://github.com/mudler/vllm.cpp/issues/1117) | `LTX25-A2VID-RECIPE` | `A2VidPipelineTwoStage` (`a2vid_two_stage.py:53` @ `fd4ded7f`) has no recipe row, so `pipeline_kind = a2vid_two_stage` gets the generic table refusal (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1328-1332`) naming the pair rather than the missing machinery. [#922](https://github.com/mudler/vllm.cpp/issues/922) is CLOSED and closed the audio CONDITIONING, not the recipe: a supplied take rides `distilled_two_stage`, which [`ltx25-a2v-audio-input.md`](specs/ltx25-a2v-audio-input.md):438-446 already records as a different trajectory. Measured at `daeff67f2`: `git grep -n '"a2vid' -- src include tests docs examples` returns TWO hits, both upstream anchors inside `Fail`-message assertions (`tests/vllm/multimodal/test_ltx2_video.cpp:4363,:4427`), against a control of 4 for `"one_stage"` in `include/` alone. Four differences from the recipe it rides, each read at the pin: stage 1 is CFG/STG/modality-guided and caller-configured (`:230-240`, fed from `utils/args.py:947-1006`, `--a2v-guidance-scale` defaulting to `video_guider.modality_scale` = 3.0 at `utils/constants.py:54,:64`) where `distilled_two_stage` fixes `allow_guidance_override = false`; stage 1's schedule is scheduler-derived (`:225-227`) against our fixed `DistilledSigmas()`; stage 1 is plain Euler (`:229-258` passes no `stepper`, `utils/blocks.py:526-527`) against our `kEulerAncestral` on 2.5; and the AUDIO guider is the DEFAULT positive-only one (`:237-239`, `ltx-core components/guiders.py:200-210`) rather than the params table's cfg-7.0 row. Two non-schedule facts that must not be guessed: `--audio-path` is `required=True` (`:312-317`), and the distilled LoRA rides stage 2 ALONE (`:114` against `:107`) with `--distilled-lora` `required=True` (`utils/args.py:1140-1153`). Unblocked by `Ltx2GuidedDenoise` landing at `daeff67f2` (#1092/#1102), which [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) `## Owed` names this arm against. Spec [`ltx25-a2vid-recipe.md`](specs/ltx25-a2vid-recipe.md) | enhancement | | [#1118](https://github.com/mudler/vllm.cpp/issues/1118) | `LTX25-A2VID-RECIPE` | LoRA adapters fuse ONCE, at load, into ONE weight set: `src/vllm/multimodal/ltx2_video.cpp:816-820` is the only `dit_options.loras.push_back` in the tree and it runs under `if (!lora_path.empty())`, so every phase of every recipe sees the same fused `im.dit.weights`. Three upstream pipelines build TWO `DiffusionStage`s from the same checkpoint with DIFFERENT adapter sets at `fd4ded7f`: `a2vid_two_stage.py:107` against `:114-119`, `ti2vid_two_stages.py:140` against `:151`, and `ti2vid_two_stages_hq.py:154,:165` at two separate strengths (`:92-101`). `distilled.py:131` builds ONE stage set, which is why `distilled_two_stage`, `dfr` and `retake` have never needed this and no gate has ever asked. Consequence for the arm landing with #1117: that row mirrors `--distilled-lora required=True` (`utils/args.py:1140-1153`) by refusing an `a2vid_two_stage` load with no `lora_path`, so the 3-step stage-2 schedule cannot run on a checkpoint with no distilled adapter — but it CANNOT mirror upstream leaving stage 1 WITHOUT that adapter, so stage 1's guided schedule runs against base + distilled LoRA where upstream runs it against the base alone. That divergence RENDERS, and the PIXELS it renders are not upstream's: it moves the trajectory, so the frames themselves differ, while the frame count, the shapes, the sample rate and the errors are all exactly what they were — nothing in the SHAPE of the result says anything is wrong, which is why it is filed rather than left in a comment. It is not undetectable, and saying so would be the more damaging error: the instrument that WOULD see it is a real-weights comparison against upstream's own render on the same checkpoint, take and seed, upstream's stage 1 on the base weights against ours on base + distilled. Two fix shapes, neither chosen: a second `Ltx2DitWeights` per adapter set (what upstream pays, two `from_checkpoint` calls at `a2vid_two_stage.py:103,:115`) or unfused runtime LoRA selectable per phase; the first doubles resident DiT weights and the second changes the GEMM path. Bounds #1093 and #921, which need the same seam. Listed under `## Owed` in [`ltx25-a2vid-recipe.md`](specs/ltx25-a2vid-recipe.md) | enhancement | | [#1134](https://github.com/mudler/vllm.cpp/issues/1134) | `BACKEND-ROCM` | The `VT_ATTN_DECODE_D128=1` ctest registration added by #767 cannot show the ROCm `d=128` decode arm REACHED the new kernel, and is empty off ROCm. `RegisteredDevices()` (`tests/vt/test_backend_cross_device.cpp:84-96`) enumerates `{kCUDA, kMETAL, kVULKAN, kXPU, kROCM}` and excludes `kCPU`, so on a CPU-only runner — which is what CI has — the new "Qwen3 geometry (bf16, GQA 2, head_dim 128)" case reports 1 test case, 0 assertions, exit 0, for BOTH registrations. On ROCm hardware the case's only backend assertion is `OpProviderStats::declines == 0`, and `OpProviderStats` counts at PROVIDER granularity, so it is identical with the flag set and unset; the NMSE bound passes on either kernel because the arm is correctness-complete. The two compose: there is no machine in this project on which the flag-ON registration distinguishes itself from the flag-OFF one. Disclosed in [`specs/rocm-decode-attn-d128.md`](specs/rocm-decode-attn-d128.md) §4, its `## Owed` section and its result banner, and §9 stop condition 2 is left OPEN rather than claimed discharged. Closing it needs a kernel-selection counter in `src/vt/rocm/rocm_paged_attn.hip` asserted to DIFFER between the two registrations; the CPU-runner half wants `kCPU` in `RegisteredDevices()` or a non-zero-assertion floor per [#463](https://github.com/mudler/vllm.cpp/issues/463). Not a duplicate of #463 (that is the unset-weights-env-var shape and does not describe the `declines` granularity half), #785 (a kernel that never LAUNCHES behind a dead `#if`, a code defect not a coverage one) or #900 (same family, LTX-2.5 subject) | bug | +| [#1146](https://github.com/mudler/vllm.cpp/issues/1146) | `ENV-LEASE-RUNTIME-STAGING` | #1129 is closed and its recorded cause is FALSIFIED by the worker image: it says a leased worker "cannot start Python" and lists `python3`, `pip`, `gcc`, `curl` and `git` as ABSENT, measured in one `dgx:gpu0` probe (job `ff28ada1-0cd3-4867-bf9b-f67050d0608b`). Measured 2026-08-17 on `thor:gpu0` through five `rc run` jobs (`6f4bdb03`, `9c0ebeac`, `8beba132`, `f60d945f`, `63c60a90`), that worker runs as `uid=0(root)` with `/usr/bin/gcc`, `/usr/bin/python3` and a working `apt-get`, and a relocated CUDA runtime staged on `/workspace` imports torch 2.13.0+cu130, reports `cuda available = True` on `NVIDIA Thor` capability (11,0), runs a bf16 matmul, and compiles and executes a Triton kernel (`TRITON_JIT_OK = 4096.0 PASS`, `PROBE5_RC=0`). So the image is provisionable per job, which is none of the three fleet-side fixes #1129 names as the only ways forward. Four walls stand between a staged runtime and a running one: the `pip --target` must run FROM the worker because the submitting host is `x86_64` and the workers are `aarch64`. `Python.h` is absent until `apt-get install python3-dev`. The NAS mount presents `file_mode=0664` so Triton cannot execute its own `ptxas-blackwell`, and `TRITON_PTXAS_PATH` does NOT fix it because it redirects only the plain `ptxas`. So `PYTHONPATH` is ORDERED, `/tmp/tp` before the NAS tree. SCOPE, and it is the point of the row: this is `thor:gpu0` at capability (11,0) ONLY, the GB10 is `sm_121a` and UNMEASURED, only `torch`, `triton` and `numpy` are staged so the pinned vLLM oracle is still NOT shown to run and #1129's consequence for the oracle-dependent rows is NARROWED rather than closed, the `+cu130` versus `release 12.8, V12.8.93` skew is recorded as observed and not adjudicated, and a prebuilt wheel does NOT shorten the route because an aarch64 vLLM wheel exists in general while our pin is neither among the wheels published for the one nightly commit nor on PyPI (the per-commit 404s prove nothing, because that URL scheme was never confirmed against a known-good case). Recipe, job IDs and staged-script sha256 values in [`lease-runtime-staging.md`](specs/lease-runtime-staging.md) | verification | | [#1033](https://github.com/mudler/vllm.cpp/issues/1033) | `GATE-ISSUE-INDEX-TABLE-SHAPE` | `check_table_shapes` (`scripts/check-agent-record.py:1292`) never ran on `.agents/issue-index.md`, so a malformed row there was invisible to every gate. The function already counted unescaped pipes per table line with exactly the right regex; its call site (`:1527-1530`) passed `roadmap_v1.md`, `coordination.md`, `*MATRIX_PATHS` and `*spec_paths`, and simply did not pass this path. Nothing else in the tree counts this file's cells, which made the index the ONLY markdown table in the record set with no shape gate. It is also the one record surface every change must write, with rows long enough to hide a stray pipe in a code span. Arming it reds exactly one row on `origin/main` at `100026481`: a pipe histogram over the index's 289 table lines reads `{5: 288, 9: 1}`, and the outlier is line 279, the [#1003](https://github.com/mudler/vllm.cpp/issues/1003) `ORACLE-LLAMACPP-REPIN-STOCK` row that arrived with `283c7e492` ([#1051](https://github.com/mudler/vllm.cpp/pull/1051)), carrying four unescaped pipes inside code spans at columns 2705, 3106, 3115 and 3338. FIXED IN FLOW: the path is added, the four pipes are escaped, and three cases in `tests/scripts/test_agent_record.py` hold it — one capturing the paths `main()` really hands the gate, one running it on the shipped file, one mutating a copy so the instrument is proven to fire. Repairing the row EDITS an append-only file, so `check-issue-index-append-only.py` is red on the branch and the exception is argued in the commit body, as `ff264cb82` ([#1025](https://github.com/mudler/vllm.cpp/pull/1025)) argued the same one: appending a corrected copy would leave the broken row in place and add a duplicate key, so the file only becomes well-formed by editing it where it sits. TWO PREMISES OF THE REPORT MEASURED FALSE and are recorded in the spec rather than quietly dropped: the checker does NOT stop at the first finding (one `errors` list, three findings in one run, exit 1 once), and the four pipes are NOT in a `git diff` piped into `grep` — that span does not exist in the row. Spec [`gate-issue-index-table-shape.md`](specs/gate-issue-index-table-shape.md) | bug | diff --git a/.agents/specs/gpu-lease-methodology.md b/.agents/specs/gpu-lease-methodology.md index ce3e30c75..5cd8c11eb 100644 --- a/.agents/specs/gpu-lease-methodology.md +++ b/.agents/specs/gpu-lease-methodology.md @@ -92,8 +92,19 @@ The limit is now precise, measured on 2026-08-17 through two carries bytes and not executables. The worker reads and writes the shared `/workspace`, refuses direct execution from it because the mount pins `file_mode=0664`, and runs staged content through `sh FILE`, through the dynamic -loader, or after a copy to `/tmp`. It cannot produce or fetch a runtime, because -it has no compiler, no downloader and no Python. +loader, or after a copy to `/tmp`. + +**The last clause of this section said the worker cannot produce or fetch a +runtime, because it has no compiler, no downloader and no Python. That is a +`dgx:gpu0` reading and it does not generalise.** Later the same day, five +`rc run` jobs on `thor:gpu0` measured a worker running as `uid=0(root)` with +`/usr/bin/gcc`, `/usr/bin/python3` and a working `apt-get`, and a `torch` and +`triton` tree staged on `/workspace` imported, initialized CUDA and compiled and +ran a Triton kernel. See +[`lease-runtime-staging.md`](lease-runtime-staging.md) +([#1146](https://github.com/mudler/vllm.cpp/issues/1146)), which also states what +that result does not establish: it is `thor:gpu0` at capability (11,0) only, the +GB10 is `sm_121a` and UNMEASURED, and the pinned vLLM oracle is not staged. ## The correction has to reach the spec that owns the blocker @@ -113,6 +124,11 @@ green. Both sites therefore take the calibrated form that close it, the relocated virtual environment is UNMEASURED, and the load-bearing reason is that nothing has staged a runtime on the NAS. +**The UNMEASURED clause in that form was answered on the same day, and both sites +were corrected again.** A relocated runtime does start inside a worker, on +`thor:gpu0`. What is still not staged is the ORACLE. See +[`lease-runtime-staging.md`](lease-runtime-staging.md) and #1146. + The same substitution repairs the derivation in the how-to. The old sentence read "no host toolchain, the worker has no compiler, SO no lease-compliant path". Those premises stopped entailing that conclusion once the later probe @@ -168,9 +184,12 @@ is the row gate. ## Owed -- #1129 stays open. No vLLM leg runs on `dgx.casa` by a lease-compliant path - today, because nothing has staged a runtime on the NAS. Whether a relocated - CUDA virtual environment starts inside a worker is UNMEASURED. +- #1129 is now closed, and its recorded cause was falsified on 2026-08-17. No + vLLM leg runs on `dgx.casa` by a lease-compliant path today, because nothing + has staged the ORACLE on the NAS. The "UNMEASURED" clause this line used to + carry is answered: a relocated CUDA runtime does start inside a worker, on + `thor:gpu0` and not yet on `dgx:gpu0`. `ENV-LEASE-RUNTIME-STAGING` and + [#1146](https://github.com/mudler/vllm.cpp/issues/1146) own the rest. ## Now @@ -178,5 +197,7 @@ The rule is stated in `AGENTS.md` and the conditional is in `.agents/environment.md`, keyed on the device and naming the three fleet devices. The narrowing of #1129 now reads the same way in `.agents/environment.md` and in `.agents/specs/mtp-k-gt-1.md`, so the blocked -row's owner is told that staging is untried rather than futile. The next step -belongs to whoever takes #1129, which is staging a runtime the lease can start. +row's owner is told that staging is untried rather than futile. Staging was then +tried, and it worked on `thor:gpu0` +([`lease-runtime-staging.md`](lease-runtime-staging.md), #1146). The next step is +the same probe on `dgx:gpu0`, and after it the pinned oracle itself. diff --git a/.agents/specs/lease-runtime-staging.md b/.agents/specs/lease-runtime-staging.md new file mode 100644 index 000000000..980c7e2d1 --- /dev/null +++ b/.agents/specs/lease-runtime-staging.md @@ -0,0 +1,267 @@ +# Staging a CUDA runtime a lease can start + +Row: `ENV-LEASE-RUNTIME-STAGING`. +Issue: [#1146](https://github.com/mudler/vllm.cpp/issues/1146). + +## Scope + +Record one measurement and correct the records it falsifies. + +The measurement: a relocated CUDA Python runtime, staged on `/workspace`, starts +inside an `rc` lease on `thor:gpu0`. It imports `torch`, initializes CUDA, runs a +bf16 matmul, and compiles and executes a Triton kernel. + +In scope: + +- This spec, holding the four walls, the working recipe, and the evidence. +- The correction to `.agents/environment.md`, which records the leased worker as + carrying no `python3`, no `pip` and no `gcc`. +- The correction to `.agents/specs/mtp-k-gt-1.md` and + `.agents/specs/gpu-lease-methodology.md`, which both carry + [#1129](https://github.com/mudler/vllm.cpp/issues/1129)'s reasoning as the live + cause of the oracle blocker. +- One appended row in `.agents/issue-index.md` for #1146. + +Out of scope: + +- Any measurement on `dgx:gpu0`. See `## What this does NOT establish`. +- Staging the pinned vLLM oracle. Only `torch` and `triton` are staged, and the + oracle is the thing #1129 actually blocks. +- Any product code. This row touches records and documents only. +- Any checker change. This row changes no checker semantics, so it owes no + red-before mutation under `## Changing the rules or a checker`. +- Editing #1129's existing index row. That file is append-only. + +## What #1129 recorded, and what is now false + +#1129 is closed. It records that no lease-compliant path can run a CUDA Python +runtime on this fleet, because the leased worker "cannot start Python". Its +Direction 2 probe recorded `python3=ABSENT`, `pip=ABSENT`, `gcc=ABSENT`, +`curl=ABSENT` and `git=ABSENT` inside the worker. + +That reading came from one probe on `dgx:gpu0`, job +`ff28ada1-0cd3-4867-bf9b-f67050d0608b`, taken on 2026-08-17. Three of its five +names are now contradicted, and the contradictions come from TWO different boxes. +Keep them apart, because a name proven on one worker proves nothing about the +other. + +On `thor:gpu0`, job `8beba132` reports `python3` and `gcc` present. The worker +runs as `uid=0(root)`, carries `/usr/bin/python3`, `/usr/bin/gcc` and +`/usr/bin/apt-get`, and installs a package as root (`APT_UPDATE_RC=0`, +`APT_INSTALL_RC=0`). Job `fd5654c0` then ran `python3 -m pip install --target` +there to completion, so `pip` is present on `thor:gpu0` too. + +On `dgx:gpu0`, job `609c4944-594b-4617-967b-fb3d3d8c09f6` invoked +`python3 -m pip install --quiet --target /workspace/oracle-probe/site torch`, so +`python3` and `pip` are present on that worker as well. Read that job no further: +it ended on `max_runtime exceeded (35m0s)`, so it is evidence that `pip` STARTS +on `dgx:gpu0` and not that this install finished. + +`curl` and `git` were probed on neither worker after #1129, so this row says +nothing about them. + +The measurement was taken correctly. The fleet changed under it. So the recorded +cause no longer holds, and the three fleet-side fixes #1129 names as the only +ways forward are not the only ways forward. The image is provisionable per job. + +## Measured, on `thor:gpu0`, 2026-08-17 + +``` +torch.__file__ = /workspace/oracle-probe/site/torch/__init__.py +torch.__version__= 2.13.0+cu130 +cuda available = True +device count = 1 +device 0 = NVIDIA Thor +capability = (11, 0) +CUBLAS_OK (bf16 1024x1024 matmul executed) +triton.__version__ = 3.7.1 +TRITON_JIT_OK = 4096.0 (expect 4096.0) PASS +PROBE5_RC=0 +``` + +## The four walls, in the order they appear + +1. **The runtime must be installed FROM the worker.** The submitting host here is + `x86_64` and the workers are `aarch64`, so a local `pip install --target` onto + the NAS writes the wrong architecture into the exact path the worker imports + from, and it fails there as a confusing import error. Run the `pip --target` + inside `rc run`. +2. **`Python.h` is absent.** Triton's JIT shells out to `/usr/bin/gcc` to build + `cuda_utils.c` and dies with `fatal error: Python.h: No such file or + directory`. `apt-get install -y python3-dev` succeeds as root, and the headers + copy to `/workspace/oracle-probe/pyhdr` (1.6 MB) so the next job does not + reinstall them. Put them on `CPATH`. +3. **The NAS grants no exec bit, and `TRITON_PTXAS_PATH` does not save you.** The + mount presents `file_mode=0664`, so Triton cannot execute its own + `ptxas-blackwell`. Setting `TRITON_PTXAS_PATH` to a `/tmp` copy is **not + sufficient**: it redirects only the plain `ptxas`, while Triton selects the + Blackwell variant from its own package directory. The whole `triton` package + (651 MB) has to sit on a filesystem that grants exec bits. +4. **So `PYTHONPATH` is ordered, not single.** `PYTHONPATH=/tmp/tp:/workspace/oracle-probe/site` + puts the exec-capable `triton` first and leaves the 4.5 GB of torch on the NAS + where it costs nothing to re-stage. + +## The working recipe + +```sh +# once per worker container +apt-get update -qq && apt-get install -y -qq python3-dev +mkdir -p /tmp/tp && cp -a /workspace/oracle-probe/site/triton /tmp/tp/ +chmod -R +x /tmp/tp/triton/backends/nvidia/bin/ + +export PYTHONPATH=/tmp/tp:/workspace/oracle-probe/site +export CPATH=/workspace/oracle-probe/pyhdr/python3.12:${CPATH:-} +``` + +## The prebuilt-wheel route is closed for our pin + +The spec says the oracle needs `nvcc`. A reader can reasonably ask whether a +prebuilt wheel sidesteps that, so the question was measured on 2026-08-17 rather +than argued. It does not. + +**An aarch64 vLLM wheel exists in general, so the architecture is not the +obstacle.** On the worker, `python3 -m pip download --no-deps vllm` fetched +`vllm-0.27.1-cp38-abi3-manylinux_2_28_aarch64.whl`, 307,180,998 bytes, +`VLLM_DL_RC=0`. The same job reported `aarch64 linux-aarch64` from +`platform.machine()` and `sysconfig.get_platform()`. + +**Our pin is not reachable that way.** `https://wheels.vllm.ai/nightly/vllm/` is +a PEP 503 index, and it lists FOUR wheels for exactly ONE commit, `402547d7f`, +aarch64 and x86_64, each also in its percent-encoded form. It is a moving +pointer and not an archive, so the pin +`5559679229bc961848b121ccdeaa8fa5d79bec98` is absent from it. The pin is also a +development commit, `0.23.1rc1.dev1511+g555967922`, so it is not on PyPI. + +**The 404s are NOT the evidence, and this is the part to read carefully.** Four +candidate filenames under `https://wheels.vllm.ai//` returned 404 by GET +and by HEAD, on the pin and on a current `main` sha `c1e438728c55`. That URL +scheme was never confirmed against a known-good case, and the host's own root +404s while `/nightly` returns 200. A 404 from an unconfirmed path therefore +proves nothing about whether a per-commit wheel exists. The load-bearing +evidence is the nightly index listing one commit. + +**So the consequence is narrow.** Reproducing the pinned oracle needs a source +build, which is why it needs `nvcc`, or a deliberate advance of the pin to a +commit a wheel exists for. A wheel cannot substitute for either. **This does NOT +establish that vLLM never retains per-commit wheels.** Nobody measured that, and +the unconfirmed URL scheme is exactly why. + +## What this does NOT establish + +Read this section before you quote any line above it. Every claim here is +narrower than the sentence a reader wants to write from it. + +- **This is measured on `thor:gpu0` at capability (11,0) ONLY.** The GB10 is + `sm_121a` and is UNMEASURED. Nothing here licenses a claim about the Spark. + A probe is queued at lowest priority behind a human hold on `dgx:gpu0`. Until + that probe returns, `dgx:gpu0` keeps the reading its own last probe gave it. +- **The pinned vLLM oracle is NOT staged.** Only `torch`, `triton` and `numpy` + are, so this does not show that the oracle runs. vLLM at `555967922` is a + source build with compiled extensions, and it needs `nvcc`, which the worker + still lacks. #1129's consequence for the oracle-dependent rows is therefore + NARROWED and not closed. A prebuilt wheel does not remove the `nvcc` + requirement either, and that is measured rather than assumed. See + `## The prebuilt-wheel route is closed for our pin`. +- **The CUDA version skew is recorded as observed, not adjudicated.** The torch + wheel is `+cu130` while the staged `ptxas` reports `release 12.8, V12.8.93`. It + compiled and ran a correct kernel here. Nobody has read whether the skew + changes a numerical result, and this row does not. +- **`numpy` WAS absent from the staged tree, and no longer is.** Job `fd5654c0` + installed it into the same tree on 2026-08-17: `NUMPY_RC=0`, then + `numpy 2.5.2 /workspace/oracle-probe/site/numpy/__init__.py`, + `NUMPY_IMPORT_RC=0`. The five jobs above ran BEFORE that install, which is why + each of their logs carries torch's `Failed to initialize NumPy: No module + named 'numpy'` warning. A job that stages the tree today should not see that + warning, and a job that still does is reading a different tree. + +## Risks + +The one checker these edits can break is `test_gpu_lock_one_truth` (#777), which +requires exactly one `**GPU mutex:**` bullet in `.agents/environment.md`. These +edits add no second mutex statement and do not touch that bullet. + +The corrected sections say what one worker image did on one day. The worker image +can change again, in either direction, which is how this row came to exist. Each +section therefore carries its box, its date and its job IDs, so the next reader +can tell a stale reading from a wrong one. + +The narrower risk is a reader who takes the `thor` result for a fleet result. The +`## What this does NOT establish` section above exists for exactly that reader, +and the same scope statement rides in every record this row edits. + +## Gates + +```sh +scripts/agent-preflight.sh +``` + +The full preflight is the row gate. This row adds no test, because it adds no +behavior: it records a measurement and corrects three documents. The `rc` jobs +below are the evidence, and they are not reproducible in CI, which has no fleet +device. + +## Evidence + +Six `rc run` jobs on `thor:gpu0`, 2026-08-17. The first five staged and ran the +runtime. The sixth added `numpy` and asked the wheel question, and it was +submitted separately as `claude/mudler-ubuntu-box/vllm-probe`. + +Every job after the first ran a script staged on the NAS rather than fed on +stdin. That is a harness detail and not one of the four walls: `@triton.jit` +calls `inspect.getsourcelines`, which raises `OSError: could not get source code` +on a function read from stdin. The sha256 values are taken over the staged files +at `/mnt/nas_share/rc/oracle-probe/`, which is the same folder the worker reads +as `/workspace/oracle-probe/`. + +| Job | Script | sha256 prefix | Result | +|---|---|---|---| +| `6f4bdb03-0162-4ed3-a922-f1a42da10fbd` | `probe.sh` | `9831e189910c` | torch imports, CUDA available, bf16 matmul runs, Triton refuses a stdin kernel | +| `9c0ebeac-8edb-4317-9112-2583bf85f38d` | `probe2.sh` | `0e9cd0f6d323` | wall 2: `fatal error: Python.h: No such file or directory` | +| `8beba132-c682-4b11-aa47-a682f2cce096` | `hdr.sh` | `fb02d1a1f45f` | `uid=0(root)`, `apt-get` installs `python3-dev`, headers staged, wall 3 appears as `PermissionError` on `ptxas-blackwell` | +| `f60d945f-f5ec-49c5-9452-c214a50a0043` | `hdr2.sh` | `37c9dcad4c64` | `TRITON_PTXAS_PATH` to a `/tmp` copy is insufficient, and the staged `ptxas` reports `release 12.8, V12.8.93` | +| `63c60a90-29b8-4115-88c5-d82be2126136` | `hdr3.sh` | `e11a540d23b8` | wall 4 removed, `TRITON_JIT_OK = 4096.0 PASS`, `PROBE5_RC=0` | +| `fd5654c0-d522-498c-8800-ca4df9a36944` | `vllmprobe.sh` | `58fa03543810` | `NUMPY_RC=0`, `numpy 2.5.2` imports from the staged tree, `NUMPY_IMPORT_RC=0`, `VLLM_DL_RC=0` on a 307,180,998-byte aarch64 vLLM wheel, `aarch64 linux-aarch64` | + +The kernel each Triton job ran is `tritontest.py`, sha256 prefix `df9861c9f86d`. + +One `rc run` job on `dgx:gpu0` is cited above for a different purpose: +`609c4944-594b-4617-967b-fb3d3d8c09f6` invoked `python3 -m pip install --target` +and then ended on `max_runtime exceeded (35m0s)`. It is evidence that `python3` +and `pip` exist on that worker. It is not evidence that its install completed. + +The `wheels.vllm.ai` reads have no `rc` job, because they are host-side network +reads rather than device work. + +## Stop conditions + +- Stop if a correction needs a second `**GPU mutex:**` bullet. Return + `NEEDS_DECISION`. Never widen `test_gpu_lock_one_truth` to pass. +- Stop if a correction would state the `thor` result as a `dgx` result. The + scope limit is the point of this row. +- Stop if a correction needs an edit to an existing `.agents/issue-index.md` + row. That file is append-only. + +## Owed + +- [#1146](https://github.com/mudler/vllm.cpp/issues/1146) stays open. Re-run the + identical staged probe on `dgx:gpu0` at `sm_121a`. +- Stage the pinned vLLM oracle itself, which is what #1129 actually blocks. It + needs `nvcc` first, because no wheel carries our pin. +- Read whether the `+cu130` and `12.8` skew changes a numerical result. +- Confirm the `https://wheels.vllm.ai//` URL scheme against a known-good + case before anyone reads the four 404s as evidence of absence. Until then those + 404s carry no weight, and the nightly index is the only load-bearing reading. + +Paid since this spec was written: `numpy` is staged (job `fd5654c0`), so the +former `## Owed` line asking for it is removed rather than left to read as debt. + +## Now + +The four walls and the working recipe are recorded here. +`.agents/environment.md`, `.agents/specs/mtp-k-gt-1.md` and +`.agents/specs/gpu-lease-methodology.md` no longer carry "cannot start Python" as +the live cause, and each now names the box and the date its reading came from. +The staged tree now holds `torch`, `triton` and `numpy`. It does not hold the +oracle, and a prebuilt wheel cannot put it there for our pin, so `nvcc` and a +source build remain the route. The next step is the `dgx:gpu0` probe, which the +coordinator has queued. diff --git a/.agents/specs/mtp-k-gt-1.md b/.agents/specs/mtp-k-gt-1.md index 77b6b924a..8a0a798c5 100644 --- a/.agents/specs/mtp-k-gt-1.md +++ b/.agents/specs/mtp-k-gt-1.md @@ -719,7 +719,7 @@ upstream's own supported configuration. | Owed | What it must show | Who | |---|---|---| | DGX three-way greedy gate at k=2, 3, 4 on Qwen3.6-27B and 35B | **PART-PAID 2026-08-16, and the remaining half is the vLLM leg.** On the 27B NVFP4 at the DEFAULT bf16 GDN state, depth REACHES the verify path at k=2, 3 and 4 on real weights and the per-depth counters are populated at EVERY depth up to k. What is NOT established is `our-ON == our-OFF`: it is FALSE here on 3 of 4 prompts, at the SAME token positions for every k and for the padded control alike, which is the signature of a fixed spec-ON/OFF difference rather than a depth defect. Attributing it needs the oracle leg, which did not run. Do NOT read this row as a passed token gate. **2026-08-17 narrowed it to 3 forwards.** Only the FIRST divergence per arm and prompt is adjudicable, which reduces 1718 divergent positions to 18 and then to 3 distinct probe points. Prompt 1 position 1 resolves to THREE different tokens under three values of k, which no depth defect can produce. `scripts/mtp-k-gt-1-neartie-gap.py` is committed and decides each candidate against `kNearTieMnats = 500`. It still has not RUN: the oracle cannot be loaded while a foreign multi-tens-of-GiB container is resident. **2026-08-17 third pass: the box was CLEAN, the run happened, and the divergence REPRODUCED exactly on independently generated streams (1718 positions, 18 adjudicable, 3 probe points, same tokens), so it is deterministic rather than a flake. The adjudication STILL did not report.** Its failure was an INSTRUMENT failure and is recorded as such rather than as a verdict: the reimaged host carries no C compiler at all, so Triton's JIT died after the weights loaded and vLLM surfaced it as `Engine core initialization failed`. Repaired by running the oracle inside a container carrying the toolchain, which is MEASURED loading the engine past that point; the box then stopped answering SSH mid-leg. Do NOT read this row as a passed token gate | `SPEC-MTP-K-GT-1`, [#81](https://github.com/mudler/vllm.cpp/issues/81) M1 | -| **THE BLOCKER: the pinned oracle cannot load a 27B on `dgx.casa` at all, and `gpu_memory_utilization` is NOT the lever** | Once the toolchain fix let an oracle reach this step for the first time, it consumed the entire host in the step AFTER `torch.compile`. **Measured 2026-08-17 at 0.75: about 110 GiB of HOST RAM held while `nvidia-smi` reported 26 GiB on the device, 45 minutes hung at loadavg 260 with 0 GiB available, its own timeout firing (`ADJUDICATE_EXIT=124`); killing the container took the box from 118 of 119 GiB used to 4 of 119 in under ten seconds.** The attribution to that 0.75 was then **TESTED AND REFUTED**: a third window ran the byte-identical instrument at `--gpu-mem-util 0.30` with a 5-second `MemAvailable` sampler and collapsed the same way (87683 MB free at 09:00:47, **0** at 09:02:25). Weight loading finished with 66 GiB free and compilation with 88 GiB free, so it is neither. **The 0.30 run also REBOOTED the box while the 0.75 run only thrashed** (`boot_id` `5bbdc432...` to `bd5c6e7a...`, `journalctl --list-boots` gap 09:10:15Z to 09:13:55Z), so a lower fraction is not a safety margin. Owed: identify the actual step by varying `max_num_batched_tokens` and `cudagraph_capture_sizes` ONE AT A TIME with the sampler running, then record a configuration that demonstrably reaches KV-cache allocation and completes a generate, in `.agents/environment.md` beside the toolchain recipe. Until then no vLLM leg of any row can run on this host. **The container re-check this row left open is now DONE, and it CONFIRMS the blocker rather than refuting it ([#1129](https://github.com/mudler/vllm.cpp/issues/1129), 2026-08-17).** The host recipe reaches its container over `ssh`, which bypasses the `rc` lease and makes the fleet report `dgx:gpu0` free while somebody is on it. The lease-compliant route was then probed directly, and the leased worker carries no `gcc`, `nvcc`, `ninja`, `cmake`, `python3`, `pip`, `docker` or `sudo`, no `/usr/include/stdio.h` and no `/usr/local/cuda*`, and it cannot see `/home/mudler`, so it cannot reach `~/venvs/vllm-oracle-pin-555967922`. `rc run` has no `--image` flag. **A second probe on the same day NARROWED that reading and did not close it, and one clause of it was measured FALSE.** The lease carries bytes, and bytes are enough to run: `/lib/ld-linux-aarch64.so.1 /workspace/` started a staged ELF, and a copy to `/tmp` followed by `chmod +x` started both a staged script and a staged ELF, each exit 0 (`.agents/environment.md`, "The lease carries bytes, and the exec bit is a mount option"). The clause "could not start it if it could" is therefore RETRACTED here. What holds is narrower and is still a blocker: **no vLLM leg of any row runs on `dgx.casa` by a lease-compliant path today, because nothing has staged a runtime on the NAS**, and only a host-side actor reached over `ssh` can place one there. Whether a relocated CUDA virtual environment then starts inside a worker is UNMEASURED, because such an environment holds absolute paths in its shebangs and its `RECORD` files, so read staging as untried rather than futile. The blocker still widens past #81 to every oracle-dependent row, [#1003](https://github.com/mudler/vllm.cpp/issues/1003), [#915](https://github.com/mudler/vllm.cpp/issues/915) and [#821](https://github.com/mudler/vllm.cpp/issues/821) included. The migration is NOT designed here. The only recorded lever is `/workspace`, which is NAS-backed, writable from the worker and the same folder on `dgx` and `thor`. #1129 owns the design | `SPEC-MTP-K-GT-1`, [#81](https://github.com/mudler/vllm.cpp/issues/81) M1, [#1129](https://github.com/mudler/vllm.cpp/issues/1129) | +| **THE BLOCKER: the pinned oracle cannot load a 27B on `dgx.casa` at all, and `gpu_memory_utilization` is NOT the lever** | Once the toolchain fix let an oracle reach this step for the first time, it consumed the entire host in the step AFTER `torch.compile`. **Measured 2026-08-17 at 0.75: about 110 GiB of HOST RAM held while `nvidia-smi` reported 26 GiB on the device, 45 minutes hung at loadavg 260 with 0 GiB available, its own timeout firing (`ADJUDICATE_EXIT=124`); killing the container took the box from 118 of 119 GiB used to 4 of 119 in under ten seconds.** The attribution to that 0.75 was then **TESTED AND REFUTED**: a third window ran the byte-identical instrument at `--gpu-mem-util 0.30` with a 5-second `MemAvailable` sampler and collapsed the same way (87683 MB free at 09:00:47, **0** at 09:02:25). Weight loading finished with 66 GiB free and compilation with 88 GiB free, so it is neither. **The 0.30 run also REBOOTED the box while the 0.75 run only thrashed** (`boot_id` `5bbdc432...` to `bd5c6e7a...`, `journalctl --list-boots` gap 09:10:15Z to 09:13:55Z), so a lower fraction is not a safety margin. Owed: identify the actual step by varying `max_num_batched_tokens` and `cudagraph_capture_sizes` ONE AT A TIME with the sampler running, then record a configuration that demonstrably reaches KV-cache allocation and completes a generate, in `.agents/environment.md` beside the toolchain recipe. Until then no vLLM leg of any row can run on this host. **The container re-check this row left open is now DONE, and it CONFIRMS the blocker rather than refuting it ([#1129](https://github.com/mudler/vllm.cpp/issues/1129), 2026-08-17).** The host recipe reaches its container over `ssh`, which bypasses the `rc` lease and makes the fleet report `dgx:gpu0` free while somebody is on it. The lease-compliant route was then probed directly, and the `dgx:gpu0` leased worker carries no `gcc`, `nvcc`, `ninja`, `cmake`, `python3`, `pip`, `docker` or `sudo`, no `/usr/include/stdio.h` and no `/usr/local/cuda*`, and it cannot see `/home/mudler`, so it cannot reach `~/venvs/vllm-oracle-pin-555967922`. That list is one box on one day and it does NOT generalise across the fleet, which the third probe below measured. `rc run` has no `--image` flag. **A second probe on the same day NARROWED that reading and did not close it, and one clause of it was measured FALSE.** The lease carries bytes, and bytes are enough to run: `/lib/ld-linux-aarch64.so.1 /workspace/` started a staged ELF, and a copy to `/tmp` followed by `chmod +x` started both a staged script and a staged ELF, each exit 0 (`.agents/environment.md`, "The lease carries bytes, and the exec bit is a mount option"). The clause "could not start it if it could" is therefore RETRACTED here. What holds is narrower and is still a blocker: **no vLLM leg of any row runs on `dgx.casa` by a lease-compliant path today, because nothing has staged a runtime on the NAS**, and only a host-side actor reached over `ssh` can place one there. **A THIRD probe the same day answered the UNMEASURED clause, and it is narrower than it sounds.** A relocated CUDA runtime staged on `/workspace` DOES start inside a worker: on `thor:gpu0`, five `rc run` jobs imported torch 2.13.0+cu130, reported `cuda available = True` on `NVIDIA Thor` capability (11,0), ran a bf16 matmul and compiled and executed a Triton kernel (`TRITON_JIT_OK = 4096.0 PASS`). That worker runs as `uid=0(root)` with a working `apt-get` and `/usr/bin/gcc`, so #1129's recorded "cannot start Python" cause no longer holds for that image ([#1146](https://github.com/mudler/vllm.cpp/issues/1146), [`lease-runtime-staging.md`](lease-runtime-staging.md)). What it does NOT establish is what this row needs. It is `thor:gpu0` at capability (11,0) ONLY, and the GB10 is `sm_121a` and UNMEASURED. Only `torch`, `triton` and `numpy` are staged, NOT the pinned oracle, which is a source build with compiled extensions and needs `nvcc`. **A prebuilt wheel does not shorten that route, measured 2026-08-17:** an aarch64 vLLM wheel exists in general, but our pin is not among the wheels published for the one nightly commit and is not on PyPI, so a source build or a deliberate pin advance is the only way. So the vLLM leg is still blocked here, by a narrower and now precisely named reason. The blocker still widens past #81 to every oracle-dependent row, [#1003](https://github.com/mudler/vllm.cpp/issues/1003), [#915](https://github.com/mudler/vllm.cpp/issues/915) and [#821](https://github.com/mudler/vllm.cpp/issues/821) included. The migration is NOT designed here. The only recorded lever is `/workspace`, which is NAS-backed, writable from the worker and the same folder on `dgx` and `thor`. `ENV-LEASE-RUNTIME-STAGING` owns the design | `SPEC-MTP-K-GT-1`, [#81](https://github.com/mudler/vllm.cpp/issues/81) M1, [#1129](https://github.com/mudler/vllm.cpp/issues/1129) | | The cleanup trap in the DGX drivers does not stop the run | `run_all_inner.sh` and `run_oracle_inner.sh` both use `trap cleanup EXIT INT TERM` where `cleanup` resets the clocks and returns WITHOUT exiting, so bash resumes the script after the handler. **Observed 2026-08-17: `SIGTERM` reset the clocks and the driver then started its next leg, which immediately began re-filling a box that had 0 GiB available.** The chain had to be `SIGKILL`ed and the container stopped separately. A cleanup trap that does not terminate is not a stop button, and on a box that reboots rather than OOM-killing that difference is the box. Owed: `exit` from the signal path, and a `docker kill` of the current leg's named container inside `cleanup` so the container cannot outlive its driver | `SPEC-MTP-K-GT-1`, [#81](https://github.com/mudler/vllm.cpp/issues/81) M1 | | The PADDED CONTROL arm of that gate, and the RATE assertion it carries | **PAID 2026-08-16 on the 27B, and its throughput VOID LIFTED 2026-08-17.** The first pass could not quote `padded_k3`/`padded_k4` throughput because they started at loadavg 10.77 and 20.41 against real arms at 1.5 to 2.9. The third pass re-ran all seven arms inside ONE window in a load band of 0.16 to 1.86, every leg exit 0, so the real-arm against padded-control comparison is now quotable rather than merely computed. Detail below. Margin fixed BEFORE the run at 0.10 absolute per depth. The real loop accepts at 0.507 to 0.750 at every depth >= 1. The padded control accepts at 0.000 at every depth >= 1 while its depth-0 rate MATCHES the real arm (0.892 to 0.925 against 0.868 to 0.878), which is what a control that isolates columns >= 1 must look like. Every margin clears by at least 0.41. The control measuring 0 is recorded as a fact about THIS prompt set and did not license restoring the count assertion. Still owed on the 35B | `SPEC-MTP-K-GT-1`, [#81](https://github.com/mudler/vllm.cpp/issues/81) M1 | | Silent de-graphing when the actual depth differs from the configured k, AND the `S`-only slot-ring key ([#1020](https://github.com/mudler/vllm.cpp/issues/1020)) | The spec-graph predicate reads the step's ACTUAL uniform query length instead of `num_spec()` (`runner.cpp:1383`), and the graph slot ring is keyed on `(S, q)` in the SAME change. The re-key is owed on its own merits and NOT only as a consequence of widening the predicate, which is the correction section 4.2a records: `uniform_decode = input.pure_decode \|\| (spec_graph && ...)` (`qwen3_5_moe.cpp:143-148`, `qwen3_5_dense.cpp:172-177`) already routes TWO query lengths to one `impl_->slots[S]` (`qwen3_5.cpp:9281`, dense `:9708`), and `SizeSlot` invalidates on `fa_cols` and `aux_taps` only (`:9309-9316` and `:9361-9367`). At k=1, 8 requests pure-decode and 4 requests spec both key on `S = 8`. That is pre-existing since SPEC-DSPARK W8 (#442) and this row does not widen it, but it is not the benign thing the first spec revision claimed. Plus a measured before-and-after on the capture-set size and persistent logits memory, and a counter or log for the eager fallback so it can never again be invisible | `SPEC-MTP-K-GT-1`, [#1020](https://github.com/mudler/vllm.cpp/issues/1020) | @@ -753,10 +753,20 @@ vLLM leg of any row runs on `dgx.casa` by a lease-compliant path today because nothing has staged a runtime on the NAS, and NOT because a lease cannot execute one ([#1129](https://github.com/mudler/vllm.cpp/issues/1129), and `.agents/environment.md` "The lease carries bytes, and the exec bit is a mount -option"). Whether a relocated CUDA virtual environment starts inside a worker is -UNMEASURED, so staging is worth attempting rather than futile. Resume this row -only after #1129 has a path, because the adjudication instrument is committed and -correct and has nothing to run on. +option"). **A third probe then answered the UNMEASURED clause: staging works.** +On `thor:gpu0`, a `torch`, `triton` and `numpy` tree staged on `/workspace` imports, +initializes CUDA and compiles and runs a Triton kernel, and that worker is root +with `apt-get` and `gcc`, so #1129's "cannot start Python" cause is falsified for +that image ([#1146](https://github.com/mudler/vllm.cpp/issues/1146), +[`lease-runtime-staging.md`](lease-runtime-staging.md)). This row still cannot +resume on it. The result is `thor:gpu0` at capability (11,0) only, the GB10 is +`sm_121a` and UNMEASURED, and the staged tree holds `torch`, `triton` and `numpy` +and not the pinned oracle, which needs `nvcc`. A prebuilt wheel does not shorten +that route: an aarch64 vLLM wheel exists in general, and our pin is neither among +the wheels published for the one nightly commit nor on PyPI, so a source build or +a deliberate pin advance is the only way. Resume this row only after the oracle +itself has a lease-compliant path, because the adjudication instrument is +committed and correct and has nothing to run on. ## Outcome (partial, CPU half)