From 0fde3db29f698f0ab34ef4b203e5780b2a696211 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 18:28:39 +0000 Subject: [PATCH 1/7] spec(ENV-LEASE-RUNTIME-STAGING): a relocated CUDA runtime starts inside a lease on thor:gpu0 (#1146) #1129 is closed and records that a leased worker "cannot start Python", with `python3`, `pip`, `gcc`, `curl` and `git` all ABSENT. That reading came from one probe on `dgx:gpu0`. Measured 2026-08-17 on `thor:gpu0` through five `rc run` jobs, the worker runs as `uid=0(root)`, carries `/usr/bin/gcc`, `/usr/bin/python3` and a working `apt-get`, and a relocated CUDA runtime staged on `/workspace` imports torch 2.13.0+cu130, reports `cuda available = True` on `NVIDIA Thor` capability (11,0), runs a bf16 matmul, and compiles and executes a Triton kernel (`TRITON_JIT_OK = 4096.0 PASS`, `PROBE5_RC=0`). This spec carries the four walls between a staged runtime and a running one, the working recipe, and the five job IDs with the staged-script sha256 prefixes, so the next person does not re-derive them. It also states in its own section what the result does not establish. The measurement is on `thor:gpu0` at capability (11,0) only, and the GB10 is `sm_121a` and UNMEASURED. Only `torch` and `triton` are staged, so the pinned vLLM oracle is still not shown to run and #1129's consequence for the oracle-dependent rows is narrowed rather than closed. The CUDA skew between the `+cu130` wheel and the `release 12.8, V12.8.93` ptxas is recorded as observed and not adjudicated. `numpy` is absent from the staged tree. The record corrections ride in the following commits of this branch. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/lease-runtime-staging.md | 191 +++++++++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100644 .agents/specs/lease-runtime-staging.md diff --git a/.agents/specs/lease-runtime-staging.md b/.agents/specs/lease-runtime-staging.md new file mode 100644 index 000000000..cb4abe1f0 --- /dev/null +++ b/.agents/specs/lease-runtime-staging.md @@ -0,0 +1,191 @@ +# Staging a CUDA runtime a lease can start + +Row: `ENV-LEASE-RUNTIME-STAGING`. +Issue: [#1146](https://github.com/mudler/vllm.cpp/issues/1146). + +## Scope + +Record one measurement and correct the records it falsifies. + +The measurement: a relocated CUDA Python runtime, staged on `/workspace`, starts +inside an `rc` lease on `thor:gpu0`. It imports `torch`, initializes CUDA, runs a +bf16 matmul, and compiles and executes a Triton kernel. + +In scope: + +- This spec, holding the four walls, the working recipe, and the evidence. +- The correction to `.agents/environment.md`, which records the leased worker as + carrying no `python3`, no `pip` and no `gcc`. +- The correction to `.agents/specs/mtp-k-gt-1.md` and + `.agents/specs/gpu-lease-methodology.md`, which both carry + [#1129](https://github.com/mudler/vllm.cpp/issues/1129)'s reasoning as the live + cause of the oracle blocker. +- One appended row in `.agents/issue-index.md` for #1146. + +Out of scope: + +- Any measurement on `dgx:gpu0`. See `## What this does NOT establish`. +- Staging the pinned vLLM oracle. Only `torch` and `triton` are staged, and the + oracle is the thing #1129 actually blocks. +- Any product code. This row touches records and documents only. +- Any checker change. This row changes no checker semantics, so it owes no + red-before mutation under `## Changing the rules or a checker`. +- Editing #1129's existing index row. That file is append-only. + +## What #1129 recorded, and what is now false + +#1129 is closed. It records that no lease-compliant path can run a CUDA Python +runtime on this fleet, because the leased worker "cannot start Python". Its +Direction 2 probe recorded `python3=ABSENT`, `pip=ABSENT`, `gcc=ABSENT`, +`curl=ABSENT` and `git=ABSENT` inside the worker. + +That reading came from one probe on `dgx:gpu0`, job +`ff28ada1-0cd3-4867-bf9b-f67050d0608b`, taken on 2026-08-17. The worker image +measured here on `thor:gpu0` the same day reports the opposite for three of those +five names. The worker runs as `uid=0(root)`, carries `/usr/bin/gcc`, +`/usr/bin/python3` and `/usr/bin/apt-get`, and installs a package as root +(`APT_UPDATE_RC=0`, `APT_INSTALL_RC=0`). + +The measurement was taken correctly. The fleet changed under it. So the recorded +cause no longer holds, and the three fleet-side fixes #1129 names as the only +ways forward are not the only ways forward. The image is provisionable per job. + +## Measured, on `thor:gpu0`, 2026-08-17 + +``` +torch.__file__ = /workspace/oracle-probe/site/torch/__init__.py +torch.__version__= 2.13.0+cu130 +cuda available = True +device count = 1 +device 0 = NVIDIA Thor +capability = (11, 0) +CUBLAS_OK (bf16 1024x1024 matmul executed) +triton.__version__ = 3.7.1 +TRITON_JIT_OK = 4096.0 (expect 4096.0) PASS +PROBE5_RC=0 +``` + +## The four walls, in the order they appear + +1. **The runtime must be installed FROM the worker.** The submitting host here is + `x86_64` and the workers are `aarch64`, so a local `pip install --target` onto + the NAS writes the wrong architecture into the exact path the worker imports + from, and it fails there as a confusing import error. Run the `pip --target` + inside `rc run`. +2. **`Python.h` is absent.** Triton's JIT shells out to `/usr/bin/gcc` to build + `cuda_utils.c` and dies with `fatal error: Python.h: No such file or + directory`. `apt-get install -y python3-dev` succeeds as root, and the headers + copy to `/workspace/oracle-probe/pyhdr` (1.6 MB) so the next job does not + reinstall them. Put them on `CPATH`. +3. **The NAS grants no exec bit, and `TRITON_PTXAS_PATH` does not save you.** The + mount presents `file_mode=0664`, so Triton cannot execute its own + `ptxas-blackwell`. Setting `TRITON_PTXAS_PATH` to a `/tmp` copy is **not + sufficient**: it redirects only the plain `ptxas`, while Triton selects the + Blackwell variant from its own package directory. The whole `triton` package + (651 MB) has to sit on a filesystem that grants exec bits. +4. **So `PYTHONPATH` is ordered, not single.** `PYTHONPATH=/tmp/tp:/workspace/oracle-probe/site` + puts the exec-capable `triton` first and leaves the 4.5 GB of torch on the NAS + where it costs nothing to re-stage. + +## The working recipe + +```sh +# once per worker container +apt-get update -qq && apt-get install -y -qq python3-dev +mkdir -p /tmp/tp && cp -a /workspace/oracle-probe/site/triton /tmp/tp/ +chmod -R +x /tmp/tp/triton/backends/nvidia/bin/ + +export PYTHONPATH=/tmp/tp:/workspace/oracle-probe/site +export CPATH=/workspace/oracle-probe/pyhdr/python3.12:${CPATH:-} +``` + +## What this does NOT establish + +Read this section before you quote any line above it. Every claim here is +narrower than the sentence a reader wants to write from it. + +- **This is measured on `thor:gpu0` at capability (11,0) ONLY.** The GB10 is + `sm_121a` and is UNMEASURED. Nothing here licenses a claim about the Spark. + A probe is queued at lowest priority behind a human hold on `dgx:gpu0`. Until + that probe returns, `dgx:gpu0` keeps the reading its own last probe gave it. +- **The pinned vLLM oracle is NOT staged.** Only `torch` and `triton` are, so + this does not show that the oracle runs. vLLM at `555967922` is a source build + with compiled extensions, and it needs `nvcc`, which the worker still lacks. + #1129's consequence for the oracle-dependent rows is therefore NARROWED and not + closed. +- **The CUDA version skew is recorded as observed, not adjudicated.** The torch + wheel is `+cu130` while the staged `ptxas` reports `release 12.8, V12.8.93`. It + compiled and ran a correct kernel here. Nobody has read whether the skew + changes a numerical result, and this row does not. +- **`numpy` is absent** from the staged tree. Torch warns about it on every + import, and vLLM would require it. + +## Risks + +The one checker these edits can break is `test_gpu_lock_one_truth` (#777), which +requires exactly one `**GPU mutex:**` bullet in `.agents/environment.md`. These +edits add no second mutex statement and do not touch that bullet. + +The corrected sections say what one worker image did on one day. The worker image +can change again, in either direction, which is how this row came to exist. Each +section therefore carries its box, its date and its job IDs, so the next reader +can tell a stale reading from a wrong one. + +The narrower risk is a reader who takes the `thor` result for a fleet result. The +`## What this does NOT establish` section above exists for exactly that reader, +and the same scope statement rides in every record this row edits. + +## Gates + +```sh +scripts/agent-preflight.sh +``` + +The full preflight is the row gate. This row adds no test, because it adds no +behavior: it records a measurement and corrects three documents. The `rc` jobs +below are the evidence, and they are not reproducible in CI, which has no fleet +device. + +## Evidence + +Five `rc run` jobs on `thor:gpu0`, 2026-08-17. Each ran a script staged on the +NAS rather than fed on stdin, because wall 1 of the first probe was that a +`@triton.jit` function read from stdin has no source file. The sha256 values are +taken over the staged files at `/mnt/nas_share/rc/oracle-probe/`, which is the +same folder the worker reads as `/workspace/oracle-probe/`. + +| Job | Script | sha256 prefix | Result | +|---|---|---|---| +| `6f4bdb03-0162-4ed3-a922-f1a42da10fbd` | `probe.sh` | `9831e189910c` | torch imports, CUDA available, bf16 matmul runs, Triton refuses a stdin kernel | +| `9c0ebeac-8edb-4317-9112-2583bf85f38d` | `probe2.sh` | `0e9cd0f6d323` | wall 2: `fatal error: Python.h: No such file or directory` | +| `8beba132-c682-4b11-aa47-a682f2cce096` | `hdr.sh` | `fb02d1a1f45f` | `uid=0(root)`, `apt-get` installs `python3-dev`, headers staged, wall 3 appears as `PermissionError` on `ptxas-blackwell` | +| `f60d945f-f5ec-49c5-9452-c214a50a0043` | `hdr2.sh` | `37c9dcad4c64` | `TRITON_PTXAS_PATH` to a `/tmp` copy is insufficient, and the staged `ptxas` reports `release 12.8, V12.8.93` | +| `63c60a90-29b8-4115-88c5-d82be2126136` | `hdr3.sh` | `e11a540d23b8` | wall 4 removed, `TRITON_JIT_OK = 4096.0 PASS`, `PROBE5_RC=0` | + +The kernel each Triton job ran is `tritontest.py`, sha256 prefix `df9861c9f86d`. + +## Stop conditions + +- Stop if a correction needs a second `**GPU mutex:**` bullet. Return + `NEEDS_DECISION`. Never widen `test_gpu_lock_one_truth` to pass. +- Stop if a correction would state the `thor` result as a `dgx` result. The + scope limit is the point of this row. +- Stop if a correction needs an edit to an existing `.agents/issue-index.md` + row. That file is append-only. + +## Owed + +- [#1146](https://github.com/mudler/vllm.cpp/issues/1146) stays open. Re-run the + identical staged probe on `dgx:gpu0` at `sm_121a`. +- Stage the pinned vLLM oracle itself, which is what #1129 actually blocks. It + needs `nvcc` first. +- Read whether the `+cu130` and `12.8` skew changes a numerical result. +- Stage `numpy`, which vLLM requires and the current tree lacks. + +## Now + +The four walls and the working recipe are recorded here. +`.agents/environment.md`, `.agents/specs/mtp-k-gt-1.md` and +`.agents/specs/gpu-lease-methodology.md` no longer carry "cannot start Python" as +the live cause, and each now names the box and the date its reading came from. +The next step is the `dgx:gpu0` probe, which the coordinator has queued. From 699150fe66254e6353cf2de0eee0f282506225e8 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 18:35:28 +0000 Subject: [PATCH 2/7] record(ENV-LEASE-RUNTIME-STAGING): the records still named "cannot start Python" as the live cause (#1146) #1129 is closed and three records carried its reasoning forward as the reason the oracle cannot run under a lease. That reasoning came from one `dgx:gpu0` probe and the `thor:gpu0` worker image contradicts three of its five names, so a reader following those records concludes that staging is futile when it has been measured working. `.agents/environment.md` now scopes the ABSENT list to the box and the day that produced it, keeps the `dgx.casa` conclusion with its narrower reason, and gains "A relocated CUDA runtime starts on `thor:gpu0`" with the reading, the recipe and the scope limit. `.agents/specs/gpu-lease-methodology.md` corrects "it has no compiler, no downloader and no Python" in `## What a lease can carry`, and its `## Owed` and `## Now` no longer say the relocation is UNMEASURED. `.agents/specs/mtp-k-gt-1.md` is where the blocked row's owner looks, because #1129's index row names `SPEC-MTP-K-GT-1`. Its `## Owed` cell and its `## Now` now say that staging works on `thor:gpu0` and that this row still cannot resume, because the staged tree holds `torch` and `triton` and not the pinned oracle. Every one of these edits states the same three limits: `thor:gpu0` at capability (11,0) only, the GB10 `sm_121a` UNMEASURED, and the oracle not staged. The point of the row is that the correction must not be over-read. `.agents/issue-index.md` gains one appended row for #1146 naming `ENV-LEASE-RUNTIME-STAGING` as its owner. #1129's existing row is untouched, because the file is append-only and GitHub holds its closed state. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/environment.md | 70 ++++++++++++++++++++++---- .agents/issue-index.md | 1 + .agents/specs/gpu-lease-methodology.md | 30 ++++++++--- .agents/specs/mtp-k-gt-1.md | 17 +++++-- 4 files changed, 96 insertions(+), 22 deletions(-) diff --git a/.agents/environment.md b/.agents/environment.md index 8b0e45889..cf9c1cd8c 100644 --- a/.agents/environment.md +++ b/.agents/environment.md @@ -72,11 +72,14 @@ labels at all. `class=train` and `gpu_model=GB10` do match. `rc run` has no `--idle-timeout`, `--max-runtime`, `--no-wait`, `--priority`, `--select` and `--timeout`. -### What a leased worker can and cannot do, measured 2026-08-17 +### What the `dgx:gpu0` leased worker can and cannot do, measured 2026-08-17 Probed with one `rc run -d dgx:gpu0 --max-runtime 2m` job (`ff28ada1-0cd3-4867-bf9b-f67050d0608b`). Verify this again before you plan work -around it, because the worker image can change under you. +around it, because the worker image can change under you. **It did change.** The +`thor:gpu0` worker measured later the same day reports the opposite for three of +the names below, so read this section as one box on one day. The `thor` reading +is in "A relocated CUDA runtime starts on `thor:gpu0`" further down. - The command runs as user `rc` in a **k3s pod**, hostname `rc-worker-`. `/.dockerenv` is absent and 8 `KUBERNETES_*` variables are set, so it is a pod @@ -86,8 +89,11 @@ around it, because the worker image can change under you. `flock`, and **`/workspace`**. - **Absent: `gcc`, `cc`, `clang`, `nvcc`, `ninja`, `cmake`, `make`, `python3`, `python`, `pip`, `docker`, `sudo`, `git`, `ssh`, `curl`, - `/usr/include/stdio.h`, and any `/usr/local/cuda*` toolkit.** A worker cannot - compile, cannot start Python, and cannot install anything. + `/usr/include/stdio.h`, and any `/usr/local/cuda*` toolkit.** This `dgx:gpu0` + worker cannot compile, cannot start Python, and cannot install anything. + **Do not carry that clause to another device.** On `thor:gpu0` the same day the + worker ran as `uid=0(root)` with `/usr/bin/gcc`, `/usr/bin/python3` and a + working `apt-get` ([#1146](https://github.com/mudler/vllm.cpp/issues/1146)). - **The host filesystem is not visible.** `/home/mudler` does not exist inside the worker. - `/workspace` is the house NAS, measured as `//192.168.68.102/Data 7.3T total, @@ -109,8 +115,12 @@ lease carries bytes, and the exec bit is a mount option" below measures staged content starting under the dynamic loader and after a copy to `/tmp`, so what blocks the oracle is that nothing has put a runtime where a lease can see it. That is why recent GPU work reached for `ssh`, and the bypass is a symptom of -this gap rather than a discipline problem. Do not design the migration here. The -row that takes #1129 owns it. +this gap rather than a discipline problem. Do not design the migration here. +`ENV-LEASE-RUNTIME-STAGING` owns the design, and +[`lease-runtime-staging.md`](specs/lease-runtime-staging.md) holds the working +recipe. That recipe stages `torch` and `triton`, not the pinned oracle, and it +ran on `thor:gpu0` and not here, so the sentence above still stands for +`dgx.casa` today. **This confirms and extends a finding that already landed, rather than making a new one.** `.agents/specs/minimax-music3.md` §13.10 probed `thor`'s worker on @@ -166,18 +176,58 @@ staging: `cp`, `cat`, `tar`, `chmod`, `perl`, `flock` and `nvidia-smi`. not close it.** The pinned oracle stays unreachable because its virtual environment lives at `~/venvs/vllm-oracle-pin-555967922` on the dgx host, which no lease can see, and only a host-side actor reached over `ssh` can place a copy -on the NAS. Whether that copy then starts is UNMEASURED. A CUDA virtual -environment holds absolute paths in its shebangs and its `RECORD` files, so -treat the relocation as an open question rather than a solved step. +on the NAS. **Whether a relocated CUDA runtime then starts is no longer +UNMEASURED. It starts, on `thor:gpu0`.** The section below has the reading. A +CUDA virtual environment still holds absolute paths in its shebangs and its +`RECORD` files, so a `pip install --target` tree is the shape that was measured +and a copied venv is not. **Three fleet-side changes would each remove the staging problem, and none of -them is ours to make.** Whoever owns the fleet picks one. +them is ours to make.** Whoever owns the fleet picks one. **A fourth route was +then measured, and it needs nobody's permission:** the `thor:gpu0` worker runs +as root with a working `apt-get`, so a job provisions its own container. 1. The worker image gains a toolchain and a Python interpreter. 2. `rc run` gains an `--image` flag, so a job selects an image that has them. 3. `/workspace` is mounted so that a file there can carry an exec bit. This one removes the copy step only, because the two routes above already execute. +### A relocated CUDA runtime starts on `thor:gpu0`, measured 2026-08-17 + +Probed with five `rc run` jobs on `thor:gpu0`: `6f4bdb03`, `9c0ebeac`, +`8beba132`, `f60d945f` and `63c60a90`. A `torch` and `triton` tree staged on +`/workspace` imports, initializes CUDA, runs a bf16 matmul, and compiles and +executes a Triton kernel. The job IDs in full, the staged-script sha256 values, +the four walls and the working recipe are in +[`lease-runtime-staging.md`](specs/lease-runtime-staging.md) +([#1146](https://github.com/mudler/vllm.cpp/issues/1146)). + +``` +torch.__version__= 2.13.0+cu130 cuda available = True +device 0 = NVIDIA Thor capability = (11, 0) +triton.__version__ = 3.7.1 TRITON_JIT_OK = 4096.0 PASS +``` + +The recipe, once per worker container: + +```sh +apt-get update -qq && apt-get install -y -qq python3-dev +mkdir -p /tmp/tp && cp -a /workspace/oracle-probe/site/triton /tmp/tp/ +chmod -R +x /tmp/tp/triton/backends/nvidia/bin/ + +export PYTHONPATH=/tmp/tp:/workspace/oracle-probe/site +export CPATH=/workspace/oracle-probe/pyhdr/python3.12:${CPATH:-} +``` + +**Read the scope before you quote it.** This is `thor:gpu0` at capability (11,0) +and nothing else. The GB10 is `sm_121a` and is UNMEASURED, so nothing here +licenses a claim about the Spark. Only `torch` and `triton` are staged, so the +pinned vLLM oracle is still not shown to run: it is a source build with compiled +extensions and it needs `nvcc`, which the worker lacks. The torch wheel is +`+cu130` while the staged `ptxas` reports `release 12.8, V12.8.93`, and that skew +is recorded as observed rather than adjudicated. `numpy` is absent from the +staged tree. + ### The `flock` orphan hazard that motivated the replacement The harness family in this repository puts the `flock` handle on a **subshell**, diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 234ccfd2d..43d65da87 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -331,3 +331,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1117](https://github.com/mudler/vllm.cpp/issues/1117) | `LTX25-A2VID-RECIPE` | `A2VidPipelineTwoStage` (`a2vid_two_stage.py:53` @ `fd4ded7f`) has no recipe row, so `pipeline_kind = a2vid_two_stage` gets the generic table refusal (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1328-1332`) naming the pair rather than the missing machinery. [#922](https://github.com/mudler/vllm.cpp/issues/922) is CLOSED and closed the audio CONDITIONING, not the recipe: a supplied take rides `distilled_two_stage`, which [`ltx25-a2v-audio-input.md`](specs/ltx25-a2v-audio-input.md):438-446 already records as a different trajectory. Measured at `daeff67f2`: `git grep -n '"a2vid' -- src include tests docs examples` returns TWO hits, both upstream anchors inside `Fail`-message assertions (`tests/vllm/multimodal/test_ltx2_video.cpp:4363,:4427`), against a control of 4 for `"one_stage"` in `include/` alone. Four differences from the recipe it rides, each read at the pin: stage 1 is CFG/STG/modality-guided and caller-configured (`:230-240`, fed from `utils/args.py:947-1006`, `--a2v-guidance-scale` defaulting to `video_guider.modality_scale` = 3.0 at `utils/constants.py:54,:64`) where `distilled_two_stage` fixes `allow_guidance_override = false`; stage 1's schedule is scheduler-derived (`:225-227`) against our fixed `DistilledSigmas()`; stage 1 is plain Euler (`:229-258` passes no `stepper`, `utils/blocks.py:526-527`) against our `kEulerAncestral` on 2.5; and the AUDIO guider is the DEFAULT positive-only one (`:237-239`, `ltx-core components/guiders.py:200-210`) rather than the params table's cfg-7.0 row. Two non-schedule facts that must not be guessed: `--audio-path` is `required=True` (`:312-317`), and the distilled LoRA rides stage 2 ALONE (`:114` against `:107`) with `--distilled-lora` `required=True` (`utils/args.py:1140-1153`). Unblocked by `Ltx2GuidedDenoise` landing at `daeff67f2` (#1092/#1102), which [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) `## Owed` names this arm against. Spec [`ltx25-a2vid-recipe.md`](specs/ltx25-a2vid-recipe.md) | enhancement | | [#1118](https://github.com/mudler/vllm.cpp/issues/1118) | `LTX25-A2VID-RECIPE` | LoRA adapters fuse ONCE, at load, into ONE weight set: `src/vllm/multimodal/ltx2_video.cpp:816-820` is the only `dit_options.loras.push_back` in the tree and it runs under `if (!lora_path.empty())`, so every phase of every recipe sees the same fused `im.dit.weights`. Three upstream pipelines build TWO `DiffusionStage`s from the same checkpoint with DIFFERENT adapter sets at `fd4ded7f`: `a2vid_two_stage.py:107` against `:114-119`, `ti2vid_two_stages.py:140` against `:151`, and `ti2vid_two_stages_hq.py:154,:165` at two separate strengths (`:92-101`). `distilled.py:131` builds ONE stage set, which is why `distilled_two_stage`, `dfr` and `retake` have never needed this and no gate has ever asked. Consequence for the arm landing with #1117: that row mirrors `--distilled-lora required=True` (`utils/args.py:1140-1153`) by refusing an `a2vid_two_stage` load with no `lora_path`, so the 3-step stage-2 schedule cannot run on a checkpoint with no distilled adapter — but it CANNOT mirror upstream leaving stage 1 WITHOUT that adapter, so stage 1's guided schedule runs against base + distilled LoRA where upstream runs it against the base alone. That divergence RENDERS, and the PIXELS it renders are not upstream's: it moves the trajectory, so the frames themselves differ, while the frame count, the shapes, the sample rate and the errors are all exactly what they were — nothing in the SHAPE of the result says anything is wrong, which is why it is filed rather than left in a comment. It is not undetectable, and saying so would be the more damaging error: the instrument that WOULD see it is a real-weights comparison against upstream's own render on the same checkpoint, take and seed, upstream's stage 1 on the base weights against ours on base + distilled. Two fix shapes, neither chosen: a second `Ltx2DitWeights` per adapter set (what upstream pays, two `from_checkpoint` calls at `a2vid_two_stage.py:103,:115`) or unfused runtime LoRA selectable per phase; the first doubles resident DiT weights and the second changes the GEMM path. Bounds #1093 and #921, which need the same seam. Listed under `## Owed` in [`ltx25-a2vid-recipe.md`](specs/ltx25-a2vid-recipe.md) | enhancement | | [#1134](https://github.com/mudler/vllm.cpp/issues/1134) | `BACKEND-ROCM` | The `VT_ATTN_DECODE_D128=1` ctest registration added by #767 cannot show the ROCm `d=128` decode arm REACHED the new kernel, and is empty off ROCm. `RegisteredDevices()` (`tests/vt/test_backend_cross_device.cpp:84-96`) enumerates `{kCUDA, kMETAL, kVULKAN, kXPU, kROCM}` and excludes `kCPU`, so on a CPU-only runner — which is what CI has — the new "Qwen3 geometry (bf16, GQA 2, head_dim 128)" case reports 1 test case, 0 assertions, exit 0, for BOTH registrations. On ROCm hardware the case's only backend assertion is `OpProviderStats::declines == 0`, and `OpProviderStats` counts at PROVIDER granularity, so it is identical with the flag set and unset; the NMSE bound passes on either kernel because the arm is correctness-complete. The two compose: there is no machine in this project on which the flag-ON registration distinguishes itself from the flag-OFF one. Disclosed in [`specs/rocm-decode-attn-d128.md`](specs/rocm-decode-attn-d128.md) §4, its `## Owed` section and its result banner, and §9 stop condition 2 is left OPEN rather than claimed discharged. Closing it needs a kernel-selection counter in `src/vt/rocm/rocm_paged_attn.hip` asserted to DIFFER between the two registrations; the CPU-runner half wants `kCPU` in `RegisteredDevices()` or a non-zero-assertion floor per [#463](https://github.com/mudler/vllm.cpp/issues/463). Not a duplicate of #463 (that is the unset-weights-env-var shape and does not describe the `declines` granularity half), #785 (a kernel that never LAUNCHES behind a dead `#if`, a code defect not a coverage one) or #900 (same family, LTX-2.5 subject) | bug | +| [#1146](https://github.com/mudler/vllm.cpp/issues/1146) | `ENV-LEASE-RUNTIME-STAGING` | #1129 is closed and its recorded cause is FALSIFIED by the worker image: it says a leased worker "cannot start Python" and lists `python3`, `pip`, `gcc`, `curl` and `git` as ABSENT, measured in one `dgx:gpu0` probe (job `ff28ada1-0cd3-4867-bf9b-f67050d0608b`). Measured 2026-08-17 on `thor:gpu0` through five `rc run` jobs (`6f4bdb03`, `9c0ebeac`, `8beba132`, `f60d945f`, `63c60a90`), that worker runs as `uid=0(root)` with `/usr/bin/gcc`, `/usr/bin/python3` and a working `apt-get`, and a relocated CUDA runtime staged on `/workspace` imports torch 2.13.0+cu130, reports `cuda available = True` on `NVIDIA Thor` capability (11,0), runs a bf16 matmul, and compiles and executes a Triton kernel (`TRITON_JIT_OK = 4096.0 PASS`, `PROBE5_RC=0`). So the image is provisionable per job, which is none of the three fleet-side fixes #1129 names as the only ways forward. Four walls stand between a staged runtime and a running one: the `pip --target` must run FROM the worker because the submitting host is `x86_64` and the workers are `aarch64`. `Python.h` is absent until `apt-get install python3-dev`. The NAS mount presents `file_mode=0664` so Triton cannot execute its own `ptxas-blackwell`, and `TRITON_PTXAS_PATH` does NOT fix it because it redirects only the plain `ptxas`. So `PYTHONPATH` is ORDERED, `/tmp/tp` before the NAS tree. SCOPE, and it is the point of the row: this is `thor:gpu0` at capability (11,0) ONLY, the GB10 is `sm_121a` and UNMEASURED, only `torch` and `triton` are staged so the pinned vLLM oracle is still NOT shown to run and #1129's consequence for the oracle-dependent rows is NARROWED rather than closed, the `+cu130` versus `release 12.8, V12.8.93` skew is recorded as observed and not adjudicated, and `numpy` is absent from the staged tree. Recipe, job IDs and staged-script sha256 values in [`lease-runtime-staging.md`](specs/lease-runtime-staging.md) | verification | diff --git a/.agents/specs/gpu-lease-methodology.md b/.agents/specs/gpu-lease-methodology.md index ce3e30c75..79be20f18 100644 --- a/.agents/specs/gpu-lease-methodology.md +++ b/.agents/specs/gpu-lease-methodology.md @@ -92,8 +92,19 @@ The limit is now precise, measured on 2026-08-17 through two carries bytes and not executables. The worker reads and writes the shared `/workspace`, refuses direct execution from it because the mount pins `file_mode=0664`, and runs staged content through `sh FILE`, through the dynamic -loader, or after a copy to `/tmp`. It cannot produce or fetch a runtime, because -it has no compiler, no downloader and no Python. +loader, or after a copy to `/tmp`. + +**The last clause of this section said the worker cannot produce or fetch a +runtime, because it has no compiler, no downloader and no Python. That is a +`dgx:gpu0` reading and it does not generalise.** Later the same day, five +`rc run` jobs on `thor:gpu0` measured a worker running as `uid=0(root)` with +`/usr/bin/gcc`, `/usr/bin/python3` and a working `apt-get`, and a `torch` and +`triton` tree staged on `/workspace` imported, initialized CUDA and compiled and +ran a Triton kernel. See +[`lease-runtime-staging.md`](lease-runtime-staging.md) +([#1146](https://github.com/mudler/vllm.cpp/issues/1146)), which also states what +that result does not establish: it is `thor:gpu0` at capability (11,0) only, the +GB10 is `sm_121a` and UNMEASURED, and the pinned vLLM oracle is not staged. ## The correction has to reach the spec that owns the blocker @@ -168,9 +179,12 @@ is the row gate. ## Owed -- #1129 stays open. No vLLM leg runs on `dgx.casa` by a lease-compliant path - today, because nothing has staged a runtime on the NAS. Whether a relocated - CUDA virtual environment starts inside a worker is UNMEASURED. +- #1129 is now closed, and its recorded cause was falsified on 2026-08-17. No + vLLM leg runs on `dgx.casa` by a lease-compliant path today, because nothing + has staged the ORACLE on the NAS. The "UNMEASURED" clause this line used to + carry is answered: a relocated CUDA runtime does start inside a worker, on + `thor:gpu0` and not yet on `dgx:gpu0`. `ENV-LEASE-RUNTIME-STAGING` and + [#1146](https://github.com/mudler/vllm.cpp/issues/1146) own the rest. ## Now @@ -178,5 +192,7 @@ The rule is stated in `AGENTS.md` and the conditional is in `.agents/environment.md`, keyed on the device and naming the three fleet devices. The narrowing of #1129 now reads the same way in `.agents/environment.md` and in `.agents/specs/mtp-k-gt-1.md`, so the blocked -row's owner is told that staging is untried rather than futile. The next step -belongs to whoever takes #1129, which is staging a runtime the lease can start. +row's owner is told that staging is untried rather than futile. Staging was then +tried, and it worked on `thor:gpu0` +([`lease-runtime-staging.md`](lease-runtime-staging.md), #1146). The next step is +the same probe on `dgx:gpu0`, and after it the pinned oracle itself. diff --git a/.agents/specs/mtp-k-gt-1.md b/.agents/specs/mtp-k-gt-1.md index 77b6b924a..69ce06c95 100644 --- a/.agents/specs/mtp-k-gt-1.md +++ b/.agents/specs/mtp-k-gt-1.md @@ -719,7 +719,7 @@ upstream's own supported configuration. | Owed | What it must show | Who | |---|---|---| | DGX three-way greedy gate at k=2, 3, 4 on Qwen3.6-27B and 35B | **PART-PAID 2026-08-16, and the remaining half is the vLLM leg.** On the 27B NVFP4 at the DEFAULT bf16 GDN state, depth REACHES the verify path at k=2, 3 and 4 on real weights and the per-depth counters are populated at EVERY depth up to k. What is NOT established is `our-ON == our-OFF`: it is FALSE here on 3 of 4 prompts, at the SAME token positions for every k and for the padded control alike, which is the signature of a fixed spec-ON/OFF difference rather than a depth defect. Attributing it needs the oracle leg, which did not run. Do NOT read this row as a passed token gate. **2026-08-17 narrowed it to 3 forwards.** Only the FIRST divergence per arm and prompt is adjudicable, which reduces 1718 divergent positions to 18 and then to 3 distinct probe points. Prompt 1 position 1 resolves to THREE different tokens under three values of k, which no depth defect can produce. `scripts/mtp-k-gt-1-neartie-gap.py` is committed and decides each candidate against `kNearTieMnats = 500`. It still has not RUN: the oracle cannot be loaded while a foreign multi-tens-of-GiB container is resident. **2026-08-17 third pass: the box was CLEAN, the run happened, and the divergence REPRODUCED exactly on independently generated streams (1718 positions, 18 adjudicable, 3 probe points, same tokens), so it is deterministic rather than a flake. The adjudication STILL did not report.** Its failure was an INSTRUMENT failure and is recorded as such rather than as a verdict: the reimaged host carries no C compiler at all, so Triton's JIT died after the weights loaded and vLLM surfaced it as `Engine core initialization failed`. Repaired by running the oracle inside a container carrying the toolchain, which is MEASURED loading the engine past that point; the box then stopped answering SSH mid-leg. Do NOT read this row as a passed token gate | `SPEC-MTP-K-GT-1`, [#81](https://github.com/mudler/vllm.cpp/issues/81) M1 | -| **THE BLOCKER: the pinned oracle cannot load a 27B on `dgx.casa` at all, and `gpu_memory_utilization` is NOT the lever** | Once the toolchain fix let an oracle reach this step for the first time, it consumed the entire host in the step AFTER `torch.compile`. **Measured 2026-08-17 at 0.75: about 110 GiB of HOST RAM held while `nvidia-smi` reported 26 GiB on the device, 45 minutes hung at loadavg 260 with 0 GiB available, its own timeout firing (`ADJUDICATE_EXIT=124`); killing the container took the box from 118 of 119 GiB used to 4 of 119 in under ten seconds.** The attribution to that 0.75 was then **TESTED AND REFUTED**: a third window ran the byte-identical instrument at `--gpu-mem-util 0.30` with a 5-second `MemAvailable` sampler and collapsed the same way (87683 MB free at 09:00:47, **0** at 09:02:25). Weight loading finished with 66 GiB free and compilation with 88 GiB free, so it is neither. **The 0.30 run also REBOOTED the box while the 0.75 run only thrashed** (`boot_id` `5bbdc432...` to `bd5c6e7a...`, `journalctl --list-boots` gap 09:10:15Z to 09:13:55Z), so a lower fraction is not a safety margin. Owed: identify the actual step by varying `max_num_batched_tokens` and `cudagraph_capture_sizes` ONE AT A TIME with the sampler running, then record a configuration that demonstrably reaches KV-cache allocation and completes a generate, in `.agents/environment.md` beside the toolchain recipe. Until then no vLLM leg of any row can run on this host. **The container re-check this row left open is now DONE, and it CONFIRMS the blocker rather than refuting it ([#1129](https://github.com/mudler/vllm.cpp/issues/1129), 2026-08-17).** The host recipe reaches its container over `ssh`, which bypasses the `rc` lease and makes the fleet report `dgx:gpu0` free while somebody is on it. The lease-compliant route was then probed directly, and the leased worker carries no `gcc`, `nvcc`, `ninja`, `cmake`, `python3`, `pip`, `docker` or `sudo`, no `/usr/include/stdio.h` and no `/usr/local/cuda*`, and it cannot see `/home/mudler`, so it cannot reach `~/venvs/vllm-oracle-pin-555967922`. `rc run` has no `--image` flag. **A second probe on the same day NARROWED that reading and did not close it, and one clause of it was measured FALSE.** The lease carries bytes, and bytes are enough to run: `/lib/ld-linux-aarch64.so.1 /workspace/` started a staged ELF, and a copy to `/tmp` followed by `chmod +x` started both a staged script and a staged ELF, each exit 0 (`.agents/environment.md`, "The lease carries bytes, and the exec bit is a mount option"). The clause "could not start it if it could" is therefore RETRACTED here. What holds is narrower and is still a blocker: **no vLLM leg of any row runs on `dgx.casa` by a lease-compliant path today, because nothing has staged a runtime on the NAS**, and only a host-side actor reached over `ssh` can place one there. Whether a relocated CUDA virtual environment then starts inside a worker is UNMEASURED, because such an environment holds absolute paths in its shebangs and its `RECORD` files, so read staging as untried rather than futile. The blocker still widens past #81 to every oracle-dependent row, [#1003](https://github.com/mudler/vllm.cpp/issues/1003), [#915](https://github.com/mudler/vllm.cpp/issues/915) and [#821](https://github.com/mudler/vllm.cpp/issues/821) included. The migration is NOT designed here. The only recorded lever is `/workspace`, which is NAS-backed, writable from the worker and the same folder on `dgx` and `thor`. #1129 owns the design | `SPEC-MTP-K-GT-1`, [#81](https://github.com/mudler/vllm.cpp/issues/81) M1, [#1129](https://github.com/mudler/vllm.cpp/issues/1129) | +| **THE BLOCKER: the pinned oracle cannot load a 27B on `dgx.casa` at all, and `gpu_memory_utilization` is NOT the lever** | Once the toolchain fix let an oracle reach this step for the first time, it consumed the entire host in the step AFTER `torch.compile`. **Measured 2026-08-17 at 0.75: about 110 GiB of HOST RAM held while `nvidia-smi` reported 26 GiB on the device, 45 minutes hung at loadavg 260 with 0 GiB available, its own timeout firing (`ADJUDICATE_EXIT=124`); killing the container took the box from 118 of 119 GiB used to 4 of 119 in under ten seconds.** The attribution to that 0.75 was then **TESTED AND REFUTED**: a third window ran the byte-identical instrument at `--gpu-mem-util 0.30` with a 5-second `MemAvailable` sampler and collapsed the same way (87683 MB free at 09:00:47, **0** at 09:02:25). Weight loading finished with 66 GiB free and compilation with 88 GiB free, so it is neither. **The 0.30 run also REBOOTED the box while the 0.75 run only thrashed** (`boot_id` `5bbdc432...` to `bd5c6e7a...`, `journalctl --list-boots` gap 09:10:15Z to 09:13:55Z), so a lower fraction is not a safety margin. Owed: identify the actual step by varying `max_num_batched_tokens` and `cudagraph_capture_sizes` ONE AT A TIME with the sampler running, then record a configuration that demonstrably reaches KV-cache allocation and completes a generate, in `.agents/environment.md` beside the toolchain recipe. Until then no vLLM leg of any row can run on this host. **The container re-check this row left open is now DONE, and it CONFIRMS the blocker rather than refuting it ([#1129](https://github.com/mudler/vllm.cpp/issues/1129), 2026-08-17).** The host recipe reaches its container over `ssh`, which bypasses the `rc` lease and makes the fleet report `dgx:gpu0` free while somebody is on it. The lease-compliant route was then probed directly, and the `dgx:gpu0` leased worker carries no `gcc`, `nvcc`, `ninja`, `cmake`, `python3`, `pip`, `docker` or `sudo`, no `/usr/include/stdio.h` and no `/usr/local/cuda*`, and it cannot see `/home/mudler`, so it cannot reach `~/venvs/vllm-oracle-pin-555967922`. That list is one box on one day and it does NOT generalise across the fleet, which the third probe below measured. `rc run` has no `--image` flag. **A second probe on the same day NARROWED that reading and did not close it, and one clause of it was measured FALSE.** The lease carries bytes, and bytes are enough to run: `/lib/ld-linux-aarch64.so.1 /workspace/` started a staged ELF, and a copy to `/tmp` followed by `chmod +x` started both a staged script and a staged ELF, each exit 0 (`.agents/environment.md`, "The lease carries bytes, and the exec bit is a mount option"). The clause "could not start it if it could" is therefore RETRACTED here. What holds is narrower and is still a blocker: **no vLLM leg of any row runs on `dgx.casa` by a lease-compliant path today, because nothing has staged a runtime on the NAS**, and only a host-side actor reached over `ssh` can place one there. **A THIRD probe the same day answered the UNMEASURED clause, and it is narrower than it sounds.** A relocated CUDA runtime staged on `/workspace` DOES start inside a worker: on `thor:gpu0`, five `rc run` jobs imported torch 2.13.0+cu130, reported `cuda available = True` on `NVIDIA Thor` capability (11,0), ran a bf16 matmul and compiled and executed a Triton kernel (`TRITON_JIT_OK = 4096.0 PASS`). That worker runs as `uid=0(root)` with a working `apt-get` and `/usr/bin/gcc`, so #1129's recorded "cannot start Python" cause no longer holds for that image ([#1146](https://github.com/mudler/vllm.cpp/issues/1146), [`lease-runtime-staging.md`](lease-runtime-staging.md)). What it does NOT establish is what this row needs. It is `thor:gpu0` at capability (11,0) ONLY, and the GB10 is `sm_121a` and UNMEASURED. Only `torch` and `triton` are staged, NOT the pinned oracle, which is a source build with compiled extensions and needs `nvcc`. So the vLLM leg is still blocked here, by a narrower and now precisely named reason. The blocker still widens past #81 to every oracle-dependent row, [#1003](https://github.com/mudler/vllm.cpp/issues/1003), [#915](https://github.com/mudler/vllm.cpp/issues/915) and [#821](https://github.com/mudler/vllm.cpp/issues/821) included. The migration is NOT designed here. The only recorded lever is `/workspace`, which is NAS-backed, writable from the worker and the same folder on `dgx` and `thor`. `ENV-LEASE-RUNTIME-STAGING` owns the design | `SPEC-MTP-K-GT-1`, [#81](https://github.com/mudler/vllm.cpp/issues/81) M1, [#1129](https://github.com/mudler/vllm.cpp/issues/1129) | | The cleanup trap in the DGX drivers does not stop the run | `run_all_inner.sh` and `run_oracle_inner.sh` both use `trap cleanup EXIT INT TERM` where `cleanup` resets the clocks and returns WITHOUT exiting, so bash resumes the script after the handler. **Observed 2026-08-17: `SIGTERM` reset the clocks and the driver then started its next leg, which immediately began re-filling a box that had 0 GiB available.** The chain had to be `SIGKILL`ed and the container stopped separately. A cleanup trap that does not terminate is not a stop button, and on a box that reboots rather than OOM-killing that difference is the box. Owed: `exit` from the signal path, and a `docker kill` of the current leg's named container inside `cleanup` so the container cannot outlive its driver | `SPEC-MTP-K-GT-1`, [#81](https://github.com/mudler/vllm.cpp/issues/81) M1 | | The PADDED CONTROL arm of that gate, and the RATE assertion it carries | **PAID 2026-08-16 on the 27B, and its throughput VOID LIFTED 2026-08-17.** The first pass could not quote `padded_k3`/`padded_k4` throughput because they started at loadavg 10.77 and 20.41 against real arms at 1.5 to 2.9. The third pass re-ran all seven arms inside ONE window in a load band of 0.16 to 1.86, every leg exit 0, so the real-arm against padded-control comparison is now quotable rather than merely computed. Detail below. Margin fixed BEFORE the run at 0.10 absolute per depth. The real loop accepts at 0.507 to 0.750 at every depth >= 1. The padded control accepts at 0.000 at every depth >= 1 while its depth-0 rate MATCHES the real arm (0.892 to 0.925 against 0.868 to 0.878), which is what a control that isolates columns >= 1 must look like. Every margin clears by at least 0.41. The control measuring 0 is recorded as a fact about THIS prompt set and did not license restoring the count assertion. Still owed on the 35B | `SPEC-MTP-K-GT-1`, [#81](https://github.com/mudler/vllm.cpp/issues/81) M1 | | Silent de-graphing when the actual depth differs from the configured k, AND the `S`-only slot-ring key ([#1020](https://github.com/mudler/vllm.cpp/issues/1020)) | The spec-graph predicate reads the step's ACTUAL uniform query length instead of `num_spec()` (`runner.cpp:1383`), and the graph slot ring is keyed on `(S, q)` in the SAME change. The re-key is owed on its own merits and NOT only as a consequence of widening the predicate, which is the correction section 4.2a records: `uniform_decode = input.pure_decode \|\| (spec_graph && ...)` (`qwen3_5_moe.cpp:143-148`, `qwen3_5_dense.cpp:172-177`) already routes TWO query lengths to one `impl_->slots[S]` (`qwen3_5.cpp:9281`, dense `:9708`), and `SizeSlot` invalidates on `fa_cols` and `aux_taps` only (`:9309-9316` and `:9361-9367`). At k=1, 8 requests pure-decode and 4 requests spec both key on `S = 8`. That is pre-existing since SPEC-DSPARK W8 (#442) and this row does not widen it, but it is not the benign thing the first spec revision claimed. Plus a measured before-and-after on the capture-set size and persistent logits memory, and a counter or log for the eager fallback so it can never again be invisible | `SPEC-MTP-K-GT-1`, [#1020](https://github.com/mudler/vllm.cpp/issues/1020) | @@ -753,10 +753,17 @@ vLLM leg of any row runs on `dgx.casa` by a lease-compliant path today because nothing has staged a runtime on the NAS, and NOT because a lease cannot execute one ([#1129](https://github.com/mudler/vllm.cpp/issues/1129), and `.agents/environment.md` "The lease carries bytes, and the exec bit is a mount -option"). Whether a relocated CUDA virtual environment starts inside a worker is -UNMEASURED, so staging is worth attempting rather than futile. Resume this row -only after #1129 has a path, because the adjudication instrument is committed and -correct and has nothing to run on. +option"). **A third probe then answered the UNMEASURED clause: staging works.** +On `thor:gpu0`, a `torch` and `triton` tree staged on `/workspace` imports, +initializes CUDA and compiles and runs a Triton kernel, and that worker is root +with `apt-get` and `gcc`, so #1129's "cannot start Python" cause is falsified for +that image ([#1146](https://github.com/mudler/vllm.cpp/issues/1146), +[`lease-runtime-staging.md`](lease-runtime-staging.md)). This row still cannot +resume on it. The result is `thor:gpu0` at capability (11,0) only, the GB10 is +`sm_121a` and UNMEASURED, and the staged tree holds `torch` and `triton` and not +the pinned oracle, which needs `nvcc`. Resume this row only after the oracle +itself has a lease-compliant path, because the adjudication instrument is +committed and correct and has nothing to run on. ## Outcome (partial, CPU half) From d1f07a29d76f3dbb935afe7d39345bb78417e1b8 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 18:38:33 +0000 Subject: [PATCH 3/7] record(ENV-LEASE-RUNTIME-STAGING): one clause still said a worker cannot fetch a runtime (#1146) "The lease carries bytes" ended by naming what the worker cannot do, and that sentence had no box on it. It is the `dgx:gpu0` reading, and the `thor:gpu0` worker does produce a runtime, because it is root and carries `apt-get` and `gcc`. The clause now names its box and points at the section that measured the other one. Found while re-reading the file after the surrounding corrections landed, which is why it rides here rather than in the commit that made it stale. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/environment.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/.agents/environment.md b/.agents/environment.md index cf9c1cd8c..e32591915 100644 --- a/.agents/environment.md +++ b/.agents/environment.md @@ -168,9 +168,11 @@ job's working directory `/` is not writable. **So the lease carries bytes, and bytes are enough to run.** A runtime staged on `/workspace` can start under the dynamic loader, or after a copy to `/tmp`. What -the worker cannot do is produce or fetch that runtime, because it has no `curl`, -`wget`, `git`, `gcc`, `nvcc`, `cmake` or `python3`. Present and useful for -staging: `cp`, `cat`, `tar`, `chmod`, `perl`, `flock` and `nvidia-smi`. +this `dgx:gpu0` worker cannot do is produce or fetch that runtime, because it has +no `curl`, `wget`, `git`, `gcc`, `nvcc`, `cmake` or `python3`. Present and useful +for staging: `cp`, `cat`, `tar`, `chmod`, `perl`, `flock` and `nvidia-smi`. **The +`thor:gpu0` worker does produce one**, because it is root and carries `apt-get` +and `gcc`. That is the section two below. **This narrows [#1129](https://github.com/mudler/vllm.cpp/issues/1129) and does not close it.** The pinned oracle stays unreachable because its virtual From a4b66622916350949bb075e2c29c4d0056a149bf Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 18:39:21 +0000 Subject: [PATCH 4/7] spec(ENV-LEASE-RUNTIME-STAGING): the stdin failure is a harness detail, not a fifth wall (#1146) The `## Evidence` preamble called the first probe's failure "wall 1", which collides with the numbered wall 1 in `## The four walls` and would leave a reader counting five. It is neither: `@triton.jit` calls `inspect.getsourcelines`, which raises `OSError: could not get source code` for a function read from stdin, so every job after the first ran a staged script. The preamble now says that, and says it is a harness detail. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/lease-runtime-staging.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/.agents/specs/lease-runtime-staging.md b/.agents/specs/lease-runtime-staging.md index cb4abe1f0..9748ff667 100644 --- a/.agents/specs/lease-runtime-staging.md +++ b/.agents/specs/lease-runtime-staging.md @@ -148,11 +148,13 @@ device. ## Evidence -Five `rc run` jobs on `thor:gpu0`, 2026-08-17. Each ran a script staged on the -NAS rather than fed on stdin, because wall 1 of the first probe was that a -`@triton.jit` function read from stdin has no source file. The sha256 values are -taken over the staged files at `/mnt/nas_share/rc/oracle-probe/`, which is the -same folder the worker reads as `/workspace/oracle-probe/`. +Five `rc run` jobs on `thor:gpu0`, 2026-08-17. Every job after the first ran a +script staged on the NAS rather than fed on stdin. That is a harness detail and +not one of the four walls: `@triton.jit` calls `inspect.getsourcelines`, which +raises `OSError: could not get source code` on a function read from stdin. The +sha256 values are taken over the staged files at +`/mnt/nas_share/rc/oracle-probe/`, which is the same folder the worker reads as +`/workspace/oracle-probe/`. | Job | Script | sha256 prefix | Result | |---|---|---|---| From 054af38d929e31e24df23bc9325793709cefd9f5 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 18:40:53 +0000 Subject: [PATCH 5/7] record(ENV-LEASE-RUNTIME-STAGING): the narrative section still stated UNMEASURED as the current form (#1146) `## The correction has to reach the spec that owns the blocker` recounts what the previous row wrote, and it ends by naming the calibrated form both sites take, including "the relocated virtual environment is UNMEASURED". A reader reaches that sentence as a statement about today. It now carries the answer beside it, and it names what is still not staged, which is the oracle. Also corrects a pointer in `.agents/environment.md`: the `thor:gpu0` reading is the section immediately below, not two below. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/environment.md | 2 +- .agents/specs/gpu-lease-methodology.md | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/.agents/environment.md b/.agents/environment.md index e32591915..d6e5b3d44 100644 --- a/.agents/environment.md +++ b/.agents/environment.md @@ -172,7 +172,7 @@ this `dgx:gpu0` worker cannot do is produce or fetch that runtime, because it ha no `curl`, `wget`, `git`, `gcc`, `nvcc`, `cmake` or `python3`. Present and useful for staging: `cp`, `cat`, `tar`, `chmod`, `perl`, `flock` and `nvidia-smi`. **The `thor:gpu0` worker does produce one**, because it is root and carries `apt-get` -and `gcc`. That is the section two below. +and `gcc`. That is the section below. **This narrows [#1129](https://github.com/mudler/vllm.cpp/issues/1129) and does not close it.** The pinned oracle stays unreachable because its virtual diff --git a/.agents/specs/gpu-lease-methodology.md b/.agents/specs/gpu-lease-methodology.md index 79be20f18..5cd8c11eb 100644 --- a/.agents/specs/gpu-lease-methodology.md +++ b/.agents/specs/gpu-lease-methodology.md @@ -124,6 +124,11 @@ green. Both sites therefore take the calibrated form that close it, the relocated virtual environment is UNMEASURED, and the load-bearing reason is that nothing has staged a runtime on the NAS. +**The UNMEASURED clause in that form was answered on the same day, and both sites +were corrected again.** A relocated runtime does start inside a worker, on +`thor:gpu0`. What is still not staged is the ORACLE. See +[`lease-runtime-staging.md`](lease-runtime-staging.md) and #1146. + The same substitution repairs the derivation in the how-to. The old sentence read "no host toolchain, the worker has no compiler, SO no lease-compliant path". Those premises stopped entailing that conclusion once the later probe From 1ea99c2a90ee3d904381c16af469ea92bb7d3d91 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 18:42:31 +0000 Subject: [PATCH 6/7] spec(ENV-LEASE-RUNTIME-STAGING): name the two probed binaries instead of counting three (#1146) The spec and `.agents/environment.md` both said the `thor:gpu0` worker reports the opposite for "three" of #1129's five ABSENT names. Two were probed there, `python3` and `gcc`, and `apt-get` is a third binary that #1129's list never named. `pip`, `curl` and `git` were not probed on `thor:gpu0` at all. A count nobody can check is the shape of claim this row exists to remove, so both sites now name the binaries and say which names this row is silent about. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/environment.md | 6 +++--- .agents/specs/lease-runtime-staging.md | 9 +++++---- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/.agents/environment.md b/.agents/environment.md index d6e5b3d44..a421300c6 100644 --- a/.agents/environment.md +++ b/.agents/environment.md @@ -77,9 +77,9 @@ labels at all. `class=train` and `gpu_model=GB10` do match. `rc run` has no Probed with one `rc run -d dgx:gpu0 --max-runtime 2m` job (`ff28ada1-0cd3-4867-bf9b-f67050d0608b`). Verify this again before you plan work around it, because the worker image can change under you. **It did change.** The -`thor:gpu0` worker measured later the same day reports the opposite for three of -the names below, so read this section as one box on one day. The `thor` reading -is in "A relocated CUDA runtime starts on `thor:gpu0`" further down. +`thor:gpu0` worker measured later the same day carries `python3` and `gcc`, which +this list calls absent, so read this section as one box on one day. The `thor` +reading is in "A relocated CUDA runtime starts on `thor:gpu0`" further down. - The command runs as user `rc` in a **k3s pod**, hostname `rc-worker-`. `/.dockerenv` is absent and 8 `KUBERNETES_*` variables are set, so it is a pod diff --git a/.agents/specs/lease-runtime-staging.md b/.agents/specs/lease-runtime-staging.md index 9748ff667..3b703cce4 100644 --- a/.agents/specs/lease-runtime-staging.md +++ b/.agents/specs/lease-runtime-staging.md @@ -41,10 +41,11 @@ Direction 2 probe recorded `python3=ABSENT`, `pip=ABSENT`, `gcc=ABSENT`, That reading came from one probe on `dgx:gpu0`, job `ff28ada1-0cd3-4867-bf9b-f67050d0608b`, taken on 2026-08-17. The worker image -measured here on `thor:gpu0` the same day reports the opposite for three of those -five names. The worker runs as `uid=0(root)`, carries `/usr/bin/gcc`, -`/usr/bin/python3` and `/usr/bin/apt-get`, and installs a package as root -(`APT_UPDATE_RC=0`, `APT_INSTALL_RC=0`). +measured here on `thor:gpu0` the same day reports the opposite for two of those +five names, `python3` and `gcc`. It runs as `uid=0(root)`, carries +`/usr/bin/python3`, `/usr/bin/gcc` and `/usr/bin/apt-get`, and installs a package +as root (`APT_UPDATE_RC=0`, `APT_INSTALL_RC=0`). `pip`, `curl` and `git` were not +probed on `thor:gpu0`, so this row says nothing about them. The measurement was taken correctly. The fleet changed under it. So the recorded cause no longer holds, and the three fleet-side fixes #1129 names as the only From ae9ead8bafe1462501dd69cd79c7a3e11e0df412 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 18:57:05 +0000 Subject: [PATCH 7/7] record(ENV-LEASE-RUNTIME-STAGING): numpy is staged, and no prebuilt wheel carries our pin (#1146) Two measurements landed after this spec was written, and each falsifies a line in it. Job `fd5654c0-d522-498c-8800-ca4df9a36944` on `thor:gpu0` carries both. **`numpy` is no longer absent.** `python3 -m pip install --target` put it in the same staged tree: `NUMPY_RC=0`, then `numpy 2.5.2` importing from `/workspace/oracle-probe/site/numpy/__init__.py`, `NUMPY_IMPORT_RC=0`. The five earlier jobs ran before that install, which is why each of their logs carries torch's `Failed to initialize NumPy` warning. The spec now says so, so a reader who still sees that warning knows they are reading a different tree. The `## Owed` line asking for numpy is removed rather than left to read as debt. **A prebuilt wheel does not remove the `nvcc` requirement.** The spec said the oracle needs `nvcc`, and a reader could reasonably ask whether a wheel sidesteps that. It does not, and the new section says why with the evidence separated by strength. An aarch64 vLLM wheel exists in general, measured: `pip download --no-deps vllm` fetched a 307,180,998-byte `manylinux_2_28_aarch64` wheel, `VLLM_DL_RC=0`, on a worker reporting `aarch64 linux-aarch64`. Our pin is not reachable that way, because `https://wheels.vllm.ai/nightly/vllm/` lists four wheels for exactly one commit and is a moving pointer rather than an archive, and because `0.23.1rc1.dev1511+g555967922` is a development version that is not on PyPI. The four 404s under a per-commit URL scheme are recorded as carrying NO weight. That scheme was never confirmed against a known-good case, and the host's own root 404s while `/nightly` returns 200. The load-bearing reading is the nightly index listing one commit. Nobody established that vLLM never retains per-commit wheels, and this record does not say so. Also separates the two boxes' evidence, in the same spirit as the earlier ABSENT-names correction. `python3` and `gcc` were probed on `thor:gpu0` and `pip` now with them. On `dgx:gpu0`, job `609c4944` invoked `python3 -m pip install --target` and then hit `max_runtime exceeded (35m0s)`, so it proves `pip` STARTS there and not that its install finished. `curl` and `git` were probed on neither. `.agents/issue-index.md` keeps exactly one #1146 row. Its scope clause is corrected in place rather than appended twice, because the row has never been on `main`, so no union merge can see two versions of it, and a second row for the same issue is what `check_issue_index` refuses as a duplicate. No landed row is touched. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/environment.md | 33 ++++--- .agents/issue-index.md | 2 +- .agents/specs/lease-runtime-staging.md | 119 ++++++++++++++++++++----- .agents/specs/mtp-k-gt-1.md | 11 ++- 4 files changed, 126 insertions(+), 39 deletions(-) diff --git a/.agents/environment.md b/.agents/environment.md index a421300c6..2f4dd52ec 100644 --- a/.agents/environment.md +++ b/.agents/environment.md @@ -196,11 +196,11 @@ as root with a working `apt-get`, so a job provisions its own container. ### A relocated CUDA runtime starts on `thor:gpu0`, measured 2026-08-17 -Probed with five `rc run` jobs on `thor:gpu0`: `6f4bdb03`, `9c0ebeac`, -`8beba132`, `f60d945f` and `63c60a90`. A `torch` and `triton` tree staged on -`/workspace` imports, initializes CUDA, runs a bf16 matmul, and compiles and -executes a Triton kernel. The job IDs in full, the staged-script sha256 values, -the four walls and the working recipe are in +Probed with six `rc run` jobs on `thor:gpu0`: `6f4bdb03`, `9c0ebeac`, `8beba132`, +`f60d945f`, `63c60a90` and `fd5654c0`. A `torch`, `triton` and `numpy` tree +staged on `/workspace` imports, initializes CUDA, runs a bf16 matmul, and +compiles and executes a Triton kernel. The job IDs in full, the staged-script +sha256 values, the four walls and the working recipe are in [`lease-runtime-staging.md`](specs/lease-runtime-staging.md) ([#1146](https://github.com/mudler/vllm.cpp/issues/1146)). @@ -223,12 +223,23 @@ export CPATH=/workspace/oracle-probe/pyhdr/python3.12:${CPATH:-} **Read the scope before you quote it.** This is `thor:gpu0` at capability (11,0) and nothing else. The GB10 is `sm_121a` and is UNMEASURED, so nothing here -licenses a claim about the Spark. Only `torch` and `triton` are staged, so the -pinned vLLM oracle is still not shown to run: it is a source build with compiled -extensions and it needs `nvcc`, which the worker lacks. The torch wheel is -`+cu130` while the staged `ptxas` reports `release 12.8, V12.8.93`, and that skew -is recorded as observed rather than adjudicated. `numpy` is absent from the -staged tree. +licenses a claim about the Spark. Only `torch`, `triton` and `numpy` are staged, +so the pinned vLLM oracle is still not shown to run: it is a source build with +compiled extensions and it needs `nvcc`, which the worker lacks. The torch wheel +is `+cu130` while the staged `ptxas` reports `release 12.8, V12.8.93`, and that +skew is recorded as observed rather than adjudicated. + +**A prebuilt wheel does not remove the `nvcc` requirement, and that is measured.** +An aarch64 vLLM wheel exists in general: `pip download --no-deps vllm` on the +worker fetched `vllm-0.27.1-cp38-abi3-manylinux_2_28_aarch64.whl`, 307,180,998 +bytes. Our pin is not reachable that way, because +`https://wheels.vllm.ai/nightly/vllm/` lists wheels for exactly ONE commit and is +a moving pointer rather than an archive, and because the pin is a development +version that is not on PyPI. Four 404s under a per-commit URL scheme were also +seen, and they prove nothing, because that scheme was never confirmed against a +known-good case. So reproducing the pinned oracle needs a source build or a +deliberate pin advance. Nobody established that vLLM never retains per-commit +wheels. ### The `flock` orphan hazard that motivated the replacement diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 9ec67e442..fe5027de2 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -331,5 +331,5 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1117](https://github.com/mudler/vllm.cpp/issues/1117) | `LTX25-A2VID-RECIPE` | `A2VidPipelineTwoStage` (`a2vid_two_stage.py:53` @ `fd4ded7f`) has no recipe row, so `pipeline_kind = a2vid_two_stage` gets the generic table refusal (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1328-1332`) naming the pair rather than the missing machinery. [#922](https://github.com/mudler/vllm.cpp/issues/922) is CLOSED and closed the audio CONDITIONING, not the recipe: a supplied take rides `distilled_two_stage`, which [`ltx25-a2v-audio-input.md`](specs/ltx25-a2v-audio-input.md):438-446 already records as a different trajectory. Measured at `daeff67f2`: `git grep -n '"a2vid' -- src include tests docs examples` returns TWO hits, both upstream anchors inside `Fail`-message assertions (`tests/vllm/multimodal/test_ltx2_video.cpp:4363,:4427`), against a control of 4 for `"one_stage"` in `include/` alone. Four differences from the recipe it rides, each read at the pin: stage 1 is CFG/STG/modality-guided and caller-configured (`:230-240`, fed from `utils/args.py:947-1006`, `--a2v-guidance-scale` defaulting to `video_guider.modality_scale` = 3.0 at `utils/constants.py:54,:64`) where `distilled_two_stage` fixes `allow_guidance_override = false`; stage 1's schedule is scheduler-derived (`:225-227`) against our fixed `DistilledSigmas()`; stage 1 is plain Euler (`:229-258` passes no `stepper`, `utils/blocks.py:526-527`) against our `kEulerAncestral` on 2.5; and the AUDIO guider is the DEFAULT positive-only one (`:237-239`, `ltx-core components/guiders.py:200-210`) rather than the params table's cfg-7.0 row. Two non-schedule facts that must not be guessed: `--audio-path` is `required=True` (`:312-317`), and the distilled LoRA rides stage 2 ALONE (`:114` against `:107`) with `--distilled-lora` `required=True` (`utils/args.py:1140-1153`). Unblocked by `Ltx2GuidedDenoise` landing at `daeff67f2` (#1092/#1102), which [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) `## Owed` names this arm against. Spec [`ltx25-a2vid-recipe.md`](specs/ltx25-a2vid-recipe.md) | enhancement | | [#1118](https://github.com/mudler/vllm.cpp/issues/1118) | `LTX25-A2VID-RECIPE` | LoRA adapters fuse ONCE, at load, into ONE weight set: `src/vllm/multimodal/ltx2_video.cpp:816-820` is the only `dit_options.loras.push_back` in the tree and it runs under `if (!lora_path.empty())`, so every phase of every recipe sees the same fused `im.dit.weights`. Three upstream pipelines build TWO `DiffusionStage`s from the same checkpoint with DIFFERENT adapter sets at `fd4ded7f`: `a2vid_two_stage.py:107` against `:114-119`, `ti2vid_two_stages.py:140` against `:151`, and `ti2vid_two_stages_hq.py:154,:165` at two separate strengths (`:92-101`). `distilled.py:131` builds ONE stage set, which is why `distilled_two_stage`, `dfr` and `retake` have never needed this and no gate has ever asked. Consequence for the arm landing with #1117: that row mirrors `--distilled-lora required=True` (`utils/args.py:1140-1153`) by refusing an `a2vid_two_stage` load with no `lora_path`, so the 3-step stage-2 schedule cannot run on a checkpoint with no distilled adapter — but it CANNOT mirror upstream leaving stage 1 WITHOUT that adapter, so stage 1's guided schedule runs against base + distilled LoRA where upstream runs it against the base alone. That divergence RENDERS, and the PIXELS it renders are not upstream's: it moves the trajectory, so the frames themselves differ, while the frame count, the shapes, the sample rate and the errors are all exactly what they were — nothing in the SHAPE of the result says anything is wrong, which is why it is filed rather than left in a comment. It is not undetectable, and saying so would be the more damaging error: the instrument that WOULD see it is a real-weights comparison against upstream's own render on the same checkpoint, take and seed, upstream's stage 1 on the base weights against ours on base + distilled. Two fix shapes, neither chosen: a second `Ltx2DitWeights` per adapter set (what upstream pays, two `from_checkpoint` calls at `a2vid_two_stage.py:103,:115`) or unfused runtime LoRA selectable per phase; the first doubles resident DiT weights and the second changes the GEMM path. Bounds #1093 and #921, which need the same seam. Listed under `## Owed` in [`ltx25-a2vid-recipe.md`](specs/ltx25-a2vid-recipe.md) | enhancement | | [#1134](https://github.com/mudler/vllm.cpp/issues/1134) | `BACKEND-ROCM` | The `VT_ATTN_DECODE_D128=1` ctest registration added by #767 cannot show the ROCm `d=128` decode arm REACHED the new kernel, and is empty off ROCm. `RegisteredDevices()` (`tests/vt/test_backend_cross_device.cpp:84-96`) enumerates `{kCUDA, kMETAL, kVULKAN, kXPU, kROCM}` and excludes `kCPU`, so on a CPU-only runner — which is what CI has — the new "Qwen3 geometry (bf16, GQA 2, head_dim 128)" case reports 1 test case, 0 assertions, exit 0, for BOTH registrations. On ROCm hardware the case's only backend assertion is `OpProviderStats::declines == 0`, and `OpProviderStats` counts at PROVIDER granularity, so it is identical with the flag set and unset; the NMSE bound passes on either kernel because the arm is correctness-complete. The two compose: there is no machine in this project on which the flag-ON registration distinguishes itself from the flag-OFF one. Disclosed in [`specs/rocm-decode-attn-d128.md`](specs/rocm-decode-attn-d128.md) §4, its `## Owed` section and its result banner, and §9 stop condition 2 is left OPEN rather than claimed discharged. Closing it needs a kernel-selection counter in `src/vt/rocm/rocm_paged_attn.hip` asserted to DIFFER between the two registrations; the CPU-runner half wants `kCPU` in `RegisteredDevices()` or a non-zero-assertion floor per [#463](https://github.com/mudler/vllm.cpp/issues/463). Not a duplicate of #463 (that is the unset-weights-env-var shape and does not describe the `declines` granularity half), #785 (a kernel that never LAUNCHES behind a dead `#if`, a code defect not a coverage one) or #900 (same family, LTX-2.5 subject) | bug | -| [#1146](https://github.com/mudler/vllm.cpp/issues/1146) | `ENV-LEASE-RUNTIME-STAGING` | #1129 is closed and its recorded cause is FALSIFIED by the worker image: it says a leased worker "cannot start Python" and lists `python3`, `pip`, `gcc`, `curl` and `git` as ABSENT, measured in one `dgx:gpu0` probe (job `ff28ada1-0cd3-4867-bf9b-f67050d0608b`). Measured 2026-08-17 on `thor:gpu0` through five `rc run` jobs (`6f4bdb03`, `9c0ebeac`, `8beba132`, `f60d945f`, `63c60a90`), that worker runs as `uid=0(root)` with `/usr/bin/gcc`, `/usr/bin/python3` and a working `apt-get`, and a relocated CUDA runtime staged on `/workspace` imports torch 2.13.0+cu130, reports `cuda available = True` on `NVIDIA Thor` capability (11,0), runs a bf16 matmul, and compiles and executes a Triton kernel (`TRITON_JIT_OK = 4096.0 PASS`, `PROBE5_RC=0`). So the image is provisionable per job, which is none of the three fleet-side fixes #1129 names as the only ways forward. Four walls stand between a staged runtime and a running one: the `pip --target` must run FROM the worker because the submitting host is `x86_64` and the workers are `aarch64`. `Python.h` is absent until `apt-get install python3-dev`. The NAS mount presents `file_mode=0664` so Triton cannot execute its own `ptxas-blackwell`, and `TRITON_PTXAS_PATH` does NOT fix it because it redirects only the plain `ptxas`. So `PYTHONPATH` is ORDERED, `/tmp/tp` before the NAS tree. SCOPE, and it is the point of the row: this is `thor:gpu0` at capability (11,0) ONLY, the GB10 is `sm_121a` and UNMEASURED, only `torch` and `triton` are staged so the pinned vLLM oracle is still NOT shown to run and #1129's consequence for the oracle-dependent rows is NARROWED rather than closed, the `+cu130` versus `release 12.8, V12.8.93` skew is recorded as observed and not adjudicated, and `numpy` is absent from the staged tree. Recipe, job IDs and staged-script sha256 values in [`lease-runtime-staging.md`](specs/lease-runtime-staging.md) | verification | +| [#1146](https://github.com/mudler/vllm.cpp/issues/1146) | `ENV-LEASE-RUNTIME-STAGING` | #1129 is closed and its recorded cause is FALSIFIED by the worker image: it says a leased worker "cannot start Python" and lists `python3`, `pip`, `gcc`, `curl` and `git` as ABSENT, measured in one `dgx:gpu0` probe (job `ff28ada1-0cd3-4867-bf9b-f67050d0608b`). Measured 2026-08-17 on `thor:gpu0` through five `rc run` jobs (`6f4bdb03`, `9c0ebeac`, `8beba132`, `f60d945f`, `63c60a90`), that worker runs as `uid=0(root)` with `/usr/bin/gcc`, `/usr/bin/python3` and a working `apt-get`, and a relocated CUDA runtime staged on `/workspace` imports torch 2.13.0+cu130, reports `cuda available = True` on `NVIDIA Thor` capability (11,0), runs a bf16 matmul, and compiles and executes a Triton kernel (`TRITON_JIT_OK = 4096.0 PASS`, `PROBE5_RC=0`). So the image is provisionable per job, which is none of the three fleet-side fixes #1129 names as the only ways forward. Four walls stand between a staged runtime and a running one: the `pip --target` must run FROM the worker because the submitting host is `x86_64` and the workers are `aarch64`. `Python.h` is absent until `apt-get install python3-dev`. The NAS mount presents `file_mode=0664` so Triton cannot execute its own `ptxas-blackwell`, and `TRITON_PTXAS_PATH` does NOT fix it because it redirects only the plain `ptxas`. So `PYTHONPATH` is ORDERED, `/tmp/tp` before the NAS tree. SCOPE, and it is the point of the row: this is `thor:gpu0` at capability (11,0) ONLY, the GB10 is `sm_121a` and UNMEASURED, only `torch`, `triton` and `numpy` are staged so the pinned vLLM oracle is still NOT shown to run and #1129's consequence for the oracle-dependent rows is NARROWED rather than closed, the `+cu130` versus `release 12.8, V12.8.93` skew is recorded as observed and not adjudicated, and a prebuilt wheel does NOT shorten the route because an aarch64 vLLM wheel exists in general while our pin is neither among the wheels published for the one nightly commit nor on PyPI (the per-commit 404s prove nothing, because that URL scheme was never confirmed against a known-good case). Recipe, job IDs and staged-script sha256 values in [`lease-runtime-staging.md`](specs/lease-runtime-staging.md) | verification | | [#1033](https://github.com/mudler/vllm.cpp/issues/1033) | `GATE-ISSUE-INDEX-TABLE-SHAPE` | `check_table_shapes` (`scripts/check-agent-record.py:1292`) never ran on `.agents/issue-index.md`, so a malformed row there was invisible to every gate. The function already counted unescaped pipes per table line with exactly the right regex; its call site (`:1527-1530`) passed `roadmap_v1.md`, `coordination.md`, `*MATRIX_PATHS` and `*spec_paths`, and simply did not pass this path. Nothing else in the tree counts this file's cells, which made the index the ONLY markdown table in the record set with no shape gate. It is also the one record surface every change must write, with rows long enough to hide a stray pipe in a code span. Arming it reds exactly one row on `origin/main` at `100026481`: a pipe histogram over the index's 289 table lines reads `{5: 288, 9: 1}`, and the outlier is line 279, the [#1003](https://github.com/mudler/vllm.cpp/issues/1003) `ORACLE-LLAMACPP-REPIN-STOCK` row that arrived with `283c7e492` ([#1051](https://github.com/mudler/vllm.cpp/pull/1051)), carrying four unescaped pipes inside code spans at columns 2705, 3106, 3115 and 3338. FIXED IN FLOW: the path is added, the four pipes are escaped, and three cases in `tests/scripts/test_agent_record.py` hold it — one capturing the paths `main()` really hands the gate, one running it on the shipped file, one mutating a copy so the instrument is proven to fire. Repairing the row EDITS an append-only file, so `check-issue-index-append-only.py` is red on the branch and the exception is argued in the commit body, as `ff264cb82` ([#1025](https://github.com/mudler/vllm.cpp/pull/1025)) argued the same one: appending a corrected copy would leave the broken row in place and add a duplicate key, so the file only becomes well-formed by editing it where it sits. TWO PREMISES OF THE REPORT MEASURED FALSE and are recorded in the spec rather than quietly dropped: the checker does NOT stop at the first finding (one `errors` list, three findings in one run, exit 1 once), and the four pipes are NOT in a `git diff` piped into `grep` — that span does not exist in the row. Spec [`gate-issue-index-table-shape.md`](specs/gate-issue-index-table-shape.md) | bug | diff --git a/.agents/specs/lease-runtime-staging.md b/.agents/specs/lease-runtime-staging.md index 3b703cce4..980c7e2d1 100644 --- a/.agents/specs/lease-runtime-staging.md +++ b/.agents/specs/lease-runtime-staging.md @@ -40,12 +40,25 @@ Direction 2 probe recorded `python3=ABSENT`, `pip=ABSENT`, `gcc=ABSENT`, `curl=ABSENT` and `git=ABSENT` inside the worker. That reading came from one probe on `dgx:gpu0`, job -`ff28ada1-0cd3-4867-bf9b-f67050d0608b`, taken on 2026-08-17. The worker image -measured here on `thor:gpu0` the same day reports the opposite for two of those -five names, `python3` and `gcc`. It runs as `uid=0(root)`, carries -`/usr/bin/python3`, `/usr/bin/gcc` and `/usr/bin/apt-get`, and installs a package -as root (`APT_UPDATE_RC=0`, `APT_INSTALL_RC=0`). `pip`, `curl` and `git` were not -probed on `thor:gpu0`, so this row says nothing about them. +`ff28ada1-0cd3-4867-bf9b-f67050d0608b`, taken on 2026-08-17. Three of its five +names are now contradicted, and the contradictions come from TWO different boxes. +Keep them apart, because a name proven on one worker proves nothing about the +other. + +On `thor:gpu0`, job `8beba132` reports `python3` and `gcc` present. The worker +runs as `uid=0(root)`, carries `/usr/bin/python3`, `/usr/bin/gcc` and +`/usr/bin/apt-get`, and installs a package as root (`APT_UPDATE_RC=0`, +`APT_INSTALL_RC=0`). Job `fd5654c0` then ran `python3 -m pip install --target` +there to completion, so `pip` is present on `thor:gpu0` too. + +On `dgx:gpu0`, job `609c4944-594b-4617-967b-fb3d3d8c09f6` invoked +`python3 -m pip install --quiet --target /workspace/oracle-probe/site torch`, so +`python3` and `pip` are present on that worker as well. Read that job no further: +it ended on `max_runtime exceeded (35m0s)`, so it is evidence that `pip` STARTS +on `dgx:gpu0` and not that this install finished. + +`curl` and `git` were probed on neither worker after #1129, so this row says +nothing about them. The measurement was taken correctly. The fleet changed under it. So the recorded cause no longer holds, and the three fleet-side fixes #1129 names as the only @@ -100,6 +113,39 @@ export PYTHONPATH=/tmp/tp:/workspace/oracle-probe/site export CPATH=/workspace/oracle-probe/pyhdr/python3.12:${CPATH:-} ``` +## The prebuilt-wheel route is closed for our pin + +The spec says the oracle needs `nvcc`. A reader can reasonably ask whether a +prebuilt wheel sidesteps that, so the question was measured on 2026-08-17 rather +than argued. It does not. + +**An aarch64 vLLM wheel exists in general, so the architecture is not the +obstacle.** On the worker, `python3 -m pip download --no-deps vllm` fetched +`vllm-0.27.1-cp38-abi3-manylinux_2_28_aarch64.whl`, 307,180,998 bytes, +`VLLM_DL_RC=0`. The same job reported `aarch64 linux-aarch64` from +`platform.machine()` and `sysconfig.get_platform()`. + +**Our pin is not reachable that way.** `https://wheels.vllm.ai/nightly/vllm/` is +a PEP 503 index, and it lists FOUR wheels for exactly ONE commit, `402547d7f`, +aarch64 and x86_64, each also in its percent-encoded form. It is a moving +pointer and not an archive, so the pin +`5559679229bc961848b121ccdeaa8fa5d79bec98` is absent from it. The pin is also a +development commit, `0.23.1rc1.dev1511+g555967922`, so it is not on PyPI. + +**The 404s are NOT the evidence, and this is the part to read carefully.** Four +candidate filenames under `https://wheels.vllm.ai//` returned 404 by GET +and by HEAD, on the pin and on a current `main` sha `c1e438728c55`. That URL +scheme was never confirmed against a known-good case, and the host's own root +404s while `/nightly` returns 200. A 404 from an unconfirmed path therefore +proves nothing about whether a per-commit wheel exists. The load-bearing +evidence is the nightly index listing one commit. + +**So the consequence is narrow.** Reproducing the pinned oracle needs a source +build, which is why it needs `nvcc`, or a deliberate advance of the pin to a +commit a wheel exists for. A wheel cannot substitute for either. **This does NOT +establish that vLLM never retains per-commit wheels.** Nobody measured that, and +the unconfirmed URL scheme is exactly why. + ## What this does NOT establish Read this section before you quote any line above it. Every claim here is @@ -109,17 +155,24 @@ narrower than the sentence a reader wants to write from it. `sm_121a` and is UNMEASURED. Nothing here licenses a claim about the Spark. A probe is queued at lowest priority behind a human hold on `dgx:gpu0`. Until that probe returns, `dgx:gpu0` keeps the reading its own last probe gave it. -- **The pinned vLLM oracle is NOT staged.** Only `torch` and `triton` are, so - this does not show that the oracle runs. vLLM at `555967922` is a source build - with compiled extensions, and it needs `nvcc`, which the worker still lacks. - #1129's consequence for the oracle-dependent rows is therefore NARROWED and not - closed. +- **The pinned vLLM oracle is NOT staged.** Only `torch`, `triton` and `numpy` + are, so this does not show that the oracle runs. vLLM at `555967922` is a + source build with compiled extensions, and it needs `nvcc`, which the worker + still lacks. #1129's consequence for the oracle-dependent rows is therefore + NARROWED and not closed. A prebuilt wheel does not remove the `nvcc` + requirement either, and that is measured rather than assumed. See + `## The prebuilt-wheel route is closed for our pin`. - **The CUDA version skew is recorded as observed, not adjudicated.** The torch wheel is `+cu130` while the staged `ptxas` reports `release 12.8, V12.8.93`. It compiled and ran a correct kernel here. Nobody has read whether the skew changes a numerical result, and this row does not. -- **`numpy` is absent** from the staged tree. Torch warns about it on every - import, and vLLM would require it. +- **`numpy` WAS absent from the staged tree, and no longer is.** Job `fd5654c0` + installed it into the same tree on 2026-08-17: `NUMPY_RC=0`, then + `numpy 2.5.2 /workspace/oracle-probe/site/numpy/__init__.py`, + `NUMPY_IMPORT_RC=0`. The five jobs above ran BEFORE that install, which is why + each of their logs carries torch's `Failed to initialize NumPy: No module + named 'numpy'` warning. A job that stages the tree today should not see that + warning, and a job that still does is reading a different tree. ## Risks @@ -149,13 +202,16 @@ device. ## Evidence -Five `rc run` jobs on `thor:gpu0`, 2026-08-17. Every job after the first ran a -script staged on the NAS rather than fed on stdin. That is a harness detail and -not one of the four walls: `@triton.jit` calls `inspect.getsourcelines`, which -raises `OSError: could not get source code` on a function read from stdin. The -sha256 values are taken over the staged files at -`/mnt/nas_share/rc/oracle-probe/`, which is the same folder the worker reads as -`/workspace/oracle-probe/`. +Six `rc run` jobs on `thor:gpu0`, 2026-08-17. The first five staged and ran the +runtime. The sixth added `numpy` and asked the wheel question, and it was +submitted separately as `claude/mudler-ubuntu-box/vllm-probe`. + +Every job after the first ran a script staged on the NAS rather than fed on +stdin. That is a harness detail and not one of the four walls: `@triton.jit` +calls `inspect.getsourcelines`, which raises `OSError: could not get source code` +on a function read from stdin. The sha256 values are taken over the staged files +at `/mnt/nas_share/rc/oracle-probe/`, which is the same folder the worker reads +as `/workspace/oracle-probe/`. | Job | Script | sha256 prefix | Result | |---|---|---|---| @@ -164,9 +220,18 @@ sha256 values are taken over the staged files at | `8beba132-c682-4b11-aa47-a682f2cce096` | `hdr.sh` | `fb02d1a1f45f` | `uid=0(root)`, `apt-get` installs `python3-dev`, headers staged, wall 3 appears as `PermissionError` on `ptxas-blackwell` | | `f60d945f-f5ec-49c5-9452-c214a50a0043` | `hdr2.sh` | `37c9dcad4c64` | `TRITON_PTXAS_PATH` to a `/tmp` copy is insufficient, and the staged `ptxas` reports `release 12.8, V12.8.93` | | `63c60a90-29b8-4115-88c5-d82be2126136` | `hdr3.sh` | `e11a540d23b8` | wall 4 removed, `TRITON_JIT_OK = 4096.0 PASS`, `PROBE5_RC=0` | +| `fd5654c0-d522-498c-8800-ca4df9a36944` | `vllmprobe.sh` | `58fa03543810` | `NUMPY_RC=0`, `numpy 2.5.2` imports from the staged tree, `NUMPY_IMPORT_RC=0`, `VLLM_DL_RC=0` on a 307,180,998-byte aarch64 vLLM wheel, `aarch64 linux-aarch64` | The kernel each Triton job ran is `tritontest.py`, sha256 prefix `df9861c9f86d`. +One `rc run` job on `dgx:gpu0` is cited above for a different purpose: +`609c4944-594b-4617-967b-fb3d3d8c09f6` invoked `python3 -m pip install --target` +and then ended on `max_runtime exceeded (35m0s)`. It is evidence that `python3` +and `pip` exist on that worker. It is not evidence that its install completed. + +The `wheels.vllm.ai` reads have no `rc` job, because they are host-side network +reads rather than device work. + ## Stop conditions - Stop if a correction needs a second `**GPU mutex:**` bullet. Return @@ -181,9 +246,14 @@ The kernel each Triton job ran is `tritontest.py`, sha256 prefix `df9861c9f86d`. - [#1146](https://github.com/mudler/vllm.cpp/issues/1146) stays open. Re-run the identical staged probe on `dgx:gpu0` at `sm_121a`. - Stage the pinned vLLM oracle itself, which is what #1129 actually blocks. It - needs `nvcc` first. + needs `nvcc` first, because no wheel carries our pin. - Read whether the `+cu130` and `12.8` skew changes a numerical result. -- Stage `numpy`, which vLLM requires and the current tree lacks. +- Confirm the `https://wheels.vllm.ai//` URL scheme against a known-good + case before anyone reads the four 404s as evidence of absence. Until then those + 404s carry no weight, and the nightly index is the only load-bearing reading. + +Paid since this spec was written: `numpy` is staged (job `fd5654c0`), so the +former `## Owed` line asking for it is removed rather than left to read as debt. ## Now @@ -191,4 +261,7 @@ The four walls and the working recipe are recorded here. `.agents/environment.md`, `.agents/specs/mtp-k-gt-1.md` and `.agents/specs/gpu-lease-methodology.md` no longer carry "cannot start Python" as the live cause, and each now names the box and the date its reading came from. -The next step is the `dgx:gpu0` probe, which the coordinator has queued. +The staged tree now holds `torch`, `triton` and `numpy`. It does not hold the +oracle, and a prebuilt wheel cannot put it there for our pin, so `nvcc` and a +source build remain the route. The next step is the `dgx:gpu0` probe, which the +coordinator has queued. diff --git a/.agents/specs/mtp-k-gt-1.md b/.agents/specs/mtp-k-gt-1.md index 69ce06c95..8a0a798c5 100644 --- a/.agents/specs/mtp-k-gt-1.md +++ b/.agents/specs/mtp-k-gt-1.md @@ -719,7 +719,7 @@ upstream's own supported configuration. | Owed | What it must show | Who | |---|---|---| | DGX three-way greedy gate at k=2, 3, 4 on Qwen3.6-27B and 35B | **PART-PAID 2026-08-16, and the remaining half is the vLLM leg.** On the 27B NVFP4 at the DEFAULT bf16 GDN state, depth REACHES the verify path at k=2, 3 and 4 on real weights and the per-depth counters are populated at EVERY depth up to k. What is NOT established is `our-ON == our-OFF`: it is FALSE here on 3 of 4 prompts, at the SAME token positions for every k and for the padded control alike, which is the signature of a fixed spec-ON/OFF difference rather than a depth defect. Attributing it needs the oracle leg, which did not run. Do NOT read this row as a passed token gate. **2026-08-17 narrowed it to 3 forwards.** Only the FIRST divergence per arm and prompt is adjudicable, which reduces 1718 divergent positions to 18 and then to 3 distinct probe points. Prompt 1 position 1 resolves to THREE different tokens under three values of k, which no depth defect can produce. `scripts/mtp-k-gt-1-neartie-gap.py` is committed and decides each candidate against `kNearTieMnats = 500`. It still has not RUN: the oracle cannot be loaded while a foreign multi-tens-of-GiB container is resident. **2026-08-17 third pass: the box was CLEAN, the run happened, and the divergence REPRODUCED exactly on independently generated streams (1718 positions, 18 adjudicable, 3 probe points, same tokens), so it is deterministic rather than a flake. The adjudication STILL did not report.** Its failure was an INSTRUMENT failure and is recorded as such rather than as a verdict: the reimaged host carries no C compiler at all, so Triton's JIT died after the weights loaded and vLLM surfaced it as `Engine core initialization failed`. Repaired by running the oracle inside a container carrying the toolchain, which is MEASURED loading the engine past that point; the box then stopped answering SSH mid-leg. Do NOT read this row as a passed token gate | `SPEC-MTP-K-GT-1`, [#81](https://github.com/mudler/vllm.cpp/issues/81) M1 | -| **THE BLOCKER: the pinned oracle cannot load a 27B on `dgx.casa` at all, and `gpu_memory_utilization` is NOT the lever** | Once the toolchain fix let an oracle reach this step for the first time, it consumed the entire host in the step AFTER `torch.compile`. **Measured 2026-08-17 at 0.75: about 110 GiB of HOST RAM held while `nvidia-smi` reported 26 GiB on the device, 45 minutes hung at loadavg 260 with 0 GiB available, its own timeout firing (`ADJUDICATE_EXIT=124`); killing the container took the box from 118 of 119 GiB used to 4 of 119 in under ten seconds.** The attribution to that 0.75 was then **TESTED AND REFUTED**: a third window ran the byte-identical instrument at `--gpu-mem-util 0.30` with a 5-second `MemAvailable` sampler and collapsed the same way (87683 MB free at 09:00:47, **0** at 09:02:25). Weight loading finished with 66 GiB free and compilation with 88 GiB free, so it is neither. **The 0.30 run also REBOOTED the box while the 0.75 run only thrashed** (`boot_id` `5bbdc432...` to `bd5c6e7a...`, `journalctl --list-boots` gap 09:10:15Z to 09:13:55Z), so a lower fraction is not a safety margin. Owed: identify the actual step by varying `max_num_batched_tokens` and `cudagraph_capture_sizes` ONE AT A TIME with the sampler running, then record a configuration that demonstrably reaches KV-cache allocation and completes a generate, in `.agents/environment.md` beside the toolchain recipe. Until then no vLLM leg of any row can run on this host. **The container re-check this row left open is now DONE, and it CONFIRMS the blocker rather than refuting it ([#1129](https://github.com/mudler/vllm.cpp/issues/1129), 2026-08-17).** The host recipe reaches its container over `ssh`, which bypasses the `rc` lease and makes the fleet report `dgx:gpu0` free while somebody is on it. The lease-compliant route was then probed directly, and the `dgx:gpu0` leased worker carries no `gcc`, `nvcc`, `ninja`, `cmake`, `python3`, `pip`, `docker` or `sudo`, no `/usr/include/stdio.h` and no `/usr/local/cuda*`, and it cannot see `/home/mudler`, so it cannot reach `~/venvs/vllm-oracle-pin-555967922`. That list is one box on one day and it does NOT generalise across the fleet, which the third probe below measured. `rc run` has no `--image` flag. **A second probe on the same day NARROWED that reading and did not close it, and one clause of it was measured FALSE.** The lease carries bytes, and bytes are enough to run: `/lib/ld-linux-aarch64.so.1 /workspace/` started a staged ELF, and a copy to `/tmp` followed by `chmod +x` started both a staged script and a staged ELF, each exit 0 (`.agents/environment.md`, "The lease carries bytes, and the exec bit is a mount option"). The clause "could not start it if it could" is therefore RETRACTED here. What holds is narrower and is still a blocker: **no vLLM leg of any row runs on `dgx.casa` by a lease-compliant path today, because nothing has staged a runtime on the NAS**, and only a host-side actor reached over `ssh` can place one there. **A THIRD probe the same day answered the UNMEASURED clause, and it is narrower than it sounds.** A relocated CUDA runtime staged on `/workspace` DOES start inside a worker: on `thor:gpu0`, five `rc run` jobs imported torch 2.13.0+cu130, reported `cuda available = True` on `NVIDIA Thor` capability (11,0), ran a bf16 matmul and compiled and executed a Triton kernel (`TRITON_JIT_OK = 4096.0 PASS`). That worker runs as `uid=0(root)` with a working `apt-get` and `/usr/bin/gcc`, so #1129's recorded "cannot start Python" cause no longer holds for that image ([#1146](https://github.com/mudler/vllm.cpp/issues/1146), [`lease-runtime-staging.md`](lease-runtime-staging.md)). What it does NOT establish is what this row needs. It is `thor:gpu0` at capability (11,0) ONLY, and the GB10 is `sm_121a` and UNMEASURED. Only `torch` and `triton` are staged, NOT the pinned oracle, which is a source build with compiled extensions and needs `nvcc`. So the vLLM leg is still blocked here, by a narrower and now precisely named reason. The blocker still widens past #81 to every oracle-dependent row, [#1003](https://github.com/mudler/vllm.cpp/issues/1003), [#915](https://github.com/mudler/vllm.cpp/issues/915) and [#821](https://github.com/mudler/vllm.cpp/issues/821) included. The migration is NOT designed here. The only recorded lever is `/workspace`, which is NAS-backed, writable from the worker and the same folder on `dgx` and `thor`. `ENV-LEASE-RUNTIME-STAGING` owns the design | `SPEC-MTP-K-GT-1`, [#81](https://github.com/mudler/vllm.cpp/issues/81) M1, [#1129](https://github.com/mudler/vllm.cpp/issues/1129) | +| **THE BLOCKER: the pinned oracle cannot load a 27B on `dgx.casa` at all, and `gpu_memory_utilization` is NOT the lever** | Once the toolchain fix let an oracle reach this step for the first time, it consumed the entire host in the step AFTER `torch.compile`. **Measured 2026-08-17 at 0.75: about 110 GiB of HOST RAM held while `nvidia-smi` reported 26 GiB on the device, 45 minutes hung at loadavg 260 with 0 GiB available, its own timeout firing (`ADJUDICATE_EXIT=124`); killing the container took the box from 118 of 119 GiB used to 4 of 119 in under ten seconds.** The attribution to that 0.75 was then **TESTED AND REFUTED**: a third window ran the byte-identical instrument at `--gpu-mem-util 0.30` with a 5-second `MemAvailable` sampler and collapsed the same way (87683 MB free at 09:00:47, **0** at 09:02:25). Weight loading finished with 66 GiB free and compilation with 88 GiB free, so it is neither. **The 0.30 run also REBOOTED the box while the 0.75 run only thrashed** (`boot_id` `5bbdc432...` to `bd5c6e7a...`, `journalctl --list-boots` gap 09:10:15Z to 09:13:55Z), so a lower fraction is not a safety margin. Owed: identify the actual step by varying `max_num_batched_tokens` and `cudagraph_capture_sizes` ONE AT A TIME with the sampler running, then record a configuration that demonstrably reaches KV-cache allocation and completes a generate, in `.agents/environment.md` beside the toolchain recipe. Until then no vLLM leg of any row can run on this host. **The container re-check this row left open is now DONE, and it CONFIRMS the blocker rather than refuting it ([#1129](https://github.com/mudler/vllm.cpp/issues/1129), 2026-08-17).** The host recipe reaches its container over `ssh`, which bypasses the `rc` lease and makes the fleet report `dgx:gpu0` free while somebody is on it. The lease-compliant route was then probed directly, and the `dgx:gpu0` leased worker carries no `gcc`, `nvcc`, `ninja`, `cmake`, `python3`, `pip`, `docker` or `sudo`, no `/usr/include/stdio.h` and no `/usr/local/cuda*`, and it cannot see `/home/mudler`, so it cannot reach `~/venvs/vllm-oracle-pin-555967922`. That list is one box on one day and it does NOT generalise across the fleet, which the third probe below measured. `rc run` has no `--image` flag. **A second probe on the same day NARROWED that reading and did not close it, and one clause of it was measured FALSE.** The lease carries bytes, and bytes are enough to run: `/lib/ld-linux-aarch64.so.1 /workspace/` started a staged ELF, and a copy to `/tmp` followed by `chmod +x` started both a staged script and a staged ELF, each exit 0 (`.agents/environment.md`, "The lease carries bytes, and the exec bit is a mount option"). The clause "could not start it if it could" is therefore RETRACTED here. What holds is narrower and is still a blocker: **no vLLM leg of any row runs on `dgx.casa` by a lease-compliant path today, because nothing has staged a runtime on the NAS**, and only a host-side actor reached over `ssh` can place one there. **A THIRD probe the same day answered the UNMEASURED clause, and it is narrower than it sounds.** A relocated CUDA runtime staged on `/workspace` DOES start inside a worker: on `thor:gpu0`, five `rc run` jobs imported torch 2.13.0+cu130, reported `cuda available = True` on `NVIDIA Thor` capability (11,0), ran a bf16 matmul and compiled and executed a Triton kernel (`TRITON_JIT_OK = 4096.0 PASS`). That worker runs as `uid=0(root)` with a working `apt-get` and `/usr/bin/gcc`, so #1129's recorded "cannot start Python" cause no longer holds for that image ([#1146](https://github.com/mudler/vllm.cpp/issues/1146), [`lease-runtime-staging.md`](lease-runtime-staging.md)). What it does NOT establish is what this row needs. It is `thor:gpu0` at capability (11,0) ONLY, and the GB10 is `sm_121a` and UNMEASURED. Only `torch`, `triton` and `numpy` are staged, NOT the pinned oracle, which is a source build with compiled extensions and needs `nvcc`. **A prebuilt wheel does not shorten that route, measured 2026-08-17:** an aarch64 vLLM wheel exists in general, but our pin is not among the wheels published for the one nightly commit and is not on PyPI, so a source build or a deliberate pin advance is the only way. So the vLLM leg is still blocked here, by a narrower and now precisely named reason. The blocker still widens past #81 to every oracle-dependent row, [#1003](https://github.com/mudler/vllm.cpp/issues/1003), [#915](https://github.com/mudler/vllm.cpp/issues/915) and [#821](https://github.com/mudler/vllm.cpp/issues/821) included. The migration is NOT designed here. The only recorded lever is `/workspace`, which is NAS-backed, writable from the worker and the same folder on `dgx` and `thor`. `ENV-LEASE-RUNTIME-STAGING` owns the design | `SPEC-MTP-K-GT-1`, [#81](https://github.com/mudler/vllm.cpp/issues/81) M1, [#1129](https://github.com/mudler/vllm.cpp/issues/1129) | | The cleanup trap in the DGX drivers does not stop the run | `run_all_inner.sh` and `run_oracle_inner.sh` both use `trap cleanup EXIT INT TERM` where `cleanup` resets the clocks and returns WITHOUT exiting, so bash resumes the script after the handler. **Observed 2026-08-17: `SIGTERM` reset the clocks and the driver then started its next leg, which immediately began re-filling a box that had 0 GiB available.** The chain had to be `SIGKILL`ed and the container stopped separately. A cleanup trap that does not terminate is not a stop button, and on a box that reboots rather than OOM-killing that difference is the box. Owed: `exit` from the signal path, and a `docker kill` of the current leg's named container inside `cleanup` so the container cannot outlive its driver | `SPEC-MTP-K-GT-1`, [#81](https://github.com/mudler/vllm.cpp/issues/81) M1 | | The PADDED CONTROL arm of that gate, and the RATE assertion it carries | **PAID 2026-08-16 on the 27B, and its throughput VOID LIFTED 2026-08-17.** The first pass could not quote `padded_k3`/`padded_k4` throughput because they started at loadavg 10.77 and 20.41 against real arms at 1.5 to 2.9. The third pass re-ran all seven arms inside ONE window in a load band of 0.16 to 1.86, every leg exit 0, so the real-arm against padded-control comparison is now quotable rather than merely computed. Detail below. Margin fixed BEFORE the run at 0.10 absolute per depth. The real loop accepts at 0.507 to 0.750 at every depth >= 1. The padded control accepts at 0.000 at every depth >= 1 while its depth-0 rate MATCHES the real arm (0.892 to 0.925 against 0.868 to 0.878), which is what a control that isolates columns >= 1 must look like. Every margin clears by at least 0.41. The control measuring 0 is recorded as a fact about THIS prompt set and did not license restoring the count assertion. Still owed on the 35B | `SPEC-MTP-K-GT-1`, [#81](https://github.com/mudler/vllm.cpp/issues/81) M1 | | Silent de-graphing when the actual depth differs from the configured k, AND the `S`-only slot-ring key ([#1020](https://github.com/mudler/vllm.cpp/issues/1020)) | The spec-graph predicate reads the step's ACTUAL uniform query length instead of `num_spec()` (`runner.cpp:1383`), and the graph slot ring is keyed on `(S, q)` in the SAME change. The re-key is owed on its own merits and NOT only as a consequence of widening the predicate, which is the correction section 4.2a records: `uniform_decode = input.pure_decode \|\| (spec_graph && ...)` (`qwen3_5_moe.cpp:143-148`, `qwen3_5_dense.cpp:172-177`) already routes TWO query lengths to one `impl_->slots[S]` (`qwen3_5.cpp:9281`, dense `:9708`), and `SizeSlot` invalidates on `fa_cols` and `aux_taps` only (`:9309-9316` and `:9361-9367`). At k=1, 8 requests pure-decode and 4 requests spec both key on `S = 8`. That is pre-existing since SPEC-DSPARK W8 (#442) and this row does not widen it, but it is not the benign thing the first spec revision claimed. Plus a measured before-and-after on the capture-set size and persistent logits memory, and a counter or log for the eager fallback so it can never again be invisible | `SPEC-MTP-K-GT-1`, [#1020](https://github.com/mudler/vllm.cpp/issues/1020) | @@ -754,14 +754,17 @@ nothing has staged a runtime on the NAS, and NOT because a lease cannot execute one ([#1129](https://github.com/mudler/vllm.cpp/issues/1129), and `.agents/environment.md` "The lease carries bytes, and the exec bit is a mount option"). **A third probe then answered the UNMEASURED clause: staging works.** -On `thor:gpu0`, a `torch` and `triton` tree staged on `/workspace` imports, +On `thor:gpu0`, a `torch`, `triton` and `numpy` tree staged on `/workspace` imports, initializes CUDA and compiles and runs a Triton kernel, and that worker is root with `apt-get` and `gcc`, so #1129's "cannot start Python" cause is falsified for that image ([#1146](https://github.com/mudler/vllm.cpp/issues/1146), [`lease-runtime-staging.md`](lease-runtime-staging.md)). This row still cannot resume on it. The result is `thor:gpu0` at capability (11,0) only, the GB10 is -`sm_121a` and UNMEASURED, and the staged tree holds `torch` and `triton` and not -the pinned oracle, which needs `nvcc`. Resume this row only after the oracle +`sm_121a` and UNMEASURED, and the staged tree holds `torch`, `triton` and `numpy` +and not the pinned oracle, which needs `nvcc`. A prebuilt wheel does not shorten +that route: an aarch64 vLLM wheel exists in general, and our pin is neither among +the wheels published for the one nightly commit nor on PyPI, so a source build or +a deliberate pin advance is the only way. Resume this row only after the oracle itself has a lease-compliant path, because the adjudication instrument is committed and correct and has nothing to run on.