record(ENV-LEASE-RUNTIME-STAGING): a relocated CUDA runtime starts inside a lease on thor:gpu0 (#1146)

.agents/environment.md

@@ -72,11 +72,14 @@ labels at all. `class=train` and `gpu_model=GB10` do match. `rc run` has no
 `--idle-timeout`, `--max-runtime`, `--no-wait`, `--priority`, `--select` and
 `--timeout`.
 
-### What a leased worker can and cannot do, measured 2026-08-17
+### What the `dgx:gpu0` leased worker can and cannot do, measured 2026-08-17
 
 Probed with one `rc run -d dgx:gpu0 --max-runtime 2m` job
 (`ff28ada1-0cd3-4867-bf9b-f67050d0608b`). Verify this again before you plan work
-around it, because the worker image can change under you.
+around it, because the worker image can change under you. **It did change.** The
+`thor:gpu0` worker measured later the same day carries `python3` and `gcc`, which
+this list calls absent, so read this section as one box on one day. The `thor`
+reading is in "A relocated CUDA runtime starts on `thor:gpu0`" further down.
 
 - The command runs as user `rc` in a **k3s pod**, hostname `rc-worker-<id>`.
   `/.dockerenv` is absent and 8 `KUBERNETES_*` variables are set, so it is a pod
@@ -86,8 +89,11 @@ around it, because the worker image can change under you.
   `flock`, and **`/workspace`**.
 - **Absent: `gcc`, `cc`, `clang`, `nvcc`, `ninja`, `cmake`, `make`, `python3`,
   `python`, `pip`, `docker`, `sudo`, `git`, `ssh`, `curl`,
-  `/usr/include/stdio.h`, and any `/usr/local/cuda*` toolkit.** A worker cannot
-  compile, cannot start Python, and cannot install anything.
+  `/usr/include/stdio.h`, and any `/usr/local/cuda*` toolkit.** This `dgx:gpu0`
+  worker cannot compile, cannot start Python, and cannot install anything.
+  **Do not carry that clause to another device.** On `thor:gpu0` the same day the
+  worker ran as `uid=0(root)` with `/usr/bin/gcc`, `/usr/bin/python3` and a
+  working `apt-get` ([#1146](https://github.com/mudler/vllm.cpp/issues/1146)).
 - **The host filesystem is not visible.** `/home/mudler` does not exist inside
   the worker.
 - `/workspace` is the house NAS, measured as `//192.168.68.102/Data 7.3T total,
@@ -109,8 +115,12 @@ lease carries bytes, and the exec bit is a mount option" below measures staged
 content starting under the dynamic loader and after a copy to `/tmp`, so what
 blocks the oracle is that nothing has put a runtime where a lease can see it.
 That is why recent GPU work reached for `ssh`, and the bypass is a symptom of
-this gap rather than a discipline problem. Do not design the migration here. The
-row that takes #1129 owns it.
+this gap rather than a discipline problem. Do not design the migration here.
+`ENV-LEASE-RUNTIME-STAGING` owns the design, and
+[`lease-runtime-staging.md`](specs/lease-runtime-staging.md) holds the working
+recipe. That recipe stages `torch` and `triton`, not the pinned oracle, and it
+ran on `thor:gpu0` and not here, so the sentence above still stands for
+`dgx.casa` today.
 
 **This confirms and extends a finding that already landed, rather than making a
 new one.** `.agents/specs/minimax-music3.md` §13.10 probed `thor`'s worker on
@@ -158,26 +168,79 @@ job's working directory `/` is not writable.
 
 **So the lease carries bytes, and bytes are enough to run.** A runtime staged on
 `/workspace` can start under the dynamic loader, or after a copy to `/tmp`. What
-the worker cannot do is produce or fetch that runtime, because it has no `curl`,
-`wget`, `git`, `gcc`, `nvcc`, `cmake` or `python3`. Present and useful for
-staging: `cp`, `cat`, `tar`, `chmod`, `perl`, `flock` and `nvidia-smi`.
+this `dgx:gpu0` worker cannot do is produce or fetch that runtime, because it has
+no `curl`, `wget`, `git`, `gcc`, `nvcc`, `cmake` or `python3`. Present and useful
+for staging: `cp`, `cat`, `tar`, `chmod`, `perl`, `flock` and `nvidia-smi`. **The
+`thor:gpu0` worker does produce one**, because it is root and carries `apt-get`
+and `gcc`. That is the section below.
 
 **This narrows [#1129](https://github.com/mudler/vllm.cpp/issues/1129) and does
 not close it.** The pinned oracle stays unreachable because its virtual
 environment lives at `~/venvs/vllm-oracle-pin-555967922` on the dgx host, which
 no lease can see, and only a host-side actor reached over `ssh` can place a copy
-on the NAS. Whether that copy then starts is UNMEASURED. A CUDA virtual
-environment holds absolute paths in its shebangs and its `RECORD` files, so
-treat the relocation as an open question rather than a solved step.
+on the NAS. **Whether a relocated CUDA runtime then starts is no longer
+UNMEASURED. It starts, on `thor:gpu0`.** The section below has the reading. A
+CUDA virtual environment still holds absolute paths in its shebangs and its
+`RECORD` files, so a `pip install --target` tree is the shape that was measured
+and a copied venv is not.
 
 **Three fleet-side changes would each remove the staging problem, and none of
-them is ours to make.** Whoever owns the fleet picks one.
+them is ours to make.** Whoever owns the fleet picks one. **A fourth route was
+then measured, and it needs nobody's permission:** the `thor:gpu0` worker runs
+as root with a working `apt-get`, so a job provisions its own container.
 
 1. The worker image gains a toolchain and a Python interpreter.
 2. `rc run` gains an `--image` flag, so a job selects an image that has them.
 3. `/workspace` is mounted so that a file there can carry an exec bit. This one
    removes the copy step only, because the two routes above already execute.
 
+### A relocated CUDA runtime starts on `thor:gpu0`, measured 2026-08-17
+
+Probed with six `rc run` jobs on `thor:gpu0`: `6f4bdb03`, `9c0ebeac`, `8beba132`,
+`f60d945f`, `63c60a90` and `fd5654c0`. A `torch`, `triton` and `numpy` tree
+staged on `/workspace` imports, initializes CUDA, runs a bf16 matmul, and
+compiles and executes a Triton kernel. The job IDs in full, the staged-script
+sha256 values, the four walls and the working recipe are in
+[`lease-runtime-staging.md`](specs/lease-runtime-staging.md)
+([#1146](https://github.com/mudler/vllm.cpp/issues/1146)).
+
+```
+torch.__version__= 2.13.0+cu130      cuda available = True
+device 0         = NVIDIA Thor       capability     = (11, 0)
+triton.__version__ = 3.7.1           TRITON_JIT_OK  = 4096.0 PASS
+```
+
+The recipe, once per worker container:
+
+```sh
+apt-get update -qq && apt-get install -y -qq python3-dev
+mkdir -p /tmp/tp && cp -a /workspace/oracle-probe/site/triton /tmp/tp/
+chmod -R +x /tmp/tp/triton/backends/nvidia/bin/
+
+export PYTHONPATH=/tmp/tp:/workspace/oracle-probe/site
+export CPATH=/workspace/oracle-probe/pyhdr/python3.12:${CPATH:-}
+```
+
+**Read the scope before you quote it.** This is `thor:gpu0` at capability (11,0)
+and nothing else. The GB10 is `sm_121a` and is UNMEASURED, so nothing here
+licenses a claim about the Spark. Only `torch`, `triton` and `numpy` are staged,
+so the pinned vLLM oracle is still not shown to run: it is a source build with
+compiled extensions and it needs `nvcc`, which the worker lacks. The torch wheel
+is `+cu130` while the staged `ptxas` reports `release 12.8, V12.8.93`, and that
+skew is recorded as observed rather than adjudicated.
+
+**A prebuilt wheel does not remove the `nvcc` requirement, and that is measured.**
+An aarch64 vLLM wheel exists in general: `pip download --no-deps vllm` on the
+worker fetched `vllm-0.27.1-cp38-abi3-manylinux_2_28_aarch64.whl`, 307,180,998
+bytes. Our pin is not reachable that way, because
+`https://wheels.vllm.ai/nightly/vllm/` lists wheels for exactly ONE commit and is
+a moving pointer rather than an archive, and because the pin is a development
+version that is not on PyPI. Four 404s under a per-commit URL scheme were also
+seen, and they prove nothing, because that scheme was never confirmed against a
+known-good case. So reproducing the pinned oracle needs a source build or a
+deliberate pin advance. Nobody established that vLLM never retains per-commit
+wheels.
+
 ### The `flock` orphan hazard that motivated the replacement
 
 The harness family in this repository puts the `flock` handle on a **subshell**,

.agents/issue-index.md

@@ -331,4 +331,5 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1117](https://github.com/mudler/vllm.cpp/issues/1117) | `LTX25-A2VID-RECIPE` | `A2VidPipelineTwoStage` (`a2vid_two_stage.py:53` @ `fd4ded7f`) has no recipe row, so `pipeline_kind = a2vid_two_stage` gets the generic table refusal (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1328-1332`) naming the pair rather than the missing machinery. [#922](https://github.com/mudler/vllm.cpp/issues/922) is CLOSED and closed the audio CONDITIONING, not the recipe: a supplied take rides `distilled_two_stage`, which [`ltx25-a2v-audio-input.md`](specs/ltx25-a2v-audio-input.md):438-446 already records as a different trajectory. Measured at `daeff67f2`: `git grep -n '"a2vid' -- src include tests docs examples` returns TWO hits, both upstream anchors inside `Fail`-message assertions (`tests/vllm/multimodal/test_ltx2_video.cpp:4363,:4427`), against a control of 4 for `"one_stage"` in `include/` alone. Four differences from the recipe it rides, each read at the pin: stage 1 is CFG/STG/modality-guided and caller-configured (`:230-240`, fed from `utils/args.py:947-1006`, `--a2v-guidance-scale` defaulting to `video_guider.modality_scale` = 3.0 at `utils/constants.py:54,:64`) where `distilled_two_stage` fixes `allow_guidance_override = false`; stage 1's schedule is scheduler-derived (`:225-227`) against our fixed `DistilledSigmas()`; stage 1 is plain Euler (`:229-258` passes no `stepper`, `utils/blocks.py:526-527`) against our `kEulerAncestral` on 2.5; and the AUDIO guider is the DEFAULT positive-only one (`:237-239`, `ltx-core components/guiders.py:200-210`) rather than the params table's cfg-7.0 row. Two non-schedule facts that must not be guessed: `--audio-path` is `required=True` (`:312-317`), and the distilled LoRA rides stage 2 ALONE (`:114` against `:107`) with `--distilled-lora` `required=True` (`utils/args.py:1140-1153`). Unblocked by `Ltx2GuidedDenoise` landing at `daeff67f2` (#1092/#1102), which [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) `## Owed` names this arm against. Spec [`ltx25-a2vid-recipe.md`](specs/ltx25-a2vid-recipe.md) | enhancement |
 | [#1118](https://github.com/mudler/vllm.cpp/issues/1118) | `LTX25-A2VID-RECIPE` | LoRA adapters fuse ONCE, at load, into ONE weight set: `src/vllm/multimodal/ltx2_video.cpp:816-820` is the only `dit_options.loras.push_back` in the tree and it runs under `if (!lora_path.empty())`, so every phase of every recipe sees the same fused `im.dit.weights`. Three upstream pipelines build TWO `DiffusionStage`s from the same checkpoint with DIFFERENT adapter sets at `fd4ded7f`: `a2vid_two_stage.py:107` against `:114-119`, `ti2vid_two_stages.py:140` against `:151`, and `ti2vid_two_stages_hq.py:154,:165` at two separate strengths (`:92-101`). `distilled.py:131` builds ONE stage set, which is why `distilled_two_stage`, `dfr` and `retake` have never needed this and no gate has ever asked. Consequence for the arm landing with #1117: that row mirrors `--distilled-lora required=True` (`utils/args.py:1140-1153`) by refusing an `a2vid_two_stage` load with no `lora_path`, so the 3-step stage-2 schedule cannot run on a checkpoint with no distilled adapter — but it CANNOT mirror upstream leaving stage 1 WITHOUT that adapter, so stage 1's guided schedule runs against base + distilled LoRA where upstream runs it against the base alone. That divergence RENDERS, and the PIXELS it renders are not upstream's: it moves the trajectory, so the frames themselves differ, while the frame count, the shapes, the sample rate and the errors are all exactly what they were — nothing in the SHAPE of the result says anything is wrong, which is why it is filed rather than left in a comment. It is not undetectable, and saying so would be the more damaging error: the instrument that WOULD see it is a real-weights comparison against upstream's own render on the same checkpoint, take and seed, upstream's stage 1 on the base weights against ours on base + distilled. Two fix shapes, neither chosen: a second `Ltx2DitWeights` per adapter set (what upstream pays, two `from_checkpoint` calls at `a2vid_two_stage.py:103,:115`) or unfused runtime LoRA selectable per phase; the first doubles resident DiT weights and the second changes the GEMM path. Bounds #1093 and #921, which need the same seam. Listed under `## Owed` in [`ltx25-a2vid-recipe.md`](specs/ltx25-a2vid-recipe.md) | enhancement |
 | [#1134](https://github.com/mudler/vllm.cpp/issues/1134) | `BACKEND-ROCM` | The `VT_ATTN_DECODE_D128=1` ctest registration added by #767 cannot show the ROCm `d=128` decode arm REACHED the new kernel, and is empty off ROCm. `RegisteredDevices()` (`tests/vt/test_backend_cross_device.cpp:84-96`) enumerates `{kCUDA, kMETAL, kVULKAN, kXPU, kROCM}` and excludes `kCPU`, so on a CPU-only runner — which is what CI has — the new "Qwen3 geometry (bf16, GQA 2, head_dim 128)" case reports 1 test case, 0 assertions, exit 0, for BOTH registrations. On ROCm hardware the case's only backend assertion is `OpProviderStats::declines == 0`, and `OpProviderStats` counts at PROVIDER granularity, so it is identical with the flag set and unset; the NMSE bound passes on either kernel because the arm is correctness-complete. The two compose: there is no machine in this project on which the flag-ON registration distinguishes itself from the flag-OFF one. Disclosed in [`specs/rocm-decode-attn-d128.md`](specs/rocm-decode-attn-d128.md) §4, its `## Owed` section and its result banner, and §9 stop condition 2 is left OPEN rather than claimed discharged. Closing it needs a kernel-selection counter in `src/vt/rocm/rocm_paged_attn.hip` asserted to DIFFER between the two registrations; the CPU-runner half wants `kCPU` in `RegisteredDevices()` or a non-zero-assertion floor per [#463](https://github.com/mudler/vllm.cpp/issues/463). Not a duplicate of #463 (that is the unset-weights-env-var shape and does not describe the `declines` granularity half), #785 (a kernel that never LAUNCHES behind a dead `#if`, a code defect not a coverage one) or #900 (same family, LTX-2.5 subject) | bug |
+| [#1146](https://github.com/mudler/vllm.cpp/issues/1146) | `ENV-LEASE-RUNTIME-STAGING` | #1129 is closed and its recorded cause is FALSIFIED by the worker image: it says a leased worker "cannot start Python" and lists `python3`, `pip`, `gcc`, `curl` and `git` as ABSENT, measured in one `dgx:gpu0` probe (job `ff28ada1-0cd3-4867-bf9b-f67050d0608b`). Measured 2026-08-17 on `thor:gpu0` through five `rc run` jobs (`6f4bdb03`, `9c0ebeac`, `8beba132`, `f60d945f`, `63c60a90`), that worker runs as `uid=0(root)` with `/usr/bin/gcc`, `/usr/bin/python3` and a working `apt-get`, and a relocated CUDA runtime staged on `/workspace` imports torch 2.13.0+cu130, reports `cuda available = True` on `NVIDIA Thor` capability (11,0), runs a bf16 matmul, and compiles and executes a Triton kernel (`TRITON_JIT_OK = 4096.0 PASS`, `PROBE5_RC=0`). So the image is provisionable per job, which is none of the three fleet-side fixes #1129 names as the only ways forward. Four walls stand between a staged runtime and a running one: the `pip --target` must run FROM the worker because the submitting host is `x86_64` and the workers are `aarch64`. `Python.h` is absent until `apt-get install python3-dev`. The NAS mount presents `file_mode=0664` so Triton cannot execute its own `ptxas-blackwell`, and `TRITON_PTXAS_PATH` does NOT fix it because it redirects only the plain `ptxas`. So `PYTHONPATH` is ORDERED, `/tmp/tp` before the NAS tree. SCOPE, and it is the point of the row: this is `thor:gpu0` at capability (11,0) ONLY, the GB10 is `sm_121a` and UNMEASURED, only `torch`, `triton` and `numpy` are staged so the pinned vLLM oracle is still NOT shown to run and #1129's consequence for the oracle-dependent rows is NARROWED rather than closed, the `+cu130` versus `release 12.8, V12.8.93` skew is recorded as observed and not adjudicated, and a prebuilt wheel does NOT shorten the route because an aarch64 vLLM wheel exists in general while our pin is neither among the wheels published for the one nightly commit nor on PyPI (the per-commit 404s prove nothing, because that URL scheme was never confirmed against a known-good case). Recipe, job IDs and staged-script sha256 values in [`lease-runtime-staging.md`](specs/lease-runtime-staging.md) | verification |
 | [#1033](https://github.com/mudler/vllm.cpp/issues/1033) | `GATE-ISSUE-INDEX-TABLE-SHAPE` | `check_table_shapes` (`scripts/check-agent-record.py:1292`) never ran on `.agents/issue-index.md`, so a malformed row there was invisible to every gate. The function already counted unescaped pipes per table line with exactly the right regex; its call site (`:1527-1530`) passed `roadmap_v1.md`, `coordination.md`, `*MATRIX_PATHS` and `*spec_paths`, and simply did not pass this path. Nothing else in the tree counts this file's cells, which made the index the ONLY markdown table in the record set with no shape gate. It is also the one record surface every change must write, with rows long enough to hide a stray pipe in a code span. Arming it reds exactly one row on `origin/main` at `100026481`: a pipe histogram over the index's 289 table lines reads `{5: 288, 9: 1}`, and the outlier is line 279, the [#1003](https://github.com/mudler/vllm.cpp/issues/1003) `ORACLE-LLAMACPP-REPIN-STOCK` row that arrived with `283c7e492` ([#1051](https://github.com/mudler/vllm.cpp/pull/1051)), carrying four unescaped pipes inside code spans at columns 2705, 3106, 3115 and 3338. FIXED IN FLOW: the path is added, the four pipes are escaped, and three cases in `tests/scripts/test_agent_record.py` hold it — one capturing the paths `main()` really hands the gate, one running it on the shipped file, one mutating a copy so the instrument is proven to fire. Repairing the row EDITS an append-only file, so `check-issue-index-append-only.py` is red on the branch and the exception is argued in the commit body, as `ff264cb82` ([#1025](https://github.com/mudler/vllm.cpp/pull/1025)) argued the same one: appending a corrected copy would leave the broken row in place and add a duplicate key, so the file only becomes well-formed by editing it where it sits. TWO PREMISES OF THE REPORT MEASURED FALSE and are recorded in the spec rather than quietly dropped: the checker does NOT stop at the first finding (one `errors` list, three findings in one run, exit 1 once), and the four pipes are NOT in a `git diff` piped into `grep` — that span does not exist in the row. Spec [`gate-issue-index-table-shape.md`](specs/gate-issue-index-table-shape.md) | bug |