record(ENV-LEASE-RUNTIME-STAGING): the leased worker compiles, downloads and runs Python, and AGENTS.md said it could do none of the three

.agents/environment.md

@@ -80,20 +80,30 @@ around it, because the worker image can change under you. **It did change.** The
 `thor:gpu0` worker measured later the same day carries `python3` and `gcc`, which
 this list calls absent, so read this section as one box on one day. The `thor`
 reading is in "A relocated CUDA runtime starts on `thor:gpu0`" further down.
+**The `dgx:gpu0` worker then changed as well**, so read the whole section as
+history and take the toolchain from
+[#1213](https://github.com/mudler/vllm.cpp/issues/1213).
 
 - The command runs as user `rc` in a **k3s pod**, hostname `rc-worker-<id>`.
   `/.dockerenv` is absent and 8 `KUBERNETES_*` variables are set, so it is a pod
   rather than a docker container. The toolchain question is therefore a
   worker-image question, not a per-job one.
 - Present: `bash`, `sh`, `ls`, **`nvidia-smi`** (which reports the GB10 by UUID),
   `flock`, and **`/workspace`**.
-- **Absent: `gcc`, `cc`, `clang`, `nvcc`, `ninja`, `cmake`, `make`, `python3`,
-  `python`, `pip`, `docker`, `sudo`, `git`, `ssh`, `curl`,
-  `/usr/include/stdio.h`, and any `/usr/local/cuda*` toolkit.** This `dgx:gpu0`
-  worker cannot compile, cannot start Python, and cannot install anything.
-  **Do not carry that clause to another device.** On `thor:gpu0` the same day the
-  worker ran as `uid=0(root)` with `/usr/bin/gcc`, `/usr/bin/python3` and a
-  working `apt-get` ([#1146](https://github.com/mudler/vllm.cpp/issues/1146)).
+- **Absent on 2026-08-17: `gcc`, `cc`, `clang`, `nvcc`, `ninja`, `cmake`,
+  `make`, `python3`, `python`, `pip`, `docker`, `sudo`, `git`, `ssh`, `curl`,
+  `/usr/include/stdio.h`, and any `/usr/local/cuda*` toolkit.** That probe read
+  the worker as unable to compile, to start Python, or to install anything.
+  **That reading is SUPERSEDED.** `rc describe dgx:gpu0` now states that the job
+  runs as root in an Ubuntu 24.04 container carrying `git`, `curl`, `wget`,
+  `ssh`, `gcc`, `g++`, `make`, `cmake`, `ninja`, `pkg-config`, `python3`, `pip`
+  and `venv`, and it instructs the reader to install anything missing. The one
+  limit the sheet names is the CUDA toolkit, which a job apt-installs per run
+  ([#1213](https://github.com/mudler/vllm.cpp/issues/1213)).
+  **Do not carry a one-box reading to another device either.** On `thor:gpu0`
+  the same day the worker ran as `uid=0(root)` with `/usr/bin/gcc`,
+  `/usr/bin/python3` and a working `apt-get`
+  ([#1146](https://github.com/mudler/vllm.cpp/issues/1146)).
 - **The host filesystem is not visible.** `/home/mudler` does not exist inside
   the worker.
 - `/workspace` is the house NAS, measured as `//192.168.68.102/Data 7.3T total,
@@ -170,12 +180,16 @@ exec bit, and sit on a 3.6T overlay with 2.5T available. `/dev/shm` is 64M. The
 job's working directory `/` is not writable.
 
 **So the lease carries bytes, and bytes are enough to run.** A runtime staged on
-`/workspace` can start under the dynamic loader, or after a copy to `/tmp`. What
-this `dgx:gpu0` worker cannot do is produce or fetch that runtime, because it has
-no `curl`, `wget`, `git`, `gcc`, `nvcc`, `cmake` or `python3`. Present and useful
+`/workspace` can start under the dynamic loader, or after a copy to `/tmp`.
+**This paragraph used to add that the `dgx:gpu0` worker cannot produce or fetch
+that runtime, because it had no `curl`, `wget`, `git`, `gcc`, `nvcc`, `cmake` or
+`python3`. That clause is SUPERSEDED.** The worker runs as root and carries every
+one of those names except the CUDA toolkit, and a job apt-installs `nvcc` from
+`developer.download.nvidia.com` per run
+([#1213](https://github.com/mudler/vllm.cpp/issues/1213)). Present and useful
 for staging: `cp`, `cat`, `tar`, `chmod`, `perl`, `flock` and `nvidia-smi`. **The
-`thor:gpu0` worker does produce one**, because it is root and carries `apt-get`
-and `gcc`. That is the section below.
+`thor:gpu0` worker produces one as well**, because it is root and carries
+`apt-get` and `gcc`. That is the section below.
 
 **This narrows [#1129](https://github.com/mudler/vllm.cpp/issues/1129) and does
 not close it.** The HOST venv at `~/venvs/vllm-oracle-pin-555967922` stays

.agents/issue-index.md

@@ -369,3 +369,4 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1189](https://github.com/mudler/vllm.cpp/issues/1189) | `VT-QUANT-FP8-GROUP` | Block-wise (128x128) FP8 so `Qwen/Qwen3.8-27B-FP8` runs instead of being refused. `weight_block_size` appears nowhere in `src/` or `include/`, so `469f38395` refuses the arm by name (#1166). Six independently landable milestones; M1 lands here. M1 is `vt::QuantFp8Group`, the dynamic per-token per-group activation quant, CPU and CUDA. The numerics mirror the kernel that ACTUALLY EXECUTES on a CUDA-alike platform with a contiguous input, which is the C++ custom op at `csrc/libtorch_stable/quantization/w8a8/fp8/per_token_group_quant.cu:42-96` and NOT the Triton kernel at `fp8_utils.py:95-150`: `fp8_utils.py:635-650` calls the former and returns before the latter. The two arms differ, and the difference is measurable rather than cosmetic. The CUDA kernel divides twice, `local_absmax / max_8bit` at `:68` and `static_cast<float>(src) / y_s` at `:85`; the Triton kernel multiplies by `(1.0 / fp8_max)` at `fp8_utils.py:145` under a comment that names the 1-ULP gap. Upstream's own test tolerates the gap with `rtol=0.15` (`test_block_fp8.py:112-114`), so a value comparison cannot tell the two apart and only a byte comparison against a spelled-out reference can. `eps` is the reduction's INITIAL value (`:47`), not a post-clamp, which is what keeps an all-zero group from dividing by zero. Scope refused here and owed to later milestones: the block-scaled GEMM (M2), `Fp8BlockWeight` and the loader (M3), `Fp8BlockLinearMethod` and the Qwen3.5 wiring (M4), the mainloop-scaled CUTLASS kernel and the column-major/TMA-aligned scale layouts (M5), merged `gate_up`/QKV (M6). M1 lands UNREACHED: no production entry point dispatches `vt::QuantFp8Group` at its merge commit, M4 owns the wiring, and `.agents/specs/vt-quant-fp8-group.md` lists it under `## Owed`. The CUDA arm compiles and its on-hardware leg is owed too, because the row took no GPU lease by design: the CPU arm is the gateable one | feature |
 | [#1190](https://github.com/mudler/vllm.cpp/issues/1190) | `ENV-AGNOSTIC-CAMPAIGN` | One operator's hosts, share paths and addresses are written into 227 tracked files where a `.env` placeholder belongs, so a second developer who follows the protocol documents is told to reach a box on one home network. Re-derived at `fd64c76ee` with `git grep -cIE 'dgx\.casa\|nas_share\|192\.168\.\|thor:gpu0'`: `dgx.casa` 203 files, `nas_share` 32, `192.168.` 24, `thor:gpu0` 11. The mechanism was already there and almost unused: `.env.example` is tracked, `.env` and `.agents/developer-preferences.md` are ignored, `scripts/agent-onboard.py --env-set` already records one answered value and refuses an undeclared key, and `${VLLM_SOURCE}` and `${GPU_LOCK}` already resolve in 59 and 50 files while `${DEVICE_ARCH}`, `${DEVICE_TOOLKIT_ROOT}` and `${DEVICE_COMPILER}` resolve in none. This SCOPING row lands the rule and the mechanism, not the sweep. The rule is a substitution test: replace the literal with a second developer's value and if the sentence stays true it is CONFIGURATION and becomes `${KEY}`, and if it becomes false it is PROVENANCE and stays literal. That reverses the issue's own ranking, because all 34 hits in the densest guide file `.agents/environment.md` are provenance or named-profile definition and none is configuration, so density does not predict the defect and a blind `sed` would falsify records. Landed: three new keys `GATE_CHECKOUT`, `SHARED_STORAGE_ROOT` and `GATE_DEVICE` derived from the literals that recur and map to no existing key; the create-on-first-use route in `scripts/agent-start.py`, which printed `environment: missing` as a status label and then listed next actions that never mentioned it, so the fallback in practice was a host name copied from a document; the matching obligation in `AGENTS.md`; and one worked example. The worked example `scripts/dgx-bringup.sh` found a live defect rather than a cosmetic one: it defaulted `CUTLASS_DIR` to `$HOME/cutlass_probe` while `.agents/environment.md:389` records `$HOME/cutlass-4.5.0` as mandatory on the same box, and a configure that misses CUTLASS silently drops the sm120a NVFP4 GEMM and FlashAttention-2, which that file measures as moving the SACRED `test_qwen27_paged_engine` from 235/235 to 234/235 with the source untouched, so a stale hard-coded default is a false green. Its new test also caught that `set -a; . ./.env; set +a` over an `.env.example` copy blanks a value the caller exported, so the process-environment contract is now executable. Waves `ENV-AGNOSTIC-W1-TOOLING` through `ENV-AGNOSTIC-W5-LEDGERS` own the sweep, partition all 227 files, and open their own issues. Spec [`env-agnostic.md`](specs/env-agnostic.md) | bug |
 | [#1193](https://github.com/mudler/vllm.cpp/issues/1193) | `SPEC-DSPARK-QWEN3-ROUTING` | A Qwen3 DSpark draft declaring `architectures=["DSparkDraftModel"]` with `model_type` `qwen3` has no route. The pin forces every DSpark draft that is not `Qwen3DSparkModel` or `Gemma4DSparkModel` onto `model_type` `deepseek_v4` (`vllm/config/speculative.py:934-944` @ `555967922`), and vLLM PR 52197 (merged 2026-08-17 at `7075ddac`) replaced that with a leading branch normalizing the pair to `Qwen3DSparkModel`. We diverge from BOTH: the forced rewrite was never ported, so nothing in `src/vllm/entrypoints/model_loader.cpp` reads a draft config's `architectures` key at all, and `SpeculativeConfig::IsDsparkDraft` (`include/vllm/config/speculative.h:120-136`) has no production caller — every reference outside its header is in `tests/vllm/config/test_speculative_dspark.cpp:132-140`, and `ResolveSpecConfig` branches on `cli.method` alone. The checkpoint is real and gateable here: `RadixArk/Qwen3.8-27B-DSpark` at revision `85ef153be924f17ce4bf62726954eeaa4a73e854` carries exactly that config shape in one 2718576122-byte shard, drafting five layers for a 64-layer Qwen3.8-27B target | bug |
+| [#1213](https://github.com/mudler/vllm.cpp/issues/1213) | `ENV-LEASE-RUNTIME-STAGING` | `AGENTS.md` stated that a leased worker "has no compiler, no downloader and no Python, so it cannot produce a runtime in place", and `.agents/environment.md` carried the matching clause twice for `dgx:gpu0`. All three negatives are false. `rc describe dgx:gpu0` states that a job runs as root in an Ubuntu 24.04 container carrying `git`, `curl`, `wget`, `ssh`, `gcc`, `g++`, `make`, `cmake`, `ninja`, `pkg-config`, `python3`, `pip` and `venv`, and it instructs the reader to install anything missing; the one limit it names is the absent CUDA toolkit. Two jobs then compiled inside a lease on 2026-08-18: `claude/mudler-ubuntu-box/qwen38-gate` apt-installed `cuda-nvcc-13-0` from the `ubuntu2404/sbsa` lane and built this tree 1791/1791 to `BUILD_RC=0` (`/mnt/nas_share/rc/qwen38-gate/out-main/cfg.log` records `nvcc` 13.0.88 and `CUDA feature cutlass-fp8: ENABLED for [121a]`), and `/mnt/nas_share/rc/mtp_test/build.sh` cloned `github.com/mudler/llama.cpp` from inside a job and left a 97 MB `libggml-cuda.so` on the share. **Why it matters:** "the lease cannot produce a runtime in place" is the stated basis for treating the pinned vLLM oracle as unreachable from a lease, and that oracle is the denominator for every speed-parity number the project owes, so the premise needs re-testing rather than inheriting. This claims nothing about a model run; #1185 owns that and stays open. FIXED IN FLOW: the `AGENTS.md` paragraph and both `.agents/environment.md` clauses now say what the measurement supports, and the four real limits (no preinstalled CUDA toolkit, global installs leak until the pod restarts, CIFS `/workspace` holds no symlink so build in `/tmp` and `cp -rL`, and `-j 4` because unconstrained parallelism OOM-reboots the box) plus the host-versus-container egress distinction ride with the correction. | record |

AGENTS.md

@@ -454,9 +454,20 @@ speed axis as VOID because of it. That is the #777 failure again, in which this
 repository carried two GPU mutexes and neither serialised the other. A bypass
 also makes the fleet report the box free while somebody is on it.
 
-**A lease carries bytes, not executables.** The leased worker reads and writes
-the shared `/workspace`, and it has no compiler, no downloader and no Python, so
-it cannot produce a runtime in place. Plan staging around that limit.
+**A lease runs as root, and the worker provisions itself.** The leased worker
+reads and writes the shared `/workspace`, and on `dgx:gpu0` it is an Ubuntu 24.04
+container carrying `git`, `curl`, `wget`, `gcc`, `cmake`, `ninja`, `python3` and
+`pip`. It installs what it lacks and it compiles in place. On 2026-08-18 one job
+apt-installed `cuda-nvcc-13-0` and built this tree for `sm_121a`
+([#1213](https://github.com/mudler/vllm.cpp/issues/1213)).
+
+**Four limits are still real, and they shape staging.** No CUDA toolkit is
+preinstalled, so a job that compiles CUDA installs one first. A global install
+leaks into the next job, so put project dependencies in a virtual environment
+under `/workspace`. `/workspace` is CIFS and holds no symlink, so build in `/tmp`
+and copy out with `cp -rL`. Unconstrained parallelism has OOM-rebooted this box,
+so use `-j 4`. The HOST `dgx.casa` has no egress to `github.com` while the
+container does, so name the side you mean.
 [`.agents/environment.md`](.agents/environment.md) carries the fleet, the
 measurement, and the procedure.