From adf6c792c12acc47558e45be5ac7b556e0884596 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 09:02:23 +0000 Subject: [PATCH] record(ENV-LEASE-RUNTIME-STAGING): a lease compiles, downloads and runs Python, and three records said it could do none of the three (#1213) `AGENTS.md` told every agent that "the leased worker ... has no compiler, no downloader and no Python, so it cannot produce a runtime in place". `.agents/environment.md` carried the matching clause twice for `dgx:gpu0`. All three negatives are false, and the last clause is the consequential one. The clause is the stated basis for treating the pinned vLLM oracle as unreachable from a lease, and that oracle is the denominator for every speed-parity number the project owes. A reader who inherits it plans staging around a wall that is not there, and reaches for `ssh` instead, which is the bypass the lease rule exists to remove. `rc describe dgx:gpu0` prints an operator usage sheet. Under "What you get inside a job" it states that `rc run` executes the command in a container on the host, as root, with the GPU attached, that the container is Ubuntu 24.04, and that it carries `git`, `curl`, `wget`, `ssh`, `gcc`, `g++`, `make`, `cmake`, `ninja`, `pkg-config`, `python3`, `pip` and `venv`. It then instructs "Anything missing, install it - you are root", with `apt-get install` and `python3 -m venv` as its two examples. It names one real limitation, the absent CUDA toolkit. Two jobs then compiled inside a lease on 2026-08-18, so this is not read from a sheet alone. `claude/mudler-ubuntu-box/qwen38-gate` ran `/workspace/qwen38-gate/build.sh`, which apt-installed `cuda-nvcc-13-0`, `cuda-cudart-dev-13-0`, `libcublas-dev-13-0` and `cuda-crt-13-0` from the `ubuntu2404/sbsa` lane after `curl`ing NVIDIA's keyring, configured with `-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a` against a host-staged CUTLASS v4.5.0, and built 1791 of 1791 targets to `BUILD_RC=0` and `BUILD_OK`. Its configure log is durable at `/mnt/nas_share/rc/qwen38-gate/out-main/cfg.log` and records `/usr/local/cuda-13.0/bin/nvcc`, `NVIDIA 13.0.88`, and `CUDA feature cutlass-fp8: ENABLED for [121a]`. Independently, `/mnt/nas_share/rc/mtp_test/build.sh` cloned `https://github.com/mudler/llama.cpp` from inside a job, and left a 97,101,264 byte `libggml-cuda.so` on the share, so downloader access is not a one-off. The correction does not flip into the opposite overstatement, and each part of that is deliberate. The HOST `dgx.casa` still has no egress to `github.com`, which is why `build.sh` stages CUTLASS host-side, while the CONTAINER reached `developer.download.nvidia.com` and `github.com`, so both records now name the side they mean. No CUDA toolkit is preinstalled, so a job installs `nvcc` per run. A global install leaks into the next job until the pod restarts, so project dependencies belong in a virtual environment under `/workspace`. `/workspace` is CIFS and holds no symlink, so a build happens in container-local `/tmp` and is copied out with `cp -rL`. Unconstrained parallelism has OOM-rebooted this box, so `-j 4`. **This claims nothing about the oracle, because nobody has tried.** #1185 already measured that the pinned oracle builds, installs and imports inside a lease on `dgx:gpu0`, and that a model run stays untested. This change claims only what is measured: a CUDA build of THIS tree completed inside a lease, so the premise behind the remaining blocker needs re-testing rather than inheriting. The re-test is recommended in #1213 and is not performed here. The 2026-08-17 probe readings stay in `.agents/environment.md` with their date and job ID, marked SUPERSEDED rather than deleted. That probe was taken correctly and the worker image changed under it, which is the same shape as #1146, and a reader has to be able to tell a stale reading from a wrong one. Records and documents only. `git diff 1f4878fdc -- src/ include/ tests/` is 0 lines. Gate, on `1f4878fdc97a06e6a21d15f856e8fd437d8634ea`: ```sh scripts/agent-preflight.sh --staged ``` `1 gate(s) failed: test_cpu_x86_llamacpp_floor`, rc=1. That harness test is untouched by this change and reds under box load; it reported `waiting for quiet: 15s busy=111% builders=0 load=107.93` while another session's build held the box. Every other gate is `ok`, including `check-agent-record`, `test_agent_record`, `check-issue-index append-only`, `test_gpu_lock_one_truth`, `check-env-doc`, `doc-checkpoint --staged` and `now-current --staged`. Closes nothing. #1213 stays open until the oracle premise is re-tested. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/environment.md | 38 ++++++++++++++++++++++++++------------ .agents/issue-index.md | 1 + AGENTS.md | 17 ++++++++++++++--- 3 files changed, 41 insertions(+), 15 deletions(-) diff --git a/.agents/environment.md b/.agents/environment.md index 5a1564e1b..be5c1537d 100644 --- a/.agents/environment.md +++ b/.agents/environment.md @@ -80,6 +80,9 @@ around it, because the worker image can change under you. **It did change.** The `thor:gpu0` worker measured later the same day carries `python3` and `gcc`, which this list calls absent, so read this section as one box on one day. The `thor` reading is in "A relocated CUDA runtime starts on `thor:gpu0`" further down. +**The `dgx:gpu0` worker then changed as well**, so read the whole section as +history and take the toolchain from +[#1213](https://github.com/mudler/vllm.cpp/issues/1213). - The command runs as user `rc` in a **k3s pod**, hostname `rc-worker-`. `/.dockerenv` is absent and 8 `KUBERNETES_*` variables are set, so it is a pod @@ -87,13 +90,20 @@ reading is in "A relocated CUDA runtime starts on `thor:gpu0`" further down. worker-image question, not a per-job one. - Present: `bash`, `sh`, `ls`, **`nvidia-smi`** (which reports the GB10 by UUID), `flock`, and **`/workspace`**. -- **Absent: `gcc`, `cc`, `clang`, `nvcc`, `ninja`, `cmake`, `make`, `python3`, - `python`, `pip`, `docker`, `sudo`, `git`, `ssh`, `curl`, - `/usr/include/stdio.h`, and any `/usr/local/cuda*` toolkit.** This `dgx:gpu0` - worker cannot compile, cannot start Python, and cannot install anything. - **Do not carry that clause to another device.** On `thor:gpu0` the same day the - worker ran as `uid=0(root)` with `/usr/bin/gcc`, `/usr/bin/python3` and a - working `apt-get` ([#1146](https://github.com/mudler/vllm.cpp/issues/1146)). +- **Absent on 2026-08-17: `gcc`, `cc`, `clang`, `nvcc`, `ninja`, `cmake`, + `make`, `python3`, `python`, `pip`, `docker`, `sudo`, `git`, `ssh`, `curl`, + `/usr/include/stdio.h`, and any `/usr/local/cuda*` toolkit.** That probe read + the worker as unable to compile, to start Python, or to install anything. + **That reading is SUPERSEDED.** `rc describe dgx:gpu0` now states that the job + runs as root in an Ubuntu 24.04 container carrying `git`, `curl`, `wget`, + `ssh`, `gcc`, `g++`, `make`, `cmake`, `ninja`, `pkg-config`, `python3`, `pip` + and `venv`, and it instructs the reader to install anything missing. The one + limit the sheet names is the CUDA toolkit, which a job apt-installs per run + ([#1213](https://github.com/mudler/vllm.cpp/issues/1213)). + **Do not carry a one-box reading to another device either.** On `thor:gpu0` + the same day the worker ran as `uid=0(root)` with `/usr/bin/gcc`, + `/usr/bin/python3` and a working `apt-get` + ([#1146](https://github.com/mudler/vllm.cpp/issues/1146)). - **The host filesystem is not visible.** `/home/mudler` does not exist inside the worker. - `/workspace` is the house NAS, measured as `//192.168.68.102/Data 7.3T total, @@ -170,12 +180,16 @@ exec bit, and sit on a 3.6T overlay with 2.5T available. `/dev/shm` is 64M. The job's working directory `/` is not writable. **So the lease carries bytes, and bytes are enough to run.** A runtime staged on -`/workspace` can start under the dynamic loader, or after a copy to `/tmp`. What -this `dgx:gpu0` worker cannot do is produce or fetch that runtime, because it has -no `curl`, `wget`, `git`, `gcc`, `nvcc`, `cmake` or `python3`. Present and useful +`/workspace` can start under the dynamic loader, or after a copy to `/tmp`. +**This paragraph used to add that the `dgx:gpu0` worker cannot produce or fetch +that runtime, because it had no `curl`, `wget`, `git`, `gcc`, `nvcc`, `cmake` or +`python3`. That clause is SUPERSEDED.** The worker runs as root and carries every +one of those names except the CUDA toolkit, and a job apt-installs `nvcc` from +`developer.download.nvidia.com` per run +([#1213](https://github.com/mudler/vllm.cpp/issues/1213)). Present and useful for staging: `cp`, `cat`, `tar`, `chmod`, `perl`, `flock` and `nvidia-smi`. **The -`thor:gpu0` worker does produce one**, because it is root and carries `apt-get` -and `gcc`. That is the section below. +`thor:gpu0` worker produces one as well**, because it is root and carries +`apt-get` and `gcc`. That is the section below. **This narrows [#1129](https://github.com/mudler/vllm.cpp/issues/1129) and does not close it.** The HOST venv at `~/venvs/vllm-oracle-pin-555967922` stays diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 38997a380..cae09070f 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -367,3 +367,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1181](https://github.com/mudler/vllm.cpp/issues/1181) | `FIX-READ-F32-SCALAR-GUARD` | `ReadF32Scalar` (`src/vllm/model_executor/models/qwen3_5_weights.cpp:312-318` @ `ab6e65216`) bounds its input with `t.data != nullptr && t.nbytes >= sizeof(float)`, a LOWER bound, and then `memcpy`s four bytes into a `float`. Two silent wrong-value paths follow and neither fails: an ARRAY is reduced to element 0, so a block-wise FP8 scale grid of shape `[ceil(N/128), ceil(K/128)]` passes and stands in for the whole weight (measured under [#1166](https://github.com/mudler/vllm.cpp/issues/1166) on `Qwen/Qwen3.8-27B-FP8` @ `017b9c7af6b5689d5dd426a76e0bc077eb5ca20a`, `q_proj.weight_scale_inv` is `[96, 40]`), and ANY dtype is reinterpreted, since that same tensor is `BF16` and its four bytes are two bf16 values read as one float. Both return a finite plausible float, so the output is fluent, plausible and wrong, which is what a token gate cannot see. Upstream makes both facts structural rather than optional: a per-tensor scale is a distinct parameter TYPE that asserts `loaded_weight.shape[0] == 1` (`vllm/model_executor/parameter.py:260-272,304-309` @ `555967922`, plus the `_assert_and_load` shape assert at `:93-96`), the slot is allocated `torch.float32` so a narrow on-disk dtype is VALUE-converted rather than reinterpreted (`utils/fp8_utils.py:1276`), and the declared strategy TENSOR/CHANNEL/BLOCK picks the parameter type before a byte is read (`compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py:63,128`). The AUDIT corrects the issue's own framing twice. The 27 grep hits across five files are 5 definitions, 20 call sites and 2 comment references, and both counts are short: `ReadCtF32Scalar` (`include/vllm/model_executor/models/dense_weight_loaders.h:376`) is a SIXTH copy of the same defect under another name, reached from a SIXTH model file (`src/vllm/model_executor/models/qwen3_weights.cpp:100,126-128` through `LoadCtNvfp4W4A16`). Of the six, three check nothing, `LnReadF32Scalar`/`ShReadF32Scalar` check dtype but not count, and only `nemotron_h_weights.cpp:557-573` is correct, which makes it the model the shared guard generalizes. No call site legitimately passes a multi-element or non-F32 tensor, and every existing fixture emits rank-0 or `{1}` `F32`, so nothing in the tree needed the leniency. It is NOT merely latent: `dense_weight_loaders.h:73-74` and `docs/BENCHMARKS.md:52` both record `unsloth/Qwen3.6-27B-NVFP4` @ `ccdaab7e` as FP8 W8A8 throughout with BF16 PER-OUTPUT-CHANNEL scales, and `LoadAttnDense` branches on the weight dtype alone (`qwen3_5_dense_weights.cpp:478-480`), so those projections enter the per-tensor arm and hit both defects at once under the tensor name the loader actually asked for, with no misspelling to stop them. Fixed in flow by one `dense_loaders::ReadF32Scalar(get, name)` that refuses `numel != 1` naming the shape, refuses a non-`F32` dtype naming the dtype, and requires exactly four readable bytes, with the other five copies deleted onto it and `nemotron_h`'s `Loader`-based twin kept as the one tracked exception. A narrow dtype is refused rather than converted, because a one-element BF16 scale has never been read correctly here and the BF16 layout that IS shipped is per-channel, which the count check refuses first. Per-channel FP8, block-wise FP8 and any explicit narrow-dtype conversion stay owed. Spec [`read-f32-scalar-guard.md`](specs/read-f32-scalar-guard.md) | bug | | [#1185](https://github.com/mudler/vllm.cpp/issues/1185) | `ENV-ORACLE-WHEEL-IN-LEASE` | The pinned vLLM oracle BUILDS, installs, imports and sees the GPU inside an `rc` lease on `dgx:gpu0`, measured 2026-08-18, which falsifies the `nvcc` clause four records carried. [`lease-runtime-staging.md`](specs/lease-runtime-staging.md) said the oracle "needs `nvcc`, which the worker still lacks", and `.agents/environment.md`, [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) and [`gpu-lease-methodology.md`](specs/gpu-lease-methodology.md) each derived a blocker from it. The build job (`buildvllm.sh`, staged sha256 prefix `15e140d41f44e7c2`) asserted the checkout against the pin BEFORE compiling, printing `PIN CONFIRMED` at `5559679229bc961848b121ccdeaa8fa5d79bec98` and aborting otherwise, took `nvcc` from the toolkit row `MODEL-NEMOTRON-H-ABI-A3-E2E` staged (`NVCC_RC=0`, CUDA `release 13.3, V13.3.73`) and produced `WHEEL_RC=0`, `PERSIST_RC=0` and a 434 MiB `vllm-0.1.dev1+g555967922.cu133-cp312-cp312-linux_aarch64.whl`, sha256 `7c58b339741a288fbb313f4f5196c9c92a9e3b3c3ebe2ea970b0ff50bb9bcba4`. The identity job (`oracleenv.sh`, prefix `6119f5223f5d818c`) asserted from `cd /`, outside any source tree: `vllm.__version__ = 0.1.dev1+g555967922`, `IDENTITY_RC=0`, `cuda True NVIDIA GB10`, `CUDA_RC=0`. SCOPE, and it carries the same weight as the result: RUNNING A MODEL IS UNTESTED. Only build, install, import and `torch.cuda.is_available()` are measured, and [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) records that the last time an oracle reached this far it consumed the host in the step AFTER `torch.compile` and REBOOTED the box, at `gpu_memory_utilization` 0.75 and again at 0.30, so the fraction is not the lever. The version string is an OPEN discrepancy: `.agents/upstream-sync.md` records `vllm_runtime_version = 0.23.1rc1.dev1511+g555967922`, the commit segment matches and satisfies the pin's binding `+g` rule, and the prefix differs because a shallow fetch stops `setuptools_scm` counting commits since the last tag, so a full-string gate needs a deeper fetch or a recorded pretend-version. The venv is NOT staged, because that job was killed at a 90-minute ceiling mid-copy and its partial tree was removed, so only the WHEEL is durable. Four staging walls, all artifacts of the NAS rather than of CUDA: `cp -a` preserves `file_mode=0664` so `nvcc` exited 126. CIFS `nounix` stores no symlink so `include` and `lib64` vanished and CMake reported `Could NOT find CUDA (missing: CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) (found version "13.3")`, naming the version and denying the toolkit in one line. 32 library links `libfoo.so` and `libfoo.so.MAJOR` had to be rebuilt because only the `libfoo.so.X.Y.Z` real files survived. And `markupsafe` existed as a dist-info with NO package files from a `pip --target` killed at a 35-minute ceiling, so Marlin codegen died on `ModuleNotFoundError`. The `rc` worker container is REUSED between jobs, so a repair inside a staging branch is skipped on the next run (`nvcc already in place`) and an environment repair must be unconditional and assert its postcondition. CONSEQUENCE for the rows #1129 blocked, [#1003](https://github.com/mudler/vllm.cpp/issues/1003), [#915](https://github.com/mudler/vllm.cpp/issues/915), [#821](https://github.com/mudler/vllm.cpp/issues/821) and [#81](https://github.com/mudler/vllm.cpp/issues/81): UNBLOCKED FOR THE BUILD STEP and STILL BLOCKED FOR A MODEL RUN. None can take a measurement until a model run is demonstrated. Job details, walls and non-claims in [`oracle-wheel-in-lease.md`](specs/oracle-wheel-in-lease.md) | verification | | [#1193](https://github.com/mudler/vllm.cpp/issues/1193) | `SPEC-DSPARK-QWEN3-ROUTING` | A Qwen3 DSpark draft declaring `architectures=["DSparkDraftModel"]` with `model_type` `qwen3` has no route. The pin forces every DSpark draft that is not `Qwen3DSparkModel` or `Gemma4DSparkModel` onto `model_type` `deepseek_v4` (`vllm/config/speculative.py:934-944` @ `555967922`), and vLLM PR 52197 (merged 2026-08-17 at `7075ddac`) replaced that with a leading branch normalizing the pair to `Qwen3DSparkModel`. We diverge from BOTH: the forced rewrite was never ported, so nothing in `src/vllm/entrypoints/model_loader.cpp` reads a draft config's `architectures` key at all, and `SpeculativeConfig::IsDsparkDraft` (`include/vllm/config/speculative.h:120-136`) has no production caller — every reference outside its header is in `tests/vllm/config/test_speculative_dspark.cpp:132-140`, and `ResolveSpecConfig` branches on `cli.method` alone. The checkpoint is real and gateable here: `RadixArk/Qwen3.8-27B-DSpark` at revision `85ef153be924f17ce4bf62726954eeaa4a73e854` carries exactly that config shape in one 2718576122-byte shard, drafting five layers for a 64-layer Qwen3.8-27B target | bug | +| [#1213](https://github.com/mudler/vllm.cpp/issues/1213) | `ENV-LEASE-RUNTIME-STAGING` | `AGENTS.md` stated that a leased worker "has no compiler, no downloader and no Python, so it cannot produce a runtime in place", and `.agents/environment.md` carried the matching clause twice for `dgx:gpu0`. All three negatives are false. `rc describe dgx:gpu0` states that a job runs as root in an Ubuntu 24.04 container carrying `git`, `curl`, `wget`, `ssh`, `gcc`, `g++`, `make`, `cmake`, `ninja`, `pkg-config`, `python3`, `pip` and `venv`, and it instructs the reader to install anything missing; the one limit it names is the absent CUDA toolkit. Two jobs then compiled inside a lease on 2026-08-18: `claude/mudler-ubuntu-box/qwen38-gate` apt-installed `cuda-nvcc-13-0` from the `ubuntu2404/sbsa` lane and built this tree 1791/1791 to `BUILD_RC=0` (`/mnt/nas_share/rc/qwen38-gate/out-main/cfg.log` records `nvcc` 13.0.88 and `CUDA feature cutlass-fp8: ENABLED for [121a]`), and `/mnt/nas_share/rc/mtp_test/build.sh` cloned `github.com/mudler/llama.cpp` from inside a job and left a 97 MB `libggml-cuda.so` on the share. **Why it matters:** "the lease cannot produce a runtime in place" is the stated basis for treating the pinned vLLM oracle as unreachable from a lease, and that oracle is the denominator for every speed-parity number the project owes, so the premise needs re-testing rather than inheriting. This claims nothing about a model run; #1185 owns that and stays open. FIXED IN FLOW: the `AGENTS.md` paragraph and both `.agents/environment.md` clauses now say what the measurement supports, and the four real limits (no preinstalled CUDA toolkit, global installs leak until the pod restarts, CIFS `/workspace` holds no symlink so build in `/tmp` and `cp -rL`, and `-j 4` because unconstrained parallelism OOM-reboots the box) plus the host-versus-container egress distinction ride with the correction. | record | diff --git a/AGENTS.md b/AGENTS.md index 325d5f144..f4a111604 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -443,9 +443,20 @@ speed axis as VOID because of it. That is the #777 failure again, in which this repository carried two GPU mutexes and neither serialised the other. A bypass also makes the fleet report the box free while somebody is on it. -**A lease carries bytes, not executables.** The leased worker reads and writes -the shared `/workspace`, and it has no compiler, no downloader and no Python, so -it cannot produce a runtime in place. Plan staging around that limit. +**A lease runs as root, and the worker provisions itself.** The leased worker +reads and writes the shared `/workspace`, and on `dgx:gpu0` it is an Ubuntu 24.04 +container carrying `git`, `curl`, `wget`, `gcc`, `cmake`, `ninja`, `python3` and +`pip`. It installs what it lacks and it compiles in place. On 2026-08-18 one job +apt-installed `cuda-nvcc-13-0` and built this tree for `sm_121a` +([#1213](https://github.com/mudler/vllm.cpp/issues/1213)). + +**Four limits are still real, and they shape staging.** No CUDA toolkit is +preinstalled, so a job that compiles CUDA installs one first. A global install +leaks into the next job, so put project dependencies in a virtual environment +under `/workspace`. `/workspace` is CIFS and holds no symlink, so build in `/tmp` +and copy out with `cp -rL`. Unconstrained parallelism has OOM-rebooted this box, +so use `-j 4`. The HOST `dgx.casa` has no egress to `github.com` while the +container does, so name the side you mean. [`.agents/environment.md`](.agents/environment.md) carries the fleet, the measurement, and the procedure.