The `rc` leased worker on `dgx:gpu0` compiles, downloads and runs Python, and `AGENTS.md` says it can do none of the three

[localai-bot]

The claim

AGENTS.md, in ## Work on a GPU happens inside a lease, states:

A lease carries bytes, not executables. The leased worker reads and writes
the shared /workspace, and it has no compiler, no downloader and no Python, so
it cannot produce a runtime in place. Plan staging around that limit.

.agents/environment.md carries the matching claim in two places. The section
"What the dgx:gpu0 leased worker can and cannot do, measured 2026-08-17" lists
gcc, cmake, ninja, make, python3, pip, git and curl as absent and
concludes "This dgx:gpu0 worker cannot compile, cannot start Python, and cannot
install anything." The section "The lease carries bytes, and the exec bit is a
mount option" repeats it: "What this dgx:gpu0 worker cannot do is produce or
fetch that runtime, because it has no curl, wget, git, gcc, nvcc,
cmake or python3."

All three negatives are false on dgx:gpu0. The consequential one is the last
clause, "it cannot produce a runtime in place".

Evidence

1. The rc operator usage sheet. rc describe dgx:gpu0 prints a host-wide
note. Under "What you get inside a job" it states that rc run executes the
command in a container on the host, as root, with the GPU attached, that the
container is Ubuntu 24.04, and that it already carries:

bash git curl wget ssh  |  gcc g++ make cmake ninja pkg-config
python3 pip venv        |  jq rsync tar unzip  |  kubectl  |  nvidia-smi

The sheet then instructs "Anything missing, install it - you are root", and
gives apt-get install and python3 -m venv /workspace/venv as the two worked
examples. It names one real limitation: "There is no CUDA toolkit (no
nvcc). The driver is injected and nvidia-smi works, but compiling CUDA needs
the toolkit - install it, or keep the build on the host."

2. A completed CUDA build of this tree, inside a lease. On 2026-08-18 a job
submitted as claude/mudler-ubuntu-box/qwen38-gate on dgx:gpu0 ran
/workspace/qwen38-gate/build.sh. That script apt-installs
cuda-nvcc-13-0 cuda-cudart-dev-13-0 libcublas-dev-13-0 cuda-crt-13-0 from the
ubuntu2404/sbsa lane after curling NVIDIA's keyring, configures with
-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a against a host-staged
CUTLASS v4.5.0, and builds with ninja -j 4. It compiled 1791 of 1791 targets to
BUILD_RC=0 and BUILD_OK.

The configure log is durable at /mnt/nas_share/rc/qwen38-gate/out-main/cfg.log,
which the job writes as /workspace/qwen38-gate/out-main/cfg.log. It records the
compilers the container resolved and the arch the build enabled:

-- The CXX compiler identification is GNU 13.3.0
-- Looking for a CUDA compiler - /usr/local/cuda-13.0/bin/nvcc
-- The CUDA compiler identification is NVIDIA 13.0.88
-- CUDA target architectures: 121a
--   CUDA feature cutlass-fp8: ENABLED for [121a]

The source tarball for that run is src-main-fd64c76ee.tar.gz, and
src-main.sha reads fd64c76ee45ba49b070ea83024f6678ddd7f64a6.

3. A second, independent instance. /mnt/nas_share/rc/mtp_test/build.sh is
another session's script. It git clones https://github.com/mudler/llama.cpp
and curls NVIDIA's repo from inside a job, installs the same four CUDA
packages, and builds with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="110;121".
rc describe dgx:gpu0 showed that job, bash /workspace/mtp_test/build.sh, as
succeeded in 9m56s. /mnt/nas_share/rc/mtp_test/bin/ holds the artifacts,
including libggml-cuda.so at 97,101,264 bytes, dated 2026-08-18 07:01. So
downloader access from inside the container is not a one-off.

What stays true, and must not be flipped into the opposite overstatement

• The host dgx.casa has no egress to github.com or huggingface.co. That
  is why build.sh stages CUTLASS host-side rather than fetching it. The
  container reached developer.download.nvidia.com, and per evidence 3 also
  github.com. Host and container are different network positions, and every
  record has to say which one it means.
• There is genuinely no CUDA toolkit preinstalled. nvcc must be
  apt-installed per job. The container is long-lived, so the install may persist
  until the pod restarts.
• Installing globally leaks into other jobs until the pod restarts. Project
  dependencies belong in a venv under /workspace.
• /workspace is CIFS and cannot hold symlinks, so a build has to happen in
  container-local /tmp and be copied out with cp -rL.
• Unconstrained parallelism has OOM-rebooted this box. Use -j 4.

Why it matters

"The lease cannot produce a runtime in place" is the stated basis for treating
the pinned vLLM oracle as unreachable from a lease. That oracle is the
denominator for every speed-parity number the project owes. If a lease can
install and compile, the blocker is far narrower than the records say.

This issue does not claim the oracle now builds. #1185 already measured that
the pinned oracle builds, installs and imports inside a lease on dgx:gpu0, and
a model run stays untested. This issue claims only what is measured here: a CUDA
build of this tree completed inside a lease, so the premise behind the remaining
blocker needs re-testing rather than inheriting.

Recommendation

Re-test the premise with a probe rather than by argument. Do not perform it as
part of the record correction.

Owner

ENV-LEASE-RUNTIME-STAGING owns the staging problem
(.agents/specs/lease-runtime-staging.md, #1146). The record correction to
AGENTS.md and .agents/environment.md rides in flow with this issue.