mudler · localai-bot · Aug 17, 2026 · Aug 17, 2026 · Aug 17, 2026 · Aug 17, 2026
diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -19,6 +19,109 @@ from relative link targets repointed for this file's location.
 
 # Benchmarks
 
+## MODEL-NEMOTRON-H-ABI-A3-E2E — the A3 token gate did NOT run, and the cause on record was NOT the cause (2026-08-17, `row/MODEL-NEMOTRON-H-ABI-A3-E2E`, base `origin/main` `a6df72777`, #810)
+
+**No number is recorded, on any axis. This entry exists so the pending cause is
+the measured one rather than the inherited one.**
+
+`.agents/specs/nemotron-h-a2p-paged-forward.md` §10 recorded the A3 gate as
+pending on **contention**: `dgx.casa` observed at loadavg 211 with 3 of 119 GB
+available, which is a real reason a 20.1 GiB checkpoint cannot load. Re-measured
+on 2026-08-17 under an `rc` lease, that box answers at **loadavg 0.36, 115 of
+119 GB available, GPU utilisation 0%**, with nothing of ours running on it. The
+recorded cause is no longer true, and this is the #775 shape the governing spec
+§5.5 warns about: a pending reason that outlives its own truth and gets
+subtracted by everyone who reads it afterwards.
+
+**★ THIS ENTRY WAS FIRST WRITTEN WITH A FALSE BLOCKER AND IS CORRECTED IN PLACE.**
+It claimed "no CUDA binary can be built for that host" over three closed paths.
+**Two of those three were wrong**, and the error has one root: *the dgx HOST and
+the `rc` worker CONTAINER are different machines, and host findings were reported
+as container findings.* The correction is kept beside the claim rather than
+substituted for it, because the failure mode is the interesting part.
+
+1. **The host toolchain — this part was RIGHT and stands.** Measured on
+   `ssh dgx.casa`: `nvcc ABSENT   cmake ABSENT   g++ ABSENT   ninja ABSENT`.
+   Already filed as [#1019](https://github.com/mudler/vllm.cpp/issues/1019),
+   whose title says the dgx profile "mandates a CUDA toolkit and a CUTLASS path
+   that do not exist". **But the host is not where work runs**, so this never
+   blocked the gate the way this entry first said.
+2. **The `rc` worker container — the original claim here was FALSE.** It said
+   `gcc`, `g++`, `cc`, `cmake`, `ninja`, `make`, `python3`, `git` were all
+   absent, that `/usr/include/stdio.h` did not exist, and that DNS failed.
+   Measured inside `rc run`, the container is Ubuntu 24.04, runs as **uid 0**,
+   and carries `gcc`, `g++`, `cmake`, `ninja`, `make`, `python3`, `pip3`, `git`
+   and `apt`; `nvidia-smi` reports the GB10; DNS resolves and
+   `developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/arm64/` answers
+   `HTTP/2 200`. **Only `nvcc` is genuinely absent**, and apt's own
+   `nvidia-cuda-toolkit` is 12.0.140, too old for sm_121a (GB10 needs 12.8+/13.x)
+   — so the toolkit is installed from the NVIDIA repo, which is a step, not a
+   wall.
+3. **`docker` and `sudo` are NOT NEEDED AT ALL, and asking for them was the
+   error compounding.** You are root in the container; the recorded
+   `sudo -n docker run …` recipe in `$HOME/a2r/gb10_build.sh` belongs to the
+   pre-`rc` era.
+
+**The lesson, stated plainly for the next reader:** a probe that runs somewhere
+other than where the work will run answers a question nobody asked. Re-derive
+inside `rc run`, and do not let a host `ssh` result stand in for it.
+
+**What WAS verified on the gate host**, all of it cheap and under short leases:
+
+- **The checkpoint is present under BOTH paths, and `.env` is CORRECT.** An
+  earlier revision of this entry claimed the opposite — that only
+  `/home/mudler/ckpt/...` held it and that "there are no cifs/nfs mounts at
+  all". `findmnt` on the host says otherwise:
+
+  ```
+  /usr/local/nas_share //192.168.68.102/Data cifs rw,relatime,vers=3.1.1,
+    cache=strict,username=anonymous,uid=0,forceuid,gid=10001,forcegid,
+    addr=192.168.68.102,file_mode=0664,dir_mode=0775,iocharset=utf8,soft,nounix
+  ```
+
+  and the checkpoint resolves at
+  `/usr/local/nas_share/checkpoints/nemotron-3.5-lightning-30b-nvfp4`,
+  `/usr/local/nas_share/checkpoints/nemotron-3.5-lightning-30b-gguf` and
+  `/home/mudler/ckpt/nemotron-3.5-lightning-30b-nvfp4`. So
+  `CHECKPOINT_ROOT=/usr/local/nas_share/checkpoints` in `.env` is right and must
+  not be "fixed".
+- **Whether the container can SEE that path is still OPEN.** No probe has
+  answered it: the one queued to do so was killed to free the queue. Nothing is
+  claimed here either way, and step 6 of the A3 job is what will answer it.
+- **21 583 809 748 bytes**, 52 safetensors shards.
+- The revision is **verified, not copied**:
+  `model-00001-of-00052.safetensors` hashes to
+  `672c8bda10fdec0256e0819e112d2aa3a936cc3e5d311a05fd3ff773ca9a44b9`, which is
+  what that file's own HF sidecar records for commit
+  `29f2d1746d8f41e316523194b19018707749b1b1`, the golden's revision.
+
+**The driver was built and its instrument proven armed, off the gate host.**
+`examples/nemotron_h_gen` compiles and links against the real `vllm::shared` in
+a CPU build (`BUILD_RC=0`, 0 compile errors). Its guards were then exercised
+against a **real engine** on `opt-125m-bf16-st` rather than argued for, because
+the driver is model-agnostic and the assertions are what a NemotronH run will
+depend on:
+
+| arm | result |
+|---|---|
+| golden width 8, all 8 match | `STRICT PASS`, exit **0** |
+| golden width 8, 0 of 8 match | `DIVERGENCE`, exit **1** |
+| `--steps 4` against golden width 8, **4/4 matched** | `SHORT`, exit **4** |
+| 5 malformed goldens (empty array, empty prompt, empty tokens, no array, truncated) | parse refusal, exit **2** each |
+
+The third row is the one worth keeping: it matched **every token it looked at**
+and still refused, because it had looked at half the golden. A driver without
+that check would have reported `4/4` as a pass.
+
+The committed golden's geometry, read by the driver itself (`--golden-info`):
+**3 entries, prompt widths 5 / 8 / 13, golden width 32 each**, so a full A3 run
+compares **96 tokens**. Any "compared" number below 96 is a short run and exits 4.
+
+**Also blocked, and by design rather than by a host:** the governing spec §5.2
+arm 2 (the three prompts submitted concurrently and interleaved). G-SAFE refuses
+`input.num_reqs > 1` and A2-B owns that clause, so only arm 1 (multi-step,
+single request) is reachable even once a build host exists.
+
 ## KIMI-BF16-STREAM — bf16 residual stream end-to-end REFUTED (122→4/128, KDA repeat-loop destabilization, no speed win); STRICT is NOT reachable by residual-precision (§14-§20 all closed); 122/128 @ 18.9 tok/s (0.90× vLLM) is the coherent best; SERVER runner fold scoped (runner aborts on Kimi's KV today) (2026-08-07, `row/KIMI-BF16-STREAM-CLOSE`, base `origin/main` `2f029a10`, GB10 sm_121a, PR #118)
 
 The #113 follow-on tested the §19-named residual #1 — the bf16 residual stream END-TO-END, framed as

diff --git a/.agents/specs/nemotron-h-a2p-paged-forward.md b/.agents/specs/nemotron-h-a2p-paged-forward.md
@@ -865,12 +865,43 @@ nobody routes this architecture through a block that ropes.
 
 - **The §5.4 A3 end-to-end token gate**, and the §5.7 sm_121a leg with it. Owned
   by this row, tracked on [#810](https://github.com/mudler/vllm.cpp/issues/810).
-  Nothing about the released checkpoint is claimed until it runs.
+  Nothing about the released checkpoint is claimed until it runs. **The recorded
+  PENDING CAUSE IS NO LONGER TRUE and was re-measured rather than inherited**
+  (2026-08-17): §10 records contention — `dgx.casa` at loadavg 211 with 3 of
+  119 GB — and the box now answers at **loadavg 0.36 with 115 of 119 GB
+  available and the GPU at 0%**, with the checkpoint present and its revision
+  verified against its own LFS record. So the contention cause is dead; this is
+  the #775 shape the governing spec §5.5 warns about — a pending cause outliving
+  its own truth — and it is corrected rather than re-quoted.
+
+  **What replaced it was ALSO wrong for one revision, and that matters more.**
+  This bullet briefly claimed "there is no CUDA toolchain to build a gate binary
+  with", citing an `rc` worker container with no compiler, no libc headers and no
+  egress. **That was a host measurement reported as a container measurement.**
+  Inside `rc run` the container is Ubuntu 24.04 running as uid 0, with `gcc`,
+  `g++`, `cmake`, `ninja`, `make`, `python3`, `git` and `apt` all present, the
+  GB10 visible to `nvidia-smi`, and working DNS. **Only `nvcc` is absent**, and
+  apt's `nvidia-cuda-toolkit` 12.0.140 is too old for sm_121a, so CUDA 13.x is
+  installed from `developer.download.nvidia.com/…/ubuntu2404/arm64` — a step, not
+  a wall. No `docker` and no `sudo` are involved. The host toolchain finding
+  ([#1019](https://github.com/mudler/vllm.cpp/issues/1019)) is real but does not
+  gate anything, because the host is not where work runs.
+
+  **What is actually outstanding** is narrower: `nvcc` must be installed into the
+  build container, and whether that container can see
+  `$CHECKPOINT_ROOT` is an OPEN question no probe has yet answered.
 - **`examples/nemotron_h_gen`** (§3.6) and the `docs/USAGE.md` weights block that
-  rides with it. Not written: the ABI surface it would exercise
-  (`vllm_engine_load` + `vllm_complete_tokens`) is the same surface the A3 gate
-  drives, so writing the example before that gate can run would ship a client
-  for a path nobody has watched produce a token. Owned by this row, tracked on
+  rides with it — **both LANDED 2026-08-17** by `MODEL-NEMOTRON-H-ABI-A3-E2E`.
+  The reason recorded here for deferring them — that shipping a client for a
+  path nobody has watched produce a token is premature — was overtaken by the
+  measurement above: the gate is blocked on a host toolchain rather than on
+  anything about the code, and a driver that does not exist cannot be run the
+  moment that host is repaired. The example therefore lands with its counting
+  guards proven ARMED against a real engine on a small local checkpoint — a
+  full-width match exits 0, a divergence exits 1, a row that matched every token
+  it looked at but looked at HALF exits 4, and five malformed-golden shapes each
+  exit 2 — and with **no claim whatsoever** about what the released 30B
+  checkpoint emits. Owned by this row, tracked on
   [#810](https://github.com/mudler/vllm.cpp/issues/810).
 - **The device `lm_head`** stays A2-Q2b's, which is why
   `scripts/runner-routing-allowlist.txt` is narrowed rather than removed and why

diff --git a/.agents/specs/nemotron-h-abi-e2e.md b/.agents/specs/nemotron-h-abi-e2e.md
@@ -1104,24 +1104,78 @@ lifecycle write.
 
 ## 9. Now
 
-**State at this commit:** spec only. No product code, no lifecycle change. Per
-§1.4 the implementation is a **separate** pull request by a **different** agent;
-this one carries the spec.
-
-A1 is claimable now against `main` + `bc570da0d`, with PR #868 as its base
-(§7 R2). A2 is blocked on #496 W2, and that block must be re-verified rather than
-inherited from this sentence.
+**State at 2026-08-17.** A1 (`c1d02bfbe`), A2-R (`598226e96`), A2-Q2a
+(`6abc769c6`) and A2-P (`a6df72777`) have all landed. G-SAFE is narrowed to
+`input.num_reqs <= 1` and `ForwardNemotronHForCausalLM` selects
+`NemotronHPagedForward` whenever the runner supplies paged KV and recurrent
+state, which was verified in the tree rather than inherited from this sentence.
+
+**§6.1's driver EXISTS: `examples/nemotron_h_gen`.** One project include
+(`vllm.h`), linked `vllm::shared`, no `example-abi-allowlist.txt` row, modelled
+on `kimi_linear_gen` and not on the two allowlisted examples §6.1 warns about.
+It builds and links against the real shared library, and its counting guards are
+proven armed against a **real engine** on a small local checkpoint rather than
+argued for: a full-width match exits 0, a divergence exits 1, a row that matched
+every token it looked at while looking at HALF the golden's width exits 4, and
+five malformed-golden shapes each exit 2. That last one is the guard this
+section's own §5.2 is really about — a comparison over too few elements reports
+a perfect score, and here it cannot.
+
+**§5.2's A3 token gate has NOT RUN. Its recorded cause was wrong twice, and both
+corrections are kept here because the second one is a trap this section can save
+the next reader from.**
+
+*First cause, dead:* contention. Re-measured, `dgx.casa` is idle (loadavg 0.36,
+115 of 119 GB available, GPU 0%), the checkpoint is present, and its first shard
+hashes to revision `29f2d174`'s own LFS record.
+
+*Second cause, also dead:* "nothing can build a gate binary". **That was a HOST
+measurement reported as a CONTAINER measurement, and the two are different
+machines for this purpose.** The host genuinely has no `nvcc`/`cmake` since the
+14 Aug reimage ([#1019](https://github.com/mudler/vllm.cpp/issues/1019)) — but
+the host is not where work runs. Inside `rc run` the worker container is Ubuntu
+24.04 as **uid 0**, carrying `gcc`, `g++`, `cmake`, `ninja`, `make`, `python3`,
+`git`, `apt`, with the GB10 visible and DNS working. **Only `nvcc` is missing**,
+and apt's `nvidia-cuda-toolkit` 12.0.140 is too old for sm_121a, so CUDA 13.x
+comes from `developer.download.nvidia.com/…/ubuntu2404/arm64`. Neither `docker`
+nor `sudo` is involved.
+
+> **Rule this cost two cycles to learn: re-derive every environment fact INSIDE
+> `rc run`.** A probe that runs somewhere other than where the work will run
+> answers a question nobody asked, and it fails toward a confident verdict about
+> the code's environment rather than toward an obvious error.
+
+*What is genuinely outstanding:* install `nvcc` in the build container, and
+whether that container can see `$CHECKPOINT_ROOT`, which is **OPEN** — no probe
+has answered it, and none is claimed. `docs/BENCHMARKS.md` records the gate as
+**pending a named resource**, never as a pass.
+
+**§5.2 arm 2 (multi-request) is additionally blocked by design, not by a host.**
+G-SAFE refuses `input.num_reqs > 1` and A2-B owns that clause, so the three
+prompts cannot yet be submitted concurrently and interleaved. Arm 1 (multi-step,
+single request, all 32 tokens) is what the driver is built for.
+
+**§6.2's allowlist entry STAYS, and that was decided on evidence.**
+`nemotron_h.cpp:1031-1034` still refuses the NVFP4 `lm_head` on a non-CPU queue,
+so the forward's last step is a host projection and it returns `HostLogits`.
+Deleting `scripts/runner-routing-allowlist.txt:26` was tried in a scratch copy:
+`check-runner-routing-consistency.py` goes from `OK` to `ERROR` naming
+`ForwardNemotronHForCausalLM returns HostLogits`, exit 1, tree restored
+byte-for-byte. A2-Q2b removes the entry; widening the allowlist to satisfy the
+checker is the defect the checker exists to stop.
 
 **Three things to read before the first edit**, because each has already cost
 somebody a cycle: §5.5, so the six `#873` gates are subtracted rather than
 chased; §5.4, so the token gate is planned for `dgx.casa` and Thor and not for
 the local box, which cannot see anything device-side; and §7 R2, so #775 is
 consumed rather than re-fixed.
 
-**Next action:** a fresh implementer claims A1 from §1, captures the §3.1 red
-first, and lands A1 with the G-SAFE interlock opened through
-`vllm::ModelAs<NemotronHLoadedModel>`. A fresh reviewer — never the implementer —
-runs the §3.4 mutations and reports M3 as a pair.
+**Next action:** repair a build host (#1019), then run
+`nemotron-h-gen --model <staged dir> --golden
+tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json` with
+`VT_NEMOTRON35_SNAPSHOT` unset, and record the resolved directory. Expect 96
+tokens compared over 3 prompts of width 32; a "compared" number below 96 is a
+short run and the driver exits 4 rather than reporting it as a match.
 
 ## 10. Outcome
 

diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
@@ -10,7 +10,7 @@
 | **Binary release (ACTIVE; Windows pre-alpha pending)** | v0.0.2 shipped eight primary archive/checksum/provenance triplets + two indexes (26 assets) from source SHA `7020de93652ca920424a10ac5255b34810dd2f24`, run `31466516224` | Windows W14-W16 implemented. **PENDING:** native hosted gates, merged-SHA ten-tuple dry run, matching-hardware evidence, v0.0.3-pre.1 publication, 32-asset audit | W12 optional/non-primary |
 | **Container images (ACTIVE; arm64 cuda verified on GB10 + Orin 2026-08-11)** | `ENG-RELEASE-CONTAINERS` ([spec](../.agents/specs/container-images.md)) | cpu amd64 783 MB; cuda arm64 **1.71 GB**. GB10 `sm_121a`: `/health`+`/version`+SIGTERM on `--gpus all`. Orin `sm_87` (Tegra): Qwen3-0.6B **generates**, GPU **GR3D 95-97%** | n/a |
 | **Developer/row protocol** | Contribution entry point; `ENG-NOW-DERIVED` #374 @`dbd0d51c` | Entry-point gates retained. #374 W1-W5 DONE; benchmark/runtime/parity `VOID`; row specs now carry `## Now` | n/a |
-| **NemotronH paged forward** (`MODEL-NEMOTRON-H-ABI-A2P`, [#810](https://github.com/mudler/vllm.cpp/issues/810)) | **No number on any axis, by the unit's own rule** ([spec](../.agents/specs/nemotron-h-a2p-paged-forward.md) §5) | **A3 token gate PENDING on CONTENTION:** `dgx.casa` at loadavg 211 with 3 of 119 GB available | CPU gate 12/12, 9/9 mutations RED |
+| **NemotronH paged forward** (`MODEL-NEMOTRON-H-ABI-A2P`, [#810](https://github.com/mudler/vllm.cpp/issues/810)) | **No number on any axis, by the unit's own rule** ([spec](../.agents/specs/nemotron-h-a2p-paged-forward.md) §5) | **A3 gate PENDING: `nvcc` + checkpoint visibility in the rc container.** Two earlier causes here (contention, then "cannot build") were measured FALSE, see [benchmark-record](../.agents/benchmark-record.md) | CPU gate 12/12, 9/9 mutations RED. A3 driver `examples/nemotron_h_gen` exists; guards armed on a real engine (pass 0, divergence 1, short 4) |
 | **LoRA runtime W2** (`LORA-RUNTIME`, #278) | **No number owed:** correctness-only; a grid PENDS the W7 model gate |
 | **ARCH audit: ABI is text-only** | 4 capabilities (H3 video, Laguna, Kimi-Linear, DeepSeek-V4) reachable only from `examples/`, none registry-backed. No gate asks whether a CONSUMER can reach a capability. Documentation only |
 | **DSR fix: server TU profiler guards (2026-08-09)** | **No number owed:** comments only. #189 moved the server body into the shared layer with its 5 `VT_BENCH_PROFILE_CONTROL` guards, taking DSR 32 -> 37; they are `DSR-ALLOW`'d per site, baseline unchanged at 32 |