From ad1891fdf70d1ca49e8edf301004e18211e0d0b4 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 12:18:32 +0000 Subject: [PATCH 1/2] feat(#810 A3): NemotronH gets its ABI driver, and the reason its gate has not run gets corrected MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The A2-P spec's `## Owed` lists three things. Two land here. The third did not run, and the reason on record for why it has not run turned out to be false. `examples/nemotron_h_gen` is the driver the governing spec §6.1 describes: modelled on `kimi_linear_gen`, one project include (`vllm.h`), linked against `vllm::shared`, and absent from `example-abi-allowlist.txt` because it reaches no internal header. `deepseek_v4_gen` and `laguna_gen` were not copied; both drive a bespoke forward through internal headers and are the transition state that allowlist exists to retire. The golden is JSON rather than Kimi's `.npy` plus raw-i32 pair, so the driver carries a scanner for that one committed shape. It refuses a shape it does not recognise rather than returning an empty vector, because an empty vector downstream is a comparison over zero elements and a comparison over zero elements reports a perfect score. That is also why a pass needs three things at once, checked in the driver instead of left to whoever reads the log: something was compared, everything compared matched, and every row was compared to the golden's full width. Those guards were exercised against a real engine rather than argued for. On a small local checkpoint the driver exits 0 on a full-width match, 1 on a divergence, 2 on each of five malformed goldens, and 4 when `--steps 4` is run against a width-8 golden. That last one matched every token it looked at and still refused, which is the case a driver without the check would have reported as 4/4. `docs/USAGE.md` gains the weights block AGENTS.md wants in the change that makes a capability reachable: repo, revision, staged path, byte total, the sha256 of the first shard, and every arm named including the four refused ones. The revision was verified rather than copied. Its stale row claiming the forward refuses the first step is corrected, because a row that reads as broken after A2-P fixed it is as wrong as one that reads as working. The larger correction is the pending cause. A2-P recorded the gate as pending on CONTENTION, at loadavg 211 with 3 of 119 GB free. Re-measured under a lease that box is idle: loadavg 0.36, 115 of 119 GB, GPU 0%, checkpoint present and revision-verified against its own LFS record. The gate is blocked on something else, that nothing on that host can build a CUDA binary, and the three ways to try are enumerated in the benchmark record with the host facts separated from the agent-access one. #775 is the precedent: a cause left standing long enough stops being read, and by then it has changed underneath. `scripts/runner-routing-allowlist.txt:26` stays, and that was decided by mutation rather than by reading. Deleting it takes the routing checker from OK to ERROR naming `ForwardNemotronHForCausalLM returns HostLogits`; the tree was restored byte-for-byte. A2-Q2b removes it, and the allowlist was not widened. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/benchmark-record.md | 73 ++++ .agents/specs/nemotron-h-a2p-paged-forward.md | 30 +- .agents/specs/nemotron-h-abi-e2e.md | 58 ++- docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 2 +- docs/USAGE.md | 76 +++- examples/CMakeLists.txt | 11 + examples/nemotron_h_gen/main.cpp | 366 ++++++++++++++++++ 8 files changed, 599 insertions(+), 19 deletions(-) create mode 100644 examples/nemotron_h_gen/main.cpp diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 342b283d9..41a301245 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -19,6 +19,79 @@ from relative link targets repointed for this file's location. # Benchmarks +## MODEL-NEMOTRON-H-ABI-A3-E2E — the A3 token gate did NOT run, and the cause on record was NOT the cause (2026-08-17, `row/MODEL-NEMOTRON-H-ABI-A3-E2E`, base `origin/main` `a6df72777`, #810) + +**No number is recorded, on any axis. This entry exists so the pending cause is +the measured one rather than the inherited one.** + +`.agents/specs/nemotron-h-a2p-paged-forward.md` §10 recorded the A3 gate as +pending on **contention**: `dgx.casa` observed at loadavg 211 with 3 of 119 GB +available, which is a real reason a 20.1 GiB checkpoint cannot load. Re-measured +on 2026-08-17 under an `rc` lease, that box answers at **loadavg 0.36, 115 of +119 GB available, GPU utilisation 0%**, with nothing of ours running on it. The +recorded cause is no longer true, and this is the #775 shape the governing spec +§5.5 warns about: a pending reason that outlives its own truth and gets +subtracted by everyone who reads it afterwards. + +**What actually blocks the gate is that no CUDA binary can be built for that +host.** Three paths were checked, and all three are closed: + +1. **The host toolchain.** No `nvcc`, no `cmake`, no `g++` since the 14 Aug + reimage. Already filed as + [#1019](https://github.com/mudler/vllm.cpp/issues/1019), whose title says the + dgx profile "mandates a CUDA toolkit and a CUTLASS path that do not exist". +2. **The `rc` worker container**, which is what `rc run` schedules into. Probed + directly: `gcc`, `g++`, `cc`, `cmake`, `ninja`, `make`, `nvcc`, `python3`, + `git` and `docker` are all ABSENT, `/usr/include/stdio.h` does not exist, and + `getent hosts pypi.org` fails, so it cannot even fetch one. `/workspace` maps + to `/usr/local/nas_share/rc` on the host and carries no checkpoint. +3. **The recorded container build** (`$HOME/a2r/gb10_build.sh`: + `sudo -n docker run --runtime=nvidia nvidia/cuda:13.0.1-devel-ubuntu24.04`). + This is the only remaining path and it needs the host docker socket. `id` on + dgx reports `groups=1002(mudler),900(admin)`, **not `docker`**, so it + requires `sudo`, and `sudo docker` is refused by this agent's own permission + layer. That last clause is an agent-access fact, not a host fact, and is + reported as such rather than filed against the box. + +**What WAS verified on the gate host**, all of it cheap and under short leases: + +- The checkpoint is present at `/home/mudler/ckpt/nemotron-3.5-lightning-30b-nvfp4` + (**not** `$CHECKPOINT_ROOT=/usr/local/nas_share/checkpoints`, which does not + exist there; there are no cifs/nfs mounts at all). +- **21 583 809 748 bytes**, 52 safetensors shards. +- The revision is **verified, not copied**: + `model-00001-of-00052.safetensors` hashes to + `672c8bda10fdec0256e0819e112d2aa3a936cc3e5d311a05fd3ff773ca9a44b9`, which is + what that file's own HF sidecar records for commit + `29f2d1746d8f41e316523194b19018707749b1b1`, the golden's revision. + +**The driver was built and its instrument proven armed, off the gate host.** +`examples/nemotron_h_gen` compiles and links against the real `vllm::shared` in +a CPU build (`BUILD_RC=0`, 0 compile errors). Its guards were then exercised +against a **real engine** on `opt-125m-bf16-st` rather than argued for, because +the driver is model-agnostic and the assertions are what a NemotronH run will +depend on: + +| arm | result | +|---|---| +| golden width 8, all 8 match | `STRICT PASS`, exit **0** | +| golden width 8, 0 of 8 match | `DIVERGENCE`, exit **1** | +| `--steps 4` against golden width 8, **4/4 matched** | `SHORT`, exit **4** | +| 5 malformed goldens (empty array, empty prompt, empty tokens, no array, truncated) | parse refusal, exit **2** each | + +The third row is the one worth keeping: it matched **every token it looked at** +and still refused, because it had looked at half the golden. A driver without +that check would have reported `4/4` as a pass. + +The committed golden's geometry, read by the driver itself (`--golden-info`): +**3 entries, prompt widths 5 / 8 / 13, golden width 32 each**, so a full A3 run +compares **96 tokens**. Any "compared" number below 96 is a short run and exits 4. + +**Also blocked, and by design rather than by a host:** the governing spec §5.2 +arm 2 (the three prompts submitted concurrently and interleaved). G-SAFE refuses +`input.num_reqs > 1` and A2-B owns that clause, so only arm 1 (multi-step, +single request) is reachable even once a build host exists. + ## KIMI-BF16-STREAM — bf16 residual stream end-to-end REFUTED (122→4/128, KDA repeat-loop destabilization, no speed win); STRICT is NOT reachable by residual-precision (§14-§20 all closed); 122/128 @ 18.9 tok/s (0.90× vLLM) is the coherent best; SERVER runner fold scoped (runner aborts on Kimi's KV today) (2026-08-07, `row/KIMI-BF16-STREAM-CLOSE`, base `origin/main` `2f029a10`, GB10 sm_121a, PR #118) The #113 follow-on tested the §19-named residual #1 — the bf16 residual stream END-TO-END, framed as diff --git a/.agents/specs/nemotron-h-a2p-paged-forward.md b/.agents/specs/nemotron-h-a2p-paged-forward.md index 8cb19cc10..860d53e5b 100644 --- a/.agents/specs/nemotron-h-a2p-paged-forward.md +++ b/.agents/specs/nemotron-h-a2p-paged-forward.md @@ -865,12 +865,32 @@ nobody routes this architecture through a block that ropes. - **The §5.4 A3 end-to-end token gate**, and the §5.7 sm_121a leg with it. Owned by this row, tracked on [#810](https://github.com/mudler/vllm.cpp/issues/810). - Nothing about the released checkpoint is claimed until it runs. + Nothing about the released checkpoint is claimed until it runs. **The recorded + PENDING CAUSE IS NO LONGER TRUE and was re-measured rather than inherited** + (2026-08-17): §10 records contention — `dgx.casa` at loadavg 211 with 3 of + 119 GB — and the box now answers at **loadavg 0.36 with 115 of 119 GB + available and the GPU at 0%**, with the checkpoint present and its revision + verified against its own LFS record. The gate is blocked on something else + entirely: **there is no CUDA toolchain to build a gate binary with.** The host + carries no `nvcc` and no `cmake` since the 14 Aug reimage + ([#1019](https://github.com/mudler/vllm.cpp/issues/1019)), and the `rc` worker + container it schedules through carries no compiler, no libc headers and no + network egress, so the recorded recipe's `docker run nvidia/cuda:*-devel` on + the host is the only remaining path. This is the #775 shape the governing spec + §5.5 warns about — a pending cause outliving its own truth — so it is + corrected here rather than re-quoted. - **`examples/nemotron_h_gen`** (§3.6) and the `docs/USAGE.md` weights block that - rides with it. Not written: the ABI surface it would exercise - (`vllm_engine_load` + `vllm_complete_tokens`) is the same surface the A3 gate - drives, so writing the example before that gate can run would ship a client - for a path nobody has watched produce a token. Owned by this row, tracked on + rides with it — **both LANDED 2026-08-17** by `MODEL-NEMOTRON-H-ABI-A3-E2E`. + The reason recorded here for deferring them — that shipping a client for a + path nobody has watched produce a token is premature — was overtaken by the + measurement above: the gate is blocked on a host toolchain rather than on + anything about the code, and a driver that does not exist cannot be run the + moment that host is repaired. The example therefore lands with its counting + guards proven ARMED against a real engine on a small local checkpoint — a + full-width match exits 0, a divergence exits 1, a row that matched every token + it looked at but looked at HALF exits 4, and five malformed-golden shapes each + exit 2 — and with **no claim whatsoever** about what the released 30B + checkpoint emits. Owned by this row, tracked on [#810](https://github.com/mudler/vllm.cpp/issues/810). - **The device `lm_head`** stays A2-Q2b's, which is why `scripts/runner-routing-allowlist.txt` is narrowed rather than removed and why diff --git a/.agents/specs/nemotron-h-abi-e2e.md b/.agents/specs/nemotron-h-abi-e2e.md index ad01fd5bb..6d6b1252a 100644 --- a/.agents/specs/nemotron-h-abi-e2e.md +++ b/.agents/specs/nemotron-h-abi-e2e.md @@ -1104,13 +1104,47 @@ lifecycle write. ## 9. Now -**State at this commit:** spec only. No product code, no lifecycle change. Per -§1.4 the implementation is a **separate** pull request by a **different** agent; -this one carries the spec. - -A1 is claimable now against `main` + `bc570da0d`, with PR #868 as its base -(§7 R2). A2 is blocked on #496 W2, and that block must be re-verified rather than -inherited from this sentence. +**State at 2026-08-17.** A1 (`c1d02bfbe`), A2-R (`598226e96`), A2-Q2a +(`6abc769c6`) and A2-P (`a6df72777`) have all landed. G-SAFE is narrowed to +`input.num_reqs <= 1` and `ForwardNemotronHForCausalLM` selects +`NemotronHPagedForward` whenever the runner supplies paged KV and recurrent +state, which was verified in the tree rather than inherited from this sentence. + +**§6.1's driver EXISTS: `examples/nemotron_h_gen`.** One project include +(`vllm.h`), linked `vllm::shared`, no `example-abi-allowlist.txt` row, modelled +on `kimi_linear_gen` and not on the two allowlisted examples §6.1 warns about. +It builds and links against the real shared library, and its counting guards are +proven armed against a **real engine** on a small local checkpoint rather than +argued for: a full-width match exits 0, a divergence exits 1, a row that matched +every token it looked at while looking at HALF the golden's width exits 4, and +five malformed-golden shapes each exit 2. That last one is the guard this +section's own §5.2 is really about — a comparison over too few elements reports +a perfect score, and here it cannot. + +**§5.2's A3 token gate has NOT RUN, and its cause is not the recorded one.** +Re-measured rather than inherited: `dgx.casa` is idle (loadavg 0.36, 115 of +119 GB available, GPU 0%), the 20.1 GiB checkpoint is present, and its first +shard hashes to revision `29f2d174`'s own LFS record. The blocker is that +**nothing on that host can build a gate binary**: no `nvcc` and no `cmake` since +the 14 Aug reimage ([#1019](https://github.com/mudler/vllm.cpp/issues/1019)), +and the `rc` worker container it now schedules through has no compiler, no libc +headers and no egress. Recorded in `docs/BENCHMARKS.md` as **pending a named +resource**, never as a pass. + +**§5.2 arm 2 (multi-request) is additionally blocked by design, not by a host.** +G-SAFE refuses `input.num_reqs > 1` and A2-B owns that clause, so the three +prompts cannot yet be submitted concurrently and interleaved. Arm 1 (multi-step, +single request, all 32 tokens) is what the driver is built for and what is owed +the moment a build host exists. + +**§6.2's allowlist entry STAYS, and that was decided on evidence.** +`nemotron_h.cpp:1031-1034` still refuses the NVFP4 `lm_head` on a non-CPU queue, +so the forward's last step is a host projection and it returns `HostLogits`. +Deleting `scripts/runner-routing-allowlist.txt:26` was tried in a scratch copy: +`check-runner-routing-consistency.py` goes from `OK` to `ERROR` naming +`ForwardNemotronHForCausalLM returns HostLogits`, exit 1, tree restored +byte-for-byte. A2-Q2b removes the entry; widening the allowlist to satisfy the +checker is the defect the checker exists to stop. **Three things to read before the first edit**, because each has already cost somebody a cycle: §5.5, so the six `#873` gates are subtracted rather than @@ -1118,10 +1152,12 @@ chased; §5.4, so the token gate is planned for `dgx.casa` and Thor and not for the local box, which cannot see anything device-side; and §7 R2, so #775 is consumed rather than re-fixed. -**Next action:** a fresh implementer claims A1 from §1, captures the §3.1 red -first, and lands A1 with the G-SAFE interlock opened through -`vllm::ModelAs`. A fresh reviewer — never the implementer — -runs the §3.4 mutations and reports M3 as a pair. +**Next action:** repair a build host (#1019), then run +`nemotron-h-gen --model --golden +tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json` with +`VT_NEMOTRON35_SNAPSHOT` unset, and record the resolved directory. Expect 96 +tokens compared over 3 prompts of width 32; a "compared" number below 96 is a +short run and the driver exits 4 rather than reporting it as a match. ## 10. Outcome diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index ea50a632b..1e8c2550f 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -10,7 +10,7 @@ | **Binary release (ACTIVE; Windows pre-alpha pending)** | v0.0.2 shipped eight primary archive/checksum/provenance triplets + two indexes (26 assets) from source SHA `7020de93652ca920424a10ac5255b34810dd2f24`, run `31466516224` | Windows W14-W16 implemented. **PENDING:** native hosted gates, merged-SHA ten-tuple dry run, matching-hardware evidence, v0.0.3-pre.1 publication, 32-asset audit | W12 optional/non-primary | | **Container images (ACTIVE; arm64 cuda verified on GB10 + Orin 2026-08-11)** | `ENG-RELEASE-CONTAINERS` ([spec](../.agents/specs/container-images.md)) | cpu amd64 783 MB; cuda arm64 **1.71 GB**. GB10 `sm_121a`: `/health`+`/version`+SIGTERM on `--gpus all`. Orin `sm_87` (Tegra): Qwen3-0.6B **generates**, GPU **GR3D 95-97%** | n/a | | **Developer/row protocol** | Contribution entry point; `ENG-NOW-DERIVED` #374 @`dbd0d51c` | Entry-point gates retained. #374 W1-W5 DONE; benchmark/runtime/parity `VOID`; row specs now carry `## Now` | n/a | -| **NemotronH paged forward** (`MODEL-NEMOTRON-H-ABI-A2P`, [#810](https://github.com/mudler/vllm.cpp/issues/810)) | **No number on any axis, by the unit's own rule** ([spec](../.agents/specs/nemotron-h-a2p-paged-forward.md) §5) | **A3 token gate PENDING on CONTENTION:** `dgx.casa` at loadavg 211 with 3 of 119 GB available | CPU gate 12/12, 9/9 mutations RED | +| **NemotronH paged forward** (`MODEL-NEMOTRON-H-ABI-A2P`, [#810](https://github.com/mudler/vllm.cpp/issues/810)) | **No number on any axis, by the unit's own rule** ([spec](../.agents/specs/nemotron-h-a2p-paged-forward.md) §5) | **A3 token gate PENDING on a BUILD HOST, not contention** (re-measured 2026-08-17: dgx idle, checkpoint revision-verified, no nvcc/cmake, [#1019](https://github.com/mudler/vllm.cpp/issues/1019)) | CPU gate 12/12, 9/9 mutations RED. A3 driver `examples/nemotron_h_gen` exists; guards armed on a real engine (pass 0, divergence 1, short 4) | | **LoRA runtime W2** (`LORA-RUNTIME`, #278) | **No number owed:** correctness-only; a grid PENDS the W7 model gate | | **ARCH audit: ABI is text-only** | 4 capabilities (H3 video, Laguna, Kimi-Linear, DeepSeek-V4) reachable only from `examples/`, none registry-backed. No gate asks whether a CONSUMER can reach a capability. Documentation only | | **DSR fix: server TU profiler guards (2026-08-09)** | **No number owed:** comments only. #189 moved the server body into the shared layer with its 5 `VT_BENCH_PROFILE_CONTROL` guards, taking DSR 32 -> 37; they are `DSR-ALLOW`'d per site, baseline unchanged at 32 | diff --git a/docs/STATUS.md b/docs/STATUS.md index 451c34eae..db7f770d7 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -131,7 +131,7 @@ token-for-token correctness against the pinned oracle. | GLM-4 dense (sandwich norms, partial rope) | Correctness-complete, speed-pending | Token-exact 16/16 (GLM-4-9B-0414); first GLM-family model; partial interleaved RoPE + Gemma2 sandwich norms + biased qkv | | GLM-4.7-Flash (MLA + GLM MoE) | Correctness-complete, speed-pending | Token-exact 8/8 (GLM-4.7-Flash, 31.2B); reuses the DeepSeek-V2 MLA stack; first e2e coverage of the q_lora query branch + noaux_tc sigmoid router with routed-scaling | | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **RUNNER FOLD LANDS (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; golden 122/128 (near-tie profile); FA2 MLA default-ON; `vllm_complete_tokens` (ABI v13).** Grouped-router top-k block-parallel (byte-identical); no binding speed number: ckpt is tiktoken-only, so no warm-server harness. STRICT stays CLOSED. Server 19.0 tok/s wall (~0.90× vLLM floor) = speed open | paged suite 8/8·206; SACRED post-fold 35B 315/315 + 27B 235/235; thin ABI client (ratchet 8) | -| Nemotron-3.5-Lightning-30B-A3B (Mamba2 + GQA + relu2 MoE) | **Paged forward lands (#810 A2-P); e2e token gate PENDING on gate-host contention** | K/V and the conv + SSM rows now live in the runner's pages, so decode step 2 keeps state; G-SAFE narrows to `num_reqs <= 1`. CPU gate 12/12, 9/9 mutations RED | +| Nemotron-3.5-Lightning-30B-A3B (Mamba2 + GQA + relu2 MoE) | **Paged forward + ABI driver land (#810 A2-P, A3); e2e token gate PENDING on a BUILD HOST, not contention** | `examples/nemotron_h_gen` reaches it through `include/vllm.h` alone; G-SAFE narrows to `num_reqs <= 1`. Gate cause re-measured: box idle, no nvcc/cmake ([#1019](https://github.com/mudler/vllm.cpp/issues/1019)) | | Gemma-3 dense (GeGLU, dual rope, sandwich norms) | Correctness-complete, speed-pending | STRICT token-exact 48/48 greedy (gemma-3-1b-it); first Gemma-family model; GeGLU (gelu_pytorch_tanh) + dual per-layer RoPE theta + Gemma-RMSNorm sandwich norms + sqrt(hidden) embed-scale + query_pre_attn_scalar scaling | | Gemma-2 dense (attn + final logit soft-cap) | Correctness-complete, speed-pending | Near-tie-band 48/48 (gemma-2-2b-it): 44/48 strict on vLLM's greedy + 4/48 at 0.0-nat ties in vLLM's own logits; proves the attention + final logit soft-cap primitives (attn_logit_softcapping 50 + final 30); the inverse of Gemma-3 (both soft-caps, no QK-norm) | | Gemma-1 dense (the original Gemma) | Correctness-complete, speed-pending | STRICT token-exact 48/48 greedy (gemma-2b); two fused norms/layer, head_dim scale, GeGLU + sqrt(hidden) embed-scale, tied lm_head; no soft-cap/QK-norm/sliding. **D1 (2026-07-31): the whole Gemma family (1/2/3/4) folded to the default-ON bf16 merged-QKV descriptor (`MergedQkvEnabled`); re-gated Gemma-2 SACRED 48/48 (global+sliding) + Gemma-4 STRICT 32/32 — its existing gate held** | diff --git a/docs/USAGE.md b/docs/USAGE.md index af7294a03..983875773 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -535,7 +535,7 @@ tokens quietly. | Architecture | Why it refuses | |---|---| | `KimiK3ForConditionalGeneration` | Needs ~1.56 TB (MXFP4); no host here can run it | -| `NemotronHForCausalLM` | The hybrid forward is ported (#517 W4) and the weight loader materializes the real checkpoint, but that forward is a HOST reference: it recomputes K/V over the whole sequence every step, carries no recurrent state between steps and treats a batch as one causal sequence. Engine construction now SUCCEEDS — the KV allocation reads the model's own recurrent spec (#810) — and the first step then refuses by name, naming the paged/batched decode path as the missing piece rather than returning plausible wrong tokens. **That refusal is UNCHANGED by A2-R (#810)**: A2-R adds a partial device arm (embedding lookup, the 52 layer norms + `norm_f`, and the 6 GQA attention blocks; Mamba2, MoE and `lm_head` stay on the host), but it is non-paged and single-request, so it creates none of the capability the refusal guards and is not reachable through `include/vllm.h`. It is exercised only by `test_nemotron_h_forward`, and it records no throughput number. Safetensors resolve and parse; a GGUF file is refused by name, since no GGUF arm exists for it | +| `NemotronHForCausalLM` | **Only BATCHED decode still refuses.** A2-P (#810) narrowed this: `ForwardNemotronHForCausalLM` now selects the paged forward whenever the runner supplies paged KV and recurrent state, so K/V go into the runner's pages and the conv/SSM rows are carried across steps, and `examples/nemotron_h_gen` reaches all of it through `include/vllm.h` alone. What is left is `num_reqs > 1`, refused by name because one request's pages and one request's recurrent state are carried per step and a multi-request step would be decoded as ONE concatenated causal sequence — plausible wrong tokens rather than a failure. Owed to A2-B. **The end-to-end token gate against the pinned oracle has NOT run**, so no claim is made here about what this checkpoint emits; `docs/BENCHMARKS.md` records that as pending rather than as silence. `lm_head` and the FP8 Mamba2 projections still compute on the host, and a GGUF file is refused by name since no GGUF arm exists for it. See *Nemotron-3.5-Lightning-30B: the exact weights, and which arms run* below | This is a deliberate state, not a bug: registering the architecture is what lets the config parse and weight-name mapping be tested before the forward exists. @@ -2269,6 +2269,80 @@ Set `VLLM_MUSE_GGUF=` (or `VLLM_MUSE_GGUF_LOAD=` for the full materialization) to run `test_muse_glimmer_gguf` against a real checkpoint; without them the gate runs off committed header-only manifests. +## Nemotron-3.5-Lightning-30B: the exact weights, and which arms run + +`NemotronHForCausalLM` is a hybrid: 6 GQA attention layers over a paged KV cache +and 23 Mamba2 layers over a recurrent conv/SSM state, with MoE blocks between +them. `examples/nemotron_h_gen` (`nemotron-h-gen`) drives it through the public +C ABI and nothing else — `vllm_engine_load` + `vllm_complete_tokens` — against +the committed oracle golden: + +```sh +nemotron-h-gen --model "$CHECKPOINT_ROOT/nemotron-3.5-lightning-30b-nvfp4" \ + --golden tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json +``` + +`--golden-info` parses the golden and prints its geometry without loading a +model, which is how you check the battery's shape before spending a 20.1 GiB +load. `--load-only` stops after `vllm_engine_load`. + +### The checkpoint + +| field | value | +|---|---| +| repo | [nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4) — first party | +| revision | `29f2d1746d8f41e316523194b19018707749b1b1` | +| staged as | `$CHECKPOINT_ROOT/nemotron-3.5-lightning-30b-nvfp4` (a `hf download --local-dir` tree) | +| on-disk total | 21 583 809 748 bytes (20.1 GiB) | +| weights | `model-000{01..52}-of-00052.safetensors` + `model.safetensors.index.json` | +| quantization | `config.json` (1 337 760 B) + `hf_quant_config.json` (928 085 B), the `modelopt_mixed` layout | +| tokenizer | `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`, `chat_template.jinja` | +| sha256 (first shard) | `672c8bda10fdec0256e0819e112d2aa3a936cc3e5d311a05fd3ff773ca9a44b9` for `model-00001-of-00052.safetensors` (743 427 168 B) | + +**A repo id alone is not a pin** — checkpoints get re-quantized in place under an +unchanged name — so the revision is recorded, and it was verified rather than +copied: the first shard on the gate host hashes to the value above, which is +that revision's own LFS record for the file +(`.cache/huggingface/download/model-00001-of-00052.safetensors.metadata`, whose +sidecar names commit `29f2d174`). `tests/parity/hf_snapshot.h` resolves the +directory and refuses a tree staged at any other revision, so +`VT_NEMOTRON35_SNAPSHOT` is left UNSET for a gate run: setting it takes the +explicit-directory escape, which is deliberately not revision-checked. + + hf download nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \ + --revision 29f2d1746d8f41e316523194b19018707749b1b1 \ + --local-dir "$CHECKPOINT_ROOT/nemotron-3.5-lightning-30b-nvfp4" + +### The arms, and what each one costs you today + +The loader materializes all 18 487 tensors in the memory format the checkpoint +ships them in, so nothing is silently widened at load. What differs between arms +is **where the arithmetic happens**, and that is not something a token +comparison can see, so it is written down here instead. + +| arm | state | +|---|---| +| bf16 layers, norms, the 6 GQA attention blocks | **device** | +| MoE experts, NVFP4 W4A16 g16 | **device** (Marlin arena) | +| FP8 W8A8 static Mamba2 input projections | **host** — the device arm is owed, [#940](https://github.com/mudler/vllm.cpp/issues/940) | +| `lm_head`, NVFP4 W4A16 g16 | **host** — it refuses a non-CPU queue by name, so the forward's last step is a host projection and the model still returns host logits. Owed to A2-Q2b, [#810](https://github.com/mudler/vllm.cpp/issues/810) | + +And the arms that are **refused by name** rather than substituted: + +| arm | the refusal | +|---|---| +| GGUF k-quants / i-quants | not ported. A GGUF path is refused at load naming `.agents/specs/nemotron-h-model.md` §5b W7, because silently dequantizing to a supported path is exactly what a token gate cannot see | +| the MTP draft head | deferred by name at load (W5) | +| batched decode (`num_reqs > 1`) | refused at the forward. One request's KV pages and one request's recurrent state are carried per step; a multi-request step would be decoded as ONE concatenated causal sequence and would return plausible wrong tokens instead of failing. Owed to A2-B, [#810](https://github.com/mudler/vllm.cpp/issues/810) | + +### What has NOT been measured + +**No token gate result exists for this checkpoint yet.** The example above is the +vehicle for it and the golden is committed, but the run itself is pending; the +current state is recorded in `docs/BENCHMARKS.md` rather than left as silence, +and nothing about the released checkpoint's output is claimed here until it is +green. + ## MiniMax-H3: video + audio generation ### The exact weights (so a render is reproducible) diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt index 6379905b8..6ac37b929 100644 --- a/examples/CMakeLists.txt +++ b/examples/CMakeLists.txt @@ -30,6 +30,17 @@ add_executable(kimi-linear-gen kimi_linear_gen/main.cpp) target_link_libraries(kimi-linear-gen PRIVATE vllm::shared) vllm_cpp_set_warnings(kimi-linear-gen) +# nemotron-h-gen: the Nemotron-3.5-Lightning-30B (NemotronHForCausalLM) A3 +# end-to-end token battery against the PINNED vLLM ORACLE golden, and a THIN +# CLIENT of the public C ABI in exactly the kimi-linear-gen shape above: +# vllm_engine_load + the v13 vllm_complete_tokens over the golden's +# pre-tokenized prompts, linking the packaged shared library so it reaches no +# internal header and needs no example-abi-allowlist row. #810, +# .agents/specs/nemotron-h-abi-e2e.md §6.1. +add_executable(nemotron-h-gen nemotron_h_gen/main.cpp) +target_link_libraries(nemotron-h-gen PRIVATE vllm::shared) +vllm_cpp_set_warnings(nemotron-h-gen) + # vllm-bench (M2.1): the throughput/latency benchmark harness — the gate-#1 # measurement tool. Links the full C++ engine (vllm::vllm) so it can build both a # synthetic CPU engine (no --model) and a real loaded engine (--model on dgx). diff --git a/examples/nemotron_h_gen/main.cpp b/examples/nemotron_h_gen/main.cpp new file mode 100644 index 000000000..605f29bca --- /dev/null +++ b/examples/nemotron_h_gen/main.cpp @@ -0,0 +1,366 @@ +// nemotron-h-gen — THIN PUBLIC-ABI CLIENT (ONE SURFACE / ARCH-ONE-SURFACE). +// +// The Nemotron-3.5-Lightning-30B (`NemotronHForCausalLM`) greedy token battery +// against the PINNED vLLM ORACLE golden, driven ENTIRELY through the flat C ABI +// (include/vllm.h): `vllm_engine_load` builds the full engine — the A2 weight +// loader's 18487 tensors in the format the checkpoint ships them in, plus the +// shared paged runner whose caches select `NemotronHPagedForward` — and +// `vllm_complete_tokens` (ABI v13) generates from the golden's pre-tokenized +// prompts. This file includes "vllm.h" and NOTHING else from the project, links +// `vllm::shared`, and is deliberately NOT on `scripts/example-abi-allowlist.txt`: +// per .agents/specs/nemotron-h-abi-e2e.md §6.1 it is modelled on +// examples/kimi_linear_gen, never on deepseek_v4_gen or laguna_gen, both of +// which drive a bespoke forward through internal headers and are the transition +// state that allowlist exists to retire. +// +// nemotron-h-gen --model --golden +// [--steps N] [--prompts M] [--max-model-len N] [--load-only] +// +// The golden is the A3 gate's operand: +// tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json, captured from +// the pinned oracle (`vllm 0.23.1rc1.dev1511+g555967922`, the setuptools_scm +// spelling of pin 5559679229bc) at `temperature 0.0, max_tokens 32` over three +// prompts. It carries `prompt_token_ids` per prompt, so this driver needs no +// tokenizer agreement established first, and `token_ids`, the 32 tokens each +// prompt must reproduce. +// +// ── WHY THIS PRINTS COUNTS, LOUDLY ────────────────────────────────────────── +// A comparison over zero elements reports a perfect score. This driver +// therefore refuses rather than passes when it compared nothing: it asserts a +// NON-ZERO prompt count and, per prompt, that the number of tokens actually +// compared equals the golden's own width. `--steps` below that width is a +// short run and is reported as such, never as a match. +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm.h" + +namespace { + +// ── A deliberately small reader for THIS golden's shape ───────────────────── +// The golden is a fixed, committed document, not arbitrary JSON: an object with +// a "golden" array whose entries carry "prompt", "prompt_token_ids" and +// "token_ids". A full JSON parser in an example would be a second +// implementation to maintain; a scanner that finds a named key and reads the +// integer array or string that follows it is enough, and it FAILS LOUDLY on a +// shape it does not recognise rather than returning an empty vector that would +// read downstream as "nothing to compare". + +std::string ReadWholeFile(const std::string& path) { + std::ifstream f(path, std::ios::binary); + if (!f) throw std::runtime_error("cannot open " + path); + std::ostringstream ss; + ss << f.rdbuf(); + return ss.str(); +} + +// Position just past the closing quote of the key `""` at or after `from`. +size_t FindKey(const std::string& s, const std::string& key, size_t from) { + const std::string needle = "\"" + key + "\""; + return s.find(needle, from); +} + +// Read the integer array that follows `"":` starting the search at `from`. +// Returns the parsed values and sets `end` past the closing bracket. +std::vector ReadIntArrayAfter(const std::string& s, + const std::string& key, size_t from, + size_t* end) { + const size_t k = FindKey(s, key, from); + if (k == std::string::npos) + throw std::runtime_error("golden: key \"" + key + "\" not found"); + const size_t open = s.find('[', k); + if (open == std::string::npos) + throw std::runtime_error("golden: \"" + key + "\" is not an array"); + const size_t close = s.find(']', open); + if (close == std::string::npos) + throw std::runtime_error("golden: \"" + key + "\" array is unterminated"); + std::vector out; + size_t i = open + 1; + while (i < close) { + while (i < close && (s[i] == ' ' || s[i] == '\n' || s[i] == '\r' || + s[i] == '\t' || s[i] == ',')) + ++i; + if (i >= close) break; + size_t j = i; + if (s[j] == '-') ++j; + while (j < close && s[j] >= '0' && s[j] <= '9') ++j; + if (j == i) + throw std::runtime_error("golden: non-numeric entry in \"" + key + "\""); + out.push_back(static_cast(std::atoll(s.substr(i, j - i).c_str()))); + i = j; + } + if (end != nullptr) *end = close + 1; + return out; +} + +// Read the string value that follows `"":`. Used for the provenance lines +// this driver echoes so the run's evidence names the oracle and the revision it +// is being held to, rather than only the tokens. +std::string ReadStringAfter(const std::string& s, const std::string& key, + size_t from) { + const size_t k = FindKey(s, key, from); + if (k == std::string::npos) return ""; + const size_t colon = s.find(':', k); + if (colon == std::string::npos) return ""; + const size_t open = s.find('"', colon); + if (open == std::string::npos) return ""; + const size_t close = s.find('"', open + 1); + if (close == std::string::npos) return ""; + return s.substr(open + 1, close - open - 1); +} + +struct GoldenEntry { + std::vector prompt_token_ids; + std::vector token_ids; +}; + +struct Golden { + std::string vllm_version; + std::string model; + std::string revision; + std::vector entries; +}; + +Golden ReadGolden(const std::string& path) { + const std::string s = ReadWholeFile(path); + Golden g; + g.vllm_version = ReadStringAfter(s, "vllm", 0); + g.model = ReadStringAfter(s, "model", 0); + g.revision = ReadStringAfter(s, "revision", 0); + const size_t arr = FindKey(s, "golden", 0); + if (arr == std::string::npos) + throw std::runtime_error("golden: no \"golden\" array in " + path); + size_t cursor = arr; + // Each entry is delimited by its own "prompt_token_ids"; when that key stops + // appearing the array is exhausted. This is why a truncated golden yields + // FEWER entries rather than a silently empty one. + while (true) { + const size_t p = FindKey(s, "prompt_token_ids", cursor); + if (p == std::string::npos) break; + GoldenEntry e; + size_t after_prompt = 0; + e.prompt_token_ids = ReadIntArrayAfter(s, "prompt_token_ids", p, &after_prompt); + size_t after_tokens = 0; + e.token_ids = ReadIntArrayAfter(s, "token_ids", after_prompt, &after_tokens); + if (e.prompt_token_ids.empty()) + throw std::runtime_error("golden: an entry has an EMPTY prompt_token_ids"); + if (e.token_ids.empty()) + throw std::runtime_error("golden: an entry has an EMPTY token_ids"); + g.entries.push_back(std::move(e)); + cursor = after_tokens; + } + return g; +} + +std::string JoinIds(const std::vector& v, int n) { + std::string out; + for (int i = 0; i < n && i < static_cast(v.size()); ++i) { + out += std::to_string(v[static_cast(i)]); + if (i + 1 < n) out += ","; + } + return out; +} + +} // namespace + +int main(int argc, char** argv) { + std::string model, golden_path; + // The golden's own sampling block is `{temperature: 0.0, max_tokens: 32}`; + // 32 is the default here so the gate's shape comes from the oracle rather + // than from a flag somebody has to remember to pass. + int steps = 32; + int prompts = 0; // 0 => every entry the golden carries + // The battery's longest prompt is 13 tokens plus 32 continuations. A bounded + // max_model_len keeps the engine's per-request token tables sized for the + // battery rather than for the checkpoint's full context. + int max_model_len = 4096; + bool load_only = false; + // Parse the golden and print its geometry WITHOUT loading a model. It exists + // because the alternative way to find out whether this driver reads the + // golden correctly is a 20.1 GiB load, and because the geometry it prints — + // entry count and per-entry widths — is the number every count assertion + // below is measured against. + bool golden_info = false; + for (int i = 1; i < argc; ++i) { + const std::string a = argv[i]; + auto next = [&]() -> const char* { return (i + 1 < argc) ? argv[++i] : ""; }; + if (a == "--model") model = next(); + else if (a == "--golden") golden_path = next(); + else if (a == "--steps") steps = std::atoi(next()); + else if (a == "--prompts") prompts = std::atoi(next()); + else if (a == "--max-model-len") max_model_len = std::atoi(next()); + else if (a == "--load-only") load_only = true; + else if (a == "--golden-info") golden_info = true; + else { std::fprintf(stderr, "unknown arg %s\n", a.c_str()); return 2; } + } + if (model.empty() && !golden_info) { + std::fprintf(stderr, + "usage: --model --golden " + "[--steps N] [--prompts M] [--max-model-len N] [--load-only]\n" + " --golden --golden-info (parse only, no " + "model)\n"); + return 2; + } + if (golden_info && golden_path.empty()) { + std::fprintf(stderr, "--golden-info needs --golden \n"); + return 2; + } + + std::fprintf(stderr, "[nemotron-h] libvllm %s (ABI %d, header %d)\n", + vllm_version(), vllm_abi_version(), VLLM_ABI_VERSION); + if (!model.empty()) + std::fprintf(stderr, "[nemotron-h] model dir: %s\n", model.c_str()); + + Golden gold; + if (!golden_path.empty()) { + try { + gold = ReadGolden(golden_path); + } catch (const std::exception& e) { + std::fprintf(stderr, "[nemotron-h] golden FAILED to parse: %s\n", e.what()); + return 2; + } + // Provenance, printed BEFORE the run: a token match is only evidence about + // the oracle if the reader can see which oracle and which revision. + std::fprintf(stderr, + "[nemotron-h] golden %s\n" + " oracle vllm=%s oracle model=%s revision=%s\n" + " entries=%zu\n", + golden_path.c_str(), gold.vllm_version.c_str(), + gold.model.c_str(), gold.revision.c_str(), gold.entries.size()); + if (gold.entries.empty()) { + std::fprintf(stderr, + "[nemotron-h] REFUSING: the golden carries ZERO entries, so " + "a comparison here would report a perfect score over nothing\n"); + return 2; + } + // The geometry every count assertion below is measured against, printed + // per entry so a truncated or mis-parsed golden is visible BEFORE a load + // rather than as a suspiciously small "compared" number afterwards. + for (size_t i = 0; i < gold.entries.size(); ++i) { + std::fprintf(stderr, + " entry %zu: prompt_token_ids=%zu golden_width=%zu\n", + i, gold.entries[i].prompt_token_ids.size(), + gold.entries[i].token_ids.size()); + } + } + if (golden_info) return 0; + + vllm_model_params mp = vllm_model_params_default(); + mp.model_path = model.c_str(); + mp.max_model_len = max_model_len; + vllm_engine* eng = nullptr; + const auto t0 = std::chrono::steady_clock::now(); + const vllm_status lst = vllm_engine_load(&mp, &eng); + const auto t1 = std::chrono::steady_clock::now(); + if (lst != VLLM_OK) { + std::fprintf(stderr, "[nemotron-h] engine load FAILED: %s\n", + vllm_last_error()); + return 1; + } + std::fprintf(stderr, "[nemotron-h] engine loaded in %.1fs\n", + std::chrono::duration(t1 - t0).count()); + if (load_only) { + vllm_engine_free(eng); + return 0; + } + if (golden_path.empty()) { + std::fprintf(stderr, + "[nemotron-h] no --golden given: loaded only, nothing compared\n"); + vllm_engine_free(eng); + return 0; + } + + vllm_sampling_params sp = vllm_sampling_params_default(); + sp.temperature = 0.0f; // greedy — the oracle's own sampling block + sp.max_tokens = steps; + // The golden runs to a fixed max_tokens, so an EOS inside the window would + // truncate our side and compare fewer tokens than the oracle recorded. + sp.ignore_eos = 1; + + const int n_prompts = + (prompts > 0) ? std::min(prompts, static_cast(gold.entries.size())) + : static_cast(gold.entries.size()); + + int total_compared = 0, total_matched = 0, rows_full = 0, rows_short = 0; + for (int pi = 0; pi < n_prompts; ++pi) { + const GoldenEntry& e = gold.entries[static_cast(pi)]; + std::vector gen(static_cast(steps), 0); + int32_t n_gen = 0; + const auto ts = std::chrono::steady_clock::now(); + const vllm_status st = vllm_complete_tokens( + eng, e.prompt_token_ids.data(), + static_cast(e.prompt_token_ids.size()), &sp, gen.data(), + static_cast(gen.size()), &n_gen, nullptr); + const auto te = std::chrono::steady_clock::now(); + if (st != VLLM_OK) { + std::fprintf(stderr, "[nemotron-h] prompt %d FAILED: %s\n", pi, + vllm_last_error()); + vllm_engine_free(eng); + return 1; + } + + const int expected = static_cast(e.token_ids.size()); + const int n = std::min(expected, static_cast(n_gen)); + int row_match = 0; + for (int t = 0; t < n; ++t) { + ++total_compared; + if (gen[static_cast(t)] == e.token_ids[static_cast(t)]) { + ++total_matched; + ++row_match; + } + } + // The element count asserted against the expected geometry, per prompt. + // `compared` short of `golden_width` is a SHORT RUN and is named as one. + const bool full = (n == expected); + if (full) ++rows_full; else ++rows_short; + std::fprintf(stderr, + "[nemotron-h] prompt %d: prompt_tokens=%zu generated=%d " + "golden_width=%d compared=%d matched=%d wall=%.2fs%s\n", + pi, e.prompt_token_ids.size(), static_cast(n_gen), expected, + n, row_match, std::chrono::duration(te - ts).count(), + full ? "" : " <-- SHORT RUN, not a full row"); + if (row_match != n || !full) { + std::fprintf(stderr, " got: %s\n exp: %s\n", + JoinIds(gen, n).c_str(), JoinIds(e.token_ids, n).c_str()); + } + } + + std::fprintf(stderr, + "\n[nemotron-h] TOKEN MATCH: %d/%d over %d prompt(s) " + "(full rows=%d, short rows=%d)\n", + total_matched, total_compared, n_prompts, rows_full, rows_short); + vllm_engine_free(eng); + + // A pass needs three things to be true at once, and each is checked here + // rather than left to the reader of the log: something was compared, every + // compared token matched, and every row was compared to its FULL golden + // width. Dropping any one of them is how a mute instrument reports a pass. + if (total_compared == 0) { + std::fprintf(stderr, + "[nemotron-h] REFUSING: ZERO tokens compared — a mute " + "instrument, not a pass\n"); + return 3; + } + if (rows_short != 0) { + std::fprintf(stderr, + "[nemotron-h] SHORT: %d row(s) compared fewer tokens than the " + "golden carries; not a full-width result\n", + rows_short); + return 4; + } + if (total_matched != total_compared) { + std::fprintf(stderr, "[nemotron-h] DIVERGENCE\n"); + return 1; + } + std::fprintf(stderr, "[nemotron-h] STRICT PASS\n"); + return 0; +} From d72c6355b1a964d208f69ba470985892642c4fc8 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 17 Aug 2026 17:42:27 +0000 Subject: [PATCH 2/2] docs(#810 A3): correct the false blocker this row landed, and say which machine it was measured on The previous commit recorded a pending cause built on a probe that ran on the dgx HOST while describing the `rc` worker CONTAINER. For this purpose those are different machines, and reporting one as the other produced a confident, specific and wrong conclusion: that no CUDA binary could be built at all. Measured inside `rc run`, the container is Ubuntu 24.04 running as uid 0 and carries gcc, g++, cmake, ninja, make, python3, git and apt, with the GB10 visible to nvidia-smi and working DNS. Only `nvcc` is genuinely absent, and apt's own nvidia-cuda-toolkit is 12.0.140, too old for sm_121a, so CUDA 13.x installs from the NVIDIA repo. Neither docker nor sudo is involved anywhere, and the ask for them is withdrawn. The HOST toolchain finding is real and stands, but it gates nothing, because the host is not where work runs. The checkpoint sentences were wrong the same way. `findmnt` on the host shows `/usr/local/nas_share` mounted from `//192.168.68.102/Data` over cifs, and the checkpoint resolves under BOTH that path and `~/ckpt`, so `.env`'s CHECKPOINT_ROOT is correct and must not be "fixed". The mount line is quoted in the benchmark record rather than paraphrased, so the record carries the measurement instead of an attribution. Whether the CONTAINER can see that path is left OPEN and is claimed by nobody: the probe queued to answer it was killed to free the queue, and step 6 of the A3 job is now the only thing that will settle it. Corrected in place rather than quietly replaced. A known-false claim left inside a landed record is worse than the original error, because the next reader cannot tell it apart from the parts that were right, and the two-machine confusion is the part worth keeping. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/benchmark-record.md | 74 +++++++++++++------ .agents/specs/nemotron-h-a2p-paged-forward.md | 29 +++++--- .agents/specs/nemotron-h-abi-e2e.md | 40 +++++++--- docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 2 +- 5 files changed, 103 insertions(+), 44 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 41a301245..5a6fa2fc4 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -33,31 +33,61 @@ recorded cause is no longer true, and this is the #775 shape the governing spec §5.5 warns about: a pending reason that outlives its own truth and gets subtracted by everyone who reads it afterwards. -**What actually blocks the gate is that no CUDA binary can be built for that -host.** Three paths were checked, and all three are closed: - -1. **The host toolchain.** No `nvcc`, no `cmake`, no `g++` since the 14 Aug - reimage. Already filed as - [#1019](https://github.com/mudler/vllm.cpp/issues/1019), whose title says the - dgx profile "mandates a CUDA toolkit and a CUTLASS path that do not exist". -2. **The `rc` worker container**, which is what `rc run` schedules into. Probed - directly: `gcc`, `g++`, `cc`, `cmake`, `ninja`, `make`, `nvcc`, `python3`, - `git` and `docker` are all ABSENT, `/usr/include/stdio.h` does not exist, and - `getent hosts pypi.org` fails, so it cannot even fetch one. `/workspace` maps - to `/usr/local/nas_share/rc` on the host and carries no checkpoint. -3. **The recorded container build** (`$HOME/a2r/gb10_build.sh`: - `sudo -n docker run --runtime=nvidia nvidia/cuda:13.0.1-devel-ubuntu24.04`). - This is the only remaining path and it needs the host docker socket. `id` on - dgx reports `groups=1002(mudler),900(admin)`, **not `docker`**, so it - requires `sudo`, and `sudo docker` is refused by this agent's own permission - layer. That last clause is an agent-access fact, not a host fact, and is - reported as such rather than filed against the box. +**★ THIS ENTRY WAS FIRST WRITTEN WITH A FALSE BLOCKER AND IS CORRECTED IN PLACE.** +It claimed "no CUDA binary can be built for that host" over three closed paths. +**Two of those three were wrong**, and the error has one root: *the dgx HOST and +the `rc` worker CONTAINER are different machines, and host findings were reported +as container findings.* The correction is kept beside the claim rather than +substituted for it, because the failure mode is the interesting part. + +1. **The host toolchain — this part was RIGHT and stands.** Measured on + `ssh dgx.casa`: `nvcc ABSENT cmake ABSENT g++ ABSENT ninja ABSENT`. + Already filed as [#1019](https://github.com/mudler/vllm.cpp/issues/1019), + whose title says the dgx profile "mandates a CUDA toolkit and a CUTLASS path + that do not exist". **But the host is not where work runs**, so this never + blocked the gate the way this entry first said. +2. **The `rc` worker container — the original claim here was FALSE.** It said + `gcc`, `g++`, `cc`, `cmake`, `ninja`, `make`, `python3`, `git` were all + absent, that `/usr/include/stdio.h` did not exist, and that DNS failed. + Measured inside `rc run`, the container is Ubuntu 24.04, runs as **uid 0**, + and carries `gcc`, `g++`, `cmake`, `ninja`, `make`, `python3`, `pip3`, `git` + and `apt`; `nvidia-smi` reports the GB10; DNS resolves and + `developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/arm64/` answers + `HTTP/2 200`. **Only `nvcc` is genuinely absent**, and apt's own + `nvidia-cuda-toolkit` is 12.0.140, too old for sm_121a (GB10 needs 12.8+/13.x) + — so the toolkit is installed from the NVIDIA repo, which is a step, not a + wall. +3. **`docker` and `sudo` are NOT NEEDED AT ALL, and asking for them was the + error compounding.** You are root in the container; the recorded + `sudo -n docker run …` recipe in `$HOME/a2r/gb10_build.sh` belongs to the + pre-`rc` era. + +**The lesson, stated plainly for the next reader:** a probe that runs somewhere +other than where the work will run answers a question nobody asked. Re-derive +inside `rc run`, and do not let a host `ssh` result stand in for it. **What WAS verified on the gate host**, all of it cheap and under short leases: -- The checkpoint is present at `/home/mudler/ckpt/nemotron-3.5-lightning-30b-nvfp4` - (**not** `$CHECKPOINT_ROOT=/usr/local/nas_share/checkpoints`, which does not - exist there; there are no cifs/nfs mounts at all). +- **The checkpoint is present under BOTH paths, and `.env` is CORRECT.** An + earlier revision of this entry claimed the opposite — that only + `/home/mudler/ckpt/...` held it and that "there are no cifs/nfs mounts at + all". `findmnt` on the host says otherwise: + + ``` + /usr/local/nas_share //192.168.68.102/Data cifs rw,relatime,vers=3.1.1, + cache=strict,username=anonymous,uid=0,forceuid,gid=10001,forcegid, + addr=192.168.68.102,file_mode=0664,dir_mode=0775,iocharset=utf8,soft,nounix + ``` + + and the checkpoint resolves at + `/usr/local/nas_share/checkpoints/nemotron-3.5-lightning-30b-nvfp4`, + `/usr/local/nas_share/checkpoints/nemotron-3.5-lightning-30b-gguf` and + `/home/mudler/ckpt/nemotron-3.5-lightning-30b-nvfp4`. So + `CHECKPOINT_ROOT=/usr/local/nas_share/checkpoints` in `.env` is right and must + not be "fixed". +- **Whether the container can SEE that path is still OPEN.** No probe has + answered it: the one queued to do so was killed to free the queue. Nothing is + claimed here either way, and step 6 of the A3 job is what will answer it. - **21 583 809 748 bytes**, 52 safetensors shards. - The revision is **verified, not copied**: `model-00001-of-00052.safetensors` hashes to diff --git a/.agents/specs/nemotron-h-a2p-paged-forward.md b/.agents/specs/nemotron-h-a2p-paged-forward.md index 860d53e5b..de0198134 100644 --- a/.agents/specs/nemotron-h-a2p-paged-forward.md +++ b/.agents/specs/nemotron-h-a2p-paged-forward.md @@ -870,15 +870,26 @@ nobody routes this architecture through a block that ropes. (2026-08-17): §10 records contention — `dgx.casa` at loadavg 211 with 3 of 119 GB — and the box now answers at **loadavg 0.36 with 115 of 119 GB available and the GPU at 0%**, with the checkpoint present and its revision - verified against its own LFS record. The gate is blocked on something else - entirely: **there is no CUDA toolchain to build a gate binary with.** The host - carries no `nvcc` and no `cmake` since the 14 Aug reimage - ([#1019](https://github.com/mudler/vllm.cpp/issues/1019)), and the `rc` worker - container it schedules through carries no compiler, no libc headers and no - network egress, so the recorded recipe's `docker run nvidia/cuda:*-devel` on - the host is the only remaining path. This is the #775 shape the governing spec - §5.5 warns about — a pending cause outliving its own truth — so it is - corrected here rather than re-quoted. + verified against its own LFS record. So the contention cause is dead; this is + the #775 shape the governing spec §5.5 warns about — a pending cause outliving + its own truth — and it is corrected rather than re-quoted. + + **What replaced it was ALSO wrong for one revision, and that matters more.** + This bullet briefly claimed "there is no CUDA toolchain to build a gate binary + with", citing an `rc` worker container with no compiler, no libc headers and no + egress. **That was a host measurement reported as a container measurement.** + Inside `rc run` the container is Ubuntu 24.04 running as uid 0, with `gcc`, + `g++`, `cmake`, `ninja`, `make`, `python3`, `git` and `apt` all present, the + GB10 visible to `nvidia-smi`, and working DNS. **Only `nvcc` is absent**, and + apt's `nvidia-cuda-toolkit` 12.0.140 is too old for sm_121a, so CUDA 13.x is + installed from `developer.download.nvidia.com/…/ubuntu2404/arm64` — a step, not + a wall. No `docker` and no `sudo` are involved. The host toolchain finding + ([#1019](https://github.com/mudler/vllm.cpp/issues/1019)) is real but does not + gate anything, because the host is not where work runs. + + **What is actually outstanding** is narrower: `nvcc` must be installed into the + build container, and whether that container can see + `$CHECKPOINT_ROOT` is an OPEN question no probe has yet answered. - **`examples/nemotron_h_gen`** (§3.6) and the `docs/USAGE.md` weights block that rides with it — **both LANDED 2026-08-17** by `MODEL-NEMOTRON-H-ABI-A3-E2E`. The reason recorded here for deferring them — that shipping a client for a diff --git a/.agents/specs/nemotron-h-abi-e2e.md b/.agents/specs/nemotron-h-abi-e2e.md index 6d6b1252a..7b77b415c 100644 --- a/.agents/specs/nemotron-h-abi-e2e.md +++ b/.agents/specs/nemotron-h-abi-e2e.md @@ -1121,21 +1121,39 @@ five malformed-golden shapes each exit 2. That last one is the guard this section's own §5.2 is really about — a comparison over too few elements reports a perfect score, and here it cannot. -**§5.2's A3 token gate has NOT RUN, and its cause is not the recorded one.** -Re-measured rather than inherited: `dgx.casa` is idle (loadavg 0.36, 115 of -119 GB available, GPU 0%), the 20.1 GiB checkpoint is present, and its first -shard hashes to revision `29f2d174`'s own LFS record. The blocker is that -**nothing on that host can build a gate binary**: no `nvcc` and no `cmake` since -the 14 Aug reimage ([#1019](https://github.com/mudler/vllm.cpp/issues/1019)), -and the `rc` worker container it now schedules through has no compiler, no libc -headers and no egress. Recorded in `docs/BENCHMARKS.md` as **pending a named -resource**, never as a pass. +**§5.2's A3 token gate has NOT RUN. Its recorded cause was wrong twice, and both +corrections are kept here because the second one is a trap this section can save +the next reader from.** + +*First cause, dead:* contention. Re-measured, `dgx.casa` is idle (loadavg 0.36, +115 of 119 GB available, GPU 0%), the checkpoint is present, and its first shard +hashes to revision `29f2d174`'s own LFS record. + +*Second cause, also dead:* "nothing can build a gate binary". **That was a HOST +measurement reported as a CONTAINER measurement, and the two are different +machines for this purpose.** The host genuinely has no `nvcc`/`cmake` since the +14 Aug reimage ([#1019](https://github.com/mudler/vllm.cpp/issues/1019)) — but +the host is not where work runs. Inside `rc run` the worker container is Ubuntu +24.04 as **uid 0**, carrying `gcc`, `g++`, `cmake`, `ninja`, `make`, `python3`, +`git`, `apt`, with the GB10 visible and DNS working. **Only `nvcc` is missing**, +and apt's `nvidia-cuda-toolkit` 12.0.140 is too old for sm_121a, so CUDA 13.x +comes from `developer.download.nvidia.com/…/ubuntu2404/arm64`. Neither `docker` +nor `sudo` is involved. + +> **Rule this cost two cycles to learn: re-derive every environment fact INSIDE +> `rc run`.** A probe that runs somewhere other than where the work will run +> answers a question nobody asked, and it fails toward a confident verdict about +> the code's environment rather than toward an obvious error. + +*What is genuinely outstanding:* install `nvcc` in the build container, and +whether that container can see `$CHECKPOINT_ROOT`, which is **OPEN** — no probe +has answered it, and none is claimed. `docs/BENCHMARKS.md` records the gate as +**pending a named resource**, never as a pass. **§5.2 arm 2 (multi-request) is additionally blocked by design, not by a host.** G-SAFE refuses `input.num_reqs > 1` and A2-B owns that clause, so the three prompts cannot yet be submitted concurrently and interleaved. Arm 1 (multi-step, -single request, all 32 tokens) is what the driver is built for and what is owed -the moment a build host exists. +single request, all 32 tokens) is what the driver is built for. **§6.2's allowlist entry STAYS, and that was decided on evidence.** `nemotron_h.cpp:1031-1034` still refuses the NVFP4 `lm_head` on a non-CPU queue, diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 1e8c2550f..32b3817a0 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -10,7 +10,7 @@ | **Binary release (ACTIVE; Windows pre-alpha pending)** | v0.0.2 shipped eight primary archive/checksum/provenance triplets + two indexes (26 assets) from source SHA `7020de93652ca920424a10ac5255b34810dd2f24`, run `31466516224` | Windows W14-W16 implemented. **PENDING:** native hosted gates, merged-SHA ten-tuple dry run, matching-hardware evidence, v0.0.3-pre.1 publication, 32-asset audit | W12 optional/non-primary | | **Container images (ACTIVE; arm64 cuda verified on GB10 + Orin 2026-08-11)** | `ENG-RELEASE-CONTAINERS` ([spec](../.agents/specs/container-images.md)) | cpu amd64 783 MB; cuda arm64 **1.71 GB**. GB10 `sm_121a`: `/health`+`/version`+SIGTERM on `--gpus all`. Orin `sm_87` (Tegra): Qwen3-0.6B **generates**, GPU **GR3D 95-97%** | n/a | | **Developer/row protocol** | Contribution entry point; `ENG-NOW-DERIVED` #374 @`dbd0d51c` | Entry-point gates retained. #374 W1-W5 DONE; benchmark/runtime/parity `VOID`; row specs now carry `## Now` | n/a | -| **NemotronH paged forward** (`MODEL-NEMOTRON-H-ABI-A2P`, [#810](https://github.com/mudler/vllm.cpp/issues/810)) | **No number on any axis, by the unit's own rule** ([spec](../.agents/specs/nemotron-h-a2p-paged-forward.md) §5) | **A3 token gate PENDING on a BUILD HOST, not contention** (re-measured 2026-08-17: dgx idle, checkpoint revision-verified, no nvcc/cmake, [#1019](https://github.com/mudler/vllm.cpp/issues/1019)) | CPU gate 12/12, 9/9 mutations RED. A3 driver `examples/nemotron_h_gen` exists; guards armed on a real engine (pass 0, divergence 1, short 4) | +| **NemotronH paged forward** (`MODEL-NEMOTRON-H-ABI-A2P`, [#810](https://github.com/mudler/vllm.cpp/issues/810)) | **No number on any axis, by the unit's own rule** ([spec](../.agents/specs/nemotron-h-a2p-paged-forward.md) §5) | **A3 gate PENDING: `nvcc` + checkpoint visibility in the rc container.** Two earlier causes here (contention, then "cannot build") were measured FALSE, see [benchmark-record](../.agents/benchmark-record.md) | CPU gate 12/12, 9/9 mutations RED. A3 driver `examples/nemotron_h_gen` exists; guards armed on a real engine (pass 0, divergence 1, short 4) | | **LoRA runtime W2** (`LORA-RUNTIME`, #278) | **No number owed:** correctness-only; a grid PENDS the W7 model gate | | **ARCH audit: ABI is text-only** | 4 capabilities (H3 video, Laguna, Kimi-Linear, DeepSeek-V4) reachable only from `examples/`, none registry-backed. No gate asks whether a CONSUMER can reach a capability. Documentation only | | **DSR fix: server TU profiler guards (2026-08-09)** | **No number owed:** comments only. #189 moved the server body into the shared layer with its 5 `VT_BENCH_PROFILE_CONTROL` guards, taking DSR 32 -> 37; they are `DSR-ALLOW`'d per site, baseline unchanged at 32 | diff --git a/docs/STATUS.md b/docs/STATUS.md index db7f770d7..14e11ec01 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -131,7 +131,7 @@ token-for-token correctness against the pinned oracle. | GLM-4 dense (sandwich norms, partial rope) | Correctness-complete, speed-pending | Token-exact 16/16 (GLM-4-9B-0414); first GLM-family model; partial interleaved RoPE + Gemma2 sandwich norms + biased qkv | | GLM-4.7-Flash (MLA + GLM MoE) | Correctness-complete, speed-pending | Token-exact 8/8 (GLM-4.7-Flash, 31.2B); reuses the DeepSeek-V2 MLA stack; first e2e coverage of the q_lora query branch + noaux_tc sigmoid router with routed-scaling | | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **RUNNER FOLD LANDS (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; golden 122/128 (near-tie profile); FA2 MLA default-ON; `vllm_complete_tokens` (ABI v13).** Grouped-router top-k block-parallel (byte-identical); no binding speed number: ckpt is tiktoken-only, so no warm-server harness. STRICT stays CLOSED. Server 19.0 tok/s wall (~0.90× vLLM floor) = speed open | paged suite 8/8·206; SACRED post-fold 35B 315/315 + 27B 235/235; thin ABI client (ratchet 8) | -| Nemotron-3.5-Lightning-30B-A3B (Mamba2 + GQA + relu2 MoE) | **Paged forward + ABI driver land (#810 A2-P, A3); e2e token gate PENDING on a BUILD HOST, not contention** | `examples/nemotron_h_gen` reaches it through `include/vllm.h` alone; G-SAFE narrows to `num_reqs <= 1`. Gate cause re-measured: box idle, no nvcc/cmake ([#1019](https://github.com/mudler/vllm.cpp/issues/1019)) | +| Nemotron-3.5-Lightning-30B-A3B (Mamba2 + GQA + relu2 MoE) | **Paged forward + ABI driver land (#810 A2-P, A3); e2e token gate PENDING on `nvcc` + checkpoint visibility in the rc container** | `examples/nemotron_h_gen` reaches it through `include/vllm.h` alone; G-SAFE narrows to `num_reqs <= 1`. Two earlier pending causes on this row were measured false; the box is idle and the checkpoint is revision-verified | | Gemma-3 dense (GeGLU, dual rope, sandwich norms) | Correctness-complete, speed-pending | STRICT token-exact 48/48 greedy (gemma-3-1b-it); first Gemma-family model; GeGLU (gelu_pytorch_tanh) + dual per-layer RoPE theta + Gemma-RMSNorm sandwich norms + sqrt(hidden) embed-scale + query_pre_attn_scalar scaling | | Gemma-2 dense (attn + final logit soft-cap) | Correctness-complete, speed-pending | Near-tie-band 48/48 (gemma-2-2b-it): 44/48 strict on vLLM's greedy + 4/48 at 0.0-nat ties in vLLM's own logits; proves the attention + final logit soft-cap primitives (attn_logit_softcapping 50 + final 30); the inverse of Gemma-3 (both soft-caps, no QK-norm) | | Gemma-1 dense (the original Gemma) | Correctness-complete, speed-pending | STRICT token-exact 48/48 greedy (gemma-2b); two fused norms/layer, head_dim scale, GeGLU + sqrt(hidden) embed-scale, tied lm_head; no soft-cap/QK-norm/sliding. **D1 (2026-07-31): the whole Gemma family (1/2/3/4) folded to the default-ON bf16 merged-QKV descriptor (`MergedQkvEnabled`); re-gated Gemma-2 SACRED 48/48 (global+sliding) + Gemma-4 STRICT 32/32 — its existing gate held** |