diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 81c3aa0bd..d3f285af6 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -19,6 +19,109 @@ from relative link targets repointed for this file's location. # Benchmarks +## MODEL-NEMOTRON-H-ABI-A3-E2E — the A3 token gate did NOT run, and the cause on record was NOT the cause (2026-08-17, `row/MODEL-NEMOTRON-H-ABI-A3-E2E`, base `origin/main` `a6df72777`, #810) + +**No number is recorded, on any axis. This entry exists so the pending cause is +the measured one rather than the inherited one.** + +`.agents/specs/nemotron-h-a2p-paged-forward.md` §10 recorded the A3 gate as +pending on **contention**: `dgx.casa` observed at loadavg 211 with 3 of 119 GB +available, which is a real reason a 20.1 GiB checkpoint cannot load. Re-measured +on 2026-08-17 under an `rc` lease, that box answers at **loadavg 0.36, 115 of +119 GB available, GPU utilisation 0%**, with nothing of ours running on it. The +recorded cause is no longer true, and this is the #775 shape the governing spec +§5.5 warns about: a pending reason that outlives its own truth and gets +subtracted by everyone who reads it afterwards. + +**★ THIS ENTRY WAS FIRST WRITTEN WITH A FALSE BLOCKER AND IS CORRECTED IN PLACE.** +It claimed "no CUDA binary can be built for that host" over three closed paths. +**Two of those three were wrong**, and the error has one root: *the dgx HOST and +the `rc` worker CONTAINER are different machines, and host findings were reported +as container findings.* The correction is kept beside the claim rather than +substituted for it, because the failure mode is the interesting part. + +1. **The host toolchain — this part was RIGHT and stands.** Measured on + `ssh dgx.casa`: `nvcc ABSENT cmake ABSENT g++ ABSENT ninja ABSENT`. + Already filed as [#1019](https://github.com/mudler/vllm.cpp/issues/1019), + whose title says the dgx profile "mandates a CUDA toolkit and a CUTLASS path + that do not exist". **But the host is not where work runs**, so this never + blocked the gate the way this entry first said. +2. **The `rc` worker container — the original claim here was FALSE.** It said + `gcc`, `g++`, `cc`, `cmake`, `ninja`, `make`, `python3`, `git` were all + absent, that `/usr/include/stdio.h` did not exist, and that DNS failed. + Measured inside `rc run`, the container is Ubuntu 24.04, runs as **uid 0**, + and carries `gcc`, `g++`, `cmake`, `ninja`, `make`, `python3`, `pip3`, `git` + and `apt`; `nvidia-smi` reports the GB10; DNS resolves and + `developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/arm64/` answers + `HTTP/2 200`. **Only `nvcc` is genuinely absent**, and apt's own + `nvidia-cuda-toolkit` is 12.0.140, too old for sm_121a (GB10 needs 12.8+/13.x) + — so the toolkit is installed from the NVIDIA repo, which is a step, not a + wall. +3. **`docker` and `sudo` are NOT NEEDED AT ALL, and asking for them was the + error compounding.** You are root in the container; the recorded + `sudo -n docker run …` recipe in `$HOME/a2r/gb10_build.sh` belongs to the + pre-`rc` era. + +**The lesson, stated plainly for the next reader:** a probe that runs somewhere +other than where the work will run answers a question nobody asked. Re-derive +inside `rc run`, and do not let a host `ssh` result stand in for it. + +**What WAS verified on the gate host**, all of it cheap and under short leases: + +- **The checkpoint is present under BOTH paths, and `.env` is CORRECT.** An + earlier revision of this entry claimed the opposite — that only + `/home/mudler/ckpt/...` held it and that "there are no cifs/nfs mounts at + all". `findmnt` on the host says otherwise: + + ``` + /usr/local/nas_share //192.168.68.102/Data cifs rw,relatime,vers=3.1.1, + cache=strict,username=anonymous,uid=0,forceuid,gid=10001,forcegid, + addr=192.168.68.102,file_mode=0664,dir_mode=0775,iocharset=utf8,soft,nounix + ``` + + and the checkpoint resolves at + `/usr/local/nas_share/checkpoints/nemotron-3.5-lightning-30b-nvfp4`, + `/usr/local/nas_share/checkpoints/nemotron-3.5-lightning-30b-gguf` and + `/home/mudler/ckpt/nemotron-3.5-lightning-30b-nvfp4`. So + `CHECKPOINT_ROOT=/usr/local/nas_share/checkpoints` in `.env` is right and must + not be "fixed". +- **Whether the container can SEE that path is still OPEN.** No probe has + answered it: the one queued to do so was killed to free the queue. Nothing is + claimed here either way, and step 6 of the A3 job is what will answer it. +- **21 583 809 748 bytes**, 52 safetensors shards. +- The revision is **verified, not copied**: + `model-00001-of-00052.safetensors` hashes to + `672c8bda10fdec0256e0819e112d2aa3a936cc3e5d311a05fd3ff773ca9a44b9`, which is + what that file's own HF sidecar records for commit + `29f2d1746d8f41e316523194b19018707749b1b1`, the golden's revision. + +**The driver was built and its instrument proven armed, off the gate host.** +`examples/nemotron_h_gen` compiles and links against the real `vllm::shared` in +a CPU build (`BUILD_RC=0`, 0 compile errors). Its guards were then exercised +against a **real engine** on `opt-125m-bf16-st` rather than argued for, because +the driver is model-agnostic and the assertions are what a NemotronH run will +depend on: + +| arm | result | +|---|---| +| golden width 8, all 8 match | `STRICT PASS`, exit **0** | +| golden width 8, 0 of 8 match | `DIVERGENCE`, exit **1** | +| `--steps 4` against golden width 8, **4/4 matched** | `SHORT`, exit **4** | +| 5 malformed goldens (empty array, empty prompt, empty tokens, no array, truncated) | parse refusal, exit **2** each | + +The third row is the one worth keeping: it matched **every token it looked at** +and still refused, because it had looked at half the golden. A driver without +that check would have reported `4/4` as a pass. + +The committed golden's geometry, read by the driver itself (`--golden-info`): +**3 entries, prompt widths 5 / 8 / 13, golden width 32 each**, so a full A3 run +compares **96 tokens**. Any "compared" number below 96 is a short run and exits 4. + +**Also blocked, and by design rather than by a host:** the governing spec §5.2 +arm 2 (the three prompts submitted concurrently and interleaved). G-SAFE refuses +`input.num_reqs > 1` and A2-B owns that clause, so only arm 1 (multi-step, +single request) is reachable even once a build host exists. + ## KIMI-BF16-STREAM — bf16 residual stream end-to-end REFUTED (122→4/128, KDA repeat-loop destabilization, no speed win); STRICT is NOT reachable by residual-precision (§14-§20 all closed); 122/128 @ 18.9 tok/s (0.90× vLLM) is the coherent best; SERVER runner fold scoped (runner aborts on Kimi's KV today) (2026-08-07, `row/KIMI-BF16-STREAM-CLOSE`, base `origin/main` `2f029a10`, GB10 sm_121a, PR #118) The #113 follow-on tested the §19-named residual #1 — the bf16 residual stream END-TO-END, framed as diff --git a/.agents/specs/nemotron-h-a2p-paged-forward.md b/.agents/specs/nemotron-h-a2p-paged-forward.md index 8cb19cc10..de0198134 100644 --- a/.agents/specs/nemotron-h-a2p-paged-forward.md +++ b/.agents/specs/nemotron-h-a2p-paged-forward.md @@ -865,12 +865,43 @@ nobody routes this architecture through a block that ropes. - **The §5.4 A3 end-to-end token gate**, and the §5.7 sm_121a leg with it. Owned by this row, tracked on [#810](https://github.com/mudler/vllm.cpp/issues/810). - Nothing about the released checkpoint is claimed until it runs. + Nothing about the released checkpoint is claimed until it runs. **The recorded + PENDING CAUSE IS NO LONGER TRUE and was re-measured rather than inherited** + (2026-08-17): §10 records contention — `dgx.casa` at loadavg 211 with 3 of + 119 GB — and the box now answers at **loadavg 0.36 with 115 of 119 GB + available and the GPU at 0%**, with the checkpoint present and its revision + verified against its own LFS record. So the contention cause is dead; this is + the #775 shape the governing spec §5.5 warns about — a pending cause outliving + its own truth — and it is corrected rather than re-quoted. + + **What replaced it was ALSO wrong for one revision, and that matters more.** + This bullet briefly claimed "there is no CUDA toolchain to build a gate binary + with", citing an `rc` worker container with no compiler, no libc headers and no + egress. **That was a host measurement reported as a container measurement.** + Inside `rc run` the container is Ubuntu 24.04 running as uid 0, with `gcc`, + `g++`, `cmake`, `ninja`, `make`, `python3`, `git` and `apt` all present, the + GB10 visible to `nvidia-smi`, and working DNS. **Only `nvcc` is absent**, and + apt's `nvidia-cuda-toolkit` 12.0.140 is too old for sm_121a, so CUDA 13.x is + installed from `developer.download.nvidia.com/…/ubuntu2404/arm64` — a step, not + a wall. No `docker` and no `sudo` are involved. The host toolchain finding + ([#1019](https://github.com/mudler/vllm.cpp/issues/1019)) is real but does not + gate anything, because the host is not where work runs. + + **What is actually outstanding** is narrower: `nvcc` must be installed into the + build container, and whether that container can see + `$CHECKPOINT_ROOT` is an OPEN question no probe has yet answered. - **`examples/nemotron_h_gen`** (§3.6) and the `docs/USAGE.md` weights block that - rides with it. Not written: the ABI surface it would exercise - (`vllm_engine_load` + `vllm_complete_tokens`) is the same surface the A3 gate - drives, so writing the example before that gate can run would ship a client - for a path nobody has watched produce a token. Owned by this row, tracked on + rides with it — **both LANDED 2026-08-17** by `MODEL-NEMOTRON-H-ABI-A3-E2E`. + The reason recorded here for deferring them — that shipping a client for a + path nobody has watched produce a token is premature — was overtaken by the + measurement above: the gate is blocked on a host toolchain rather than on + anything about the code, and a driver that does not exist cannot be run the + moment that host is repaired. The example therefore lands with its counting + guards proven ARMED against a real engine on a small local checkpoint — a + full-width match exits 0, a divergence exits 1, a row that matched every token + it looked at but looked at HALF exits 4, and five malformed-golden shapes each + exit 2 — and with **no claim whatsoever** about what the released 30B + checkpoint emits. Owned by this row, tracked on [#810](https://github.com/mudler/vllm.cpp/issues/810). - **The device `lm_head`** stays A2-Q2b's, which is why `scripts/runner-routing-allowlist.txt` is narrowed rather than removed and why diff --git a/.agents/specs/nemotron-h-abi-e2e.md b/.agents/specs/nemotron-h-abi-e2e.md index ad01fd5bb..7b77b415c 100644 --- a/.agents/specs/nemotron-h-abi-e2e.md +++ b/.agents/specs/nemotron-h-abi-e2e.md @@ -1104,13 +1104,65 @@ lifecycle write. ## 9. Now -**State at this commit:** spec only. No product code, no lifecycle change. Per -§1.4 the implementation is a **separate** pull request by a **different** agent; -this one carries the spec. - -A1 is claimable now against `main` + `bc570da0d`, with PR #868 as its base -(§7 R2). A2 is blocked on #496 W2, and that block must be re-verified rather than -inherited from this sentence. +**State at 2026-08-17.** A1 (`c1d02bfbe`), A2-R (`598226e96`), A2-Q2a +(`6abc769c6`) and A2-P (`a6df72777`) have all landed. G-SAFE is narrowed to +`input.num_reqs <= 1` and `ForwardNemotronHForCausalLM` selects +`NemotronHPagedForward` whenever the runner supplies paged KV and recurrent +state, which was verified in the tree rather than inherited from this sentence. + +**§6.1's driver EXISTS: `examples/nemotron_h_gen`.** One project include +(`vllm.h`), linked `vllm::shared`, no `example-abi-allowlist.txt` row, modelled +on `kimi_linear_gen` and not on the two allowlisted examples §6.1 warns about. +It builds and links against the real shared library, and its counting guards are +proven armed against a **real engine** on a small local checkpoint rather than +argued for: a full-width match exits 0, a divergence exits 1, a row that matched +every token it looked at while looking at HALF the golden's width exits 4, and +five malformed-golden shapes each exit 2. That last one is the guard this +section's own §5.2 is really about — a comparison over too few elements reports +a perfect score, and here it cannot. + +**§5.2's A3 token gate has NOT RUN. Its recorded cause was wrong twice, and both +corrections are kept here because the second one is a trap this section can save +the next reader from.** + +*First cause, dead:* contention. Re-measured, `dgx.casa` is idle (loadavg 0.36, +115 of 119 GB available, GPU 0%), the checkpoint is present, and its first shard +hashes to revision `29f2d174`'s own LFS record. + +*Second cause, also dead:* "nothing can build a gate binary". **That was a HOST +measurement reported as a CONTAINER measurement, and the two are different +machines for this purpose.** The host genuinely has no `nvcc`/`cmake` since the +14 Aug reimage ([#1019](https://github.com/mudler/vllm.cpp/issues/1019)) — but +the host is not where work runs. Inside `rc run` the worker container is Ubuntu +24.04 as **uid 0**, carrying `gcc`, `g++`, `cmake`, `ninja`, `make`, `python3`, +`git`, `apt`, with the GB10 visible and DNS working. **Only `nvcc` is missing**, +and apt's `nvidia-cuda-toolkit` 12.0.140 is too old for sm_121a, so CUDA 13.x +comes from `developer.download.nvidia.com/…/ubuntu2404/arm64`. Neither `docker` +nor `sudo` is involved. + +> **Rule this cost two cycles to learn: re-derive every environment fact INSIDE +> `rc run`.** A probe that runs somewhere other than where the work will run +> answers a question nobody asked, and it fails toward a confident verdict about +> the code's environment rather than toward an obvious error. + +*What is genuinely outstanding:* install `nvcc` in the build container, and +whether that container can see `$CHECKPOINT_ROOT`, which is **OPEN** — no probe +has answered it, and none is claimed. `docs/BENCHMARKS.md` records the gate as +**pending a named resource**, never as a pass. + +**§5.2 arm 2 (multi-request) is additionally blocked by design, not by a host.** +G-SAFE refuses `input.num_reqs > 1` and A2-B owns that clause, so the three +prompts cannot yet be submitted concurrently and interleaved. Arm 1 (multi-step, +single request, all 32 tokens) is what the driver is built for. + +**§6.2's allowlist entry STAYS, and that was decided on evidence.** +`nemotron_h.cpp:1031-1034` still refuses the NVFP4 `lm_head` on a non-CPU queue, +so the forward's last step is a host projection and it returns `HostLogits`. +Deleting `scripts/runner-routing-allowlist.txt:26` was tried in a scratch copy: +`check-runner-routing-consistency.py` goes from `OK` to `ERROR` naming +`ForwardNemotronHForCausalLM returns HostLogits`, exit 1, tree restored +byte-for-byte. A2-Q2b removes the entry; widening the allowlist to satisfy the +checker is the defect the checker exists to stop. **Three things to read before the first edit**, because each has already cost somebody a cycle: §5.5, so the six `#873` gates are subtracted rather than @@ -1118,10 +1170,12 @@ chased; §5.4, so the token gate is planned for `dgx.casa` and Thor and not for the local box, which cannot see anything device-side; and §7 R2, so #775 is consumed rather than re-fixed. -**Next action:** a fresh implementer claims A1 from §1, captures the §3.1 red -first, and lands A1 with the G-SAFE interlock opened through -`vllm::ModelAs`. A fresh reviewer — never the implementer — -runs the §3.4 mutations and reports M3 as a pair. +**Next action:** repair a build host (#1019), then run +`nemotron-h-gen --model --golden +tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json` with +`VT_NEMOTRON35_SNAPSHOT` unset, and record the resolved directory. Expect 96 +tokens compared over 3 prompts of width 32; a "compared" number below 96 is a +short run and the driver exits 4 rather than reporting it as a match. ## 10. Outcome diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index aabcdae2e..cc516c4e5 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -10,7 +10,7 @@ | **Binary release (ACTIVE; Windows pre-alpha pending)** | v0.0.2 shipped eight primary archive/checksum/provenance triplets + two indexes (26 assets) from source SHA `7020de93652ca920424a10ac5255b34810dd2f24`, run `31466516224` | Windows W14-W16 implemented. **PENDING:** native hosted gates, merged-SHA ten-tuple dry run, matching-hardware evidence, v0.0.3-pre.1 publication, 32-asset audit | W12 optional/non-primary | | **Container images (ACTIVE; arm64 cuda verified on GB10 + Orin 2026-08-11)** | `ENG-RELEASE-CONTAINERS` ([spec](../.agents/specs/container-images.md)) | cpu amd64 783 MB; cuda arm64 **1.71 GB**. GB10 `sm_121a`: `/health`+`/version`+SIGTERM on `--gpus all`. Orin `sm_87` (Tegra): Qwen3-0.6B **generates**, GPU **GR3D 95-97%** | n/a | | **Developer/row protocol** | Contribution entry point; `ENG-NOW-DERIVED` #374 @`dbd0d51c` | Entry-point gates retained. #374 W1-W5 DONE; benchmark/runtime/parity `VOID`; row specs now carry `## Now` | n/a | -| **NemotronH paged forward** (`MODEL-NEMOTRON-H-ABI-A2P`, [#810](https://github.com/mudler/vllm.cpp/issues/810)) | **No number on any axis, by the unit's own rule** ([spec](../.agents/specs/nemotron-h-a2p-paged-forward.md) §5) | **A3 token gate PENDING on CONTENTION:** `dgx.casa` at loadavg 211 with 3 of 119 GB available | CPU gate 12/12, 9/9 mutations RED | +| **NemotronH paged forward** (`MODEL-NEMOTRON-H-ABI-A2P`, [#810](https://github.com/mudler/vllm.cpp/issues/810)) | **No number on any axis, by the unit's own rule** ([spec](../.agents/specs/nemotron-h-a2p-paged-forward.md) §5) | **A3 gate PENDING: `nvcc` + checkpoint visibility in the rc container.** Two earlier causes here (contention, then "cannot build") were measured FALSE, see [benchmark-record](../.agents/benchmark-record.md) | CPU gate 12/12, 9/9 mutations RED. A3 driver `examples/nemotron_h_gen` exists; guards armed on a real engine (pass 0, divergence 1, short 4) | | **LoRA runtime W2** (`LORA-RUNTIME`, #278) | **No number owed:** correctness-only; a grid PENDS the W7 model gate | | **ARCH audit: ABI is text-only** | 4 capabilities (H3 video, Laguna, Kimi-Linear, DeepSeek-V4) reachable only from `examples/`, none registry-backed. No gate asks whether a CONSUMER can reach a capability. Documentation only | | **DSR fix: server TU profiler guards (2026-08-09)** | **No number owed:** comments only. #189 moved the server body into the shared layer with its 5 `VT_BENCH_PROFILE_CONTROL` guards, taking DSR 32 -> 37; they are `DSR-ALLOW`'d per site, baseline unchanged at 32 | diff --git a/docs/STATUS.md b/docs/STATUS.md index f09f36303..7d46436ef 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -131,7 +131,7 @@ token-for-token correctness against the pinned oracle. | GLM-4 dense (sandwich norms, partial rope) | Correctness-complete, speed-pending | Token-exact 16/16 (GLM-4-9B-0414); first GLM-family model; partial interleaved RoPE + Gemma2 sandwich norms + biased qkv | | GLM-4.7-Flash (MLA + GLM MoE) | Correctness-complete, speed-pending | Token-exact 8/8 (GLM-4.7-Flash, 31.2B); reuses the DeepSeek-V2 MLA stack; first e2e coverage of the q_lora query branch + noaux_tc sigmoid router with routed-scaling | | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **RUNNER FOLD LANDS (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; golden 122/128 (near-tie profile); FA2 MLA default-ON; `vllm_complete_tokens` (ABI v13).** Grouped-router top-k block-parallel (byte-identical); no binding speed number: ckpt is tiktoken-only, so no warm-server harness. STRICT stays CLOSED. Server 19.0 tok/s wall (~0.90× vLLM floor) = speed open | paged suite 8/8·206; SACRED post-fold 35B 315/315 + 27B 235/235; thin ABI client (ratchet 8) | -| Nemotron-3.5-Lightning-30B-A3B (Mamba2 + GQA + relu2 MoE) | **Paged forward lands (#810 A2-P); e2e token gate PENDING on gate-host contention** | K/V and the conv + SSM rows now live in the runner's pages, so decode step 2 keeps state; G-SAFE narrows to `num_reqs <= 1`. CPU gate 12/12, 9/9 mutations RED | +| Nemotron-3.5-Lightning-30B-A3B (Mamba2 + GQA + relu2 MoE) | **Paged forward + ABI driver land (#810 A2-P, A3); e2e token gate PENDING on `nvcc` + checkpoint visibility in the rc container** | `examples/nemotron_h_gen` reaches it through `include/vllm.h` alone; G-SAFE narrows to `num_reqs <= 1`. Two earlier pending causes on this row were measured false; the box is idle and the checkpoint is revision-verified | | Gemma-3 dense (GeGLU, dual rope, sandwich norms) | Correctness-complete, speed-pending | STRICT token-exact 48/48 greedy (gemma-3-1b-it); first Gemma-family model; GeGLU (gelu_pytorch_tanh) + dual per-layer RoPE theta + Gemma-RMSNorm sandwich norms + sqrt(hidden) embed-scale + query_pre_attn_scalar scaling | | Gemma-2 dense (attn + final logit soft-cap) | Correctness-complete, speed-pending | Near-tie-band 48/48 (gemma-2-2b-it): 44/48 strict on vLLM's greedy + 4/48 at 0.0-nat ties in vLLM's own logits; proves the attention + final logit soft-cap primitives (attn_logit_softcapping 50 + final 30); the inverse of Gemma-3 (both soft-caps, no QK-norm) | | Gemma-1 dense (the original Gemma) | Correctness-complete, speed-pending | STRICT token-exact 48/48 greedy (gemma-2b); two fused norms/layer, head_dim scale, GeGLU + sqrt(hidden) embed-scale, tied lm_head; no soft-cap/QK-norm/sliding. **D1 (2026-07-31): the whole Gemma family (1/2/3/4) folded to the default-ON bf16 merged-QKV descriptor (`MergedQkvEnabled`); re-gated Gemma-2 SACRED 48/48 (global+sliding) + Gemma-4 STRICT 32/32 — its existing gate held** | diff --git a/docs/USAGE.md b/docs/USAGE.md index 3816f41b7..e0af37c99 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -535,7 +535,7 @@ tokens quietly. | Architecture | Why it refuses | |---|---| | `KimiK3ForConditionalGeneration` | Needs ~1.56 TB (MXFP4); no host here can run it | -| `NemotronHForCausalLM` | The hybrid forward is ported (#517 W4) and the weight loader materializes the real checkpoint, but that forward is a HOST reference: it recomputes K/V over the whole sequence every step, carries no recurrent state between steps and treats a batch as one causal sequence. Engine construction now SUCCEEDS — the KV allocation reads the model's own recurrent spec (#810) — and the first step then refuses by name, naming the paged/batched decode path as the missing piece rather than returning plausible wrong tokens. **That refusal is UNCHANGED by A2-R (#810)**: A2-R adds a partial device arm (embedding lookup, the 52 layer norms + `norm_f`, and the 6 GQA attention blocks; Mamba2, MoE and `lm_head` stay on the host), but it is non-paged and single-request, so it creates none of the capability the refusal guards and is not reachable through `include/vllm.h`. It is exercised only by `test_nemotron_h_forward`, and it records no throughput number. Safetensors resolve and parse; a GGUF file is refused by name, since no GGUF arm exists for it | +| `NemotronHForCausalLM` | **Only BATCHED decode still refuses.** A2-P (#810) narrowed this: `ForwardNemotronHForCausalLM` now selects the paged forward whenever the runner supplies paged KV and recurrent state, so K/V go into the runner's pages and the conv/SSM rows are carried across steps, and `examples/nemotron_h_gen` reaches all of it through `include/vllm.h` alone. What is left is `num_reqs > 1`, refused by name because one request's pages and one request's recurrent state are carried per step and a multi-request step would be decoded as ONE concatenated causal sequence — plausible wrong tokens rather than a failure. Owed to A2-B. **The end-to-end token gate against the pinned oracle has NOT run**, so no claim is made here about what this checkpoint emits; `docs/BENCHMARKS.md` records that as pending rather than as silence. `lm_head` and the FP8 Mamba2 projections still compute on the host, and a GGUF file is refused by name since no GGUF arm exists for it. See *Nemotron-3.5-Lightning-30B: the exact weights, and which arms run* below | This is a deliberate state, not a bug: registering the architecture is what lets the config parse and weight-name mapping be tested before the forward exists. @@ -2294,6 +2294,80 @@ Set `VLLM_MUSE_GGUF=` (or `VLLM_MUSE_GGUF_LOAD=` for the full materialization) to run `test_muse_glimmer_gguf` against a real checkpoint; without them the gate runs off committed header-only manifests. +## Nemotron-3.5-Lightning-30B: the exact weights, and which arms run + +`NemotronHForCausalLM` is a hybrid: 6 GQA attention layers over a paged KV cache +and 23 Mamba2 layers over a recurrent conv/SSM state, with MoE blocks between +them. `examples/nemotron_h_gen` (`nemotron-h-gen`) drives it through the public +C ABI and nothing else — `vllm_engine_load` + `vllm_complete_tokens` — against +the committed oracle golden: + +```sh +nemotron-h-gen --model "$CHECKPOINT_ROOT/nemotron-3.5-lightning-30b-nvfp4" \ + --golden tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json +``` + +`--golden-info` parses the golden and prints its geometry without loading a +model, which is how you check the battery's shape before spending a 20.1 GiB +load. `--load-only` stops after `vllm_engine_load`. + +### The checkpoint + +| field | value | +|---|---| +| repo | [nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4) — first party | +| revision | `29f2d1746d8f41e316523194b19018707749b1b1` | +| staged as | `$CHECKPOINT_ROOT/nemotron-3.5-lightning-30b-nvfp4` (a `hf download --local-dir` tree) | +| on-disk total | 21 583 809 748 bytes (20.1 GiB) | +| weights | `model-000{01..52}-of-00052.safetensors` + `model.safetensors.index.json` | +| quantization | `config.json` (1 337 760 B) + `hf_quant_config.json` (928 085 B), the `modelopt_mixed` layout | +| tokenizer | `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`, `chat_template.jinja` | +| sha256 (first shard) | `672c8bda10fdec0256e0819e112d2aa3a936cc3e5d311a05fd3ff773ca9a44b9` for `model-00001-of-00052.safetensors` (743 427 168 B) | + +**A repo id alone is not a pin** — checkpoints get re-quantized in place under an +unchanged name — so the revision is recorded, and it was verified rather than +copied: the first shard on the gate host hashes to the value above, which is +that revision's own LFS record for the file +(`.cache/huggingface/download/model-00001-of-00052.safetensors.metadata`, whose +sidecar names commit `29f2d174`). `tests/parity/hf_snapshot.h` resolves the +directory and refuses a tree staged at any other revision, so +`VT_NEMOTRON35_SNAPSHOT` is left UNSET for a gate run: setting it takes the +explicit-directory escape, which is deliberately not revision-checked. + + hf download nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \ + --revision 29f2d1746d8f41e316523194b19018707749b1b1 \ + --local-dir "$CHECKPOINT_ROOT/nemotron-3.5-lightning-30b-nvfp4" + +### The arms, and what each one costs you today + +The loader materializes all 18 487 tensors in the memory format the checkpoint +ships them in, so nothing is silently widened at load. What differs between arms +is **where the arithmetic happens**, and that is not something a token +comparison can see, so it is written down here instead. + +| arm | state | +|---|---| +| bf16 layers, norms, the 6 GQA attention blocks | **device** | +| MoE experts, NVFP4 W4A16 g16 | **device** (Marlin arena) | +| FP8 W8A8 static Mamba2 input projections | **host** — the device arm is owed, [#940](https://github.com/mudler/vllm.cpp/issues/940) | +| `lm_head`, NVFP4 W4A16 g16 | **host** — it refuses a non-CPU queue by name, so the forward's last step is a host projection and the model still returns host logits. Owed to A2-Q2b, [#810](https://github.com/mudler/vllm.cpp/issues/810) | + +And the arms that are **refused by name** rather than substituted: + +| arm | the refusal | +|---|---| +| GGUF k-quants / i-quants | not ported. A GGUF path is refused at load naming `.agents/specs/nemotron-h-model.md` §5b W7, because silently dequantizing to a supported path is exactly what a token gate cannot see | +| the MTP draft head | deferred by name at load (W5) | +| batched decode (`num_reqs > 1`) | refused at the forward. One request's KV pages and one request's recurrent state are carried per step; a multi-request step would be decoded as ONE concatenated causal sequence and would return plausible wrong tokens instead of failing. Owed to A2-B, [#810](https://github.com/mudler/vllm.cpp/issues/810) | + +### What has NOT been measured + +**No token gate result exists for this checkpoint yet.** The example above is the +vehicle for it and the golden is committed, but the run itself is pending; the +current state is recorded in `docs/BENCHMARKS.md` rather than left as silence, +and nothing about the released checkpoint's output is claimed here until it is +green. + ## MiniMax-H3: video + audio generation ### The exact weights (so a render is reproducible) diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt index 6379905b8..6ac37b929 100644 --- a/examples/CMakeLists.txt +++ b/examples/CMakeLists.txt @@ -30,6 +30,17 @@ add_executable(kimi-linear-gen kimi_linear_gen/main.cpp) target_link_libraries(kimi-linear-gen PRIVATE vllm::shared) vllm_cpp_set_warnings(kimi-linear-gen) +# nemotron-h-gen: the Nemotron-3.5-Lightning-30B (NemotronHForCausalLM) A3 +# end-to-end token battery against the PINNED vLLM ORACLE golden, and a THIN +# CLIENT of the public C ABI in exactly the kimi-linear-gen shape above: +# vllm_engine_load + the v13 vllm_complete_tokens over the golden's +# pre-tokenized prompts, linking the packaged shared library so it reaches no +# internal header and needs no example-abi-allowlist row. #810, +# .agents/specs/nemotron-h-abi-e2e.md §6.1. +add_executable(nemotron-h-gen nemotron_h_gen/main.cpp) +target_link_libraries(nemotron-h-gen PRIVATE vllm::shared) +vllm_cpp_set_warnings(nemotron-h-gen) + # vllm-bench (M2.1): the throughput/latency benchmark harness — the gate-#1 # measurement tool. Links the full C++ engine (vllm::vllm) so it can build both a # synthetic CPU engine (no --model) and a real loaded engine (--model on dgx). diff --git a/examples/nemotron_h_gen/main.cpp b/examples/nemotron_h_gen/main.cpp new file mode 100644 index 000000000..605f29bca --- /dev/null +++ b/examples/nemotron_h_gen/main.cpp @@ -0,0 +1,366 @@ +// nemotron-h-gen — THIN PUBLIC-ABI CLIENT (ONE SURFACE / ARCH-ONE-SURFACE). +// +// The Nemotron-3.5-Lightning-30B (`NemotronHForCausalLM`) greedy token battery +// against the PINNED vLLM ORACLE golden, driven ENTIRELY through the flat C ABI +// (include/vllm.h): `vllm_engine_load` builds the full engine — the A2 weight +// loader's 18487 tensors in the format the checkpoint ships them in, plus the +// shared paged runner whose caches select `NemotronHPagedForward` — and +// `vllm_complete_tokens` (ABI v13) generates from the golden's pre-tokenized +// prompts. This file includes "vllm.h" and NOTHING else from the project, links +// `vllm::shared`, and is deliberately NOT on `scripts/example-abi-allowlist.txt`: +// per .agents/specs/nemotron-h-abi-e2e.md §6.1 it is modelled on +// examples/kimi_linear_gen, never on deepseek_v4_gen or laguna_gen, both of +// which drive a bespoke forward through internal headers and are the transition +// state that allowlist exists to retire. +// +// nemotron-h-gen --model --golden +// [--steps N] [--prompts M] [--max-model-len N] [--load-only] +// +// The golden is the A3 gate's operand: +// tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json, captured from +// the pinned oracle (`vllm 0.23.1rc1.dev1511+g555967922`, the setuptools_scm +// spelling of pin 5559679229bc) at `temperature 0.0, max_tokens 32` over three +// prompts. It carries `prompt_token_ids` per prompt, so this driver needs no +// tokenizer agreement established first, and `token_ids`, the 32 tokens each +// prompt must reproduce. +// +// ── WHY THIS PRINTS COUNTS, LOUDLY ────────────────────────────────────────── +// A comparison over zero elements reports a perfect score. This driver +// therefore refuses rather than passes when it compared nothing: it asserts a +// NON-ZERO prompt count and, per prompt, that the number of tokens actually +// compared equals the golden's own width. `--steps` below that width is a +// short run and is reported as such, never as a match. +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm.h" + +namespace { + +// ── A deliberately small reader for THIS golden's shape ───────────────────── +// The golden is a fixed, committed document, not arbitrary JSON: an object with +// a "golden" array whose entries carry "prompt", "prompt_token_ids" and +// "token_ids". A full JSON parser in an example would be a second +// implementation to maintain; a scanner that finds a named key and reads the +// integer array or string that follows it is enough, and it FAILS LOUDLY on a +// shape it does not recognise rather than returning an empty vector that would +// read downstream as "nothing to compare". + +std::string ReadWholeFile(const std::string& path) { + std::ifstream f(path, std::ios::binary); + if (!f) throw std::runtime_error("cannot open " + path); + std::ostringstream ss; + ss << f.rdbuf(); + return ss.str(); +} + +// Position just past the closing quote of the key `""` at or after `from`. +size_t FindKey(const std::string& s, const std::string& key, size_t from) { + const std::string needle = "\"" + key + "\""; + return s.find(needle, from); +} + +// Read the integer array that follows `"":` starting the search at `from`. +// Returns the parsed values and sets `end` past the closing bracket. +std::vector ReadIntArrayAfter(const std::string& s, + const std::string& key, size_t from, + size_t* end) { + const size_t k = FindKey(s, key, from); + if (k == std::string::npos) + throw std::runtime_error("golden: key \"" + key + "\" not found"); + const size_t open = s.find('[', k); + if (open == std::string::npos) + throw std::runtime_error("golden: \"" + key + "\" is not an array"); + const size_t close = s.find(']', open); + if (close == std::string::npos) + throw std::runtime_error("golden: \"" + key + "\" array is unterminated"); + std::vector out; + size_t i = open + 1; + while (i < close) { + while (i < close && (s[i] == ' ' || s[i] == '\n' || s[i] == '\r' || + s[i] == '\t' || s[i] == ',')) + ++i; + if (i >= close) break; + size_t j = i; + if (s[j] == '-') ++j; + while (j < close && s[j] >= '0' && s[j] <= '9') ++j; + if (j == i) + throw std::runtime_error("golden: non-numeric entry in \"" + key + "\""); + out.push_back(static_cast(std::atoll(s.substr(i, j - i).c_str()))); + i = j; + } + if (end != nullptr) *end = close + 1; + return out; +} + +// Read the string value that follows `"":`. Used for the provenance lines +// this driver echoes so the run's evidence names the oracle and the revision it +// is being held to, rather than only the tokens. +std::string ReadStringAfter(const std::string& s, const std::string& key, + size_t from) { + const size_t k = FindKey(s, key, from); + if (k == std::string::npos) return ""; + const size_t colon = s.find(':', k); + if (colon == std::string::npos) return ""; + const size_t open = s.find('"', colon); + if (open == std::string::npos) return ""; + const size_t close = s.find('"', open + 1); + if (close == std::string::npos) return ""; + return s.substr(open + 1, close - open - 1); +} + +struct GoldenEntry { + std::vector prompt_token_ids; + std::vector token_ids; +}; + +struct Golden { + std::string vllm_version; + std::string model; + std::string revision; + std::vector entries; +}; + +Golden ReadGolden(const std::string& path) { + const std::string s = ReadWholeFile(path); + Golden g; + g.vllm_version = ReadStringAfter(s, "vllm", 0); + g.model = ReadStringAfter(s, "model", 0); + g.revision = ReadStringAfter(s, "revision", 0); + const size_t arr = FindKey(s, "golden", 0); + if (arr == std::string::npos) + throw std::runtime_error("golden: no \"golden\" array in " + path); + size_t cursor = arr; + // Each entry is delimited by its own "prompt_token_ids"; when that key stops + // appearing the array is exhausted. This is why a truncated golden yields + // FEWER entries rather than a silently empty one. + while (true) { + const size_t p = FindKey(s, "prompt_token_ids", cursor); + if (p == std::string::npos) break; + GoldenEntry e; + size_t after_prompt = 0; + e.prompt_token_ids = ReadIntArrayAfter(s, "prompt_token_ids", p, &after_prompt); + size_t after_tokens = 0; + e.token_ids = ReadIntArrayAfter(s, "token_ids", after_prompt, &after_tokens); + if (e.prompt_token_ids.empty()) + throw std::runtime_error("golden: an entry has an EMPTY prompt_token_ids"); + if (e.token_ids.empty()) + throw std::runtime_error("golden: an entry has an EMPTY token_ids"); + g.entries.push_back(std::move(e)); + cursor = after_tokens; + } + return g; +} + +std::string JoinIds(const std::vector& v, int n) { + std::string out; + for (int i = 0; i < n && i < static_cast(v.size()); ++i) { + out += std::to_string(v[static_cast(i)]); + if (i + 1 < n) out += ","; + } + return out; +} + +} // namespace + +int main(int argc, char** argv) { + std::string model, golden_path; + // The golden's own sampling block is `{temperature: 0.0, max_tokens: 32}`; + // 32 is the default here so the gate's shape comes from the oracle rather + // than from a flag somebody has to remember to pass. + int steps = 32; + int prompts = 0; // 0 => every entry the golden carries + // The battery's longest prompt is 13 tokens plus 32 continuations. A bounded + // max_model_len keeps the engine's per-request token tables sized for the + // battery rather than for the checkpoint's full context. + int max_model_len = 4096; + bool load_only = false; + // Parse the golden and print its geometry WITHOUT loading a model. It exists + // because the alternative way to find out whether this driver reads the + // golden correctly is a 20.1 GiB load, and because the geometry it prints — + // entry count and per-entry widths — is the number every count assertion + // below is measured against. + bool golden_info = false; + for (int i = 1; i < argc; ++i) { + const std::string a = argv[i]; + auto next = [&]() -> const char* { return (i + 1 < argc) ? argv[++i] : ""; }; + if (a == "--model") model = next(); + else if (a == "--golden") golden_path = next(); + else if (a == "--steps") steps = std::atoi(next()); + else if (a == "--prompts") prompts = std::atoi(next()); + else if (a == "--max-model-len") max_model_len = std::atoi(next()); + else if (a == "--load-only") load_only = true; + else if (a == "--golden-info") golden_info = true; + else { std::fprintf(stderr, "unknown arg %s\n", a.c_str()); return 2; } + } + if (model.empty() && !golden_info) { + std::fprintf(stderr, + "usage: --model --golden " + "[--steps N] [--prompts M] [--max-model-len N] [--load-only]\n" + " --golden --golden-info (parse only, no " + "model)\n"); + return 2; + } + if (golden_info && golden_path.empty()) { + std::fprintf(stderr, "--golden-info needs --golden \n"); + return 2; + } + + std::fprintf(stderr, "[nemotron-h] libvllm %s (ABI %d, header %d)\n", + vllm_version(), vllm_abi_version(), VLLM_ABI_VERSION); + if (!model.empty()) + std::fprintf(stderr, "[nemotron-h] model dir: %s\n", model.c_str()); + + Golden gold; + if (!golden_path.empty()) { + try { + gold = ReadGolden(golden_path); + } catch (const std::exception& e) { + std::fprintf(stderr, "[nemotron-h] golden FAILED to parse: %s\n", e.what()); + return 2; + } + // Provenance, printed BEFORE the run: a token match is only evidence about + // the oracle if the reader can see which oracle and which revision. + std::fprintf(stderr, + "[nemotron-h] golden %s\n" + " oracle vllm=%s oracle model=%s revision=%s\n" + " entries=%zu\n", + golden_path.c_str(), gold.vllm_version.c_str(), + gold.model.c_str(), gold.revision.c_str(), gold.entries.size()); + if (gold.entries.empty()) { + std::fprintf(stderr, + "[nemotron-h] REFUSING: the golden carries ZERO entries, so " + "a comparison here would report a perfect score over nothing\n"); + return 2; + } + // The geometry every count assertion below is measured against, printed + // per entry so a truncated or mis-parsed golden is visible BEFORE a load + // rather than as a suspiciously small "compared" number afterwards. + for (size_t i = 0; i < gold.entries.size(); ++i) { + std::fprintf(stderr, + " entry %zu: prompt_token_ids=%zu golden_width=%zu\n", + i, gold.entries[i].prompt_token_ids.size(), + gold.entries[i].token_ids.size()); + } + } + if (golden_info) return 0; + + vllm_model_params mp = vllm_model_params_default(); + mp.model_path = model.c_str(); + mp.max_model_len = max_model_len; + vllm_engine* eng = nullptr; + const auto t0 = std::chrono::steady_clock::now(); + const vllm_status lst = vllm_engine_load(&mp, &eng); + const auto t1 = std::chrono::steady_clock::now(); + if (lst != VLLM_OK) { + std::fprintf(stderr, "[nemotron-h] engine load FAILED: %s\n", + vllm_last_error()); + return 1; + } + std::fprintf(stderr, "[nemotron-h] engine loaded in %.1fs\n", + std::chrono::duration(t1 - t0).count()); + if (load_only) { + vllm_engine_free(eng); + return 0; + } + if (golden_path.empty()) { + std::fprintf(stderr, + "[nemotron-h] no --golden given: loaded only, nothing compared\n"); + vllm_engine_free(eng); + return 0; + } + + vllm_sampling_params sp = vllm_sampling_params_default(); + sp.temperature = 0.0f; // greedy — the oracle's own sampling block + sp.max_tokens = steps; + // The golden runs to a fixed max_tokens, so an EOS inside the window would + // truncate our side and compare fewer tokens than the oracle recorded. + sp.ignore_eos = 1; + + const int n_prompts = + (prompts > 0) ? std::min(prompts, static_cast(gold.entries.size())) + : static_cast(gold.entries.size()); + + int total_compared = 0, total_matched = 0, rows_full = 0, rows_short = 0; + for (int pi = 0; pi < n_prompts; ++pi) { + const GoldenEntry& e = gold.entries[static_cast(pi)]; + std::vector gen(static_cast(steps), 0); + int32_t n_gen = 0; + const auto ts = std::chrono::steady_clock::now(); + const vllm_status st = vllm_complete_tokens( + eng, e.prompt_token_ids.data(), + static_cast(e.prompt_token_ids.size()), &sp, gen.data(), + static_cast(gen.size()), &n_gen, nullptr); + const auto te = std::chrono::steady_clock::now(); + if (st != VLLM_OK) { + std::fprintf(stderr, "[nemotron-h] prompt %d FAILED: %s\n", pi, + vllm_last_error()); + vllm_engine_free(eng); + return 1; + } + + const int expected = static_cast(e.token_ids.size()); + const int n = std::min(expected, static_cast(n_gen)); + int row_match = 0; + for (int t = 0; t < n; ++t) { + ++total_compared; + if (gen[static_cast(t)] == e.token_ids[static_cast(t)]) { + ++total_matched; + ++row_match; + } + } + // The element count asserted against the expected geometry, per prompt. + // `compared` short of `golden_width` is a SHORT RUN and is named as one. + const bool full = (n == expected); + if (full) ++rows_full; else ++rows_short; + std::fprintf(stderr, + "[nemotron-h] prompt %d: prompt_tokens=%zu generated=%d " + "golden_width=%d compared=%d matched=%d wall=%.2fs%s\n", + pi, e.prompt_token_ids.size(), static_cast(n_gen), expected, + n, row_match, std::chrono::duration(te - ts).count(), + full ? "" : " <-- SHORT RUN, not a full row"); + if (row_match != n || !full) { + std::fprintf(stderr, " got: %s\n exp: %s\n", + JoinIds(gen, n).c_str(), JoinIds(e.token_ids, n).c_str()); + } + } + + std::fprintf(stderr, + "\n[nemotron-h] TOKEN MATCH: %d/%d over %d prompt(s) " + "(full rows=%d, short rows=%d)\n", + total_matched, total_compared, n_prompts, rows_full, rows_short); + vllm_engine_free(eng); + + // A pass needs three things to be true at once, and each is checked here + // rather than left to the reader of the log: something was compared, every + // compared token matched, and every row was compared to its FULL golden + // width. Dropping any one of them is how a mute instrument reports a pass. + if (total_compared == 0) { + std::fprintf(stderr, + "[nemotron-h] REFUSING: ZERO tokens compared — a mute " + "instrument, not a pass\n"); + return 3; + } + if (rows_short != 0) { + std::fprintf(stderr, + "[nemotron-h] SHORT: %d row(s) compared fewer tokens than the " + "golden carries; not a full-width result\n", + rows_short); + return 4; + } + if (total_matched != total_compared) { + std::fprintf(stderr, "[nemotron-h] DIVERGENCE\n"); + return 1; + } + std::fprintf(stderr, "[nemotron-h] STRICT PASS\n"); + return 0; +}