From 9bffa2b603ee8e7d29ccbba7e03ce862b7b9d857 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 14 Aug 2026 12:38:37 +0000 Subject: [PATCH 1/6] feat(MODEL-NEMOTRON-H): the WEIGHT LOADER -- the real checkpoint runs, in the formats it ships in (#517) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the blocker §6b named as "still owed after W4": nothing materialized the 18487 enumerated tensors, so every checkpoint load left NemotronHHostWeights unmaterialized and the forward refused by name. Spec §6d is the authority. RED first, off the real 20.1 GiB checkpoint at the CONTENT-pinned revision 29f2d174 (#569): `Status: FAILURE!`, exit 1, THREW "host weights are not materialized", peak RSS after "load" 29 MiB. Note the instrument trap in the same output -- `assertions: 3 | 3 passed | 0 failed` beside a red gate, because the case THREW. EVERY WEIGHT IS HELD IN THE FORMAT THE CHECKPOINT SHIPS IT IN, and that is arithmetic rather than taste: the 5888 routed-expert projections alone are 29.4e9 parameters, 16.5 GiB packed against 58.7 GB at bf16. A dequantize-at-load loader does not fit on any box this project owns, and on a unified-memory box that is a reboot rather than a failed load. Measured host mirror 18013 MiB against 18013 MiB read out of the shards. The host reference forward composes vt::MatmulBT and has no NVFP4 and no FP8 entry point, so it widens a quantized operand TRANSIENTLY at the GEMM call site through the shared model_loader/nvfp4_dequant.h seam. That arm is DECLARED -- named in the header, named at the call site, counted by the load report -- and the quantized GEMMs stay kMoeGroupedGemmNvfp4Marlin and the fp8-linear registration, which W6 selects. NemotronHOwned::View now refuses a non-dense weight by name, so a packed buffer cannot be reinterpreted as the model dtype. Gated STRUCTURALLY, not only by tokens, because a checkpoint read as uniform NVFP4 stays numerically plausible and still matches tokens while moving the wrong bytes: enumerated / in the index 18487 / 18487 materialized + deferred 18217 + 270 = 18487 (MTP tower -> W5, by name) NVFP4 W4A16 g16 5935 projections / 17805 tensors FP8 W8A8 static 46 projections / 138 tensors fp8 KV scales 12 unquantized bf16 / f32 216 / 46 the five rows sum to 18217 == materialized widenings 69, and no more host bytes 18,888,922,112 (17.59 GiB) peak RSS 17.70 GiB after load, 18.38 GiB after a forward {W4A16_NVFP4: 5935, FP8: 46} is exactly the histogram W1 measured over the 5981 quantized_layers entries, now confirmed against the TENSORS. A finding worth carrying: A_log, D and dt_bias ship BF16 on disk and the forward requires them f32; those 69 widenings are upstream's own polarity and are counted so they cannot spread. EVIDENCE, not the W6 token gate: one forward per committed oracle prompt, argmax of the last position against the golden's FIRST generated token -- 3/3 (6993, 1032, 1349) on x86_64 AND on Jetson Thor. W6 still owns the token gate. Five mutations, each applied alone and restored with its SHA-256 re-verified: uniform-NVFP4 and the dropped KV scales refuse by name; the un-widened SSM scalars red the widening count and then the dtype; and the flipped nibble order and the ignored weight_scale_2 each give 0/3 goldens while 45 of 46 assertions STILL PASS -- a loader gate built only out of counts would have called both clean. The load lives in nemotron_h_weights.cpp, beside the enumeration it must agree with tensor-for-tensor, which is where every other architecture puts its load and which avoids a CMakeLists.txt edit that would have owed a docs/USAGE.md change with nothing honest to write in it. Reported, not repaired: test_op_parity is RED on this base for a reason owned by MODEL-MUSIC-MUSIC3 (#672) -- RunGoldenPass does `m["op"]` on every manifest.json under the goldens root and minimax_music3_oracle's has no `op` key. It landed at 34dc57876, an ancestor of this base. Squashed off origin/main so check-doc-checkpoint's PER-COMMIT rule is satisfied trivially, the same call §5e made for W3; `git diff` against the merged branch tree is EMPTY. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/nemotron-h-model.md | 250 ++++++++- docs/FEATURES.md | 2 +- scripts/runner-routing-allowlist.txt | 2 +- src/vllm/model_executor/models/nemotron_h.cpp | 161 +++++- .../models/nemotron_h_forward.h | 88 ++- .../model_executor/models/nemotron_h_loader.h | 132 +++++ .../models/nemotron_h_registry.cpp | 37 +- .../models/nemotron_h_weights.cpp | 503 +++++++++++++++++- tests/CMakeLists.txt | 20 + tests/vllm/models/test_nemotron_h_loader.cpp | 319 +++++++++++ 10 files changed, 1478 insertions(+), 36 deletions(-) create mode 100644 src/vllm/model_executor/models/nemotron_h_loader.h create mode 100644 tests/vllm/models/test_nemotron_h_loader.cpp diff --git a/.agents/specs/nemotron-h-model.md b/.agents/specs/nemotron-h-model.md index 6dfe71a2f..098bdf41a 100644 --- a/.agents/specs/nemotron-h-model.md +++ b/.agents/specs/nemotron-h-model.md @@ -1298,6 +1298,214 @@ fold measures 1.91e-07 peak-relative; "25.3x the signal" for `scale_logits` measures 0.568. The qualitative claims are right and were independently verified; the numbers appear to be mean-relative or from an earlier fixture. +## 6d. The WEIGHT LOADER — the checkpoint runs (2026-08-14) + +§6b closed with "the **weight loader** — nothing materializes the 18487 +enumerated tensors, so no checkpoint can be run and the forward refuses by name +on every load", and §7 said the loader "§4's table does not name as a W of its +own and should". This is that brick, built on `row/MODEL-NEMOTRON-H-LOADER` off +`22367c551`. + +**RED first, off the real checkpoint.** Resolved through +`parity::Nemotron35LightningSnapshot()` (content-pinned, #569) to +`/mnt/nas_share/checkpoints/nemotron-3.5-lightning-30b-nvfp4`: `Status: +FAILURE!`, exit 1, `THREW: NemotronHForCausalLM forward: host weights are not +materialized`, peak RSS after "load" **29 MiB**. Note the instrument trap in the +same output — `assertions: 3 | 3 passed | 0 failed` beside a red gate, because +the case THREW ([[doctest-assertions-line-hides-thrown-cases]]). + +### The design decision, and why it is arithmetic rather than taste + +**Every weight is held in the memory format the checkpoint SHIPS it in.** Not +because widening is untidy, but because a dequantize-at-load loader does not fit +on any box this project owns: the 5888 routed-expert projections alone are +29.4e9 parameters, **16.5 GiB packed against 58.7 GB at bf16**. On a +unified-memory box that is not a failed load, it is a reboot +([[gb10-unified-memory-oom-reboots-box]]). + +Measured host mirror: **18013 MiB**, against 18013 MiB read out of the shards — +the mirror is the checkpoint, not a widened copy of it. Peak RSS **17.70 GiB** +after the load and **18.38 GiB** at the end of a forward. + +The consequence is that the HOST reference forward, which composes +`vt::MatmulBT` and has no NVFP4 and no FP8 entry point, widens a quantized +operand TRANSIENTLY at the GEMM call site through the shared +`model_loader/nvfp4_dequant.h` seam. That is a **declared** arm, named in +`nemotron_h_loader.h` and `nemotron_h.cpp` and reported by the load report — not +a silent fallback of the kind §8's stop condition forbids. The quantized GEMMs +remain `kMoeGroupedGemmNvfp4Marlin` and the fp8-linear registration, which W6 +selects on the device path. `NemotronHOwned::View` now REFUSES a non-dense +weight by name, so a packed buffer cannot be reinterpreted as the model dtype by +a caller that did not think about it. + +One consequential wiring change followed: the routed-expert loop in +`NemotronHMoeMixer` now visits its (token, slot) pairs **expert-major** rather +than token-major, so one dequant of an NVFP4 expert serves all of that expert's +rows. Each pair's own `NonGatedExpert` call is unchanged (one row against one +expert), and each pair writes a disjoint `expert_out` slot and reads nothing +another wrote, so the ORDER cannot change the result; `test_nemotron_h_forward` +stays 13/13 and 254/254. + +### The structural gate, as hard numbers + +`tests/vllm/models/test_nemotron_h_loader.cpp`. It is gated STRUCTURALLY and not +only by tokens, because a checkpoint read as uniform NVFP4 stays numerically +plausible and still matches tokens while moving the wrong bytes. + +| Row | Measured | +|---|---| +| enumerated / in `model.safetensors.index.json` | **18487 / 18487** | +| materialized + deferred | **18217 + 270 = 18487** | +| deferred BY NAME (MTP tower, W5) | **270**, every tag naming W5 | +| NVFP4 W4A16 g16 | **5935 projections / 17805 tensors** | +| FP8 W8A8 static | **46 projections / 138 tensors** | +| fp8 KV scales (`k_scale`/`v_scale`) | **12** | +| unquantized bf16 / f32 on disk | **216 / 46** | +| the five scheme rows sum to `materialized` | 18217 == 18217 | +| widenings (bf16 on disk, f32 in memory) | **69**, and no more | +| host bytes | **18,888,922,112** (17.59 GiB) | + +The `{W4A16_NVFP4: 5935, FP8: 46}` split is exactly the histogram W1 measured +over all 5981 `quantized_layers` entries, now confirmed against the TENSORS +rather than the config. The 69 widenings are the three f32-by-contract SSM +scalars (`A_log`, `D`, `dt_bias`) on 23 mamba layers — upstream's own polarity +(`-torch.exp(self.A_log.float())`) and what `vt::Mamba2ChunkScan` validates. + +**One finding worth carrying: `A_log`, `D` and `dt_bias` ship BF16 on disk.** +The forward requires them f32, so the loader widens; a loader that inherited the +model dtype instead produces a numerically plausible model. M3 below is the +instrument. + +### EVIDENCE, not the W6 token gate + +For each of the three committed oracle prompts, ONE forward over its +`prompt_token_ids` and the argmax of the last position against the golden's +FIRST generated token: + +| Prompt | argmax | oracle | +|---|---|---| +| `The capital of France is` | 6993 | 6993 | +| `Write the first five Fibonacci numbers:` | 1032 | 1032 | +| `Explain what a state space model is, in one sentence:` | 1349 | 1349 | + +**3/3.** W6 still owns the token gate (identical prompts, counts, batching and +sampling, oracle identity asserted, full 32-token greedy decode); this consumes +one token per prompt and makes no speed claim. It is here because it is the only +check that can fail for a reason the structural gate cannot see — every count can +be right while a group scale is transposed or a nibble order is flipped. + +The doctest 2.5.2 `const char*` trap this row already repaired once (§"W1 +land-prep" LOW-2) bit again in the first run of exactly this line: the verdict +printed as `oracle 69931`, the `1` being doctest rendering a `const char*` +lvalue. Bound to a `std::string`. + +### Mutation proof (IMP-MUTATE) + +Each applied ALONE to the restored tree, rebuilt, run against the real +checkpoint, then restored and the file's SHA-256 re-verified. + +| Mutation | Result | +|---|---| +| M1 mamba `in_proj` read as NVFP4 instead of FP8 (the "uniform NVFP4" defect) | **FAILURE!** — THREW `'backbone.layers.0.mixer.in_proj.weight' ships dtype F8_E4M3, not the U8 its scheme declares`; 1 case / 0 passed, assertions 2 | +| M2 the 12 fp8-KV scales silently dropped | **FAILURE!** — THREW `'backbone.layers.5.mixer.k_proj.k_scale' (consumer 'attn.k_scale[fp8-kv]') is enumerated but no host slot claimed it`; assertions 2 | +| M3 `A_log`/`D`/`dt_bias` inherit the model dtype (no widening) | **FAILURE!** — `rep.widened_tensors == 69` red, then THREW `weight 'mixer.A_log' has the wrong dtype for this arm`; assertions 35, 1 failed | +| M4 NVFP4 nibble order flipped to `kHighFirst` | **FAILURE!** — **0/3** goldens (argmax 66822 / 60300 / 31645 vs 6993 / 1032 / 1349); assertions 46, exactly **1** failed | +| M5 `weight_scale_2` read (so the accounting stays right) and then ignored | **FAILURE!** — **0/3** goldens (argmax 1321 / 2142 / 1321); assertions 46, exactly **1** failed | + +M1's first form — BOTH mamba projections switched — could not be COMPILED: +`-Werror=unused-function` rejected the now-unreferenced `LoadFp8`. That is a +real, if accidental, second gate, the same one W4's M8/M9/M14 hit; it was re-run +with `out_proj` left on `LoadFp8`. + +**M4 and M5 are the pair that keeps the golden arm honest.** Both leave EVERY +structural count correct — 45 of 46 assertions still pass, and the one that +fails is `matched == total`. A loader gate built only out of counts would have +called both of them clean, and both are exactly the "still numerically +plausible, still the right shape, wrong bytes" class §1 warns about. M1-M3 +conversely are invisible to the golden arm, because they refuse before a token +exists. + +**Restoration.** After each, the file was rewritten from the captured original +and its SHA-256 re-verified (`186ae4cea4d7…` for `nemotron_h_loader.cpp`, +`554d1c7fc65b…` for `nemotron_h.cpp`); `git status --porcelain` showed only the +spec and `docs/FEATURES.md` edits this section is part of. + +### Gate evidence + +Local x86_64 CPU-only host (GNU 13.3, Ninja, `VLLM_CPP_CUDA=OFF`), disk recorded +beside every number because this box has hit 100% mid-run before and a build +that ENOSPCs leaves the PREVIOUS binary in place +([[stale-binary-prints-green-status]]). + +| Arm | Result | disk free | +|---|---|---| +| Release `-Werror`, clean full build | **exit 0, 0 `warning:` lines, 0 `No space left` lines**, 891/891 targets | 53G / 88% | +| `test_nemotron_h_loader` (Release, live checkpoint) | **1/1 cases, 46/46 assertions, `Status: SUCCESS!`**, 10:07 wall, VmHWM 19,270,444 KiB | 36G / 92% | +| `test_nemotron_h_forward` (Release) | 13/13, 254/254, `Status: SUCCESS!` | 53G | +| `test_nemotron_h_scaffold` (Release) | 12/12, 38285/38285, `Status: SUCCESS!` | 53G | +| Debug (`-g0`, asserts unmasked) forward | **13/13, 254/254, `Status: SUCCESS!`** | 36G | +| Debug (`-g0`) scaffold | **12/12, 38285/38285, `Status: SUCCESS!`** | 36G | +| Debug (`-g0`) **loader, live checkpoint** | **1/1, 46/46, `Status: SUCCESS!`**, 3/3 goldens, 15:21 wall, VmHWM 19,277,028 KiB | 26G / 95% | +| full `ctest -j4` | **449 of 451 passed**, then **450 of 451**: `test_engine_core_proc` failed under `-j4` and passes ALONE (14/14, 113/113, exit 0) — the known starvation set, and this box was running the Debug loader arm at the time. `test_op_parity` FAILED, **pre-existing on the base** (below). Skipped: `test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`; `test_nemotron_h_loader` passed vacuously with no `CHECKPOINT_ROOT` in that shell | 26G / 94% | + +The forward and scaffold counts are IDENTICAL to W4's (13/254, 12/38285), which +is the evidence that the expert-major reorder is result-neutral rather than an +assertion that it is. + +**Jetson Thor (`kairos-4db2`, aarch64, sm_110) — a CUDA build AND the real +checkpoint.** Transferred by `git archive` and md5-verified on both ends +(`1cdc92214540646cc1fb7c9e87c87b23`); the 20.1 GiB checkpoint staged to +`/home/mudler/nemo-loader/ckpt` with its `.cache/huggingface/download/*.metadata` +sidecars intact, so the CONTENT pin resolves there too. Container +`vllmcpp-build:aarch64`, `--runtime=nvidia`, `NVIDIA_DISABLE_REQUIRE=1`, nvcc +13.0.88, `-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=110 +-DVLLM_CPP_TRITON=OFF`, no CUTLASS — the six `DISABLED (no requested arch in +[110] provides it)` lines are CORRECT for sm_110, not a silent fallback. Disk +408-439G free / 50-54% throughout; RAM 122 GiB, and the box was loaded ALONE. + +| Thor arm | Result | +|---|---| +| build | **`BUILD_EXIT=0`, 0 `warning:` lines, 0 `No space left` lines** | +| `test_nemotron_h_forward` | **13/13, 254/254, `Status: SUCCESS!`** — identical to x86_64 | +| `test_nemotron_h_scaffold` | **12/12, 38285/38285, `Status: SUCCESS!`** — identical to x86_64 | +| `test_nemotron_h_loader`, **live checkpoint at `/w/ckpt`** | **1/1, 46/46, `Status: SUCCESS!`**, `LOADER_EXIT=0`, **3/3** goldens, 3:00 wall | +| peak RSS on Thor | **17.85 GiB** after load, **18.53 GiB** at the end | + +Two cross-architecture agreements worth naming, because neither was arranged. +`host bytes: 18013 MiB, source 18013 MiB` is byte-for-byte what x86_64 reported, +and the logits range is **`[-9.4375, 8.75]` on both**. The 3:00 wall against +x86_64's 10:07 is the storage, not the CPU: Thor reads the checkpoint off local +NVMe and the x86 box reads it over SMB from the NAS. + +`/usr/bin/time` is NOT in that container — the first attempt exited **127** and +would have read as a failed run to anyone grepping only for `SUCCESS`. The RSS +above is the test's own `/proc/self/status` `VmHWM`, which is why it has one. + +**One honestly weak property, recorded rather than discovered.** With no +checkpoint the gate emits a `MESSAGE` naming the missing export and RETURNS, +so CTest records a PASS, not a *Skipped* — a CI run cannot tell it apart from a +real pass without reading the log. That is the convention +`test_nemotron_h_scaffold`'s live case already set for this same checkpoint on +this same row, and matching it beat inventing a second one; the alternative is +`test_modelopt_mixed_precision_checkpoint`'s exit-77, which needs a custom +`main`. Named here so the next reader does not have to work it out. + +**`test_op_parity` is RED on the base and not this row's.** `RunGoldenPass` +(`tests/parity/test_op_parity.cpp:1852-1860`) walks every subdirectory of the +goldens root and does `std::string op = m["op"];` on any `manifest.json` it +finds. `tests/parity/goldens/minimax_music3_oracle/manifest.json` has no `op` +key — its keys are `captured_on, checkpoint, environment, generated_by, issue, +model, oracle, request, result, spec, spec_disagreements, spec_facts` — so the +pass throws `[json.exception.type_error.302] type must be string, but is null`. +That manifest landed at `34dc57876` (`oracle(MODEL-MUSIC-MUSIC3)`, #672 / #708), +which `git merge-base --is-ancestor 34dc57876 22367c551` confirms is an ancestor +of this branch's base, and this branch's diff touches no file under +`tests/parity/` or `src/vt/`. Reported rather than repaired: `tests/parity/` is +outside this task's authority, the fix belongs to MODEL-MUSIC-MUSIC3 (either an +`op` key or the `manifest.json` renamed so the op walker skips it as the +tokenizer golden dirs already are), and repairing a pre-existing break inside a +scoped loader change would hide it — the same call §5d finding 5 made. + ## 7. Now **State at this commit:** **W1 and W3 have LANDED on `main`; W2 is in @@ -1325,20 +1533,34 @@ no `STATUS`/`BENCHMARKS` write. **Oracle gateability is CLOSED** — §5a record the pinned oracle loading and running the checkpoint on GB10 with three greedy goldens committed, so W6 has a denominator whenever it is reached. -**Next action:** **W4 is written and gated on `row/MODEL-NEMOTRON-H-W4B` (§6b) -and needs a FRESH REVIEW** — never the agent that wrote it — which should mutate -the self-certification (§6b M15) and the two anti-vacuity guards that were -rewritten, because those are the claims this W changes rather than adds. - -After W4 lands, the next brick is **the WEIGHT LOADER**, which §4's table does -not name as a W of its own and should: W5 (MTP), W6 (the e2e token gate) and W7 -(GGUF) all sit behind it, and until it exists the forward refuses by name on -every checkpoint load. Also carried forward, not resolved: the two OWED GPU items -in §6a (`kMoeGroupedGemmNvfp4Marlin` on the real g16 tensors, and the end-to-end -NemotronH MoE block on GB10), and the OWED GGUF k-quant arm tracked as W7 (§5b). - -The row stays `INVENTORIED`: W4 changes no lifecycle state, because nothing runs -end to end yet. +**W4 has LANDED** (`ce8c8bf67`, #718), and the WEIGHT LOADER §7 named as "the +next brick" is built and gated on `row/MODEL-NEMOTRON-H-LOADER` — **§6d is the +authority on it**. The forward no longer refuses on a checkpoint load: 18487 of +18487 tensors are accounted (18217 materialized in their shipped formats, 270 +deferred by name to W5), the real 20.1 GiB checkpoint runs at **17.70 GiB peak +RSS**, and its first greedy token matches the pinned oracle's committed golden on +**3 of 3** prompts. + +**Next action:** the loader needs a **FRESH REVIEW** — never the agent that wrote +it. The two claims it changes rather than adds, and which a review should mutate, +are (a) `NemotronHOwned::View`'s refusal of a non-dense weight, which is the only +thing standing between a packed NVFP4 buffer and a plausible-garbage GEMM operand, +and (b) the expert-major reorder in `NemotronHMoeMixer`, whose result-neutrality +is claimed from the disjointness of the output slots rather than measured. + +Then W5 (the MTP head, whose 270 tensors the loader already names as owed), W6 +(the e2e token gate against the committed goldens, now unblocked — it has weights), +and W7 (GGUF). Carried forward, not resolved: the two OWED GPU items in §6a +(`kMoeGroupedGemmNvfp4Marlin` on the real g16 tensors, and the end-to-end +NemotronH MoE block on GB10) — the loader now produces exactly those g16 tensors, +so the first of them is reachable — and the OWED GGUF k-quant arm (§5b). + +**Reported, outside this task's authority to fix.** `test_op_parity` is RED on +this row's base for a reason belonging to MODEL-MUSIC-MUSIC3 (#672); §6d's gate +evidence has the diagnosis. + +The row stays `INVENTORIED`: the loader changes no lifecycle state, because the +forward is still the HOST reference and nothing runs on the paged runner (W6). ## 8. Stop conditions diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 5c256b052..3804de4d1 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -138,7 +138,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on, via the `laguna-gen` CLI; the registered engine forward VT_CHECKs non-bf16 (`ARCH-ONE-SURFACE` fold) | | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Folded onto the shared paged runner (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (the intrinsic near-tie profile); FA2 paged MLA default-ON; SACRED post-fold green** | Served via `vllm_engine_load` + `vllm_complete_tokens` (ABI v13); server 19.0 tok/s wall vs vLLM ~21 (~0.90×), speed residual open | | `KimiK3ForConditionalGeneration` | Kimi-K3 (2.8T MoE) | scaffold: registry+config+enumeration gated, forward refuses | HW-infeasible (~1.56 TB); no run | -| `NemotronHForCausalLM` | Nemotron-3.5-Lightning-30B-A3B-NVFP4 (`nvidia` @`29f2d174`) | registry+config+enumeration+KV-shape gated; the hybrid Mamba2/GQA/relu2-MoE forward COMPUTES, gated elementwise vs independent double references. No weight loader, so nothing runs end to end (spec #517 W4) | no run; GGUF k-quants refuse by name and are owed | +| `NemotronHForCausalLM` | Nemotron-3.5-Lightning-30B-A3B-NVFP4 (`nvidia` @`29f2d174`) | config+enumeration+KV-shape gated; the hybrid Mamba2/GQA/relu2-MoE forward COMPUTES. The loader materializes 18487/18487 in their SHIPPED formats (5935 NVFP4 W4A16 g16, 46 FP8 W8A8, bf16), 270 MTP deferred to W5 (#517) | CPU host forward returns logits, 17.7 GiB peak RSS; 3/3 first greedy tokens match the oracle goldens, W6 owns the token gate; GGUF owed | | `MuseGlimmerForCausalLM` | real tensors, **bf16 depth 4/52 only**: 5 prefill argmax positions match a torch transcription of vllm#51655 and HF. GGUF full depth generates coherently (#347, #359) but is **NOT token-exact** | text forward + loader vs an fp32 reference, per-mechanism property tests, scaffold 11/11, GGUF gate 17/17. An ABSENT config key now takes the architecture's constant (#412): GGUF post-norms ran at 1e-5, not 1e-8 | no vLLM denominator (pin cannot load it); SECONDARY llama.cpp, same GGUF, GB10 CPU: prefill tie **0.997x**, decode 0.232x, RSS 1.92x (#333) | | `MuseGlimmerForConditionalGeneration` | vision: **no reference run of any kind**; enumeration gated vs the released 30B index (1436/1436). Image/video need bf16 safetensors: `mmproj-kquant.gguf` is refused by name | perception encoder loaded and wired, so an image or video prompt runs; `perception_emb_norm` now armed by default (#405). Reachability plus placeholder scatter only, no image or video correctness | not measurable; anchored to open vllm#51655 | | `LlamaModel` | landed tiny synthetic embedding fixture (engine path == direct pooler path, identical vectors; f64 LAST+normalize reference); real checkpoint (e5-mistral class) is a NAMED residual | pooling/embed only, text paths refuse by task; `vllm_embed` + `/v1/embeddings` | n/a (CPU correctness-grade embeddings) | diff --git a/scripts/runner-routing-allowlist.txt b/scripts/runner-routing-allowlist.txt index b96ab9646..e68a11dc6 100644 --- a/scripts/runner-routing-allowlist.txt +++ b/scripts/runner-routing-allowlist.txt @@ -23,4 +23,4 @@ # --- KNOWN OFF-FRAMEWORK: private resident/graph decode, pending framework-routing --- laguna # LagunaModel::ForwardDevice is a HOST stub (returns HostLogits); resident/graph decode fully off-framework (private f32-host-vector KV, bespoke DecodeAttnGqaKernel, host RoPE, host logit download) -> pending framework-routing (see AGENTS.md decode/runtime seam) qwen3_vl # ForwardQwen3VLForConditionalGeneration returns HostLogits + private VLGenerateCore host ArgMax loop (MM-ENGINE-FORWARD partial seam) bypasses the runner -> pending framework-routing (see AGENTS.md decode/runtime seam) -nemotron_h # W4 (#517) ports the hybrid forward as the HOST CPU reference (nemotron_h.cpp asserts a CPU queue); ForwardNemotronHForCausalLM reaches it through ModelRegistry::Forward and returns HostLogits. It cannot yet return a device-resident ForwardLogits because there is no NemotronH weight LOADER at all — every checkpoint load leaves the host weights unmaterialized and the forward REFUSES BY NAME (spec .agents/specs/nemotron-h-model.md §5b). The device/paged runner path is W6, which is what removes this entry -> pending framework-routing (see AGENTS.md decode/runtime seam) +nemotron_h # W4 (#517) ports the hybrid forward as the HOST CPU reference (nemotron_h.cpp asserts a CPU queue); ForwardNemotronHForCausalLM reaches it through ModelRegistry::Forward and returns HostLogits. NARROWED (spec §6d): the weight loader now materializes all 18487 released tensors and a real checkpoint runs, so "there is no loader" is no longer why this entry exists. What remains is that the forward is the HOST reference — it holds K and V for the whole prompt, pages nothing, and produces host logits, so there is no device-resident ForwardLogits to return. The device/paged runner path is W6, which is what removes this entry -> pending framework-routing (see AGENTS.md decode/runtime seam) diff --git a/src/vllm/model_executor/models/nemotron_h.cpp b/src/vllm/model_executor/models/nemotron_h.cpp index 6f94eeda4..f320e9a66 100644 --- a/src/vllm/model_executor/models/nemotron_h.cpp +++ b/src/vllm/model_executor/models/nemotron_h.cpp @@ -24,6 +24,11 @@ #include #include +// The SHARED ModelOpt dequant seam (DequantNvfp4ToBf16 / DequantFp8ToBf16 / +// kNvfp4GroupSize). The host reference forward has no NVFP4 and no FP8 GEMM, so +// a quantized operand is widened here rather than in a hand-rolled sibling of +// the utility every other ModelOpt consumer in this tree already uses. +#include "vllm/model_executor/model_loader/nvfp4_dequant.h" #include "vt/ops.h" #include "vt/recipes.h" @@ -185,6 +190,77 @@ void RequireWeight(const NemotronHOwned& w, const char* what, DType want, what + "' has the wrong shape"); } +// A DENSE operand for one GEMM, in `want`. +// +// Dense weights are viewed in place — no copy, byte-identical to the W4 path. +// A quantized weight (NVFP4 W4A16 g16 experts / lm_head, FP8 W8A8 static mamba +// projections) is dequantized through the SHARED ModelOpt seam into a TRANSIENT +// buffer that lives exactly as long as this GEMM. +// +// This is the declared host arm, not a silent fallback. The host reference +// forward composes `vt::MatmulBT`, which has no NVFP4 and no FP8 entry point; +// the quantized GEMMs are the CUDA `kMoeGroupedGemmNvfp4Marlin` and fp8-linear +// registrations W6 selects. Two properties keep it honest: the weight KEEPS its +// quantized memory format in host memory (so RSS and the load report describe +// the checkpoint, not a widened copy of it), and the dequant is counted and +// named by the loader's report rather than being discoverable only by reading +// this function. +struct DenseOperand { + std::vector owned; // non-empty only when a dequant happened + Tensor view; +}; + +DenseOperand DenseFor(const NemotronHOwned& w, DType want, vt::Device dev) { + DenseOperand d; + if (w.IsDense()) { + d.view = w.View(dev); + return d; + } + // The shared seam produces bf16; widen losslessly for the f32 reference arm. + d.owned = w.DenseBf16(); + const int64_t n = w.Numel(); + if (want == DType::kF32) { + std::vector wide(static_cast(n) * sizeof(float)); + const auto* src = reinterpret_cast(d.owned.data()); + auto* dst = reinterpret_cast(wide.data()); + for (int64_t i = 0; i < n; ++i) dst[i] = vt::BF16ToF32(src[i]); + d.owned = std::move(wide); + } + d.view.data = d.owned.data(); + d.view.dtype = want; + d.view.device = dev; + d.view.rank = static_cast(w.shape.size()); + VT_CHECK(d.view.rank >= 1 && d.view.rank <= vt::kMaxRank, + "NemotronH forward: dequantized operand rank out of range"); + int64_t stride = 1; + for (int i = d.view.rank - 1; i >= 0; --i) { + d.view.shape[i] = w.shape[static_cast(i)]; + d.view.stride[i] = stride; + stride *= w.shape[static_cast(i)]; + } + return d; +} + +// An OWNED dense `NemotronHOwned` at `want`, so one dequant can serve several +// GEMMs (the expert-major MoE loop). Same seam and same declared-arm reasoning +// as `DenseFor`; a dense input is returned unchanged. +NemotronHOwned DenseCopy(const NemotronHOwned& w, DType want) { + if (w.IsDense()) return w; + NemotronHOwned out; + out.dtype = want; + out.shape = w.shape; + out.bytes = w.DenseBf16(); + if (want == DType::kF32) { + const int64_t n = w.Numel(); + std::vector wide(static_cast(n) * sizeof(float)); + const auto* src = reinterpret_cast(out.bytes.data()); + auto* dst = reinterpret_cast(wide.data()); + for (int64_t i = 0; i < n; ++i) dst[i] = vt::BF16ToF32(src[i]); + out.bytes = std::move(wide); + } + return out; +} + // out[M,N] = a[M,K] @ b^T, b [N,K] — the torch-Linear orientation every weight // above is stored in. Buf Linear(Queue& q, const Buf& a, const NemotronHOwned& w, int64_t M, int64_t K, @@ -192,9 +268,9 @@ Buf Linear(Queue& q, const Buf& a, const NemotronHOwned& w, int64_t M, int64_t K RequireWeight(w, what, a.dtype, {N, K}); Buf out(a.dtype, {M, N}); Tensor at = a.t(q.device, {M, K}); - Tensor wt = w.View(q.device); + const DenseOperand wd = DenseFor(w, a.dtype, q.device); Tensor ot = out.t(q.device); - vt::MatmulBT(q, ot, at, wt); + vt::MatmulBT(q, ot, at, wd.view); return out; } @@ -239,6 +315,16 @@ int64_t NemotronHOwned::Numel() const { } vt::Tensor NemotronHOwned::View(vt::Device device) const { + // A view over packed NVFP4 nibbles or e4m3 bytes, TYPED as the logical dtype, + // reads finite plausible garbage of the right shape — no kernel and no shape + // check can catch it, and a token gate cannot see it. So the dense view + // refuses a quantized weight by name and `DenseFor` is the only way to a GEMM + // operand. + VT_CHECK(form == NemotronHWeightForm::kDense, + "NemotronHOwned::View: this weight is held in its SHIPPED quantized " + "form (NVFP4 W4A16 g16 or FP8 W8A8 static); a dense view of it would " + "reinterpret packed bytes as the model dtype. Materialize it with " + "DenseBf16() at the call site instead."); Tensor t; t.data = const_cast(bytes.data()); t.dtype = dtype; @@ -255,6 +341,43 @@ vt::Tensor NemotronHOwned::View(vt::Device device) const { return t; } +std::vector NemotronHOwned::DenseBf16() const { + VT_CHECK(shape.size() == 2, + "NemotronHOwned::DenseBf16: only a 2-D [out, in] projection is " + "quantized in this architecture"); + const int64_t rows = shape[0]; + const int64_t cols = shape[1]; + std::vector out(static_cast(rows * cols) * sizeof(uint16_t)); + auto* dst = reinterpret_cast(out.data()); + switch (form) { + case NemotronHWeightForm::kNvfp4W4A16G16: + VT_CHECK(bytes.size() == static_cast(rows * cols / 2), + "NemotronHOwned::DenseBf16: NVFP4 payload is not [rows, cols/2]"); + VT_CHECK(scale.size() == static_cast(rows * cols / kNvfp4GroupSize), + "NemotronHOwned::DenseBf16: NVFP4 group scales are not " + "[rows, cols/16]"); + // The SHARED ModelOpt seam, at the DEFAULTED nibble order — ModelOpt and + // compressed-tensors checkpoints are low-first + // (.agents/specs/nvfp4-nibble-order.md), and this is one. + DequantNvfp4ToBf16(bytes.data(), scale.data(), global_scale, rows, cols, + dst); + return out; + case NemotronHWeightForm::kFp8W8A8Static: + VT_CHECK(bytes.size() == static_cast(rows * cols), + "NemotronHOwned::DenseBf16: FP8 payload is not [rows, cols]"); + // Weight-only: `input_scale` is carried, not applied. Nothing on the host + // path quantizes the activation, so applying it here would scale the + // product by a factor upstream applies to the OTHER operand. + DequantFp8ToBf16(bytes.data(), global_scale, rows * cols, dst); + return out; + case NemotronHWeightForm::kDense: + break; + } + VT_CHECK(false, + "NemotronHOwned::DenseBf16: called on a weight that is already dense"); + return out; +} + NemotronHOwned NemotronHOwned::FromF32(const std::vector& values, vt::DType dtype, std::vector shape) { CheckActDType(dtype); @@ -585,16 +708,44 @@ std::vector NemotronHMoeMixer(const NemotronHMoeWeights& w, const Buf x = PackF32(hidden_normed, act_dtype, {T, H}); Buf expert_out(act_dtype, {T, Kk, H}); const size_t esz = vt::SizeOf(act_dtype); + // EXPERT-MAJOR iteration over the (token, slot) pairs. Each pair's own + // `NonGatedExpert` call is unchanged — one row against one expert, so every + // output row's arithmetic is byte-identical to the token-major order this + // replaces — but visiting the pairs grouped by expert lets ONE dequant of a + // quantized expert serve all of that expert's rows. On the released + // checkpoint an expert is an NVFP4 W4A16 g16 pair (~10e6 elements); the + // token-major order re-materializes it per row. + // + // Ordering cannot change the RESULT: each pair writes its own disjoint + // `expert_out` slot and reads nothing another pair wrote. + std::vector> pairs(static_cast(E)); for (int64_t t = 0; t < T; ++t) { for (int64_t j = 0; j < Kk; ++j) { const int32_t e = topk_id[static_cast(t * Kk + j)]; VT_CHECK(e >= 0 && e < E, "NemotronH moe: router emitted an invalid expert id"); + pairs[static_cast(e)].push_back(t * Kk + j); + } + } + for (int64_t e = 0; e < E; ++e) { + const std::vector& slots = pairs[static_cast(e)]; + if (slots.empty()) continue; + const NemotronHExpertWeights& src = w.experts[static_cast(e)]; + // The dense stand-in exists only while this expert is being served, so peak + // host RSS carries ONE dequantized expert, not 128 of them. + NemotronHExpertWeights dense; + const NemotronHExpertWeights* use = &src; + if (!src.up_proj.IsDense() || !src.down_proj.IsDense()) { + dense.up_proj = DenseCopy(src.up_proj, act_dtype); + dense.down_proj = DenseCopy(src.down_proj, act_dtype); + use = &dense; + } + for (int64_t slot : slots) { + const int64_t t = slot / Kk; Buf row(act_dtype, {1, H}); std::memcpy(row.bytes.data(), x.bytes.data() + static_cast(t * H) * esz, static_cast(H) * esz); - const Buf y = NonGatedExpert(queue, row, w.experts[static_cast(e)], 1, H, I, - "mixer.experts"); - std::memcpy(expert_out.bytes.data() + static_cast((t * Kk + j) * H) * esz, + const Buf y = NonGatedExpert(queue, row, *use, 1, H, I, "mixer.experts"); + std::memcpy(expert_out.bytes.data() + static_cast(slot * H) * esz, y.bytes.data(), static_cast(H) * esz); } } diff --git a/src/vllm/model_executor/models/nemotron_h_forward.h b/src/vllm/model_executor/models/nemotron_h_forward.h index 3176d8903..89991bb2a 100644 --- a/src/vllm/model_executor/models/nemotron_h_forward.h +++ b/src/vllm/model_executor/models/nemotron_h_forward.h @@ -79,17 +79,93 @@ inline constexpr bool kNemotronHAttentionHasNoRope = true; // explicitly so a gate can assert the memory format instead of inferring it from // matching tokens — a dtype that is too WIDE is numerically correct and // therefore invisible to a token comparison. +// The MEMORY FORMAT a materialized weight is held in. This is the whole point +// of the enum existing: the released checkpoint is `quant_algo: +// MIXED_PRECISION` and is NOT uniform — routed/shared experts and `lm_head` are +// NVFP4 W4A16 group-16, the 46 mamba `in_proj`/`out_proj` are FP8 W8A8 static, +// and attention, conv1d, gates, norms and embeddings are plain bf16. Holding +// any of those in a WIDER form than it ships is numerically correct, invisible +// to a token gate, and moves the wrong bytes (AGENTS.md). +// +// It is also what makes the checkpoint fit at all. The 5888 routed-expert +// projections alone are 29.4e9 parameters; dequantized to bf16 at load they are +// 58.7 GB, against 16.5 GiB packed. So `kNvfp4W4A16G16` is not an optimization +// — a load that widens it does not run on any box this project owns. +enum class NemotronHWeightForm : uint8_t { + // `bytes` holds Numel() elements of `dtype`, contiguous. Every W4-era weight. + kDense, + // ModelOpt `W4A16_NVFP4`, `group_size=16` (spec §1, config_groups group_1): + // bytes U8 [rows, cols/2] two E2M1 nibbles per byte, LOW nibble + // first (the torchao/ModelOpt convention, + // `Nvfp4NibbleOrder::kLowFirst`) + // scale F8_E4M3 [rows, cols/16] one linear e4m3 scale per 16 inputs + // global_scale F32 scalar `weight_scale_2`, MULTIPLIED (not + // reciprocated) + // `shape`/`dtype` stay LOGICAL: [rows, cols] at the model dtype, which is what + // a dequantized view yields and what every shape check compares against. + kNvfp4W4A16G16, + // ModelOpt FP8 W8A8 static (config_groups group_0, the 46 mamba projections): + // bytes F8_E4M3 [rows, cols] one IEEE e4m3 byte per element + // global_scale F32 scalar `weight_scale`, MULTIPLIED + // input_scale F32 scalar the STATIC activation scale + // The host reference forward is weight-only (W4A16-shaped): it consumes + // `global_scale` and carries `input_scale` without applying it, because + // nothing here quantizes the activation. That is recorded rather than + // silently dropped — W6's device path is where the activation scale is live. + kFp8W8A8Static, +}; + struct NemotronHOwned { + // The payload. Its meaning is `form`'s (see NemotronHWeightForm): dense + // elements of `dtype` for kDense, packed nibbles for NVFP4, e4m3 bytes for + // FP8. std::vector bytes; + // The LOGICAL dtype — what a dense view of this weight yields. Not the + // storage dtype when `form != kDense`. vt::DType dtype = vt::DType::kF32; + // The LOGICAL shape, likewise: [out, in] for every projection, whatever the + // packing. std::vector shape; + NemotronHWeightForm form = NemotronHWeightForm::kDense; + // NVFP4 only: the per-16-element e4m3 group scales, [rows, cols/16] bytes. + std::vector scale; + // NVFP4 `weight_scale_2` / FP8 `weight_scale`. Multiplied, never reciprocated. + float global_scale = 1.0F; + // FP8 W8A8 `input_scale`, carried for the device path (W6). `has_input_scale` + // distinguishes "the checkpoint shipped 1.0" from "no scale shipped", because + // a defaulted 1.0 that silently stands in for a missing tensor is exactly the + // class of load defect a token gate absorbs. + float input_scale = 1.0F; + bool has_input_scale = false; + bool Empty() const { return bytes.empty(); } + bool IsDense() const { return form == NemotronHWeightForm::kDense; } int64_t Numel() const; + // Bytes actually resident for this weight, payload + scales. The number a + // load report adds up; not derivable from `shape` once `form != kDense`. + int64_t HostBytes() const { + return static_cast(bytes.size() + scale.size()); + } // A non-owning contiguous view over the current buffer. Rebuilt on each call - // so it survives moves/reallocations of the owning struct. + // so it survives moves/reallocations of the owning struct. DENSE ONLY: a view + // over packed nibbles typed as bf16 reads plausible garbage, so this refuses + // rather than handing one out. vt::Tensor View(vt::Device device) const; + // Materialize a DENSE bf16 copy of a quantized weight, through the SHARED + // ModelOpt dequant seam (`model_loader/nvfp4_dequant.h`), which is the one + // this tree's other ModelOpt consumers use. bf16 is the target because that is + // what the shared seam produces AND the released checkpoint's model dtype; a + // caller running the f32 reference arm widens the result losslessly. + // + // This exists because the HOST reference forward has NO NVFP4 and NO FP8 GEMM + // — those are the CUDA `kMoeGroupedGemmNvfp4Marlin` / fp8-linear arms W6 + // selects. It is a DECLARED host dequant, counted and reported by the loader, + // never a silent fallback: the weight keeps its quantized form in memory, and + // only the GEMM operand is widened, transiently, at the call site. + std::vector DenseBf16() const; + // Pack canonical f32 values into `dtype`. This is the seam a real loader // replaces: it hands over checkpoint bytes that are ALREADY in the model // dtype, and no packing happens per forward. @@ -127,6 +203,16 @@ struct NemotronHAttentionWeights { NemotronHOwned k_proj; // [num_key_value_heads*head_dim, hidden_size] NemotronHOwned v_proj; // [num_key_value_heads*head_dim, hidden_size] NemotronHOwned o_proj; // [hidden_size, num_attention_heads*head_dim] + // The fp8 KV-cache scales the checkpoint ships as `k_proj.k_scale` / + // `v_proj.v_scale` (`quantization_config.kv_cache_scheme`, num_bits 8, type + // float). MATERIALIZED but UNUSED on this path: the host reference forward + // holds K and V in the model dtype for the whole prompt and pages nothing, so + // there is no fp8 KV store to scale. W6's paged device path is where they + // become live. They are loaded rather than skipped so the tensor accounting + // is honest — 12 tensors dropped on the floor is 12 tensors nobody notices. + float k_scale = 1.0F; + float v_scale = 1.0F; + bool has_kv_scales = false; }; // One NON-GATED expert: `ckpt_names=("up_proj","down_proj","")` diff --git a/src/vllm/model_executor/models/nemotron_h_loader.h b/src/vllm/model_executor/models/nemotron_h_loader.h new file mode 100644 index 000000000..3aedbdc02 --- /dev/null +++ b/src/vllm/model_executor/models/nemotron_h_loader.h @@ -0,0 +1,132 @@ +// Nemotron-H (`NemotronHForCausalLM`) — the WEIGHT LOADER (implemented in +// `nemotron_h_weights.cpp`, beside the enumeration it must agree with) +// ([spec](../../../../.agents/specs/nemotron-h-model.md) §5b, §6b "Still owed +// after W4"; issue #517). +// +// W3 enumerated all 18487 released tensors and W4 made the architecture +// compute. Neither MATERIALIZED anything, so every checkpoint load left +// `NemotronHHostWeights::materialized` false and the forward refused by name. +// This is that brick. +// +// ─── THE CHECKPOINT IS NOT UNIFORM, AND THAT IS THE WHOLE JOB ──────────────── +// +// `quant_algo: MIXED_PRECISION`, `quant_method: modelopt`. One file carries +// three memory formats, and the repo name ("...-NVFP4") describes only the +// biggest of them: +// +// routed experts, shared experts, lm_head NVFP4 W4A16, group_size=16 +// .weight U8 [out, in/2] two E2M1 nibbles per byte +// .weight_scale F8_E4M3 [out, in/16] one linear e4m3 per 16 inputs +// .weight_scale_2 F32 scalar the global scale, MULTIPLIED +// +// mamba in_proj / out_proj (46 targets) FP8 W8A8 static +// .weight F8_E4M3 [out, in] +// .weight_scale F32 scalar +// .input_scale F32 [1] the STATIC activation scale +// +// attention q/k/v/o, conv1d, gates, norms, embeddings plain bf16 / f32 +// KV cache fp8: k_proj.k_scale, v_proj.v_scale +// +// Reading it as UNIFORM NVFP4 is the failure a token gate cannot see: the +// answer stays numerically plausible, the tokens still match, and the load +// moves the wrong bytes. So every weight is held in the format it SHIPS in, and +// the load report states the composition as hard numbers rather than leaving it +// to be inferred. +// +// ─── WHY THE QUANTIZED FORMS ARE KEPT, NOT DEQUANTIZED AT LOAD ─────────────── +// +// Two reasons, and the first one is arithmetic. The 5888 routed-expert +// projections are 29.4e9 parameters: 16.5 GiB packed, 58.7 GB at bf16. A +// dequantize-at-load loader does not fit on any box this project owns, and on a +// unified-memory box it takes the machine down rather than failing. The second +// is the rule: a widened weight is numerically correct and invisible to a token +// comparison while moving twice the bytes (AGENTS.md). +// +// The host reference forward has no NVFP4 and no FP8 GEMM — those are the CUDA +// `kMoeGroupedGemmNvfp4Marlin` and fp8-linear arms W6 selects. It therefore +// widens a quantized operand TRANSIENTLY at the GEMM call site, through the +// shared `model_loader/nvfp4_dequant.h` seam, and the report below names and +// counts that arm so it is a declared property of this path rather than +// something a reader has to find. +// +// ─── WHAT IS DEFERRED, BY NAME ─────────────────────────────────────────────── +// +// The 270-tensor MTP tower (`mtp.layers.*`, unquantized bf16 because the +// `ignore` list carries `mtp*`) is W5 and is NOT materialized. It is counted and +// named in the report rather than skipped silently: 270 tensors dropped on the +// floor is 270 tensors nobody notices. The GGUF arm (W7) refuses earlier, in +// the registry TU. +#pragma once + +#include +#include +#include + +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/nemotron_h.h" +#include "vllm/model_executor/models/nemotron_h_forward.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vt/dtype.h" + +namespace vllm { + +// What the load actually did, in numbers a gate can assert. Every count is over +// TENSORS as the checkpoint ships them, so `materialized + deferred` is the +// released tensor count and nothing hides in a rounding. +struct NemotronHLoadReport { + int64_t enumerated = 0; // what EnumerateNemotronHTensors named + int64_t in_index = 0; // what model.safetensors.index.json ships + int64_t materialized = 0; // read into a named host slot + int64_t deferred = 0; // named, owed to a later W, deliberately not read + + // Composition by the SHIPPED scheme. `*_weights` counts logical projections, + // `*_tensors` the on-disk tensors they are made of (three each). + int64_t nvfp4_weights = 0; + int64_t nvfp4_tensors = 0; + int64_t fp8_weights = 0; + int64_t fp8_tensors = 0; + // The fp8 KV scheme's own tensors (`k_proj.k_scale`, `v_proj.v_scale`), which + // belong to no projection and are their own row of the scheme table. + int64_t fp8_kv_scale_tensors = 0; + // The unquantized remainder, by the dtype it ships in. + int64_t bf16_tensors = 0; + int64_t f32_tensors = 0; + // Tensors whose HOST dtype is wider than their DISK dtype. On the released + // checkpoint this is exactly the 69 SSM scalars (`A_log`, `D`, `dt_bias` on 23 + // mamba layers), bf16 on disk and f32 in memory because upstream keeps them + // f32 (`self.A = -torch.exp(self.A_log.float())`) and `vt::Mamba2ChunkScan` + // validates them f32. Counted so the widening cannot spread unnoticed. + int64_t widened_tensors = 0; + + int64_t source_bytes = 0; // bytes read out of the safetensors mappings + int64_t host_bytes = 0; // bytes resident in NemotronHHostWeights afterwards + + // The distinct consumer tags that were deferred, and the W that owns them. + std::vector deferred_by_name; +}; + +// The model dtype, mirroring transformers' own resolution: newer versions +// serialize `dtype` and the legacy spelling is `torch_dtype`. Refuses anything +// this forward cannot represent BY NAME rather than substituting a dtype. +vt::DType ResolveNemotronHModelDType(const HfConfig& config); + +// Materialize `NemotronHHostWeights` from an already-opened safetensors +// checkpoint. `shards` is the shared `ModelSource` seam every other +// architecture's loader consumes; source pages are released as each tensor is +// copied out (`MaybeReleaseSourcePages`), so peak RSS tracks the owned mirror +// rather than mirror-plus-mapping. +// +// Throws (refuses by name) on: a tensor the enumeration names and the +// checkpoint does not ship, a tensor the checkpoint ships and the enumeration +// does not name, a dtype or shape that disagrees with the scheme its consumer +// declares, or a missing scale companion. +NemotronHHostWeights LoadNemotronHHostWeights( + const std::vector& shards, const NemotronHParams& params, + vt::DType act_dtype, NemotronHLoadReport* report); + +// The report of the load that produced `model`. The load happens inside the +// type-erased `ModelRegistry::Load` factory, so a structural gate has no other +// way to reach the numbers. Throws if `model` is not a NemotronH model. +const NemotronHLoadReport& NemotronHLoadReportOf(const LoadedModel& model); + +} // namespace vllm diff --git a/src/vllm/model_executor/models/nemotron_h_registry.cpp b/src/vllm/model_executor/models/nemotron_h_registry.cpp index c1476d247..5eab704fa 100644 --- a/src/vllm/model_executor/models/nemotron_h_registry.cpp +++ b/src/vllm/model_executor/models/nemotron_h_registry.cpp @@ -26,6 +26,7 @@ #include "vllm/model_executor/models/nemotron_h.h" #include "vllm/model_executor/models/nemotron_h_forward.h" +#include "vllm/model_executor/models/nemotron_h_loader.h" #include "vllm/model_executor/models/qwen3_5.h" // ForwardLogits #include "vllm/model_executor/models/qwen3_5_common.h" // HostLogits #include "vllm/v1/kv_cache_dtype.h" @@ -64,10 +65,17 @@ class NemotronHLoadedModel final : public LoadedModel { // (the unit gate) constructs the weights itself and reaches the same forward. NemotronHHostWeights& weights() { return weights_; } const NemotronHHostWeights& weights() const { return weights_; } + // What the load did, in numbers. Kept on the model rather than returned by + // value because the load happens inside the type-erased registry factory and + // a gate has no other way to reach it; `NemotronHLoadReportOf` is the + // accessor. + NemotronHLoadReport& report() { return report_; } + const NemotronHLoadReport& report() const { return report_; } private: NemotronHParams params_; NemotronHHostWeights weights_; + NemotronHLoadReport report_; }; std::unique_ptr LoadNemotronHForCausalLM( @@ -83,11 +91,23 @@ std::unique_ptr LoadNemotronHForCausalLM( ".agents/specs/nemotron-h-model.md §5b W7)"); } // The config descent IS the validation, and it refuses by name on anything - // this bring-up cannot represent. W4 owns materializing the tensors - // EnumerateNemotronHTensors names. + // this bring-up cannot represent. NemotronHParams params = ParseNemotronHParams(config); - return std::make_unique(registration, - std::move(params)); + auto model = + std::make_unique(registration, params); + if (source.safetensors == nullptr) { + throw std::runtime_error( + "Model architecture NemotronHForCausalLM: the safetensors source " + "carries no shards"); + } + // MATERIALIZE. The 18487 enumerated tensors are read into the host weights in + // the memory format the checkpoint SHIPS them in — NVFP4 W4A16 g16 experts and + // lm_head, FP8 W8A8 static mamba projections, bf16 everything else — and the + // MTP tower is deferred by name (W5). See nemotron_h_loader.h. + model->weights() = LoadNemotronHHostWeights( + *source.safetensors, params, ResolveNemotronHModelDType(config), + &model->report()); + return model; } void PrepareNemotronHForCausalLM(LoadedModel& model, const HfConfig& config, @@ -123,6 +143,15 @@ const ModelFactory kNemotronHFactory{ } // namespace +const NemotronHLoadReport& NemotronHLoadReportOf(const LoadedModel& model) { + const auto* nh = dynamic_cast(&model); + if (nh == nullptr) { + throw std::runtime_error( + "NemotronHLoadReportOf: this LoadedModel is not a NemotronH model"); + } + return nh->report(); +} + v1::KVCacheConfig MakeNemotronHKVCache(const HfConfig& config, int block_size, int num_blocks) { const NemotronHParams p = ParseNemotronHParams(config); diff --git a/src/vllm/model_executor/models/nemotron_h_weights.cpp b/src/vllm/model_executor/models/nemotron_h_weights.cpp index ae8a9d81b..72dc020b3 100644 --- a/src/vllm/model_executor/models/nemotron_h_weights.cpp +++ b/src/vllm/model_executor/models/nemotron_h_weights.cpp @@ -1,14 +1,35 @@ -// Nemotron-H W3: config descent + the on-disk weight name map. See nemotron_h.h -// for the port anchors and the scope boundary against W1/W4. +// Nemotron-H: config descent, the on-disk weight name map, and the WEIGHT LOAD +// that materializes it. See nemotron_h.h for the port anchors and +// nemotron_h_loader.h for the MIXED_PRECISION scheme table, why each weight is +// held in the format it SHIPS in, and what is deferred by name. +// +// The loader lives in this TU rather than a `nemotron_h_loader.cpp` of its own +// for two reasons. It is where every other architecture puts its weight load — +// `llama_weights.cpp`, `laguna_weights.cpp`, `deepseek_v2_weights.cpp` are all +// loaders — and the enumeration it must agree with tensor-for-tensor is right +// here, so the two cannot drift into separate files with separate ideas of what +// the checkpoint ships. A new TU would also have meant editing the top-level +// `CMakeLists.txt`, which `check-doc-checkpoint.py:83-93` classifies as a +// user-facing build entrypoint owing a `docs/USAGE.md` change; nothing here is +// user-facing yet (the forward is the host reference, W6 owns the runner path), +// so that edit would have documented nothing. #include "vllm/model_executor/models/nemotron_h.h" +#include "vllm/model_executor/models/nemotron_h_loader.h" #include +#include +#include +#include +#include #include #include #include +#include #include +#include "vllm/model_executor/model_loader/nvfp4_dequant.h" + namespace vllm { namespace { @@ -340,6 +361,337 @@ void ClaimMlp(std::vector& out, const NemotronHParams& p, } } +// ─── the WEIGHT LOADER (see nemotron_h_loader.h) ───────────────────────────── + + +// The loader's own refusal. Distinct from the config descent's `Refuse` +// above, and deliberately so: the two name different phases, and a message +// reading "NemotronHForCausalLM:" for a TENSOR problem would send a reader to +// the config. +[[noreturn]] void RefuseLoad(const std::string& detail) { + throw std::runtime_error("NemotronHForCausalLM weight load: " + detail); +} + +// One tensor's on-disk view, keyed by the name the checkpoint ships. +using TensorIndex = std::map; + +int64_t Numel(const std::vector& shape) { + int64_t n = 1; + for (int64_t d : shape) n *= d; + return n; +} + +std::string ShapeStr(const std::vector& shape) { + std::string s = "["; + for (size_t i = 0; i < shape.size(); ++i) { + if (i != 0) s += ", "; + s += std::to_string(shape[i]); + } + return s + "]"; +} + +// The loader's accumulating state: the index it reads from, the names it has +// consumed (so the accounting is BOTH ways), and the report it fills. +struct Loader { + const TensorIndex& index; + NemotronHLoadReport& report; + std::set consumed; + + const StTensor& Need(const std::string& name) { + const auto it = index.find(name); + if (it == index.end()) { + RefuseLoad("the checkpoint does not ship '" + name + + "', which this architecture's enumeration names"); + } + if (!consumed.insert(name).second) { + RefuseLoad("'" + name + "' was claimed twice"); + } + report.source_bytes += static_cast(it->second->nbytes); + return *it->second; + } + + void Expect(const StTensor& t, const std::string& name, const char* dtype, + const std::vector& shape) { + if (t.dtype != dtype) { + RefuseLoad("'" + name + "' ships dtype " + t.dtype + ", not the " + dtype + + " its scheme declares"); + } + if (t.shape != shape) { + RefuseLoad("'" + name + "' ships shape " + ShapeStr(t.shape) + ", not " + + ShapeStr(shape)); + } + } +}; + +// Release the source pages of a range the loader has finished with. The +// mappings stay open (the caller owns them), so without this the whole 20.1 GiB +// checkpoint stays resident alongside the owned mirror. +void Consumed(const StTensor& t) { MaybeReleaseSourcePages(t.data, t.nbytes); } + +// ─── dense copies ─────────────────────────────────────────────────────────── + +// Copy a plain (unquantized) tensor into `want`. `disk_shape` is what the +// checkpoint ships; `logical_shape` is what the forward indexes it as — they +// differ only for `conv1d.weight`, which ships [Cd, 1, K] and is consumed as the +// squeezed [Cd, K] (mamba_mixer2.py's `self.conv1d.weight.view(conv_dim, K)`). +NemotronHOwned CopyDense(Loader& ld, const std::string& name, vt::DType want, + const std::vector& disk_shape, + std::vector logical_shape) { + const StTensor& t = ld.Need(name); + if (t.shape != disk_shape) { + RefuseLoad("'" + name + "' ships shape " + ShapeStr(t.shape) + ", not " + + ShapeStr(disk_shape)); + } + const int64_t n = Numel(disk_shape); + NemotronHOwned w; + w.dtype = want; + w.shape = std::move(logical_shape); + w.bytes.assign(static_cast(n) * vt::SizeOf(want), 0); + if (t.dtype == "BF16") { + ld.report.bf16_tensors += 1; + if (t.nbytes != static_cast(n) * 2) { + RefuseLoad("'" + name + "' is BF16 but its byte count does not match its shape"); + } + const auto* src = reinterpret_cast(t.data); + if (want == vt::DType::kBF16) { + std::memcpy(w.bytes.data(), src, static_cast(n) * 2); + } else if (want == vt::DType::kF32) { + // A WIDENING. Counted, because the only ones the released checkpoint asks + // for are the three f32-by-contract SSM scalars; a widening anywhere else + // is a defect that a token gate would absorb. + ld.report.widened_tensors += 1; + auto* dst = reinterpret_cast(w.bytes.data()); + for (int64_t i = 0; i < n; ++i) dst[i] = vt::BF16ToF32(src[i]); + } else { + RefuseLoad("'" + name + "' cannot be materialized at the requested dtype"); + } + } else if (t.dtype == "F32") { + ld.report.f32_tensors += 1; + if (t.nbytes != static_cast(n) * 4) { + RefuseLoad("'" + name + "' is F32 but its byte count does not match its shape"); + } + const auto* src = reinterpret_cast(t.data); + if (want == vt::DType::kF32) { + std::memcpy(w.bytes.data(), src, static_cast(n) * 4); + } else if (want == vt::DType::kBF16) { + // A NARROWING, which is what a bf16 model dtype asks for on a tensor the + // producer happened to store wide. Never silent: it is the model dtype + // every other layer inherits, and the router (the one f32 consumer) asks + // for f32 explicitly. + auto* dst = reinterpret_cast(w.bytes.data()); + for (int64_t i = 0; i < n; ++i) dst[i] = vt::F32ToBF16(src[i]); + } else { + RefuseLoad("'" + name + "' cannot be materialized at the requested dtype"); + } + } else { + RefuseLoad("'" + name + "' ships dtype " + t.dtype + + ", which is not an unquantized dtype this loader reads"); + } + Consumed(t); + return w; +} + +NemotronHOwned CopyDense(Loader& ld, const std::string& name, vt::DType want, + const std::vector& shape) { + return CopyDense(ld, name, want, shape, shape); +} + +// A per-tensor f32 scalar companion. ModelOpt writes `weight_scale_2` and +// `weight_scale` as rank-0 and `input_scale` / `k_scale` / `v_scale` as [1]; +// both spellings are one number and both are accepted, nothing else is. +float ReadF32Scalar(Loader& ld, const std::string& name) { + const StTensor& t = ld.Need(name); + if (t.dtype != "F32") { + RefuseLoad("'" + name + "' ships dtype " + t.dtype + ", not the F32 a scale is"); + } + if (!(t.shape.empty() || t.shape == std::vector{1})) { + RefuseLoad("'" + name + "' ships shape " + ShapeStr(t.shape) + + ", not the scalar a per-tensor scale is"); + } + if (t.nbytes != sizeof(float)) { + RefuseLoad("'" + name + "' is a scalar F32 but does not carry 4 bytes"); + } + float v = 0.0F; + std::memcpy(&v, t.data, sizeof(float)); + Consumed(t); + return v; +} + +// ─── the two quantized schemes ────────────────────────────────────────────── + +// `W4A16_NVFP4`, `group_size=16`. `rows`/`cols` are the LOGICAL [out, in]. +NemotronHOwned LoadNvfp4(Loader& ld, const std::string& prefix, vt::DType logical, + int64_t rows, int64_t cols) { + if (cols % kNvfp4GroupSize != 0) { + RefuseLoad("'" + prefix + "' has in_features " + std::to_string(cols) + + ", which is not a multiple of the NVFP4 group size 16"); + } + NemotronHOwned w; + w.form = NemotronHWeightForm::kNvfp4W4A16G16; + w.dtype = logical; + w.shape = {rows, cols}; + + const std::string wname = prefix + ".weight"; + const StTensor& packed = ld.Need(wname); + ld.Expect(packed, wname, "U8", {rows, cols / 2}); + w.bytes.assign(packed.data, packed.data + packed.nbytes); + Consumed(packed); + + const std::string sname = prefix + ".weight_scale"; + const StTensor& gs = ld.Need(sname); + // The GROUP scale, not a per-tensor one. Binding a per-tensor scale here (or + // this one to the wrong projection) produces a finite, correctly-shaped, + // wrongly-scaled matrix — the x1.10-class error a token gate absorbs. + ld.Expect(gs, sname, "F8_E4M3", {rows, cols / kNvfp4GroupSize}); + w.scale.assign(gs.data, gs.data + gs.nbytes); + Consumed(gs); + + w.global_scale = ReadF32Scalar(ld, prefix + ".weight_scale_2"); + + ld.report.nvfp4_weights += 1; + ld.report.nvfp4_tensors += 3; + return w; +} + +// FP8 W8A8 static. Weight-only on this path: `input_scale` is carried, not +// applied (nemotron_h_forward.h records why). +NemotronHOwned LoadFp8(Loader& ld, const std::string& prefix, vt::DType logical, + int64_t rows, int64_t cols) { + NemotronHOwned w; + w.form = NemotronHWeightForm::kFp8W8A8Static; + w.dtype = logical; + w.shape = {rows, cols}; + + const std::string wname = prefix + ".weight"; + const StTensor& q = ld.Need(wname); + ld.Expect(q, wname, "F8_E4M3", {rows, cols}); + w.bytes.assign(q.data, q.data + q.nbytes); + Consumed(q); + + w.global_scale = ReadF32Scalar(ld, prefix + ".weight_scale"); + w.input_scale = ReadF32Scalar(ld, prefix + ".input_scale"); + w.has_input_scale = true; + + ld.report.fp8_weights += 1; + ld.report.fp8_tensors += 3; + return w; +} + +// ─── the blocks ───────────────────────────────────────────────────────────── + +void LoadMamba(Loader& ld, const NemotronHParams& p, const std::string& mixer, + vt::DType adt, NemotronHMambaWeights& out) { + const int64_t H = p.hidden_size; + const int64_t I = p.mamba_intermediate_size(); + const int64_t Cd = p.conv_dim(); + const int64_t K = p.conv_kernel; + const int64_t Hh = p.mamba_num_heads; + + out.in_proj = LoadFp8(ld, mixer + ".in_proj", adt, p.in_proj_out_features(), H); + out.out_proj = LoadFp8(ld, mixer + ".out_proj", adt, H, I); + // The conv weight ships [Cd, 1, K] and is consumed squeezed. + out.conv1d_weight = + CopyDense(ld, mixer + ".conv1d.weight", adt, {Cd, 1, K}, {Cd, K}); + if (p.use_conv_bias) { + out.conv1d_bias = CopyDense(ld, mixer + ".conv1d.bias", adt, {Cd}); + } + // f32 BY CONTRACT, and bf16 on disk. Upstream keeps these f32 whatever the + // model dtype (`self.A = -torch.exp(self.A_log.float())`, and D/dt_bias feed + // the same f32 scan); `vt::Mamba2ChunkScan` validates all three as f32. This + // is the annotated f32 escape AGENTS.md allows, it is upstream's own polarity, + // and `report.widened_tensors` counts it so it stays exactly these three. + out.A_log = CopyDense(ld, mixer + ".A_log", vt::DType::kF32, {Hh}); + out.D = CopyDense(ld, mixer + ".D", vt::DType::kF32, {Hh}); + out.dt_bias = CopyDense(ld, mixer + ".dt_bias", vt::DType::kF32, {Hh}); + // Mixer2RMSNormGated over the SSM intermediate width, NOT hidden_size. + out.norm_weight = CopyDense(ld, mixer + ".norm.weight", adt, {I}); +} + +void LoadAttention(Loader& ld, const NemotronHParams& p, const std::string& mixer, + vt::DType adt, bool fp8_kv, NemotronHAttentionWeights& out) { + const int64_t H = p.hidden_size; + const int64_t qd = p.q_proj_out_features(); + const int64_t kvd = p.kv_proj_out_features(); + out.q_proj = CopyDense(ld, mixer + ".q_proj.weight", adt, {qd, H}); + out.k_proj = CopyDense(ld, mixer + ".k_proj.weight", adt, {kvd, H}); + out.v_proj = CopyDense(ld, mixer + ".v_proj.weight", adt, {kvd, H}); + out.o_proj = CopyDense(ld, mixer + ".o_proj.weight", adt, {H, qd}); + if (fp8_kv) { + out.k_scale = ReadF32Scalar(ld, mixer + ".k_proj.k_scale"); + out.v_scale = ReadF32Scalar(ld, mixer + ".v_proj.v_scale"); + out.has_kv_scales = true; + ld.report.fp8_kv_scale_tensors += 2; + } +} + +void LoadExpert(Loader& ld, const std::string& prefix, vt::DType adt, int64_t H, + int64_t I, bool quantized, NemotronHExpertWeights& out) { + // `ckpt_names=("up_proj","down_proj","")` (nemotron_h.py:220): there is no + // gate_proj anywhere in this checkpoint. + if (quantized) { + out.up_proj = LoadNvfp4(ld, prefix + ".up_proj", adt, I, H); + out.down_proj = LoadNvfp4(ld, prefix + ".down_proj", adt, H, I); + } else { + out.up_proj = CopyDense(ld, prefix + ".up_proj.weight", adt, {I, H}); + out.down_proj = CopyDense(ld, prefix + ".down_proj.weight", adt, {H, I}); + } +} + +void LoadMoe(Loader& ld, const NemotronHParams& p, const std::string& mixer, + vt::DType adt, bool quantized, NemotronHMoeWeights& out) { + const int64_t H = p.hidden_size; + const int64_t E = p.n_routed_experts; + // The ROUTER IS F32 AND THAT IS MIRRORED, NOT INHERITED: + // `GateLinear(..., out_dtype=torch.float32, force_fp32_compute=True)` + // (nemotron_h.py:150-156). The released backbone even ships it F32 on disk; + // the MTP tower's twin ships BF16, which is exactly why the dtype is REQUESTED + // here rather than taken from whatever the producer wrote. + out.gate = CopyDense(ld, mixer + ".gate.weight", vt::DType::kF32, {E, H}); + out.e_score_correction_bias = CopyDense( + ld, mixer + ".gate.e_score_correction_bias", vt::DType::kF32, {E}); + out.experts.resize(static_cast(E)); + for (int64_t e = 0; e < E; ++e) { + LoadExpert(ld, mixer + ".experts." + std::to_string(e), adt, H, + p.moe_intermediate_size, quantized, + out.experts[static_cast(e)]); + } + if (p.n_shared_experts > 0) { + LoadExpert(ld, mixer + ".shared_experts", adt, H, + p.moe_shared_expert_intermediate_size * p.n_shared_experts, + quantized, out.shared); + out.has_shared = true; + } +} + +void LoadMlp(Loader& ld, const NemotronHParams& p, const std::string& mixer, + vt::DType adt, bool quantized, NemotronHMlpWeights& out) { + NemotronHExpertWeights e; + LoadExpert(ld, mixer, adt, p.hidden_size, p.intermediate_size, quantized, e); + out.up_proj = std::move(e.up_proj); + out.down_proj = std::move(e.down_proj); +} + +int64_t HostBytesOf(const NemotronHHostWeights& h) { + int64_t n = h.embeddings.HostBytes() + h.norm_f.HostBytes() + + h.lm_head.HostBytes(); + for (const NemotronHLayerWeights& l : h.layers) { + n += l.norm.HostBytes(); + n += l.mamba.in_proj.HostBytes() + l.mamba.out_proj.HostBytes() + + l.mamba.conv1d_weight.HostBytes() + l.mamba.conv1d_bias.HostBytes() + + l.mamba.A_log.HostBytes() + l.mamba.D.HostBytes() + + l.mamba.dt_bias.HostBytes() + l.mamba.norm_weight.HostBytes(); + n += l.attn.q_proj.HostBytes() + l.attn.k_proj.HostBytes() + + l.attn.v_proj.HostBytes() + l.attn.o_proj.HostBytes(); + n += l.moe.gate.HostBytes() + l.moe.e_score_correction_bias.HostBytes(); + for (const NemotronHExpertWeights& e : l.moe.experts) { + n += e.up_proj.HostBytes() + e.down_proj.HostBytes(); + } + n += l.moe.shared.up_proj.HostBytes() + l.moe.shared.down_proj.HostBytes(); + n += l.mlp.up_proj.HostBytes() + l.mlp.down_proj.HostBytes(); + } + return n; +} + } // namespace std::string_view NemotronHBlockName(NemotronHBlock block) { @@ -375,7 +727,7 @@ NemotronHParams ParseNemotronHParams(const HfConfig& config) { {NemotronHBlock::kMamba, NemotronHBlock::kMoe, NemotronHBlock::kAttention, NemotronHBlock::kMlp}); if (p.layers_block_type.empty()) { - Refuse("layers_block_type resolved to an empty schedule"); + RefuseLoad("layers_block_type resolved to an empty schedule"); } p.num_nextn_predict_layers = GetInt(doc, "num_nextn_predict_layers", 0); p.mtp_layers_block_type = @@ -385,7 +737,7 @@ NemotronHParams ParseNemotronHParams(const HfConfig& config) { {NemotronHBlock::kAttention, NemotronHBlock::kMoe}); if (p.num_nextn_predict_layers > 0 && p.mtp_layers_block_type.empty()) { // Mirror of validate_layer_type (configuration_nemotron_h.py:206-212). - Refuse( + RefuseLoad( "mtp_layers_block_type is required when num_nextn_predict_layers > 0"); } @@ -436,7 +788,7 @@ NemotronHParams ParseNemotronHParams(const HfConfig& config) { GetDoubleAliased(doc, "time_step_floor", "mamba_dt_init_floor", 1e-4); if (p.mamba_num_heads <= 0 || p.mamba_head_dim <= 0 || p.n_groups <= 0 || p.ssm_state_size <= 0 || p.conv_kernel <= 1) { - Refuse("the Mamba2 geometry is degenerate (mamba_num_heads, mamba_head_dim, " + RefuseLoad("the Mamba2 geometry is degenerate (mamba_num_heads, mamba_head_dim, " "n_groups, ssm_state_size must be positive and conv_kernel > 1)"); } @@ -458,12 +810,12 @@ NemotronHParams ParseNemotronHParams(const HfConfig& config) { // The `fc1_latent_proj`/`fc2_latent_proj` pair (nemotron_h.py:191-207) is // out of scope for this row (spec §0), and silently ignoring the key would // build a differently-shaped MoE with no error. - Refuse("moe_latent_size is set, but the latent MoE " + RefuseLoad("moe_latent_size is set, but the latent MoE " "(fc1_latent_proj/fc2_latent_proj) is out of scope for this row " "(see .agents/specs/nemotron-h-model.md §0)"); } if (p.mlp_hidden_act != "relu2") { - Refuse("mlp_hidden_act '" + p.mlp_hidden_act + + RefuseLoad("mlp_hidden_act '" + p.mlp_hidden_act + "' is not implemented (this architecture is the non-gated relu2 " "expert; see .agents/specs/nemotron-h-model.md W2)"); } @@ -472,7 +824,7 @@ NemotronHParams ParseNemotronHParams(const HfConfig& config) { if (doc.contains("intermediate_size") && doc.at("intermediate_size").is_array()) { // `get_nemotron_h_config_for_layer` / NemotronHPuzzleForCausalLM // (nemotron_h.py:283-288) is explicitly out of scope (spec §0). - Refuse("a per-layer intermediate_size list (NemotronHPuzzleForCausalLM) is " + RefuseLoad("a per-layer intermediate_size list (NemotronHPuzzleForCausalLM) is " "out of scope for this row (see .agents/specs/nemotron-h-model.md §0)"); } p.intermediate_size = GetInt(doc, "intermediate_size", 21504); @@ -484,7 +836,7 @@ NemotronHParams ParseNemotronHParams(const HfConfig& config) { const std::string& ssm = p.mamba_ssm_cache_dtype; if (!(ssm.empty() || ssm == "auto" || ssm == "float32" || ssm == "float" || ssm == "float16" || ssm == "half" || ssm == "bfloat16")) { - Refuse("mamba_ssm_cache_dtype '" + ssm + + RefuseLoad("mamba_ssm_cache_dtype '" + ssm + "' is not supported (expected auto, float32/float, float16/half or " "bfloat16)"); } @@ -516,7 +868,7 @@ vt::DType NemotronHSsmCacheDType(const NemotronHParams& params, if (dtype == "bfloat16") return vt::DType::kBF16; // ParseNemotronHParams already refused anything else by name; this is the // unreachable arm kept so the mapping cannot silently widen. - Refuse("mamba_ssm_cache_dtype '" + dtype + "' is not supported"); + RefuseLoad("mamba_ssm_cache_dtype '" + dtype + "' is not supported"); } std::vector EnumerateNemotronHTensors( @@ -600,4 +952,135 @@ std::vector EnumerateNemotronHTensors( return out; } + +vt::DType ResolveNemotronHModelDType(const HfConfig& config) { + // transformers serializes the model dtype as `dtype` (the released + // NemotronH config.json ships `"dtype": "bfloat16"`); `torch_dtype` is the + // legacy spelling and is the fallback, not the other way round. HfConfig + // parses only the legacy key, so the modern one is read from the raw + // document here. + std::string name; + if (config.raw.contains("dtype") && config.raw.at("dtype").is_string()) { + name = config.raw.at("dtype").get(); + } + if (name.empty()) name = config.torch_dtype; + if (name.empty() || name == "auto") name = "bfloat16"; + if (name == "bfloat16") return vt::DType::kBF16; + if (name == "float32" || name == "float") return vt::DType::kF32; + // f16 is deliberately refused rather than substituted: `vt::MoeRelu2` has no + // f16 output arm (spec §6a), so a silent widen-to-bf16 would run a different + // model than the checkpoint declares. + throw std::runtime_error( + "NemotronHForCausalLM weight load: model dtype '" + name + + "' is not supported (this forward composes ops whose output dtypes are " + "bf16 — the released checkpoint's — and f32)"); +} + +NemotronHHostWeights LoadNemotronHHostWeights( + const std::vector& shards, const NemotronHParams& params, + vt::DType act_dtype, NemotronHLoadReport* report) { + NemotronHLoadReport local; + NemotronHLoadReport& rep = report != nullptr ? *report : local; + rep = NemotronHLoadReport{}; + + TensorIndex index; + for (const SafetensorsFile& shard : shards) { + for (const std::string& name : shard.Names()) { + if (!index.emplace(name, &shard.Get(name)).second) { + RefuseLoad("'" + name + "' appears in more than one shard"); + } + } + } + rep.in_index = static_cast(index.size()); + + const std::vector enumerated = + EnumerateNemotronHTensors(params); + rep.enumerated = static_cast(enumerated.size()); + + NemotronHHostWeights host; + host.act_dtype = act_dtype; + const NemotronHParams& p = params; + const bool quantized = p.quant.present; + + Loader ld{index, rep, {}}; + + // --- root --- + host.embeddings = CopyDense(ld, "backbone.embeddings.weight", act_dtype, + {p.vocab_size, p.hidden_size}); + host.norm_f = CopyDense(ld, "backbone.norm_f.weight", act_dtype, {p.hidden_size}); + if (!p.tie_word_embeddings) { + host.lm_head = quantized + ? LoadNvfp4(ld, "lm_head", act_dtype, p.vocab_size, + p.hidden_size) + : CopyDense(ld, "lm_head.weight", act_dtype, + {p.vocab_size, p.hidden_size}); + } + + // --- the 52 backbone layers --- + const int64_t L = p.num_hidden_layers(); + host.layers.resize(static_cast(L)); + for (int64_t i = 0; i < L; ++i) { + NemotronHLayerWeights& lw = host.layers[static_cast(i)]; + const std::string layer = "backbone.layers." + std::to_string(i); + const std::string mixer = layer + ".mixer"; + lw.block = p.layers_block_type[static_cast(i)]; + lw.norm = CopyDense(ld, layer + ".norm.weight", act_dtype, {p.hidden_size}); + switch (lw.block) { + case NemotronHBlock::kMamba: + LoadMamba(ld, p, mixer, act_dtype, lw.mamba); + break; + case NemotronHBlock::kAttention: + LoadAttention(ld, p, mixer, act_dtype, + quantized && p.quant.fp8_kv_cache, lw.attn); + break; + case NemotronHBlock::kMoe: + LoadMoe(ld, p, mixer, act_dtype, quantized, lw.moe); + break; + case NemotronHBlock::kMlp: + LoadMlp(ld, p, mixer, act_dtype, quantized, lw.mlp); + break; + } + } + + rep.materialized = static_cast(ld.consumed.size()); + + // --- the MTP tower: DEFERRED BY NAME (W5) --------------------------------- + // + // 270 unquantized bf16 tensors (`ignore` carries `mtp*`). W5 owns the head; + // nothing here can consume it, and materializing it would cost 2.65 GB of + // host memory nothing reads. It is counted and NAMED rather than skipped — + // the whole point of the enumeration is that no tensor is in the "nobody + // thought of it" state. + std::set deferred_tags; + for (const NemotronHTensor& t : enumerated) { + if (ld.consumed.count(t.name) != 0) continue; + if (t.name.rfind("mtp.", 0) != 0) { + RefuseLoad("'" + t.name + "' (consumer '" + t.consumer + + "') is enumerated but no host slot claimed it; every enumerated " + "tensor must be materialized or deferred by name"); + } + rep.deferred += 1; + deferred_tags.insert(t.consumer); + } + for (const std::string& tag : deferred_tags) { + rep.deferred_by_name.push_back( + tag + " (MTP head, W5 of .agents/specs/nemotron-h-model.md)"); + } + + // The other direction: a tensor the checkpoint ships that nothing named. + if (rep.materialized + rep.deferred != rep.enumerated) { + RefuseLoad("accounting mismatch: " + std::to_string(rep.materialized) + + " materialized + " + std::to_string(rep.deferred) + " deferred != " + + std::to_string(rep.enumerated) + " enumerated"); + } + if (rep.enumerated != rep.in_index) { + RefuseLoad("the enumeration names " + std::to_string(rep.enumerated) + + " tensors but the checkpoint ships " + std::to_string(rep.in_index)); + } + + rep.host_bytes = HostBytesOf(host); + host.materialized = true; + return host; +} + } // namespace vllm diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index d6933ac93..3f39da519 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -543,6 +543,26 @@ vllm_cpp_add_test(test_nemotron_h_forward vllm/models/test_nemotron_h_forward.cp # USER_USAGE_PREFIXES surface (#515). Same call as the W1 resolver header. target_include_directories(test_nemotron_h_forward PRIVATE ${CMAKE_SOURCE_DIR}/src) +# Nemotron-H WEIGHT LOADER gate (#517, .agents/specs/nemotron-h-model.md §5b and +# §6b "Still owed after W4"): materializes the 18487 released tensors from the +# REAL 20.1 GiB checkpoint and runs the registered forward on them. Gated +# STRUCTURALLY, not by tokens — every tensor materialized or deferred BY NAME, +# the per-scheme composition (5935 NVFP4 W4A16 g16 triples + 46 FP8 W8A8 static +# triples + the bf16/f32 remainder) asserted against the checkpoint's own memory +# format, and the loaded dtypes asserted to be the SHIPPED ones rather than +# wider. The checkpoint resolves through the CONTENT-pinned +# parity::Nemotron35LightningSnapshot (#569) and the gate SKIPS loudly when it is +# absent, so CI never needs the NAS asset. Not the W6 token gate: no golden is +# consumed and no speed claim is made. +vllm_cpp_add_test(test_nemotron_h_loader vllm/models/test_nemotron_h_loader.cpp) +target_compile_definitions(test_nemotron_h_loader PRIVATE + NEMOTRON_H_GOLDENS_DIR="${CMAKE_CURRENT_SOURCE_DIR}/parity/goldens/nemotron_35_lightning_greedy") +# nemotron_h.h / nemotron_h_loader.h are MODEL-PRIVATE headers under src/, not +# include/vllm/ (#515); tests/parity carries the content-pinned snapshot +# resolver. +target_include_directories(test_nemotron_h_loader PRIVATE + ${CMAKE_SOURCE_DIR}/tests/parity ${CMAKE_SOURCE_DIR}/src) + # Kimi-Linear CPU REFERENCE FORWARD gates (W2-W6): the per-op reference forwards # (KDA layer vs the kimi_kda host-ref composition; NoPE-MLA vs a materialized-MHA # reference; sigmoid noaux_tc router + block vs a hand-computed case) and the whole diff --git a/tests/vllm/models/test_nemotron_h_loader.cpp b/tests/vllm/models/test_nemotron_h_loader.cpp new file mode 100644 index 000000000..cc0459339 --- /dev/null +++ b/tests/vllm/models/test_nemotron_h_loader.cpp @@ -0,0 +1,319 @@ +// Nemotron-H (`NemotronHForCausalLM`) WEIGHT LOADER gate — issue #517, spec +// `.agents/specs/nemotron-h-model.md` (§5b, §6b "Still owed after W4"). +// +// W3 made the architecture KNOWN and enumerated all 18487 released tensors; W4 +// made it COMPUTE. Nothing MATERIALIZED those tensors, so every checkpoint load +// left `NemotronHHostWeights` unmaterialized and the forward refused by name. +// This gate is that missing brick, and it gates it STRUCTURALLY rather than by +// tokens: +// +// (1) every one of the 18487 released tensors is either MATERIALIZED into a +// named host slot or DEFERRED BY NAME with the W that owns it — "nobody +// thought of it" is not a state; +// (2) the per-scheme composition is asserted against the checkpoint's own +// memory format: 5935 NVFP4 W4A16 group-16 triples, 46 FP8 W8A8 static +// triples, and the bf16/f32 remainder. A checkpoint read as UNIFORM NVFP4 +// is still numerically plausible and still matches tokens while moving the +// wrong bytes, which is exactly what a token gate cannot see; +// (3) the loaded DTYPES are the shipped ones, not wider. A too-WIDE dtype is +// numerically correct, invisible to a token comparison, and doubles the +// bytes (AGENTS.md). The three deliberate widenings (`A_log`, `D`, +// `dt_bias`, bf16 on disk -> f32 in host memory) are upstream's own +// polarity and are asserted INDIVIDUALLY so they cannot spread; +// (4) the scale tensors are bound to the right consumers — a group scale on +// the wrong projection is a x1.10-class error a token gate absorbs; +// (5) the forward reached through the SHARED `ModelRegistry::Forward` seam +// produces finite, non-degenerate logits over the real vocabulary. +// +// The checkpoint is resolved through `parity::Nemotron35LightningSnapshot()`, +// which pins by CONTENT (#569): it sweeps every staged file's +// `.cache/huggingface/download/.metadata` `commit_hash` against revision +// 29f2d174. The resolved directory is PRINTED, because `VT_NEMOTRON35_SNAPSHOT` +// is an ungated escape hatch and a gate that does not say which directory it +// read cannot be reproduced. Absent checkpoint => a loud SKIP, never a +// substitution. +// +// This is NOT the W6 token gate. It consumes no golden and makes no speed +// claim; the committed `nemotron_35_lightning_greedy/oracle.json` stays W6's. +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "hf_snapshot.h" +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/model_executor/models/nemotron_h.h" +#include "vllm/model_executor/models/nemotron_h_forward.h" +#include "vllm/model_executor/models/nemotron_h_loader.h" +#include "vllm/model_executor/models/qwen3_5.h" // ForwardLogits, *KvCache +#include "vllm/transformers_utils/hf_config.h" +#include "vllm/v1/attention/backend.h" // CommonAttentionMetadata +#include "vllm/v1/attention/backends/gdn_attn.h" // GDNAttentionMetadata +#include "vt/device.h" +#include "vt/dtype.h" + +#ifndef NEMOTRON_H_GOLDENS_DIR +#define NEMOTRON_H_GOLDENS_DIR \ + "tests/parity/goldens/nemotron_35_lightning_greedy" +#endif + +namespace { + +// Peak resident set size of this process, in KiB, straight out of the kernel. +// VmHWM is the high-water mark, so it survives a buffer that has already been +// freed by the time the check runs — which is the number a load has to report. +int64_t VmHwmKiB() { + std::ifstream in("/proc/self/status"); + std::string key; + while (in >> key) { + if (key == "VmHWM:") { + int64_t kib = 0; + in >> kib; + return kib; + } + std::string rest; + std::getline(in, rest); + } + return -1; +} + +} // namespace + +TEST_CASE("NemotronH: the REAL checkpoint loads and the forward produces logits") { + std::string why; + const std::string dir = parity::Nemotron35LightningSnapshot(&why); + if (dir.empty()) { + MESSAGE("SKIP: no Nemotron-3.5-Lightning checkpoint at the pinned revision " + "29f2d1746d8f41e316523194b19018707749b1b1 -- " + << why + << ". Export CHECKPOINT_ROOT (set -a; . ./.env; set +a) or point " + "VT_NEMOTRON35_SNAPSHOT at the staged directory."); + return; + } + // The resolved directory is EVIDENCE, not a debug aid: VT_NEMOTRON35_SNAPSHOT + // is deliberately never revision-checked, so a run that does not name the + // directory it read cannot be reproduced or falsified. + MESSAGE("resolved checkpoint directory: " << dir); + + const vllm::HfConfig config = vllm::LoadHfConfig(dir + "/config.json"); + const vllm::NemotronHParams params = vllm::ParseNemotronHParams(config); + const std::vector enumerated = + vllm::EnumerateNemotronHTensors(params); + CHECK(enumerated.size() == 18487); + + // Open every shard. The index is the authority on which files exist; opening + // them here (rather than inside the loader) is the SHARED ModelSource seam + // every other architecture's loader consumes. + const std::map weight_map = + vllm::LoadSafetensorsIndex(dir + "/model.safetensors.index.json"); + CHECK(weight_map.size() == 18487); + std::set shard_names; + for (const auto& [tensor, shard] : weight_map) { + (void)tensor; + shard_names.insert(shard); + } + MESSAGE("shards: " << shard_names.size()); + + const int64_t rss_before_kib = VmHwmKiB(); + std::vector shards; + shards.reserve(shard_names.size()); + for (const std::string& name : shard_names) { + shards.push_back(vllm::SafetensorsFile::Open(dir + "/" + name)); + } + + const vllm::ModelSource source = vllm::ModelSource::FromSafetensors(shards); + std::unique_ptr model = + vllm::ModelRegistry::Load(config, source); + REQUIRE(model != nullptr); + + const int64_t rss_after_kib = VmHwmKiB(); + MESSAGE("peak RSS before load: " << rss_before_kib / 1024 << " MiB"); + MESSAGE("peak RSS after load: " << rss_after_kib / 1024 << " MiB"); + + // ─── (1) EVERY tensor is materialized or deferred BY NAME ────────────────── + const vllm::NemotronHLoadReport& rep = vllm::NemotronHLoadReportOf(*model); + MESSAGE("host bytes: " << rep.host_bytes / (1024 * 1024) << " MiB, source " + << rep.source_bytes / (1024 * 1024) << " MiB"); + CHECK(rep.enumerated == 18487); + CHECK(rep.in_index == 18487); + CHECK(rep.materialized + rep.deferred == 18487); + // The MTP tower (W5): 270 unquantized bf16 tensors the `ignore` list's `mtp*` + // wildcard leaves unquantized. Deferred, counted, and NAMED — never silently + // dropped. + CHECK(rep.deferred == 270); + CHECK(rep.materialized == 18217); + REQUIRE(!rep.deferred_by_name.empty()); + for (const std::string& tag : rep.deferred_by_name) { + CHECK(tag.find("W5") != std::string::npos); + } + + // ─── (2) the per-scheme composition, against the CHECKPOINT's own format ─── + // + // Reading MIXED_PRECISION as uniform NVFP4 is numerically plausible and + // token-invisible. These five rows are the scheme table of spec §1 turned into + // numbers, and they must sum to every materialized tensor with nothing left + // over. + // + // NVFP4 W4A16 g16: 23*128*2 routed + 23*2 shared + lm_head = 5935 projections, + // three tensors each (weight / weight_scale / weight_scale_2). Exactly the + // `{W4A16_NVFP4: 5935}` half of the histogram W1 measured over all 5981 + // `quantized_layers` entries. + CHECK(rep.nvfp4_weights == 5935); + CHECK(rep.nvfp4_tensors == 5935 * 3); + // FP8 W8A8 static: the 23 mamba `in_proj` + 23 `out_proj` = 46 targets, the + // other half of that histogram. If this reads 0 the loader took the whole + // checkpoint as NVFP4; if it reads 5981 it took the whole thing as FP8. + CHECK(rep.fp8_weights == 46); + CHECK(rep.fp8_tensors == 46 * 3); + // The fp8 KV scheme: one k_scale + one v_scale on each of the 6 attention + // layers. + CHECK(rep.fp8_kv_scale_tensors == 12); + // The unquantized remainder, by the dtype it SHIPS in: 216 bf16 (embeddings, + // norm_f, 52 layer norms, and the 6 mamba/attention tensor families) and 46 + // f32 (the 23 routers and their 23 score-correction biases). + CHECK(rep.bf16_tensors == 216); + CHECK(rep.f32_tensors == 46); + CHECK(rep.nvfp4_tensors + rep.fp8_tensors + rep.fp8_kv_scale_tensors + + rep.bf16_tensors + rep.f32_tensors == + rep.materialized); + + // ─── (3) the loaded dtypes are the SHIPPED ones, not wider ──────────────── + // + // A too-WIDE dtype is numerically correct, invisible to a token comparison, + // and moves twice the bytes. The only widening the released checkpoint asks + // for is the three f32-by-contract SSM scalars on each of the 23 mamba + // layers — upstream's own polarity (`-torch.exp(self.A_log.float())`) and + // what `vt::Mamba2ChunkScan` validates. 23 * 3 = 69, and no more. + CHECK(rep.widened_tensors == 69); + + // The two arithmetic facts that make keeping the quantized forms mandatory + // rather than tidy: the host mirror is within a few percent of the on-disk + // bytes, and the routed experts alone would be 58.7 GB at bf16. + CHECK(rep.host_bytes < 24LL * 1024 * 1024 * 1024); + CHECK(rep.host_bytes > 16LL * 1024 * 1024 * 1024); + // Peak RSS is the number a unified-memory box lives or dies by. + MESSAGE("peak RSS after load (GiB): " << static_cast(rss_after_kib) / + (1024.0 * 1024.0)); + + // The forward, through the SHARED registry seam — never a private entry point. + const std::vector token_ids{1, 2, 3, 4}; + const std::vector positions{0, 1, 2, 3}; + const std::vector logits_indices{3}; + const vllm::v1::CommonAttentionMetadata attn_meta{}; + const vllm::v1::GDNAttentionMetadata gdn_meta{}; + std::vector attn_kv; + std::vector gdn_state; + vt::Queue queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + const vllm::ModelForwardInput input{.token_ids = token_ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + const vllm::ForwardLogits logits = + vllm::ModelRegistry::Forward(*model, input); + CHECK(logits.rows == 1); + CHECK(logits.vocab == params.vocab_size); + REQUIRE(logits.host.size() == + static_cast(logits.rows * logits.vocab)); + + // Finite AND non-degenerate: a loader that materialized zeros would return a + // perfectly finite constant row, and an argmax over it is still a token. + // Aggregated rather than one assertion per vocabulary entry: 131072 REQUIREs + // swamp the assertion count, and a changed assertion COUNT is itself the + // signal a mutation is read by ([[doctest-assertions-line-hides-thrown-cases]]). + double lo = logits.host[0]; + double hi = logits.host[0]; + int64_t nonfinite = 0; + for (float v : logits.host) { + if (!std::isfinite(v)) ++nonfinite; + lo = std::min(lo, v); + hi = std::max(hi, v); + } + MESSAGE("logits range: [" << lo << ", " << hi << "]"); + CHECK(nonfinite == 0); + CHECK(hi - lo > 1.0); + + // ─── (6) EVIDENCE, not the W6 token gate: the first greedy token of each of + // the three committed oracle prompts ──────────────────────────────── + // + // W6 formally owns the token gate — identical prompts, counts, batching and + // sampling against the pinned oracle, with the oracle identity asserted. This + // is a much smaller claim on the same artifact: for each committed prompt, + // ONE forward over its `prompt_token_ids` and the argmax of the last position + // against the golden's FIRST generated token. + // + // It is here because it is the one check that can fail for a reason the + // structural gate above cannot see. Every count can be right — 5935 NVFP4 + // triples bound to the right projections, 46 FP8 triples, 69 widenings and no + // more — while a group scale is transposed or a nibble order is flipped, and + // the answer stays finite and correctly shaped. A wrong argmax on all three + // prompts is what that looks like from outside. + const std::filesystem::path goldens = + std::filesystem::path(NEMOTRON_H_GOLDENS_DIR) / "oracle.json"; + std::ifstream gin(goldens.string()); + REQUIRE_MESSAGE(gin.good(), "cannot open " << goldens.string()); + nlohmann::json oracle; + gin >> oracle; + // The goldens name the revision they belong to; the checkpoint resolver above + // pinned the same one by CONTENT. Assert they agree rather than assuming it. + CHECK(oracle.at("revision").get() == + std::string(parity::kNemotron35LightningNvfP4Revision)); + + int matched = 0; + int total = 0; + for (const nlohmann::json& g : oracle.at("golden")) { + const std::vector prompt = + g.at("prompt_token_ids").get>(); + const int32_t want = g.at("token_ids").at(0).get(); + std::vector pos(prompt.size()); + for (size_t i = 0; i < pos.size(); ++i) pos[i] = static_cast(i); + const std::vector last{static_cast(prompt.size()) - 1}; + const vllm::ModelForwardInput in{.token_ids = prompt, + .positions = pos, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = config, + .queue = queue, + .logits_indices = last, + .num_reqs = 1}; + const vllm::ForwardLogits out = vllm::ModelRegistry::Forward(*model, in); + REQUIRE(out.host.size() == static_cast(params.vocab_size)); + const int32_t got = static_cast( + std::max_element(out.host.begin(), out.host.end()) - out.host.begin()); + ++total; + if (got == want) ++matched; + // `std::string`, NOT a `const char*` ternary: doctest 2.5.2 has no + // stringifier for a `const char*` lvalue and prints it as `1`, which is + // exactly what this line did on its first run — "oracle 69931" instead of + // "oracle 6993 MATCH". The same trap this row already repaired once (spec + // §"W1 land-prep", LOW-2). + const std::string verdict = got == want ? " MATCH" : " DIFFERS"; + MESSAGE("prompt \"" << g.at("prompt").get() + << "\": argmax " << got << ", oracle " << want + << verdict); + } + MESSAGE("first-token agreement with the committed oracle goldens: " + << matched << "/" << total); + CHECK(matched == total); + + MESSAGE("peak RSS at end: " << VmHwmKiB() / 1024 << " MiB"); +} From bcbf5741a677c19295f3a0d5a68a96c18d2f6f2d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 14 Aug 2026 18:59:58 +0000 Subject: [PATCH 2/6] fix(MODEL-NEMOTRON-H): the loader read the mapping through a pointer it may not form, and NOTHING new ran in CI (#517) FOLLOWING_AGENTS_PROTOCOL Repairs the two MEDIUM findings, three LOWs and two NITs a fresh review returned against PR #752 at `9bffa2b60`. The review CONFIRMED the substance -- 18217 of 18487 tensors materialized, 270 `mtp.*` deferred by name, host bytes 18,888,922,112 re-derived bit-exact from the 52 safetensors headers, 3/3 first-token agreement on x86_64 and Thor -- and every number it confirmed is unchanged here. F1. `CopyDense` read the safetensors mmap through `reinterpret_cast(t.data)` / `(t.data)`. `StTensor::data` is `8 + + ` and NOT ONE of those terms is required to be even, so forming either pointer is undefined whether or not the access faults -- and the same file's scalar read already used `memcpy`, so it was internally inconsistent. This is #627's class, whose THIRD recurrence closed on main the same day at `fc903b8dd` (#674) after main had been RED on `sanitize-cpu (address,undefined)` since #641; that repair is mirrored exactly. Reads go through `vt::LoadUnaligned` (the seam #301 left behind, eight sibling loaders already on it) and the two bulk `memcpy`s drop their typed pointer, which they never needed. No sanitizer configuration touched, no scope widened. It also makes docs/FEATURES.md:83 true again -- that row already claims the `*_weights.cpp` loaders never form a typed pointer into the mapping. Anchors re-derived at HEAD and asserted unique (count == 1), because recorded line numbers go stale within a PR. A standalone `-fsanitize=address,undefined -fno-sanitize-recover=all` probe over an ODD address: the shipped cast exits 1 with `load of misaligned address 0x521000000101 ... requires 2 byte alignment`; `vt::LoadUnaligned` exits 0 and matches the memcpy oracle exactly. The reviewer measured this LATENT on this checkpoint (0 of 216 BF16, 0 of 6085 F32 misaligned) and that is recorded rather than used as a reason to leave it. F2. The seams were unreachable without the 20.1 GiB checkpoint: deleting `View`'s `VT_CHECK` left the live gate at 46/46 and both offline suites green, and with no `CHECKPOINT_ROOT` the loader test recorded `Passed 0.00 sec` with ZERO assertions -- indistinguishable from a real pass. The guard is not decorative: without it `View` hands out a 128-element bf16 tensor over a 64-byte NVFP4 buffer, 192 bytes OUT OF BOUNDS. (a) `test_nemotron_h_quantized_forms.cpp`, a new OFFLINE gate needing no checkpoint and registered unconditionally, asserts `View` refuses by name and that `DenseBf16` reproduces a dequant derived INDEPENDENTLY from the upstream formula -- E2M1 table, fp8-e4m3 decoder and bf16 RNE round all written from their format definitions, anchored by hand (0x38->1.0, nibble 0x7->6.0, bf16(1.5)=0x3FC0). Every fixture is dyadic so every expected value is EXACT in bf16 and the comparison is on the BIT PATTERN, which keeps doctest::Approx's 1.19e-5 floor out of it entirely. The nibble order, the per-16 group scale and the MULTIPLIED weight_scale_2 are each asserted, and a fifth case reaches the file-private DenseFor/DenseCopy through NemotronHMlpMixer against a dense arm built from the INDEPENDENT reference, so a dequant defect cannot cancel itself out. RED-before, compile exit status printed beside every result and the binary sha with it (all four differ, so no stale binary): baseline 5 cases / 130 assertions SUCCESS; nibble order -> kHighFirst FAILURE, 96 of 96 elements differ, probe pair exactly swapped; weight_scale_2 ignored FAILURE, 36 assertions; View guard neutralized FAILURE; FP8 input_scale applied FAILURE. The first two are spec 6d's M4/M5 -- the pair a counts-only gate calls clean -- which previously needed the 20.1 GiB checkpoint and three oracle prompts and are now caught in 0.01 s with no checkpoint. (b) the checkpoint-gated skip is LOUD, NAMED and COUNTED, on the `PendingRunnerOps()` idiom: a registry of gated cases, a recorded verdict each, and a closing accounting case. With CHECKPOINT_ROOT unset it now reports 2 cases / 7 assertions and logs "checkpoint-gated cases: 0 ran, 1 skipped, of 1". Armed: not recording the skip REDs 4 of 6 assertions, and restoring the pre-repair silent return REDs 2 of 2. L1: `LoadMamba` now branches on `quantized` like LoadExpert/LoadMlp and the enumeration do, so a released bf16 NemotronH no longer refuses with a DTYPE message for what is a declared scheme. `mamba_proj_bias` remains owed and is named rather than left to be found. L2: the two "consumes no golden" scope statements now say what the code does -- one forward per prompt, argmax of the last position vs the golden's FIRST token; the 32-token decode stays W6's. L3: `186ae4cea4d7...` turned out to be a REAL sha, of the pre-rename TU at `c4029deed`; `08a65696d` then folded it into `nemotron_h_weights.cpp` as a documented pure move. The table now names both identities and says plainly that M1-M3 were not re-run after the move. N1: 16.5 GiB is 16.5 GB (15.4 GiB), fixed in both headers and the spec with the arithmetic written out. N2: the report counts quantized WEIGHTS, not dequant EVENTS. Carried into the spec rather than fixed: the FP8 arm is weight-only on the host path (input_scale carried, unapplied) so it is not a bit-mirror of vLLM's W8A8 static GEMM; and the golden arm's reach is 3 prompts x 1 token, at most 78 of 128 experts per layer, never the dense mlp, never the MTP tower. Gate: Release -Werror clean build exit 0, 0 warnings, 0 ENOSPC (44G free, 90%); new suite 5/5 130/130; loader with CHECKPOINT_ROOT unset 2/2 7/7; -DVLLM_CPP_SANITIZE='address,undefined' build of the three Nemotron-H targets clean, all three SUCCESS, no sanitizer finding. The live checkpoint gate is deliberately NOT re-run here -- the operator runs it on GB10. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/nemotron-h-model.md | 240 +++++++- docs/FEATURES.md | 2 +- src/vllm/model_executor/models/nemotron_h.cpp | 11 +- .../models/nemotron_h_forward.h | 3 +- .../model_executor/models/nemotron_h_loader.h | 5 +- .../models/nemotron_h_weights.cpp | 52 +- tests/CMakeLists.txt | 27 +- tests/vllm/models/test_nemotron_h_loader.cpp | 112 +++- .../test_nemotron_h_quantized_forms.cpp | 511 ++++++++++++++++++ 9 files changed, 927 insertions(+), 36 deletions(-) create mode 100644 tests/vllm/models/test_nemotron_h_quantized_forms.cpp diff --git a/.agents/specs/nemotron-h-model.md b/.agents/specs/nemotron-h-model.md index 098bdf41a..afc8e262b 100644 --- a/.agents/specs/nemotron-h-model.md +++ b/.agents/specs/nemotron-h-model.md @@ -1319,10 +1319,17 @@ the case THREW ([[doctest-assertions-line-hides-thrown-cases]]). **Every weight is held in the memory format the checkpoint SHIPS it in.** Not because widening is untidy, but because a dequantize-at-load loader does not fit on any box this project owns: the 5888 routed-expert projections alone are -29.4e9 parameters, **16.5 GiB packed against 58.7 GB at bf16**. On a +29.4e9 parameters, **16.5 GB packed against 58.7 GB at bf16**. On a unified-memory box that is not a failed load, it is a reboot ([[gb10-unified-memory-oom-reboots-box]]). +Both of those are **decimal GB**, and the unit was wrong here and in two headers +until the fresh review of this row caught it (`16.5 GiB`). The packed figure is +14.7e9 bytes of nibbles (0.5 B/param) plus 1.84e9 bytes of group scales (1 B per +16 params) = 16.54e9 bytes, which is **15.4 GiB**. A number quoted three times +starts being treated as measured ([[a-number-quoted-often-becomes-treated-as-measured]]), +so it is written out rather than left to be re-derived. + Measured host mirror: **18013 MiB**, against 18013 MiB read out of the shards — the mirror is the checkpoint, not a widened copy of it. Peak RSS **17.70 GiB** after the load and **18.38 GiB** at the end of a forward. @@ -1426,9 +1433,29 @@ conversely are invisible to the golden arm, because they refuse before a token exists. **Restoration.** After each, the file was rewritten from the captured original -and its SHA-256 re-verified (`186ae4cea4d7…` for `nemotron_h_loader.cpp`, -`554d1c7fc65b…` for `nemotron_h.cpp`); `git status --porcelain` showed only the -spec and `docs/FEATURES.md` edits this section is part of. +and its SHA-256 re-verified; `git status --porcelain` showed only the spec and +`docs/FEATURES.md` edits this section is part of. + +The filenames in that record were STALE, and the fresh review of this row was +right to refuse a proof naming a file that does not exist. Re-anchored at HEAD: + +| Mutated file, as the record named it | What it is now | SHA-256 | +|---|---|---| +| `nemotron_h_loader.cpp` (M1–M3), the TU as it stood at `c4029deed` | `nemotron_h_weights.cpp` | `186ae4cea4d7…` **pre-rename** | +| `nemotron_h_weights.cpp` at the landed head `9bffa2b60` | same file | `f8a87b15fb…` | +| `nemotron_h.cpp` (M4, M5), unchanged since `cf95f59b8` | same file | `554d1c7fc65b…` | + +M1–M3 ARE anchored — `186ae4cea4d7…` is a real, reachable SHA +(`git show c4029deed:src/vllm/model_executor/models/nemotron_h_loader.cpp | sha256sum`). +What happened is that `08a65696d` then folded that TU into +`nemotron_h_weights.cpp` as a documented **pure move** (its only edit being +`Refuse` → `RefuseLoad` at 27 call sites), so the record kept a filename the +tree no longer has. The mutations were NOT re-run after the move, and this +section says so rather than implying they were. M4/M5's anchor never moved and +the reviewer reproduced both argmax triples exactly. + +`f8a87b15fb…` is the sha of the reviewed head. The F1 repair below changes that +file, so it is recorded as the anchor of the mutation campaign, not of `main`. ### Gate evidence @@ -1481,14 +1508,16 @@ NVMe and the x86 box reads it over SMB from the NAS. would have read as a failed run to anyone grepping only for `SUCCESS`. The RSS above is the test's own `/proc/self/status` `VmHWM`, which is why it has one. -**One honestly weak property, recorded rather than discovered.** With no -checkpoint the gate emits a `MESSAGE` naming the missing export and RETURNS, -so CTest records a PASS, not a *Skipped* — a CI run cannot tell it apart from a -real pass without reading the log. That is the convention -`test_nemotron_h_scaffold`'s live case already set for this same checkpoint on -this same row, and matching it beat inventing a second one; the alternative is +**One honestly weak property, recorded rather than discovered — and since +REPAIRED (§6e F2b).** With no checkpoint the gate emitted a `MESSAGE` naming the +missing export and RETURNED, so CTest recorded `Passed 0.00 sec` with **zero +assertions** — indistinguishable from a real pass without reading the log. The +convention was inherited from `test_nemotron_h_scaffold`'s live case, and +matching it beat inventing a second one; the alternative is `test_modelopt_mixed_precision_checkpoint`'s exit-77, which needs a custom -`main`. Named here so the next reader does not have to work it out. +`main`. The fresh review was right that recording it is not the same as fixing +it, and §6e replaces it with a named, counted skip plus a closing accounting +case. **`test_op_parity` is RED on the base and not this row's.** `RunGoldenPass` (`tests/parity/test_op_parity.cpp:1852-1860`) walks every subdirectory of the @@ -1506,6 +1535,195 @@ outside this task's authority, the fix belongs to MODEL-MUSIC-MUSIC3 (either an tokenizer golden dirs already are), and repairing a pre-existing break inside a scoped loader change would hide it — the same call §5d finding 5 made. +## 6e. The fresh review of §6d, and the repairs (2026-08-14) + +PR #752 was reviewed at the immutable head `9bffa2b60` by an agent that neither +wrote nor gated it. The review **confirmed the substance** — 18217 of 18487 +enumerated tensors materialized, 270 `mtp.*` deferred by name, every weight in +its packed form (host bytes `18,888,922,112`, re-derived bit-exact from the 52 +safetensors headers), 5935 NVFP4 W4A16 g16 / 46 FP8 W8A8 / 12 fp8 KV scales / +216 BF16 + 46 F32 / 69 widenings, and 3/3 first-token agreement on x86_64 and +Thor — and returned two MEDIUM findings, three LOW and two NIT. All seven are +repaired below; none was declined. + +### F1 — a pointer the loader may not form + +`CopyDense` read the safetensors mapping through +`reinterpret_cast(t.data)` and +`reinterpret_cast(t.data)`. `StTensor::data` is +`8 + + ` and **none of +those three terms is required to be even**, so forming either pointer is +undefined whether or not the access faults. The same file's scalar read already +used `memcpy`, so it was internally inconsistent. + +This is the class [#627](https://github.com/mudler/vllm.cpp/issues/627) tracks, +whose THIRD recurrence closed on `main` the same day at `fc903b8dd` (#674) after +`main` had sat RED on `sanitize-cpu (address,undefined)` since #641. That repair +is mirrored exactly: reads go through `vt::LoadUnaligned` — the seam #301 left +behind, already used by eight sibling loaders — and the two bulk `memcpy`s drop +their typed pointer entirely (`memcpy` never needed one). No sanitizer +configuration was touched and no scope was widened. + +Anchors re-derived at HEAD and asserted unique (count == 1), because recorded +line numbers go stale within a PR: `nemotron_h_weights.cpp:455` (BF16 arm) and +`:473` (F32 arm) at the reviewed head. + +**Evidence.** A standalone probe (`-fsanitize=address,undefined +-fno-sanitize-recover=all`, GNU 13.3) over a deliberately ODD address, running +both read forms of this file: + +| Form | Result | +|---|---| +| the shipped `reinterpret_cast` | **exit 1**, `runtime error: load of misaligned address 0x521000000101 for type 'const short unsigned int', which requires 2 byte alignment` | +| `vt::LoadUnaligned` | **exit 0**, `data % 2 = 1`, sum `2006464` **matching the memcpy oracle exactly** | + +The reviewer measured this as LATENT rather than live — 0 of 216 BF16 and 0 of +6085 F32 tensors land misaligned on this particular checkpoint — and that is +recorded rather than used as a reason to leave it: the loader builds for +`build-test-cpu-arm64`, and UBSan could never have caught it in CI because of +F2. A project-configured `-DVLLM_CPP_SANITIZE='address,undefined'` build of the +three Nemotron-H targets is clean (below). + +**Not changed, and why:** the remaining `reinterpret_cast`s in `nemotron_h.cpp` +(:224, :256, :351) and the two write-side casts in `nemotron_h_weights.cpp` are +over `std::vector::data()`, which the default allocator returns +suitably aligned for any scalar. They are not views into a mapping and are not +in F1's class. + +### F2 — the guard was correct but UNARMED, and none of §6d's new code ran in CI + +The review proved the seams were unreachable without the 20.1 GiB checkpoint: +deleting the `VT_CHECK` in `NemotronHOwned::View` left the live gate at 46/46 +with 3/3 goldens and both offline suites green, and a `ctest` run with no +`CHECKPOINT_ROOT` recorded `test_nemotron_h_loader ... Passed 0.00 sec` with +**zero assertions** — byte-for-byte what a real pass looks like from outside. +The guard is not decorative: with it removed, `View` hands out a 128-element +bf16 tensor over a 64-byte NVFP4 buffer, **192 bytes out of bounds**. + +**(a) `tests/vllm/models/test_nemotron_h_quantized_forms.cpp`** — a new OFFLINE +gate needing no checkpoint, registered unconditionally in `tests/CMakeLists.txt` +so it runs on every CI arm. It asserts that `View` refuses a non-dense weight BY +NAME (message names the form and points at `DenseBf16`), that a dense weight is +still viewed unchanged, and that `DenseBf16` reproduces a dequant derived +**independently from the upstream formula** — the E2M1 table, an fp8-e4m3 +decoder and a bf16 RNE round all written out in the test from their format +definitions rather than called out of `vt`/`vllm`, and anchored by hand +(0x38 → 1.0, 0x40 → 2.0, nibble 0x7 → 6.0, bf16(1.5) = 0x3FC0). Every fixture +value is dyadic, so every expected result is EXACT in bf16 and the comparison is +on the BIT PATTERN — `doctest::Approx`'s ~1.19e-5 absolute floor +([[doctest-approx-scale-term-floor]]) never enters. The three properties a +counts-only gate cannot see are asserted individually: the nibble order (element +2j is the LOW nibble), the per-16 group scale (two groups of one row, different +fp8 scales), and `weight_scale_2` MULTIPLIED (not reciprocated, not ignored). A +fifth case reaches the file-private `DenseFor`/`DenseCopy` through +`NemotronHMlpMixer` and proves a quantized weight is actually widened at the +GEMM call site — against a dense arm built from the INDEPENDENT reference, so a +dequant defect cannot cancel itself out, and with the weight still packed +afterwards. + +**RED-before, by mutation.** Every result carries the compile exit status beside +it, because a mutation that fails to BUILD reads as a passing test +([[mutation-build-failure-reads-as-a-passing-test]]), and `Status:` is grepped +as well as `assertions:`, because a thrown case prints "N passed | 0 failed" +beside a red status ([[doctest-assertions-line-hides-thrown-cases]]). The binary +SHA is printed too, so a stale binary cannot print a green status +([[stale-binary-prints-green-status]]) — all four differ. + +| Mutation (applied alone to `nemotron_h.cpp`, restored + sha-verified after) | BUILD_EXIT | binary sha | Result | +|---|---|---|---| +| baseline (unmutated) | 0 | — | 5 cases / **130 assertions**, `Status: SUCCESS!`, exit 0 | +| **M4′** nibble order → `kHighFirst` | **0** | `f7e75e690a62` | **`Status: FAILURE!`**, 3 passed / **2 failed**, 4 assertions failed, exit 1 — **96 of 96** elements differ, and the probe pair reads exactly swapped (`16576` vs `16128`) | +| **M5′** `weight_scale_2` ignored (passed as `1.0F`) | **0** | `c797cc44ec4b` | **`Status: FAILURE!`**, 3 passed / 2 failed, **36** assertions failed, exit 1 | +| `View`'s `VT_CHECK` neutralized to `VT_CHECK(true, …)` | **0** | `0303eeaeff7d` | **`Status: FAILURE!`**, 4 passed / 1 failed, 5 assertions failed, exit 1 | +| FP8 `input_scale` APPLIED (it must be carried, not applied) | **0** | `630495efa1a9` | **`Status: FAILURE!`**, 4 passed / 1 failed, 3 assertions failed, exit 1 | + +M4′ and M5′ are §6d's M4 and M5, the pair that a counts-only gate calls clean. +They previously needed the 20.1 GiB checkpoint and three oracle prompts to +detect; they are now caught in **0.01 s with no checkpoint at all**. + +**(b) the checkpoint-gated skip is now LOUD, NAMED and COUNTED.** +`test_nemotron_h_loader` declares its checkpoint-gated cases in a +`CheckpointGatedCases()` registry — the `PendingRunnerOps()` idiom +(`tests/parity/test_op_parity.cpp:1834`) — records a verdict for each, asserts +in the skip path that the case is declared and that its reason is non-empty, and +closes with an accounting case that REQUIREs every declared case reached exactly +one verdict and MESSAGEs the tally. + +With `CHECKPOINT_ROOT` unset it now reports **2 cases / 7 assertions, +`Status: SUCCESS!`** and logs `checkpoint-gated cases: 0 ran, 1 skipped, of 1`, +instead of `Passed 0.00 sec` with nothing on the record. The accounting is +itself armed: + +| Mutation to `test_nemotron_h_loader.cpp` | BUILD_EXIT | binary sha | Result | +|---|---|---|---| +| the skip is announced but not RECORDED | 0 | `0fc726deb80f` | **`Status: FAILURE!`**, 0 of 2 cases passed, 4 of 6 assertions failed | +| the pre-repair SILENT early return restored | 0 | `89cb559244b8` | **`Status: FAILURE!`**, 1 of 2 cases passed, **2 of 2 assertions failed** | + +### The LOWs and NITs + +**L1 — `LoadMamba` did not branch on `quantized`.** `ClaimMamba` deliberately +supports the unquantized case (hard-coding the FP8 companions there enumerated +92 tensors a released bf16 checkpoint does not ship), but the loader called +`LoadFp8` unconditionally, so such a checkpoint refused with `'…in_proj.weight' +ships dtype BF16, not the F8_E4M3 its scheme declares` — a DTYPE message for +what is really the declared scheme. Repaired by branching, exactly as +`LoadExpert`/`LoadMlp` and the enumeration do. **Still owed on that arm and named +here rather than left to be found:** `mamba_proj_bias` is enumerated +(`in_proj.bias`/`out_proj.bias`) and has no host slot, so a checkpoint that sets +it refuses through the accounting path; the released one sets it false. + +**L2 — stale scope statements.** Both +`tests/vllm/models/test_nemotron_h_loader.cpp:36` ("It consumes no golden…") and +`tests/CMakeLists.txt:555` ("no golden is consumed") contradicted the file, +which consumes `oracle.json` at :268-316 while the CMake block defines +`NEMOTRON_H_GOLDENS_DIR`. Both now say what the code does: ONE forward per +prompt and the argmax of the last position against that prompt's FIRST token; +the full 32-token greedy decode stays W6's. + +**L3 — the restoration proof named a nonexistent file.** Repaired in §6d's +Mutation-proof table above; `186ae4cea4d7…` turned out to be a REAL sha, of the +pre-rename TU at `c4029deed`. + +**N1 — "16.5 GiB" is 16.5 GB.** Repaired in `nemotron_h_loader.h:39`, +`nemotron_h_forward.h:92` and §6d above, with the arithmetic written out. + +**N2 — "counted … by the loader's report".** The report counts quantized +WEIGHTS (5935 + 46), which is the population the widening applies to, not +dequant EVENTS, which are per GEMM call and a property of the workload. +`nemotron_h.cpp:205-206` now says that. + +### Carried forward — recorded, not repaired + +Two limits the review established, which belong in the record because neither is +visible from the code or the counts: + +1. **The FP8 arm is weight-only on the host path.** `input_scale` is carried and + never applied, so it is NOT a bit-mirror of vLLM's W8A8 *static* GEMM, which + quantizes the activation with it. Nothing on the host path quantizes an + activation, so applying it here would scale the product by a factor upstream + applies to the OTHER operand. W6's device path is where the activation scale + becomes live, and where the mirror claim can be made. +2. **The golden arm's reach is 3 prompts x 1 token.** That touches at most 78 of + the 128 experts per layer, never the dense `mlp` block (no released in-scope + checkpoint ships one), and never the MTP tower (deferred to W5). It is + evidence, not the token gate; W6 owns the token gate. + +### Re-gate evidence (the repair head) + +Local x86_64 CPU-only host (GNU 13.3, Ninja, `VLLM_CPP_CUDA=OFF`), disk recorded +beside every number ([[enospc-makes-checkers-emit-false-policy-refusals]]). + +| Arm | Result | disk free | +|---|---|---| +| Release `-Werror`, clean full build | **exit 0, 0 `warning:` lines, 0 `No space left` lines** | 44G / 90% | +| `test_nemotron_h_quantized_forms` (NEW) | **5/5 cases, 130/130 assertions, `Status: SUCCESS!`** | 44G | +| `test_nemotron_h_loader`, `CHECKPOINT_ROOT` UNSET | **2/2, 7/7, `Status: SUCCESS!`**, "0 ran, 1 skipped, of 1" | 44G | +| `-DVLLM_CPP_SANITIZE='address,undefined'`, the three Nemotron-H targets | build exit 0 / 0 warnings; `quantized_forms` 5/5 130/130, `loader` 2/2 7/7, `forward` 13/13 254/254, all `Status: SUCCESS!`, exit 0, **no sanitizer finding** | 41G | + +The live checkpoint gate was deliberately NOT re-run here: the operator runs it +on GB10, and running it on a box already at 90% would risk an ENOSPC that leaves +the previous binary in place. + ## 7. Now **State at this commit:** **W1 and W3 have LANDED on `main`; W2 is in diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 3804de4d1..a02c27b73 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -138,7 +138,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on, via the `laguna-gen` CLI; the registered engine forward VT_CHECKs non-bf16 (`ARCH-ONE-SURFACE` fold) | | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Folded onto the shared paged runner (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (the intrinsic near-tie profile); FA2 paged MLA default-ON; SACRED post-fold green** | Served via `vllm_engine_load` + `vllm_complete_tokens` (ABI v13); server 19.0 tok/s wall vs vLLM ~21 (~0.90×), speed residual open | | `KimiK3ForConditionalGeneration` | Kimi-K3 (2.8T MoE) | scaffold: registry+config+enumeration gated, forward refuses | HW-infeasible (~1.56 TB); no run | -| `NemotronHForCausalLM` | Nemotron-3.5-Lightning-30B-A3B-NVFP4 (`nvidia` @`29f2d174`) | config+enumeration+KV-shape gated; the hybrid Mamba2/GQA/relu2-MoE forward COMPUTES. The loader materializes 18487/18487 in their SHIPPED formats (5935 NVFP4 W4A16 g16, 46 FP8 W8A8, bf16), 270 MTP deferred to W5 (#517) | CPU host forward returns logits, 17.7 GiB peak RSS; 3/3 first greedy tokens match the oracle goldens, W6 owns the token gate; GGUF owed | +| `NemotronHForCausalLM` | Nemotron-3.5-Lightning-30B-A3B-NVFP4 (`nvidia` @`29f2d174`) | config+enumeration+KV-shape gated; hybrid Mamba2/GQA/relu2-MoE forward COMPUTES. Loader materializes 18487/18487 in SHIPPED formats (5935 NVFP4 g16, 46 FP8 W8A8, bf16) AND unquantized bf16; 270 MTP owed W5 (#517) | CPU host forward returns logits, 17.7 GiB peak RSS; 3/3 first greedy tokens match the oracle goldens (W6 owns the token gate); quantized forms gated offline; GGUF owed | | `MuseGlimmerForCausalLM` | real tensors, **bf16 depth 4/52 only**: 5 prefill argmax positions match a torch transcription of vllm#51655 and HF. GGUF full depth generates coherently (#347, #359) but is **NOT token-exact** | text forward + loader vs an fp32 reference, per-mechanism property tests, scaffold 11/11, GGUF gate 17/17. An ABSENT config key now takes the architecture's constant (#412): GGUF post-norms ran at 1e-5, not 1e-8 | no vLLM denominator (pin cannot load it); SECONDARY llama.cpp, same GGUF, GB10 CPU: prefill tie **0.997x**, decode 0.232x, RSS 1.92x (#333) | | `MuseGlimmerForConditionalGeneration` | vision: **no reference run of any kind**; enumeration gated vs the released 30B index (1436/1436). Image/video need bf16 safetensors: `mmproj-kquant.gguf` is refused by name | perception encoder loaded and wired, so an image or video prompt runs; `perception_emb_norm` now armed by default (#405). Reachability plus placeholder scatter only, no image or video correctness | not measurable; anchored to open vllm#51655 | | `LlamaModel` | landed tiny synthetic embedding fixture (engine path == direct pooler path, identical vectors; f64 LAST+normalize reference); real checkpoint (e5-mistral class) is a NAMED residual | pooling/embed only, text paths refuse by task; `vllm_embed` + `/v1/embeddings` | n/a (CPU correctness-grade embeddings) | diff --git a/src/vllm/model_executor/models/nemotron_h.cpp b/src/vllm/model_executor/models/nemotron_h.cpp index f320e9a66..c65e01cb3 100644 --- a/src/vllm/model_executor/models/nemotron_h.cpp +++ b/src/vllm/model_executor/models/nemotron_h.cpp @@ -202,9 +202,14 @@ void RequireWeight(const NemotronHOwned& w, const char* what, DType want, // the quantized GEMMs are the CUDA `kMoeGroupedGemmNvfp4Marlin` and fp8-linear // registrations W6 selects. Two properties keep it honest: the weight KEEPS its // quantized memory format in host memory (so RSS and the load report describe -// the checkpoint, not a widened copy of it), and the dequant is counted and -// named by the loader's report rather than being discoverable only by reading -// this function. +// the checkpoint, not a widened copy of it), and the arm is NAMED by the loader +// header and reported by the load report rather than being discoverable only by +// reading this function. +// +// What the report counts is the QUANTIZED WEIGHTS — 5935 NVFP4 W4A16 g16 +// projections and 46 FP8 W8A8 static ones, the population this widening can be +// applied to. It does NOT count dequant EVENTS: a dequant here is transient and +// per GEMM call, so its count is a property of the workload, not of the load. struct DenseOperand { std::vector owned; // non-empty only when a dequant happened Tensor view; diff --git a/src/vllm/model_executor/models/nemotron_h_forward.h b/src/vllm/model_executor/models/nemotron_h_forward.h index 89991bb2a..8bcf25b94 100644 --- a/src/vllm/model_executor/models/nemotron_h_forward.h +++ b/src/vllm/model_executor/models/nemotron_h_forward.h @@ -89,7 +89,8 @@ inline constexpr bool kNemotronHAttentionHasNoRope = true; // // It is also what makes the checkpoint fit at all. The 5888 routed-expert // projections alone are 29.4e9 parameters; dequantized to bf16 at load they are -// 58.7 GB, against 16.5 GiB packed. So `kNvfp4W4A16G16` is not an optimization +// 58.7 GB, against 16.5 GB packed (15.4 GiB — nibbles plus group scales, and +// both figures here are DECIMAL GB). So `kNvfp4W4A16G16` is not an optimization // — a load that widens it does not run on any box this project owns. enum class NemotronHWeightForm : uint8_t { // `bytes` holds Numel() elements of `dtype`, contiguous. Every W4-era weight. diff --git a/src/vllm/model_executor/models/nemotron_h_loader.h b/src/vllm/model_executor/models/nemotron_h_loader.h index 3aedbdc02..44916b10f 100644 --- a/src/vllm/model_executor/models/nemotron_h_loader.h +++ b/src/vllm/model_executor/models/nemotron_h_loader.h @@ -36,7 +36,10 @@ // ─── WHY THE QUANTIZED FORMS ARE KEPT, NOT DEQUANTIZED AT LOAD ─────────────── // // Two reasons, and the first one is arithmetic. The 5888 routed-expert -// projections are 29.4e9 parameters: 16.5 GiB packed, 58.7 GB at bf16. A +// projections are 29.4e9 parameters: 16.5 GB packed (14.7 GB of nibbles at +// 0.5 B/param plus 1.84 GB of group scales at 1 B per 16 — 15.4 GiB), against +// 58.7 GB at bf16. Both figures are DECIMAL GB, and that unit is the point: the +// same number written GiB is 7% wrong in the direction that flatters us. A // dequantize-at-load loader does not fit on any box this project owns, and on a // unified-memory box it takes the machine down rather than failing. The second // is the rule: a widened weight is numerically correct and invisible to a token diff --git a/src/vllm/model_executor/models/nemotron_h_weights.cpp b/src/vllm/model_executor/models/nemotron_h_weights.cpp index 72dc020b3..4d7aff5d6 100644 --- a/src/vllm/model_executor/models/nemotron_h_weights.cpp +++ b/src/vllm/model_executor/models/nemotron_h_weights.cpp @@ -29,6 +29,7 @@ #include #include "vllm/model_executor/model_loader/nvfp4_dequant.h" +#include "vt/unaligned.h" // LoadUnaligned — safetensors offsets carry no alignment namespace vllm { namespace { @@ -452,16 +453,29 @@ NemotronHOwned CopyDense(Loader& ld, const std::string& name, vt::DType want, if (t.nbytes != static_cast(n) * 2) { RefuseLoad("'" + name + "' is BF16 but its byte count does not match its shape"); } - const auto* src = reinterpret_cast(t.data); + // NOT `reinterpret_cast(t.data)`. `t.data` addresses the + // safetensors mmap at `8 + + ` (safetensors_reader.h) and NONE of those three terms is + // required to be even, so that pointer is one this loader may not form — + // undefined regardless of whether the access happens to fault, and a real + // fault on the strict-alignment targets this builds for + // (build-test-cpu-arm64, Jetson/Orin). Reads go through the shared + // `vt::LoadUnaligned` seam (a plain memcpy; #301 left it behind, #627 tracks + // the class, and fc903b8dd/#674 took the same repair in the LTX-2.5 VAE + // loader after `main` sat RED on `sanitize-cpu (address,undefined)` for it). + // The bulk `memcpy` below needs no typed pointer at all. if (want == vt::DType::kBF16) { - std::memcpy(w.bytes.data(), src, static_cast(n) * 2); + std::memcpy(w.bytes.data(), t.data, static_cast(n) * 2); } else if (want == vt::DType::kF32) { // A WIDENING. Counted, because the only ones the released checkpoint asks // for are the three f32-by-contract SSM scalars; a widening anywhere else // is a defect that a token gate would absorb. ld.report.widened_tensors += 1; auto* dst = reinterpret_cast(w.bytes.data()); - for (int64_t i = 0; i < n; ++i) dst[i] = vt::BF16ToF32(src[i]); + for (int64_t i = 0; i < n; ++i) { + dst[i] = vt::BF16ToF32( + vt::LoadUnaligned(t.data + static_cast(i) * 2)); + } } else { RefuseLoad("'" + name + "' cannot be materialized at the requested dtype"); } @@ -470,16 +484,20 @@ NemotronHOwned CopyDense(Loader& ld, const std::string& name, vt::DType want, if (t.nbytes != static_cast(n) * 4) { RefuseLoad("'" + name + "' is F32 but its byte count does not match its shape"); } - const auto* src = reinterpret_cast(t.data); + // Same seam, same reason as the BF16 arm above: a 4-byte-aligned + // `const float*` into the mapping is not a pointer this loader may form. if (want == vt::DType::kF32) { - std::memcpy(w.bytes.data(), src, static_cast(n) * 4); + std::memcpy(w.bytes.data(), t.data, static_cast(n) * 4); } else if (want == vt::DType::kBF16) { // A NARROWING, which is what a bf16 model dtype asks for on a tensor the // producer happened to store wide. Never silent: it is the model dtype // every other layer inherits, and the router (the one f32 consumer) asks // for f32 explicitly. auto* dst = reinterpret_cast(w.bytes.data()); - for (int64_t i = 0; i < n; ++i) dst[i] = vt::F32ToBF16(src[i]); + for (int64_t i = 0; i < n; ++i) { + dst[i] = vt::F32ToBF16( + vt::LoadUnaligned(t.data + static_cast(i) * 4)); + } } else { RefuseLoad("'" + name + "' cannot be materialized at the requested dtype"); } @@ -580,15 +598,29 @@ NemotronHOwned LoadFp8(Loader& ld, const std::string& prefix, vt::DType logical, // ─── the blocks ───────────────────────────────────────────────────────────── void LoadMamba(Loader& ld, const NemotronHParams& p, const std::string& mixer, - vt::DType adt, NemotronHMambaWeights& out) { + vt::DType adt, bool quantized, NemotronHMambaWeights& out) { const int64_t H = p.hidden_size; const int64_t I = p.mamba_intermediate_size(); const int64_t Cd = p.conv_dim(); const int64_t K = p.conv_kernel; const int64_t Hh = p.mamba_num_heads; - out.in_proj = LoadFp8(ld, mixer + ".in_proj", adt, p.in_proj_out_features(), H); - out.out_proj = LoadFp8(ld, mixer + ".out_proj", adt, H, I); + // BRANCH ON `quantized`, exactly as `LoadExpert`/`LoadMlp` and the + // `ClaimFp8`/`ClaimNvfp4` enumeration do. `ClaimMamba` deliberately supports + // the unquantized case — hard-coding the FP8 companions there enumerated 92 + // tensors a released bf16 checkpoint does not ship — so loading these two + // projections as FP8 unconditionally made the loader disagree with its own + // enumeration: a bf16 NemotronH refused with `'…in_proj.weight' ships dtype + // BF16, not the F8_E4M3 its scheme declares`, a DTYPE message for what is + // really the scheme the config declared. + if (quantized) { + out.in_proj = LoadFp8(ld, mixer + ".in_proj", adt, p.in_proj_out_features(), H); + out.out_proj = LoadFp8(ld, mixer + ".out_proj", adt, H, I); + } else { + out.in_proj = + CopyDense(ld, mixer + ".in_proj.weight", adt, {p.in_proj_out_features(), H}); + out.out_proj = CopyDense(ld, mixer + ".out_proj.weight", adt, {H, I}); + } // The conv weight ships [Cd, 1, K] and is consumed squeezed. out.conv1d_weight = CopyDense(ld, mixer + ".conv1d.weight", adt, {Cd, 1, K}, {Cd, K}); @@ -1027,7 +1059,7 @@ NemotronHHostWeights LoadNemotronHHostWeights( lw.norm = CopyDense(ld, layer + ".norm.weight", act_dtype, {p.hidden_size}); switch (lw.block) { case NemotronHBlock::kMamba: - LoadMamba(ld, p, mixer, act_dtype, lw.mamba); + LoadMamba(ld, p, mixer, act_dtype, quantized, lw.mamba); break; case NemotronHBlock::kAttention: LoadAttention(ld, p, mixer, act_dtype, diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 3f39da519..d0e805dd6 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -552,8 +552,12 @@ target_include_directories(test_nemotron_h_forward PRIVATE ${CMAKE_SOURCE_DIR}/s # format, and the loaded dtypes asserted to be the SHIPPED ones rather than # wider. The checkpoint resolves through the CONTENT-pinned # parity::Nemotron35LightningSnapshot (#569) and the gate SKIPS loudly when it is -# absent, so CI never needs the NAS asset. Not the W6 token gate: no golden is -# consumed and no speed claim is made. +# absent -- a NAMED, COUNTED skip with a closing accounting case, because a skip +# that prints `Passed 0.00 sec` with zero assertions is indistinguishable from a +# real pass -- so CI never needs the NAS asset. Not the W6 token gate and no +# speed claim: it consumes the committed oracle.json for ONE forward per prompt +# and the argmax of the last position against that prompt's FIRST token; the +# full 32-token greedy decode stays W6's. vllm_cpp_add_test(test_nemotron_h_loader vllm/models/test_nemotron_h_loader.cpp) target_compile_definitions(test_nemotron_h_loader PRIVATE NEMOTRON_H_GOLDENS_DIR="${CMAKE_CURRENT_SOURCE_DIR}/parity/goldens/nemotron_35_lightning_greedy") @@ -563,6 +567,25 @@ target_compile_definitions(test_nemotron_h_loader PRIVATE target_include_directories(test_nemotron_h_loader PRIVATE ${CMAKE_SOURCE_DIR}/tests/parity ${CMAKE_SOURCE_DIR}/src) +# Nemotron-H MIXED_PRECISION MEMORY FORMS, OFFLINE (#517, +# .agents/specs/nemotron-h-model.md §6d). The loader's design claim — every +# weight held in the format it SHIPS in, widened only TRANSIENTLY at the GEMM +# call site — lived entirely behind the 20.1 GiB checkpoint gate above, so on a +# runner with no CHECKPOINT_ROOT it was not executed at all: deleting +# NemotronHOwned::View's refusal left the whole suite green. This gate needs NO +# checkpoint and therefore runs on EVERY arm. It asserts that View REFUSES a +# non-dense weight by name, and that DenseBf16 reproduces a dequant derived +# INDEPENDENTLY from the upstream ModelOpt formula — the nibble order, the +# per-16 group scale, the multiplied weight_scale_2, and the FP8 input_scale +# that is carried and NOT applied. Those are exactly §6d's M4/M5 defects, which +# leave every structural count correct. CPU-only, no golden, no speed claim. +vllm_cpp_add_test(test_nemotron_h_quantized_forms + vllm/models/test_nemotron_h_quantized_forms.cpp) +# nemotron_h.h / nemotron_h_forward.h are MODEL-PRIVATE headers under src/, not +# include/vllm/ (#515). +target_include_directories(test_nemotron_h_quantized_forms PRIVATE + ${CMAKE_SOURCE_DIR}/src) + # Kimi-Linear CPU REFERENCE FORWARD gates (W2-W6): the per-op reference forwards # (KDA layer vs the kimi_kda host-ref composition; NoPE-MLA vs a materialized-MHA # reference; sigmoid noaux_tc router + block vs a hand-computed case) and the whole diff --git a/tests/vllm/models/test_nemotron_h_loader.cpp b/tests/vllm/models/test_nemotron_h_loader.cpp index cc0459339..d6a100411 100644 --- a/tests/vllm/models/test_nemotron_h_loader.cpp +++ b/tests/vllm/models/test_nemotron_h_loader.cpp @@ -33,8 +33,15 @@ // read cannot be reproduced. Absent checkpoint => a loud SKIP, never a // substitution. // -// This is NOT the W6 token gate. It consumes no golden and makes no speed -// claim; the committed `nemotron_35_lightning_greedy/oracle.json` stays W6's. +// This is NOT the W6 token gate, and it makes no speed claim. It DOES consume +// the committed `nemotron_35_lightning_greedy/oracle.json`, but only for its +// smallest possible claim: ONE forward per prompt, and the argmax of the last +// position against that prompt's FIRST generated token (case (6) below). The +// full 32-token greedy decode against the pinned oracle — identical prompts, +// counts, batching and sampling, with the oracle identity asserted — stays +// W6's. The first-token arm is here because it is the only check that can fail +// for a reason the structural gate cannot see: every count can be right while a +// group scale is transposed or a nibble order is flipped. #include #include @@ -89,19 +96,80 @@ int64_t VmHwmKiB() { return -1; } +// Every checkpoint-gated case in this TU, by name. A skip is a RESULT and has +// to look like one: as this file shipped, a `ctest` run with no +// `CHECKPOINT_ROOT` recorded `test_nemotron_h_loader ... Passed 0.00 sec` with +// ZERO assertions, which is byte-for-byte what a real pass looks like from +// outside. An instrument that cannot say how many things it examined has not +// reported ([[the-state-was-not-the-one-you-believed]]). +// +// So the skip path runs assertions of its own — the case name is in this +// registry, the refusal named a reason, and the count of skipped cases is +// stated — and the closing case below asserts that every registered case +// reached exactly one verdict. Modelled on `PendingRunnerOps()` +// (tests/parity/test_op_parity.cpp:1834): a listed thing SKIPS loudly, an +// unlisted one hard-FAILS. +const std::set& CheckpointGatedCases() { + static const std::set kCases = { + "real_checkpoint_loads_and_forwards", + }; + return kCases; +} + +// Verdicts recorded so far, by case name: "RAN" or "SKIPPED()". +std::map& Verdicts() { + static std::map v; + return v; +} + +// Record a LOUD, NAMED, COUNTED skip and return the number of cases skipped so +// far. Asserts the case is one this TU declared, so a renamed case cannot skip +// itself into invisibility. +int NoteSkip(const std::string& case_name, const std::string& why) { + REQUIRE_MESSAGE(CheckpointGatedCases().count(case_name) == 1, + "'" << case_name + << "' is not a declared checkpoint-gated case -- add it to " + "CheckpointGatedCases() before skipping it"); + REQUIRE_MESSAGE(!why.empty(), "a skip must state WHY; an empty reason is not one"); + Verdicts()[case_name] = "SKIPPED(" + why + ")"; + int n = 0; + for (const auto& [name, verdict] : Verdicts()) { + (void)name; + if (verdict.rfind("SKIPPED", 0) == 0) ++n; + } + return n; +} + +void NoteRan(const std::string& case_name) { + REQUIRE(CheckpointGatedCases().count(case_name) == 1); + Verdicts()[case_name] = "RAN"; +} + } // namespace TEST_CASE("NemotronH: the REAL checkpoint loads and the forward produces logits") { + const std::string kCase = "real_checkpoint_loads_and_forwards"; std::string why; const std::string dir = parity::Nemotron35LightningSnapshot(&why); if (dir.empty()) { - MESSAGE("SKIP: no Nemotron-3.5-Lightning checkpoint at the pinned revision " - "29f2d1746d8f41e316523194b19018707749b1b1 -- " - << why - << ". Export CHECKPOINT_ROOT (set -a; . ./.env; set +a) or point " - "VT_NEMOTRON35_SNAPSHOT at the staged directory."); + const int skipped = NoteSkip(kCase, why); + MESSAGE("SKIPPED " << skipped << " of " << CheckpointGatedCases().size() + << " checkpoint-gated case(s). '" << kCase + << "': no Nemotron-3.5-Lightning checkpoint at the pinned " + "revision 29f2d1746d8f41e316523194b19018707749b1b1 -- " + << why + << ". Export CHECKPOINT_ROOT (set -a; . ./.env; set +a) or " + "point VT_NEMOTRON35_SNAPSHOT at the staged directory. " + "Both gate hosts mount it at " + "/usr/local/nas_share/checkpoints/" + "nemotron-3.5-lightning-30b-nvfp4."); + // The load-bearing assertion of the skip path: this run examined ZERO + // checkpoint tensors, and says so with a check rather than with silence. + CHECK(skipped == 1); + CHECK(Verdicts().at(kCase).rfind("SKIPPED", 0) == 0); return; } + NoteRan(kCase); // The resolved directory is EVIDENCE, not a debug aid: VT_NEMOTRON35_SNAPSHOT // is deliberately never revision-checked, so a run that does not name the // directory it read cannot be reproduced or falsified. @@ -317,3 +385,33 @@ TEST_CASE("NemotronH: the REAL checkpoint loads and the forward produces logits" MESSAGE("peak RSS at end: " << VmHwmKiB() / 1024 << " MiB"); } + +// The accounting. Declared LAST so it runs after every checkpoint-gated case +// above (doctest registers in declaration order within a TU). It exists so this +// suite always states what it did: with the checkpoint present it reports every +// case RAN, and without it every case SKIPPED, with the reason. Neither is +// `Passed 0.00 sec` with nothing on the record. +TEST_CASE("NemotronH loader: every checkpoint-gated case reached exactly one verdict") { + const std::set& declared = CheckpointGatedCases(); + CHECK(Verdicts().size() == declared.size()); + int ran = 0; + int skipped = 0; + for (const std::string& name : declared) { + const auto it = Verdicts().find(name); + REQUIRE_MESSAGE(it != Verdicts().end(), + "checkpoint-gated case '" + << name + << "' recorded NO verdict -- it neither ran nor skipped, " + "which is the state this registry exists to make " + "impossible"); + MESSAGE(name << ": " << it->second); + if (it->second == "RAN") { + ++ran; + } else { + ++skipped; + } + } + MESSAGE("checkpoint-gated cases: " << ran << " ran, " << skipped << " skipped, of " + << declared.size()); + CHECK(ran + skipped == static_cast(declared.size())); +} diff --git a/tests/vllm/models/test_nemotron_h_quantized_forms.cpp b/tests/vllm/models/test_nemotron_h_quantized_forms.cpp new file mode 100644 index 000000000..0ee4dcd45 --- /dev/null +++ b/tests/vllm/models/test_nemotron_h_quantized_forms.cpp @@ -0,0 +1,511 @@ +// Nemotron-H — the MIXED_PRECISION memory forms, OFFLINE (#517, spec +// `.agents/specs/nemotron-h-model.md` §6d). +// +// ─── WHY THIS FILE EXISTS ─────────────────────────────────────────────────── +// +// The weight loader's whole design claim is that every weight is held in the +// format the checkpoint SHIPS it in — NVFP4 W4A16 group-16 for the routed and +// shared experts and `lm_head`, FP8 W8A8 static for the 46 mamba projections, +// plain bf16/f32 for the rest — and that the HOST reference forward widens a +// quantized operand only TRANSIENTLY, at the GEMM call site. Two seams carry +// that claim: `NemotronHOwned::View`, which must REFUSE a non-dense weight +// rather than reinterpret packed nibbles as the model dtype, and +// `NemotronHOwned::DenseBf16`, the declared dequant. +// +// Every one of those seams was reachable ONLY through the 20.1 GiB checkpoint +// gate. On a runner with no `CHECKPOINT_ROOT` that gate returns early, so +// deleting the `View` guard outright left the whole suite green — and the guard +// is not decorative: without it `View` hands out a 128-element bf16 tensor over +// a 64-byte NVFP4 buffer, 192 bytes OUT OF BOUNDS. This file makes the seams +// gateable with no checkpoint at all, so they run on EVERY CI arm. +// +// ─── HOW THE REFERENCE IS DERIVED ─────────────────────────────────────────── +// +// NOT from the code under test, and not through a helper both arms share +// ([[gate-comparing-shared-helper-proves-consistency-not-correctness]]). The +// expected values are computed here from upstream's own formula — +// `nvfp4_emulation_utils.dequantize_to_dtype` (swizzle=False) and ModelOpt's +// `W4A16_NVFP4` recipe, as transcribed in +// `model_loader/nvfp4_dequant.h` @ pin e24d1b24: +// +// scale[o, g] = f32(weight_scale[o, g]) * weight_scale_2 // f32, MULTIPLIED +// out[o, i] = bf16( e2m1_lut[nibble(o, i)] * scale[o, i/16] ) +// +// with element `2j` in the LOW nibble of byte `j` (torchao `pack_uint4`; +// `.agents/specs/nvfp4-nibble-order.md`), and for FP8 W8A8 static: +// +// out[i] = bf16( f8_e4m3(weight[i]) * weight_scale ) // input_scale UNUSED +// +// The fp8-e4m3 decode and the bf16 round are written out here from their format +// definitions rather than called out of `vt`/`vllm`, so a defect in either +// cannot cancel itself out. Both are ALSO anchored by literal spot values +// (0x38 -> 1.0, 0x40 -> 2.0, nibble 0x7 -> 6.0) that are checked by hand below. +// +// TOLERANCE: there is none, deliberately. Every fixture value is a small dyadic +// rational, so every expected result is EXACT in bf16 and the comparison is on +// the bf16 BIT PATTERN. That sidesteps `doctest::Approx`'s ~1.19e-5 absolute +// floor ([[doctest-approx-scale-term-floor]]) entirely: a band that cannot be +// wrong is better than a band nobody re-derived. +// +// THE TWO DEFECTS THIS IS BUILT TO CATCH are §6d's M4 and M5 — the NVFP4 nibble +// order flipped to `kHighFirst`, and `weight_scale_2` read (so the accounting +// stays right) and then ignored. Both leave EVERY structural count of the +// checkpoint gate correct, so a counts-only gate calls them clean. The fixtures +// below are chosen so each one moves a value: the two nibbles of a byte never +// carry the same magnitude, the two groups of a row never carry the same scale, +// and `weight_scale_2` is never 1. +// +// CPU-only. No checkpoint, no golden, no speed claim. +#include + +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/model_loader/nvfp4_dequant.h" // kNvfp4GroupSize +#include "vllm/model_executor/models/nemotron_h.h" +#include "vllm/model_executor/models/nemotron_h_forward.h" +#include "vt/device.h" +#include "vt/dtype.h" + +namespace { + +using vllm::NemotronHMlpWeights; +using vllm::NemotronHOwned; +using vllm::NemotronHParams; +using vllm::NemotronHWeightForm; +using vt::Device; +using vt::DeviceType; +using vt::DType; +using vt::Queue; + +Device Cpu() { return Device{DeviceType::kCPU, 0}; } +Queue CpuQ() { return Queue{Cpu(), nullptr}; } + +// ─── the independent reference ────────────────────────────────────────────── + +// IEEE fp8-e4m3fn -> f32, from the format (1 sign, 4 exp bias 7, 3 mantissa; no +// inf; 0x7F/0xFF NaN; subnormals at exponent 0). Written here, NOT called out of +// `vllm::F8E4M3ToF32`, because that is the decoder the dequant under test uses. +double RefF8E4M3(uint8_t byte) { + const int sign = (byte & 0x80U) != 0 ? -1 : 1; + const int exp = static_cast((byte >> 3) & 0x0FU); + const int man = static_cast(byte & 0x07U); + if (exp == 0) return sign * (man / 8.0) * 0.0078125; // 2^-6 subnormal step + return sign * (1.0 + man / 8.0) * std::ldexp(1.0, exp - 7); +} + +// The E2M1 magnitude table, transcribed from nvfp4_emulation_utils.py:20-22. +// Index is the 3 low magnitude bits; bit 3 is the sign. +double RefE2M1(uint8_t nibble) { + static const double kMag[8] = {0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0}; + const double m = kMag[nibble & 0x07U]; + return (nibble & 0x08U) != 0 ? -m : m; +} + +// f32 -> bf16 bit pattern, round-to-nearest-even, from the bf16 definition +// (truncate the low 16 bits with an RNE carry). Independent of `vt::F32ToBF16`. +uint16_t RefBf16Bits(double value) { + const float f = static_cast(value); + uint32_t u = 0; + std::memcpy(&u, &f, sizeof(u)); + const uint32_t lsb = (u >> 16) & 1U; + u += 0x7FFFU + lsb; + return static_cast(u >> 16); +} + +// The whole ModelOpt W4A16_NVFP4 dequant, independently. `packed` is +// [rows, cols/2] with element 2j in the LOW nibble; `gscale` is [rows, cols/16] +// fp8-e4m3 bytes; `ws2` is the per-tensor `weight_scale_2`, MULTIPLIED. +std::vector RefDequantNvfp4(const std::vector& packed, + const std::vector& gscale, float ws2, + int64_t rows, int64_t cols) { + std::vector out(static_cast(rows * cols)); + for (int64_t o = 0; o < rows; ++o) { + for (int64_t i = 0; i < cols; ++i) { + const size_t byte = static_cast(o * (cols / 2) + i / 2); + const uint8_t nib = (i % 2 == 0) ? (packed[byte] & 0x0FU) + : static_cast(packed[byte] >> 4); + const size_t g = static_cast(o * (cols / vllm::kNvfp4GroupSize) + + i / vllm::kNvfp4GroupSize); + const double s = RefF8E4M3(gscale[g]) * static_cast(ws2); + out[static_cast(o * cols + i)] = RefBf16Bits(RefE2M1(nib) * s); + } + } + return out; +} + +std::vector RefDequantFp8(const std::vector& bytes, float scale) { + std::vector out(bytes.size()); + for (size_t i = 0; i < bytes.size(); ++i) { + out[i] = RefBf16Bits(RefF8E4M3(bytes[i]) * static_cast(scale)); + } + return out; +} + +// ─── fixtures ─────────────────────────────────────────────────────────────── +// +// Both nibbles of a byte carry DIFFERENT magnitudes (so a flipped nibble order +// moves every element), and the two groups of a row carry DIFFERENT fp8 scales +// (so a transposed or per-tensor group scale moves the second group). +const uint8_t kGroupScaleA = 0x38; // 1.0 +const uint8_t kGroupScaleB = 0x40; // 2.0 +const float kWeightScale2 = 0.25F; // never 1: M5 ignores it, and that must move + +// [rows, cols/2] packed nibbles, deterministic and never nibble-symmetric. +std::vector PackedNibbles(int64_t rows, int64_t cols, uint32_t salt) { + std::vector p(static_cast(rows * cols / 2)); + for (size_t b = 0; b < p.size(); ++b) { + const uint8_t lo = static_cast((b * 5U + salt) % 16U); + uint8_t hi = static_cast((b * 3U + salt + 7U) % 16U); + // Never equal in MAGNITUDE, so swapping the two halves of the byte cannot + // leave the pair unchanged. + if ((hi & 0x07U) == (lo & 0x07U)) hi = static_cast((hi + 1U) % 16U); + p[b] = static_cast(lo | (hi << 4)); + } + return p; +} + +// One fp8 group scale per 16 inputs, alternating so no two adjacent groups of a +// row share a value. +std::vector GroupScales(int64_t rows, int64_t cols) { + std::vector s(static_cast(rows * cols / vllm::kNvfp4GroupSize)); + for (size_t i = 0; i < s.size(); ++i) s[i] = (i % 2 == 0) ? kGroupScaleA : kGroupScaleB; + return s; +} + +NemotronHOwned MakeNvfp4(int64_t rows, int64_t cols, uint32_t salt, DType logical) { + NemotronHOwned w; + w.form = NemotronHWeightForm::kNvfp4W4A16G16; + w.dtype = logical; + w.shape = {rows, cols}; + w.bytes = PackedNibbles(rows, cols, salt); + w.scale = GroupScales(rows, cols); + w.global_scale = kWeightScale2; + return w; +} + +NemotronHOwned MakeFp8(int64_t rows, int64_t cols, uint32_t salt, DType logical) { + NemotronHOwned w; + w.form = NemotronHWeightForm::kFp8W8A8Static; + w.dtype = logical; + w.shape = {rows, cols}; + w.bytes.resize(static_cast(rows * cols)); + for (size_t i = 0; i < w.bytes.size(); ++i) { + // Stay inside the finite, non-NaN e4m3 range: exponent field 1..14. + const uint8_t exp = static_cast(1U + (i * 3U + salt) % 14U); + const uint8_t man = static_cast((i * 5U + salt) % 8U); + const uint8_t sign = static_cast(((i + salt) % 3U == 0) ? 0x80U : 0U); + w.bytes[i] = static_cast(sign | (exp << 3) | man); + } + w.global_scale = 0.5F; + w.input_scale = 8.0F; // CARRIED, never applied on the host path + w.has_input_scale = true; + return w; +} + +// A dense bf16 weight whose CONTENT is the INDEPENDENT reference dequant of `q`. +// This is what makes the reach case sensitive to M4/M5: if `DenseBf16` flips a +// nibble order or drops `weight_scale_2`, this arm does not move with it. +NemotronHOwned DenseFromReference(const NemotronHOwned& q) { + const int64_t rows = q.shape[0]; + const int64_t cols = q.shape[1]; + const std::vector ref = + q.form == NemotronHWeightForm::kNvfp4W4A16G16 + ? RefDequantNvfp4(q.bytes, q.scale, q.global_scale, rows, cols) + : RefDequantFp8(q.bytes, q.global_scale); + NemotronHOwned d; + d.form = NemotronHWeightForm::kDense; + d.dtype = DType::kBF16; + d.shape = q.shape; + d.bytes.resize(ref.size() * sizeof(uint16_t)); + std::memcpy(d.bytes.data(), ref.data(), d.bytes.size()); + return d; +} + +std::vector Bits(const std::vector& bytes) { + std::vector out(bytes.size() / sizeof(uint16_t)); + std::memcpy(out.data(), bytes.data(), out.size() * sizeof(uint16_t)); + return out; +} + +// How many elements of two bf16 series differ, and the first index that does. +int CountDiff(const std::vector& a, const std::vector& b, + size_t* first) { + int n = 0; + *first = a.size(); + for (size_t i = 0; i < a.size() && i < b.size(); ++i) { + if (a[i] != b[i]) { + if (n == 0) *first = i; + ++n; + } + } + return n; +} + +} // namespace + +// ─── (0) the reference itself ─────────────────────────────────────────────── +// +// The reference above is only worth something if it is right, so it is anchored +// to values that can be read off the format by hand before it judges anything. +TEST_CASE("NemotronH quantized forms: the independent reference is anchored by hand") { + // fp8-e4m3fn: 0x38 = 0 0111 000 -> 2^(7-7) * 1.0 = 1.0. + CHECK(RefF8E4M3(0x38) == doctest::Approx(1.0)); + CHECK(RefF8E4M3(0x40) == doctest::Approx(2.0)); // 0 1000 000 -> 2^1 + CHECK(RefF8E4M3(0x3C) == doctest::Approx(1.5)); // 0 0111 100 -> 1.5 + CHECK(RefF8E4M3(0x30) == doctest::Approx(0.5)); // 0 0110 000 -> 2^-1 + CHECK(RefF8E4M3(0xB8) == doctest::Approx(-1.0)); // sign bit set + CHECK(RefF8E4M3(0x00) == doctest::Approx(0.0)); + + // E2M1: {0, .5, 1, 1.5, 2, 3, 4, 6}, sign in bit 3. + CHECK(RefE2M1(0x0) == doctest::Approx(0.0)); + CHECK(RefE2M1(0x1) == doctest::Approx(0.5)); + CHECK(RefE2M1(0x7) == doctest::Approx(6.0)); + CHECK(RefE2M1(0xF) == doctest::Approx(-6.0)); + + // bf16 bit patterns of exactly representable values. + CHECK(RefBf16Bits(1.0) == 0x3F80); + CHECK(RefBf16Bits(-1.0) == 0xBF80); + CHECK(RefBf16Bits(0.0) == 0x0000); + CHECK(RefBf16Bits(1.5) == 0x3FC0); + CHECK(RefBf16Bits(6.0) == 0x40C0); + // And it agrees with the runtime's own converter on those values, which is a + // CONSISTENCY check on top of the hand anchors, not a substitute for them. + CHECK(RefBf16Bits(1.5) == vt::F32ToBF16(1.5F)); + CHECK(RefBf16Bits(-6.0) == vt::F32ToBF16(-6.0F)); +} + +// ─── (1) View REFUSES a non-dense weight, BY NAME ─────────────────────────── +// +// A view over packed nibbles typed as the model dtype is finite, correctly +// shaped, plausible garbage — no kernel, no shape check and no token gate can +// see it. Worse, the buffer is HALF the size the view claims: `View` on a +// [8, 16] NVFP4 weight would describe 128 bf16 elements (256 bytes) over a +// 64-byte payload, 192 bytes out of bounds. +TEST_CASE("NemotronHOwned::View refuses a weight held in its shipped quantized form") { + const NemotronHOwned nvfp4 = MakeNvfp4(8, 16, 1, DType::kBF16); + REQUIRE(nvfp4.bytes.size() == 64); // [8, 16/2] + REQUIRE(nvfp4.scale.size() == 8); // [8, 16/16] + REQUIRE_FALSE(nvfp4.IsDense()); + // What the refusal is protecting: the LOGICAL extent against the payload. + REQUIRE(static_cast(nvfp4.Numel()) * sizeof(uint16_t) == + nvfp4.bytes.size() * 4); + + bool threw = false; + std::string msg; + try { + (void)nvfp4.View(Cpu()); + } catch (const std::runtime_error& e) { + threw = true; + msg = e.what(); + } + CHECK(threw); + // BY NAME: the message has to say which form it is holding and where to go, + // or the refusal is just a crash with extra steps. + CHECK(msg.find("NemotronHOwned::View") != std::string::npos); + CHECK(msg.find("NVFP4") != std::string::npos); + CHECK(msg.find("DenseBf16") != std::string::npos); + + const NemotronHOwned fp8 = MakeFp8(4, 16, 2, DType::kBF16); + REQUIRE_FALSE(fp8.IsDense()); + bool threw_fp8 = false; + try { + (void)fp8.View(Cpu()); + } catch (const std::runtime_error&) { + threw_fp8 = true; + } + CHECK(threw_fp8); + + // The dense form is still handed out, unchanged — the guard refuses the + // quantized forms, not every weight. + const NemotronHOwned dense = DenseFromReference(nvfp4); + REQUIRE(dense.IsDense()); + const vt::Tensor t = dense.View(Cpu()); + CHECK(t.rank == 2); + CHECK(t.shape[0] == 8); + CHECK(t.shape[1] == 16); + CHECK(t.dtype == DType::kBF16); + CHECK(t.data == static_cast(dense.bytes.data())); +} + +// ─── (2) the NVFP4 dequant, against the independent reference ─────────────── +TEST_CASE("NemotronHOwned::DenseBf16 reproduces the ModelOpt NVFP4 W4A16 g16 dequant") { + // 3 rows x 32 cols = TWO groups per row, so the group scale is per-group and + // not per-row or per-tensor, and 48 bytes of nibbles. + const NemotronHOwned w = MakeNvfp4(3, 32, 11, DType::kBF16); + REQUIRE(w.bytes.size() == 48); + REQUIRE(w.scale.size() == 6); + + const std::vector got = Bits(w.DenseBf16()); + const std::vector want = + RefDequantNvfp4(w.bytes, w.scale, w.global_scale, 3, 32); + REQUIRE(got.size() == want.size()); + REQUIRE(got.size() == 96); + size_t first = 0; + const int diff = CountDiff(got, want, &first); + // Plain decimal, NOT `std::hex`: doctest 2.5.2's MESSAGE stream renders the + // manipulator as `{?}` and then leaves the stream in hex, which is how this + // line first printed `got 0x{?}16000`. + CHECK_MESSAGE(diff == 0, + "first differing element " + << first << " of " << got.size() << ": got bf16 bits " + << static_cast(first < got.size() ? got[first] : 0) + << ", want " << static_cast(first < want.size() ? want[first] : 0) + << " (" << diff << " of " << got.size() << " differ)"); + + // ── the three properties a counts-only gate cannot see ────────────────── + + // NIBBLE ORDER (§6d M4). Element 2j is the LOW nibble. Assert it on a byte + // whose two halves are known, rather than inferring it from the bulk compare. + NemotronHOwned probe; + probe.form = NemotronHWeightForm::kNvfp4W4A16G16; + probe.dtype = DType::kBF16; + probe.shape = {1, 16}; + probe.bytes.assign(8, 0x00); + probe.bytes[0] = 0x71; // low nibble 0x1 -> 0.5, high nibble 0x7 -> 6.0 + probe.scale.assign(1, kGroupScaleA); // 1.0 + probe.global_scale = 1.0F; + std::vector p = Bits(probe.DenseBf16()); + REQUIRE(p.size() == 16); + CHECK(p[0] == RefBf16Bits(0.5)); // element 0 came from the LOW nibble + CHECK(p[1] == RefBf16Bits(6.0)); // element 1 from the HIGH nibble + CHECK(p[0] != p[1]); // the pair is not symmetric, so a flip moves it + + // THE GROUP SCALE is per-16, and bound to the right group. Same nibbles in + // both groups of a row, different fp8 scales -> the second group is 2x. + NemotronHOwned two; + two.form = NemotronHWeightForm::kNvfp4W4A16G16; + two.dtype = DType::kBF16; + two.shape = {1, 32}; + two.bytes.assign(16, 0x22); // every element = 1.0 before scaling + two.scale = {kGroupScaleA, kGroupScaleB}; // 1.0 then 2.0 + two.global_scale = 1.0F; + const std::vector t2 = Bits(two.DenseBf16()); + REQUIRE(t2.size() == 32); + for (size_t i = 0; i < 16; ++i) CHECK(t2[i] == RefBf16Bits(1.0)); + for (size_t i = 16; i < 32; ++i) CHECK(t2[i] == RefBf16Bits(2.0)); + + // `weight_scale_2` IS MULTIPLIED, not reciprocated and not ignored (§6d M5). + NemotronHOwned scaled = two; + scaled.global_scale = kWeightScale2; // 0.25 + const std::vector t3 = Bits(scaled.DenseBf16()); + REQUIRE(t3.size() == 32); + for (size_t i = 0; i < 16; ++i) CHECK(t3[i] == RefBf16Bits(0.25)); + for (size_t i = 16; i < 32; ++i) CHECK(t3[i] == RefBf16Bits(0.5)); + // Reciprocated would be 4.0/8.0, ignored would be 1.0/2.0 — both distinct. + CHECK(t3[0] != RefBf16Bits(4.0)); + CHECK(t3[0] != RefBf16Bits(1.0)); + + // The weight KEEPS its packed form: dequant is transient, per call. + CHECK(w.bytes.size() == 48); + CHECK(w.form == NemotronHWeightForm::kNvfp4W4A16G16); + CHECK(w.HostBytes() == 48 + 6); +} + +// ─── (3) the FP8 W8A8 static arm, and the scale it does NOT apply ─────────── +TEST_CASE("NemotronHOwned::DenseBf16 reproduces the FP8 W8A8 static dequant") { + const NemotronHOwned w = MakeFp8(4, 16, 3, DType::kBF16); + REQUIRE(w.bytes.size() == 64); + + const std::vector got = Bits(w.DenseBf16()); + const std::vector want = RefDequantFp8(w.bytes, w.global_scale); + REQUIRE(got.size() == want.size()); + size_t first = 0; + CHECK_MESSAGE(CountDiff(got, want, &first) == 0, + "first differing element " << first); + + // `input_scale` is CARRIED, NOT APPLIED: nothing on the host path quantizes + // the activation, so applying it here would scale the product by a factor + // upstream applies to the OTHER operand. `has_input_scale` distinguishes + // "the checkpoint shipped 1.0" from "no scale shipped". + CHECK(w.has_input_scale); + CHECK(w.input_scale == 8.0F); + NemotronHOwned other = w; + other.input_scale = 1.0F / 8.0F; + const std::vector got2 = Bits(other.DenseBf16()); + size_t f2 = 0; + CHECK_MESSAGE(CountDiff(got, got2, &f2) == 0, + "input_scale moved the dequant at element " << f2); + + // `weight_scale` IS applied. Doubling it doubles every non-zero element. + NemotronHOwned doubled = w; + doubled.global_scale = w.global_scale * 2.0F; + const std::vector got3 = Bits(doubled.DenseBf16()); + size_t f3 = 0; + CHECK(CountDiff(got, got3, &f3) > 0); + const std::vector want3 = RefDequantFp8(w.bytes, w.global_scale * 2.0F); + size_t f4 = 0; + CHECK_MESSAGE(CountDiff(got3, want3, &f4) == 0, "first differing element " << f4); +} + +// ─── (4) a quantized weight actually REACHES a GEMM ───────────────────────── +// +// The two cases above gate `DenseBf16` in isolation. This one gates the WIRING: +// `DenseFor`/`DenseCopy` are file-private to nemotron_h.cpp, so the only way to +// prove a quantized weight is widened at the GEMM call site (rather than viewed, +// refused, or silently skipped) is to run a mixer on one. +// +// The dense arm's content comes from the INDEPENDENT reference, never from +// `DenseBf16`, so a flipped nibble order or a dropped `weight_scale_2` breaks +// this case too instead of cancelling out. Both arms then feed byte-identical +// bf16 operands to the same `vt::MatmulBT`, so the outputs are bit-identical and +// the comparison needs no tolerance. +TEST_CASE("NemotronH forward: a quantized MLP is widened at the GEMM call site") { + NemotronHParams p; + p.hidden_size = 16; + p.intermediate_size = 32; + p.mlp_hidden_act = "relu2"; + p.mlp_bias = false; + + const NemotronHOwned up_q = MakeNvfp4(p.intermediate_size, p.hidden_size, 5, + DType::kBF16); + const NemotronHOwned down_q = MakeNvfp4(p.hidden_size, p.intermediate_size, 9, + DType::kBF16); + + NemotronHMlpWeights quant; + quant.up_proj = up_q; + quant.down_proj = down_q; + NemotronHMlpWeights dense; + dense.up_proj = DenseFromReference(up_q); + dense.down_proj = DenseFromReference(down_q); + REQUIRE_FALSE(quant.up_proj.IsDense()); + REQUIRE(dense.up_proj.IsDense()); + + const int64_t T = 3; + std::vector hidden(static_cast(T * p.hidden_size)); + for (size_t i = 0; i < hidden.size(); ++i) { + hidden[i] = static_cast(static_cast((i * 7) % 13) - 6.0) * 0.125F; + } + + Queue q = CpuQ(); + const std::vector from_quant = + vllm::NemotronHMlpMixer(quant, p, hidden, T, DType::kBF16, q); + const std::vector from_dense = + vllm::NemotronHMlpMixer(dense, p, hidden, T, DType::kBF16, q); + REQUIRE(from_quant.size() == static_cast(T * p.hidden_size)); + REQUIRE(from_dense.size() == from_quant.size()); + CHECK(std::memcmp(from_quant.data(), from_dense.data(), + from_quant.size() * sizeof(float)) == 0); + + // NOT VACUOUS: the block computes something, and a DIFFERENT quantized weight + // gives a different answer. Otherwise an all-zeros forward would pass. + bool nonzero = false; + for (float v : from_quant) nonzero = nonzero || v != 0.0F; + CHECK(nonzero); + NemotronHMlpWeights other = quant; + other.up_proj.global_scale = quant.up_proj.global_scale * 2.0F; + const std::vector moved = + vllm::NemotronHMlpMixer(other, p, hidden, T, DType::kBF16, q); + CHECK(std::memcmp(from_quant.data(), moved.data(), + from_quant.size() * sizeof(float)) != 0); + + // And the weight is STILL packed afterwards: the widening was transient, not a + // dequantize-at-load in disguise. + CHECK(quant.up_proj.form == NemotronHWeightForm::kNvfp4W4A16G16); + CHECK(quant.up_proj.bytes.size() == + static_cast(p.intermediate_size * p.hidden_size / 2)); +} From 07ecb31febb7d79b695d6da2cbfdf53914f7fd24 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 14 Aug 2026 22:00:39 +0000 Subject: [PATCH 3/6] fix(#818): the merge of #784 and the #517 weight loader was CLEAN and did not build the behaviour either side had MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FOLLOWING_AGENTS_PROTOCOL Found by RE-RUNNING the full gate after merging `origin/main`, not by reading the diff -- `git merge` reported no conflict at all, because the two changes touch different files. #784 (`b1cd4d8f6`, part of #730) rewrote `test_nemotron_h_scaffold`'s refusal subcase to call the REAL `reg.factory->load_weights(reg, config, source)` rather than downcast a fabricated `struct StubModel : vllm::LoadedModel`. UBSan was right about the stub, and the rewrite is correct ON MAIN, where `LoadNemotronHForCausalLM` reads only `source.kind` and never touches `source.safetensors`. §6d then gave NemotronH a loader that refuses an empty source by name. Merged, #784's `ModelSource source; source.kind = kSafetensors;` hits exactly that refusal and the subcase THREW before it ever reached the forward: test_nemotron_h_scaffold.cpp:666: ERROR: test case THREW exception: Model architecture NemotronHForCausalLM: the safetensors source carries no shards Neither parent is red. [[merge-tree-clean-is-not-builds]]. Repaired here because here is where the two sides meet, by SPLITTING the subcase rather than deleting either side's guarantee: - an empty safetensors source REFUSES AT LOAD, by name (`NemotronHForCausalLM`, `carries no shards`) -- the guarantee moved EARLIER and is asserted where it now lives, which is a stronger claim than the one it replaces; - the forward still refuses on unmaterialized weights, reached through the exported `vllm::NemotronHForward` on a default-constructed `NemotronHHostWeights`. That state is no longer reachable through the factory at all now that the loader exists, so it is asserted on a REAL `NemotronHHostWeights`. #784's substance is kept in full: no `StubModel`, no downcast onto an object that never was a `NemotronHLoadedModel`, no UB. ARMED, not decorative; compile exit and binary sha printed beside each result and the two binaries differ. Replacing the `carries no shards` refusal with `return model;` -> BUILD_EXIT=0, sha `4e6cdbd93478`, `Status: FAILURE!`, 11 of 12 cases, 2 of 38289 assertions failed. Neutering `VT_CHECK(host.materialized, ...)` to `VT_CHECK(true, ...)` -> BUILD_EXIT=0, sha `84179116c90c`, `Status: FAILURE!`, 11 of 12, 2 of 38289. Green after: 12/12 cases, 38289/38289, `Status: SUCCESS!` -- the split adds four assertions to W4's 38285 and the CASE count stays 12. Also in this commit, carried from the review repair: the new offline suite's independent fp8-e4m3 reference had the SUBNORMAL exponent wrong (2^-7 for 2^-6). No fixture reached it, so nothing was mis-judged, but a reference that is only right where it is exercised is not a reference. Fixed and anchored by hand at 0x01/0x07/0x08, and armed -- restoring 2^-7 REDs 2 of 134 assertions at BUILD_EXIT=0, sha `182816a73a4c`. `docs/FEATURES.md` deliberately does NOT claim the unquantized bf16 mamba arm L1 added: it is reachable and consistent with its own enumeration, but no released bf16 NemotronH checkpoint is within reach and no synthetic one exists, so it is argued rather than measured. An ungated branch is visible debt, not a supported surface, and the spec says so. Full gate at the merged head: clean Release `-Werror` build from an empty tree BUILD_EXIT=0, 0 warnings, 0 errors, 0 ENOSPC; `ctest -j4` 468 of 469 (the one failure being exactly this #818, repaired after that run); the four Nemotron-H suites 12/12+13/13+2/2+5/5 all SUCCESS afterwards. `test_op_parity` PASSES at this merge base -- main's own red (#755/#672) is gone. Closes #818. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/nemotron-h-model.md | 87 ++++++++++-- docs/FEATURES.md | 2 +- tests/vllm/models/test_nemotron_h_loader.cpp | 5 + .../test_nemotron_h_quantized_forms.cpp | 10 +- .../vllm/models/test_nemotron_h_scaffold.cpp | 124 ++++++++---------- 5 files changed, 146 insertions(+), 82 deletions(-) diff --git a/.agents/specs/nemotron-h-model.md b/.agents/specs/nemotron-h-model.md index afc8e262b..531edce4e 100644 --- a/.agents/specs/nemotron-h-model.md +++ b/.agents/specs/nemotron-h-model.md @@ -1667,10 +1667,18 @@ supports the unquantized case (hard-coding the FP8 companions there enumerated `LoadFp8` unconditionally, so such a checkpoint refused with `'…in_proj.weight' ships dtype BF16, not the F8_E4M3 its scheme declares` — a DTYPE message for what is really the declared scheme. Repaired by branching, exactly as -`LoadExpert`/`LoadMlp` and the enumeration do. **Still owed on that arm and named -here rather than left to be found:** `mamba_proj_bias` is enumerated -(`in_proj.bias`/`out_proj.bias`) and has no host slot, so a checkpoint that sets -it refuses through the accounting path; the released one sets it false. +`LoadExpert`/`LoadMlp` and the enumeration do. + +**That arm is REACHABLE and consistent with its enumeration, and it is NOT +GATED — recorded here rather than left to be found.** No released bf16 +NemotronH checkpoint is within this project's reach and no synthetic one exists, +so the branch is argued from the enumeration (`ClaimFp8`'s own `quantized` +gate) rather than measured. Two things are owed with it: a checkpoint or +synthetic fixture that executes it, and `mamba_proj_bias`, which is enumerated +(`in_proj.bias`/`out_proj.bias`) with no host slot, so a checkpoint that sets it +still refuses through the accounting path. The released checkpoint sets it +false. `docs/FEATURES.md` deliberately does NOT claim the unquantized arm: an +ungated branch is visible debt, not a supported surface. **L2 — stale scope statements.** Both `tests/vllm/models/test_nemotron_h_loader.cpp:36` ("It consumes no golden…") and @@ -1708,6 +1716,54 @@ visible from the code or the counts: checkpoint ships one), and never the MTP tower (deferred to W5). It is evidence, not the token gate; W6 owns the token gate. +### A merge that was CLEAN and did not BUILD the behaviour either side had (#818) + +Found by re-running the full gate after merging `origin/main` — not by reading +the diff, which showed no conflict at all because the two changes touch +different files. + +`#784` (`b1cd4d8f6`, part of #730) rewrote `test_nemotron_h_scaffold`'s refusal +subcase to call the REAL `reg.factory->load_weights(reg, config, source)` +instead of downcasting a fabricated `struct StubModel : vllm::LoadedModel`. +UBSan was right about the stub, and the rewrite is correct **on `main`**, where +`LoadNemotronHForCausalLM` reads only `source.kind` and never touches +`source.safetensors`. §6d then gave NemotronH a loader that refuses an empty +source BY NAME. Merged, #784's `ModelSource source; source.kind = +kSafetensors;` hits exactly that refusal and the subcase THREW before ever +reaching the forward: + +``` +test_nemotron_h_scaffold.cpp:666: ERROR: test case THREW exception: + Model architecture NemotronHForCausalLM: the safetensors source carries no shards +``` + +Neither parent is red. This is [[merge-tree-clean-is-not-builds]], and it is +repaired here because here is where the two sides meet. The subcase is SPLIT +rather than either side's guarantee deleted: + +- an empty safetensors source REFUSES AT LOAD, by name (`NemotronHForCausalLM`, + `carries no shards`). The guarantee moved EARLIER and is asserted where it now + lives, which is a stronger claim than the one it replaces; +- the forward still refuses on unmaterialized weights, reached through the + exported `vllm::NemotronHForward` on a default-constructed + `NemotronHHostWeights`. That state is no longer reachable through the factory + at all now that the loader exists — `load_weights` either materializes or + refuses — so it is asserted on a REAL `NemotronHHostWeights`. #784's substance + is kept in full: no `StubModel`, no downcast onto an object that never was a + `NemotronHLoadedModel`, no UB. + +Both halves are ARMED, not decorative. Compile exit and binary sha printed +beside each, and the two binaries differ: + +| Mutation | BUILD_EXIT | binary sha | Result | +|---|---|---|---| +| the `carries no shards` refusal replaced by `return model;` | 0 | `4e6cdbd93478` | **`Status: FAILURE!`**, 11 of 12 cases, 2 of 38289 assertions failed | +| `VT_CHECK(host.materialized, …)` neutralized to `VT_CHECK(true, …)` | 0 | `84179116c90c` | **`Status: FAILURE!`**, 11 of 12 cases, 2 of 38289 assertions failed | + +The suite goes from 38285 assertions to **38289** — the split adds four, and the +case count stays 12, which is the number to watch: a changed CASE count is +signal ([[doctest-assertions-line-hides-thrown-cases]]). + ### Re-gate evidence (the repair head) Local x86_64 CPU-only host (GNU 13.3, Ninja, `VLLM_CPP_CUDA=OFF`), disk recorded @@ -1715,14 +1771,27 @@ beside every number ([[enospc-makes-checkers-emit-false-policy-refusals]]). | Arm | Result | disk free | |---|---|---| -| Release `-Werror`, clean full build | **exit 0, 0 `warning:` lines, 0 `No space left` lines** | 44G / 90% | -| `test_nemotron_h_quantized_forms` (NEW) | **5/5 cases, 130/130 assertions, `Status: SUCCESS!`** | 44G | -| `test_nemotron_h_loader`, `CHECKPOINT_ROOT` UNSET | **2/2, 7/7, `Status: SUCCESS!`**, "0 ran, 1 skipped, of 1" | 44G | +| Release `-Werror`, **clean full build from an empty tree**, at the merged head | **`BUILD_EXIT=0`, 0 `warning:` lines, 0 `error:` lines, 0 `No space left` lines** | 44G / 90% | +| full `ctest -j4`, 469 tests | **468 passed, 1 failed** in 798 s. The one failure is `test_nemotron_h_scaffold` = the merge-induced #818 above, repaired after this run. Skipped: `test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`. **`test_op_parity` PASSED** — the `main`-inherited red (#755/#672) is gone at this merge base | 23G / 95% | +| `test_nemotron_h_scaffold` (after the #818 repair) | **12/12 cases, 38289/38289 assertions, `Status: SUCCESS!`** | 23G | +| `test_nemotron_h_forward` | **13/13, 254/254, `Status: SUCCESS!`** — identical to W4's | 23G | +| `test_nemotron_h_loader`, `CHECKPOINT_ROOT` UNSET | **2/2, 7/7, `Status: SUCCESS!`**, logs "0 ran, 1 skipped, of 1" | 23G | +| `test_nemotron_h_quantized_forms` (NEW) | **5/5, 134/134, `Status: SUCCESS!`** | 23G | +| `ctest -R '^test_nemotron_h'` after the repair | **4 of 4 passed** | 23G | | `-DVLLM_CPP_SANITIZE='address,undefined'`, the three Nemotron-H targets | build exit 0 / 0 warnings; `quantized_forms` 5/5 130/130, `loader` 2/2 7/7, `forward` 13/13 254/254, all `Status: SUCCESS!`, exit 0, **no sanitizer finding** | 41G | +| `scripts/agent-preflight.sh --staged` | all gates OK except `test_cpu_x86_llamacpp_floor`, which is ENVIRONMENTAL and base-inherited: it reproduces on the SHARED CHECKOUT at `main` with `NO_QUIET_WINDOW after 30s (busy=102% load=62.28 88.03 93.73)` on a box at load average 91-130 ([[cpu-x86-floor-test-reds-under-box-load]]) | — | + +The three test TUs edited after the clean full build (`test_nemotron_h_scaffold`, +`test_nemotron_h_loader`, `test_nemotron_h_quantized_forms`) are leaf `.cpp` +files with no dependents, and each was recompiled from source afterwards at +`-Werror` with 0 warnings — a TU is compiled whole or not at all, so that is a +clean compile of each, not an incremental one. No header changed after the clean +build. The live checkpoint gate was deliberately NOT re-run here: the operator runs it -on GB10, and running it on a box already at 90% would risk an ENOSPC that leaves -the previous binary in place. +on GB10, and this box finished the run at **95% full**, where an ENOSPC leaves +the PREVIOUS binary in place and prints a green status +([[stale-binary-prints-green-status]], [[enospc-makes-checkers-emit-false-policy-refusals]]). ## 7. Now diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 60183ca29..b48cc710a 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -139,7 +139,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on, via the `laguna-gen` CLI; the registered engine forward VT_CHECKs non-bf16 (`ARCH-ONE-SURFACE` fold) | | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Folded onto the shared paged runner (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (the intrinsic near-tie profile); FA2 paged MLA default-ON; SACRED post-fold green** | Served via `vllm_engine_load` + `vllm_complete_tokens` (ABI v13); server 19.0 tok/s wall vs vLLM ~21 (~0.90×), speed residual open | | `KimiK3ForConditionalGeneration` | Kimi-K3 (2.8T MoE) | scaffold: registry+config+enumeration gated, forward refuses | HW-infeasible (~1.56 TB); no run | -| `NemotronHForCausalLM` | Nemotron-3.5-Lightning-30B-A3B-NVFP4 (`nvidia` @`29f2d174`) | config+enumeration+KV-shape gated; hybrid Mamba2/GQA/relu2-MoE forward COMPUTES. Loader materializes 18487/18487 in SHIPPED formats (5935 NVFP4 g16, 46 FP8 W8A8, bf16) AND unquantized bf16; 270 MTP owed W5 (#517) | CPU host forward returns logits, 17.7 GiB peak RSS; 3/3 first greedy tokens match the oracle goldens (W6 owns the token gate); quantized forms gated offline; GGUF owed | +| `NemotronHForCausalLM` | Nemotron-3.5-Lightning-30B-A3B-NVFP4 (`nvidia` @`29f2d174`) | config+enumeration+KV-shape gated; hybrid Mamba2/GQA/relu2-MoE forward COMPUTES. Loader materializes 18487/18487 in their SHIPPED formats (5935 NVFP4 g16, 46 FP8 W8A8, bf16); 270 MTP owed to W5 (#517) | CPU host forward returns logits, 17.7 GiB peak RSS; 3/3 first greedy tokens match the oracle goldens (W6 owns the token gate); quantized memory forms gated offline; GGUF owed | | `MuseGlimmerForCausalLM` | real tensors, **bf16 depth 4/52 only**: 5 prefill argmax positions match a torch transcription of vllm#51655 and HF. GGUF full depth generates coherently (#347, #359) but is **NOT token-exact** | text forward + loader vs an fp32 reference, per-mechanism property tests, scaffold 11/11, GGUF gate 17/17. An ABSENT config key now takes the architecture's constant (#412): GGUF post-norms ran at 1e-5, not 1e-8 | no vLLM denominator (pin cannot load it); SECONDARY llama.cpp, same GGUF, GB10 CPU: prefill tie **0.997x**, decode 0.232x, RSS 1.92x (#333) | | `MuseGlimmerForConditionalGeneration` | vision: **no reference run of any kind**; enumeration gated vs the released 30B index (1436/1436). Image/video need bf16 safetensors: `mmproj-kquant.gguf` is refused by name | perception encoder loaded and wired, so an image or video prompt runs; `perception_emb_norm` now armed by default (#405). Reachability plus placeholder scatter only, no image or video correctness | not measurable; anchored to open vllm#51655 | | `LlamaModel` | landed tiny synthetic embedding fixture (engine path == direct pooler path, identical vectors; f64 LAST+normalize reference); real checkpoint (e5-mistral class) is a NAMED residual | pooling/embed only, text paths refuse by task; `vllm_embed` + `/v1/embeddings` | n/a (CPU correctness-grade embeddings) | diff --git a/tests/vllm/models/test_nemotron_h_loader.cpp b/tests/vllm/models/test_nemotron_h_loader.cpp index d6a100411..47438a130 100644 --- a/tests/vllm/models/test_nemotron_h_loader.cpp +++ b/tests/vllm/models/test_nemotron_h_loader.cpp @@ -391,6 +391,11 @@ TEST_CASE("NemotronH: the REAL checkpoint loads and the forward produces logits" // suite always states what it did: with the checkpoint present it reports every // case RAN, and without it every case SKIPPED, with the reason. Neither is // `Passed 0.00 sec` with nothing on the record. +// +// It presumes the WHOLE TU ran, which is how ctest invokes this binary. Running +// it alone under `--test-case=` deliberately FAILS: "no verdict recorded" is +// exactly the state it exists to refuse, and making it pass in that case would +// hand back the vacuous green it was written to remove. TEST_CASE("NemotronH loader: every checkpoint-gated case reached exactly one verdict") { const std::set& declared = CheckpointGatedCases(); CHECK(Verdicts().size() == declared.size()); diff --git a/tests/vllm/models/test_nemotron_h_quantized_forms.cpp b/tests/vllm/models/test_nemotron_h_quantized_forms.cpp index 0ee4dcd45..3be70a6b0 100644 --- a/tests/vllm/models/test_nemotron_h_quantized_forms.cpp +++ b/tests/vllm/models/test_nemotron_h_quantized_forms.cpp @@ -94,7 +94,8 @@ double RefF8E4M3(uint8_t byte) { const int sign = (byte & 0x80U) != 0 ? -1 : 1; const int exp = static_cast((byte >> 3) & 0x0FU); const int man = static_cast(byte & 0x07U); - if (exp == 0) return sign * (man / 8.0) * 0.0078125; // 2^-6 subnormal step + // Subnormal: (m/8) * 2^(1-bias) = (m/8) * 2^-6. Normal: (1 + m/8) * 2^(e-7). + if (exp == 0) return sign * (man / 8.0) * std::ldexp(1.0, -6); return sign * (1.0 + man / 8.0) * std::ldexp(1.0, exp - 7); } @@ -260,6 +261,13 @@ TEST_CASE("NemotronH quantized forms: the independent reference is anchored by h CHECK(RefF8E4M3(0x30) == doctest::Approx(0.5)); // 0 0110 000 -> 2^-1 CHECK(RefF8E4M3(0xB8) == doctest::Approx(-1.0)); // sign bit set CHECK(RefF8E4M3(0x00) == doctest::Approx(0.0)); + // Subnormals (exponent field 0), which no fixture below reaches — anchored so + // the reference is right rather than merely unexercised. 0x01 = (1/8)*2^-6. + CHECK(RefF8E4M3(0x01) == doctest::Approx(0.001953125)); + CHECK(RefF8E4M3(0x07) == doctest::Approx(0.013671875)); // the largest subnormal + CHECK(RefF8E4M3(0x08) == doctest::Approx(0.015625)); // the smallest normal + // ...and the subnormal ladder is monotone into the normals. + CHECK(RefF8E4M3(0x07) < RefF8E4M3(0x08)); // E2M1: {0, .5, 1, 1.5, 2, 3, 4, 6}, sign in bit 3. CHECK(RefE2M1(0x0) == doctest::Approx(0.0)); diff --git a/tests/vllm/models/test_nemotron_h_scaffold.cpp b/tests/vllm/models/test_nemotron_h_scaffold.cpp index f527d6f1b..7f27f027c 100644 --- a/tests/vllm/models/test_nemotron_h_scaffold.cpp +++ b/tests/vllm/models/test_nemotron_h_scaffold.cpp @@ -24,6 +24,7 @@ // resolves the checkpoint through `parity::Nemotron35LightningSnapshot()` // (env `VT_NEMOTRON35_SNAPSHOT`) and SKIPS loudly when it is absent. #include "vllm/model_executor/models/nemotron_h.h" +#include "vllm/model_executor/models/nemotron_h_forward.h" #include @@ -676,84 +677,65 @@ TEST_CASE("NemotronH: the unported arms REFUSE BY NAME") { std::runtime_error); } - SUBCASE("the forward still REFUSES on a checkpoint load, rather than returning zeros") { - // UPDATED BY W4 (#517). W3 pinned this refusal when - // `ForwardNemotronHForCausalLM` was an unconditional VT_CHECK reading - // "forward is not implemented yet". W4 ports the forward MECHANISM - // (nemotron_h.cpp) and reaches it through this same - // `ModelRegistry::Forward` seam, so the unconditional refusal is gone — but - // a checkpoint STILL cannot be run, because there is no NemotronH weight - // LOADER at all. Every load therefore leaves `NemotronHHostWeights` - // unmaterialized and the forward refuses THERE instead, naming the piece - // that is actually missing rather than the whole feature. + SUBCASE("a safetensors source with NO SHARDS refuses at LOAD, by name") { + // MERGE-INDUCED, and repaired where the two sides met. Both parents were + // green alone and the merge is what broke it + // ([[merge-tree-clean-is-not-builds]]): #784 rewrote this subcase to call + // the REAL `load_weights` instead of downcasting a `StubModel` (UBSan was + // right, #730), which is correct on a tree where + // `LoadNemotronHForCausalLM` reads only `source.kind`; §6d then gave + // NemotronH a weight loader, so the same call on an EMPTY source now + // refuses at LOAD with `the safetensors source carries no shards` and the + // subcase threw before reaching the forward. // - // The guarantee this subcase exists for is UNCHANGED and is the one that - // matters: reaching the registered forward without a materialized - // checkpoint THROWS and NAMES the gap. A forward that silently returned - // `{}` would produce zero logits and a plausible-looking garbage token. - // What moved is only which piece the message names. - // The model handed to `factory->forward` MUST be the one - // `factory->load_weights` produced. `ForwardNemotronHForCausalLM` - // (nemotron_h_registry.cpp:100) opens the handle with - // `static_cast(model)` — the universal registry seam, - // shared verbatim by every other arch's forward — and that downcast is - // undefined behaviour on any object that is not really a - // `NemotronHLoadedModel`. Two DISTINCT lines, and the sanitizer names the - // second, not the first: the cast is nemotron_h_registry.cpp:102, while the - // member call made THROUGH the resulting reference — where the vptr check - // actually fires, and what the report quoted below is anchored to — is - // nemotron_h_registry.cpp:112:30, `nh.params()`. - // This subcase used to fabricate a bare - // `struct StubModel : vllm::LoadedModel` instead, which W3 got away with - // only because the forward was then an unconditional `VT_CHECK(false)` that - // never touched `model`. W4 added the downcast, and the stub turned it into - // a live type-confusion: UBSan's vptr check reported "member call on address - // ... which does not point to an object of type 'NemotronHLoadedModel'" and - // `-fno-sanitize-recover=all` aborted the process (issue #730). - // - // `LoadNemotronHForCausalLM` reads only `source.kind` and the config — there - // is no NemotronH weight loader yet — so it builds a REAL - // `NemotronHLoadedModel` with `NemotronHHostWeights` unmaterialized without - // touching a checkpoint. That is exactly the state this subcase is about, - // and it reaches the refusal through the real object rather than a - // look-alike, so the guarantee below is now asserted on the production type. + // The guarantee is not weakened, it MOVED EARLIER and is asserted where it + // now lives. #784's substance is kept: no `StubModel`, no downcast onto an + // object that never was a `NemotronHLoadedModel`, no UB. vllm::ModelSource source; source.kind = vllm::ModelSource::Kind::kSafetensors; - const std::unique_ptr loaded = - reg.factory->load_weights(reg, config, source); - REQUIRE(loaded != nullptr); - vllm::LoadedModel& model = *loaded; + CHECK_THROWS_WITH_AS(reg.factory->load_weights(reg, config, source), + doctest::Contains("NemotronHForCausalLM"), + std::runtime_error); + CHECK_THROWS_WITH_AS(reg.factory->load_weights(reg, config, source), + doctest::Contains("carries no shards"), + std::runtime_error); + } + + SUBCASE("the forward REFUSES on unmaterialized weights, rather than returning zeros") { + // The original guarantee, unchanged and still the one that matters: reaching + // the forward without a materialized checkpoint THROWS and NAMES the gap. A + // forward that silently returned `{}` would produce zero logits and a + // plausible-looking garbage token. + // + // It is reached through `vllm::NemotronHForward` — the exported host + // reference the registry's own `ForwardNemotronHForCausalLM` calls — on a + // default-constructed `NemotronHHostWeights`, which is precisely the + // unmaterialized state. That state is no longer reachable THROUGH the + // factory at all now that the loader exists: `load_weights` either + // materializes or refuses (above). Asserting it on a real + // `NemotronHHostWeights` keeps the property gated without inventing a + // look-alike model object to carry it. + const vllm::NemotronHParams params = vllm::ParseNemotronHParams(config); + const vllm::NemotronHHostWeights host; // materialized == false + REQUIRE_FALSE(host.materialized); + vt::Queue queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; const std::vector token_ids{0}; - const std::vector positions{0}; const std::vector logits_indices{0}; - const vllm::v1::CommonAttentionMetadata attn_meta{}; - const vllm::v1::GDNAttentionMetadata gdn_meta{}; - std::vector attn_kv; - std::vector gdn_state; - vt::Queue queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; - const vllm::ModelForwardInput input{.token_ids = token_ids, - .positions = positions, - .attn_meta = attn_meta, - .gdn_meta = gdn_meta, - .attn_kv = attn_kv, - .gdn_state = gdn_state, - .config = config, - .queue = queue, - .logits_indices = logits_indices, - .num_reqs = 1}; - // The message must NAME the missing piece, not just fail. After W4 the - // missing piece is the WEIGHT LOAD, not the forward. - CHECK_THROWS_WITH_AS(reg.factory->forward(model, input), - doctest::Contains("host weights are not materialized"), - std::runtime_error); + // The message must NAME the missing piece, not just fail. + CHECK_THROWS_WITH_AS( + vllm::NemotronHForward(host, params, token_ids, logits_indices, queue), + doctest::Contains("host weights are not materialized"), + std::runtime_error); // ...and it must still say so in NemotronH's own name, so a refusal from // some shared helper cannot be mistaken for this one. - CHECK_THROWS_WITH_AS(reg.factory->forward(model, input), - doctest::Contains("NemotronHForCausalLM forward"), - std::runtime_error); - CHECK_THROWS_WITH_AS(reg.factory->forward(model, input), - doctest::Contains("nemotron-h-model.md"), - std::runtime_error); + CHECK_THROWS_WITH_AS( + vllm::NemotronHForward(host, params, token_ids, logits_indices, queue), + doctest::Contains("NemotronHForCausalLM forward"), + std::runtime_error); + CHECK_THROWS_WITH_AS( + vllm::NemotronHForward(host, params, token_ids, logits_indices, queue), + doctest::Contains("nemotron-h-model.md"), + std::runtime_error); } } From 93d1681333e026855b8dd0756b9f84850f223e28 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 14 Aug 2026 22:04:12 +0000 Subject: [PATCH 4/6] record(MODEL-NEMOTRON-H): the weights load, the model is NOT reachable from the ABI (#810), and #815 does not overlap F1 (#517) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FOLLOWING_AGENTS_PROTOCOL Record only. No executable line moves. TWO facts a reader of §6d would otherwise get wrong. 1. #810. NemotronH does NOT run end to end through `include/vllm.h` -- not on `main`, and not with this branch merged. Both refuse at `src/vllm/v1/worker/gpu/runner.cpp:525`, `runner: Qwen3.5 MambaSpec shapes disagree with model config`, a line this branch does not touch. Established independently on the real 21 GiB checkpoint on GB10: the weights load fine (17.7 GiB RSS) and then ENGINE CONSTRUCTION refuses. So "the real checkpoint runs" means the HOST REFERENCE FORWARD runs on real weights, and the spec now says that in those words so nobody reads it as "the server runs it". That single `VT_CHECK` is the ONLY load-time blocker -- neutering it alone made `vllm_engine_load` succeed and reach the forward. The spec records why that is NOT a fix: `ForwardNemotronHForCausalLM` ignores `attn_kv`, `gdn_state`, `gdn_meta` and `num_reqs`, so a server past that check emits silently wrong tokens from decode step 2 -- precisely what a 1-token golden arm cannot see. The refusal is the only thing currently making the gap visible. #810 owns it, W6 owns the paged/device runner, and it is explicitly out of scope here. 2. PR #815 / `row/FIX-UNALIGNED-LOADERS-772` closes the SAME CLASS as F1 and does NOT overlap it. Checked, not assumed: its complete file list carries no `nemotron_h_*` file, and `git log -S'nemotron_h_weights'` over `origin/main..origin/row/FIX-UNALIGNED-LOADERS-772` is empty. The reason is structural -- #772's sweep was taken over `main`, and this site does not exist on `main`; it arrives with PR #752 itself. Established with a POSITIVE CONTROL rather than from a silent grep: the same regex that is silent on the repaired `nemotron_h_weights.cpp` FIRES on `voxtral.cpp:51`, `voxtral.cpp:344` and `qwen3_vl.cpp:78`, which is exactly #815's set. The two are complementary; whichever lands second re-resolves `docs/FEATURES.md` and `tests/CMakeLists.txt` by key, the only two files both touch. Anchors re-derived at HEAD with uniqueness asserted (count == 1) for all ten sites this row edits, and the drift is why: F1's two anchors have moved 455 -> 468 and 473 -> 490 since the reviewed head `9bffa2b60` ([[recorded-line-anchors-go-stale-within-the-same-pr]]). Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/nemotron-h-model.md | 41 ++++++++++++++++++++++++++++--- 1 file changed, 37 insertions(+), 4 deletions(-) diff --git a/.agents/specs/nemotron-h-model.md b/.agents/specs/nemotron-h-model.md index 531edce4e..bfe8287c9 100644 --- a/.agents/specs/nemotron-h-model.md +++ b/.agents/specs/nemotron-h-model.md @@ -1585,10 +1585,25 @@ F2. A project-configured `-DVLLM_CPP_SANITIZE='address,undefined'` build of the three Nemotron-H targets is clean (below). **Not changed, and why:** the remaining `reinterpret_cast`s in `nemotron_h.cpp` -(:224, :256, :351) and the two write-side casts in `nemotron_h_weights.cpp` are -over `std::vector::data()`, which the default allocator returns -suitably aligned for any scalar. They are not views into a mapping and are not -in F1's class. +and the two write-side casts in `nemotron_h_weights.cpp` are over +`std::vector::data()`, which the default allocator returns suitably +aligned for any scalar. They are not views into a mapping and are not in F1's +class. + +**No overlap with PR #815 / `row/FIX-UNALIGNED-LOADERS-772`, checked rather than +assumed.** That branch closes the same CLASS at four other sites — +`voxtral.cpp:51`, `voxtral.cpp:344`, `qwen3_vl.cpp:78`, `qwen3_5_mtp.cpp:71`, +plus `minimax_h3_vae_loader.cpp`. Its complete file list contains no +`nemotron_h_*` file at all, and `git log -S'nemotron_h_weights'` over +`origin/main..origin/row/FIX-UNALIGNED-LOADERS-772` is empty. The reason is +structural, not luck: #772's sweep was taken over `main`, and this site does not +exist on `main` — it arrives with PR #752 itself. Established with a POSITIVE +CONTROL rather than from a silent grep ([[never-assert-absence-from-a-failed-grep]]): +the same regex that is silent on the repaired `nemotron_h_weights.cpp` FIRES on +`voxtral.cpp:51`, `voxtral.cpp:344` and `qwen3_vl.cpp:78` — exactly #815's set — +so the instrument works. The two changes are complementary and together close +the class. Whichever lands second must re-resolve `docs/FEATURES.md` and +`tests/CMakeLists.txt` BY KEY, the only two files both touch. ### F2 — the guard was correct but UNARMED, and none of §6d's new code ran in CI @@ -1715,6 +1730,24 @@ visible from the code or the counts: the 128 experts per layer, never the dense `mlp` block (no released in-scope checkpoint ships one), and never the MTP tower (deferred to W5). It is evidence, not the token gate; W6 owns the token gate. +3. **THE WEIGHTS LOAD; THE MODEL IS NOT REACHABLE FROM THE PUBLIC ABI (#810).** + An independent investigation established that NemotronH does not run end to + end through `include/vllm.h` — not on `main`, and not with this branch + merged. Both refuse at `src/vllm/v1/worker/gpu/runner.cpp:525`, + `runner: Qwen3.5 MambaSpec shapes disagree with model config`, a line this + branch does not touch. Proven on the real 21 GiB checkpoint on GB10: the + weights load fine (17.7 GiB RSS) and then ENGINE CONSTRUCTION refuses. So + "the real checkpoint runs" in §6d means the HOST REFERENCE FORWARD runs on + real weights, and it must not be read as "the server runs it". + + That single `VT_CHECK` is the ONLY load-time blocker — neutering it alone + made `vllm_engine_load` succeed and reach the forward. **That is not a fix + and must not be done**: `ForwardNemotronHForCausalLM` ignores `attn_kv`, + `gdn_state`, `gdn_meta` and `num_reqs`, so a server past that check emits + silently wrong tokens from decode step 2 — the failure mode a token gate on + a 1-token prompt cannot see. The refusal is currently the only thing making + the gap visible. #810 owns it; W6 owns the paged/device runner. Explicitly + OUT OF SCOPE here. ### A merge that was CLEAN and did not BUILD the behaviour either side had (#818) From 54e7ea6091ed58eaa8cc9c39aaba60e93d344faa Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 14 Aug 2026 22:31:34 +0000 Subject: [PATCH 5/6] evidence(MODEL-NEMOTRON-H): the final gate at the merged head -- 470/470, clean build, all preflight gates green (#517) FOLLOWING_AGENTS_PROTOCOL Record only. No executable line moves. Re-ran the row's gate MYSELF at `e3f2992af` over `origin/main` @ `c01e4be93`, rather than carrying forward the numbers from the contended run earlier in this session: Release -Werror, CLEAN full build from an empty tree BUILD_EXIT=0, 0 `warning:`, 0 `error:`, 0 `No space left`, 1393/1393 full `ctest -j4` 470 of 470 PASSED, CTEST_EXIT=0, 124 s skipped: test_modelopt_mixed_precision_checkpoint, test_voxtral_e2e scripts/agent-preflight.sh All gates green disk 103G free / 76-77% before and after Two reds reported earlier in this session are now shown to be ENVIRONMENTAL, and both are recorded rather than quietly dropped. `test_cpu_x86_llamacpp_floor` RED at load average 91-130 with `NO_QUIET_WINDOW after 30s (busy=102% ...)`, and it reproduced on the SHARED CHECKOUT at `main`, which is what made it base-inherited rather than this row's ([[cpu-x86-floor-test-reds-under-box-load]]); it passes on the idle box. `test_cpu_threadpool` likewise. `test_op_parity`, RED on `main` when this task was scoped (#755/#672), PASSES at this merge base. The earlier contended run is KEPT in the spec rather than replaced: 468 of 469 with `test_nemotron_h_scaffold` red is how #818 was found, and it was found by RE-RUNNING the gate, not by reading the merge diff -- which reported no conflict at all. A negative result is a result. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/nemotron-h-model.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/.agents/specs/nemotron-h-model.md b/.agents/specs/nemotron-h-model.md index bfe8287c9..fbb019551 100644 --- a/.agents/specs/nemotron-h-model.md +++ b/.agents/specs/nemotron-h-model.md @@ -1802,10 +1802,18 @@ signal ([[doctest-assertions-line-hides-thrown-cases]]). Local x86_64 CPU-only host (GNU 13.3, Ninja, `VLLM_CPP_CUDA=OFF`), disk recorded beside every number ([[enospc-makes-checkers-emit-false-policy-refusals]]). +FINAL, at the merged head `e3f2992af` over `origin/main` @ `c01e4be93`: + | Arm | Result | disk free | |---|---|---| -| Release `-Werror`, **clean full build from an empty tree**, at the merged head | **`BUILD_EXIT=0`, 0 `warning:` lines, 0 `error:` lines, 0 `No space left` lines** | 44G / 90% | -| full `ctest -j4`, 469 tests | **468 passed, 1 failed** in 798 s. The one failure is `test_nemotron_h_scaffold` = the merge-induced #818 above, repaired after this run. Skipped: `test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`. **`test_op_parity` PASSED** — the `main`-inherited red (#755/#672) is gone at this merge base | 23G / 95% | +| Release `-Werror`, **clean full build from an empty tree** | **`BUILD_EXIT=0`, 0 `warning:` lines, 0 `error:` lines, 0 `No space left` lines**, 1393/1393 targets | 103G / 77% | +| full `ctest -j4` | **470 of 470 PASSED, `CTEST_EXIT=0`**, 124 s. Skipped: `test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`. `test_op_parity` PASSES (the `main`-inherited red #755/#672 is gone at this base); `test_cpu_threadpool` passes on an idle box | 103G / 76% | +| `scripts/agent-preflight.sh` | **All gates green** — including `test_cpu_x86_llamacpp_floor`, which had RED earlier in this session at load average 91-130 and passes on the idle box, confirming it environmental rather than this row's ([[cpu-x86-floor-test-reds-under-box-load]]) | 103G | + +The earlier run of this same gate, on a contended box, is kept because a +negative result is a result: `ctest` 468 of 469 with `test_nemotron_h_scaffold` +red — that is how #818 was found, and it was found by RE-RUNNING the gate rather +than by reading the merge diff, which showed no conflict at all. | `test_nemotron_h_scaffold` (after the #818 repair) | **12/12 cases, 38289/38289 assertions, `Status: SUCCESS!`** | 23G | | `test_nemotron_h_forward` | **13/13, 254/254, `Status: SUCCESS!`** — identical to W4's | 23G | | `test_nemotron_h_loader`, `CHECKPOINT_ROOT` UNSET | **2/2, 7/7, `Status: SUCCESS!`**, logs "0 ran, 1 skipped, of 1" | 23G | From c6255afa1c5b8e141d032f229fe01671b2b2fb98 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 14 Aug 2026 22:50:25 +0000 Subject: [PATCH 6/6] evidence(MODEL-NEMOTRON-H): re-gate on the new base -- 471/471 and the new style checker (#517) FOLLOWING_AGENTS_PROTOCOL `origin/main` moved again between the previous gate and the push, so the whole gate ran again rather than the earlier numbers being carried forward. The move mattered for a second reason: `ddff09093` landed `scripts/check-commit-style.py`, a checker this branch had never been measured against, and it changed `scripts/agent-preflight.sh` with it. At the merged head over `origin/main` @ `ca01719e6`: Release -Werror, CLEAN full build from an empty tree BUILD_EXIT=0, 0 `warning:`, 0 `error:`, 0 `No space left`, 1395/1395 full `ctest -j4` 471 of 471 PASSED, CTEST_EXIT=0, 111 s skipped: test_modelopt_mixed_precision_checkpoint, test_voxtral_e2e the four Nemotron-H suites scaffold 12/12, forward 13/13, loader 2/2, quantized_forms 5/5 scripts/check-commit-style.py --range ca01719e6..HEAD OK: commit writing style scripts/check-commit-trailers.py --range origin/main..HEAD OK: commit trailer contract disk 100G free / 77% One instrument failure is recorded because it nearly became a false report. A `tail` of the ctest log taken one minute before the run started read a PRE-EXISTING file naming a foreign worktree (`.claude/worktrees/agent-a956d471bf9a3b7d0/build-rel`) and ending in `Errors while running CTest`. The scratchpad is shared, so a stale log looks exactly like a fresh red ([[scratchpad-is-shared-across-sessions]]). It was discarded after checking that the live log names this worktree in its own header, and the result above comes only from the run that did. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/nemotron-h-model.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/.agents/specs/nemotron-h-model.md b/.agents/specs/nemotron-h-model.md index fbb019551..9c8b9eeed 100644 --- a/.agents/specs/nemotron-h-model.md +++ b/.agents/specs/nemotron-h-model.md @@ -1802,12 +1802,18 @@ signal ([[doctest-assertions-line-hides-thrown-cases]]). Local x86_64 CPU-only host (GNU 13.3, Ninja, `VLLM_CPP_CUDA=OFF`), disk recorded beside every number ([[enospc-makes-checkers-emit-false-policy-refusals]]). -FINAL, at the merged head `e3f2992af` over `origin/main` @ `c01e4be93`: +FINAL, at the merged head over `origin/main` @ `ca01719e6` -- re-run in full +because `origin/main` moved again before the push AND landed a checker this +branch had never been measured against (`scripts/check-commit-style.py`, +POLICY-SINGLE-PR-AND-STYLE `ddff09093`): | Arm | Result | disk free | |---|---|---| -| Release `-Werror`, **clean full build from an empty tree** | **`BUILD_EXIT=0`, 0 `warning:` lines, 0 `error:` lines, 0 `No space left` lines**, 1393/1393 targets | 103G / 77% | -| full `ctest -j4` | **470 of 470 PASSED, `CTEST_EXIT=0`**, 124 s. Skipped: `test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`. `test_op_parity` PASSES (the `main`-inherited red #755/#672 is gone at this base); `test_cpu_threadpool` passes on an idle box | 103G / 76% | +| Release `-Werror`, **clean full build from an empty tree** | **`BUILD_EXIT=0`, 0 `warning:` lines, 0 `error:` lines, 0 `No space left` lines**, 1395/1395 targets | 100G / 77% | +| full `ctest -j4` | **471 of 471 PASSED, `CTEST_EXIT=0`**, 111 s. Skipped: `test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`. `test_op_parity` PASSES (the `main`-inherited red #755/#672 is gone at this base); `test_cpu_threadpool` passes on an idle box | 100G / 77% | +| the four Nemotron-H suites in that run | scaffold **12/12**, forward **13/13**, loader **2/2**, quantized_forms **5/5**, all Passed | 100G | +| `scripts/check-commit-style.py --range ca01719e6..HEAD` (NEW gate) | **`OK: commit writing style`** | — | +| `scripts/check-commit-trailers.py --range origin/main..HEAD` | **`OK: commit trailer contract`** | — | | `scripts/agent-preflight.sh` | **All gates green** — including `test_cpu_x86_llamacpp_floor`, which had RED earlier in this session at load average 91-130 and passes on the idle box, confirming it environmental rather than this row's ([[cpu-x86-floor-test-reds-under-box-load]]) | 103G | The earlier run of this same gate, on a contended box, is kept because a