feat(#810 A2-Q1): NemotronH's 23 Mamba2 blocks reach the device on the FP8 W8A8 seam (#517)

.agents/model-matrix.md

@@ -282,7 +282,7 @@ Transformers compatibility is capability-driven and excluded from finite counts.
 | `MODEL-TEXT-mimo-v2-mi-mo-v2-flash-for-causal-lm` | `MiMoV2FlashForCausalLM` | `registry.py:166`; `vllm/model_executor/models/mimo_v2.py::MiMoV2FlashForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; sliding-window attention | ☐ required | `INVENTORIED` | none | unassigned |
 | `MODEL-TEXT-mimo-v2-mi-mo-v2-for-causal-lm` | `MiMoV2ForCausalLM` | `registry.py:167`; `vllm/model_executor/models/mimo_v2.py::MiMoV2ForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; sliding-window attention | ☐ required | `INVENTORIED` | none | unassigned |
 | `MODEL-TEXT-nemotron-nemotron-for-causal-lm` | `NemotronForCausalLM` | `registry.py:168`; `vllm/model_executor/models/nemotron.py::NemotronForCausalLM` | causal generation / text | model loader/forward; paged attention/KV | ☐ required | `INVENTORIED` | none | unassigned |
-| `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | `NemotronHForCausalLM`, `NemotronHPuzzleForCausalLM` | `registry.py:169-170`; `vllm/model_executor/models/nemotron_h.py::NemotronHForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; Mamba/SSM state; sliding-window attention. **2026-08-12: row spec committed ([#517](https://github.com/mudler/vllm.cpp/issues/517), [spec](specs/nemotron-h-model.md)); BLOCKED on `KERNEL-SSM-MAMBA` [#496](https://github.com/mudler/vllm.cpp/issues/496)** ([spec](specs/mamba2-ssd.md)) — the Mamba2 SSD core is unported. Beyond that kernel this arch owes three further things, none of which exist locally: non-gated `relu²` MoE (every grouped-MoE op we have is SwiGLU-shaped), ModelOpt `MIXED_PRECISION` per-module loading (`quantization/modelopt.py:2280` — NVFP4 W4A16 g16 experts and FP8 W8A8 mamba projections in one checkpoint), and the DeepSeek-style MTP head. Driver checkpoint `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` (20.1 GiB, fits one GB10). **2026-08-17: A2-P lands the PAGED forward** ([#810](https://github.com/mudler/vllm.cpp/issues/810), [spec](specs/nemotron-h-a2p-paged-forward.md)): `ForwardNemotronHForCausalLM` selects `NemotronHPagedForward` whenever the runner supplies paged KV and recurrent state, so K/V go into the runner's pages at `attn_meta.slot_mapping` and the conv + SSM rows are gathered from and scattered to `GdnStateCache` at the metadata's state indices — the first time this architecture carries state between decode steps. G-SAFE narrows from three clauses to `num_reqs <= 1` (batching is A2-B). The row stays `INVENTORIED`: the A3 end-to-end token gate against the released checkpoint is PENDING on `dgx.casa`, and `lm_head` (A2-Q2b), the FP8 mamba projections (A2-Q1, [#940](https://github.com/mudler/vllm.cpp/issues/940)), MTP (W5) and GGUF (W7) are all still owed | ☐ required | `INVENTORIED` | none | unassigned |
+| `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | `NemotronHForCausalLM`, `NemotronHPuzzleForCausalLM` | `registry.py:169-170`; `vllm/model_executor/models/nemotron_h.py::NemotronHForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; Mamba/SSM state; sliding-window attention. **2026-08-12: row spec committed ([#517](https://github.com/mudler/vllm.cpp/issues/517), [spec](specs/nemotron-h-model.md)); BLOCKED on `KERNEL-SSM-MAMBA` [#496](https://github.com/mudler/vllm.cpp/issues/496)** ([spec](specs/mamba2-ssd.md)) — the Mamba2 SSD core is unported. Beyond that kernel this arch owes three further things, none of which exist locally: non-gated `relu²` MoE (every grouped-MoE op we have is SwiGLU-shaped), ModelOpt `MIXED_PRECISION` per-module loading (`quantization/modelopt.py:2280` — NVFP4 W4A16 g16 experts and FP8 W8A8 mamba projections in one checkpoint), and the DeepSeek-style MTP head. Driver checkpoint `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` (20.1 GiB, fits one GB10). **2026-08-17: A2-P lands the PAGED forward** ([#810](https://github.com/mudler/vllm.cpp/issues/810), [spec](specs/nemotron-h-a2p-paged-forward.md)): `ForwardNemotronHForCausalLM` selects `NemotronHPagedForward` whenever the runner supplies paged KV and recurrent state, so K/V go into the runner's pages at `attn_meta.slot_mapping` and the conv + SSM rows are gathered from and scattered to `GdnStateCache` at the metadata's state indices — the first time this architecture carries state between decode steps. G-SAFE narrows from three clauses to `num_reqs <= 1` (batching is A2-B). The row stays `INVENTORIED`: the A3 end-to-end token gate against the released checkpoint is PENDING on `dgx.casa`, and `lm_head` (A2-Q2b), MTP (W5) and GGUF (W7) are all still owed. **2026-08-18: A2-Q1 lands the FP8 W8A8 MAMBA DEVICE ARM** ([#810](https://github.com/mudler/vllm.cpp/issues/810), [spec](specs/nemotron-h-a2q1-fp8-mamba.md)), on the shared seam [#940](https://github.com/mudler/vllm.cpp/issues/940) extracted for it: `NemotronHMamba2MixerDevice` mirrors the host arm op for op and both forwards select it at runtime, so the 23 Mamba2 layers no longer download the normed hidden, dequantize 890e6 fp8 elements to bf16 per token and upload the result. The recurrent rows the gather produced are advanced IN PLACE, and G-SAFE is untouched. The GB10 per-block numeric gate and the GPU-occupancy measurement the unit is accepted on are PENDING a lease | ☐ required | `INVENTORIED` | none | unassigned |
 | `MODEL-TEXT-olmo-olmo-for-causal-lm` | `OlmoForCausalLM` | `registry.py:171`; `vllm/model_executor/models/olmo.py::OlmoForCausalLM` | causal generation / text | model loader/forward; paged attention/KV | ☐ required | `INVENTORIED` | none | unassigned |
 | `MODEL-TEXT-olmo2-olmo2-for-causal-lm` | `Olmo2ForCausalLM`, `Olmo3ForCausalLM` | `registry.py:172-173`; `vllm/model_executor/models/olmo2.py::Olmo2ForCausalLM` | causal generation / text | OLMo-2-0425-1B dense LANDED (W0-W4). Impl: `include/vllm/model_executor/models/olmo2.h` + `src/vllm/model_executor/models/{olmo2,olmo2_weights,olmo2_registry}.cpp` (TWO `REGISTER_VLLM_MODEL` for `Olmo2ForCausalLM`+`Olmo3ForCausalLM`, one shared factory; reuses the shared dense glue). ZERO new compute kernel — the two "distinctives" both reduce to WIRING over landed ops: (1) PURE POST-NORM (`norm_after`, `olmo2.py:261-277`) = standalone `vt::RmsNorm` on each sublayer OUTPUT + plain `vt::Add` residual re-join, NO pre-norm (the GLM-4 standalone-output-norm op, pre-norms dropped); (2) FULL-WIDTH QK-norm (`olmo2.py:113-117,160-172`) = two standalone `vt::RmsNorm` over `[T,q_size]`/`[T,kv_size]` before NeoX rope (NOT the per-head `kAttnQkNormRope`). Reuse: SiLU SwiGLU (`kSiluAndMul`), NeoX RoPE (`RopeFromCache`/`RopeNeox`), GQA/MHA paged glue (`dense_attn_block.h`), merged qkv/gate_up loader. Checkpoint is F32 on-disk → loader downcasts f32→bf16 (round-to-nearest-even, `vt::F32ToBF16`) to match vLLM-bf16. UNTIED lm_head (`tie_word_embeddings:false`). Loader 179 tensors, zero missing/unmapped. Tokenizer: ONE shared-TU touch — `tokenizer.cpp` accepts the OLMo-2 Split `behavior=Removed,invert=true` encoding (equiv. to `Isolated` for the full-cover cl100k regex → `kLlama3`); guarded additive branch, no other checkpoint uses it. NO-BOS-verified real ByteLevel gate. Runs EAGER (bf16, no decode graph). Gate: 16/16 vs vLLM 0.25.0 (STRICT 13/16 + near-tie 3/16, max gap 0.094 nats, 0 forward-divergent) — speed pending. **OLMo-3 W5 (batch3) IMPLEMENTED** (guarded additive edits to `olmo2.{h,cpp,weights}`, diff-inert for OLMo-2 — re-run OLMo-2 gate 16/16 UNCHANGED): per-layer interleaved routing off `config.layer_types` — sliding_attention layers use plain NeoX rope (theta 500000) + finite window (masked at the FA kernel, inert for short contexts), full_attention layers use a precomputed YaRN cos/sin cache (get_rope yarn: factor 8, original 8192, mscale=yarn_get_mscale(8)=1.2079=config attention_factor, indexed by real positions); dtype-aware loader (OLMo-3 is BF16 on-disk vs OLMo-2 F32). **BUT the pinned vLLM 0.25.0 oracle CANNOT run `allenai/OLMo-3-1025-7B`** — its transformers version predates OLMo-3's nested per-layer-type rope schema: `olmo2.py:143` does `rope_parameters["rope_theta"]` → `KeyError: 'rope_theta'` (top-level rope_theta not folded into standardized rope_parameters), and forcing it surfaces `Unrecognized keys {'sliding_attention','full_attention'}` → `TypeError: unhashable type: 'dict'`. So there is NO pinned-oracle SACRED bar for OLMo-3 (DEP-blocked, spec D5); our engine LOADS + RUNS it (bootstrap loaded clean). W5 SACRED gate pending an oracle that can construct OLMo-3's rope config. | ✅ [sweep-olmo2](specs/sweep-olmo2.md) | `PARTIAL` | `test_olmo2_paged_engine` 16/16 (dgx, 92 assertions, batch3 re-run UNCHANGED); `test_olmo3_paged_engine` present (oracle-blocked, skips); loader 179 tensors zero-unmapped; registry resolves both arch strings; registration `src/vllm/model_executor/models/olmo2_registry.cpp:126`; test `tests/vllm/models/test_model_registry.cpp:118` | `CLAIM-SWEEP-OLMO2` (Claude Code opus-4-8) |
 | `MODEL-TEXT-olmo-hybrid-olmo-hybrid-for-causal-lm` | `OlmoHybridForCausalLM` | `registry.py:174`; `vllm/model_executor/models/olmo_hybrid.py::OlmoHybridForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; Mamba/SSM state; GDN/linear-attention state | ☐ required | `INVENTORIED` | none | unassigned |

.agents/specs/nemotron-h-a2q1-fp8-mamba.md

@@ -418,23 +418,68 @@ stated removal condition is the device/paged runner path, which is A2-P's.
 
 ## 10. Now
 
-**State at this commit:** spec only. No product code, no lifecycle change.
-
-A2-Q1 is **BLOCKED on #960** and is not claimable until it lands. When it does, a
-fresh implementer claims this file, captures a RED per §5.3 first, and lands the
-mamba device arm with G-SAFE untouched. A fresh reviewer — never the implementer
-— runs the §5.3 mutations.
-
-**Three things to read before the first edit:** §3, so the Thor dependency is
-consumed rather than re-derived; §1.1, so the conv-state dtype is left to A2-P;
-and §5.2, so the bands are measured rather than chosen.
+**State at this commit:** the device arm is IMPLEMENTED and REACHED, and its
+binding measurement is PENDING a lease.
+
+`NemotronHMamba2MixerDevice` (`nemotron_h_device.cpp`) runs the whole block on
+the device on the shared FP8 W8A8 seam, and BOTH forwards select it at runtime:
+`NemotronHDeviceForward` (non-paged, discarding the recurrence) and
+`NemotronHPagedForward` (production, advancing the gathered recurrent rows in
+place). The selection is a runtime op-table query plus a weight-form predicate,
+never a preprocessor guard, so a dense NemotronH or a device without the fp8 pair
+keeps the host bounce.
+
+**Three design points differ from §4 and the reasons are recorded here rather
+than left to be re-derived:**
+
+1. **The `Fp8Weight` pair is built ON FIRST DEVICE USE, not by the loader**
+   (§4.1/§4.2 put it in the loader). Building it in a `ResidentSlot` the weights
+   own is A2-Q2a's newer idiom and it is strictly better here: it does not double
+   the 890 MB fp8 tower in host memory at load, it does not upload anything on a
+   host-only run, and it leaves `rep.host_bytes` — the literal `18888922112` that
+   `test_nemotron_h_loader.cpp:310` pins — untouched, so §4.2's re-derivation
+   obligation does not arise at all. The e4m3 staging copy is released as soon as
+   `ResidentFp8` has uploaded it, so the peak cost of the conversion is one
+   projection.
+2. **The upload IS accounted** (§4.4 expected the report to say it was short).
+   `dense_fp8::ResidentFp8` still does not call `load_stats::AddDeviceUpload`
+   — that is [#974](https://github.com/mudler/vllm.cpp/issues/974), unchanged, and
+   the shared header is not touched — so A2-Q1 accounts what IT uploads at the
+   site that causes it, exactly as `ResidentWeight` and `ResidentNvfp4` do. The
+   counter is then also the instrument the residency case reads, because an arm
+   that re-uploaded the tower every step returns identical numbers to one that
+   uploads it once.
+3. **The paged selection carries an `ssm_dtype == f32` term.**
+   `vt::GdnStateGather` widens the page into an f32 working buffer by op
+   contract, and the HOST arm then narrows it back to `ssm_dtype` before the
+   mixer sees it. On a checkpoint whose `mamba_ssm_cache_dtype` is not f32 the
+   two arms would round differently and the per-block numeric gate would be
+   comparing two different computations. The released checkpoint resolves f32.
+
+**The comparison is NOT bit-comparable by construction, and §5.2's "measure the
+band" is therefore binding rather than cautionary.** The host reference is
+W8A16 — `DenseFor` dequantizes the fp8 weight to bf16 and leaves the activation
+alone, as `DenseBf16` says outright — while the device arm is W8A8 as vLLM is.
+The difference between them is the e4m3 activation quantization, and every band
+in `tests/vllm/models/test_nemotron_h_mamba_device.cpp` is measured in the run
+against a defect the fixture separates.
+
+**What is still owed, and it is the acceptance test rather than a formality:**
+the §5.1 per-block numeric gate on the real checkpoint on both hosts, the A3
+`96/96 mode=decode STRICT PASS` re-run, the §5.3 mutations, and the GPU busy
+fraction — which must RISE from the measured 6.31% baseline, sampled with its
+denominator reported. `scripts/nemotron-h-a2q1-dgx-gate.sh` is the recipe.
 
 ## 11. Owed
 
-- [#974](https://github.com/mudler/vllm.cpp/issues/974) — the fp8 resident
-  helpers upload without `AddDeviceUpload` / `AdoptDeviceBytesAsHost`. A2-Q1
-  consumes them and reports its accounting as short (§4.4); the fix is not this
-  unit's.
+- [#974](https://github.com/mudler/vllm.cpp/issues/974) — `dense_fp8::ResidentFp8`
+  uploads without `AddDeviceUpload` / `AdoptDeviceBytesAsHost`. A2-Q1 consumes it
+  and accounts its own upload at its own call site (§10.2); the fix INSIDE the
+  shared header is not this unit's, and every other caller of the seam is
+  byte-unchanged.
+- The §5.1 real-checkpoint per-block numeric gate, the §5.3 mutation pass and the
+  GPU-occupancy measurement are PENDING a GB10 lease. They are the unit's
+  acceptance test, not paperwork.
 - The device arm has **no production caller** until A2-P wires it through
   `ModelRegistry::Forward` (§7). Tracked on
   [#810](https://github.com/mudler/vllm.cpp/issues/810).

docs/BENCHMARKS.md

@@ -11,6 +11,7 @@
 | **Container images (ACTIVE; arm64 cuda verified on GB10 + Orin 2026-08-11)** | `ENG-RELEASE-CONTAINERS` ([spec](../.agents/specs/container-images.md)) | cpu amd64 783 MB; cuda arm64 **1.71 GB**. GB10 `sm_121a`: `/health`+`/version`+SIGTERM on `--gpus all`. Orin `sm_87` (Tegra): Qwen3-0.6B **generates**, GPU **GR3D 95-97%** | n/a |
 | **Developer/row protocol** | Contribution entry point; `ENG-NOW-DERIVED` #374 @`dbd0d51c` | Entry-point gates retained. #374 W1-W5 DONE; benchmark/runtime/parity `VOID`; row specs now carry `## Now` | n/a |
 | **NemotronH paged forward** (`MODEL-NEMOTRON-H-ABI-A2P`, [#810](https://github.com/mudler/vllm.cpp/issues/810)) | **No speed number, by the unit's own rule** ([spec](../.agents/specs/nemotron-h-a2p-paged-forward.md) §5) | **A3 host gate PASSES 96/96 `STRICT PASS`.** GB10 read 4/24; cause and fix [#1157](https://github.com/mudler/vllm.cpp/issues/1157), sm_121a re-run pending a lease | CPU gate 12/12. Load 209.0 s, peak RSS 20 142 392 KB |
+| **NemotronH FP8 mamba on device** (`A2-Q1`, [#810](https://github.com/mudler/vllm.cpp/issues/810)) | GB10 sm_121a, released 20.1 GiB checkpoint, c1 greedy | **PENDING a lease.** The unit's acceptance test is the GPU busy fraction rising from the 6.31% baseline, not a ratio, and it is unmeasured here | Host arm dequantized 890e6 fp8 elements per token; `VT_NEMOTRON_H_DEVICE_MAMBA=0` is the same-binary A/B |
 | **LoRA runtime W2** (`LORA-RUNTIME`, #278) | **No number owed:** correctness-only; a grid PENDS the W7 model gate |
 | **ARCH audit: ABI is text-only** | 4 capabilities (H3 video, Laguna, Kimi-Linear, DeepSeek-V4) reachable only from `examples/`, none registry-backed. No gate asks whether a CONSUMER can reach a capability. Documentation only |
 | **CUDA-graph break seam W1** (`ENG-CUDAGRAPH-BREAK`, [#1192](https://github.com/mudler/vllm.cpp/issues/1192)) | **No number owed, and none taken:** coverage and correctness row, no throughput gate declared | Capability, not a rate: mid-forward capture re-begin holds on a leased GPU; scoped forward matches eager, 500 logits, 0 differing. Probe committed, recipe and sha256 in the [record](../.agents/benchmark-record.md) |

docs/ENVIRONMENT.md

@@ -242,6 +242,7 @@ Read-only observability; none change output.
 | `VT_SERVER_PREFILL_PROGRESS` | off | `=1` prints chunked-prefill progress to stderr, rate-limited to roughly 2 Hz per request. `=0` explicitly disables it even when `VT_SERVER_VERBOSE=1` |
 | `VT_NEMOTRON_H_DIAG` | off | `=1` prints one line per step and per layer of `NemotronHPagedForward` to stderr: the decode/prefill split, the recurrent slot indices and the has-initial mask, then per Mamba2 layer the L2 norm of the state gathered IN and the state written OUT, and per layer the last row's mixer and residual norms. It is what separated a broken recurrent carry from a stale input id on this model ([#1157](https://github.com/mudler/vllm.cpp/issues/1157)): the carry read identical on host and GB10 while layer 0's embedding row did not. Every download it does is inside the guard, so an unset run pays nothing |
 | `VT_NEMOTRON_H_DEVICE_MOE` | on | `=0` routes NemotronH's NVFP4 MoE layers back through the host reference instead of the device Marlin arm. A bisect lever for a device-vs-host divergence on this model, not a configuration: it trades the whole MoE tower's throughput for the arm the CPU gate proves token-exact |
+| `VT_NEMOTRON_H_DEVICE_MAMBA` | on | `=0` routes NemotronH's 23 FP8 W8A8 Mamba2 layers back through the host reference instead of the device arm ([#810](https://github.com/mudler/vllm.cpp/issues/810) A2-Q1). The same-binary A/B every measurement of that arm needs, and a bisect lever for a device-vs-host divergence: with it off the identical binary takes the host bounce, in which each projection is dequantized from fp8 to bf16 on every call, so a throughput or GPU-occupancy difference is attributable to the arm and not to a rebuild |
 | `VT_GDN_VALIDATE` | off | Run the GDN validation/cross-check path (slower; for kernel debugging) |
 | `VT_FP4_AUTOTUNE_VERBOSE` | off | Log the NVFP4 GEMM autotuner's tactic selection |
 | `VT_H3_PROGRESS` | unset | Trace the MiniMax-H3 denoise loop's phases to stderr: which forward path was taken (device vs the CPU reference), how long the ONE-TIME device weight staging took, and per-step forward seconds with the sequence length. A real-checkpoint run spends its minutes in exactly one of those phases, and this says which without guessing — it was added after GPU-utilization counters proved unreliable on Tegra-class boards |