From 4ecfe77e29cb57ddcddd2897f6ba28dd5f5299dd Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 08:06:05 +0000 Subject: [PATCH 1/5] spec(MODEL-NEMOTRON-H): the paged runner rebuilds the recurrent half from a config NemotronH does not ship (#810) The weights load and then ENGINE CONSTRUCTION refuses, on the real 21 GiB checkpoint on GB10, with a message naming the wrong model: vllm_engine_load: vt: runner: Qwen3.5 MambaSpec shapes disagree with model config at src/vllm/v1/worker/gpu/runner.cpp:525 `GPUModelRunner::initialize_kv_cache` has two halves with opposite polarity. The attention half is spec-driven and its own comment at runner.cpp:539-560 says why -- "upstream never reconstructs the KV shape from the HF config". The recurrent half rebuilds the geometry from `config_.linear_*` (:494-507), fields NemotronH does not ship, and uses the model's published MambaSpec only as an assertion oracle (:523-527). Per-layer membership is a Qwen3.5 string (:660-662), so with an empty `layer_types` all 52 layers are allocated as full attention: zero recurrent buffers and ~8.7x the pages the 6 real GQA layers need. This spec is committed before implementation and writes no product code. Three things it establishes rather than asserts. The upstream anchor settles the polarity: vLLM enumerates the INSTANTIATED modules out of the static forward context (gpu_model_runner.py:7786) and asks each one for its own spec (:7801); the runner allocates untyped int8 pages (:7312, :7430) and the LAYER creates the typed conv/SSM views (abstract.py: 38-43). The upstream runner literally cannot perform our cross-check, because it never holds a conv shape. Group membership is `layer_names`, fanned out at :2548-2549 -- which our `KVCacheGroupSpec::layer_names` mirrors and our runner never reads. The conv-state questions are answered from the pin rather than deferred. Our (dim, state_len) is upstream's DS layout, an env-selectable first-class mode (mamba_utils.py:152-157), not a divergence -- and upstream ships the byte equality test to port. Conv state is NOT f32 upstream: it carries the cache dtype (:96-107), which makes the registry's bf16 a correct mirror and makes the local f32 requirement in `vt::CausalConv1dFwd` the thing that has to move. The existing gate asserts this row's claim and its shape half is INERT. test_runner.cpp:505 is titled "runner: MambaSpec is the allocation source of truth" and passes on a tree where it is not, because MakeConfig() and MakeKvConfig(c) derive both sides from the same config. Its DTYPE half is genuinely armed, which is the shape of the repair. That asymmetry is the RED-first anchor: a subcase whose spec shapes are not derivable from the config reds today at :525. The scope decision is three work items in two landable units, and it is driven by one safety constraint. Neutering the check ALONE makes vllm_engine_load succeed and reach a forward that consumes three of ModelForwardInput's eighteen fields -- ignoring attn_kv, gdn_state, gdn_meta, gdn_state_slots and num_reqs. So fixing the allocation without the forward is strictly MORE dangerous than today's refusal, and the spec makes that a hard gate rather than advice: the allocation fix lands with a by-name refusal in the forward, narrowed and never deleted until every clause it guards is implemented. The in-tree idiom to mirror already exists at kimi_linear_registry.cpp:99-102. Also recorded: #775's unconditional static_cast is in the exact function this work rewrites and keeps main red on sanitize-cpu, so it is repaired in flow with the class filed separately; the Kimi-Linear config synthesis at hf_config.cpp:484-528 exists so this same check passes, and a third such branch is forbidden rather than tempting; and the e2e stop condition is the committed 32-token goldens through include/vllm.h with VT_NEMOTRON35_SNAPSHOT unset, driven by an ABI-only example modelled on kimi_linear_gen, never on the allowlisted deepseek_v4_gen/laguna_gen shape. Gate: `scripts/agent-preflight.sh` All gates green, disk 67G free / 85%. The `test_cpu_x86_llamacpp_floor` red seen on an earlier pass is environmental and base-inherited (#618): it reproduces on the CLEAN shared checkout at main with zero modifications, NO_QUIET_WINDOW exit 4 at load average 66.8. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 1 + .agents/specs/nemotron-h-abi-e2e.md | 914 ++++++++++++++++++++++++++++ 2 files changed, 915 insertions(+) create mode 100644 .agents/specs/nemotron-h-abi-e2e.md diff --git a/.agents/issue-index.md b/.agents/issue-index.md index c30465bd0..e26c36e38 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -209,3 +209,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#776](https://github.com/mudler/vllm.cpp/issues/776) | `GATE-OP-PARITY-MANIFEST` | `test_op_parity` THREW `json.exception.type_error.302` out of the CPU golden pass instead of failing an assertion, so the walker's `no runner for op` guard — the check that caught #559's missing runner arm — stopped running for every golden after the offender. The artifact was `tests/parity/goldens/minimax_music3_oracle/manifest.json`, and that half is #755, already fixed by `043e56862`. #755 closed the walker's INPUT set; it did not close its EXCEPTION surface, and reproducing #776 on the fixed tree shows the difference: nulling one tensor `dtype` in `rmsnorm_f32_8x128` still threw `type_error.302` at the TEST_CASE line and cut the pass from 142 assertions to 37, leaving 45 committed goldens unchecked. Both remaining throw sites — `json::parse` and any runner field read — now funnel through `GuardGoldenStage`, which turns a `std::exception` into a `FAIL_CHECK` naming `goldens//manifest.json` and continues. `doctest::detail::TestFailureException` is deliberately not a `std::exception` (`third_party/doctest/doctest.h:2563`), so the #559 `FAIL` still aborts loudly and the widening cannot mute it. FIXED IN FLOW | bug | | [#740](https://github.com/mudler/vllm.cpp/issues/740) | `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` | MoE loader: bf16 / 3-D-stacked routed experts — the arm every published Qwen MoE repo ships, and the one Qwen3.8 needs | feature | | [#861](https://github.com/mudler/vllm.cpp/issues/861) | `GATE-SQUASH-SEPARATOR` | `PR_BODY` did not remove the `---------` separator: GitHub writes it above the `Co-authored-by:` block it appends, under both settings, so `617d6f452` landed failing the trailer gate with ONE block and one marker. #829 and #850 concluded otherwise from a simulation that omitted it. The fuse now steps over the separator when trailers sit on both sides (spec [`squash-separator.md`](specs/squash-separator.md)) | bug | +| [#810](https://github.com/mudler/vllm.cpp/issues/810) | `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | NemotronH is not reachable through `include/vllm.h`: `GPUModelRunner::initialize_kv_cache` rebuilds the RECURRENT half of the allocation from `config_.linear_*` instead of the `MambaSpec` the model published, so every non-Qwen3.5 hybrid is refused by Qwen3.5's name at `runner.cpp:525`, and per-layer membership comes from `config_.layer_types[l] == "linear_attention"` rather than `KVCacheGroupSpec::layer_names`. Spec [`nemotron-h-abi-e2e.md`](specs/nemotron-h-abi-e2e.md); the attention half at `runner.cpp:539-607` is already spec-driven and is the model to mirror. Note the SAFETY constraint recorded there: neutering the check alone reaches a forward that ignores `attn_kv`/`gdn_state`/`num_reqs`, which is strictly more dangerous than the refusal | bug | diff --git a/.agents/specs/nemotron-h-abi-e2e.md b/.agents/specs/nemotron-h-abi-e2e.md new file mode 100644 index 000000000..7e174fc96 --- /dev/null +++ b/.agents/specs/nemotron-h-abi-e2e.md @@ -0,0 +1,914 @@ +# NemotronH end-to-end through the public ABI — the runner's recurrent half is Qwen3.5-shaped + +**Issue:** [#810](https://github.com/mudler/vllm.cpp/issues/810). +**Parent row:** `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` +([#517](https://github.com/mudler/vllm.cpp/issues/517), parent spec +[nemotron-h-model.md](nemotron-h-model.md)) — this spec owns what that spec's +§6e carried forward as item 3 and explicitly put out of its own scope. +**Also repaired here:** [#775](https://github.com/mudler/vllm.cpp/issues/775) — +see §7 R2. +**Base:** `origin/main` @ `2e9d95e74c7aee133e21771182a6a587fe74c67b`. +**Pinned oracle:** `${VLLM_SOURCE}` = `/home/mudler/_git/vllm` @ +`5559679229bc961848b121ccdeaa8fa5d79bec98` (vLLM 0.26.0.dev0), verified at HEAD +while writing this spec, per [upstream-sync.md](../upstream-sync.md). +**Weight loader:** landed at `bc570da0d`. +**Lifecycle at this commit:** the row stays `INVENTORIED`. A spec commit changes +no lifecycle state and therefore owes no `STATUS`/`BENCHMARKS`/`NOW` write; §8 +records what the implementing changes owe. + +**No product code is written by this spec.** It is committed before +implementation, per AGENTS.md §"Spec before code". + +--- + +## 0. The established fact this spec starts from + +An independent investigation proved, on `main` with the weight loader merged, on +the real 21 GiB checkpoint on GB10: + +``` +vllm-cli: model load failed (status 2): vllm_engine_load: vt: runner: Qwen3.5 MambaSpec + shapes disagree with model config at src/vllm/v1/worker/gpu/runner.cpp:525 +``` + +The weights load (17.7 GiB RSS) and then **engine construction** refuses. + +`GPUModelRunner::initialize_kv_cache` (`src/vllm/v1/worker/gpu/runner.cpp:439`, +body to `:785`) has two halves with opposite polarity. + +**The attention half is spec-driven, and its comment says why.** +`runner.cpp:539-560` records that *"upstream never reconstructs the KV shape from +the HF config: every layer's cache bytes come from its SPEC"*, and names the +defect it was written to remove — *"we previously hardcoded `num_blocks * 2 * +block * Hkv * Dh` with Hkv/Dh read from `config_`"*. The implementation at +`:566-607` takes `block_size`, `num_kv_heads`, `head_size`, `dtype` and +`page_size_bytes()` off the `AttentionSpec`. + +**The recurrent half is config-driven and uses the spec only as an assertion +oracle.** `runner.cpp:494-507` rebuilds the geometry from HF-config fields +NemotronH does not ship: + +```cpp +const int64_t Hk = config_.linear_num_key_heads; // :498 -> 0 +const int64_t Kw = config_.linear_conv_kernel_dim; // :502 -> 0 +const int64_t conv_dim = 2 * key_dim + value_dim; // :505 -> 0 +const int64_t conv_state_len = ... (Kw - 1) ...; // :507 -> -1 +``` + +and then, at `:523-527`, refuses when the model's own published spec disagrees +with that reconstruction: + +```cpp +VT_CHECK(mamba_spec->shapes[0] == expected_conv_shape && + mamba_spec->shapes[1] == expected_ssm_shape, + "runner: Qwen3.5 MambaSpec shapes disagree with model config"); +``` + +`MakeNemotronHKVCache` (`src/vllm/model_executor/models/nemotron_h_registry.cpp:155-217`) +correctly published `{6144, 3}` and `{64, 64, 128}` at `:204-215`. + +Note what the surrounding lines already get right: `:528-529` takes the state +**dtypes** off the spec, and the GDN allocation at `:663-679` sizes buffers with +**shapes from the config and dtypes from the spec**. So the file is already +half-converted; A1 finishes a conversion the attention half completed. + +**The comment at `:494-497` claims the polarity the code does not have.** It +states that *"as in upstream, MambaSpec is the source of truth for the recurrent +tensors' order, shapes, dtypes, and page bytes"*. That is the intended design and +it is what §2 confirms upstream does. The code validates against config-derived +numbers instead. A1 makes the comment true. + +### 0.1 Per-layer membership is a Qwen3.5 string + +`runner.cpp:660-662`: + +```cpp +const bool is_gdn = + has_mamba_group && !config_.layer_types.empty() && + config_.layer_types[static_cast(l)] == "linear_attention"; +``` + +NemotronH's `layer_types` is empty, so all **52** layers are treated as full +attention: zero recurrent buffers, and ~8.7x the attention pages actually needed +(52 against the 6 real GQA layers at indices {5, 12, 19, 26, 33, 42}). + +`KVCacheGroupSpec::layer_names` (`include/vllm/v1/kv_cache_interface.h:343`) is +already populated by `MakeNemotronHKVCache` with the real per-layer names +(`nemotron_h_registry.cpp:178`, `:206`). **`initialize_kv_cache` never reads +it** — the runner keys purely on `kind()` and layer index. Group membership is +the durable signal; a config spelling is not. + +There are now **three** independent per-layer-hybrid signals in this tree that +must agree and are never cross-checked: `config_.layer_types[l]` in the runner, +`layer.is_linear_attention` — a *weights* flag — in the Qwen3.5 forward +(`src/vllm/model_executor/models/qwen3_5.cpp:7053-7065`), and +`KVCacheGroupSpec::layer_names` in the spec. A1 does not unify all three, but it +moves the runner onto the one the model publishes, which is the one that cannot +be absent. + +### 0.2 Why Kimi-Linear works and NemotronH does not — the anti-pattern to refuse + +Only three architectures produce a `MambaSpec` at all: Qwen3.5/3.6 +(`qwen3_5_common.cpp:82`), Kimi-Linear (`kimi_linear_registry.cpp:153`) and +NemotronH (`nemotron_h_registry.cpp:210`). Qwen3.5 passes the `:525` check +because the check *is* Qwen3.5's geometry. Kimi-Linear passes only because +`src/vllm/transformers_utils/hf_config.cpp:484-528` **synthesizes** the +`linear_*` fields and fabricates a `layer_types` vector of `"linear_attention"` / +`"full_attention"` strings out of its nested `linear_attn_config`. Its comment at +`:484-488` states the purpose outright: so that *"the runner's MambaSpec +consistency check (`runner.cpp` `expected_conv_shape`/`expected_ssm_shape`) and +its per-layer linear-attention/full-attention allocation loop see the same +geometry `MakeKimiLinearKVCache` declares"*. + +There is no equivalent synthesis for NemotronH, and **this spec forbids adding +one.** Its config ships `layers_block_type`, `mamba_num_heads`, `mamba_head_dim`, +`ssm_state_size` and `conv_kernel`, and no `layer_types` and no `linear_*`. A +third synthesis branch would work, and it is how this defect reaches model number +four. The model already published the truth; the runner should read it. + +### ★ THE SAFETY CONSTRAINT — it shapes every scope decision below + +Neutering **only** the `VT_CHECK` at `runner.cpp:525` makes `vllm_engine_load` +**succeed** and reach the forward. `ForwardNemotronHForCausalLM` +(`nemotron_h_registry.cpp:120-133`) consumes exactly **three** of +`ModelForwardInput`'s eighteen fields — `token_ids`, `logits_indices`, `queue`. It +ignores `positions`, `attn_meta`, `gdn_meta`, `attn_kv`, `gdn_state`, +`gdn_state_slots`, `num_reqs`, `pure_decode`, `num_speculative_tokens`, +`gather_logits` and the rest (struct at +`include/vllm/model_executor/models/model_registry.h:229-285`). A server past +that check would decode step 2 onward with **fresh recurrent state and no KV** — +plausible tokens, wrong tokens, no refusal. + +**Fixing the allocation without the forward is strictly MORE DANGEROUS than +today's refusal.** The refusal is currently the only thing making the gap +visible. + +This is a **hard gate on this spec, not advice**: + +> **G-SAFE.** No change may land that removes or weakens the `runner.cpp:525` +> refusal unless the same commit either (a) lands the device/paged forward that +> consumes `attn_kv` / `gdn_state` / `gdn_meta` / `gdn_state_slots` / `num_reqs`, +> or (b) installs a by-name refusal in `ForwardNemotronHForCausalLM` for any +> step where `attn_kv` or `gdn_state` is non-empty, or `num_reqs > 1`. The +> refusal names the architecture and the missing piece, per AGENTS.md. +> +> A reviewer who cannot point at the interlock in the same diff returns FAIL. +> The interlock is **narrowed, never deleted**, until every clause it guards is +> implemented and gated. + +**The interlock is not novel — mirror the in-tree idiom.** +`ForwardKimiLinearForCausalLM` (`kimi_linear_registry.cpp:88-113`) already gates +its paged fold on `!input.attn_kv.empty() && !input.gdn_state.empty()` at +`:99-102`, keeping the non-paged seam alive below it. That is structurally the +same predicate, inverted. Copy its shape rather than inventing one. + +--- + +## 1. Scope decision — three work items, two landable units + +**Decision: SEVERAL, not one.** The four candidate slices (S1 allocation, S2 +per-layer routing, S3 the device/paged forward, S5 the ABI token gate) do not +split four ways, and they do not collapse into one either. + +### A1 — the shared-runner refactor plus the safety interlock + +**S1 and S2 are ONE unit, not two.** They are the same refactor of the same +function, and neither is separately testable on the driver model: + +- S1 without S2 computes correct conv/SSM shapes for **zero** layers — with + `layer_types` empty, `:660-662` classifies every layer as full attention, so + nothing consumes the corrected shapes. The change would be green and inert. +- S2 without S1 still trips the `VT_CHECK` at `:525` before the routing loop at + `:629-722` is reached. The change would be unreachable. + +They fail together, they are proven by the same test pair, and splitting them +manufactures a commit that cannot be gated. **A1 = S1 + S2 + the G-SAFE +interlock + the #775 repair, in one commit.** + +| In A1 | Out of A1 | +|---|---| +| conv/SSM shapes, dtypes and per-slot bytes read from `mamba_spec->shapes` / `->dtypes`, mirroring `runner.cpp:539-607` | any change to `dense_attn`, the block table, the scheduler, or the KV manager | +| the config cross-check at `:523-527` **deleted**, not widened; the three `Qwen3.5`-named messages at `:516`, `:525`, `:535` renamed to name the architecture | any new HF-config field, and specifically **no** NemotronH branch in `hf_config.cpp` (§0.2) | +| per-layer recurrent membership derived from `KVCacheGroupSpec::layer_names` | the device forward (A2) | +| the G-SAFE by-name refusal in `ForwardNemotronHForCausalLM` | the MTP head (parent W5) and GGUF (parent W7) | +| the #775 `static_cast` repair at `nemotron_h_registry.cpp:122` (§7 R2) | a tree-wide sweep of the same cast class (§7 R2) | +| a by-name refusal in `MakeKVCacheMaybeSpec` for a speculative non-Qwen3.5 config (§1.2) | making speculative NemotronH actually work | + +**What A1 unblocks:** every hybrid whose config does not speak Qwen3.5's +`linear_*` / `layer_types` dialect can allocate, without a per-model synthesis +branch. It is a precondition for A2 and for any future Mamba2 architecture. + +### A2 + A3 — the device/paged forward and its e2e proof, one PR + +**S3 and S5 land together.** A device forward with no e2e gate is unfalsifiable, +and an e2e gate with no device forward has nothing to run. `vllm_complete_tokens` +already exists at `VLLM_ABI_VERSION 20` (`include/vllm.h`, added at v13), so S5 +needs **no ABI growth** — there is no separable "grow the surface" task to carve +out. Keeping them in one PR also means the commit that deletes the G-SAFE +interlock is the commit that proves it is no longer needed. + +**S3 closes four gaps at once**, which is why it is one item and not four: + +1. **No paged attention.** `NemotronHAttentionMixer` + (`src/vllm/model_executor/models/nemotron_h.cpp:585-630`) recomputes Q, K and + V from scratch on every call — `:611-613` — then runs one dense causal + `vt::Attention` over the whole `[T, ·]`. No `PagedKvCache`, no slot mapping, + no `ReshapeAndCache`. `NemotronHGreedyDecode` (`:970-992`) therefore runs a + full forward over the growing sequence per generated token. The fp8 KV scales + the loader materialized are documented as unused for exactly this reason + (`nemotron_h_forward.h:207-216`). +2. **No batching.** `token_ids` is the step's concatenated scheduled tokens + **across requests**, and the forward treats it as ONE causal sequence. Two + concurrent requests silently cross-contaminate — a correctness defect that + produces fluent output. +3. **The CPU-queue-only `VT_CHECK`** at `nemotron_h.cpp:822-825`, inside + `NemotronHForward`, which names this spec's predecessor as its owner. +4. **`PrepareNemotronHForCausalLM` is a pure no-op** + (`nemotron_h_registry.cpp:113-118` — three `(void)` casts), so nothing warms, + sizes or validates per-step state. + +Each is a symptom of the same thing — the forward is a host reference that owns +its own K/V and its own state — so repairing one without the others produces a +half-paged forward harder to reason about than either end state. + +**Permitted split, if a reviewer would be better served by parts** (AGENTS.md: +size is a review judgement, not a counter): A2 may split into **A2a** +(single-request paged decode; `num_reqs > 1` still refused) and **A2b** +(batching). The rule that makes this safe is the one already stated: at A2a the +interlock is **narrowed** to the `num_reqs > 1` clause, not removed. A2a with +the interlock deleted is not an acceptable split. + +### 1.1 Dependency order + +``` +A1 (runner refactor + interlock + #775) ──► A2 (device/paged forward) ──► A3 (ABI e2e token gate) + │ │ └── same PR as A2 + │ ├── requires KERNEL-SSM-MAMBA (#496) W2, the CUDA arm + │ └── requires the conv-state dtype reconciliation (§5.3b) + └── requires nothing beyond main + bc570da0d +``` + +A1 is independent of #496 and can land immediately. A2 cannot: the parent spec's +§0 hard blocker holds — the CUDA Mamba2 SSD arm is W2 of +[#496](https://github.com/mudler/vllm.cpp/issues/496) +([spec](mamba2-ssd.md)). **Re-verify #496 W2's state against the current head and +the current issue before claiming A2; do not implement against a described +dependency** ([`porting.md`](../porting.md)). + +### 1.2 `MakeKVCacheMaybeSpec` — decided: refusal in A1, repair owed to parent W5 + +`src/vllm/entrypoints/model_loader.cpp:835-846` hard-routes **any** speculative +config to `MakeQwen3_5KVCacheSpec` at `:842-843`, unconditionally on +architecture. All four speculative methods reach it (`mtp`, `dflash`, `ngram`, +`dspark`). Same defect class as #810 — a Qwen3.5-shaped assumption in shared +code — but not on A1's critical path, because no speculative NemotronH config +can exist until the parent spec's W5 (the MTP head) lands, and +`MakeNemotronHKVCache` deliberately passes no `num_speculative_blocks` +(`nemotron_h_registry.cpp:202-203`). + +**Decision: in scope for A1 as a refusal only.** A1 refuses a speculative config +for an architecture that is not Qwen3.5, naming the architecture and the missing +piece, rather than silently handing it Qwen3.5 KV specs. That is cheap, it is +the AGENTS.md rule for an unimplemented arm, and it converts a silent-wrong-answer +path into visible debt. **The real repair — routing a speculative config through +the model's own KV-cache factory — is owed to parent W5** and is out of scope +here. Record it in the parent spec's W5 section when A1 lands. + +### 1.3 Adjacent debt found while scoping — recorded, not fixed + +`runner.cpp:483-491` recognizes only `kFullAttention`, `kMlaAttention` and +`kMamba`. A `SlidingWindowSpec` or `ChunkedLocalAttentionSpec` group would match +nothing and leave `full_attn_group_id_ == -1`. No registry KV builder produces +either today (`ChunkedLocalAttentionSpec` is constructed only at +`src/vllm/model_executor/layers/attention/chunked_local_attention.cpp:143`), so +this is latent, not live. **Out of scope. File an issue when A1 is claimed** — +AGENTS.md requires an issue for a bug found in flow, and this one is not +fixable in flow because no model exercises it. + +### 1.4 Explicitly out of scope + +Speed (no ratio is recorded by this row); the MTP head; the GGUF k-quant arm +(parent §5b/W7, still refused by name); `NemotronHPuzzleForCausalLM`; TP +sharding of `n_groups`; any change to the Qwen3.5 or Kimi-Linear model files. + +--- + +## 2. Upstream anchors — `file:line` on both sides, at `555967922` + +Verified against the checkout at the pin (`git rev-parse HEAD` → +`5559679229bc961848b121ccdeaa8fa5d79bec98`). **Nemotron-H is still in the old +layout at this pin**: `vllm/model_executor/models/nemotron_h.py`, registered at +`vllm/model_executor/models/registry.py:179`. The newer `vllm/models//` +package holds only `deepseek_v32`, `deepseek_v4`, `inkling` and `minimax_m3` — so +an `ls` of either directory alone would have been misleading, and both were +searched. + +### 2.1 The polarity we have inverted — upstream derives the spec from the MODULES + +This is the load-bearing anchor of the whole spec. + +| Question | Upstream | Ours | +|---|---|---| +| Where does a hybrid's KV spec come from? | `gpu_model_runner.py:7774` `get_kv_cache_spec()` — docstring: *"parsing the kv cache format from each Attention module in the static forward context"* | `ModelRegistry::MakeKVCache` (`model_registry.cpp:329-334`), per-model factory | +| How are the layers enumerated? | `gpu_model_runner.py:7785-7787`: `get_layers_from_vllm_config(self.vllm_config, AttentionLayerBase)`, then `for layer_name, attn_module in attn_layers.items()` | `config_.layer_types[l]` string compare (`runner.cpp:660-662`) | +| Who produces the shape? | the **layer instance**: `gpu_model_runner.py:7801` `attn_module.get_kv_cache_spec(...)`; Mamba's is `mamba/abstract.py:63-79`, built from `self.get_state_shape()` / `self.get_state_dtype()` | the runner, from `config_` (`runner.cpp:494-507`) | +| Is the HF config ever re-read for shapes? | **only** for page-size padding, on the platform, not in the runner: `vllm/platforms/interface.py:859-860` → `cache_config.mamba_page_size_padded`, consumed back at `mamba/abstract.py:66` | yes, and it is the refusal | + +The layer registers *itself* by prefix at +`vllm/model_executor/layers/mamba/mamba_mixer2.py:496-499` +(`compilation_config.static_forward_context[prefix] = self`), and the registry +is filtered by `isinstance` at `vllm/config/vllm.py:2442-2449`. `MambaBase` is an +`AttentionLayerBase` (`mamba/abstract.py:18`) with abstract `get_state_shape` +(`:46`), `get_state_dtype` (`:60`) and `mamba_type` (`:56`). + +**No `layer_types` or `hybrid_override_pattern` string is parsed anywhere in +`gpu_model_runner.py`.** The only reads of `hybrid_override_pattern` are inside +`nemotron_h.py` at `:275`, `:572`, `:578`, `:595` — module-construction time +only, to choose a layer class from `ALL_DECODER_LAYER_TYPES` (`:531-536`, +`M`/`-`/`*`/`E`). By the time the runner asks, the pattern has already become +instantiated objects. + +### 2.2 The runner allocates BYTES; the LAYER creates typed views + +`vllm/v1/worker/gpu_model_runner.py:7311-7313`: + +```python +tensor = torch.zeros(kv_cache_tensor.size, dtype=torch.int8, device=self.device) +``` + +and the Mamba branch of the reshape, `:7429-7441`, hands the layer an untyped +page view: + +```python +elif isinstance(kv_cache_spec, MambaSpec): + page_size_bytes = kv_cache_spec.page_size_bytes + kv_caches[layer_name] = raw_tensor[: num_blocks * page_size_bytes].view( + num_blocks, 1, 1, page_size_bytes) +``` + +The unpack into typed conv/SSM views happens in the layer, +`vllm/model_executor/layers/mamba/abstract.py:38-43` (method from `:29`): + +```python +for shape, dtype in zip(self.get_state_shape(), self.get_state_dtype()): + nbytes = prod(shape) * get_dtype_size(dtype) + state = pages[:, offset : offset + nbytes].view(dtype) + states.append(state.view(-1, *shape)) +``` + +**The upstream runner literally cannot perform our cross-check** — it never +holds a conv shape. That is why `MambaSpec` carries `shapes` and `dtypes` +(`vllm/v1/kv_cache_interface.py:690-696`) and `page_size_bytes` is their sum +(`:699-703`), and why our `MambaSpec` +(`include/vllm/v1/kv_cache_interface.h:302-323`) mirrors those fields and +deliberately carries **no** `num_heads` / `head_dim` / `conv_dim` — the geometry +is implicit in `shapes`, exactly as upstream. A1 stops re-deriving what the +mirror deliberately did not store. + +Hybrid layout reconciliation (attention blocks-first vs K/V-first when a mamba +group coexists) is `gpu_model_runner.py:7443-7449` and +`_update_hybrid_attention_mamba_layout` at `:7477-7509` — read before changing +our tensor views. + +### 2.3 Group membership and per-layer routing + +- Grouping key is the **spec object**, `vllm/v1/core/kv_cache_utils.py:1209-1211`: + `same_type_layers[layer_spec].append(layer_name)`; entry point + `get_kv_cache_groups` at `:1760`, hybrid path + `_get_kv_cache_groups_uniform_page_size` at `:1140`. +- `KVCacheGroupSpec` is `layer_names: list[str]` + `kv_cache_spec`, + `vllm/v1/kv_cache_interface.py:938-947` — the exact structure our `:343` + mirrors and does not read. +- Backend groups are built from the modules **again**, not the config: + `gpu_model_runner.py:7024-7027`; `AttentionGroup` carries + `layer_names` + `kv_cache_group_id` (`vllm/v1/worker/utils.py:211-215`). +- Per-group metadata is built once and **fanned out by layer name**, + `gpu_model_runner.py:2548-2549`: + ```python + for layer_name in attn_group.layer_names: + attn_metadata_dict[layer_name] = attn_metadata_i + ``` + and each layer looks itself up by its own prefix, + `mamba_mixer2.py:712-713`: `attn_metadata = attn_metadata_raw[self.prefix]`. +- Mamba groups get **no token→slot mapping**, `gpu_model_runner.py:7238-7240`: + `if kv_cache_spec_kind == KVCacheSpecKind.MAMBA: + slot_mapping_modes.append(SlotMappingMode.NONE)`. + +**Mirror this:** per-layer membership comes from `layer_names`, which is what A1 +switches to. + +### 2.4 State across decode steps, and across requests — what A2 must mirror + +- **Batch reorder.** Decodes first: `gpu_model_runner.py:1126-1130` + `reorder_batch_to_split_decodes_and_prefills(...)`; the function is + `vllm/v1/attention/backends/utils.py:665`, order documented at `:574-575`. + Mamba builders declare `reorder_batch_threshold: int = 1` + (`vllm/v1/attention/backends/mamba_attn.py:87`). +- **Counts.** `mamba_attn.py:463-469` `split_decodes_and_prefills(...)` + (`utils.py:564`). +- **Per-request state index comes from the BLOCK TABLE**, not a slot map: + `mamba_attn.py:513-518` `mamba_get_block_table_tensor(...)` + (`utils.py:927-965`), then split by request class at `:523-532`: + ```python + state_indices_tensor_d, state_indices_tensor_p = torch.split( + state_indices_tensor, [num_decodes, num_prefills], dim=0) + ``` +- **Continuation detection.** `mamba_attn.py:554-556` + `has_initial_states_p = (num_computed_tokens[num_reqs - num_prefills : num_reqs] > 0)`. +- **Metadata fields** on the base: `mamba_attn.py:29-81` — + `has_initial_states_p:39`, `state_indices_tensor_p:42`, + `state_indices_tensor_d:47`, `cu_chunk_seqlen_p:66`, `last_chunk_indices_p:69`. + Mamba2 adds `prep_initial_states`, `chunk_size`, `seq_idx_p` + (`mamba2_attn.py:105-111`); the varlen chunk metadata builder is + `mamba2_attn.py:31-88`. +- **The mixer's forward**, `mamba_mixer2.py:151` / body from `:695`: + token split `:757-760`, output split `:808-812`; prefill conv `:832-847` + (`causal_conv1d_fn(..., has_initial_state=has_initial_states_p, + cache_indices=state_indices_tensor_p)`); per-request initial SSM states + gathered `:855-866`; varlen scan `:870-892` + (`mamba_chunk_scan_combined_varlen(..., seq_idx=seq_idx_p, + cu_chunk_seqlens=cu_chunk_seqlen_p, state_dtype=ssm_state.dtype)`); state + write-back `:977-978` `ssm_state[state_indices_tensor_p] = varlen_states`; + decode in-place slots `:1006-1010`; decode conv `:1013-1025`; decode SSM + `:737-742` with `state_batch_indices` / `dst_state_batch_indices`. + +**This is what "batched recurrent state" means, and none of it exists locally +yet.** A2 mirrors it; A2 does not invent it. + +### 2.5 Nemotron-H's own shapes and dtypes at the pin + +- `nemotron_h.py:761-798` `get_mamba_state_shape_from_config`: + `intermediate_size = mamba_num_heads * mamba_head_dim` (`:779`), then + `MambaStateShapeCalculator.mamba2_state_shape(...)` (`:781-790`). +- `nemotron_h.py:744-758` `get_mamba_state_dtype_from_config` → + `MambaStateDtypeCalculator.mamba2_state_dtype(model_dtype, mamba_cache_dtype, + mamba_ssm_cache_dtype)`. +- The shape expressions, `vllm/model_executor/layers/mamba/mamba_utils.py:174-199`: + ```python + conv_dim = intermediate_size + 2 * n_groups * state_size + conv_state_shape = cls._orient_conv_shape(divide(conv_dim, tp), conv_kernel - 1 + num_spec) + temporal_state_shape = (divide(num_heads, tp), head_dim, state_size) + ``` +- **Both classmethods are consumed only for page padding** (`interface.py:859-860`) + and by `nano_nemotron_vl.py`. The runtime shapes come from the layer instance: + `mamba_mixer2.py:1119-1139` `get_state_shape()`, `:1105-1117` + `get_state_dtype()`. + +### 2.6 The conv-state layout question — ANSWERED, and our orientation is a supported upstream mode + +`mamba_utils.py:23` defines `ConvStateLayoutType = Literal["SD", "DS"]`; +`:27-43` `get_conv_state_layout()` returns **`"SD"` by default** (the `return "SD"` +is `:43`; `is_conv_state_dim_first()` is `:46`), overridable by +`VLLM_SSM_CONV_STATE_LAYOUT` (`vllm/envs.py:227`, parsed `:1698-1700`). The +decisive expression is `:152-157`: + +```python +def _orient_conv_shape(dim, state_len): + """Return (dim, state_len) for DS layout, (state_len, dim) for SD.""" + if is_conv_state_dim_first(): + return (dim, state_len) + return (state_len, dim) +``` + +and the kernels want dim-major regardless, so SD is transposed on the way in — +`mamba_mixer2.py:714-721`: + +```python +conv_state = (self.kv_cache[0] if is_conv_state_dim_first() + else self.kv_cache[0].transpose(-1, -2)) +``` + +**So our local `(dim, state_len)` is upstream's `DS`** — a first-class, +upstream-supported layout, not a divergence. The note at +`nemotron_h_registry.cpp:194-200` is correct that the bytes are the same product, +and it can now be sharpened from "our local convention" to "upstream's `DS` +layout, `VLLM_SSM_CONV_STATE_LAYOUT=DS`, `mamba_utils.py:152-157`". Upstream even +has the byte-equality test to port: **`tests/v1/worker/test_mamba_utils.py:2136` +`test_ds_conv_layout_bias_gt_0_byte_equal_to_sd`**. + +**Owed by A2:** port that test, preserving its parameters and its revision +anchor, per AGENTS.md §"Port its tests in the same change". + +### 2.7 The conv-state DTYPE question — ANSWERED, and it exposes a live local conflict + +`mamba_utils.py:96-107` `_mamba_state_dtype`, reached from `mamba2_state_dtype` +at `:73-81`: + +```python +conv_state_dtype = get_kv_cache_torch_dtype(mamba_cache_dtype, model_dtype) +if mamba_ssm_cache_dtype == "auto": + temporal_state_dtype = conv_state_dtype +else: + temporal_state_dtype = STR_DTYPE_TO_TORCH_DTYPE[mamba_ssm_cache_dtype] +``` + +So **conv state is NOT f32 upstream**: it carries `--mamba-cache-dtype` +(default `auto` → the model dtype, bf16 here). Only the SSM/temporal state takes +`mamba_ssm_cache_dtype`, which this checkpoint sets to `float32`. The only +hard-coded `torch.float32` states at the pin are the ReplaySSM `dt_cache` ring +(`mamba_utils.py:93`) and KDA (`:137`) — neither applies. + +That makes `MakeNemotronHKVCache`'s `conv_dtype = kBF16` +(`nemotron_h_registry.cpp:167`) **a correct mirror**, and it makes the local +conflict explicit rather than latent: + +> `nemotron_h_forward.h:284-296` records that the host reference's conv state is +> **f32 "because `vt::CausalConv1dFwd` validates the conv state as f32"**, while +> the persistent page the registry declares is bf16 — and it says outright that +> reconciling the two belongs to the paged-decode work, i.e. **A2**. + +**Decision, and it follows from AGENTS.md rather than from taste.** vLLM resolves +one dtype and every layer inherits it; `f32` is a rare, annotated escape, and a +buffer that names `f32` on a model path owes a one-line reason. The upstream +answer is bf16. **A2 therefore gives `vt::CausalConv1dFwd` a bf16 conv-state +arm** and stores the conv state at the cache dtype; it does not widen the page to +f32 to satisfy a local kernel precondition. If that arm turns out to be +materially harder than it looks, the fallback is an f32 conv page with the +one-line reason and the cost **stated in bytes per token** in §10 — never an +unannotated widening. A token gate cannot see this either way, which is exactly +why it is decided here in the spec and not during implementation. + +--- + +## 3. RED-first, and why the existing gate cannot be trusted + +`runner.cpp` serves **every** model — every architecture served by `LoadedEngine` +flows through this one function (`model_loader.cpp:1007-1023` constructs the +single `GPUModelRunner`). A refactor of it owes a proof that it changes nothing +for the models that already work, and that proof must be armed. + +### 3.1 The existing gate asserts this row's claim and its shape half is INERT + +`tests/vllm/v1/worker/test_runner.cpp:505` is titled **"runner: MambaSpec is the +allocation source of truth"**, and it passes today — on a tree where the +MambaSpec is demonstrably *not* the source of truth for shapes. It is not lying; +its shape half is a **self-consistency gate** +([[gate-comparing-shared-helper-proves-consistency-not-correctness]]): + +- `MakeConfig()` (`test_runner.cpp:98-120`) populates every `linear_*` field and + a full `layer_types` vector; +- `MakeKvConfig(c)` derives the `MambaSpec` shapes from **that same `c`**; +- so `shapes == expected_*_shape` is two derivations of one config agreeing with + each other. It cannot fail, and the `VT_CHECK` at `:525` passes straight + through it. + +The **dtype** half of that same case *is* genuinely armed — `MakeKvConfig(c, +kBF16, kF16)` passes dtypes the config cannot produce, and `:528-529` honours +them. That asymmetry is the shape of the repair, in the same file. + +**This is the RED-first anchor and it is cheap.** Add a subcase whose `MambaSpec` +shapes are **not derivable from the config**: `linear_*` zeroed, `layer_types` +empty, spec shapes `{6144, 3}` and `{64, 64, 128}` — the real NemotronH values. +On `main` it fails at `runner.cpp:525` with `Qwen3.5 MambaSpec shapes disagree +with model config`. **Capture that red and put the transcript in the PR body.** +After A1 it is green. A test never seen failing has proven nothing +([`verification.md`](../verification.md)). + +Note what is *already* gated and needs no new coverage: the NemotronH **spec +side** is fully asserted at `tests/vllm/models/test_nemotron_h_scaffold.cpp:620-665` +— two groups, `{{6144,3},{64,64,128}}`, `{kBF16, kF32}`, all 6 + 23 real layer +names, `page_size_bytes()`, and `KVBytesPerBlock == attn page * 6`. **The gap is +that no test anywhere constructs a `GPUModelRunner` from a NemotronH spec.** +A1 closes exactly that gap and nothing else. + +### 3.2 The Qwen3.5 byte-identity arm — required, not optional + +A1 must carry a **byte-identity** arm proving the refactor changes nothing for +the existing hybrid. The precedent and the wording already exist in this tree: +`include/vllm/v1/kv_cache_interface.h:354-374` states a **BYTE-NEUTRALITY +CONTRACT** for the `per_layer_attn_specs` seam — "byte-identical allocation, +view, indexing and kernel dispatch to before this field existed". Mirror that +contract and that phrasing; a reviewer already knows how to read it. + +The arm asserts an **explicit table**, not spot checks, and it must be able to +say **how many** things it examined ([[the-state-was-not-the-one-you-believed]]): + +| Asserted, per Qwen3.5 arm | Why | +|---|---| +| `attn_kv().size()`, `gdn_state().size()`, `num_hidden_layers` | a changed count is the signal; a suite that cannot report its own N has not reported | +| the layer→group assignment for **every** layer index, as a literal vector | spot-checking layer 0 cannot see a routing inversion | +| every `conv_state` / `ssm_state` shape and dtype, per layer | that is the defect class | +| `full_attn_group_id()`, `gdn_group_id()` | group selection is what A1 rewires; the `fa_draft` case at `test_runner.cpp:468` must stay green | +| `fa_page_size_bytes()` and total allocated bytes | the byte-neutrality claim itself | + +Expected literals are derived at the **pre-refactor base SHA** and committed as +literals. Deriving them from the config inside the test reproduces exactly the +defect §3.1 describes — **the expected values must not be computed by the code +under test, nor by a helper sharing its inputs.** + +### 3.3 The NemotronH arm + +Same table, driven from a synthetic 52-layer NemotronH-shaped `KVCacheConfig` +(no checkpoint, so it runs in CI): + +- exactly **6** attention layers at indices **{5, 12, 19, 26, 33, 42}** and **23** + recurrent layers, asserted as the full index vector, not as counts alone; +- `attn_kv().size() == 6`, not 52 — with the ~8.7x page ratio stated in the test + so a regression reads as what it is; +- conv shape `{6144, 3}`, SSM shape `{64, 64, 128}`, taken from the spec; +- state dtypes taken from `spec->dtypes`, with §2.7's reconciliation applied. + +### 3.4 Mutations — each arm must be proven armed + +Applied **alone**, in a scratch copy, rebuilt, run, tree restored byte-for-byte. +**Print the compile exit code and the test-binary sha256 beside every result** — +a mutation that fails to build reads as a passing test, and that has happened +three times in one campaign here +([[mutation-build-failure-reads-as-a-passing-test]]). + +| # | Mutation | Must RED | +|---|---|---| +| M1 | conv-state second dim `- 1` in the refactored allocation | Qwen3.5 byte-identity arm **and** NemotronH arm | +| M2 | recurrent state dtype forced to `kBF16`, ignoring `spec->dtypes` | both arms | +| M3 | membership reverted to `config_.layer_types[l] == "linear_attention"` | **NemotronH arm RED, Qwen3.5 arm GREEN** — the asymmetry is the proof the refactor is behaviour-preserving for Qwen3.5. Report it as a pair; a mutation that reds both means Qwen3.5 behaviour changed | +| M4 | membership taken from group **index** instead of `layer_names` | NemotronH arm | +| M5 | the G-SAFE refusal in `ForwardNemotronHForCausalLM` replaced by a fall-through | the interlock test | +| M6 | the `MakeKVCacheMaybeSpec` speculative refusal removed | the §1.2 refusal test | +| M7 | the #775 `dynamic_cast` guard reverted to `static_cast` | the UBSan leg, under `-DVLLM_CPP_SANITIZE='address,undefined'` | +| M8 (A2) | the carried SSM state zeroed at the start of every step | the A3 multi-step token gate — see §5.2 | +| M9 (A2) | every request's state slot forced to 0 | the A3 batched token gate | +| M10 (A2) | the batch treated as one concatenated causal sequence | the A3 batched token gate | + +--- + +## 4. Blast radius — enumerate before editing + +`initialize_kv_cache` is declared at +`include/vllm/v1/worker/gpu/runner.h:386` and called from `runner.cpp:349` and +`:390` (the two real constructors; the two Qwen3.5 convenience constructors at +`:394-409` delegate). The engine builds exactly one runner +(`include/vllm/entrypoints/model_loader.h:468`, constructed +`model_loader.cpp:1007-1023`), so **every architecture flows through it.** + +Spec kinds that reach it today, and which the byte-identity arm must not disturb: + +| Spec | Producers | +|---|---| +| `FullAttentionSpec`, uniform | ~25 dense archs (`qwen3_5_common.cpp:72`, `llama_registry.cpp:129`, …) | +| `FullAttentionSpec`, per-layer via `per_layer_attn_specs` | Gemma-4 (`gemma4_registry.cpp:236-241`) | +| `MLAAttentionSpec` | DeepSeek-V2/V4, Kimi-K3, Kimi-Linear, MiniCPM3, GLM4-MoE-Lite | +| `MambaSpec` | **only 3**: Qwen3.5/3.6, Kimi-Linear, NemotronH | + +This same function was already generalized once, when the first dense model +forced it to stop assuming the Qwen3.5 hybrid topology; `test_runner.cpp:1197-1199` +records that both of the resulting cases *crashed* pre-generalization — one on +the empty-`layer_types` out-of-bounds index that #810 is the second instance of. +**A1 is the third instance of that generalization.** Treat it as a known class. + +--- + +## 5. Gates — and what each one cannot see + +Correctness first, always. No throughput number is recorded by this row. + +### 5.1 A1's gate — unit, deterministic, no checkpoint + +`ctest -R '^test_runner$'` plus `ctest -R '^test_nemotron_h'`, plus the full +`scripts/agent-preflight.sh`. It sees shapes, dtypes, counts, byte totals and +layer→group routing. **It cannot see whether the state is ever used** — that is +why G-SAFE exists, and A1's gate must never be reported as evidence that +NemotronH runs. + +### 5.2 A3's token gate — multi-step and multi-request, deliberately + +Two standing lessons bind this gate's design, and both point the same way. + +**A token gate cannot see a dtype that is too wide, or a dropped mechanism.** F32 +where upstream uses a narrower cache dtype is *more* precise: tokens match and we +quietly move twice the bytes ([`porting.md`](../porting.md)). A mechanism can be +absent while the argmax is unchanged ([`porting-a-model.md`](../porting-a-model.md) §3). + +**The SSM state dtype is unobservable with fresh single-leg state.** +`src/vllm/model_executor/models/nemotron_h_forward.h:298-301` records this, and +the converse: the dtype *"becomes observable exactly when a state is carried in, +which is what the two-leg gate exercises and what the W6 paged decode does on +every step"*. The corresponding two-leg unit gate already exists — +`tests/vllm/models/test_nemotron_h_forward.cpp:923-961`, "the SSM dtype is +independent, and it is CARRIED". So: + +> **The gate must exercise multi-step decode and multi-request batching. A +> longer prompt is not a substitute.** A single-step or single-request gate is +> structurally blind to the entire class of defect A2 introduces. + +A3 asserts: + +1. **Multi-step:** all 32 golden tokens per prompt, token-exact — not the first + token. The parent spec's §6d already matches 3/3 *first* tokens against a + forward carrying no state at all, which is exactly how little a first token + proves. +2. **Multi-request:** the three golden prompts submitted **concurrently and + interleaved**, each required to reproduce its own single-request golden. This + is the only assertion that can see gap 2 in §1 — a batch treated as one + causal sequence produces fluent, plausible, wrong output. +3. **M8 must red arm 1; M9 and M10 must red arm 2.** If M8 does **not** red the + 32-token gate, the gate is blind to the carried-state dtype and the row then + owes a **direct assertion** on the allocated state dtype and byte total + instead. Record the outcome either way; a mutation that stays green is a + coverage hole, not a pass. +4. **The memory format is checked explicitly against the oracle**, not inferred + from matching tokens: conv and SSM state dtypes and total state bytes compared + against what the pinned oracle reports for the same checkpoint, read from the + **running** engine's resolved config, not from source + ([`porting.md`](../porting.md)). +5. **Oracle identity asserted, aborting on mismatch.** Use `vllm-oracle-next`, + never `$HOME/venvs/vllm-oracle`, which symlinks to a 0.25.0 rollback + predating `NemotronHMoEDecoderLayer` and fails in a way that reads as "the + model is unsupported" (parent spec §5a). A v1 driver script needs + `if __name__ == "__main__":` or EngineCore's spawn re-imports it and the + failure names neither vLLM nor the caller. + +### 5.3 Upstream tests owed in the same change + +Per AGENTS.md, upstream's tests are ported in the same change, preserving +parameters, modes, fixtures, tolerances, failure cases and the revision anchor: + +- `tests/v1/worker/test_mamba_utils.py:2136` + `test_ds_conv_layout_bias_gt_0_byte_equal_to_sd` — the DS/SD byte-equality that + makes §2.6's claim a gate rather than an assertion. **A2.** +- `tests/v1/worker/test_gpu_model_runner.py:1137` + `test_hybrid_attention_mamba_tensor_shapes` — *"writing a mamba block will not + corrupt an attention block and vice versa"*. This is the closest upstream + analogue of A1's byte-identity arm; port its intent. **A1.** +- `tests/v1/attention/test_mamba_update_block_table.py:75` + `test_update_block_table_copies_block_idx_to_persistent_buffers` and `:178` + `test_state_indices_tensor_d_includes_num_speculative_blocks` — per-request + state indexing. **A2.** +- Note for the record: `tests/models/language/generation/test_hybrid.py` does + **not** list Nemotron-H at the pin (searched `SSM_MODELS`, `HYBRID_MODELS`, + `HYBRID_MODELS_REQUIRING_CHUNKED_PREFILL` at `:28-58`). There is no upstream + e2e hybrid case to port for this architecture; A3's gate is the committed + oracle golden instead. That is a search result with the paths named, not an + assumption. + +### 5.4 The gate hosts, and the discipline + +Both arms of A3 run on **both gate hosts**: the local x86_64 CPU-only Release +host (CPU backend) and `dgx.casa` / GB10 sm_121a (CUDA). Requiring both is not +ceremony — the same 32 tokens on both backends is also the CUDA==CPU equivalence +check, and it is free once the gate exists. + +GPU discipline on dgx, all mandatory: `flock $HOME/gpu.lock` — the lock is +`$HOME/gpu.lock`, **not** `/tmp/gpu.lock`; `local-ai-worker` parked and restored +at the end; one log per run; never a large oracle alongside `ctest`, because +`gpu_memory_utilization` reserves HOST RAM on GB10 and has OOM-rebooted the box; +CUDA `ctest` with `-j 1`. `df -h` before and after every build — an ENOSPC leaves +the previous binary in place and prints a green status +([[stale-binary-prints-green-status]], +[[enospc-makes-checkers-emit-false-policy-refusals]]). + +--- + +## 6. The e2e acceptance — this row's stop condition + +> **The row is done when `vllm_engine_load` followed by a generate call, both +> made through `include/vllm.h` and nothing else, reproduces the committed +> 32-token goldens at +> `tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json` token-exact +> for all three prompts, with `VT_NEMOTRON35_SNAPSHOT` UNSET so the revision +> check binds, on both gate hosts.** + +`VT_NEMOTRON35_SNAPSHOT` is deliberately never revision-checked +(`tests/parity/test_hf_snapshot_pinning.cpp:62`), so setting it during the gate +silently unpins the checkpoint. Unset it and **record the resolved directory** as +evidence. The golden carries the three oracle version strings, the model path, +revision `29f2d174`, and `prompt_token_ids` per prompt — so the pre-tokenized ABI +path needs no tokenizer agreement established first. + +### 6.1 The driver — `examples/nemotron_h_gen`, modelled on `kimi_linear_gen` + +**Model it on `examples/kimi_linear_gen`** (`examples/kimi_linear_gen/main.cpp`; +`examples/CMakeLists.txt:29-31` links it against `vllm::shared` and nothing +else). That example is a thin public-ABI client — `#include "vllm.h"` only, +`vllm_engine_load` + `vllm_complete_tokens` — and it is **not** listed in +`scripts/example-abi-allowlist.txt`, so a new example built the same way needs +**no allowlist row**. Its header comment records that its former private harness +was deleted once the paged path became production; that is exactly the end state +A2 reaches. + +**Do NOT copy `deepseek_v4_gen` or `laguna_gen`.** Both are on that allowlist, +both drive a bespoke forward through internal headers, and the allowlist's own +preamble names them as the transition state it exists to retire. A new example in +their shape would be a **new** internal-reaching example with no entry, which +reds `scripts/check-surface-coverage.py` — the gate doing its job. Satisfying it +by appending a line would be precisely the "CLI-only capability landed silently" +defect it was built to stop. + +### 6.2 The routing allowlist entry is part of done + +`scripts/runner-routing-allowlist.txt:26` carries a `nemotron_h` entry naming its +own removal condition: the forward is the host reference, it holds K and V for +the whole prompt, pages nothing, and returns `HostLogits`, so *"the device/paged +runner path is W6, which is what removes this entry"*. **A2 removes that line.** +A stale entry for a now-clean model reds the checker, so this is enforced, not +aspirational — and if A2 cannot remove it, A2 has not reached the seam it +claimed. + +--- + +## 7. Risks, and the decisions they force + +**R1 — G-SAFE is the whole risk.** An A1 that lands without the interlock +converts a loud refusal into a silently wrong server. §0 is the gate; a reviewer +who cannot point at the interlock returns FAIL. + +**R2 — #775 is open in the exact function A1 rewrites, and `main` is RED on +`sanitize-cpu` because of it.** `ForwardNemotronHForCausalLM` downcasts +unconditionally at `nemotron_h_registry.cpp:122` +(`static_cast(model)`), and UBSan names the member call +that goes through it at `:130-132`. `ModelRegistry::Forward` +(`model_registry.cpp:324-327`) performs no type check. + +> **Decision: A1 repairs #775 in the same change and closes it.** Three reasons, +> all pointing the same way. It is in the exact function A1 must edit to install +> the interlock — and the interlock is *itself* a guard placed before those +> member calls, so writing it correctly means establishing the dynamic type +> first, which is the fix. AGENTS.md is explicit that a bug found in flow is +> fixed in flow, referenced in the commit and closed. And it stops `main` staying +> red on a lane this row's own later gates need green. +> +> **The safe idiom already exists in the same file:** `NemotronHLoadReportOf` +> (`nemotron_h_registry.cpp:146-153`) uses `dynamic_cast` + throw. Mirror that. +> +> **The same unconditional cast is repo-wide** — e.g. +> `kimi_linear_registry.cpp:90`. #775 says so itself: *"if the other entry points +> have the same unconditional `static_cast`, that is a class"*. **A1 fixes the +> NemotronH site and files an issue for the class**, to be swept the way #627 and +> #772 handled the unaligned-read class. Do not silently widen A1 into a +> tree-wide sweep; do not leave the class undiscovered either. + +**R3 — A2 is blocked on #496 W2 and the block may still hold.** Re-verify +against the current head and issue state before claiming A2. If the CUDA Mamba2 +arm has not landed, A2 stops and reports; a device forward gated only against a +host reference is not the gate this row claims. + +**R4 — the conv-state dtype conflict is live, and it is A2's to resolve** (§2.7). +`nemotron_h_registry.cpp:167` declares the page `kBF16`; +`nemotron_h_forward.h:284-296` says the host path's conv state must be f32 +because `vt::CausalConv1dFwd` validates it as f32; the scaffold test asserts +bf16 at `test_nemotron_h_scaffold.cpp:652`. Upstream's answer is bf16. A2 gives +the kernel a bf16 conv-state arm; any f32 fallback carries its one-line reason +and its cost in bytes per token, in §10. + +**R5 — the fixture must be the checkpoint the changed path loads.** Pin revision +`29f2d174` explicitly. A repo silently re-quantized under the same name has cost +this project a full campaign ([[unsloth-27b-nvfp4-is-now-fp8]]). + +**R6 — `origin/main` moves under a shared checkout.** Merge an immutable SHA, +and **re-run the full gate after merging rather than reading the diff** — a clean +merge is not a merge that builds the behaviour either side had. #818 is exactly +that failure, in this very model's tests, found by re-running the gate. +Never force-push, including `--force-with-lease`. + +### 7.1 Stop conditions + +- The interlock cannot be expressed — the forward cannot distinguish a + state-carrying step from a stateless one → **`NEEDS_DECISION`**; do not land A1. +- Reading shapes from the spec changes Qwen3.5's allocated bytes → **stop.** The + byte-neutrality contract *is* the claim; if it does not hold, the refactor is + not behaviour-preserving and this spec is wrong, not the test. +- #496 W2 has not landed when A2 is claimed → **stop and report** (R3). +- Mutation M8 stays green → the gate is blind to the carried-state dtype; the + row owes a direct dtype assertion, recorded as a finding rather than absorbed. +- The goldens cannot be reproduced and the divergence looks like a near-tie → + **ask the oracle's own top-2 margin FIRST** + ([[qwen35b-first-request-only-matches-vllm]]) before declaring a defect. +- A `quantized_layers` entry names an algorithm we do not implement → refuse by + name, record as owed, never silently dequantize. +- Any temptation to add a NemotronH branch to `hf_config.cpp` → **stop.** That is + §0.2, and taking it means the refactor was not done. + +--- + +## 8. Records owed on landing + +Per the AGENTS.md "Public documents" table. A spec commit owes none of these; +these are the obligations of the implementing changes. + +| Change | Owes | +|---|---| +| **A1** | nothing in `docs/` — it edits `src/` and `tests/` only and changes no lifecycle state. It **does** owe: this spec's `## Outcome` opened with the byte-identity result and the §3.4 mutation table; a note in the parent spec's W5 section recording the §1.2 refusal and the repair still owed; #775 referenced and closed in the commit; new issues filed for §1.3 and for the `static_cast` class in R2 | +| **A2 + A3** | a lifecycle change, so: `docs/STATUS.md`, `docs/BENCHMARKS.md` (pending, failed or void is a result — silence is not), the row spec's `## Now`, and the row + checklist entry + rollup in `.agents/model-matrix.md:285` in the **same** change (`scripts/check-model-checklist.py` enforces the rollup). Plus `docs/FEATURES.md:141`, whose `NemotronHForCausalLM` row currently reads "CPU host forward returns logits … W6 owns the token gate"; the removal of `scripts/runner-routing-allowlist.txt:26`; and `docs/USAGE.md` if `scripts/check-doc-checkpoint.py` classifies the new example as a user-facing surface — note `kimi-linear-gen` is not currently documented there, so run the checker rather than assuming either way | +| **both** | `.agents/issue-index.md` row for #810 (appended by this spec's commit); #810 linked from the spec and the PR body; this spec's `## Outcome` recording what was measured, what was rejected and why, and why each default is what it is | + +`.agents/NOW.md` is authored at operator cadence and is **not** a per-row +lifecycle write. + +--- + +## 9. Now + +**State at this commit:** spec only. No product code, no lifecycle change. A1 is +claimable immediately against `main` + `bc570da0d`; A2 is blocked on #496 W2, and +that block must be re-verified rather than inherited from this sentence. + +**Next action:** a fresh implementer claims A1 from §1, captures the §3.1 red +first, and lands it with the G-SAFE interlock and the #775 repair in the same +commit. A fresh reviewer — never the implementer — runs the §3.4 mutations and +reports M3 as a pair. + +## 10. Outcome + +Not yet written. Per AGENTS.md this section is filled when the row reaches +`DONE`: what was measured, what was rejected and why, and why each default is +set the way it is — including, explicitly, the conv-state dtype decision of §2.7 +and its cost in bytes per token if it diverges from upstream. From 272e0336b1ae770e39f87e520ce75e0a6481b040 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 08:15:43 +0000 Subject: [PATCH 2/5] spec(MODEL-NEMOTRON-H): the static_cast class already has an issue -- point at #847 instead of asking for a new one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found while filing the §1.3 follow-up: #847, "34 registry entry points still downcast a type-erased LoadedModel with an unchecked static_cast", is exactly the class #775 says it belongs to. The spec asked A1 to FILE that issue, which would have produced a duplicate and lost the 34-site count somebody already measured. A1 now closes #775 at the NemotronH site and references #847 for the sweep, without widening into it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/nemotron-h-abi-e2e.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/.agents/specs/nemotron-h-abi-e2e.md b/.agents/specs/nemotron-h-abi-e2e.md index 7e174fc96..3eb13b66b 100644 --- a/.agents/specs/nemotron-h-abi-e2e.md +++ b/.agents/specs/nemotron-h-abi-e2e.md @@ -831,10 +831,14 @@ that goes through it at `:130-132`. `ModelRegistry::Forward` > > **The same unconditional cast is repo-wide** — e.g. > `kimi_linear_registry.cpp:90`. #775 says so itself: *"if the other entry points -> have the same unconditional `static_cast`, that is a class"*. **A1 fixes the -> NemotronH site and files an issue for the class**, to be swept the way #627 and -> #772 handled the unaligned-read class. Do not silently widen A1 into a -> tree-wide sweep; do not leave the class undiscovered either. +> have the same unconditional `static_cast`, that is a class"* — and **the class +> is already filed as +> [#847](https://github.com/mudler/vllm.cpp/issues/847)**, "34 registry entry +> points still downcast a type-erased `LoadedModel` with an unchecked +> `static_cast`". So A1 fixes the NemotronH site, closes #775, and references +> #847 without widening into it. Do not silently turn A1 into a tree-wide sweep; +> #847 owns that, to be swept the way #627 and #772 handled the unaligned-read +> class. **R3 — A2 is blocked on #496 W2 and the block may still hold.** Re-verify against the current head and issue state before claiming A2. If the CUDA Mamba2 @@ -886,7 +890,7 @@ these are the obligations of the implementing changes. | Change | Owes | |---|---| -| **A1** | nothing in `docs/` — it edits `src/` and `tests/` only and changes no lifecycle state. It **does** owe: this spec's `## Outcome` opened with the byte-identity result and the §3.4 mutation table; a note in the parent spec's W5 section recording the §1.2 refusal and the repair still owed; #775 referenced and closed in the commit; new issues filed for §1.3 and for the `static_cast` class in R2 | +| **A1** | nothing in `docs/` — it edits `src/` and `tests/` only and changes no lifecycle state. It **does** owe: this spec's `## Outcome` opened with the byte-identity result and the §3.4 mutation table; a note in the parent spec's W5 section recording the §1.2 refusal and the repair still owed; #775 referenced and closed in the commit; a new issue filed for §1.3 (the `static_cast` class is already #847) | | **A2 + A3** | a lifecycle change, so: `docs/STATUS.md`, `docs/BENCHMARKS.md` (pending, failed or void is a result — silence is not), the row spec's `## Now`, and the row + checklist entry + rollup in `.agents/model-matrix.md:285` in the **same** change (`scripts/check-model-checklist.py` enforces the rollup). Plus `docs/FEATURES.md:141`, whose `NemotronHForCausalLM` row currently reads "CPU host forward returns logits … W6 owns the token gate"; the removal of `scripts/runner-routing-allowlist.txt:26`; and `docs/USAGE.md` if `scripts/check-doc-checkpoint.py` classifies the new example as a user-facing surface — note `kimi-linear-gen` is not currently documented there, so run the checker rather than assuming either way | | **both** | `.agents/issue-index.md` row for #810 (appended by this spec's commit); #810 linked from the spec and the PR body; this spec's `## Outcome` recording what was measured, what was rejected and why, and why each default is what it is | From cdb84c401b0cf1025c99fa68585160a922058385 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 08:20:30 +0000 Subject: [PATCH 3/5] spec(MODEL-NEMOTRON-H): #775 is not unclaimed -- PR #868 already fixes it, and A1 consumes it Caught by checking remote BRANCHES, not just issues and the open-PR list I had read earlier in the session. #775 is OPEN, which is what an issue search reports, and `row/FIX-NEMOTRON-H-FORWARD-DOWNCAST-775` (0923af27c, forwarded to 829bbd527) has carried a complete gated repair since before this spec was written. PR #868 is open on it. This is the failure mode recorded as verify-gap-against-open-prs-not-just-issues, and the spec now records the correction rather than quietly applying it. The consequence is not cosmetic. #868 introduces `vllm::ModelAs` in the shared registry header -- a dynamic_cast plus an out-of-line refusal -- and the G-SAFE interlock this spec requires is a guard placed before exactly the member calls that helper makes safe. So A1 opens the interlock through `ModelAs` instead of reimplementing the check, and #868 becomes A1's stated base rather than A1's duplicated work. Reimplementing it would also have produced a second answer to a question #868 already answered and wrote down: comparing `registration().architecture` was rejected there, because it reports what the registration claims rather than what the object is. The class stays with #847 and stays unswept, for the reason #868 records: 34 sites across 32 model TUs, and llama_registry, qwen3_5_dense and gemma4_registry each register three architectures against one forward, so those have no single architecture name to refuse under. The sweep is not mechanical, and this row does not widen into it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/nemotron-h-abi-e2e.md | 98 +++++++++++++++++------------ 1 file changed, 57 insertions(+), 41 deletions(-) diff --git a/.agents/specs/nemotron-h-abi-e2e.md b/.agents/specs/nemotron-h-abi-e2e.md index 3eb13b66b..96d1761df 100644 --- a/.agents/specs/nemotron-h-abi-e2e.md +++ b/.agents/specs/nemotron-h-abi-e2e.md @@ -5,8 +5,10 @@ ([#517](https://github.com/mudler/vllm.cpp/issues/517), parent spec [nemotron-h-model.md](nemotron-h-model.md)) — this spec owns what that spec's §6e carried forward as item 3 and explicitly put out of its own scope. -**Also repaired here:** [#775](https://github.com/mudler/vllm.cpp/issues/775) — -see §7 R2. +**Depends on an in-flight fix:** [#775](https://github.com/mudler/vllm.cpp/issues/775) +is already repaired by **open PR +[#868](https://github.com/mudler/vllm.cpp/pull/868)** — see §7 R2. A1 consumes it, +it does not redo it. **Base:** `origin/main` @ `2e9d95e74c7aee133e21771182a6a587fe74c67b`. **Pinned oracle:** `${VLLM_SOURCE}` = `/home/mudler/_git/vllm` @ `5559679229bc961848b121ccdeaa8fa5d79bec98` (vLLM 0.26.0.dev0), verified at HEAD @@ -183,7 +185,7 @@ function, and neither is separately testable on the driver model: They fail together, they are proven by the same test pair, and splitting them manufactures a commit that cannot be gated. **A1 = S1 + S2 + the G-SAFE -interlock + the #775 repair, in one commit.** +interlock, in one commit, on top of #868.** | In A1 | Out of A1 | |---|---| @@ -191,7 +193,7 @@ interlock + the #775 repair, in one commit.** | the config cross-check at `:523-527` **deleted**, not widened; the three `Qwen3.5`-named messages at `:516`, `:525`, `:535` renamed to name the architecture | any new HF-config field, and specifically **no** NemotronH branch in `hf_config.cpp` (§0.2) | | per-layer recurrent membership derived from `KVCacheGroupSpec::layer_names` | the device forward (A2) | | the G-SAFE by-name refusal in `ForwardNemotronHForCausalLM` | the MTP head (parent W5) and GGUF (parent W7) | -| the #775 `static_cast` repair at `nemotron_h_registry.cpp:122` (§7 R2) | a tree-wide sweep of the same cast class (§7 R2) | +| the interlock opened through `vllm::ModelAs<>` from PR #868 (§7 R2) | re-fixing #775, and the tree-wide sweep of the cast class (#847) | | a by-name refusal in `MakeKVCacheMaybeSpec` for a speculative non-Qwen3.5 config (§1.2) | making speculative NemotronH actually work | **What A1 unblocks:** every hybrid whose config does not speak Qwen3.5's @@ -241,14 +243,15 @@ the interlock deleted is not an acceptable split. ### 1.1 Dependency order ``` -A1 (runner refactor + interlock + #775) ──► A2 (device/paged forward) ──► A3 (ABI e2e token gate) - │ │ └── same PR as A2 - │ ├── requires KERNEL-SSM-MAMBA (#496) W2, the CUDA arm - │ └── requires the conv-state dtype reconciliation (§5.3b) - └── requires nothing beyond main + bc570da0d +A1 (runner refactor + interlock) ──► A2 (device/paged forward) ──► A3 (ABI e2e token gate) + │ │ └── same PR as A2 + │ ├── requires KERNEL-SSM-MAMBA (#496) W2, the CUDA arm + │ └── requires the conv-state dtype reconciliation (§5.3b) + └── requires main + bc570da0d, and PR #868 for `vllm::ModelAs<>` (§7 R2) ``` -A1 is independent of #496 and can land immediately. A2 cannot: the parent spec's +A1 is independent of #496. Its only dependency is PR #868 (§7 R2), which is a +small in-flight fix rather than a campaign, so A1 is claimable now. A2 cannot: the parent spec's §0 hard blocker holds — the CUDA Mamba2 SSD arm is W2 of [#496](https://github.com/mudler/vllm.cpp/issues/496) ([spec](mamba2-ssd.md)). **Re-verify #496 W2's state against the current head and @@ -624,7 +627,7 @@ three times in one campaign here | M4 | membership taken from group **index** instead of `layer_names` | NemotronH arm | | M5 | the G-SAFE refusal in `ForwardNemotronHForCausalLM` replaced by a fall-through | the interlock test | | M6 | the `MakeKVCacheMaybeSpec` speculative refusal removed | the §1.2 refusal test | -| M7 | the #775 `dynamic_cast` guard reverted to `static_cast` | the UBSan leg, under `-DVLLM_CPP_SANITIZE='address,undefined'` | +| M7 | the interlock's `vllm::ModelAs<>` reverted to an unchecked `static_cast` | the interlock test, and the UBSan leg under `-DVLLM_CPP_SANITIZE='address,undefined'`. This mutation belongs to A1 only if #868 has landed; if A1 declares #868 its base, #868 owns its own equivalent and A1 does not restate it | | M8 (A2) | the carried SSM state zeroed at the start of every step | the A3 multi-step token gate — see §5.2 | | M9 (A2) | every request's state slot forced to 0 | the A3 batched token gate | | M10 (A2) | the batch treated as one concatenated causal sequence | the A3 batched token gate | @@ -811,34 +814,47 @@ claimed. converts a loud refusal into a silently wrong server. §0 is the gate; a reviewer who cannot point at the interlock returns FAIL. -**R2 — #775 is open in the exact function A1 rewrites, and `main` is RED on -`sanitize-cpu` because of it.** `ForwardNemotronHForCausalLM` downcasts -unconditionally at `nemotron_h_registry.cpp:122` -(`static_cast(model)`), and UBSan names the member call -that goes through it at `:130-132`. `ModelRegistry::Forward` -(`model_registry.cpp:324-327`) performs no type check. - -> **Decision: A1 repairs #775 in the same change and closes it.** Three reasons, -> all pointing the same way. It is in the exact function A1 must edit to install -> the interlock — and the interlock is *itself* a guard placed before those -> member calls, so writing it correctly means establishing the dynamic type -> first, which is the fix. AGENTS.md is explicit that a bug found in flow is -> fixed in flow, referenced in the commit and closed. And it stops `main` staying -> red on a lane this row's own later gates need green. +**R2 — the #775 repair is ALREADY IN FLIGHT as PR +[#868](https://github.com/mudler/vllm.cpp/pull/868); A1 consumes it rather than +redoing it.** An earlier draft of this spec told A1 to fix #775 in flow. That was +wrong, and the correction is recorded rather than quietly applied, because the +way it was wrong is a standing trap: an issue-and-open-PR search that stops at +issues finds #775 OPEN and concludes the work is unclaimed +([[verify-gap-against-open-prs-not-just-issues]]). + +The facts as measured. `ForwardNemotronHForCausalLM` does downcast +unconditionally at `nemotron_h_registry.cpp:122`, and `ModelRegistry::Forward` +(`model_registry.cpp:324-327`) performs no type check — §0 is accurate. But +branch `row/FIX-NEMOTRON-H-FORWARD-DOWNCAST-775` (`0923af27c`, merged forward to +`829bbd527`) already carries a complete, gated repair, and PR #868 is open on it. +What it introduces matters to A1 directly: + +- **`vllm::ModelAs` in the shared registry header** — a `dynamic_cast` + plus a branch, with the refusal authored once and out of line in + `RaiseModelTypeMismatch`. It mirrors the local `dynamic_cast`-with-refusal + idiom already used by `kv_cache_spec_registry.cpp`, + `single_type_kv_cache_manager.cpp` and `attention/backend.cpp`. +- A test handing `reg.factory->forward` a complete `ForeignLoadedModel` and + requiring a named refusal. +- The rejected alternative, written down: comparing + `model.registration().architecture` against the expected name answers what the + *registration claims*, not what the object *is*, and the realistic defect + carries the right architecture string on the wrong object. + +> **Decision: A1 depends on #868 and opens the interlock with +> `vllm::ModelAs`.** The G-SAFE refusal is a guard placed +> before exactly the member calls #868 makes safe, so writing it on top of +> `ModelAs` is both correct and one line. **Confirm #868's state before claiming +> A1.** If it has landed, use the helper. If it is still open, A1 either waits or +> declares #868 its base and says so in the PR body — it must not reimplement the +> helper, and it must not reintroduce the cast. > -> **The safe idiom already exists in the same file:** `NemotronHLoadReportOf` -> (`nemotron_h_registry.cpp:146-153`) uses `dynamic_cast` + throw. Mirror that. -> -> **The same unconditional cast is repo-wide** — e.g. -> `kimi_linear_registry.cpp:90`. #775 says so itself: *"if the other entry points -> have the same unconditional `static_cast`, that is a class"* — and **the class -> is already filed as -> [#847](https://github.com/mudler/vllm.cpp/issues/847)**, "34 registry entry -> points still downcast a type-erased `LoadedModel` with an unchecked -> `static_cast`". So A1 fixes the NemotronH site, closes #775, and references -> #847 without widening into it. Do not silently turn A1 into a tree-wide sweep; -> #847 owns that, to be swept the way #627 and #772 handled the unaligned-read -> class. +> **The class is #847, and it is not mechanical.** #868's own message records why +> the 34 sibling sites across 32 model TUs were not swept with it: +> `llama_registry`, `qwen3_5_dense` and `gemma4_registry` each register three +> architectures against one forward, so those sites have no single architecture +> name to refuse under. #847 carries the three candidate answers. A1 references +> it and does not widen into it. **R3 — A2 is blocked on #496 W2 and the block may still hold.** Re-verify against the current head and issue state before claiming A2. If the CUDA Mamba2 @@ -890,7 +906,7 @@ these are the obligations of the implementing changes. | Change | Owes | |---|---| -| **A1** | nothing in `docs/` — it edits `src/` and `tests/` only and changes no lifecycle state. It **does** owe: this spec's `## Outcome` opened with the byte-identity result and the §3.4 mutation table; a note in the parent spec's W5 section recording the §1.2 refusal and the repair still owed; #775 referenced and closed in the commit; a new issue filed for §1.3 (the `static_cast` class is already #847) | +| **A1** | nothing in `docs/` — it edits `src/` and `tests/` only and changes no lifecycle state. It **does** owe: this spec's `## Outcome` opened with the byte-identity result and the §3.4 mutation table; a note in the parent spec's W5 section recording the §1.2 refusal and the repair still owed; the §1.3 issue filed; #775/#868 referenced as a dependency, never re-fixed (the class stays with #847) | | **A2 + A3** | a lifecycle change, so: `docs/STATUS.md`, `docs/BENCHMARKS.md` (pending, failed or void is a result — silence is not), the row spec's `## Now`, and the row + checklist entry + rollup in `.agents/model-matrix.md:285` in the **same** change (`scripts/check-model-checklist.py` enforces the rollup). Plus `docs/FEATURES.md:141`, whose `NemotronHForCausalLM` row currently reads "CPU host forward returns logits … W6 owns the token gate"; the removal of `scripts/runner-routing-allowlist.txt:26`; and `docs/USAGE.md` if `scripts/check-doc-checkpoint.py` classifies the new example as a user-facing surface — note `kimi-linear-gen` is not currently documented there, so run the checker rather than assuming either way | | **both** | `.agents/issue-index.md` row for #810 (appended by this spec's commit); #810 linked from the spec and the PR body; this spec's `## Outcome` recording what was measured, what was rejected and why, and why each default is what it is | @@ -906,8 +922,8 @@ claimable immediately against `main` + `bc570da0d`; A2 is blocked on #496 W2, an that block must be re-verified rather than inherited from this sentence. **Next action:** a fresh implementer claims A1 from §1, captures the §3.1 red -first, and lands it with the G-SAFE interlock and the #775 repair in the same -commit. A fresh reviewer — never the implementer — runs the §3.4 mutations and +first, confirms #868's state per §7 R2, and lands A1 with the G-SAFE interlock +opened through `vllm::ModelAs`. A fresh reviewer — never the implementer — runs the §3.4 mutations and reports M3 as a pair. ## 10. Outcome From 6c3b6a299fc7737511cd958e5c0cc4f7c4748bc0 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 08:28:26 +0000 Subject: [PATCH 4/5] spec(MODEL-NEMOTRON-H): three developer decisions, and the inherited red an implementer must subtract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Separate pull requests, spec first (§1.4). This was owed under porting-a-model.md §0 and was unrecorded. The reason is written down so the next row claim does not re-open it: the spec is committed BEFORE implementation and a FRESH implementer works from it, and a fresh implementer is not fresh if it also authored the spec. A spec shipping alongside its own implementation is reviewed next to the code it was supposed to constrain. §0's general recommendation covers the case where a spec is a paragraph of scope; this one is the artifact another agent works from. The gate hosts are dgx.casa (GB10, sm_121a) and 192.168.68.23 (Thor, sm_110), not the local box (§5.4). The local x86_64 host has no GPU, so it cannot see anything device-side and an A3 result from it is not an A3 result; it keeps A1's unit gate, which needs neither GPU nor checkpoint. GB10 still supplies the CUDA==CPU equivalence check, so nothing is lost. Thor is not a second CUDA host -- it builds TRITON=OFF with no cutlass and every sm_110 fast-path cell resolves EMPTY, so it exercises the fallback kernels GB10's fast paths hide, which is why the loader row took evidence on both. Recorded with it: the GPU is container-only there, and the box REBOOTS instead of OOM-killing, so the 20.1 GiB checkpoint fits alone but a future spec-on run does not. The conv-state dtype mirrors upstream, and that is not a scheduling call (§2.7, §7 R4). vLLM resolves one dtype and every layer inherits it; a token gate cannot catch a dtype that is too WIDE, because it is still numerically correct -- tokens match, goldens pass, and we move twice the bytes. An f32 conv state where upstream uses bf16 is exactly that defect, and every gate this row owns is blind to it. Since mamba_utils.py:96-107 shows conv state carrying mamba_cache_dtype, the registry's kBF16 is the correct mirror and the local CausalConv1dFwd precondition forcing f32 is the bug. A2 gives that kernel a bf16 arm. The f32 fallback survives only as an annotated escape owing its cost in bytes per token. Also recorded: main is RED on six gates at 0141cb4b4 from the #865 ci.yml rewrite, filed as #873 and reproduced at that SHA with zero modifications (§5.5). An implementer subtracts them rather than chasing them -- and does not subtract them blindly, because #775 is the precedent for a lane staying red long enough that the cause changed underneath the people discounting it. If the separately dispatched #873 repair lands first, the section is historical and a red on any of the six is yours. Three findings are stated in the terms that make them load-bearing: the upstream runner cannot perform our cross-check, which is the argument for DELETING it rather than relocating it; test_runner.cpp:505's dtype half is armed while its shape half is inert, which makes it the cheapest RED-first anchor in the row; and Kimi-Linear passes only because hf_config.cpp:484-528 synthesizes the fields, by its own comment. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/nemotron-h-abi-e2e.md | 230 ++++++++++++++++++++++------ 1 file changed, 184 insertions(+), 46 deletions(-) diff --git a/.agents/specs/nemotron-h-abi-e2e.md b/.agents/specs/nemotron-h-abi-e2e.md index 96d1761df..12ee76a62 100644 --- a/.agents/specs/nemotron-h-abi-e2e.md +++ b/.agents/specs/nemotron-h-abi-e2e.md @@ -288,7 +288,27 @@ this is latent, not live. **Out of scope. File an issue when A1 is claimed** — AGENTS.md requires an issue for a bug found in flow, and this one is not fixable in flow because no model exercises it. -### 1.4 Explicitly out of scope +### 1.4 Spec and implementation are SEPARATE pull requests (decided, do not re-litigate) + +`porting-a-model.md` §0 says to ask the developer at row claim whether the spec +and the implementation share one pull request, and to record the answer. +**Asked and answered on 2026-08-15: separate. Spec first, implementation after.** + +The reason is recorded here so the next row claim does not re-open it. It is not +a preference — it falls out of two AGENTS.md rules that compose only one way: +*"the spec is committed **before** implementation, never written up afterwards"*, +and *"a **fresh implementer** works from the committed spec"*. A fresh implementer +is not fresh if it also authored the spec, and a spec shipping in the same pull +request as its implementation is being reviewed alongside the code it was +supposed to constrain — which is the review this protocol exists to avoid. The +general recommendation in `porting-a-model.md` §0 covers the ordinary case where +a spec is a paragraph of scope; this row's spec is the artifact a different agent +works from. + +So: **this PR carries the spec and no product code. A1 is a separate PR by a +different agent**, and its reviewer is a third. + +### 1.5 Explicitly out of scope Speed (no ratio is recorded by this row); the MTP head; the GGUF k-quant arm (parent §5b/W7, still refused by name); `NemotronHPuzzleForCausalLM`; TP @@ -360,7 +380,15 @@ for shape, dtype in zip(self.get_state_shape(), self.get_state_dtype()): ``` **The upstream runner literally cannot perform our cross-check** — it never -holds a conv shape. That is why `MambaSpec` carries `shapes` and `dtypes` +holds a conv shape. **That is the argument for DELETING the check, not for +relocating it.** A cross-check that upstream is structurally incapable of +expressing is not a safety net we happen to have and they happen to lack; it is +a second, independent derivation of a number the model already published, and a +second derivation is exactly the thing that can disagree. Widening it to accept +NemotronH's geometry, or moving it behind a per-architecture switch, would keep +the defect and add a maintenance surface. It goes. + +That is also why `MambaSpec` carries `shapes` and `dtypes` (`vllm/v1/kv_cache_interface.py:690-696`) and `page_size_bytes` is their sum (`:699-703`), and why our `MambaSpec` (`include/vllm/v1/kv_cache_interface.h:302-323`) mirrors those fields and @@ -520,16 +548,33 @@ conflict explicit rather than latent: > the persistent page the registry declares is bf16 — and it says outright that > reconciling the two belongs to the paged-decode work, i.e. **A2**. -**Decision, and it follows from AGENTS.md rather than from taste.** vLLM resolves -one dtype and every layer inherits it; `f32` is a rare, annotated escape, and a -buffer that names `f32` on a model path owes a one-line reason. The upstream -answer is bf16. **A2 therefore gives `vt::CausalConv1dFwd` a bf16 conv-state -arm** and stores the conv state at the cache dtype; it does not widen the page to -f32 to satisfy a local kernel precondition. If that arm turns out to be -materially harder than it looks, the fallback is an f32 conv page with the -one-line reason and the cost **stated in bytes per token** in §10 — never an -unannotated widening. A token gate cannot see this either way, which is exactly -why it is decided here in the spec and not during implementation. +**Decision (developer-directed 2026-08-15): mirror upstream. A2 gives +`vt::CausalConv1dFwd` a bf16 conv-state arm and stores the conv state at the +cache dtype.** This is a correctness obligation under AGENTS.md, **not a +scheduling call**, and it is not to be re-litigated as one when the kernel work +turns out to be real. + +The reasoning follows from the rules rather than from taste: + +- AGENTS.md: *"vLLM resolves ONE model dtype and every layer inherits it"*, and + `f32` is a rare, annotated escape. The upstream answer here is measured, not + inferred — `mamba_utils.py:96-107`, conv state carries `mamba_cache_dtype`, + default `auto` → the model dtype → bf16 for this checkpoint. +- AGENTS.md again, and this is the whole point: *"a token gate cannot catch a + dtype that is too WIDE — it is still numerically correct, so tokens match, + goldens pass, and we move twice the bytes."* An f32 conv state where upstream + uses bf16 is **precisely** that defect. Every gate this row owns is blind to + it, which is why it is settled in the spec rather than during implementation. +- **The registry is right and the precondition is the bug.** + `nemotron_h_registry.cpp:167` declares `kBF16` and mirrors upstream correctly. + What forces f32 is a *local* precondition — `vt::CausalConv1dFwd` validating + its conv state as f32 — and a local precondition contradicting a correct + mirror is a defect in the precondition, not a reason to widen the page to + satisfy it. + +**The f32 fallback survives only as an annotated escape**, never as the default: +if it is taken it owes a one-line reason at the buffer and its cost **stated in +bytes per token** in §10. A silent widening is not one of the options. --- @@ -557,7 +602,14 @@ its shape half is a **self-consistency gate** The **dtype** half of that same case *is* genuinely armed — `MakeKvConfig(c, kBF16, kF16)` passes dtypes the config cannot produce, and `:528-529` honours -them. That asymmetry is the shape of the repair, in the same file. +them. + +**Its dtype half being armed while its shape half is inert is the cheapest +RED-first anchor in this row**, and it is worth saying in exactly those terms. +The two halves sit in one `TEST_CASE`, five lines apart, testing one function. +One of them already does the right thing. So the repair needs no new harness, no +new fixture and no new file — it needs the shape half to be given inputs the +config cannot produce, which is what the dtype half is already doing next to it. **This is the RED-first anchor and it is cheap.** Add a subcase whose `MambaSpec` shapes are **not derivable from the config**: `linear_*` zeroed, `layer_types` @@ -667,10 +719,13 @@ Correctness first, always. No throughput number is recorded by this row. ### 5.1 A1's gate — unit, deterministic, no checkpoint `ctest -R '^test_runner$'` plus `ctest -R '^test_nemotron_h'`, plus the full -`scripts/agent-preflight.sh`. It sees shapes, dtypes, counts, byte totals and -layer→group routing. **It cannot see whether the state is ever used** — that is -why G-SAFE exists, and A1's gate must never be reported as evidence that -NemotronH runs. +`scripts/agent-preflight.sh` — read §5.5 first and subtract the inherited +baseline. This gate runs anywhere, including the local development box; it needs +no GPU and no checkpoint. + +It sees shapes, dtypes, counts, byte totals and layer→group routing. **It cannot +see whether the state is ever used** — that is why G-SAFE exists, and A1's gate +must never be reported as evidence that NemotronH runs. ### 5.2 A3's token gate — multi-step and multi-request, deliberately @@ -713,10 +768,15 @@ A3 asserts: against what the pinned oracle reports for the same checkpoint, read from the **running** engine's resolved config, not from source ([`porting.md`](../porting.md)). -5. **Oracle identity asserted, aborting on mismatch.** Use `vllm-oracle-next`, - never `$HOME/venvs/vllm-oracle`, which symlinks to a 0.25.0 rollback - predating `NemotronHMoEDecoderLayer` and fails in a way that reads as "the - model is unsupported" (parent spec §5a). A v1 driver script needs +5. **Oracle identity asserted BY COMMIT, aborting on mismatch.** Use + `vllm-oracle-next`, never `$HOME/venvs/vllm-oracle`, which symlinks to a + 0.25.0 rollback predating `NemotronHMoEDecoderLayer` and fails in a way that + reads as "the model is unsupported" (parent spec §5a). **Assert the commit, + not the version string:** `environment.md` records (2026-08-12) that the pin + reports `0.23.1rc1.dev1511+g555967922`, a `setuptools_scm` + nearest-ancestor-tag artefact — HEAD *is* `5559679229bc`, and the recorded + `0.26.0.dev0` string will never match. A version-string assertion here fails + with nothing wrong. A v1 driver script also needs `if __name__ == "__main__":` or EngineCore's spawn re-imports it and the failure names neither vLLM nor the caller. @@ -743,22 +803,85 @@ parameters, modes, fixtures, tolerances, failure cases and the revision anchor: oracle golden instead. That is a search result with the paths named, not an assumption. -### 5.4 The gate hosts, and the discipline - -Both arms of A3 run on **both gate hosts**: the local x86_64 CPU-only Release -host (CPU backend) and `dgx.casa` / GB10 sm_121a (CUDA). Requiring both is not -ceremony — the same 32 tokens on both backends is also the CUDA==CPU equivalence -check, and it is free once the gate exists. - -GPU discipline on dgx, all mandatory: `flock $HOME/gpu.lock` — the lock is +### 5.4 The gate hosts (developer-directed 2026-08-15), and the discipline + +**The two gate hosts are `dgx.casa` (GB10, sm_121a) and `192.168.68.23` (Jetson +Thor, sm_110).** Both arms of A3 run on both. + +**The local x86_64 box is a development arm, not a gate host.** It has no GPU, so +it cannot see anything device-side, and an A3 result from it is not an A3 result. +Run A1's unit gate and the preflight there all you like; do not report a token +gate from it. + +**GB10 still provides the CUDA==CPU equivalence check** — it runs both backends, +so nothing is lost by dropping the local box. + +**Thor is not a second CUDA host, it is the PORTABLE path.** It builds with +`-DVLLM_CPP_CUDA_ARCHITECTURES=110 -DVLLM_CPP_CUDA=ON -DVLLM_CPP_TRITON=OFF` and +**no cutlass — every sm_110 fast-path cell resolves EMPTY** +(`environment.md`, verified 2026-08-11). So Thor exercises the fallback kernels +that GB10's fast paths hide, which is exactly why the loader row took its +evidence on both, and A3 does the same. A conv/SSM path that only works through +a fast path is a path that does not work. + +Thor's operational facts, none of them optional: + +- **The GPU is reachable only from inside a container.** No host CUDA toolkit, no + `nvcc`, no `cmake`. Use `sudo docker` (the user is in group `admin`, not + `docker`), **`--runtime=nvidia`** — `--gpus all` is refused outright — and + **`-e NVIDIA_DISABLE_REQUIRE=1`**, because the image's `NVIDIA_REQUIRE_CUDA` + tops out at `driver<576` and the box runs 595.78. +- **★ THIS BOX REBOOTS INSTEAD OF OOM-KILLING.** `vm.overcommit_memory=1` with + zero swap: touching memory the kernel could not back takes the whole machine + down, with `exit=255`, `OOMKilled=false` and no `dmesg` OOM line. Observed three + times on 2026-08-11. The rule recorded there is **no >25 GB model plus a second + checkpoint**. The driver checkpoint is 20.1 GiB and A3 loads it **alone** — the + MTP drafter is parent W5 and out of scope here — so A3 fits, but a future + spec-on run on this box does not. Size for it before pressing enter. +- No `local-ai-worker` and no `~/gpu.lock` on Thor in its current state; the + worker-restore discipline below is dgx-only. +- Move code with `git archive`, never rsync ([[dgx-transfer-git-archive-not-rsync]]); + weights move dgx→Thor with `tar -ch | ssh ... tar -x` into a FRESH directory. +- `nsys` in that image is 2024.2.3 and **cannot trace CUDA here**. Irrelevant to + A3, which records no number — noted so nobody plans one. + +**GPU discipline on dgx**, all mandatory: `flock $HOME/gpu.lock` — the lock is `$HOME/gpu.lock`, **not** `/tmp/gpu.lock`; `local-ai-worker` parked and restored at the end; one log per run; never a large oracle alongside `ctest`, because -`gpu_memory_utilization` reserves HOST RAM on GB10 and has OOM-rebooted the box; -CUDA `ctest` with `-j 1`. `df -h` before and after every build — an ENOSPC leaves -the previous binary in place and prints a green status +`gpu_memory_utilization` reserves HOST RAM on GB10 and has OOM-rebooted that box +too; CUDA `ctest` with `-j 1`. `df -h` before and after every build on either +host — an ENOSPC leaves the previous binary in place and prints a green status ([[stale-binary-prints-green-status]], [[enospc-makes-checkers-emit-false-policy-refusals]]). +### 5.5 The inherited baseline — subtract it, do not chase it + +At the time this spec was written, `main` @ **`0141cb4b4`** is **RED on six +gates**, all of them consequences of the `#865` `ci.yml` rewrite and none of them +anything an implementer of this row can cause: + +``` +check-release-binary-contract check-release-workflow check-test-registration +test_check_release_binary_contract test_release_manifest test_release_pipeline +``` + +Reproduced in a detached worktree at `0141cb4b4` **with zero modifications**; +preflight was green on the immediately preceding base `e8f3b4a75` in the same +session on the same box. Filed as +[#873](https://github.com/mudler/vllm.cpp/issues/873), and a repair is being +dispatched separately. + +**Subtract them as an inherited baseline; do not chase them, and do not report +them as this row's.** Equally: do not subtract them *blindly*. #775 is the exact +precedent — a lane stayed red long enough that people learned to discount it, and +by the time the cause had changed underneath, anyone subtracting "the known red" +was subtracting the wrong thing. Re-measure at your own base and name what you +actually saw. + +**If the #873 repair lands before A1 is claimed, this section is historical +rather than operative** — the correct action is then a clean green preflight, and +a red on any of the six is yours. + --- ## 6. The e2e acceptance — this row's stop condition @@ -768,7 +891,8 @@ the previous binary in place and prints a green status > 32-token goldens at > `tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json` token-exact > for all three prompts, with `VT_NEMOTRON35_SNAPSHOT` UNSET so the revision -> check binds, on both gate hosts.** +> check binds, on **both gate hosts — `dgx.casa` (GB10, sm_121a) and +> `192.168.68.23` (Thor, sm_110, no CUTLASS, `TRITON=OFF`)**, per §5.4.** `VT_NEMOTRON35_SNAPSHOT` is deliberately never revision-checked (`tests/parity/test_hf_snapshot_pinning.cpp:62`), so setting it during the gate @@ -861,13 +985,17 @@ against the current head and issue state before claiming A2. If the CUDA Mamba2 arm has not landed, A2 stops and reports; a device forward gated only against a host reference is not the gate this row claims. -**R4 — the conv-state dtype conflict is live, and it is A2's to resolve** (§2.7). -`nemotron_h_registry.cpp:167` declares the page `kBF16`; -`nemotron_h_forward.h:284-296` says the host path's conv state must be f32 -because `vt::CausalConv1dFwd` validates it as f32; the scaffold test asserts -bf16 at `test_nemotron_h_scaffold.cpp:652`. Upstream's answer is bf16. A2 gives -the kernel a bf16 conv-state arm; any f32 fallback carries its one-line reason -and its cost in bytes per token, in §10. +**R4 — the conv-state dtype conflict is live, it is A2's to resolve, and the +answer is already decided** (§2.7). `nemotron_h_registry.cpp:167` declares the +page `kBF16`; `nemotron_h_forward.h:284-296` says the host path's conv state must +be f32 because `vt::CausalConv1dFwd` validates it as f32; the scaffold test +asserts bf16 at `test_nemotron_h_scaffold.cpp:652`. Upstream's answer is bf16 +(`mamba_utils.py:96-107`), so **A2 gives the kernel a bf16 conv-state arm**. The +risk this entry records is not *which way* — that is settled — but that the +kernel arm is unplanned work an implementer may be tempted to trade away under +schedule pressure. It is not tradeable: a too-wide dtype passes every gate this +row owns. Any f32 fallback carries its one-line reason and its cost in bytes per +token, in §10. **R5 — the fixture must be the checkpoint the changed path loads.** Pin revision `29f2d174` explicitly. A repo silently re-quantized under the same name has cost @@ -917,14 +1045,24 @@ lifecycle write. ## 9. Now -**State at this commit:** spec only. No product code, no lifecycle change. A1 is -claimable immediately against `main` + `bc570da0d`; A2 is blocked on #496 W2, and -that block must be re-verified rather than inherited from this sentence. +**State at this commit:** spec only. No product code, no lifecycle change. Per +§1.4 the implementation is a **separate** pull request by a **different** agent; +this one carries the spec. + +A1 is claimable now against `main` + `bc570da0d`, with PR #868 as its base +(§7 R2). A2 is blocked on #496 W2, and that block must be re-verified rather than +inherited from this sentence. + +**Three things to read before the first edit**, because each has already cost +somebody a cycle: §5.5, so the six `#873` gates are subtracted rather than +chased; §5.4, so the token gate is planned for `dgx.casa` and Thor and not for +the local box, which cannot see anything device-side; and §7 R2, so #775 is +consumed rather than re-fixed. **Next action:** a fresh implementer claims A1 from §1, captures the §3.1 red -first, confirms #868's state per §7 R2, and lands A1 with the G-SAFE interlock -opened through `vllm::ModelAs`. A fresh reviewer — never the implementer — runs the §3.4 mutations and -reports M3 as a pair. +first, and lands A1 with the G-SAFE interlock opened through +`vllm::ModelAs`. A fresh reviewer — never the implementer — +runs the §3.4 mutations and reports M3 as a pair. ## 10. Outcome From fe3828b646a8c4f2024649861ffa6e82a0e0742b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 08:32:16 +0000 Subject: [PATCH 5/5] spec(MODEL-NEMOTRON-H): the dependency diagram pointed at a section number the conv-dtype decision no longer lives at MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The conv-state dtype question moved into §2.7 when it stopped being a deferred reconciliation and became a decided mirror of mamba_utils.py:96-107. The A2 arrow in §1.1 kept pointing at §5.3b, which is now "Upstream tests owed in the same change" -- so the one cross-reference that tells an implementer where the unplanned kernel arm is justified sent them somewhere else. Caught by listing every section-number reference against every heading; it was the only dangling one. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/nemotron-h-abi-e2e.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/specs/nemotron-h-abi-e2e.md b/.agents/specs/nemotron-h-abi-e2e.md index 12ee76a62..0835e47d4 100644 --- a/.agents/specs/nemotron-h-abi-e2e.md +++ b/.agents/specs/nemotron-h-abi-e2e.md @@ -246,7 +246,7 @@ the interlock deleted is not an acceptable split. A1 (runner refactor + interlock) ──► A2 (device/paged forward) ──► A3 (ABI e2e token gate) │ │ └── same PR as A2 │ ├── requires KERNEL-SSM-MAMBA (#496) W2, the CUDA arm - │ └── requires the conv-state dtype reconciliation (§5.3b) + │ └── requires the conv-state dtype mirror (§2.7) └── requires main + bc570da0d, and PR #868 for `vllm::ModelAs<>` (§7 R2) ```