feat(rocm): grouped + non-grouped keep-quant expert GEMM — the MoE-path terminus (issue #41)

.agents/specs/rocm-gg-keep-quant.md

@@ -0,0 +1,58 @@
+# ROCm keep-quant expert GEMM — review rework (PR #523)
+
+## What this fixes
+
+The review sweep (localai-bot, 2026-08-13) found the original #523 shape
+registered `kMatmulBTQuant` with a loader that flips keep-quant on a BOOLEAN
+(`GgufQuantComputeAvailable()` = `OpRegistered(...)`), while the ROCm kernel
+implements 4 of the 12 formats the loader admits (Q4_0, Q8_0, Q2_K, Q3_K, Q4_K,
+Q5_K, Q6_K, IQ2_XXS, IQ3_XXS, IQ2_S, MXFP4). On a discrete card with no CPU
+fallback tier, a Q4_0/Q2_K/IQ2 model that loaded and generated fine before
+would keep blocks quantized and throw at first forward. Same boolean flipped
+`keep_f16` on, and the ROCm `MatmulBT` refuses f16 — a second regression of a
+working path.
+
+## The rework
+
+1. **Per-dtype capability in the loader** (`gguf_keep_quant.cpp`):
+   `KeepQuantDType` and the keep-f16 default now consult the running device's
+   actual support. ROCm keep-quant supports {Q8_0, Q4_K, Q5_K, Q6_K}
+   (kMatmulBTQuant + kMatmulBTQuantGrouped both); ROCm keep-f16 is OFF
+   (`MatmulBTKernelRocm` accepts bf16/f32 only). Unsupported formats keep the
+   pre-existing `expand_bf16` residency — no load fails, no forward throws, and
+   `VT_GGUF_KEEP_QUANT=1` on a Q4_0 model is a no-op rather than a regression.
+   CUDA/CPU behavior is byte-identical (their sets already cover the CPU list).
+2. **Capture-safe scratch**: the per-call `hipMalloc`/`hipFree`/
+   `hipStreamSynchronize` on the activation-quant scratch (illegal under
+   hipGraph stream capture — blocks #473/#332) becomes a grow-only per-stream
+   pool via `hipMallocAsync`, mirroring the donor's `EnsureScratch` +
+   `RetireGraphScratch` (never-freed, because a captured graph may have baked
+   the pointer). Also fixes the `qact` leak when `Check()` threw between
+   malloc and free.
+3. **The refusal messages** name the actually-unported formats (Q4_0, Q2_K,
+   Q3_K, IQ2_XXS, IQ3_XXS, IQ2_S, MXFP4) instead of double-listing Q5_K — the
+   message is now unreachable in practice (the loader pre-filters) but stays
+   correct as the last line of defense.
+4. **Teeth**: the non-grouped `kMatmulBTQuant` gains its own cross-device case
+   (it carried the headline mechanism and had no test), and both new cases
+   `REQUIRE(OpAvailable(...))` instead of skipping silently when registration
+   is dropped. The grouped case keeps its NMSE<=5e-4 vs CPU keep-quant oracle
+   bar.
+5. `Dp4a` keeps the portable four-MAC body if `__dp4a` is absent on the
+   gfx1100 toolchain (verified at build time); if `__dp4a` compiles, use it.
+
+## Gates
+
+- Focused: `test_backend_cross_device` (grouped + non-grouped keep-quant
+  cases, REQUIRE-proven registration), red-first by stash-revert.
+- Regression: the 0.8B + 0.6B M4 gates; Qwen3.6-35B-A3B Q4_K_M e2e on one
+  gfx1100 card (`--max-num-seqs 1`); a Q4_0 GGUF load on ROCm proving no
+  regression (expands, generates, no throw).
+- Full HIP ctest zero-delta vs base.
+
+## Boundaries
+
+- No change to the ported dot-product cores (review verified them against the
+  donor, DotQ6K byte-for-byte).
+- Q2_K/Q3_K/IQ2/IQ3/MXFP4 ROCm kernels remain owed and are recorded as such
+  here and in the refusal messages.

.agents/specs/rocm-grouped-quant-gemm.md

@@ -0,0 +1,77 @@
+# ROCm grouped quant expert GEMM (kMatmulBTQuantGrouped) — spike
+
+**Issue:** #41 (ROCm lane); the named blocker for MoE-bearing models after the
+GDN slice (#334–#345) and the MoE chain (#348, #509).
+**Status:** spike — no code yet.
+
+## The gap, verified
+
+On discrete ROCm the MoE path now resolves everything except the expert GEMM:
+`kMoeRouterTopK` + `kMoeSiluMul` (#348), `kSharedExpertGate`/`kMoeCombine`/
+`kMoeCombineGate` (#509) are native and gated. The remaining throw is
+`kMatmulBTQuantGrouped` — the keep-quant grouped expert GEMM that runs the
+stacked `[E*N,K]` expert towers. Without it, MoE-bearing models
+(Qwen3.5-27B-class GDN-MoE, DeepSeek-V4 GGUF) throw on discrete ROCm.
+
+`test_bench`/`test_capi` flipped green once the chain ops landed (they don't
+reach the grouped GEMM). `test_loaded_engine_dense` still fails — but on the
+**async-scheduling assertion** (`runner_supports_async()=false` on ROCm), a
+lane capability gap unrelated to this op.
+
+## What the donor actually is
+
+`src/vt/cuda/cuda_quant_dot.cu` (2069 lines). The grouped GEMM is
+`QuantDotGemmGroupedKernel` (:746) + a fused SwiGLU variant (:799) +
+Q8_0-specific kernels (:1404/:1441). Structure:
+
+1. **Shared activation quant**: input rows quantized to Q8_K once
+   (`QuantizeRowQ8_K`, CPU ref `cpu_quant_act.cpp:88`; a `QuantizeQ8_0Kernel`
+   device quantizer exists for the Q8_0 path).
+2. **Per-format integer dot superblocks**: `DotSuperblock<W>` specializations
+   (`:655`+) for Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, Q4_0, Q8_0, IQ2_XXS/IQ3_XXS —
+   each dequantizes a keep-quant weight superblock and dots against the Q8_K
+   activation block. This is the bulk and the only genuinely tricky part.
+3. **Grouped dispatch**: warp-per-(p,j), `__shfl_down_sync` reduction
+   (HIP-compatible as-is), expert row selected by `expert_ids[p]`.
+
+The CPU reference (`cpu_quant_dot.cpp` VecDot family) is complete and is the
+gate oracle. HIP needs no torch; the donor's torch surface is only the host
+glue.
+
+## Port plan (per-format PRs, red-first, CPU-oracle gated)
+
+- **W0: Q8_K activation quant + Q8_0 dot + grouped skeleton.** Smallest
+  end-to-end slice that runs a real (if low-value) grouped GEMM; establishes
+  the registration, the Q8_K quantizer port, and the cross-device gate vs
+  `VecDotQ8_0Q8_0` (cpu_quant_dot.cpp:88). RED: op unregistered today.
+- **W1: Q4_0 + Q4_K** (`VecDotQ4_0Q8_0` :50, `VecDotQ4_KQ8_K` :203) — the
+  dominant GGUF expert formats.
+- **W2: Q5_K/Q6_K/Q2_K** and the fused SwiGLU variant (the ds4 epilogue).
+- **W3: IQ2/IQ3** — lowest-value, last.
+
+Each family: hand-port from the donor's `DotSuperblock`, cross-device case vs
+the CPU VecDot oracle (NMSE ≤ 5e-4 — the same band the CUDA lane uses, since
+the integer core is bit-exact and only the float scale sum reassociates),
+focused + full gate.
+
+## Testability constraint (honest)
+
+The op is only reachable end-to-end on MoE models I cannot fit on this box
+(Qwen3.5-27B needs a multi-GB GGUF; the 0.8B has no experts). So the gate is
+the **CPU reference at the op level** (cross-device, both groupings, the
+broadcast-activation arm), and the model-level e2e stays PENDING a host with
+the checkpoint — that is a real constraint, stated, not papered over.
+
+## What is deliberately not in scope
+
+- The fused SwiGLU grouped kernel (W2, a perf/composition variant).
+- ggml's SIMD-table IQ formats' fastest paths (port the reference math first).
+- Any perf tuning — correctness first; the win over "no path at all" is
+  binary.
+
+## Stop conditions
+
+- A format's dot cannot be made NMSE-clean vs the CPU VecDot oracle → stop and
+  post the failing evidence on #41 rather than ship a wrong quant path.
+- A model-level e2e claim is ever made from op-level-only evidence → it must
+  not be; the constraint above holds.

CMakeLists.txt

@@ -1526,6 +1526,8 @@ if(VLLM_CPP_HIP)
     src/vt/rocm/rocm_gemma4_expert_geglu.hip
     src/vt/rocm/rocm_fp8_channel_gemv.hip
     src/vt/rocm/rocm_moe_router.hip
+    src/vt/rocm/rocm_moe_chain.hip
+    src/vt/rocm/rocm_grouped_gemm.hip
     src/vt/rocm/rocm_sample.hip
     src/vt/rocm/rocm_gdn_state.hip
     src/vt/rocm/rocm_gdn_conv.hip
@@ -1546,6 +1548,8 @@ if(VLLM_CPP_HIP)
       src/vt/rocm/rocm_gemma4_expert_geglu.hip
       src/vt/rocm/rocm_fp8_channel_gemv.hip
       src/vt/rocm/rocm_moe_router.hip
+      src/vt/rocm/rocm_moe_chain.hip
+      src/vt/rocm/rocm_grouped_gemm.hip
       src/vt/rocm/rocm_sample.hip
       src/vt/rocm/rocm_gdn_state.hip
       src/vt/rocm/rocm_gdn_conv.hip

docs/FEATURES.md

@@ -264,7 +264,7 @@ both refuse, naming what is missing.
 | CPU (x86, Arm i8mm; A76 assembly correct/default, llama speed gate open, and the closed 20-core floor ran a SUPERSEDED fork denominator rather than the stock `b10451` pin, re-take owed #1003) | ✅ | ◐ | ☐ | ✅ |
 | Metal (Apple Silicon) | ✅ builds under Apple Clang with project warnings promoted to errors, the Qwen3.5 MoE loader included; its layout-refusal path uses the same messages and behavior on every platform (#1054) | ☐ | ☐ | ✅ |
 | Vulkan | ◐ | ☐ | ☐ | ✅ |
-| ROCm | W0 verified on 5 gfx archs; dense and GDN run all-native. **M3: `ROCM_ATTN` registered and selected per attention group** (#1056/#1065, [spec](../.agents/specs/rocm-attn-backend.md)). CPU parity open (#269) | 44 registered ops including full GDN; ctest-green gfx1151/1103/1100/1201/1200 ([#41](https://github.com/mudler/vllm.cpp/issues/41)). APU managed allocation is unverified. [ROCM.md](ROCM.md) | ✅ | ✅ |
+| ROCm | W0 verified on 5 gfx archs; dense and GDN run all-native. **M3: `ROCM_ATTN` registered and selected per attention group** (#1056/#1065, [spec](../.agents/specs/rocm-attn-backend.md)). CPU parity open (#269) | 47 registered ops including full GDN and MoE combine/gate; ctest-green gfx1151/1103/1100/1201/1200 ([#41](https://github.com/mudler/vllm.cpp/issues/41)). APU managed allocation is unverified. [ROCM.md](ROCM.md) | ✅ | ✅ |
 | XPU / TPU | ☐ | ✅ | ◐ | ☐ |
 | Tenstorrent Blackhole | ◐ `ACTIVE`, OPT-125m 6/6; Qwen3-0.6B wired; Mistral-7B-v0.3 16/16 on P150 ([spec](../.agents/specs/tenstorrent-mistral.md)). 16x16 rerun and residual-RMS owed ([spec](../.agents/specs/tenstorrent-backend.md)) | ✅ | ☐ | ☐ |
 | Tenstorrent host-free decode | ◐ env-gated `VT_TT_HOST_FREE_DECODE`; implementer P150 79-replay/5.8x. Default inert. New batch after capture refused. Engine golden owed | ☐ | ☐ | ☐ |
@@ -360,7 +360,7 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the
 | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server |
 | Multimodal over HTTP | Image request path wired; forward + codec pending | `ROAD-V1-MM` W1-W3 landed. Open: no mm-forward on `Request.mm_features`; no image codec. Video/audio/multi-image now **refuse** with HTTP 400 rather than drop ([#686](https://github.com/mudler/vllm.cpp/issues/686)) |
 | Reranking / classify models | Engine side only | Embeddings are LIVE (`LlamaModel`, `vllm_embed`, `/v1/embeddings`); the classify/score heads are landed ops with no registered arch |
-| ROCm | W0 community-verified on 5 gfx archs; classic-dense and GDN-hybrid e2e run all-native; correctness gaps remain | 44 registered ops including the GDN state/conv/postconv/recurrence set; APU managed-allocation branch remains unverified. [ROCM.md](ROCM.md) |
+| ROCm | W0 community-verified on 5 gfx archs; classic-dense and GDN-hybrid e2e run all-native; correctness gaps remain | 47 registered ops including the GDN state/conv/postconv/recurrence set and MoE combine/gate; APU managed-allocation branch remains unverified. [ROCM.md](ROCM.md) |
 | XPU, TPU | Not started | CUDA, CPU, Metal and Vulkan are the built backends |
 | Custom logits processors on CUDA | Open, not root-caused | Segfaults in a CUDA build, 232/232 green on CPU |
 | Memory budgeting (`ROAD-V1-MEM`, #83) | M1+M2 landed (absolute bytes) | `--kv-cache-memory` sizes the KV pool from an absolute byte budget (ABI v16, group-aware divisor); `--num-blocks` overrides; `--gpu-memory-utilization` needs the M3 profile run (dgx-gated). See `specs/kv-sizing.md` |

docs/USAGE.md

@@ -91,11 +91,14 @@ portable scan), and the norm-gate/preamble ops (`kRmsNormGated`,
 GDN-hybrid models call. Compressed conv/SSM state (bf16, the vLLM
 `mamba_cache_dtype` default) is advertised via the
 `SupportsCompressedConvState`/`SupportsCompressedGdnState` backend probes.
-MoE-path coverage is partial: `MoeRouterTopK` (f32/bf16 logits, ungrouped
-softmax, no bias) and `MoeSiluMul` are native; the remaining chain
-(`kSharedExpertGate`, `kMoeCombine`/`kMoeCombineGate`, and the grouped quant
-expert GEMM) is not registered yet, so MoE-bearing models still throw on
-those ops. On a
+MoE-path coverage: `MoeRouterTopK` (f32/bf16 logits, ungrouped softmax, no
+bias), `MoeSiluMul`, `SharedExpertGate`, `MoeCombine`, and `MoeCombineGate`
+are native. All three combine/gate ops accept f32 and bf16 operands and
+refuse anything else with a named message (f16 is not a supported arm). The grouped quant expert GEMM (`kMatmulBTQuantGrouped`) and the non-grouped
+`kMatmulBTQuant` are native for the formats the lane's GGUFs carry
+(Q8_0/Q4_K/Q5_K/Q6_K); other keep-quant formats (Q4_0, Q2_K, Q3_K, the IQ
+family, MXFP4) keep their expand-bf16 residency on ROCm rather than throwing
+at forward time. On a
 discrete card there is no CPU fallback tier, so a model whose layers call an op
 that is not registered yet fails loudly with `vt: no kernel for op N on device
 type 5` — that is the memory-safety design working, not a crash. Run with

src/vllm/model_executor/model_loader/gguf_keep_quant.cpp

@@ -110,6 +110,35 @@ bool KeepF16DType(uint32_t ggml_type) { return ggml_type == 1; }
 // ggml type id 40 is the NVFP4 fork extension; see gguf_dequant.cpp case 40.
 bool KeepNvfp4DType(uint32_t ggml_type) { return ggml_type == 40; }
 
+// Device-side keep-quant capability (review sweep on #523): the master
+// boolean `GgufQuantComputeAvailable()` only says the OP is registered; a
+// device's kernel set can be narrower than the CPU admission list, and on a
+// discrete backend with no CPU fallback tier a format the device cannot
+// execute must keep its pre-existing expand-bf16 residency -- flipping it to
+// a keep-quant block throws at FORWARD time with the whole model resident.
+// Per-device sets name what the registered kernels actually implement.
+bool DeviceKeepQuantSupported(vt::DType dt, vt::DeviceType dev) {
+  switch (dev) {
+    case vt::DeviceType::kROCM:
+      // src/vt/rocm/rocm_grouped_gemm.hip implements exactly these on both the
+      // grouped and non-grouped arms; Q4_0/Q2_K/Q3_K/IQ2_*/IQ3_*/MXFP4 are
+      // owed (recorded in .agents/specs/rocm-gg-keep-quant.md).
+      return dt == vt::DType::kQ8_0 || dt == vt::DType::kQ4_K ||
+             dt == vt::DType::kQ5_K || dt == vt::DType::kQ6_K;
+    default:
+      // CUDA falls back to the CPU kernel for anything it lacks
+      // (cuda_quant_dot.cu:1841-1846); the CPU list IS the CPU capability.
+      return true;
+  }
+}
+
+// keep-f16 needs an f16-capable MatmulBT on the running device; the ROCm
+// kernel accepts bf16/bf16 and f32/f32 only, so an F16 file weight must
+// expand there rather than be kept and refused at first forward (same review).
+bool DeviceKeepF16Supported(vt::DeviceType dev) {
+  return dev != vt::DeviceType::kROCM;
+}
+
 bool KeepQuantDType(uint32_t ggml_type, vt::DType* out) {
   vt::DType dt = vt::DType::kF32;
   if (!vt::BlockDTypeFromGgmlTypeId(ggml_type, &dt)) return false;
@@ -142,9 +171,14 @@ GgufResidency RouteGgufTensor(bool keep_quant, bool keep_f16, bool nvfp4_fp4,
     const int64_t k = KeepQuantKDim(role, shape);
     vt::DType dt = vt::DType::kF32;
     // ggml_row_size's precondition: a row is a whole number of blocks. A weight
-    // whose K is ragged cannot be dotted block-wise, so it expands.
+    // whose K is ragged cannot be dotted block-wise, so it expands. The device
+    // gate (review #523): a format the RUNNING device cannot execute keeps its
+    // pre-existing expand-bf16 residency instead of flipping to a keep-quant
+    // block that throws at forward time on a card with no CPU fallback tier.
     if (k > 0 && KeepQuantDType(ggml_type, &dt) &&
-        k % vt::BlockElems(dt) == 0) {
+        k % vt::BlockElems(dt) == 0 &&
+        DeviceKeepQuantSupported(
+            dt, vllm::platforms::CurrentPlatform().device_type())) {
       return GgufResidency::kKeepQuant;
     }
   }
@@ -231,7 +265,9 @@ GgufLoadPolicy GgufLoadPolicy::FromEnv() {
   //
   // VT_GGUF_KEEP_F16=0 is the opt-out; rides expand_nk so it is CPU-only and off
   // under VT_CPU_REF regardless (the oracle load stays byte-identical).
-  p.keep_f16 = EnvOnOr("VT_GGUF_KEEP_F16", p.expand_nk) && p.expand_nk;
+  p.keep_f16 = EnvOnOr("VT_GGUF_KEEP_F16", p.expand_nk) && p.expand_nk &&
+               DeviceKeepF16Supported(
+                   vllm::platforms::CurrentPlatform().device_type());
   // `QUANT-GGUF-NVFP4` column C. Same shape as the keep-quant default: ON
   // wherever the running device can execute the NVFP4 GEMM (CUDA today; a CPU
   // build keeps expanding, which is correct but unquantized), with