feat(rocm): MoE combine/gate ops — SharedExpertGate, MoeCombine, MoeCombineGate (issue #41) by VikashLoomba · Pull Request #509 · mudler/vllm.cpp

VikashLoomba · 2026-08-12T17:43:56Z

Row

BACKEND-ROCM — the next links in the generic MoE path, after the router/silu-mul (#348). Issue #41. Claim CLAIM-ROCM-GDN-KERNELS continues.

What changed

NEW src/vt/rocm/rocm_moe_chain.hip with three ops, hand-translated from cuda_moe.cu (MoeCombineKernel :473, MoeCombineGateKernel :555) + the SharedExpertGate CPU oracle (cpu_ops.cpp:2387):

kSharedExpertGate — out[t,c] = sigmoid(gl[t]) * sd[t,c] (bf16 out, f32/bf16 sd)
kMoeCombine — weighted top-k expert sum + optional shared term
kMoeCombineGate — combine with the shared-expert sigmoid gate folded in, the shared term rounded through bf16 exactly as the donor

All grid-stride, f32 math, bf16/f32 dtype arms via boundary conversions. New cross-device case gates all three (MoeCombineGate's oracle is the host-computed composite — no CPU op registration exists for it).

Evidence (4× gfx1100, ROCm 7.14, Release)

MoE combine/gate case: 9/9 assertions, runs not skips
ctest -R 'rocm|cross_device': 4/4
full ctest: pre-existing failure set shrinks 7 → 5 — test_bench and test_capi now PASS (they failed at op 77 / the router dtype before the chain). test_loaded_engine_dense now fails only on the async-scheduling assertion (runner_supports_async()=false on ROCm), a lane capability gap, not a kernel throw.
preflight --staged + trailers green

Speed claims

This PR makes NO speed claim.

Honest gaps

Named remaining blocker: the grouped quant expert GEMM (kMatmulBTQuantGrouped, the DeepSeek-V4 keep-quant family) — the heavy lift for Qwen3.5-27B-class GDN-MoE models, a proper kernel project of its own.
The router's grouped/bias/noaux_tc forms still throw by design (same guard as feat(rocm): gfx1201 hipBLAS ops + Gemma-4-26B-A4B MoE (BF16/FP8) #140); the covered path is ungrouped softmax.

Current-main integration cutoff (2026-08-18)

Rebased from 65951efbf8e941f669b95ae4e15f2bd290a3fde2 to fa2e628078c663f0d4923396593ab9086bb60113 on recorded current-main cutoff 65d6cdaed3e20e9bc70b4f9374fccafefefa7bd0. Exact three-commit replay; main’s ROCM_ATTN M3 fact and this PR’s 47-op MoE projection remain intact. Fresh cutoff-head builds, focused gates, full repository preflight, and independent mutation review all pass.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: pi:kimi-k3 [pi]
Assisted-by: codex:gpt-5.6-sol [codex]
Assisted-by: pi:gpt-5.6-sol [pi]

localai-bot · 2026-08-13T17:13:59Z

Reviewed as part of a sweep over the open external PRs. The translations are faithful — I checked all three donor anchors at your base SHA and they resolve exactly (cuda_moe.cu:473 is MoeCombineKernel, :555 is MoeCombineGateKernel, cpu_ops.cpp:2387 is SharedExpertGateKernel), and the kernel bodies match the donors call for call, including the __bfloat162float(__float2bfloat16(sv)) double-round. Registration goes through the existing registrar with no parallel path.

Two things to fix. Both are the same shape as findings on #506 and #523 — the calculation ported cleanly, the guards around it did not — so it is probably worth reading the three together.

1. The donor's dtype refusals were dropped, and f16 is reachable.

MoeCombineKernelCuda (cuda_moe.cu:520-524) and MoeCombineGateKernelCuda (:597-604) each open with VT_CHECKs refusing anything but f32/bf16, with a named message. The ROCm entry points have none, and dispatch is a bare if (expert_out.dtype == DType::kBF16) ... else <float>.

f16 gets through: vt::MoeCombine gates on IsFloat (src/vt/ops.cpp:21 — kF32 || kF16 || kBF16) for both expert_out and shared, and Tensor::Ptr<T>() (include/vt/tensor.h:66-68) is an unchecked static_cast with no dtype assertion. So an f16 expert_out [T,K,H] passes the seam, falls into the else branch, and the kernel reads 4 bytes per element out of a 2-bytes-per-element allocation — T*K*H*2 bytes past the end, garbage out, no error anywhere. AGENTS.md is explicit that an unimplemented arm is refused with a message naming the missing piece.

2. The new test exercises the dtype arm the model never runs.

On the live path expert_out is bf16 (qwen3_5.cpp:5463, DBuf ddown(d, DType::kBF16, {P, H})), dout is bf16, and the unfused arm passes a bf16 shared (:5326). The new case in test_backend_cross_device.cpp only ever constructs DType::kF32 for expert_out, shared, sd and out, so MoeCombine and MoeCombineGate are tested exclusively as <float,float,float>. The comment says "f32 and bf16 arms"; only SharedExpertGate actually exercises a bf16 store.

Consequence: an inverted predicate in by_out/by_shared/by_sd, or a wrong bf16 Ld/St boundary, is a 2x out-of-bounds read on every MoE layer of every token — and the case still reports 9/9 green, because those branches are never entered.

Smaller ones: CMakeLists.txt:1396-1397 has src/vt/rocm/rocm_moe_chain.hip twice with mangled indentation (inert, but it is exactly the shape that merges badly against another additive ROCm PR touching the same list); docs/FEATURES.md:232,320 still say "44 registered ops" where the count is now 47; and the MoeCombine f32 arm is gated at NMSE 5e-4 where the donor asserts bit-exactness by design (cuda_moe.cu:465-468 — single store-rounding, same as the CPU reference) and the ROCm kernel is thread-per-element with no cross-lane reduction, so CHECK(got == ref) is achievable and is the stronger gate. Your SharedExpertGate check already does exactly that.

Credit where due: recomputing the MoeCombineGate oracle from scratch on host rather than routing it through a shared helper is the right call, and your stated reason for it (no CPU registration for kMoeCombineGate) checks out — cpu_ops.cpp registers kMoeCombine only.

No AMD hardware here, so your 9/9 and the ctest results could not be reproduced and I am not disputing them; both findings are static, read from the seam and the model path.

…ms -- the mudler#509 review rework Review sweep findings (localai-bot, 2026-08-13), all accepted: 1. The donor's dtype refusals were dropped: cuda_moe.cu:520-524/:597-604 open with VT_CHECKs refusing non-f32/bf16; without them an f16 expert_out passes the seam's IsFloat gate and Tensor::Ptr<T>()'s unchecked cast reads 4 bytes per element from a 2-byte allocation. Refusals added to all three ROCm entry points (SharedExpertGate included: its 4-arm dispatch has the same f16 hazard on sd). 2. The case now exercises the PRODUCTION dtype mix, not only f32: the model path runs expert_out bf16 (qwen3_5.cpp DBuf ddown), shared bf16, out bf16. The new bf16 arm is asserted BIT-EXACT against the CPU reference (both sides thread-per-element, same sequential K order, single store rounding, -ffp-contract=off), and the f32 MoeCombine arm is tightened from NMSE to bit-exact per the donor's design comment (cuda_moe.cu:465-468). Writing the bf16 arm caught a construction bug in the first version of it (an f32-typed tensor over a bf16 buffer -- exactly the OOB class the review predicted the missing arm hid); fixed and re-verified against an independent host composite (0/320) plus a raw-hipMalloc scratch replica of both backends. 3. CMakeLists.txt: the mangled duplicate rocm_moe_chain.hip line removed. 4. docs/FEATURES.md op count 44 -> 47 (counted: the registration sites). Gates (gfx1100, flock): test_backend_cross_device 20/20 (bit-exact bf16 + f32 arms). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]

VikashLoomba · 2026-08-14T09:07:52Z

Both findings accepted and reworked (commits dfd1213 + ebb4c6b, rebased onto current main):

dtype refusals restored: all three ROCm entry points now open with the donor's VT_CHECKs (f32/bf16 only, named messages) — the f16-through-IsFloat → unchecked-Ptr<T> OOB read is refused loudly. SharedExpertGate got the same treatment (its 4-arm dispatch had the same hazard on sd).
Production dtype arms: the case now exercises the mix the model actually runs (expert_out bf16 / shared bf16 / out bf16, per qwen3_5.cpp) in addition to f32 — and both MoeCombine arms are asserted bit-exact, not NMSE (thread-per-element, no cross-lane reduction, -ffp-contract=off; the donor's design comment says the same). Writing the bf16 arm immediately caught a construction bug in my first version of it (an f32-typed tensor over a bf16 buffer — the exact class you predicted the missing arm hid), which I root-caused against an independent host composite before believing either side.

Also: the mangled duplicate rocm_moe_chain.hip line in CMakeLists.txt is removed; FEATURES.md op count corrected 44 → 47 (counted the registration sites); and the rebase onto main surfaced main's new MoeCombineFn(..., float routed_scale) — plumbed through the ROCm kernel in the CPU reference's order (scale the routed sum, then add shared), with the bf16 arm now running at scale 0.7 so the multiply is exercised.

Gates: test_backend_cross_device 20/20 on gfx1100.

…ms -- the mudler#509 review rework Review sweep findings (localai-bot, 2026-08-13), all accepted: 1. The donor's dtype refusals were dropped: cuda_moe.cu:520-524/:597-604 open with VT_CHECKs refusing non-f32/bf16; without them an f16 expert_out passes the seam's IsFloat gate and Tensor::Ptr<T>()'s unchecked cast reads 4 bytes per element from a 2-byte allocation. Refusals added to all three ROCm entry points (SharedExpertGate included: its 4-arm dispatch has the same f16 hazard on sd). 2. The case now exercises the PRODUCTION dtype mix, not only f32: the model path runs expert_out bf16 (qwen3_5.cpp DBuf ddown), shared bf16, out bf16. The new bf16 arm is asserted BIT-EXACT against the CPU reference (both sides thread-per-element, same sequential K order, single store rounding, -ffp-contract=off), and the f32 MoeCombine arm is tightened from NMSE to bit-exact per the donor's design comment (cuda_moe.cu:465-468). Writing the bf16 arm caught a construction bug in the first version of it (an f32-typed tensor over a bf16 buffer -- exactly the OOB class the review predicted the missing arm hid); fixed and re-verified against an independent host composite (0/320) plus a raw-hipMalloc scratch replica of both backends. 3. CMakeLists.txt: the mangled duplicate rocm_moe_chain.hip line removed. 4. docs/FEATURES.md op count 44 -> 47 (counted: the registration sites). Gates (gfx1100, flock): test_backend_cross_device 20/20 (bit-exact bf16 + f32 arms). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]

Fold research 6195: ProductGetBlasStreamIsCapturing calls the exact HipBlasHooks hook. HIP product probe begins capture and asserts true; always-false hook mutation is RED. Host fake-capture case unchanged. 1a1153d6 is not a review target. Adjacent mudler#785/mudler#523/mudler#509/mudler#834 noted in spec; no pickup. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.6 [Hermes]

… OFF Implements [`.agents/specs/rocm-decode-attn-d128.md`](https://github.com/mudler/vllm.cpp/blob/main/.agents/specs/rocm-decode-attn-d128.md), which landed ahead of this change as #564. The ROCm half of #382. The CUDA half merged as #425 (`66399617`); this mirrors it, adopting that arm's flag, default and stated reason rather than inventing new ones. ## What changes `d == 128` — the Qwen3-dense / Llama / Mistral head size — reaches the fast decode kernels instead of falling through to the generic `PagedAttnOnline`. `LoadRowEplBf16`/`StoreRowEplBf16` gain an `EPL=4` (`uint2`) case beside the existing `EPL=8`/`16`; the dispatch gates and the two launch switches gain a `d == 128` arm. No new kernel and no new algorithm — the kernel bodies were already generic over `EPL`. **Default OFF, opt in with `VT_ATTN_DECODE_D128=1`** — the same env var, default and reason as the merged CUDA arm. The arm is correctness-complete but not byte-exact against the kernel it replaces: warp-strided online softmax reduces the KV sequence in a different **order** than `PagedAttnOnline`'s per-tile loop, so a greedy anchor can move at an exact bf16 tie. Shipping OFF keeps every existing golden byte-identical. The flip owes the near-tie razor, a distributional gate and regen under the ratified-tie rule, and per the spec must be argued **per backend** — see the reversal below. That is what keeps #382 open. ## Reviewer note Spec §4 item 3 writes the gate as `(d == 128 && (decode_d128 || decode_wmma))`. This commit implements it **without** the `decode_wmma` disjunct, which is what the same item's "Forward reference" paragraph instructs: `VT_ATTN_DECODE_WMMA` does not exist in the tree, and the flag lands with the rocWMMA arm on its own branch. The difference is intentional; it is visible in the diff before the note explaining it is. ## Evidence gfx1200 (RX 9060 XT, RDNA4, 32 CU), ROCm 7.2.3, `$GPU_LOCK` held. All figures are a same-binary flag A/B — no rebuild between arms — at 1024-token synthetic prompt, 128 generated, greedy, seed 0, 2 reps per cell agreeing within ~1%. | Model | head_dim | decode path | TPOT OFF | TPOT ON | speedup | |---|---|---|---|---|---| | Qwen3-0.6B | 128 | `qg=2` fused | 42.53 ms | 11.78 ms | **3.61x** | | Qwen3-1.7B | 128 | `qg=2` fused | 52.85 ms | 21.93 ms | **2.41x** | | Qwen3-4B | 128 | `qg=4` per-head | 81.89 ms | 39.22 ms | **2.09x** | | Qwen3.5-0.8B | 256 | — (control) | 23.76 ms | 23.55 ms | 1.01x | Qwen3-4B has no GQA fusion at any head_dim, so its 2.09x isolates the `EPL` widening from the fusion. **Qwen3.5-0.8B is the negative control and it earned its keep.** Its `head_dim` is 256, so the `d == 128` gate provably cannot reach it. Its first OFF rep came in a 33% outlier at 31.14 ms, which a blind 2-rep average would have reported as a ~1.2x "win" for a model the flag cannot affect. Re-run three times: 23.86 / 23.75 / 23.68 against ON's 23.52 / 23.57. End-to-end output throughput rises less than TPOT on the same runs (0.6B 2.48x, 1.7B 2.05x, 4B 2.02x) because they carry a 1024-token prefill the flag does not touch. TPOT isolates decode; throughput dilutes it. ### Concurrency — the advantage grows, it does not compress Qwen3-1.7B, `--num-prompts` = 2x concurrency: | Conc | tok/s OFF | tok/s ON | ratio | TPOT ratio | |---|---|---|---|---| | 1 | 12.89 | 24.66 | 1.91x | 2.40x | | 2 | 23.27 | 47.45 | 2.04x | 2.45x | | 4 | 39.10 | 86.86 | 2.22x | 2.46x | | 8 | 58.97 | 147.35 | **2.50x** | **2.77x** | | 16 | 78.43 | 227.08 | **2.90x** | **3.18x** | This refuted the prediction made before the run, which reasoned that a tiny grid at concurrency 1 flatters the fast kernel. The dominant effect is the reverse: from c8 to c16 the fallback scales only **1.33x** against the arm's **1.54x**, and scaling efficiency at c16 relative to perfect-linear-from-c1 is **38% OFF against 58% ON**. `PagedAttnOnline` is the batch-scaling bottleneck, not merely slow per call, so the win is largest in the regime a server actually runs in. The c1 row reproduces an independently-run four-model sweep to within ~1% (52.85/21.93 there vs 53.40/22.26 here). ### Correctness - `ctest -R 'rocm|cross_device'` **5/5**, chained directly to the exact-SHA push. - New case: "paged attention at Qwen3 geometry (bf16, GQA 2, head_dim 128) matches the CPU oracle", looped over `RegisteredDevices()`, NMSE <= 5e-4 vs the CPU oracle plus `OpProviderStats::declines == 0`. Genuinely new coverage — the existing generic cross-device test runs at `d=8, f32` and never reached any bf16 `EPL`-templated kernel, so none of them had bf16 coverage in this suite. (The merged CUDA arm shipped with no test at all.) - Because the arm ships OFF **and** its flag is read into a `static const bool` — once per process — the default registration can only ever gate the fallback. `tests/CMakeLists.txt` adds a second invocation with the flag set, same shape as the existing `test_dense_gateup_fused_marlin_off_*` pair. Verified non-vacuous against the #463 trap: 1 case, 6 assertions, not zero. - Full `ctest` 448/455. The 7 failures are **proven** pre-existing, not asserted: a clean `main` `2784dd7b` worktree built from source with none of this code fails the identical set (only `test_op_parity`'s index shifts 403 -> 404, from the added registration). They are a missing `shellcheck`, an mmap-RSS assertion, a JSON type error, and the `SharedExpertGate` ROCm registration gap owed to unmerged #509. - `agent-preflight` fails 9, a strict **subset** of that same baseline's 10 (differing only by `role-undeclared`). `check-commit-trailers` and `check-doc-checkpoint` both pass against this base. ## Carried finding #382 measured this same `EPL=4` arm **1.6x slower** on sm_110 / Jetson AGX Thor, where gfx1200 measures it 2-3.6x faster. Recorded, not reconciled — different kernels, different fallbacks, different memory systems. It is why the default-ON flip must be argued per backend rather than once, and it is preserved in the spec rather than averaged away. ## Against the pinned oracle: 6.35x to 1.75x slower on per-token decode Measured after the tables above, with **both sides in the same container**, oracle = vLLM `555967922` in its production configuration via `vllm bench serve`. Qwen3-0.6B, 1024 in / 128 out, concurrency 1, **8 prompts**, warmup discarded, **3 reps**: | | TPOT reps | mean | vs oracle | |---|---|---|---| | ours, flag unset | 42.54 / 42.46 / 42.19 | 42.40 ms | 6.35x slower | | ours, `VT_ATTN_DECODE_D128=1` | 11.97 / 11.38 / 11.66 | **11.67 ms** | **1.75x slower** | | vLLM `555967922` | 6.57 / 6.90 / 6.58 | 6.68 ms | — | Running our binary against the container's ROCm rather than the host's is a substitution, so it was proved inert first: in-container matches native at 42.79 vs 42.53 ms unset, and 12.03 vs 11.78 ms with the flag. The prompt count is load-bearing. At `--num-prompts 2` the oracle returned TPOT **6.96 ms and 13.45 ms on consecutive reps**, a ~2x spread averaging to a plausible-looking and entirely fictional number. At 8 prompts with a discarded warmup both sides hold to ~±0.3 ms. This number lived only in a PR comment, which a squash merge does not carry into the tree. It is now in the spec's §5 and appended to `.agents/benchmark-record.md`, with its caveats attached rather than trailing. ## The container/glibc blocker was RETRACTED An earlier revision of this body, and the spec's §6, said the oracle re-measure was blocked on a Nix-glibc vs container-glibc ABI mismatch. **That diagnosis was wrong and is retracted.** Our binary runs inside the pinned oracle container; the earlier failures were self-inflicted (`LD_LIBRARY_PATH` exported container-wide, which breaks the container's own tools, plus a bind mount that silently yielded nothing and looked exactly like a missing ELF interpreter). §6 now reads "not run — **not blocked**", and the WMMA-spec cross-reference is gone. A false blocker in the record is worse than no record, because it stops the next person from trying. ## Not claimed **This does not close #488.** That issue asks for a **per-call** kernel comparison and explicitly asserts no cause. The number above is **per-token latency** with asymmetric harnesses — the oracle over HTTP via `vllm bench serve`, ours in-process — so TPOT is the only comparable axis, and TTFT, E2EL and end-to-end throughput carry the oracle's HTTP and tokenizer overhead and are directional only. It is not the same-tool per-call trace `AGENTS.md` wants before a throughput claim. `rocprofv3` is present in the container and our binary traces under it; what is still owed is decode-phase windowing on the oracle side, or the trace compares our decode against vLLM's model load and graph capture. One board, one shape. **`docs/BENCHMARKS.md`'s ROCm axis stays PENDING**, and the row this PR adds is marked DIRECTIONAL and sits beside the existing row rather than overwriting it. **The flag-ON arm still has no proof it REACHES the new kernel, now filed as #1134.** `RegisteredDevices()` (`tests/vt/test_backend_cross_device.cpp:84-96`) enumerates `{kCUDA, kMETAL, kVULKAN, kXPU, kROCM}` and excludes `kCPU`, so on a CPU-only runner — which is what CI has — the new case reports 1 test case, 0 assertions, exit 0, for **both** registrations. On ROCm the case's only backend assertion is `OpProviderStats::declines == 0`, counted at **provider** granularity, so it is identical with the flag set and unset. §9 stop condition 2 is left OPEN. The spec disclosed this honestly; what was missing is the issue `AGENTS.md` requires for a known gap not fixed in flow. Searched before filing: not a duplicate of #463 (the unset-weights-env-var shape, which does not describe the `declines` half), #785 (a kernel that never LAUNCHES behind a dead `#if`) or #900 (same family, LTX-2.5 subject). Also out of scope and named in the spec's new `## Owed` section: the dtype gap (ROCm's decode-opt is bf16-only at every head_dim, so 4 of 5 dtype combinations still fall to `PagedAttnOnline` at `d=128` — pre-existing, inherited, not introduced), `qg=4`/`qg=8` fusion, `d=128` prefill, and the rocWMMA arm. ## Record repairs carried in the final commit `docs(BACKEND-ROCM): retract the blocker, keep the oracle number, and file the gap`, on top of joral's commits, which are untouched. It carries the retraction above; the oracle number into §5 and `.agents/benchmark-record.md`; `docs/BENCHMARKS.md` and `docs/STATUS.md` reconciled; §7's stale "**two** flag-on ctest registrations" corrected to one, matching what `8aedd780` already fixed in §4 item 3 and the Test-coverage section; a literal `## Owed` heading over the owed list; and #1134 filed and appended to `.agents/issue-index.md`. Two comment-only edits at `rocm_paged_attn.hip:330` and `:455`, which still enumerated "EPL=8 → d=256, EPL=16 → d=512" without the new `EPL=4` case although the top-of-file comment at `:264` had been updated. The branch is **rebased** onto `origin/main` `d1e5e9bc` — it was 39 commits behind, and the rebase drops the earlier `merge: upstream/main` commit. Gates rerun from the worktree with explicit SHAs: `check-commit-trailers`, `check-commit-style`, `check-doc-checkpoint`, `check-public-doc-tables`, `check-agent-record` and `check-pr-size`, all OK. Not rebuilt and not re-run on hardware: every gfx1200 figure here is joral's, unchanged. Row: BACKEND-ROCM Issue: #382 Issue: #1134 Spec: #564 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] Assisted-by: ClaudeCode:claude-opus-5 [ClaudeCode]

…ms -- the mudler#509 review rework Review sweep findings (localai-bot, 2026-08-13), all accepted: 1. The donor's dtype refusals were dropped: cuda_moe.cu:520-524/:597-604 open with VT_CHECKs refusing non-f32/bf16; without them an f16 expert_out passes the seam's IsFloat gate and Tensor::Ptr<T>()'s unchecked cast reads 4 bytes per element from a 2-byte allocation. Refusals added to all three ROCm entry points (SharedExpertGate included: its 4-arm dispatch has the same f16 hazard on sd). 2. The case now exercises the PRODUCTION dtype mix, not only f32: the model path runs expert_out bf16 (qwen3_5.cpp DBuf ddown), shared bf16, out bf16. The new bf16 arm is asserted BIT-EXACT against the CPU reference (both sides thread-per-element, same sequential K order, single store rounding, -ffp-contract=off), and the f32 MoeCombine arm is tightened from NMSE to bit-exact per the donor's design comment (cuda_moe.cu:465-468). Writing the bf16 arm caught a construction bug in the first version of it (an f32-typed tensor over a bf16 buffer -- exactly the OOB class the review predicted the missing arm hid); fixed and re-verified against an independent host composite (0/320) plus a raw-hipMalloc scratch replica of both backends. 3. CMakeLists.txt: the mangled duplicate rocm_moe_chain.hip line removed. 4. docs/FEATURES.md op count 44 -> 47 (counted: the registration sites). Gates (gfx1100, flock): test_backend_cross_device 20/20 (bit-exact bf16 + f32 arms). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]

VikashLoomba · 2026-08-18T06:42:58Z

Current-main rebase and fresh review

Rebased from 6fc195df to 0961356b on current-main pin f22c6cc8 and force-updated with an explicit lease.

Range-diff: kernel/test commits replay exactly; the only context change preserves main's adjacent Tenstorrent documentation row byte-for-byte.
Fresh HIP build and full cross-device gate: 21/21 cases, 352/352 assertions.
Production dtype refusals and grouped/non-grouped registration checks remain non-vacuous on ROCm.
Repository preflight: All gates green.
Fresh independent integration review: PASS; final tree clean.

PR #523 is restacked on this exact immutable parent.

…mbineGate) (mudler#41) The next links in the generic MoE path after the router/silu-mul. Hand- translated from cuda_moe.cu (MoeCombineKernel :473, MoeCombineGateKernel :555) and the SharedExpertGate CPU oracle (cpu_ops.cpp:2387). Grid-stride, f32 math, bf16/f32 dtype arms via boundary conversions; the combine-gate folds the shared-expert sigmoid gate rounded through bf16 exactly as the donor. Evidence (4x gfx1100, ROCm 7.14, Release): - new MoE combine/gate cross-device case: 9/9 assertions (MoeCombineGate's oracle is the host-computed composite — no CPU op registration exists) - ctest -R 'rocm|cross_device': 4/4 - full ctest: pre-existing failure set shrinks 7 -> 5; test_bench and test_capi now PASS (they failed at op 77 / the router dtype before the chain). test_loaded_engine_dense now fails only on the async-scheduling assertion (a lane capability gap, not a kernel throw). - Named remaining blocker: the grouped quant expert GEMM (kMatmulBTQuantGrouped), the DeepSeek-V4 keep-quant family. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]

…ms -- the mudler#509 review rework Review sweep findings (localai-bot, 2026-08-13), all accepted: 1. The donor's dtype refusals were dropped: cuda_moe.cu:520-524/:597-604 open with VT_CHECKs refusing non-f32/bf16; without them an f16 expert_out passes the seam's IsFloat gate and Tensor::Ptr<T>()'s unchecked cast reads 4 bytes per element from a 2-byte allocation. Refusals added to all three ROCm entry points (SharedExpertGate included: its 4-arm dispatch has the same f16 hazard on sd). 2. The case now exercises the PRODUCTION dtype mix, not only f32: the model path runs expert_out bf16 (qwen3_5.cpp DBuf ddown), shared bf16, out bf16. The new bf16 arm is asserted BIT-EXACT against the CPU reference (both sides thread-per-element, same sequential K order, single store rounding, -ffp-contract=off), and the f32 MoeCombine arm is tightened from NMSE to bit-exact per the donor's design comment (cuda_moe.cu:465-468). Writing the bf16 arm caught a construction bug in the first version of it (an f32-typed tensor over a bf16 buffer -- exactly the OOB class the review predicted the missing arm hid); fixed and re-verified against an independent host composite (0/320) plus a raw-hipMalloc scratch replica of both backends. 3. CMakeLists.txt: the mangled duplicate rocm_moe_chain.hip line removed. 4. docs/FEATURES.md op count 44 -> 47 (counted: the registration sites). Gates (gfx1100, flock): test_backend_cross_device 20/20 (bit-exact bf16 + f32 arms). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]

…#684) plumbed through the ROCm arm The rebase onto current main brought main's new MoeCombineFn signature (routed_scale, default 1.0f, scaling the ROUTED sum before the shared term -- upstream apply_routed_scale_to_output). The ROCm kernel applies it in the same f32 accumulator in the same order (one standalone multiply on the finished sum, bit-identical to the CPU reference under -ffp-contract=off); the forward declaration in rocm_ops.hip is updated to match. The bf16 test arm now runs at scale 0.7 so the multiply is exercised, not just the 1.0 passthrough. Gates (gfx1100, flock): test_backend_cross_device 20/20 (bit-exact arms unchanged at 1.0, bit-exact at 0.7). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]

VikashLoomba · 2026-08-18T12:24:11Z

Final integration cutoff

Final head: fa2e628078c663f0d4923396593ab9086bb60113, exact replay on cutoff 65d6cdae.

Fresh HIP cross-device gate: 21/21 cases, 352/352 assertions.
Main’s ROCM_ATTN runner selection and this PR’s 47-op MoE projection are both preserved.
Full repository preflight: All gates green.
Independent cutoff review: PASS, exact range-diff and clean tree.

PR #523 is stacked on this exact parent.

VikashLoomba mentioned this pull request Aug 12, 2026

feat(rocm): grouped + non-grouped keep-quant expert GEMM — the MoE-path terminus (issue #41) #523

Open

1 task

joral mentioned this pull request Aug 13, 2026

spec(BACKEND-ROCM): the ROCm head_dim=128 decode arm, the ROCm half of #382 #564

Merged

localai-bot mentioned this pull request Aug 13, 2026

feat(rocm): wvSplitK skinny GEMM for decode M<=4 — the #487 decode GEMM lever #506

Open

1 task

VikashLoomba force-pushed the row/ROCM-MOE-CHAIN branch from 8a4ed36 to ebb4c6b Compare August 14, 2026 08:54

VikashLoomba force-pushed the row/ROCM-MOE-CHAIN branch from ebb4c6b to 859a666 Compare August 14, 2026 09:33

joral mentioned this pull request Aug 14, 2026

perf(rocm): head_dim=128 decode arm -- the ROCm half of #382, default OFF #767

Merged

VikashLoomba force-pushed the row/ROCM-MOE-CHAIN branch from 859a666 to d5fb6eb Compare August 14, 2026 17:25

VikashLoomba force-pushed the row/ROCM-MOE-CHAIN branch from d5fb6eb to 9aefaec Compare August 14, 2026 19:12

VikashLoomba force-pushed the row/ROCM-MOE-CHAIN branch from 9aefaec to 6fc195d Compare August 18, 2026 06:07

VikashLoomba force-pushed the row/ROCM-MOE-CHAIN branch from 6fc195d to 0961356 Compare August 18, 2026 06:41

VikashLoomba marked this pull request as ready for review August 18, 2026 06:43

VikashLoomba added 3 commits August 18, 2026 04:43

VikashLoomba force-pushed the row/ROCM-MOE-CHAIN branch from 0961356 to fa2e628 Compare August 18, 2026 12:22

Navigation Menu

feat(rocm): MoE combine/gate ops — SharedExpertGate, MoeCombine, MoeCombineGate (issue #41) - #509

feat(rocm): MoE combine/gate ops — SharedExpertGate, MoeCombine, MoeCombineGate (issue #41)#509
VikashLoomba wants to merge 3 commits into
mudler:mainfrom
VikashLoomba:row/ROCM-MOE-CHAIN

VikashLoomba commented Aug 12, 2026 •

edited

Loading

Uh oh!

localai-bot commented Aug 13, 2026

Uh oh!

VikashLoomba commented Aug 14, 2026

Uh oh!

VikashLoomba commented Aug 18, 2026

Uh oh!

VikashLoomba commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

VikashLoomba commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Row

What changed

Evidence (4× gfx1100, ROCm 7.14, Release)

Speed claims

Honest gaps

Current-main integration cutoff (2026-08-18)

Uh oh!

localai-bot commented Aug 13, 2026

Uh oh!

VikashLoomba commented Aug 14, 2026

Uh oh!

VikashLoomba commented Aug 18, 2026

Current-main rebase and fresh review

Uh oh!

VikashLoomba commented Aug 18, 2026

Final integration cutoff

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

VikashLoomba commented Aug 12, 2026 •

edited

Loading