[gfx950][FlyDSL] Add dense BF16 x MXFP4 GEMM

[LiuYinfeng01]

Summary

Add a gfx950 FlyDSL dense GEMM for BF16 activations and packed MXFP4 weights.

• consumes BF16 activations without requantization
• decodes packed E2M1 weights with E8M0 scales into transient BF16 fragments
• uses BF16 MFMA with FP32 accumulation
• avoids both per-call full-weight dequantization and a persistent BF16 weight cache
• preshuffles packed weights and scales once at model load
• pipelines A/B loads with MFMA and dispatches measured Kimi-K3 tile configurations

BF16 activation --------------------> BF16 MFMA -> BF16 output
packed MXFP4 weight -> BF16 decode --^

This fused weight-dequant + BF16 GEMM path preserves emulation semantics. Direct A4W4 is intentionally out of scope and will be proposed separately.

Design and scope

The K loop double-buffers A in LDS. Prefill issues the next A/B tile before MFMA to preserve software-pipeline overlap. Split-K decode issues the next tile after MFMA, keeping one B tile live at a time and raising compiled occupancy for the 8448 x 7168 decode kernel from 1 to 2 waves/EU. The measured 1536 x 7168 decode configuration avoids split-K and its LDS reduction.

• gfx950 only
• packed MXFP4 weights and E8M0 scales are preshuffled once by the caller
• N and K must be divisible by 256
• downstream vLLM integration selects this path before falling back to cached BF16 emulation
• the unaligned (N,K)=(7168,4224) projection remains on the designed fallback

Packed weights plus scales use 73.44% less storage than cached BF16 weights. Warmed launches with preallocated output add zero bytes of peak PyTorch allocation.

Kernel-level correctness

Reference: canonical MXFP4 weight decode to BF16 followed by BF16 F.linear on the original activation. Measurements use deterministic random packed weights, E8M0 scale 0x7f, and M=128.

N x K	Existing Triton A4W4 relative L2	This PR relative L2	Existing exact	This PR exact
1024 x 3584	11.2099%	0.001776%	1.6022%	99.9924%
3584 x 512	11.2433%	0.001118%	1.5878%	99.9996%

The existing Triton path requantizes arbitrary BF16 activations to MXFP4, while this kernel preserves BF16 activation semantics. When activations already lie on the MXFP4 grid, all paths are exactly equal to the reference.

Kernel performance

Environment: MI355X (gfx950), PyTorch 2.11.0+gitd0c8b1f, HIP 7.2.53211, FlyDSL 0.3.0. Weight preparation and JIT compilation are excluded.

Versus the existing dynamic-MXFP4 Triton path

This PR is 1.75x-4.86x faster than the faster existing AITER Triton path across the measured Kimi-K3 TP2 shapes.

N x K	M	Existing best	This PR	Speedup
1024 x 3584	1	27.881 us	10.645 us	2.62x
1024 x 3584	512	29.059 us	16.589 us	1.75x
1024 x 3584	4096	214.094 us	44.048 us	4.86x
3584 x 512	1	28.659 us	8.360 us	3.43x
3584 x 512	512	29.730 us	10.912 us	2.72x
3584 x 512	4096	144.385 us	33.172 us	4.35x

Validation

• 27 passed: op_tests/flydsl_tests/test_flydsl_gemm_a16wfp4.py
• arbitrary BF16 activations against canonical decode + BF16 F.linear
• all E2M1 codes and E8M0 edge classifications
• packed-storage and no-full-BF16-allocation checks
• Ruff, Black, py_compile, and git diff --check

Kimi-K3 model-level validation

A three-way full GSM8K evaluation separates this fused kernel from the two non-fused emulation paths.

Setup

• 8x AMD Instinct MI355X (gfx950), TP8/EP8
• amd/Kimi-K3-Quark-MXFP4-AttnFP8, snapshot 7544f63f99ddf3729fdc01fcdfa7f1c6aaf66ac8
• identical TRITON_MLA, eager-mode, FP8 KV-cache, SiTUv2 routed-MoE and serving settings
• official GSM8K test set: 1,319 questions, 8-shot, temperature 0, seed 20260813, max output 512
• 32 concurrent workers, max_num_seqs=32, max_num_batched_tokens=8192
• only the dense/shared MXFP4 linear path changes:
  1. packed per-call dequantization (VLLM_MXFP4_EMULATION_DEQUANT_AT_LOAD=0)
  2. load-time cached BF16 weight + BF16 F.linear
  3. this PR's fused A16WFP4 kernel

Accuracy

Dense/shared linear path	Strict	Accuracy	Malformed	True NaN/Inf	Wall time
Packed per-call dequantization	1262/1319	95.68%	0	0	663.5 s
Cached BF16 emulation	1261/1319	95.60%	1	0	656.4 s
PR #4772 fused A16WFP4	1266/1319	95.98%	0	0	682.1 s

The first-pass harness used a raw substring check for nan, incorrectly flagging 15 otherwise-correct answers containing words such as financial. The table recognizes only standalone NaN/Inf tokens; all three runs had the same 15 false-positive indices and no true NaN/Inf output.

The fused route was selected consistently on all eight ranks for (N,K)=(8448,7168), (1536,7168), and (7168,768). The unaligned (7168,4224) projection used cached-BF16 fallback. This covers 185/186 dense/shared linear modules and 278/279 source weight tensors.

The five-answer difference between fused and cached is small and not statistically significant; it should not be interpreted as an accuracy improvement. This run shows no model-level accuracy regression from the fused path.

End-to-end serving performance

The full GSM8K run has variable output lengths. A separate fixed-token benchmark provides the controlled latency comparison.

Full GSM8K throughput

Dense/shared linear path	Completion tokens	Wall time	Output throughput	vs. cached
Packed per-call dequantization	132,926	663.5 s	200.33 tok/s	-1.53%
Cached BF16 emulation	133,546	656.4 s	203.44 tok/s	baseline
PR #4772 fused A16WFP4	132,532	682.1 s	194.30 tok/s	-4.49%

This is an end-to-end workload observation, not a fixed-output microbenchmark. Different generated reasoning lengths and batching schedules affect wall time.

Fixed-token decode

The controlled benchmark uses concurrency 1, one warmup, five measured requests, a fixed 457-token prompt, 512 generated tokens, temperature 0, and seed 20260813 on the same TP8/EP8 stack.

Metric	Cached BF16 emulation	PR #4772 fused	Fused delta
Mean TTFT	175.352 ms	177.054 ms	+0.97%
Mean TPOT	118.561 ms	120.468 ms	+1.61%
Mean E2E	60.760 s	61.736 s	+1.61%

The packed per-call path was not included in this fixed-token run. Historical measurements on an older runtime showed why caching is needed (402.43 ms to 90.97 ms concurrency-1 TPOT), but those numbers are not directly comparable to the current stack.

This model-level experiment does not claim an end-to-end speedup over cached BF16 emulation. The kernel-level speedup above is against the existing dynamic-MXFP4 Triton path. At model level, shape mix, activation QDQ, dispatch, and fallback overhead determine the result. The demonstrated model-level benefit is retaining packed weights without a persistent full BF16 weight cache while preserving accuracy.

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:gfx1250-ffm-triton	Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x	Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4772 --add-label <label>