perf(ds4): default mixed ROCmFP MMQ prefill on gfx1151

[davide221]

Summary

Stacked on #667. This makes its registry-aware mixed ROCmFP MMQ prefill path the default on gfx1151 when DS4 uses dense or sparse approximate prefill.

The old path dequantized mixed expert weights and then called BLAS. The direct MMQ path removes that round trip and reduces the 8K trace from 325,841 to 73,165 GPU dispatches.

Exact prefill and non-gfx1151 devices keep their existing behavior. An explicit DFLASH_DS4_MIX_MMQ_PREFILL=0 remains the kill switch.

Strix Halo results

DeepSeek-V4-Flash-0731 ROCmFPX MIX, six routed experts, chunk 3072, sparse prefill, DSpark enabled, cold cache:

Prompt	Before	After	Gain
8,192	147.9 tok/s (55.4s)	209.0 tok/s (39.2s)	+41.3%
32,768	139.5 tok/s (234.9s)	185.8 tok/s (176.4s)	+33.2%
122,879	105.7 tok/s (1162.7s)	129.35 tok/s (950.0s)	+22.4%

The 8K result is a matched current-code kill-switch A/B. The longer comparisons use the same model, service flags, tokenizer-built prompt, and HTTP workload from the pre-change #667 captures. All measured contexts produced the same response SHA-256: 16fbd7d1f18d2fedb247d73edc3bc6aa040f5ab99bd3b48c35b79e543d22179b.

The gain narrows with context length because sparse attention becomes the remaining dominant cost. This does not claim Vulkan parity yet.

Reuse and safety

• The optimized loaders remain generic mixed-ROCmFP qtype MMQ code; this PR only adds a qualified DS4/gfx1151 default policy.
• Future model backends can select the same path after model/device qualification.
• Exact mode is not enabled automatically because MMQ changes reduction and codebook-quantization topology.
• Explicit environment policy is preserved, including the value 0 kill switch.

Verification

• HIP dflash_server and test_deepseek4_unit build
• test_deepseek4_unit on gfx1151
• test_inference_profile
• test_observability
• test_qwen35_roctx
• test_seq_engine_contract
• test_deepseek4_roctx
• test_server_unit: 431 passed, 0 failed
• Clean full API context sweep at 8K, 32K, and 122,879 tokens

Draft while stacked #667 lands and while we run broader output-quality and long-duration stability qualification.

[davide221]

Experimental long-context follow-up (still draft/opt-in): added a reusable D512 K==V streaming top-k HIP attention path behind DFLASH_DS4_DIRECT_INDEXER_TOPK=1 and GGML_CUDA_MLA_STREAM_TOPK=1. On gfx1151, three matched isolated graph runs improved 2627.3→2236.8 us (1.17x), 2645.5→2293.5 us (1.15x), and 2649.7→2363.3 us (1.12x). Matched ROCprof kernel time improved 2619.5→2126.6 us (-18.8%). test_deepseek4_unit and all 16 DS4 benchmark-tool tests pass. No full-model tok/s or output-parity claim yet; Lucebox8 is currently unreachable and the fallback host is booted with only ~61 GiB system RAM.

[davide221]

Pushed 8e49d7b as another opt-in burn-in candidate; the PR remains draft.

Measured on gfx1151:

• Grouped MoE combine, DS4 shape 4096 x 6 x 3072: old 4-op chain 12.16 ms, fused scalar 1.66 ms, aligned vec4 1.43 ms. CPU-reference NMSE 3.94e-15.
• Sparse grouped MMQ, K=4096, rows=2048, 256 experts, top-k 6, width 48, 20 iterations:
  • ROCmFP2: 6.078 -> 4.904 ms (-19.3%)
  • ROCmFP3: 6.325 -> 6.291 ms (neutral)
  • grouped ROCmFP4: 9.097 -> 4.058 ms (-55.4%)

The MMQ selector is format-aware and only activates for sparse grouped MoE on gfx1151 via GGML_CUDA_MMQ_MOE_ADAPTIVE_X=1; ordinary matmuls, other formats, and other devices are unchanged. Combine remains behind DFLASH_MOE_FUSED_COMBINE=1 and DFLASH_MOE_COMBINE_VEC4=1.

Validation: 76 grouped-MMID parity cases pass, masked routes remain +0, the ROCmFPX numerical suite passes, and dflash_server builds. Full DS4 throughput and response-hash A/B is still required before enabling either path by default; LB8 is currently unreachable and the fallback host cannot load the full model with its present memory split.

[davide221]

Pushed b0d5b1b: vectorized FP32 staging for the reusable D512 streaming indexed-attention candidate.

Qualification on gfx1151:

• Five alternating clean runs: F16 staging 2.603 ms mean, vectorized FP32 staging 2.363 ms mean (-9.3%).
• Same-run ROCprof: 2.382 ms -> 2.117 ms across 93 calls each (-11.1%).
• FP32-staged output is byte-identical to the F16-staged streaming output; the full DeepSeek4 unit suite passes and dflash_server builds.
• Geometry sweep rejected 8-head, 32-head, and 8-key stages; 16 heads x 16 keys remains fastest.

Enable with GGML_CUDA_MLA_STREAM_F32_STAGE=1 alongside the existing direct-indexer and streaming-attention flags. It remains opt-in because LB8 is still unreachable and the full 8K/32K/128K model-backed throughput/output A/B is pending.

[davide221]

Pushed 1b6cd67: coalesced ROCmFP3 MMQ weight loads.

For each eight FP3 values, one wave lane now reads the exact three packed bytes once and expands both adjacent int8 groups. On HIP, the three bytes come from one safe four-byte memcpy load within the 14-byte quant block.

Matched DS4 sparse grouped shape (K=4096, rows=2048, 256 experts, top-k 6, width 48): q3 mean 6.268 ms -> 6.114 ms (-2.46%); final 20-iteration run 6.111 ms. Q2/Q4 controls were unchanged. Grouped-output parity passed all 76 cases, the complete ROCmFP numerical suite passed, and dflash_server builds.

I also tested folding scale staging into the weight pass; it regressed Q2/Q3 by more than 2x and was discarded.

[davide221]

Pushed f140639: paired ROCmFP2 MMQ loads.

Adjacent FP2 four-value groups now share one aligned 16-bit HIP load per lane. On the matched sparse grouped shape, two 20-iteration runs averaged 4.894 ms before and 4.835 ms after (-1.20%). Q3/Q4 controls were unchanged. All 76 grouped cases passed, the complete ROCmFP numerical suite passed, and dflash_server builds.

[davide221]

Pushed f1039da: compact device-side sparse-MoE work queue for prefill-sized gfx1151 grouped ROCmFP MMQ (opt-in with GGML_CUDA_MMQ_MOE_PERSISTENT=1). It removes empty expert tiles without a host readback or synchronization and is model-neutral within the grouped ROCmFP path. The ordinary grid remains selected below 256 tokens.

Three alternating matched runs at the DS4 prefill shape (K=4096, rows=2048, 256 experts, top-k 6, width 3072) averaged:

• ROCmFP2: 34.834 -> 30.944 ms (-11.2%)
• ROCmFP3: 36.926 -> 32.147 ms (-12.9%)
• ROCmFP4: 25.154 -> 22.852 ms (-9.2%)

All baseline/candidate outputs across three runs had the same SHA-256. Additional outputs were byte-identical at widths 128/256/512/1024; all 76 grouped-MMID cases and the complete ROCmFP numerical suite pass, and dflash_server builds. Full-model 8K/32K/128K throughput qualification remains pending because LB8 is still not passing Tailscale data traffic; the PR remains draft.

[davide221]

Pushed 2d09e65: optional HIP hardware exponential for the reusable FP32-staged D512 streaming-MLA softmax (GGML_CUDA_MLA_STREAM_FAST_EXP=1). Five alternating runs averaged 2.359 ms for the existing FP32-staged kernel and 2.154 ms with the intrinsic (-8.7%). The stronger numerical fixture uses realistic ~unit-scale Q/K values; versus the precise FP32-staged output, NMSE was 1.97e-14 and max absolute error 4.1e-8. The complete DeepSeek4 unit suite passes and dflash_server builds. It remains opt-in and the PR remains draft pending the LB8 full-model A/B.