config(glm5): retune MXFP4 MoE token=8 to a4w4

[jiaryang]

Summary

• Follow-up to [config] retune moe for GLM-5.2 fp4 #4266: glm5_fp4_tuned_fmoe.csv already moved several inter_dim=512 tokens to flydsl_mxmoe a4w4, but token=8 was still on afp4/mfma. Non-MTP decode c8 (nextPow2(8)=8) hit that slow path.
• Retuned that single shape with gemm_moe_tune.py --mxfp4-flydsl on MI355X (gfx950).
• Best: flydsl_mxmoe_g1_a4w4_16x256x256_f16in_nt + flydsl_mxmoe_g2_a4w4_16x256x256_atomic (block_m=16, coupled us=68.581).

Selected kernels (token=8, 6144×512, E=257, topk=9)

Confirmed via fused_moe log on all three serving shapes below.

stage	before (afp4)	after (a4w4, this PR)
g1	flydsl_moe1_afp4_wfp4_bf16_t32x32x256_w3_xcd4_kw4_fp4	flydsl_mxmoe_g1_a4w4_16x256x256_f16in_nt
g2	flydsl_moe2_afp4_wfp4_bf16_t32x128x256_reduce_bnt2	flydsl_mxmoe_g2_a4w4_16x256x256_atomic

Serving A/B (MI355X TP4, GLM-5.2-MXFP4, non-MTP, c8)

Recipe: max-running-requests=8, DSA triton, kv fp8, SGLANG_ROCM_FUSED_DECODE_MLA=0, ROCM_QUICK_REDUCE_QUANTIZATION=INT4, SGLANG_OPT_USE_TOPK_V2=0.

• TTT = Total token throughput (tok/s) / 4
• TTT vs B200 = MI / B200 (higher is better)
• TPOT vs B200 = B200 / MI (higher is better)

shape	before TTT	after TTT	Δ TTT	TTT vs B200 (b→a)	before TPOT	after TPOT	Δ TPOT	TPOT vs B200 (b→a)
1024/1024	284.6	323.0	+13.5%	78% → 88% (365)	13.72	11.98	−12.7%	77% → 88% (10.6)
8192/1024	1035	1154	+11.5%	75% → 84% (1373)	15.63	13.83	−11.5%	80% → 90% (12.5)
70000/300	4777	4842	+1.4%	93% → 94% (5148)	77.3	71.6	−7.3%	84% → 91% (65)

Decode-heavy 1k/8k show the largest gain; 70k is flatter because attention/prefill dominate, but TPOT still improves.

Decode profile (TP-0, c8, 20 steps; earlier A/B)

	baseline	this change	Δ
MoE path	afp4 FlyDSL (mfma-class)	flydsl_mxmoe_*_a4w4_*	—
MoE sum (ms)	5.94	4.02	−1.9
Decode wall (ms)	13.84	12.02	−1.8

Test plan

• rm -f /tmp/aiter_configs/tuned_fmoe.csv; restart SGLang with/without AITER_CONFIG_FMOE override
• Confirm fused_moe selects a4w4 for (..., 8, 6144, 512, ...)
• Non-MTP c8 serving A/B on 1024/1024, 8192/1024, 70000/300 (table above)
• Decode profile A/B (MoE / wall)

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:gfx1250-ffm-triton	Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x	Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4798 --add-label <label>

[Copilot]

Pull request overview

This PR updates the GLM-5 FP4 tuned fused-MoE configuration for gfx950 by switching the token=8, model_dim=6144, inter_dim=512 shape from the prior afp4/mfma MoE kernels to the tuned flydsl_mxmoe a4w4 kernels, aligning it with other nearby token shapes already on a4w4.

Changes:

• Replace the gfx950 token=8, inter_dim=512 entry’s kernel selection with flydsl_mxmoe_g1_a4w4_16x256x256_f16in_nt + flydsl_mxmoe_g2_a4w4_16x256x256_atomic.
• Update the recorded us timing for that shape to 68.581 (and corresponding fields consistent with existing mxmoe rows).

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.