perf(configs): tune bf16 GEMM for gfx1250 DeepSeek-V4-Flash shapes

[yichiche]

Summary

• Adds tuned bf16 A16W16 GEMM configs for gfx1250 (96 CU) covering the DeepSeek-V4-Flash shapes: N ∈ {64, 256, 512, 1024, 2048} at K = 4096, bf16 in / bf16 out.
• Without them these shapes were untuned on gfx1250 and fell through to the torch default, where rocBLAS picks a degenerate kernel (cost flat in M).
• GPU kernel time improves by geomean 6.82x across all 120 shapes with zero regressions; output is bitwise-identical to F.linear on every shape.
• Config-only change: no kernel, no dispatch logic. Other architectures are untouched.

Motivation

get_GEMM_A16W16_config() looks up (gfx, cu_num, M, N, K, bias, dtype, outdtype, scaleAB, bpreshuffle). DeepSeek-V4-Flash (hidden_size 4096) issues bf16 GEMMs with N ∈ {64, 256, 512, 1024, 2048} at K = 4096, and none of those had a gfx1250 row, so every one of them hit the torch default.

That default is pathological on gfx1250 — the measured cost barely depends on M, which is the signature of a bad kernel pick rather than real work:

shape	M=2	M=64	M=256	M=2048	M=8192
N=64, K=4096 torch	30.4 us	30.4 us	30.5 us	36.4 us	87.5 us
N=256, K=4096 torch	179.9 us	178.6 us	179.8 us	182.0 us	187.4 us

An N=256 GEMM costs ~178 us whether it is 2 rows or 8192 rows.

Modifications

• aiter/configs/bf16_untuned_gemm.csv: +330 rows — the shape list handed to the tuner (5 N values x 66 M values).
• aiter/configs/bf16_tuned_gemm.csv: +120 tuned rows (gfx1250, cu_num=96, libtype=triton).

Rows are appended, so the existing gfx950 block keeps its current order and the diff is purely additive (450 insertions, 0 deletions). Lookup is by dict key, so file position carries no meaning.

Two notes for reviewers:

• The pre-existing gfx1250 rows in model_configs/dsv4_bf16_tuned_gemm.csv are cu_num=256 with us/tflops/bw all zero, so they never match a 96-CU gfx1250 part. The new rows are cu_num=96 and do not collide with them — python3 -m unittest op_tests.tuning_tests.test_config_shape_collision passes (12/12), including the bf16_tuned_gemm family merge.
• On gfx1250 triton_gemm() ignores solidx/kernelName and lets aiter.ops.triton.gemm.basic.gemm_a16w16 choose (it selects the gluon backend on this arch), so those columns are carried as 0/auto.

Accuracy

Every one of the 120 shapes was checked against F.linear on a call outside the timing loop: all 120 are bitwise identical (torch.equal is True; worst relative error 0.00e+00). This is a dispatch change between two implementations of the same bf16 GEMM, so no accuracy movement is expected and none was observed.

Model-level accuracy (GSM8K) has not been run.

Benchmarking

Unit benchmark on gfx1250 (96 CU), all 120 tuned shapes. Both arms go through the real tgemm.mm dispatcher and differ only by whether this PR's rows are in the config file (before → torch, after → triton).

Metric is GPU kernel time from aiter.test_common.run_perftest (torch profiler device time, arguments rotated to defeat L2 reuse).

Overall (120 shapes)

metric	geomean	min	max	regressions
GPU kernel time	6.82x	2.07x	41.92x	0 / 120

regime	shapes	speedup
M<=256 (decode-ish)	100	7.34x
M>=1024 (prefill-ish)	20	4.71x

N	speedup
64	5.70x
256	23.99x
512	5.31x
1024	4.73x
2048	4.28x

Representative shapes

N	M	torch us	triton us	speedup
64	1	30.78	4.06	7.58x
64	64	30.40	4.16	7.31x
64	8192	87.46	30.57	2.86x
256	8	178.70	4.26	41.92x
256	256	179.83	5.56	32.37x
256	8192	187.42	90.51	2.07x
512	8192	506.49	128.39	3.95x
1024	2048	261.77	49.65	5.27x
1024	8192	1004.12	176.47	5.69x
2048	2048	502.82	75.29	6.68x
2048	8192	2014.63	270.46	7.45x

Checklist

• Config-only change; no kernel or dispatch-logic edits
• op_tests/tuning_tests/test_config_shape_collision passes (12/12)
• Numerics verified bitwise-identical to F.linear on all 120 shapes
• No shape collides with the canonical file or any model_configs/*bf16_tuned_gemm*.csv
• End-to-end serving benchmark

Related

The same box also needs #4791 (fix(tuned_gemm): gate skinny GEMM default to archs with a real kernel) for shapes that are still untuned on gfx1250 — without it the skinny fallback selects an assert(false) stub. That fix is independent of this PR and does not overlap.

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:gfx1250-ffm-triton	Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x	Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4804 --add-label <label>