Skip to content

[Attention] Expose paged MQA SplitKV override - #4775

Draft
AMD-yanfeiwang wants to merge 2 commits into
ROCm:mainfrom
AMD-yanfeiwang:amd/expose-paged-mqa-split-kv
Draft

[Attention] Expose paged MQA SplitKV override#4775
AMD-yanfeiwang wants to merge 2 commits into
ROCm:mainfrom
AMD-yanfeiwang:amd/expose-paged-mqa-split-kv

Conversation

@AMD-yanfeiwang

Copy link
Copy Markdown
Contributor

Summary

  • expose an optional SplitKV override on deepgemm_fp8_paged_mqa_logits
  • preserve the existing launch heuristic exactly when the override is omitted
  • validate explicit values and reject ambiguous use with VarCtxSchedule
  • add focused regression coverage for validation, legacy resolution, and explicit pass-through

Motivation

The paged MQA kernel already accepts a runtime split count, but the public wrapper always derives it from TotalCuCount, TileQCount, and WavePerEU. This prevents callers from tuning the split independently for production shapes.

On MI355X (gfx950), a diagnostic three-process sweep for the DSV4 shape rows=6144, width=33280, ChunkK=256 found SplitKV=2 consistently faster than the current automatic value of 10:

Run Auto 10 Explicit 2 Speedup
1 2582.667 us 2546.246 us 1.01430x
2 2590.807 us 2558.686 us 1.01255x
3 2582.267 us 2547.466 us 1.01366x

Aggregate producer gain was 1.382%; all tested split values passed bitwise correctness probes. This PR exposes the mechanism only. It intentionally does not hard-code a shape policy from a single diagnostic shape.

Compatibility

SplitKV=None executes the exact previous formula, so existing callers and launch behavior are unchanged. The underlying Triton/Gluon kernel already accepts a dynamic split count; no kernel ABI or cache-key change is needed.

Validation

  • 13 passed in test_pa_mqa_logits_split_kv.py
  • public-wrapper MI355X check: zero bitwise mismatches for explicit SplitKV=2
  • Ruff, py_compile, and git diff --check pass

@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4775 --add-label <label>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant