Skip to content

arm PR103.0: CUDA GDN cache-cpy fusion (arm spec + implementation PR spec) - #112

Open
ddvnguyen wants to merge 1 commit into
baselinefrom
fork/pr103-gdn-fusion-arm
Open

arm PR103.0: CUDA GDN cache-cpy fusion (arm spec + implementation PR spec)#112
ddvnguyen wants to merge 1 commit into
baselinefrom
fork/pr103-gdn-fusion-arm

Conversation

@ddvnguyen

Copy link
Copy Markdown
Owner

Arm spec for porting the Metal GGML_METAL_FUSE_GDN_CACHE fusion (upstream d011a21) to CUDA: GDN kernel writes recurrent-state snapshots directly into the state/cache buffer, trailing cpy elided. Qwen3.8-27B is GDN-heavy (48 of 64 layers) — 48 copies + launch overhead removed per token.

What this PR contains

Docs only (docs/arms/pr103-gdn-cache-cpy-fusion.md): implementation PR spec (kernel epilogue write, fusion-table entry, GGML_CUDA_FUSE_GDN_CACHE env toggle for single-binary A/B), edge cases (other consumers of GDN output, chunked prefill, MTP/nextn draft layer), A/B test matrix, bars, correctness gates. The fusion implementation lands as a separate code PR after review.

Bars (vs arm102/PR105.0 reference)

  • A1 (kv_unified off, V q5_1, fusion on): >= 41.5 t/s (+3.5% over 40.1); 40.5-41.4 still shippable if correctness gates pass
  • Prefill regression <= 2%; n=2 agg within 49.2-52.6 band; MTP acceptance unchanged
  • Hard gates: greedy OFF-vs-ON byte-identical; run-to-run byte-identical cold/warm

Test plan

  • Implementation code PR with fusion + toggle (human-reviewed)
  • A/B matrix A1/A2/B1/B2, >= 5 loops each
  • CUDA-events profile: cpy elision evidence across 48 layers/token
  • Correctness gates pass
  • Results into arm report

Sequencing: after PR105.0 (reference topology); PR104.x builds on this base.

AI usage disclosure: YES — arm spec drafted by AI (opencode) from the Metal fusion commit semantics and local source verification; implementation code to be human-authored/owned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant