Skip to content

Add generic tensor copy and architecture-aware Gated Delta Rule support#4757

Open
lzhangzz wants to merge 7 commits into
InternLM:mainfrom
lzhangzz:gdr-1b
Open

Add generic tensor copy and architecture-aware Gated Delta Rule support#4757
lzhangzz wants to merge 7 commits into
InternLM:mainfrom
lzhangzz:gdr-1b

Conversation

@lzhangzz

Copy link
Copy Markdown
Collaborator

Summary

This change introduces a reusable, layout-aware CUDA tensor copy path and replaces the legacy Gated Delta Net implementation with a planned, registered Gated Delta Rule kernel stack.

Key Changes

  • Add generic strided tensor copy with vectorized and tiled transpose paths.
  • Preserve DLPack byte offsets, strides, and backing-storage capacity.
  • Add recurrent and chunked Gated Delta Rule planning and dispatch.
  • Add architecture-specific kernels for pre-SM90, SM90, and SM120 GPUs.
  • Support mixed decode and prefill batches, grouped recurrent state, multiple state dtypes, and context-parallel planning.
  • Integrate the new planner and kernels into GatedDeltaNetLayer.
  • Expose native copy and Gated Delta Rule planning/execution through Python bindings.
  • Add reference, correctness, and benchmark tooling for TurboMind, FLA, and FlashQLA comparisons.
  • Remove the legacy tensor copy source and old GDN benchmark.

Validation

  • Completed a Qwen3.5-27B TurboMind smoke test with TP=1 and CP=1.
  • Added coverage for recurrent and chunked execution, strided layouts, grouped state, state dtypes, finished sequences, and context-parallel behavior.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

lzhangzz added 4 commits July 16, 2026 13:25
Drop direct cute/algorithm/copy.hpp and cute/atom/copy_atom.hpp includes;
they are already pulled in via cute/tensor.hpp, and SortIncludes places them
first which triggers incomplete Copy_Atom during prefetch.hpp parse.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants