Add Ascend Cache IO aggregation copy path - #1038
Open
NaganooMei wants to merge 1 commit into
Open
Conversation
NaganooMei
requested review from
FangRun2,
HaoLi980405,
Infinite666,
Tarrei,
Zbm1996,
harrisonyhq,
mag1c-h,
wuhuxiao,
ygwpz,
yxkyong and
zbb200819
as code owners
June 18, 2026 10:04
ygwpz
reviewed
Jun 19, 2026
ygwpz
reviewed
Jun 19, 2026
NaganooMei
force-pushed
the
io-aggregation-upstream-pr-v2
branch
from
June 20, 2026 08:28
5aafeb1 to
7382b4e
Compare
NaganooMei
force-pushed
the
io-aggregation-upstream-pr-v2
branch
13 times, most recently
from
June 27, 2026 01:28
ca7a54c to
d184a64
Compare
NaganooMei
force-pushed
the
io-aggregation-upstream-pr-v2
branch
2 times, most recently
from
July 8, 2026 02:59
c59591c to
5982518
Compare
NaganooMei
force-pushed
the
io-aggregation-upstream-pr-v2
branch
from
July 8, 2026 03:40
5982518 to
a577e5c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This PR adds an Ascend CacheStore IO aggregation path for small H2D/D2H shard copies.
The original CacheStore transfer path submits one async copy per tensor fragment. For workloads where one CacheStore shard contains many tensor fragments, this creates many small H2D/D2H submissions and increases CPU/runtime scheduling overhead.
This PR introduces an Ascend-only aggregation stream. It treats one CacheStore shard as one IO object, uses a staging buffer on device, and launches FFTS SDMA descriptors to scatter/gather between the staging buffer and device tensor addresses.
For
PLATFORM=ascend/RUNTIME_ENVIRONMENT=ascendbuilds, CacheStore enables IO aggregation by default. Whenuse_layerwise=true, the vLLM connector disables CacheStore IO aggregation to preserve the existing layerwise path behavior. Non-Ascend runtimes do not compile or allow this feature.Modifications
IoAggregationStreamConfigMakeIoAggregationStream(config)CopyStreamas the CacheStore-side stream owner, but let IO aggregation use one outer stream object with internal lanes.LoadQueueandDumpQueuecan submit one CacheStore shard as a scatter/gather object instead of looping over tensor fragments.AscendIoAggregationStreamandAscendShardIOAggregator.cache_stream_number.FftsSdmaDispatcherto build and launch FFTS SDMA copy descriptors.UCM_RUNTIME_ASCEND_IO_AGGREGATION.cache_io_aggregation=true.use_gdr.Config Behavior
cache_io_aggregationdefaults to enabled in Ascend builds.cache_io_aggregationis forced to disabled by the vLLM connector whenuse_layerwise=true.cache_io_aggregation_pipeline_depthdefaults to2.cache_io_aggregation_max_ready_lanesdefaults to8.Test