Repository navigation
[executorch][cuda] Check FQN weight coverage without AOTI's per-buffer warnings - #24052
Merged
Merged
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24052
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
|
|
Gasoonjia
force-pushed
the
gasoonjia/cuda-quiet-offgraph-kv-warnings
branch
from
October 21, 2026 17:25
c4a4471 to
64ca607
Compare
shoumikhin
approved these changes
Oct 22, 2026
Gasoonjia
marked this pull request as ready for review
October 22, 2026 07:28
Gasoonjia
force-pushed
the
gasoonjia/cuda-quiet-offgraph-kv-warnings
branch
from
October 22, 2026 17:55
64ca607 to
14d0ef0
Compare
This PR needs a
|
Gasoonjia
force-pushed
the
gasoonjia/cuda-quiet-offgraph-kv-warnings
branch
from
October 22, 2026 21:07
14d0ef0 to
9ffe236
Compare
Base automatically changed from
gasoonjia/fix-dflash-final-block-test
to
main
October 22, 2026 21:10
…r warnings Every CUDA method load that binds FQN weights next to an off-graph KV cache printed one AOTI line per KV buffer, e.g. [WARNING] Found constant or module state buffer or empty module state parameter __et_offgraph_kv_layer_0_k in model, but not provided by user! The muse-glimmer batching CI log had 1672 of these lines out of 11650. CudaWeightCache::load binds weights with validate_full_update=true while it deliberately leaves the off-graph KV buffers unbound; the KV pool binds them later. AOTI's full update allows absent buffers but writes this warning to stderr for each one, on every load. When the AOTI library exports GetConstantType and GetConstantFromFolded, do the same completeness check ExecuTorch-side: folded constants, buffers, tensor constants and empty parameters may be absent, every other constant must be bound, else InvalidProgram. Then bind with validate_full_update=false. validate_full_update only gates that check in AOTI, so binding is otherwise unchanged. Libraries without the two APIs keep validate_full_update=true. Test plan: - g++ -fsyntax-only on cuda_weight_cache.cpp and cuda_backend.cpp against the CUDA 12.6 headers. - CI: CUDA unittest and muse-glimmer batching jobs pass, and their logs no longer contain the __et_offgraph_kv_ warnings.
Gasoonjia
force-pushed
the
gasoonjia/cuda-quiet-offgraph-kv-warnings
branch
from
October 22, 2026 21:10
9ffe236 to
d6d412b
Compare
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #24050.
Every CUDA method load that binds FQN weights next to an off-graph KV cache printed one AOTI warning line per KV buffer:
The
unittest-cuda-muse-glimmer-batchinglog contains 1672 of these lines, out of 11650 lines total.Why it happens.
CudaWeightCache::loadbinds the weights withvalidate_full_update=trueand intentionally leaves the off-graph KV buffers unbound, because the KV pool binds them later. AOTI's full-update check (assert_all_constants) allows absent buffers, but it writes a warning to stderr for each absent buffer on every load.Fix.
AOTInductorModelContainerGetConstantTypeandAOTInductorModelContainerGetConstantFromFolded, ExecuTorch runs the same completeness rule itself. Folded constants, buffers, tensor constants, and empty parameters may be absent. Every other constant must be bound, otherwiseloadfails withInvalidProgram.validate_full_update=false. In AOTI that flag only gatesassert_all_constants, so nothing else about binding changes.loadkeepsvalidate_full_update=trueas before.Test plan:
g++ -fsyntax-onlyoncuda_weight_cache.cppandcuda_backend.cppagainst the CUDA 12.6 headers.__et_offgraph_kv_warnings.