Repository navigation
Conversation
Summary:
Two passes ask "is this int8 on device?" by reading the producer's fake
tensor. That read is premature in `explicit_layout_pass_list`: the validator
runs before `AtenToCortexMPass`, so a LUT activation (tanh, sigmoid) or
softmax is still an `aten.*` node with an fp32 fake tensor even though
`FoldAndAnnotateQParamsPass` has already given it int8 qparams.
- `ValidateCortexMExplicitLayoutPass` aborts lowering behind one ("layout
copy ... must move an int8 tensor"). Explicit layout is the only mode that
supports Conv1d, so `Conv1d(tanh(x))` could not lower at all.
- `QuantizedClampActivationPass` leaves float bounds on a clamp behind a LUT
activation -- a run-time dtype error on the legacy pass list, wrong
numerics on explicit layout.
Add `lowered_output_dtype` to `cortex_m/utils.py` (the dtype of a node's
first output on device: folded `output_qparams` when present, else the fake
tensor) and use it in both passes. The validator's error message now names
the input node and its dtype.
A clamp the clamp pass declines (bounds that are not compile-time scalars,
per-channel output qparams) still carries int8 qparams and is accepted,
matching the legacy pass list.
Differential Revision: D126708891
Summary: The substitution helpers make the same premature fake-tensor read the layout validator did in the parent diff. `_get_permute_replacement`, `_get_minimum_replacement` and `_get_maximum_replacement` choose their kernel from the input fake tensor, but `AtenToDialectPass` gives each new `cortex_m` node the replaced node's meta dict, so an input already rewritten from `aten.tanh` still reads fp32 until the retrace at the end of `AtenToCortexMPass`. Read `lowered_output_dtype` instead, as the validator and the clamp pass now do. Rewritten nodes keep their folded `output_qparams` and report int8; a `cortex_m` dequantize has none and still reports float. The cost was coverage, not numerics. `Conv1d(tanh(x))` lowered its input-side layout move to a portable `aten::permute_copy`, and `minimum(tanh(x), y)` / `maximum(tanh(x), y)` stayed on portable kernels. That breaks the explicit-layout guarantee that every layout move is a CMSIS transpose, so a selective build that registers only the expected kernels fails to load. Differential Revision: D127046193
Summary: `FoldAndAnnotateQParamsPass` annotates the operators that compute. The ones that only rearrange -- views, squeezes, transposes and the layout copies the explicit-layout pipeline inserts around every convolution -- are left with no quantization parameters of their own, and the layout copies are created with empty metadata entirely. So `lowered_output_dtype` falls through to a fake tensor that is still the floating-point one the pre-quantization trace produced, and reports float32 for a tensor that is int8 on device. Two consumers read that helper and both are misled. The permute substitution in `AtenToCortexMPass` declines anything it cannot prove is int8, so a permute sitting behind a reshape stays on a portable kernel. And `ValidateCortexMExplicitLayoutPass` would reject a layout copy reached through one, which is the same blind spot arriving as a hard error rather than a silent one. Rearranging a tensor does not change its element type, so a plumbing node's quantization parameters are its producer's. `PropagateQParamsPass` annotates them once, in the graph, rather than at each of the places that ask. Nodes are visited in topological order, so a chain of plumbing resolves in a single sweep: each node reads a producer the same sweep has already annotated. It runs after `CortexMCanonicalizeViewCopyPermutePass`, so the plumbing it annotates is the plumbing that survives, and before `ValidateCortexMExplicitLayoutPass`, the first consumer. Later than `FoldAndAnnotateQParamsPass` on purpose: `ActivationFusionPass` both reads and writes `output_qparams` on its input nodes, and must not see these annotations. Differential Revision: D127331247
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24140
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 32d6f00 with merge base d07c72b ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
Contributor
|
@pssrawat has exported this pull request. If you are a Meta employee, you can view the originating Diff in D127331247. |
Collaborator
|
Hi, could you modify the passes which inserts movement ops without qparams to copy the qparams of their original node instead of propagating qparams in a separate pass? That is a more predictable design to me. Also, this highlights another reason for why these ops should substitute on the fake tensor dtype rather than qparams, we create an implicit condition on the meta-data which all passes must follow is really easy to miss. Trying to fix that automatically in this pass is more of a band-aid solution to me. If you change the implementation of the previous PR his change might not be necessary at all. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
FoldAndAnnotateQParamsPassannotates the operators that compute. The onesthat only rearrange -- views, squeezes, transposes and the layout copies the
explicit-layout pipeline inserts around every convolution -- are left with no
quantization parameters of their own, and the layout copies are created with
empty metadata entirely. So
lowered_output_dtypefalls through to a faketensor that is still the floating-point one the pre-quantization trace
produced, and reports float32 for a tensor that is int8 on device.
Two consumers read that helper and both are misled. The permute substitution
in
AtenToCortexMPassdeclines anything it cannot prove is int8, so a permutesitting behind a reshape stays on a portable kernel. And
ValidateCortexMExplicitLayoutPasswould reject a layout copy reached throughone, which is the same blind spot arriving as a hard error rather than a
silent one.
Rearranging a tensor does not change its element type, so a plumbing node's
quantization parameters are its producer's.
PropagateQParamsPassannotatesthem once, in the graph, rather than at each of the places that ask. Nodes are
visited in topological order, so a chain of plumbing resolves in a single
sweep: each node reads a producer the same sweep has already annotated.
It runs after
CortexMCanonicalizeViewCopyPermutePass, so the plumbing itannotates is the plumbing that survives, and before
ValidateCortexMExplicitLayoutPass, the first consumer. Later thanFoldAndAnnotateQParamsPasson purpose:ActivationFusionPassboth reads andwrites
output_qparamson its input nodes, and must not see theseannotations.
Differential Revision: D127331247