[PyTorch] [torch.compile] Prepare LayerNormLinear and LayerNormMLP for torch.compile - #42
Open
pggPL wants to merge 1 commit into
Open
[PyTorch] [torch.compile] Prepare LayerNormLinear and LayerNormMLP for torch.compile#42pggPL wants to merge 1 commit into
pggPL wants to merge 1 commit into
Conversation
…r torch.compile Restructure the two modules the same way Linear was restructured in NVIDIA#2967, so the follow-up custom-op registration is a pure wiring change: - LayerNormLinearFwdArgs / LayerNormLinearBwdArgs and LayerNormMLPFwdArgs / LayerNormMLPBwdArgs dataclasses replace the positional non_tensor_args tuple and the ad-hoc ctx.<attr> plumbing. - The autograd Functions shrink to thin wrappers around module-level _layernorm_{linear,mlp}_forward_impl / _setup_ctx / _backward_impl that take one dataclass argument. Saved tensors that alias forward inputs or outputs are deduplicated by name and rebuilt in setup_ctx. - LayerNormMLP keeps activation checkpointing: the forward saves only its inputs, and _layernorm_mlp_recompute re-runs the forward impl in backward with recompute_for_bwd set, filling the backward args from the recomputed state. - The weight workspaces move from positional autograd inputs into the args bag and are returned as extra outputs; backward-only recipe knobs (split accumulators, backward_override, debug UB flags, delayed wgrad_store) are resolved in Module.forward. - check_fp8_reduce_and_update moves to module/_common.py and is shared with Linear. No functional change. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Restructure
LayerNormLinearandLayerNormMLPthe same wayLinearwas restructured in NVIDIA#2967, so that the follow-up PR registering them astorch.compilecustom ops (on top of the framework merged in NVIDIA#3053) is a pure wiring change.No functional change intended. Part 1 of 3 (then #44 groundwork, #43 compile wiring).
Type of change
Changes
LayerNormLinearFwdArgs/LayerNormLinearBwdArgsandLayerNormMLPFwdArgs/LayerNormMLPBwdArgsdataclasses replace the positionalnon_tensor_argstuple and the ad-hocctx.<attr>plumbing._layernorm_{linear,mlp}_forward_impl/_setup_ctx/_backward_implthat take one dataclass argument (the shape the custom-op framework expects). Saved tensors that alias forward inputs or outputs are deduplicated by name (saved_tensor_aliases) and rebuilt insetup_ctx.LayerNormMLPkeeps activation checkpointing: the forward saves only its inputs, and_layernorm_mlp_recomputere-runs the forward impl in backward withrecompute_for_bwdset, filling the backward args from the recomputed state.Linear). Backward-only recipe knobs (dgrad/wgrad_use_split_accumulator,backward_override, debug UB flags, delayedwgrad_store) are resolved inModule.forward.check_fp8_reduce_and_updatemoves tomodule/_common.pyand is shared withLinear.Testing (RTX Ada, single GPU)
tests/pytorch/test_sanity.py -k "layernorm_linear or layernorm_mlp": 7344 passedtests/pytorch/test_numerics.py -k "layernorm_linear or layernorm_mlp": 738 passedtests/pytorch/test_backward_override.py: 1316 passedtests/pytorch/test_cuda_graphs.py -k "layernorm or mlp": 105 passedtest_gpt_full_activation_recompute(checkpoint recompute path): same pass/fail pattern asmain(known Ada flakiness)Checklist: