Skip to content

[PyTorch] [torch.compile] torch.compile support for LayerNormLinear and LayerNormMLP - #43

Open
pggPL wants to merge 1 commit into
layernorm_compile_prepfrom
layernorm_compile
Open

[PyTorch] [torch.compile] torch.compile support for LayerNormLinear and LayerNormMLP#43
pggPL wants to merge 1 commit into
layernorm_compile_prepfrom
layernorm_compile

Conversation

@pggPL

@pggPL pggPL commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Description

Register LayerNormLinear and LayerNormMLP as torch.compile custom ops on top of the framework merged in NVIDIA#3053, so both trace under fullgraph=True with the FP8 / MXFP8 / NVFP4 recipes. Part 3 of 3: sits on #44 (framework + module groundwork), which sits on #42 (module restructuring). This PR is only the compile wiring itself.

Type of change

  • New feature (non-breaking change which adds functionality)

Changes

layernorm_linear.py / layernorm_mlp.py (mirrors Linear)

  • compile_unsupported_reason gate on the fwd args.
  • Data-free _*_forward_fake / _*_backward_fake on TensorSpec, mirroring the impls' saved-tensor layout and alias dedup.
  • register_custom_op(...), @no_torch_dynamo eager wrapper (_*_eager), compiled dispatch + _compile_eager_fallback_reason / _forward_eager_fallback in Module.forward; @no_torch_dynamo comes off Module.forward.
  • Eager fallbacks (graph break under fullgraph=True): debug, quantized input, fsdp_group, differentiable fp8_output, CPU offload, delayed wgrad, FP8 weight caching (is_first_microbatch), fuse_wgrad_accumulation, non-opaque quantizers; MLP additionally checkpoint=True and gemm_gelu_fusion under FP8.

Tests (tests/pytorch/test_torch_compile.py): recipes x compile modes (default, reduce-overhead) for both modules, norm / activation / return_layernorm_output / zero_centered_gamma variants, FP8 primary weight, no-bias and frozen-weight slots, dynamic shapes, eager fallbacks.

Testing (RTX Ada, single GPU)

  • tests/pytorch/test_torch_compile.py: 160 passed, 46 skipped, 1 xpassed (43 new LN tests)
  • Eager regression on this tree: test_sanity.py -k "layernorm_linear or layernorm_mlp" 7344 passed, test_numerics.py -k "layernorm_linear or layernorm_mlp" 738 passed, test_backward_override.py 1316 passed, test_cuda_graphs.py -k "layernorm or mlp" 105 passed
  • pylint on the touched files: 10/10

Not covered on a single GPU: TP/SP and Userbuffers under compile.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

…nd LayerNormMLP

Register the LayerNormLinear and LayerNormMLP forward/backward as
torch.library custom ops on top of the framework from NVIDIA#3053, so both trace
under fullgraph compile with the FP8/MXFP8/NVFP4 recipes. Builds on the
module restructuring and the framework groundwork in the two preceding PRs.

layernorm_linear.py / layernorm_mlp.py:
- compile_unsupported_reason gate, data-free forward/backward fakes on
  TensorSpec, register_custom_op, @no_torch_dynamo eager wrapper, compiled
  dispatch and eager fallback in Module.forward, mirroring Linear.
- Eager fallbacks: debug, quantized input, fsdp_group, differentiable
  fp8_output, CPU offload, delayed wgrad, FP8 weight caching,
  fuse_wgrad_accumulation, non-opaque quantizers; MLP also activation
  checkpointing and gemm_gelu_fusion under FP8.

tests/pytorch/test_torch_compile.py: LN module coverage (recipes x compile
modes, norm/activation/return_layernorm_output variants, FP8 primary weight,
frozen weights / no bias, dynamic shapes, eager fallbacks).

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
@pggPL
pggPL force-pushed the layernorm_compile branch from 9422845 to 4cfffaf Compare September 3, 2026 12:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant