[PyTorch][torch.compile] Differentiable fp8_output and fp8_grad for Linear - #47
Open
pggPL wants to merge 2 commits into
Open
[PyTorch][torch.compile] Differentiable fp8_output and fp8_grad for Linear#47pggPL wants to merge 2 commits into
pggPL wants to merge 2 commits into
Conversation
…inear Quantized op outputs carry a grad-handle slot; _QuantizedOutputFn rebuilds the wrapper so autograd reaches the op through the handle. Quantized grads are flattened across the backward op boundary and rebuilt from the bwd fake specs. QuantizedTensor gets the AOTAutograd tangent coercion hooks, FP8 dequantize becomes a traceable custom op, and QuantizedTensor dispatch re-dispatches instead of disabling torch dispatch. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Declare inp as TensorOrQuantized so a Float8Tensor input is flattened by the wrapper op like the other quantized operands, and drop the quantized-input fallback. This lets Linear(fp8_output=True) feed another Linear inside one graph; the consumer's dgrad (plain or quantized) reaches the producer through the grad handle. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
te.Linearundertorch.compilecurrently falls back to eager forfp8_output=Truewith gradients enabled and forfp8_grad=Truewhen the input needs a gradient (compile_unsupported_reason: "differentiable fp8_output=True", "a quantized input grad (fp8_grad=True)"). This PR makes both work on the compiled custom-op path.Why it did not work:
Float8Tensorwrapper was rebuilt outside the op as a leaf, so autograd had no path from the wrapper back to the op.fp8_grad=Trueconsumer is E5M2 with a different quantizer, which fails with "guessed its metadata incorrectly" unless the subclass implements the coercion hooks.Tensor[]slot; this worked by accident in default mode and broke CUDA graphs (buffers not tracked as outputs).Float8Tensor.dequantize()called a rawtex.dequantizebinding, so it could not be traced inside a graph (including the backward of any torch op receiving a Float8 tangent).QuantizedTensor.__torch_dispatch__re-dispatched throughsuper().__torch_dispatch__, which disables Python dispatch and bypasses the innerFakeTensor/FunctionalTensorunder AOTAutograd (a real CUDA kernel was launched on a fake tensor: illegal memory access).Type of change
Changes
dynamo/custom_op.py: a quantized user output carries one extra "grad handle" slot (stride-0 tensor of the output's logical shape and dtype)._QuantizedOutputFnrebuilds the wrapper from the inner buffers and routes the wrapper's gradient (plain or quantized) to the handle;_slice_user_gradsreads the handle's grad. Quantized grads returned by the backward op are flattened to inner buffers and rebuilt in_autograd_backwardfrom the bwd fake specs (_unpack_bwd_result).quantized_tensor.py:__coerce_tangent_metadata__(tangent template fromQuantizer.tangent_quantizer(): rowwise-only, E5M2 for Float8 quantizers) and__coerce_same_metadata_as_tangent__(same type/dtype: fix usage and pass through; different FP8 dtype: requantize with the expected quantizer; expected plain tensor: dequantize).__torch_dispatch__re-dispatches withfunc(*args, **kwargs)after dequantizing.float8_tensor_storage.py: FP8 dequantize goes through atorch.librarycustom op with a fake kernel, so it traces undertorch.compile.module/linear.py:LinearBwdArgs.grad_outputandLinearFwdArgs.inpareTensorOrQuantized, so a quantized dgrad and aFloat8Tensorinput cross the op boundary through the wrapper op's subclass flattening; thefp8_output/fp8_grad/ quantized-input compile gates are removed.test_torch_compile.py):Linear(fp8_output=True)feeding a secondLinearinside one graph (consumer dgrad in bf16 or FP8), an externally quantized input;fp8_gradwith the dgrad consumed in-graph (TE Linear or torch op) and crossing the graph boundary; differentiablefp8_outputwith in-graphdequantize(), with an E5M2 tangent, with an E4M3 tangent (requantization), and a plain output receiving a Float8 tangent. All bit-exact against eager except the requantization case.Known PyTorch limitations, pinned by tests (the first one also means a graph break right after a differentiable
fp8_output=TrueLinear turns theFloat8Tensorinto a graph output and fails in backward unless the consumer produces a quantized grad; onmainthat configuration ran eagerly):.dequantize()outside the graph) raises AOTAutograd's tangent-metadata error; there is no coercion hook for plain -> subclass.requires_subclass_dispatch); otherwise it is lifted as a fake constant (xfail).Checklist: