How can we integrate the DeepGEMM Fp8 GEMM implementation in TE's block-wise scaling？

[BolongLin]

Hi, how can we set other Cute/Cutlass operators on TE? Like GEMM from DeepGEMM, which is a library designed for FP8 GEMMs with fine-grained scaling, as proposed in DeepSeek-V3.

[yaox12]

We will not add DeepGEMM into TE because it lacks the GEMM for wgrad (1x128 by 1x128) in back propagation. And its JIT mechanism brings non-negligible overheads in training.

We're landing DeepSeek-v3-like FP8 recipe (1x128 for activations and 128x128 for weights) in TE and we will use the block-wise GEMM from cuBLAS (to be released in CUDA 12.9), which has a comparable performance as DeepGEMM and both 1D2D (1x128 by 128x128) and 1D1D (1x128 by 1x128) support, and gets rid of the JIT overheads.

[RuiWang1998]

Hi Xin,

Could we expect an ETA on this?

[yaox12]

Hi Xin,

Could we expect an ETA on this?

See PR #1559. Note you will need CUDA 12.9 to run it because the groupwise/blockwise FP8 GEMM is shipped with CUDA 12.9 (ETA early April).

[zigzagcai]

Hi Xin,
Could we expect an ETA on this?

See PR #1559. Note you will need CUDA 12.9 to run it because the groupwise/blockwise FP8 GEMM is shipped with CUDA 12.9 (ETA early April).

Could you share a preview release of CUDA 12.9? We just want to try block-wise FP8 GEMM functionality.

[zobeideThePlayer]

Hi Xin,
Could we expect an ETA on this?

See PR #1559. Note you will need CUDA 12.9 to run it because the groupwise/blockwise FP8 GEMM is shipped with CUDA 12.9 (ETA early April).

Hi,

Checking at this PR[#1559 ], it seems that it only support linear module, but not grouped linear yet? Are there any plan to land the fine-grained scaling for grouped linear too?

[yaox12]

Hi Xin,
Could we expect an ETA on this?

See PR #1559. Note you will need CUDA 12.9 to run it because the groupwise/blockwise FP8 GEMM is shipped with CUDA 12.9 (ETA early April).

Hi,

Checking at this PR[#1559 ], it seems that it only support linear module, but not grouped linear yet? Are there any plan to land the fine-grained scaling for grouped linear too?

Yes, it's included in #1525.

[thefacetakt]

(ETA early April).

Hi @yaox12, is there by any chance an updated ETA on cuda 12.9?

[yaox12]

(ETA early April).

Hi @yaox12, is there by any chance an updated ETA on cuda 12.9?

The new ETA is some time next week. But you know, it's not under my control 😢

[thefacetakt]

(ETA early April).

Hi @yaox12, is there by any chance an updated ETA on cuda 12.9?

The new ETA is some time next week. But you know, it's not under my control 😢

Thanks!

[yaox12]

CUDA 12.9 has been released on May 1st. https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cublas-release-12-9

We have added support for new scaling modes on Hopper (sm_90), including outer vector (per-channel/per-row), per-128-element, and per-128x128-block. Note that there is currently limited support for fused epilogues.

Now you can try FP8 training with the blockwise recipe by

with fp8_autocast(fp8_recipe=recipe.Float8BlockScaling()):
    model()

[zigzagcai]

CUDA 12.9 has been released on May 1st. https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cublas-release-12-9

We have added support for new scaling modes on Hopper (sm_90), including outer vector (per-channel/per-row), per-128-element, and per-128x128-block. Note that there is currently limited support for fused epilogues.

Now you can try FP8 training with the blockwise recipe by

with fp8_autocast(fp8_recipe=recipe.Float8BlockScaling()):
model()

Wow~
So great!!!

[yjzhong89]

CUDA 12.9 has been released on May 1st. https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cublas-release-12-9

We have added support for new scaling modes on Hopper (sm_90), including outer vector (per-channel/per-row), per-128-element, and per-128x128-block. Note that there is currently limited support for fused epilogues.

Now you can try FP8 training with the blockwise recipe by

with fp8_autocast(fp8_recipe=recipe.Float8BlockScaling()):
model()

@yaox12 so this feature still cannot be used on L40s machine, and only support on Hopper (sm_90) or later?

[sbhavani]

Resolved by the TE Float8BlockScaling path rather than integrating DeepGEMM. Support landed in #1559, with GroupedLinear support in #1525; it uses cuBLAS blockwise FP8 GEMM and requires CUDA/cuBLAS >= 12.9 on SM90+ GPUs. L40S/SM89 is therefore not supported for this recipe; on Blackwell, prefer MXFP8BlockScaling or NVFP4BlockScaling. Closing as resolved.