Add NVFP4_E5M3 support and update README - #2565
Merged
Merged
Conversation
- Implement NVFP4_E5M3 dense linear and MoE methods backed by xkernels. - Introduce nvfp4_e5m3_qdq for quant-dequant operations. - Update pyproject.toml to include xkernels as a dependency. - Enhance README with installation instructions and usage examples. - Add tests for NVFP4_E5M3 configuration and functionality. Signed-off-by: Xin He <xin3.he@intel.com>
…ntegration - Implemented batched NVFP4_E5M3 MoE execution using Triton kernels for improved performance. - Updated README.md to reflect changes in MoE processing and added new usage instructions. - Refactored test_nvfp4_ue5m3_model.py to remove eager execution enforcement for better flexibility. - Introduced xkernels_ops.py for custom operator registration and optimized GEMM operations. - Added nvfp4_e5m3.py to handle quantization and dequantization of activations with new custom ops. - Created test_fused_moe_ue5m3.py to validate fused MoE functionality and performance against reference implementations. - Enhanced test_nvfp4_ue5m3.py to include fullgraph capture tests for QDQ and GEMM operations. Signed-off-by: Xin He <xin3.he@intel.com>
…n and corresponding tests Signed-off-by: Xin He <xin3.he@intel.com>
xin3he
force-pushed
the
xinhe/nvfp4_e5m3
branch
from
September 3, 2026 08:17
e8c957b to
6cdc66f
Compare
- Updated `pyproject.toml` to remove xkernels dependency and add nvidia-cutlass-dsl. - Modified `inc_nvfp4_ue5m3_linear.py` to implement weight dequantization using CuTe. - Refactored `inc_nvfp4_ue5m3_moe.py` and `inc_nvfp4_ue5m3_scheme.py` to reflect changes in implementation. - Removed `xkernels_ops.py` as it is no longer needed. - Enhanced `envs.py` to introduce a new environment variable for weight dequantization mode. - Implemented CuTe custom operators for NVFP4 E5M3 in `cute_kernels.py` and `nvfp4_e5m3_cute.py`. - Updated tests in `test_envs.py` and `test_nvfp4_ue5m3.py` to validate new functionality and configurations. - Ensured backward compatibility with reference implementations for quantization and dequantization. Signed-off-by: Xin He <xin3.he@intel.com>
…h compile factors Signed-off-by: Xin He <xin3.he@intel.com>
…ocumentation Signed-off-by: Xin He <xin3.he@intel.com>
xin3he
force-pushed
the
xinhe/nvfp4_e5m3
branch
from
September 5, 2026 12:40
22fdd8a to
d65c12e
Compare
…and comprehensive tests Signed-off-by: Xin He <xin3.he@intel.com>
xin3he
force-pushed
the
xinhe/nvfp4_e5m3
branch
from
September 7, 2026 05:21
648dd81 to
71bdc9b
Compare
for more information, see https://pre-commit.ci
…re and clarity Signed-off-by: Xin He <xin3.he@intel.com>
… data context Signed-off-by: Xin He <xin3.he@intel.com>
yiliu30
reviewed
Sep 7, 2026
xin3he
changed the title
…atic CuTe backend selection Signed-off-by: Xin He <xin3.he@intel.com>
for more information, see https://pre-commit.ci
yiliu30
approved these changes
Sep 8, 2026
…ror checks for missing CUTLASS DSL Signed-off-by: Xin He <xin3.he@intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request significantly expands the documentation and test coverage for the vLLM QDQ plugin, with a particular focus on NVFP4_E5M3 (AutoRound) support and the Sage3 Triton diffusion attention backend. It improves the main README for clarity and completeness, adds a new test script for NVFP4_E5M3 models, updates dependency requirements, and enhances the NVFP4 hardware backend documentation with format details, usage, runtime behavior, and validation status.
Documentation and Feature Expansion:
benchmark/vllm-qdq-plugin/README.md: Completely reorganized and expanded to cover both the QDQ plugin and the Sage3 Triton attention backend, including detailed quick start instructions, supported formats (now including NVFP4_E5M3), runtime controls, performance/accuracy notes, implementation details, and licensing. [1] [2] [3] [4]benchmark/vllm-qdq-plugin/src/nvfp4_hw/README.md: Greatly expanded with a thorough explanation of native NVFP4 and NVFP4_E5M3 support, runtime selection logic, backend requirements, limitations, validation status, and example commands. [1] [2] [3]Testing and Usability Improvements:
benchmark/vllm-qdq-plugin/scripts/test_nvfp4_ue5m3_model.py: Added a new script to load an AutoRound NVFP4_E5M3 model and run a test prompt, making it easier to verify model support and plugin integration.Dependency and Packaging Updates:
benchmark/vllm-qdq-plugin/pyproject.toml: Addednvidia-cutlass-dsl>=4.6.0as a required dependency to ensure CuTe backend support for QDQ, improving out-of-the-box compatibility.These changes collectively improve the plugin’s usability, documentation quality, and support for modern quantization formats, making it easier for users to get started and for developers to extend or validate the plugin.
Type of Change
feature
Description
detail description
Expected Behavior & Potential Risk
the expected behavior that triggered by this PR
How has this PR been tested?
how to reproduce the test (including hardware information)
Dependency Change?
any library dependency introduced or removed