Skip to content

[recipe] feat: Add DeepSeek-V4 training smoke configs#3923

Open
Meirtz wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
Meirtz:lmei/dsv4-train-smoke-configs
Open

[recipe] feat: Add DeepSeek-V4 training smoke configs#3923
Meirtz wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
Meirtz:lmei/dsv4-train-smoke-configs

Conversation

@Meirtz
Copy link
Copy Markdown

@Meirtz Meirtz commented May 21, 2026

What does this PR do ?

Adds minimal DeepSeek-V4 training smoke configs for exercising selected Megatron Core DSv4 architecture paths through Megatron Bridge. These configs are intended for short validation runs, not full production pretraining recipes.

Changelog

  • Add deepseek_v4_tiny_pretrain_config for a 1-layer DSv4 hybrid-attention smoke path.
  • Add deepseek_v4_flash_proxy_pretrain_config for DSv4 hybrid attention with mHC and hash-routed MoE.
  • Add deepseek_v4_flash_mtp_proxy_pretrain_config for DSv4 hybrid attention with mHC and MTP.
  • Export the new recipe helpers and add unit and functional smoke coverage.
  • Pin the Megatron Core dev submodule to a commit that contains the required DSv4 primitives.

GitHub Actions CI

See the CI section in the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Local validation:

  • uv run --no-sync --with pre-commit pre-commit run --all-files
  • Focused GB200 recipe smoke validation for the DSv4 tiny, mHC/hash-MoE proxy, and mHC/MTP proxy paths.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation? Not needed for these smoke-only recipe configs.
  • Does the PR affect components that are optional to install? No.

Additional Information

This PR intentionally limits the scope to smoke configs and tests. It does not claim full DeepSeek-V4 training support.

The recipe configs currently depend on the Megatron Core dev pin because the required DSv4 primitives are not yet fully available in the Megatron Core main pin.

Signed-off-by: Lingrui Mei <lmei@nvidia.com>
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented May 21, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Meirtz Meirtz marked this pull request as ready for review May 22, 2026 04:07
@yaoyu-33 yaoyu-33 added area:recipe Training recipes and launch configs feature New capabilities, enhancements, or enablement work full-test-suite needs-review PR is ready for code review and waiting on a reviewer labels May 22, 2026
@Meirtz Meirtz changed the title [recipe, test] feat: Add DeepSeek V4 training smoke configs [recipe] feat: Add DeepSeek-V4 training smoke configs May 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:recipe Training recipes and launch configs community-request feature New capabilities, enhancements, or enablement work full-test-suite needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants