Skip to content

fix(zero): warn when process group silently falls back to single-rank#8117

Closed
NovusEdge wants to merge 1 commit into
deepspeedai:masterfrom
NovusEdge:fix-silent-noshard-warning
Closed

fix(zero): warn when process group silently falls back to single-rank#8117
NovusEdge wants to merge 1 commit into
deepspeedai:masterfrom
NovusEdge:fix-silent-noshard-warning

Conversation

@NovusEdge

Copy link
Copy Markdown
Contributor

Summary

Adds a warning when zero.Init detects that WORLD_SIZE env var suggests multi-GPU but the actual process group has world_size=1.

This catches the sneaky case where torch.distributed wasnt initialized before model construction, so each rank loads the full model instead of sharding -> boom, OOM :D

The problem

When using deepspeed --num_gpus 8 but calling from_pretrained() before deepspeed.init_distributed(), zero.Init silently falls back to single-rank mode. No warning, just an OOM that looks like "model too big" when it actually fits fine if sharded properly.

The fix

Check if WORLD_SIZE > 1 but dp_world_size == 1 after process group setup, and warn loudly with a clear explanation of whats happening and how to fix it.

Closes #8084

When WORLD_SIZE env var indicates multi-GPU but the process group has
world_size=1, emit a warning explaining the likely cause and fix.

This catches the silent failure where zero.Init sees world_size=1 because
torch.distributed wasn't initialized before model construction, leading to
each rank loading the full model and OOM.

Closes deepspeedai#8084

Signed-off-by: NovusEdge <novusedge0@gmail.com>
@tohtana

tohtana commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Hi @NovusEdge,

Thank you for submitting this PR!
However, I found #8089 seems to supersede this PR. As #8089 is submitted earlier, let me close this PR.

I also see that the predicate in the current approach env_world_size > 1 and self.dp_world_size == 1 is too broad.
self.dp_world_size is computed from self.ds_process_group, and that group can be an explicit data_parallel_group or sequence_data_parallel_group, not necessarily the world group. A multi-rank job can intentionally use a singleton data-parallel group when another parallel dimension is enabled.
I think #8089 can handle this properly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

zero.Init silently does not shard (world_size=1) when the process group is uninitialized before from_pretrained -> per-rank full load -> OOM

2 participants