Skip to content

net: compute recv sameDevice against self, not the PXN proxyRank - #2260

Open
EylonKrause wants to merge 1 commit into
NVIDIA:masterfrom
EylonKrause:fix/net-recv-samedevice
Open

net: compute recv sameDevice against self, not the PXN proxyRank#2260
EylonKrause wants to merge 1 commit into
NVIDIA:masterfrom
EylonKrause:fix/net-recv-samedevice

Conversation

@EylonKrause

Copy link
Copy Markdown

Description

recvSetup connects its proxy to the local rank — ncclProxyConnect(comm, TRANSPORT_NET, 0, myInfo->rank, &recv->proxyConn) — and the adjacent comment notes "We don't support PXN on receive yet", so the recv proxy always runs on this rank (self). But req.sameDevice is computed from proxyRank:

req.sameDevice = (comm->peerInfo[proxyRank].cudaDev == comm->cudaDev);

When graph != NULL, ncclTopoGetNetDev sets proxyRank to a PXN intermediate rank, which can be a different rank on a different GPU. So for the receiver sameDevice may be computed against the wrong device and come out false. The recv-side gate if (ncclGdrCopy && map->sameProcess && resources->sameDevice) then skips the GDRCopy SYNC/flush mapping, defeating the optimization on PXN-capable topologies even though the recv proxy is always on the kernel's own device. (The send path legitimately connects its proxy to proxyRank, so comparing against proxyRank is correct there.)

Related Issues

None.

Changes & Impact

  • src/transport/net.cc (recvSetup): compute sameDevice against self (comm->peerInfo[myInfo->rank].cudaDev), matching where the recv proxy actually runs. The send path is unchanged. Behavior-preserving on non-PXN / single-NIC setups where proxyRank == myInfo->rank already; on PXN topologies it stops spuriously disabling recv-side GDRCopy. Non-breaking.

Performance Impact

Re-enables recv-side GDRCopy SYNC/flush on PXN-capable multi-GPU nodes where it was being spuriously disabled — a latency improvement on that path. No change elsewhere.

Testing

  • Builds clean with make src.build; all_reduce_perf regression: Out of bounds values : 0 OK.
  • Verified by inspection: the recv proxy is bound to myInfo->rank (and recvProxyConnect rejects a remote recv proxy), so self is the correct comparison; proxyRank can differ only via the graph/PXN path. The misgating manifests on multi-GPU / multi-NIC PXN nodes, which cannot be reproduced on a single-GPU machine.

The recv proxy always runs on this rank (recvSetup connects it to
myInfo->rank; PXN on receive is unsupported), but sameDevice was computed
from peerInfo[proxyRank], which under a graph/PXN topology can be a
different intermediate rank on a different GPU. That spuriously yields
sameDevice=false and disables recv-side GDRCopy SYNC/flush. The send path
legitimately uses proxyRank and is left unchanged.

Signed-off-by: EylonKrause <eylon1909@gmail.com>
@EylonKrause

Copy link
Copy Markdown
Author

Disclosure: this contribution was authored with an AI coding assistant (Claude) and reviewed before submission.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant