rccl --mca pml option fix when openmpi with ucx is used for rccl-test

[ahskabir]

Motivation

CVS's determine_mpi_pml_config() function fails to add --mca pml ucx to the mpirun command when mpi_pml is set to "ucx" in the config. Instead, it only passes -x UCX_TLS=auto (an invalid transport name) and -x UCX_NET_DEVICES= (empty). Without explicit PML=ucx, MPI falls back to a non-UCX transport that causes GPU runlist oversubscription and 2-3x bandwidth degradation.

Root Cause
In cvs/lib/rccl_lib.py, the determine_mpi_pml_config() function:
When mpi_pml="ucx": sets pml_param = "" (empty) instead of "--mca pml ucx"
Passes -x UCX_TLS=auto which is not a valid UCX transport name
Passes -x UCX_NET_DEVICES= (empty) which tells UCX to use no devices

The function only ever outputs --mca pml ob1 or empty string — the code path for --mca pml ucx does not exist.
According to OpenMPI 5.0 documentation, UCX PML should auto-select when InfiniBand/RoCE is detected — --mca pml ucx should not be required. However, CVS's invalid UCX environment variables (-x UCX_TLS=auto and -x UCX_NET_DEVICES=) cause UCX to fail initialization before OpenMPI can auto-detect it. With UCX broken, OpenMPI falls back to a non-UCX transport, which causes the GPU queue oversubscription and bandwidth degradation.

The OMPI_MCA_pml=ucx workaround forces UCX PML selection regardless of auto-detection, and UCX_NET_DEVICES=all fixes the broken device config so UCX can actually initialize. Both are needed because CVS actively passes bad values that override what would otherwise work automatically.

Technical Details

Following params are added in rccl_config.json file :
"ucx_tls": "rc,self,sm,tcp",
"_comment_ucx_tls": "When user requested UCX either leave this parameter value blank for auto assignment or assign value tcp",
"net_dev_list": "",
"_comment_net_dev_list": "Leave empty for auto-detection from backend NICs, or set explicitly e.g. ens26np0,ens27np0"

After the fix, in the rccl_config.json file user can provide mpi_pml: ucx and it correctly passes the args to mpirun

Test Plan

Testing was done by Ahsan Kabir and Ryan Lukasik separately, Ryan being the reported. I tested this bugfix first and tried many different combination of providing different type of pml like auto, ob1, ucx etc. For ucx, I built openmpi with ucx support. All of these combinations have been tried for for rccl_perf test. Same logic was added to rccl_regression.

Test Result

Testing summary is described in the comments section of https://amd-hub.atlassian.net/browse/AIMVT-248 and https://amd.atlassian.net/browse/DCCS-6464

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[speriaswamy-amd]

The core fix is working. I tested this commit on two Ruby nodes with 16 ranks and exercised explicit ucx, explicit ob1, auto selecting UCX, a controlled auto fallback to ob1, and the duplicate rccl_regression path. All paths completed with zero #wrong values; the focused 16 MiB all-reduce runs were approximately 156–157 GB/s. I also smoke-tested all 12 available rccl-tests collectives. Ruff, the 18 existing unit tests, CI, and the docs build pass.

I am requesting one small robustness change before merge: explicit ob1 currently performs UCX NIC discovery before the PML decision. Ruby has a complete mapping, so its hardware run passes, but empty and partial mapping probes fail before ob1 can launch. Please skip discovery for explicit ob1 and make discovery failures descriptive. I have also left a should-fix comment for the checked-in configuration conflict.

[speriaswamy-amd]

Please change this to "bnxt_re0:1,bnxt_re1:1"

[ahskabir]

fixed.