Skip to content

fix(deps): add explicit cufile dependency #1552

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gforsyth wants to merge 1 commit into from
Closed

fix(deps): add explicit cufile dependency #1552

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions dependencies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ files:
- test_python_common
- test_python_cuopt
- depends_on_rapids_logger
- test_cufile
py_build_cuopt_server:
output: pyproject
pyproject_dir: python/cuopt_server
Expand Down Expand Up @@ -736,6 +737,10 @@ dependencies:


cuda_wheels:
common:
- output_types: [requirements, pyproject]
packages:
- cuda-pathfinder
specific:
# cuOpt needs 'nvJitLink>={whatever-cuopt-was-built-against}' at runtime, and mixing
# old-CTK with new-nvJitLink is supported, so maintain 1 entry here per CUDA major.minor
Expand All @@ -754,14 +759,14 @@ dependencies:
cuda: "12.*"
use_cuda_wheels: "true"
packages:
- cuda-toolkit[cublas,cudart,curand,cusolver,cusparse,nvtx]==12.*
- cuda-toolkit[cublas,cudart,cufile,curand,cusolver,cusparse,nvtx]==12.*
- nvidia-cudss-cu12>=0.7,<0.8
- nvidia-nvjitlink-cu12>=12.9,<13
- matrix:
cuda: "13.*"
use_cuda_wheels: "true"
packages:
- &ctk_cu13 cuda-toolkit[cublas,cudart,curand,cusolver,cusparse,nvtx]==13.*
- &ctk_cu13 cuda-toolkit[cublas,cudart,cufile,curand,cusolver,cusparse,nvtx]==13.*
- &cudss_cu13 nvidia-cudss-cu13>=0.7,<0.8
- &nvjitlink_cu13 nvidia-nvjitlink>=13.0,<14
# if no matching matrix selectors passed, list the CUDA 13 requirement
Expand Down Expand Up @@ -848,3 +853,17 @@ dependencies:
- matrix:
packages:
- python>=3.11,<3.15
test_cufile:
specific:
# The `cufile` extra was added to the `cuda-toolkit` metapackage in 12.6.3
# Adding an explicit test dependency on the cufile package here for
# testing on versions before 12.6.3
- output_types: pyproject
matrices:
- matrix:
cuda: "12.*"
use_cuda_wheels: "true"
packages:
- nvidia-cufile-cu12
- matrix:
packages:
8 changes: 7 additions & 1 deletion python/libcuopt/libcuopt/load.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0


Expand Down Expand Up @@ -41,10 +41,16 @@ def load_library():
import libraft
import librmm
import rapids_logger
from cuda.pathfinder import load_nvidia_dynamic_lib

rapids_logger.load_library()
librmm.load_library()
libraft.load_library()
# We manually load librt.so.1 here to workaround a bad dynamic link
# in nvidia_cufile-1.15.1.6-py3-none-manylinux_2_27_aarch64.whl
# https://nvbugspro.nvidia.com/bug/6425783
ctypes.CDLL("librt.so.1", mode=ctypes.RTLD_GLOBAL)
load_nvidia_dynamic_lib("cufile")
Comment on lines +49 to +53

@coderabbitai coderabbitai Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Unscoped, unguarded librt.so.1 preload.

The comment explains this workaround is for nvidia_cufile-1.15.1.6-py3-none-manylinux_2_27_aarch64.whl, but ctypes.CDLL("librt.so.1", mode=ctypes.RTLD_GLOBAL) runs unconditionally on every architecture, and its OSError is not caught (only ModuleNotFoundError is handled by the surrounding except). If librt.so.1 is absent on some system, this raises out of load_library() uncaught, whereas before this change library loading degraded gracefully via the various fallbacks (system installation / wheel installation) elsewhere in this function.

Consider gating this preload to aarch64 (matching the referenced bug) and wrapping it in its own try/except OSError so a missing librt.so.1 doesn't hard-fail library loading.

🛠️ Proposed fix to scope and guard the workaround 
-        # We manually load librt.so.1 here to workaround a bad dynamic link
-        # in nvidia_cufile-1.15.1.6-py3-none-manylinux_2_27_aarch64.whl
-        # https://nvbugspro.nvidia.com/bug/6425783
-        ctypes.CDLL("librt.so.1", mode=ctypes.RTLD_GLOBAL)
-        load_nvidia_dynamic_lib("cufile")
+        if platform.machine() in ("aarch64", "arm64"):
+            # We manually load librt.so.1 here to workaround a bad dynamic link
+            # in nvidia_cufile-1.15.1.6-py3-none-manylinux_2_27_aarch64.whl
+            # https://nvbugspro.nvidia.com/bug/6425783
+            try:
+                ctypes.CDLL("librt.so.1", mode=ctypes.RTLD_GLOBAL)
+            except OSError:
+                pass
+        load_nvidia_dynamic_lib("cufile")

(requires import platform at the top of the file)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/libcuopt/libcuopt/load.py` around lines 49 - 53, Scope the librt.so.1
preload workaround in load_library() to aarch64 only, using platform
architecture detection, and wrap ctypes.CDLL("librt.so.1",
mode=ctypes.RTLD_GLOBAL) in a dedicated try/except OSError. Allow loading to
continue through existing fallbacks when the preload is unavailable, while
retaining the subsequent load_nvidia_dynamic_lib("cufile") behavior.

except ModuleNotFoundError:
Comment on lines 41 to 54

@coderabbitai coderabbitai Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Description: check if cuda-pathfinder is referenced anywhere outside dependencies.yaml (e.g. conda recipes)
rg -n "cuda-pathfinder|cuda_pathfinder" --type=yaml -g '!dependencies.yaml'
fd -a meta.yaml -x cat {}

Repository: NVIDIA/cuopt

Length of output: 150

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== load.py ==\n'
sed -n '1,140p' python/libcuopt/libcuopt/load.py

printf '\n== dependencies.yaml matches ==\n'
rg -n -C 3 "cuda-pathfinder|cuda_pathfinder|requirements|pyproject|conda" dependencies.yaml

printf '\n== any packaging files mentioning cuda-pathfinder ==\n'
rg -n "cuda-pathfinder|cuda_pathfinder" -g '!dependencies.yaml' .

Repository: NVIDIA/cuopt

Length of output: 21481

Move the cuda.pathfinder import out of the core load block. In python/libcuopt/libcuopt/load.py:41-54, a missing cuda-pathfinder turns the whole try into a no-op, so rapids_logger, librmm, and libraft never load on conda installs. Split the cufile-specific import/workaround into its own guarded block after the core loads.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/libcuopt/libcuopt/load.py` around lines 41 - 54, Separate the
cuda.pathfinder import and cufile-specific loading from the main core-library
load block in the module’s load routine. Keep rapids_logger, librmm, and libraft
imports and load_library calls in the existing guarded block, then add a
subsequent guarded block for load_nvidia_dynamic_lib, the librt workaround, and
cufile loading so a missing cuda-pathfinder does not prevent core libraries from
loading.

pass

Expand Down
3 changes: 2 additions & 1 deletion python/libcuopt/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@ classifiers = [
"Programming Language :: Python :: 3.14",
]
dependencies = [
"cuda-toolkit[cublas,cudart,curand,cusolver,cusparse,nvtx]==13.*",
"cuda-pathfinder",
"cuda-toolkit[cublas,cudart,cufile,curand,cusolver,cusparse,nvtx]==13.*",
"librmm==26.8.*,>=0.0.0a0",
"nvidia-cudss-cu13>=0.7,<0.8",
"nvidia-nvjitlink>=13.0,<14",
Expand Down
Loading