Skip to content

Fix tag-triggered CI by installing the cuda-python metapackage with --no-deps#2334

Merged
leofang merged 1 commit into
NVIDIA:mainfrom
Andy-Jost:ajost/fix-tag-ci-metapackage-no-deps
Jul 9, 2026
Merged

Fix tag-triggered CI by installing the cuda-python metapackage with --no-deps#2334
leofang merged 1 commit into
NVIDIA:mainfrom
Andy-Jost:ajost/fix-tag-ci-metapackage-no-deps

Conversation

@Andy-Jost

Copy link
Copy Markdown
Contributor

Summary

Tag-triggered CI (e.g. cuda-core-v1.1.0) fails in the docs build and in the wheel-test "Ensure cuda-python installable" step, even though the same commit passes on main. This installs the cuda-python metapackage wheel with --no-deps in those jobs, since the subpackages are already installed from CI artifacts.

Problem

The cuda-python metapackage pins its subpackages (currently cuda-core~=1.0.0 in cuda_python/setup.py). CI builds and installs cuda-pathfinder, cuda-bindings, and cuda-core from freshly built artifacts, then installs the cuda-python wheel on top. Because that last install is a plain pip install, pip re-resolves the pinned deps against PyPI. On main the built cuda-core version is a dev version that still satisfies ~=1.0.0, so nothing is pulled and CI is green. On a release tag the built version is 1.1.0, which does not satisfy ~=1.0.0, so pip goes to PyPI. This surfaces as two distinct tag-only failures:

  1. Docs build: pip pulls an old cuda-core (e.g. 1.0.1) from PyPI over the freshly built 1.1.0 artifact. The docs then build against the stale package and fail on modules that only exist in 1.1.0.
  2. New Python versions: when a tag introduces a new interpreter (e.g. 3.15/3.15t), no matching cuda-core exists on PyPI yet, so the re-resolve fails outright with cuda-core~=1.0.0 (from versions: none) — after the actual cuda.core tests have already passed.

Fix

Add --no-deps to the metapackage wheel install in the docs build and the Linux/Windows "Ensure cuda-python installable" steps. The subpackages are already present from artifacts, so dependency resolution here only re-fetches from PyPI and masks the correct just-built wheels. --no-deps installs the metapackage against what's already there and keeps these jobs pinned to the artifacts under test.

This intentionally skips verifying the metapackage's PyPI dependency resolution in these jobs. That check is misleading during a subpackage tag release anyway (the pins don't yet match the not-yet-published version); it belongs in a dedicated metapackage-release install test against PyPI.

Changes

  • .github/workflows/build-docs.yml: pip install --no-deps cuda_python*.whl
  • .github/workflows/test-wheel-linux.yml: --no-deps on both branches of "Ensure cuda-python installable"
  • .github/workflows/test-wheel-windows.yml: same

Test plan

  • Re-run tag CI for cuda-core-v1.1.0 after merge; docs build and "Ensure cuda-python installable" should pass.

During cuda-core tag releases, pip install cuda_python*.whl re-resolves
cuda-core~=1.0.0 from PyPI and fails or downgrades the artifact wheel.
Subpackages are already installed from CI artifacts, so skip dependency
resolution in docs build and wheel test installability checks.
@Andy-Jost Andy-Jost added this to the cuda.core v1.1.0 milestone Jul 9, 2026
@Andy-Jost Andy-Jost added bug Something isn't working P0 High priority - Must do! CI/CD CI/CD infrastructure labels Jul 9, 2026
@Andy-Jost Andy-Jost self-assigned this Jul 9, 2026
@Andy-Jost Andy-Jost added bug Something isn't working P0 High priority - Must do! CI/CD CI/CD infrastructure labels Jul 9, 2026
@Andy-Jost Andy-Jost requested review from leofang and mdboom and removed request for mdboom July 9, 2026 21:01
@github-actions

This comment has been minimized.

@leofang

leofang commented Jul 9, 2026

Copy link
Copy Markdown
Member

Let me just admin-merge.

Discussed in the meeting, this is considered a bandaid to unblock the release. What happened in the first attempt was that

  • tagged cuda-core v1.1.0
  • cuda-python still pins at v1.0.0
  • the dependency test (disabled in this PR) is not happy with the local cuda-core v1.1.0 generated in the tag-triggered workflow
  • pip tries to get cuda-core v1.0.0 from PyPI
  • we don't have public builds for Python 3.15/3.15t on PyPI, so these pipelines failed

This PR effectively nukes the dependency test (as if the whole job step is removed), which is fine as a bandaid. Long-term, we probably need to consider moving cuda-python dependency test to the release workflow (though I am not a fan of it).

@leofang leofang merged commit c23833b into NVIDIA:main Jul 9, 2026
121 checks passed
@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Doc Preview CI
Preview removed because the pull request was closed or merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working CI/CD CI/CD infrastructure P0 High priority - Must do!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants