Skip to content

fix(podman): discover CDI GPUs from server info - #4329

Draft
elezar wants to merge 1 commit into
mainfrom
codex/podman-cdi-discovery/elezar
Draft

elezar wants to merge 1 commit into
mainfrom
codex/podman-cdi-discovery/elezar

Conversation

@elezar

@elezar elezar commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Summary

Default GPU requests currently infer Podman CDI devices from the gateway host's /dev, which can differ from the Podman server's CDI inventory. Read server info at initialization and before default GPU validation and creation, preserving legacy discovery only when the server omits both CDI fields.

Related Issue

Follow-up to merged PR #4306. This PR now targets main.

No issue linked yet. This draft is being opened at the maintainer's explicit request; add a scope-matched issue before marking it ready for review.

Changes

  • Retain host.cdiSpecDirs and source-qualified host.discoveredDevices from the existing /v5.0.0/libpod/info endpoint.
  • Treat CDI field presence as authoritative, including servers that report spec directories but omit an empty device list. Keep server paths diagnostic rather than scanning them on the gateway host.
  • Refresh asynchronously for preflight and creation. Fail info/API decoding errors without selecting stale devices, and retry discovery on later requests.
  • Preserve the NVIDIA default GPU selection policy, WSL all-device fallback, and explicit selector handling.
  • Add mocked API regressions for populated/empty inventories, legacy fallback, inventory edits, and query failures; update operator docs and the debugging skill.

Testing

  • Podman driver unit and mocked API suite after rebasing onto current main: 242 tests passed.
  • Podman Clippy (all targets, warnings denied), formatting, and diff checks passed after rebasing.
  • Documentation navigation validation passed during initial implementation.
  • mise run pre-commit passed during initial implementation (sccache disabled via a temporary local override).
  • Full GPU workload E2E qualification.

A live Fedora tmachine API check on Podman 5.8.1 confirmed the legacy response omits both CDI fields. New-server response shapes came from earlier synthetic-spec verification on Podman 6.1.1; mocked tests exercise those shapes against this implementation. No real GPU execution was tested. An earlier mise run ci attempt stopped at the unrelated prover link step because this host lacks libz3. CDI syntax validation is already on main through merged PR #4306; this PR's diff contains only discovery, tests, and documentation. Unrelated testing-strategy commits are excluded.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Relevant driver and operator documentation updated

@copy-pr-bot

copy-pr-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown

@elezar
elezar force-pushed the codex/podman-cdi-discovery/elezar branch from 95e8bcb to 2870595 Compare October 8, 2026 11:40
@elezar
elezar changed the base branch from main to fix/2-podman-device-validation/alangou October 8, 2026 11:40
Base automatically changed from fix/2-podman-device-validation/alangou to main October 8, 2026 12:39
Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
elezar force-pushed the codex/podman-cdi-discovery/elezar branch from 2870595 to c521ef0 Compare October 9, 2026 09:43

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant