Skip to content

nvpci: do not warn when a GPU has no vfio-dev directory, and fix simpleLogger formatting - #129

Open
fjvbn2003 wants to merge 1 commit into
NVIDIA:mainfrom
fjvbn2003:nvpci-vfio-dev-warning
Open

fjvbn2003 wants to merge 1 commit into
NVIDIA:mainfrom
fjvbn2003:nvpci-vfio-dev-warning

Conversation

@fjvbn2003

Copy link
Copy Markdown

Problem

GPU Feature Discovery (k8s-device-plugin v0.19.3, deployed by GPU Operator v26.3.3) prints this line every time it relabels a node, on every node, for every GPU that is bound to the nvidia driver:

2026/10/17 14:14:25 WARNING: unable to detect IOMMU FD for [0000:01:00.0 open /sys/bus/pci/devices/0000:01:00.0/vfio-dev: no such file or directory]: %!v(MISSING)

There are two problems in pkg/nvpci:

  1. getIOMMUFD() returns the ReadDir error when <device>/vfio-dev does not exist, and getGPUByPciBusID() logs it as a warning. The kernel only creates vfio-dev for a device that is bound to a VFIO driver on a kernel with VFIO device cdev support. For a GPU that is bound to nvidia, which is the common case, a missing vfio-dev directory is expected and there is no IOMMU FD to report.
  2. simpleLogger.Warningf() passes the variadic slice as one argument (log.Printf("WARNING: "+format, v)), so every warning that goes through the default logger is garbled: the arguments are printed as one [...] list and the last verb becomes %!v(MISSING).

Measured on five single-GPU nodes (RTX 4070 Laptop x2, RTX 4060 Laptop, RTX 3060 Laptop, RTX 3090) with GFD's default 60 s sleep interval, over the last 24 hours:

Node GFD log lines unable to detect IOMMU FD %!v(MISSING) Creating Labels
A 5752 1438 1438 1438
B 5752 1438 1438 1438
C 5752 1438 1438 1438
D 5744 1436 1436 1436

One line in four of the GFD log is this warning.

Changes

  • getIOMMUFD() returns "", nil when vfio-dev does not exist. Other errors, and an existing vfio-dev directory without a vfio* entry, are still reported as before.
  • simpleLogger.Warningf() forwards the arguments with v....
  • Tests: TestNvpciIOMMUFDNotBoundToVFIO (no vfio-dev → empty IommuFD, no warning), TestNvpciIOMMUFDEmptyVFIODev (the remaining warning is still logged, with the expected text), and TestSimpleLoggerWarningf.

Testing

$ go test ./pkg/nvpci/ -run 'TestSimpleLogger|TestNvpciIOMMUFD' -v
--- PASS: TestSimpleLoggerWarningf
--- PASS: TestNvpciIOMMUFD
--- PASS: TestNvpciIOMMUFDNotBoundToVFIO
--- PASS: TestNvpciIOMMUFDEmptyVFIODev
$ go test ./...
ok  ... (all packages)

Without the change, TestSimpleLoggerWarningf fails with the same [...]: %!v(MISSING) output as above, and TestNvpciIOMMUFDNotBoundToVFIO fails because the warning is logged.

I did not run golangci-lint locally.

GFD gets the fix with its next go-nvlib bump.

I used an AI assistant (Claude) to help investigate and write this change; I reviewed and tested it myself.

🤖 Generated with Claude Code

…leLogger formatting

A GPU that is not bound to vfio has no vfio-dev directory, which is the
normal case, yet nvpci logged a warning for every such device on every
scan. The warning was also garbled ("%!v(MISSING)") because
simpleLogger passed its variadic arguments as a single slice. Treat a
missing vfio-dev directory as "no IOMMU FD" without a warning, and
forward the arguments with v... .

Signed-off-by: YoungjuKim <fjvbn2003@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant