Skip to content

nvpassthrough: keep a card's other functions bound across driver switches - #130

Open
swibrow wants to merge 1 commit into
NVIDIA:mainfrom
swibrow:nvpassthrough-keep-aux-bound
Open

swibrow wants to merge 1 commit into
NVIDIA:mainfrom
swibrow:nvpassthrough-keep-aux-bound

Conversation

@swibrow

@swibrow swibrow commented Oct 21, 2026

Copy link
Copy Markdown

Follow-up to kubernetes-sigs/dra-driver-nvidia-gpu#1567, where the maintainers asked for the fix to go here.

Problem

Unbind unbinds the device and every other PCI function of the same card (HDMI audio, and since #115 also the VirtualLink USB functions), and clears their driver_override. BindToDriver then binds only the device itself. So a caller that switches a GPU with Unbind + BindToDriver, as the DRA driver does, leaves the other functions without a driver after every switch, in both directions:

  • To vfio: the GPU lands on vfio-pci, but its audio function is left unbound instead of moving to vfio with it.
  • Back to the host driver: the audio function is unbound again, and nothing binds it anywhere. A host that keeps it on vfio-pci for passthrough (for example via vfio_pci ids=, so that KubeVirt can pass it to a VM as a PCI host device) loses that binding on every VM start and stop.

Fix

  • BindToDriver binds the other functions of the card as well when the target is a vfio driver (vfio-pci or a variant), reusing the loop from BindToVFIODriver (now bindAuxDevices).
  • Unbind only unbinds the device it is given. The bind side already moves the other functions to vfio, and no API binds them back to a host driver, so unbinding them only left them driverless.

With this, Unbind + BindToDriver(vfio) moves the whole card to vfio, and Unbind + BindToDriver(nvidia) returns only the GPU and leaves the other functions where they are. The DRA driver needs no code change, only the dependency bump.

The Unbind change is a behaviour change for callers that expect it to free the whole card. The only importer I could find is the DRA driver (k8s-driver-manager carries its own copy in internal/nvpassthrough). If you would rather keep Unbind as it is, an alternative is a separate method that unbinds only the given function, which the DRA driver would then call.

Tests

  • TestIsVFIODriver covers the vfio driver check, including variant drivers and both _ and - spellings.
  • make assert-fmt vet check-vendor, go test ./pkg/nvpassthrough/ and golangci-lint run ./pkg/nvpassthrough/... pass.
  • dra-driver-nvidia-gpu main builds and its gpu-kubelet-plugin tests pass with this branch via a replace directive and no other change.
  • Hardware: Talos v1.14.0, Kubernetes v1.36.1, NVIDIA 595.91.07, RTX 3090 Ti (audio function on vfio-pci via vfio_pci ids=), KubeVirt v1.9.0, running the DRA driver built against this branch with no DRA driver code change. Stopping the VM returned the GPU to nvidia and left the audio function on vfio-pci; starting it again moved the GPU to vfio-pci, and the VM came up with the GPU and the audio function (as a KubeVirt PCI host device). With v0.12.0 and v0.13.0 the audio function is left without a driver after each switch. The vfio direction of BindToDriver did not have to move the audio function in that run, since it was already on vfio-pci; that path is the same loop BindToVFIODriver uses.

…ches

Unbind unbound every other PCI function of the card and cleared its
driver_override, while BindToDriver only bound the device itself. Callers
that switch a GPU with Unbind + BindToDriver, such as the DRA driver for
NVIDIA GPUs, left the HDMI audio and USB functions without a driver after
every switch, and a host that keeps them on vfio-pci for passthrough lost
that binding on each VM start and stop.

BindToDriver now also binds the other functions when the target is a vfio
driver, reusing the loop from BindToVFIODriver, and Unbind only unbinds
the device it is given.

Signed-off-by: swibrow <15628653+swibrow@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant