Repository navigation
Conversation
…ches Unbind unbound every other PCI function of the card and cleared its driver_override, while BindToDriver only bound the device itself. Callers that switch a GPU with Unbind + BindToDriver, such as the DRA driver for NVIDIA GPUs, left the HDMI audio and USB functions without a driver after every switch, and a host that keeps them on vfio-pci for passthrough lost that binding on each VM start and stop. BindToDriver now also binds the other functions when the target is a vfio driver, reusing the loop from BindToVFIODriver, and Unbind only unbinds the device it is given. Signed-off-by: swibrow <15628653+swibrow@users.noreply.github.com>
2 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to kubernetes-sigs/dra-driver-nvidia-gpu#1567, where the maintainers asked for the fix to go here.
Problem
Unbindunbinds the device and every other PCI function of the same card (HDMI audio, and since #115 also the VirtualLink USB functions), and clears theirdriver_override.BindToDriverthen binds only the device itself. So a caller that switches a GPU withUnbind+BindToDriver, as the DRA driver does, leaves the other functions without a driver after every switch, in both directions:vfio-pci, but its audio function is left unbound instead of moving to vfio with it.vfio-pcifor passthrough (for example viavfio_pci ids=, so that KubeVirt can pass it to a VM as a PCI host device) loses that binding on every VM start and stop.Fix
BindToDriverbinds the other functions of the card as well when the target is a vfio driver (vfio-pcior a variant), reusing the loop fromBindToVFIODriver(nowbindAuxDevices).Unbindonly unbinds the device it is given. The bind side already moves the other functions to vfio, and no API binds them back to a host driver, so unbinding them only left them driverless.With this,
Unbind+BindToDriver(vfio)moves the whole card to vfio, andUnbind+BindToDriver(nvidia)returns only the GPU and leaves the other functions where they are. The DRA driver needs no code change, only the dependency bump.The
Unbindchange is a behaviour change for callers that expect it to free the whole card. The only importer I could find is the DRA driver (k8s-driver-manager carries its own copy ininternal/nvpassthrough). If you would rather keepUnbindas it is, an alternative is a separate method that unbinds only the given function, which the DRA driver would then call.Tests
TestIsVFIODrivercovers the vfio driver check, including variant drivers and both_and-spellings.make assert-fmt vet check-vendor,go test ./pkg/nvpassthrough/andgolangci-lint run ./pkg/nvpassthrough/...pass.gpu-kubelet-plugintests pass with this branch via areplacedirective and no other change.vfio-pciviavfio_pci ids=), KubeVirt v1.9.0, running the DRA driver built against this branch with no DRA driver code change. Stopping the VM returned the GPU tonvidiaand left the audio function onvfio-pci; starting it again moved the GPU tovfio-pci, and the VM came up with the GPU and the audio function (as a KubeVirt PCI host device). With v0.12.0 and v0.13.0 the audio function is left without a driver after each switch. The vfio direction ofBindToDriverdid not have to move the audio function in that run, since it was already onvfio-pci; that path is the same loopBindToVFIODriveruses.