feat: activate NVLink fabric partitions for vGPU VFs on NVSwitch systems - #193
Open
lexfrei wants to merge 12 commits into
Open
feat: activate NVLink fabric partitions for vGPU VFs on NVSwitch systems#193lexfrei wants to merge 12 commits into
lexfrei wants to merge 12 commits into
Conversation
lexfrei
marked this pull request as ready for review
July 4, 2026 08:32
This was referenced Jul 4, 2026
How to adapt the Shared NVSwitch Virtualization Model of FM to activate nvlink in multi-gpu VMs
#133
Open
Collaborator
|
Can you replace the fabric bindings with https://github.com/NVIDIA/go-nvfm ? |
Author
|
Done — switched the fabric bindings from the inline cgo ABI to go-nvfm. The plugin now drives libnvfm through go-nvfm, which dlopen's it at runtime and ships the SDK headers, so the hand-maintained ABI transcription and the link-time dependency are both gone. Behaviour is unchanged, and it builds/vets/tests clean under linux + cgo. |
This was referenced Jul 12, 2026
lexfrei
force-pushed
the
feat/fabric-partition-activation
branch
from
July 20, 2026 00:43
495619e to
a8bf9ae
Compare
lexfrei
force-pushed
the
feat/fabric-partition-activation
branch
from
September 1, 2026 10:12
a8bf9ae to
7801737
Compare
…work Ada/Hopper+ GPUs (vGPU 17+) no longer expose vGPU instances through mdev; instead, SR-IOV Virtual Functions stay bound to the "nvidia" host driver and are configured directly on the PCI function via sysfs. The existing vGPU discovery only walks /sys/bus/mdev/devices, so it finds nothing on these GPUs and every vGPU VF ends up indistinguishable from a whole-card passthrough device, since both share the same PCI vendor:device id. Add a second discovery path that walks /sys/bus/pci/devices looking for Nvidia functions bound to the "nvidia" driver, skips Physical Functions (identified by sriov_totalvfs), and treats a Virtual Function as a configured vGPU when its current_vgpu_type is non-zero. The profile name is resolved by matching that type id against the creatable_vgpu_types catalog, merged per physical card (scoped by each VF's physfn parent) since a configured VF's own list is frequently empty or reduced to the active entry, and since vGPU type ids are only unique within a single physical card. Because these VFs are ordinary PCI/IOMMU-group devices, discovered functions are added to the same iommu/BDF maps used by GPU passthrough and dispatched through the existing GenericDevicePlugin, grouped by vGPU profile name instead of PCI device id. No changes were needed to the allocate path or device contract; the mdev-backed vGPU discovery is untouched. Document the new hardware path in the README alongside the existing mdev instructions, since it applies to a different GPU generation and neither replaces the other. Signed-off-by: Aleksei Sviridkin <f@lex.la>
A fully consumed card reduces the creatable_vgpu_types list of every one of its functions down to the header, so the per-card catalog cannot resolve the types of already-configured VFs and the plugin advertised nothing exactly when all capacity was allocated. Fall back to a host-wide catalog merged across all cards; numeric ids that different cards map to different names stay ambiguous and are still skipped rather than guessed. Signed-off-by: Aleksei Sviridkin <f@lex.la>
On a host where every card is fully consumed, no creatable_vgpu_types catalog lists anything at all — sysfs alone cannot resolve configured type ids to profile names no matter how catalogs are merged. Resolve through NVML as the last resort: the supported-vGPU list of a card does not shrink as capacity is allocated. Scoped per physical card like the sysfs catalogs, cached per scan, and requires the host driver library to be loadable in the container. Signed-off-by: Aleksei Sviridkin <f@lex.la>
…other cards When a configured VF's card catalog is reduced and cannot resolve its type id, resolution fell back to a host-wide catalog merged across all cards. Numeric vGPU type ids are only unique within one physical card, so if the target card contributed no entry for the id and another card mapped the same id to a different profile, that entry looked unambiguous and the VF could be advertised under the wrong extended resource. Drop the host-wide catalog entirely. Resolve strictly card-local sysfs, then the parent Physical Function's own NVML supported-type list, then skip the VF with a log. NVML is authoritative per card and its supported list does not shrink as capacity is allocated, so it covers the fully consumed card without ever borrowing a sibling card's mapping. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
…ments The NVML name-resolution fallback needs libnvidia-ml.so.1 and the NVIDIA device nodes. The default DaemonSet mounts only /dev/vfio and runs unprivileged, so on a fully consumed node a restart could resolve no profile names and advertise no VFs. Add a dedicated manifest that provides them via a single-file bind of libnvidia-ml.so.1 plus the NVIDIA device nodes under privileged: true, leaving the default manifest minimal and unprivileged. Document in the README both the minimally privileged runtimeClassName: nvidia path and the hostPath variant, why the library must be bound as a single file rather than the whole host lib directory, and which device nodes the plugin's NVML queries actually require. Also document that vGPU profiles must be configured before the plugin starts, since it scans the Virtual Functions once at startup. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
…ation On NVSwitch systems running SR-IOV vGPU with Fabric Manager in FABRIC_MODE=2, a whole-card guest cannot initialise CUDA until its VF's NVLink fabric partition is activated through the Fabric Manager SDK (cuInit fails with error 802 otherwise). Add a self-contained pkg/fabric wrapping libnvfm (the nv_fm_agent API): - cgo binding (linux && cgo) linking libnvfm.so.1, with a portable stub for other platforms and CGO_ENABLED=0 so the whole module still builds and unit-tests on developer machines. The FM ABI is declared in the binding; NVIDIA's proprietary headers are not vendored. - Connect / GetSupportedPartitions / ActivateWithVFs / Deactivate / Close with sizeof-based struct-version handling and a full fmReturn_t -> error mapping. The connection address type (Unix socket vs host:port TCP) is selected from the address form. - NVML helpers to resolve a physical GPU's FM physicalId (module id) and MIG mode, and a VF -> single-GPU partition resolver keyed strictly on physicalId, never on PCI enumeration order. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
Activate a whole-card vGPU VF's NVLink fabric partition during GenericDevicePlugin.Allocate, before the device is handed to the virt-launcher pod, so the guest's CUDA init succeeds with per-VM NVLink isolation from the first init. For each allocated VF the plugin resolves its single-GPU fabric partition (VF -> parent PF -> FM physicalId -> partition) and activates it for that one VF; fmActivateFabricPartitionWithVFs takes one VF per GPU, so a single-GPU partition is always activated with exactly one VF. MIG-mode GPUs have NVLink disabled and are not in the fabric, so their VFs need no activation and are skipped (nvmlDeviceGetMigMode). The feature is a no-op on non-fabric systems and classic passthrough GPUs. Configuration via environment: FABRIC_PARTITION_ACTIVATION (auto/off), FABRIC_MANAGER_ADDRESS (default the FM TCP command interface), FABRIC_FAIL_MODE (closed: an FM error fails the allocation; open: log and allow), and FABRIC_RECONCILE_INTERVAL. A pod-resources reconciler deactivates a partition once its VF is gone, since the device plugin API has no release callback. Unit-tested with a fake Fabric Manager client. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
The fabric partition activation binding links libnvfm.so.1 (NEEDED in the binary), so both the builder and the runtime image need it. Fetch a pinned libnvfm from the CUDA repository package at build time (only the versioned shared object; the FM ABI is declared in-tree, headers are not required), link against it, and copy it into the distroless runtime image on the loader search path. The version is a build ARG so it can track the target host's Fabric Manager daemon. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
Document the fabric partition activation design: the FM SDK binding, the VF -> single-GPU partition resolver, single-VF activation (one VF per GPU), MIG-skip, the pod-resources reconciler for deactivation, the configuration, the runtime requirements (hostNetwork for the FM TCP command interface, pod-resources mount, FABRIC_MODE=2), and libnvfm image bundling. Includes a draft upstream PR description. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace the hand-written inline cgo Fabric Manager SDK binding with the official github.com/NVIDIA/go-nvfm library. The previous binding declared the nv_fm_agent struct layouts and entry points by hand in the cgo preamble (NVIDIA's headers cannot be redistributed) and linked libnvfm.so.1 at build time. go-nvfm ships the FM SDK headers and dlopen's libnvfm at runtime, so both the hand-maintained ABI transcription and the link-time dependency are gone. Behaviour is unchanged: same Client interface and call sequence, same single-VF partition activation, same physicalId-based resolution and MIG-mode skip, same 5s connect timeout, same TCP/Unix address handling, and the same return-code to sentinel-error mapping the allocation logic branches on. The runtime image still bundles the versioned libnvfm.so.1; go-nvfm is pointed at that SONAME because the unversioned -dev symlink is not shipped in the distroless image. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
The libnvfm build steps were written for x86_64 only, so the arm64 image this repository also publishes was broken in two independent places. The builder stage fetched the Fabric Manager package from the CUDA repository under the directory reported by uname -m. That is correct for x86_64, but the repository names its 64-bit ARM server directory sbsa, so the arm64 fetch requested a path that does not exist. Under set -eux the 404 from wget aborted the stage and the arm64 image never built at all. The repository path and the Debian architecture suffix are now chosen separately, since only one of them follows uname. The runtime stage then copied libnvfm.so.1 into /usr/lib/x86_64-linux-gnu. The loader searches the multiarch directory belonging to the image's own architecture, so on arm64 that destination names a directory that is never searched and go-nvfm's dlopen of libnvfm.so.1 would fail, leaving fabric partition activation silently unavailable. The library now goes to /usr/lib, which this base image's loader carries in its built-in search path on both published architectures, so one destination serves both. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
The design document carried a pull-request draft that restated the sections above it, and a paragraph recording which shape the Fabric Manager binding passed through before the current one. Neither constrains the code as it stands, and both date the document to the moment it was written rather than to what the plugin does. The heading covering the gaps this feature has to fill now names what the plugin does not provide, rather than what a fork does not supply. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
lexfrei
force-pushed
the
feat/fabric-partition-activation
branch
from
September 1, 2026 13:31
7801737 to
54b7d33
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #133.
On NVSwitch systems (HGX H100/H200) running SR-IOV vGPU with Fabric Manager in
FABRIC_MODE=2, a whole-card (NVLink-enabled) guest cannot initialise CUDA —cuInitfails with error802("system not yet initialized") — until its VF's NVLink fabric partition is activated through the Fabric Manager SDK. The device plugin hands the VF to thevirt-launcherpod but nothing activates the fabric partition, so those guests fail CUDA on these systems.This change activates the VF's single-GPU fabric partition during
Allocate, before the guest starts, giving working CUDA from the first init with per-VM NVLink isolation:pkg/fabricwrapslibnvfm(thenv_fm_agentAPI) via cgo, with a portable stub so non-linux /CGO_ENABLED=0builds are unaffected. The FM ABI is declared in the binding (headers not vendored).Allocateresolves the VF → parent PF → FMphysicalId(NVML module id) → single-GPU partition, matched strictly onphysicalId(never PCI order), then activates the partition for exactly that VF (fmActivateFabricPartitionWithVFstakes one VF per GPU).nvmlDeviceGetMigMode). Only whole-card VFs are activated.FABRIC_PARTITION_ACTIVATION(defaultauto), a no-op on non-NVSwitch systems and classic passthrough GPUs.FABRIC_FAIL_MODE(defaultclosed) chooses whether an FM error fails the allocation or is logged and allowed. A pod-resources reconciler deactivates a partition once its VF is gone.libnvfm.so.1; the pod needshostNetwork(to reach the FM command API on127.0.0.1:6666) and the pod-resources socket mounted, and FM inFABRIC_MODE=2.See
docs/fabric-partition-activation.mdfor the full design.Scope
This PR activates single-GPU fabric partitions — one partition per allocated VF, resolved from the VF's parent physical GPU. Multi-GPU partition activation (giving one VM several whole-card vGPUs with NVLink P2P between them — which requires activating a predefined multi-GPU partition with one VF per member GPU, plus
GetPreferredAllocationsupport so the allocated VFs land on GPUs that form a defined partition) is deliberately out of scope here and implemented in the follow-up #195. MIG-backed vGPUs need no activation at all: MIG disables NVLink, so the GPU is not on the fabric and CUDA initializes standalone.