moezdil · Oct 11, 2026 · Oct 11, 2026
diff --git a/README.md b/README.md
@@ -215,10 +215,11 @@ See the [chart README](helm/amd-gpu/README.md) for all values and the [configura
 | GPU | Architecture | Validated |
 |---|---|---|
 | AMD Instinct MI300X VF | CDNA3 | Registration (UUID, product, VRAM, CUs) |
+| AMD Instinct MI325X VF | CDNA3, gfx942 | Registration, CU slices (disjoint masks for concurrent pods), memory hook and mask pinning, dmem caps (kernel 7.0), CDI, node events, monitor and labeller, the operating modes (a compute-partition flip fails on a VF and the GPU stays registered), exporter health paths |
 | Radeon RX 9060 XT | RDNA4, gfx1200 | Registration, health, CU and memory slices, CU-mask pinning, plugin restart |
 | Radeon RX 9070 XT + Radeon iGPU | RDNA4 gfx1201 + RDNA2 gfx1036 | Multi-GPU ordering, per-GPU memory and dmem caps, WGP slices, CDI, GPU removal |
 
-RDNA3 (gfx11) and CDNA compute or memory slicing have not been validated yet.
+RDNA3 (gfx11) has not been validated yet. Several-GPU behavior on Instinct (XGMI peers, spread and binpack ordering), a real compute-partition flip and managed-memory oversubscription past the VRAM size (it needs XNACK) have not been checked on hardware either; see [Hardware verification](docs/contributing/development.md#hardware-verification).
 
 ## Development
 

diff --git a/docs/contributing/development.md b/docs/contributing/development.md
@@ -50,3 +50,15 @@ Make sure to name your branch descriptively to reflect the changes you are makin
 ## Code Review Process
 
 All contributions will undergo a code review process. Reviewers will assess the quality of the code, adherence to coding standards, and the completeness of tests. Be open to feedback and ready to make adjustments as needed.
+
+## Hardware verification
+
+Changes to discovery, AMD SMI, ROCr visibility, CU masks, the memory hook or the dmem cap need a run on a real GPU node. On an Instinct MI325X (one GPU, as a virtual function) with k3s and the HAMi scheduler these checks are the useful ones:
+
+- `go vet ./...` and `go test -race ./...` in the builder image with `--privileged --device /dev/kfd --device /dev/dri -v /sys:/sys -v /sys/kernel/debug:/sys/kernel/debug`, so the GPU-gated tests run instead of skipping.
+- A whole-GPU pod and a sliced pod (`amd.com/gpucores`, `amd.com/gpumem`): check `HSA_CU_MASK`, `ROCR_VISIBLE_DEVICES`, `HIP_VISIBLE_DEVICES`, `LD_AUDIT` and `HIP_DEVICE_MEMORY_LIMIT_0`, then a `hipMalloc` over the limit.
+- Several sliced pods at once: the masks must not overlap and the node lock annotation `hami.io/mutex.lock` must be empty afterwards.
+- The dmem cap needs kernel 6.14 or newer: `dmem.capacity` must list `drm/<bdf>/vram` and a slice's pod cgroup must show it in `dmem.max`.
+- `govulncheck ./...` should report no vulnerabilities.
+
+Not covered by that node: several GPUs (XGMI peers, spread and binpack ordering, `amd.com/link-bind`), a real compute-partition flip (AMD SMI returns status 2 on a VF), RDNA behavior, and a `hipMalloc` larger than the VRAM (it needs XNACK).
diff --git a/docs/user-guide/configuration.md b/docs/user-guide/configuration.md
@@ -11,6 +11,10 @@ The chart passes its values to the plugin as flags. The [chart README](https://g
 | `dp.logVerbosity` | `-v` | `2` | Log verbosity; 4 adds topology parsing, 5 adds annotation decoding. |
 | `dp.healthPulse` | `-pulse` | `10` (chart), `0` (flag) | Seconds between GPU health checks. `0` disables them. |
 | `dp.dmemBackend` | `-dmem_backend` | `true` | Also cap a slice's VRAM with the kernel dmem cgroup controller. |
+| `dp.reportNodeCapacity` | `-report_node_capacity` | `false` | Publish the healthy GPUs' memory (`amd.com/gpumem`, MiB) and compute units (`amd.com/gpucores`) as node capacity and allocatable. Turning it off removes the entries again. |
+| `dp.deviceMemoryScaling` | `-device_memory_scaling` | `1` | Multiplies the VRAM each GPU registers. Above `1`, pods get `HIP_OVERSUBSCRIBE=true` and `hipMalloc` is served from managed memory (needs XNACK on CDNA). |
+| `dp.excludeGPUs` | `-exclude_gpus` | empty | GPUs this node keeps for itself, as PCI BDFs (`0000:06:00.0`) or DRM cards (`card1`). With none left, the stale registration is cleared. |
+| `dp.hipLogLevel` | `LIBHIP_LOG_LEVEL` in pods | `0` | Log level of the memory hook in pods (1 error, 2 warn, 3 info, 4 debug); `0` keeps the library default. |
 | `dp.cdi.enabled`, `dp.cdi.specDir` | `-cdi_spec_dir` | off, `/var/run/cdi` | Inject GPUs through CDI. |
 | `dp.muslFailClosed.*` | `-musl_fail_closed`, `-ctr_path`, `-containerd_socket` | off | Refuse a slice whose image cannot load the memory hook (musl, static, or glibc older than 2.34). |
 | | `-resource_naming_strategy` | `single` | `single` or `mixed`; see below. |
@@ -77,7 +81,7 @@ flips a single GPU; mix spx and qpx GPUs on one node. See
 
 The memory hook limits VRAM inside the process, so it needs glibc 2.34 or newer and can be bypassed. The dmem cap is enforced by the kernel. It is on by default, and the plugin turns it off at startup, with one log line, unless the node has all of:
 
-- cgroup v2 with the `dmem` controller (a recent kernel, listed in `/sys/fs/cgroup/cgroup.controllers`),
+- cgroup v2 with the `dmem` controller (kernel 6.14 or newer, listed in `/sys/fs/cgroup/cgroup.controllers`; kernel 6.8 has none and the plugin logs that the cap is skipped),
 - the systemd cgroup driver (`/sys/fs/cgroup/kubepods.slice` exists).
 
 For each sliced GPU, the plugin writes `drm/<bdf>/vram <bytes>` to the pod cgroup's `dmem.max`. The cap is written in the background shortly after the container starts, because the pod cgroup can take a few seconds to appear, and lowering `dmem.max` below current usage does not reclaim VRAM, so memory a workload allocates before the cap lands stays allocated. A whole-GPU pod gets no dmem cap. Do not put whole-GPU pods and slices on the same card if VRAM isolation matters.
@@ -86,6 +90,10 @@ For each sliced GPU, the plugin writes `drm/<bdf>/vram <bytes>` to the pod cgrou
 
 `dp.muslFailClosed.enabled=true` makes the plugin inspect a sliced pod's image with `ctr` before it starts. If the image is musl-based, statically linked, or uses glibc older than 2.34, the memory hook would not load, so the pod is refused, unless dmem caps its VRAM anyway, in which case the plugin leaves the hook out. The containerd socket and data directory must be mounted with `mountPropagation: HostToContainer`. The chart's defaults are RKE2 paths; override `dp.muslFailClosed.ctrPath`, `containerdSocketDir`, `containerdSocket` and `containerdDataDir` for other distributions. This is experimental.
 
+The chart defaults for `dp.muslFailClosed.ctrPath` and `dp.muslFailClosed.containerdDataDir` are RKE2 paths. On k3s set `ctrPath` to `/var/lib/rancher/k3s/data/current/bin/ctr` and `containerdDataDir` to `/var/lib/rancher/k3s/agent/containerd`.
+
+The kubelet calls the plugin when it admits the pod, before it pulls the image. If the image is not on the node yet, the plugin starts pulling it in the background and refuses that first pod with `image is not on the node yet`; a retry (a Deployment recreates the pod) finds the image and starts.
+
 ## CDI
 
 With `dp.cdi.enabled=true` the plugin writes an `amd.com/gpu` CDI spec to `dp.cdi.specDir`, with one device per DRM card plus `/dev/kfd`, and returns CDI device names instead of device nodes. The container runtime must have CDI enabled and read that directory; containerd 2.x and CRI-O do by default. CDI device responses from a device plugin need Kubernetes 1.28 or newer (the `DevicePluginCDIDevices` feature gate, on by default since 1.29).

diff --git a/docs/user-guide/installation.md b/docs/user-guide/installation.md
@@ -44,7 +44,7 @@ kubectl get node <node> -o jsonpath='{.metadata.annotations.hami\.io/node-amd-re
 
 - **No `amd.com/gpu` on the node.** Read the plugin log with `kubectl -n kube-system logs ds/amd-gpu-device-plugin-daemonset`. Check that `/dev/kfd` and `/dev/dri` exist on the host and that no other plugin owns `amd.com/gpu`.
 - **A sliced pod fails with `UnexpectedAdmissionError`.** The GPU has no free CUs for the request, often because of slices rounded up to whole WGPs on RDNA. A HAMi scheduler that reads `cuPerWGP` keeps such pods `Pending` instead.
-- **A sliced pod ignores its memory limit or fails to start.** The hook needs glibc 2.34 or newer: musl and static images ignore it, and older glibc images fail with `GLIBC_2.34 not found`. Use a newer image. dmem is on by default and caps VRAM without the hook where the node supports it; for older glibc images also enable `dp.muslFailClosed` so the plugin leaves the hook out (see [Configuration](configuration.md)).
+- **A sliced pod ignores its memory limit or fails to start.** The hook needs glibc 2.34 or newer: musl and static images ignore it, and older glibc images fail with `GLIBC_2.34 not found`. Use a newer image. With `dp.muslFailClosed` on, the first pod using an image that is not on the node yet is refused with `image is not on the node yet` while the plugin pulls it; run it again. dmem is on by default and caps VRAM without the hook where the node supports it; for older glibc images also enable `dp.muslFailClosed` so the plugin leaves the hook out (see [Configuration](configuration.md)).
 
 ## Uninstall
 

diff --git a/docs/user-guide/resource-allocation.md b/docs/user-guide/resource-allocation.md
@@ -16,6 +16,8 @@ Each GPU is published `count` times; that count is how many pods may share it. B
 
 On RDNA (gfx10 and later) the CU mask is applied per WGP, a pair of CUs, and a mask that enables only one CU of a pair is ignored, which would run the pod on the whole GPU. So on RDNA the plugin allocates whole WGPs and rounds the request up. For example, 3 CUs become 4 (`0:0-3`), and the next slice starts at `0:4`. The plugin publishes the pair size as `custominfo.cuPerWGP`, so a HAMi scheduler that reads it accounts the same count. A single-WGP GPU, such as a small iGPU, can only be given out whole.
 
+A request without `amd.com/gpucores` gets all of the GPU's CUs, so the HAMi scheduler books the whole card's compute for it: a pod that sets only `amd.com/gpumem` takes the full CU range and the next such pod waits until it ends. Set `amd.com/gpucores` on every pod that should share a GPU.
+
 The mask is cooperative. The hook pins `HSA_CU_MASK` to the pod spec so the process cannot widen it, but a process that re-executes itself without the hook runs outside the slice.
 
 ## Memory

diff --git a/helm/amd-gpu/README.md b/helm/amd-gpu/README.md
@@ -24,22 +24,24 @@ helm install amd-gpu ./helm/amd-gpu -n kube-system
 | dp.serviceAccount.create | bool | `true` | Create the service account and RBAC needed to register devices and persist allocations. |
 | dp.serviceAccount.name | string | `""` | Override the service account name. With `create: false`, RBAC is bound to this existing account, or skipped when it is empty. |
 | dp.serviceAccount.annotations | object | `{}` | Service account annotations. |
+| dp.operatingMode | string | `"cu"` | `cu` registers whole GPUs sliced by CU; `partition` registers each compute partition of an Instinct GPU as a hard device. The `hami.io/amd-operating-mode` node annotation overrides it. |
+| dp.computePartition | string | `""` | In partition mode, the compute partition (`spx`, `dpx`, `qpx`, `cpx`) idle GPUs are switched to at startup. The `hami.io/amd-compute-partition` node annotation overrides it. |
 | dp.securityContext.privileged | bool | `true` | Let the plugin query AMD GPU device nodes on the host. |
 | dp.securityContext.capabilities.drop | list | `["ALL"]` | Capabilities dropped from the plugin container. |
 | dp.hostHookPath | string | `"/usr/local"` | Host path prefix; the hook is installed under `<hostHookPath>/vgpu`. |
 | dp.hookInstaller.enabled | bool | `true` | Install the image-bundled `libamvgpu.so` on the node with a postStart hook. |
 | dp.logVerbosity | int | `2` | glog verbosity; 4 adds topology parsing, 5 adds annotation decoding. |
 | dp.healthPulse | int | `10` | Seconds between per-GPU health checks; 0 disables them. |
-| dp.reportNodeCapacity | bool | `false` | Publish the healthy GPUs' memory (`amd.com/gpumem`, MiB) and compute units (`amd.com/gpucores`) as node capacity and allocatable. |
+| dp.reportNodeCapacity | bool | `false` | Publish the healthy GPUs' memory (`amd.com/gpumem`, MiB) and compute units (`amd.com/gpucores`) as node capacity and allocatable. Turning it off removes the entries again. |
 | dp.deviceMemoryScaling | float | `1` | Multiplies the VRAM each GPU registers. Above `1` oversubscribes: pods get `HIP_OVERSUBSCRIBE=true` and `hipMalloc` is served from managed memory that spills to host RAM. Slower than VRAM; needs XNACK on CDNA. |
 | dp.splitCount | int | `0` | How many workloads may share one GPU. `0` picks it per GPU: 2 on gfx12, otherwise 10. |
 | dp.allocatorPolicy | string | `"besteffort"` | Multi-GPU preferred allocation: `besteffort` (same as `binpack`) or `spread`. |
 | dp.dmemBackend | bool | `true` | Also cap sliced VRAM with the kernel dmem cgroup; skipped on nodes without dmem or the systemd cgroup driver. |
 | dp.muslFailClosed.enabled | bool | `false` | Refuse a slice whose image cannot load the memory hook (musl, static, or glibc older than 2.34), unless dmem caps it. Experimental. |
-| dp.muslFailClosed.ctrPath | string | `"/var/lib/rancher/rke2/bin/ctr"` | Host path of `ctr`. |
+| dp.muslFailClosed.ctrPath | string | `"/var/lib/rancher/rke2/bin/ctr"` | Host path of `ctr` (RKE2 default; on k3s `/var/lib/rancher/k3s/data/current/bin/ctr`). |
 | dp.muslFailClosed.containerdSocketDir | string | `"/run/k3s/containerd"` | Host directory of the containerd socket. |
 | dp.muslFailClosed.containerdSocket | string | `"/run/k3s/containerd/containerd.sock"` | containerd socket path. |
-| dp.muslFailClosed.containerdDataDir | string | `"/var/lib/rancher/rke2/agent/containerd"` | containerd data directory. |
+| dp.muslFailClosed.containerdDataDir | string | `"/var/lib/rancher/rke2/agent/containerd"` | containerd data directory (RKE2 default; on k3s `/var/lib/rancher/k3s/agent/containerd`). |
 | dp.excludeGPUs | list | `[]` | GPUs this node keeps for itself and does not register, as PCI BDFs (`0000:06:00.0`) or DRM cards (`card1`). |
 | dp.hipLogLevel | int | `0` | `LIBHIP_LOG_LEVEL` of the pods that load `libamvgpu` (1 error, 2 warn, 3 info, 4 debug); 0 keeps the library default. |
 | dp.cdi.enabled | bool | `false` | Inject GPUs through CDI instead of device nodes. |