diff --git a/docs/user-guide/configuration.md b/docs/user-guide/configuration.md index c47675f8..8f18b149 100644 --- a/docs/user-guide/configuration.md +++ b/docs/user-guide/configuration.md @@ -45,6 +45,31 @@ Mode resolution, lowest to highest precedence: The annotation is read once at plugin start; changing it requires a plugin pod restart. The plugin logs the resolved mode as `operating mode: `. +### Changing the operating mode + +Drain the node first. Pods placed under one mode keep that mode's slot count +and CU layout, so switching while GPU Pods run can leave the scheduler and the +node disagreeing about what is free. If a GPU switches while Pods still use +it, the plugin logs a warning and records a node Warning event, +`AMDGPUModeChangedInUse`, naming the Pods. It does not stop them. + +1. Cordon and drain the node (`kubectl cordon `, + `kubectl drain --ignore-daemonsets --delete-emptydir-data`), or + delete every Pod that requests `amd.com/gpu`. +2. Change the mode: set `hami.io/amd-operating-mode` on the node, or the + `dp.operatingMode` chart value. +3. Restart the plugin Pod (`kubectl -n kube-system rollout restart + ds/amd-gpu-device-plugin-daemonset`). +4. Check the log for `operating mode: `, and that + `amd.com/gpu` under `kubectl get node -o jsonpath='{.status.allocatable}'` + matches the new mode (the split count in `cu`, one per GPU in `partition`). +5. `kubectl uncordon `. + +The same applies when upgrading from a build older than the device ID fix +(#159) that was running in `partition` mode: Pods allocated by that build are +recorded under a `#` ID the new build does not recognise. Drain +the node before upgrading. + Partition counts are per GPU: `amd-smi set -g --compute-partition qpx` flips a single GPU; mix spx and qpx GPUs on one node. See [MI355X Partitioning](mi355x-partitioning.md) for kernel caveats.