docs: troubleshoot driver/library mismatch after unattended upgrades

gpu-operator/troubleshooting.rst

@@ -660,3 +660,143 @@ Refer to this `GitHub issue <https://github.com/cri-o/cri-o/issues/9521>`_ for d
 The errors will eventually resolve on their own after the driver daemonset is installed or the upgrade is complete.
 
 This issue was fixed in GPU Operator v25.10.1 and later.
+
+**************************************************************************
+Driver/Library Version Mismatch After Unattended Upgrades on Ubuntu Nodes
+**************************************************************************
+
+.. rubric:: Issue
+   :class: h4
+
+This issue applies when you use a GPU driver that is pre-installed on the host, such as when you set ``driver.enabled=false``.
+On Ubuntu, the ``unattended-upgrades`` package can upgrade the NVIDIA driver packages in the background.
+The upgrade replaces the user-space driver libraries and utilities.
+The kernel module that is already loaded stays at the previous version until you reload it or reboot the node.
+
+.. rubric:: Observation
+   :class: h4
+
+- Pods that request GPUs fail to start.
+  The pod events include an error that ends with ``nvidia-persistenced/socket: no such file or directory``.
+
+- Running ``nvidia-smi`` on the host fails:
+
+  .. code-block:: console
+
+     $ nvidia-smi
+
+  *Example Output*
+
+  .. code-block:: output
+
+     Failed to initialize NVML: Driver/library version mismatch
+
+- The version of the loaded kernel module differs from the version of the installed driver packages.
+  Display the version of the loaded kernel module:
+
+  .. code-block:: console
+
+     $ cat /proc/driver/nvidia/version
+
+  Display the versions of the installed driver packages:
+
+  .. code-block:: console
+
+     $ dpkg -l | grep -i nvidia
+
+  Check whether ``unattended-upgrades`` upgraded the driver packages:
+
+  .. code-block:: console
+
+     $ grep -i nvidia /var/log/unattended-upgrades/unattended-upgrades.log
+
+.. rubric:: Root Cause
+   :class: h4
+
+The loaded kernel module and the user-space driver libraries have different versions.
+NVML does not initialize when the versions differ.
+The ``nvidia-persistenced`` daemon from the upgraded packages also fails to start.
+
+The Container Device Interface (CDI) specifications on the node also become stale.
+The NVIDIA Container Toolkit and the NVIDIA Device Plugin generate the files in ``/var/run/cdi`` only when their pods start.
+The files list the driver libraries by their versioned file names, such as ``libcuda.so.<version>``, and the ``nvidia-persistenced`` socket.
+The upgrade removes the libraries of the previous version, and the socket does not exist, so containers that are created from these specifications fail to start.
+
+.. rubric:: Action
+   :class: h4
+
+Load the kernel module that matches the upgraded libraries.
+
+#. Drain the node:
+
+   .. code-block:: console
+
+      $ kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
+
+#. Reboot the node:
+
+   .. code-block:: console
+
+      $ sudo reboot
+
+#. After the node is ready, uncordon the node:
+
+   .. code-block:: console
+
+      $ kubectl uncordon <node-name>
+
+If you cannot reboot the node, you can reload the kernel modules instead.
+You must first stop all processes that use the GPU.
+If a module is still in use, the unload fails and you must reboot the node.
+
+.. code-block:: console
+
+   $ sudo modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia
+
+.. code-block:: console
+
+   $ sudo modprobe nvidia
+
+After you reload the modules, restart the NVIDIA Container Toolkit and NVIDIA Device Plugin pods on the node so that they regenerate the CDI specifications.
+You do not need this step after a reboot, because the pods restart with the node.
+
+.. code-block:: console
+
+   $ kubectl delete pod -n gpu-operator -l app=nvidia-container-toolkit-daemonset \
+       --field-selector spec.nodeName=<node-name>
+
+.. code-block:: console
+
+   $ kubectl delete pod -n gpu-operator -l app=nvidia-device-plugin-daemonset \
+       --field-selector spec.nodeName=<node-name>
+
+To confirm that the CDI specifications match the loaded driver, compare the library versions in the specifications on the node with the version in ``/proc/driver/nvidia/version``:
+
+.. code-block:: console
+
+   $ grep -oh 'libcuda.so.[0-9.]*[0-9]' /var/run/cdi/* | sort | uniq -c
+
+To prevent the issue, exclude the driver packages from unattended upgrades.
+Upgrade the driver during a planned maintenance window and reboot the node afterward.
+Use one of the following methods:
+
+- Hold the installed NVIDIA driver packages.
+  Replace the placeholder with the package names that ``dpkg -l | grep -i nvidia`` reports:
+
+  .. code-block:: console
+
+     $ sudo apt-mark hold <nvidia-package-name>
+
+  You must run ``sudo apt-mark unhold <nvidia-package-name>`` before you upgrade the driver.
+
+- Add the driver packages to the ``Unattended-Upgrade::Package-Blacklist`` list.
+  The entries are regular expressions that match the start of the package name:
+
+  .. code-block:: console
+
+     $ sudo tee /etc/apt/apt.conf.d/51nvidia-unattended-upgrades <<'EOT'
+     Unattended-Upgrade::Package-Blacklist {
+         "nvidia-";
+         "libnvidia-";
+     };
+     EOT