From 5c9b8880e06d44c0059b4289109d1b0cf5c082bb Mon Sep 17 00:00:00 2001 From: YoungjuKim Date: Thu, 15 Oct 2026 00:09:13 +0900 Subject: [PATCH 1/2] docs: troubleshoot driver/library mismatch after unattended upgrades Add a troubleshooting section for nodes that use a pre-installed driver on Ubuntu. The unattended-upgrades package can upgrade the NVIDIA driver packages in the background. The user-space libraries change, but the loaded kernel module stays at the old version until a reboot. GPU pods then fail with an nvidia-persistenced socket error and nvidia-smi on the host reports a driver/library version mismatch. The section shows how to confirm the mismatch, how to recover by rebooting or reloading the modules, and how to keep unattended-upgrades away from the driver packages. Signed-off-by: YoungjuKim --- gpu-operator/troubleshooting.rst | 117 +++++++++++++++++++++++++++++++ 1 file changed, 117 insertions(+) diff --git a/gpu-operator/troubleshooting.rst b/gpu-operator/troubleshooting.rst index da6258660..e9eefe512 100644 --- a/gpu-operator/troubleshooting.rst +++ b/gpu-operator/troubleshooting.rst @@ -660,3 +660,120 @@ Refer to this `GitHub issue `_ for d The errors will eventually resolve on their own after the driver daemonset is installed or the upgrade is complete. This issue was fixed in GPU Operator v25.10.1 and later. + +************************************************************************** +Driver/Library Version Mismatch After Unattended Upgrades on Ubuntu Nodes +************************************************************************** + +.. rubric:: Issue + :class: h4 + +This issue applies when you use a GPU driver that is pre-installed on the host, such as when you set ``driver.enabled=false``. +On Ubuntu, the ``unattended-upgrades`` package can upgrade the NVIDIA driver packages in the background. +The upgrade replaces the user-space driver libraries and utilities. +The kernel module that is already loaded stays at the previous version until you reload it or reboot the node. + +.. rubric:: Observation + :class: h4 + +- Pods that request GPUs fail to start. + The pod events include an error that ends with ``nvidia-persistenced/socket: no such file or directory``. + +- Running ``nvidia-smi`` on the host fails: + + .. code-block:: console + + $ nvidia-smi + + *Example Output* + + .. code-block:: output + + Failed to initialize NVML: Driver/library version mismatch + +- The version of the loaded kernel module differs from the version of the installed driver packages. + Display the version of the loaded kernel module: + + .. code-block:: console + + $ cat /proc/driver/nvidia/version + + Display the versions of the installed driver packages: + + .. code-block:: console + + $ dpkg -l | grep -i nvidia + + Check whether ``unattended-upgrades`` upgraded the driver packages: + + .. code-block:: console + + $ grep -i nvidia /var/log/unattended-upgrades/unattended-upgrades.log + +.. rubric:: Root Cause + :class: h4 + +The loaded kernel module and the user-space driver libraries have different versions. +NVML does not initialize when the versions differ. +The ``nvidia-persistenced`` daemon from the upgraded packages also fails to start. +Its socket file does not exist, so containers that expect to mount the socket fail to start. + +.. rubric:: Action + :class: h4 + +Load the kernel module that matches the upgraded libraries. + +#. Drain the node: + + .. code-block:: console + + $ kubectl drain --ignore-daemonsets --delete-emptydir-data + +#. Reboot the node: + + .. code-block:: console + + $ sudo reboot + +#. After the node is ready, uncordon the node: + + .. code-block:: console + + $ kubectl uncordon + +If you cannot reboot the node, you can reload the kernel modules instead. +You must first stop all processes that use the GPU. +If a module is still in use, the unload fails and you must reboot the node. + +.. code-block:: console + + $ sudo modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia + +.. code-block:: console + + $ sudo modprobe nvidia + +To prevent the issue, exclude the driver packages from unattended upgrades. +Upgrade the driver during a planned maintenance window and reboot the node afterward. +Use one of the following methods: + +- Hold the installed NVIDIA driver packages. + Replace the placeholder with the package names that ``dpkg -l | grep -i nvidia`` reports: + + .. code-block:: console + + $ sudo apt-mark hold + + You must run ``sudo apt-mark unhold `` before you upgrade the driver. + +- Add the driver packages to the ``Unattended-Upgrade::Package-Blacklist`` list. + The entries are regular expressions that match the start of the package name: + + .. code-block:: console + + $ sudo tee /etc/apt/apt.conf.d/51nvidia-unattended-upgrades <<'EOT' + Unattended-Upgrade::Package-Blacklist { + "nvidia-"; + "libnvidia-"; + }; + EOT From 0604fc9d31e0fb58a12bb4431adfcaf58d433612 Mon Sep 17 00:00:00 2001 From: YoungjuKim Date: Sat, 17 Oct 2026 23:28:50 +0900 Subject: [PATCH 2/2] docs: regenerate CDI specs after reloading the driver modules The CDI specifications that the toolkit and the device plugin write to /var/run/cdi are generated only when their pods start. They reference the driver libraries by versioned file names, so after an unattended driver upgrade they point to files that no longer exist. Explain this in the root cause and add a step to restart the two pods after reloading the modules without a reboot. Signed-off-by: YoungjuKim --- gpu-operator/troubleshooting.rst | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/gpu-operator/troubleshooting.rst b/gpu-operator/troubleshooting.rst index e9eefe512..151a5d052 100644 --- a/gpu-operator/troubleshooting.rst +++ b/gpu-operator/troubleshooting.rst @@ -716,7 +716,11 @@ The kernel module that is already loaded stays at the previous version until you The loaded kernel module and the user-space driver libraries have different versions. NVML does not initialize when the versions differ. The ``nvidia-persistenced`` daemon from the upgraded packages also fails to start. -Its socket file does not exist, so containers that expect to mount the socket fail to start. + +The Container Device Interface (CDI) specifications on the node also become stale. +The NVIDIA Container Toolkit and the NVIDIA Device Plugin generate the files in ``/var/run/cdi`` only when their pods start. +The files list the driver libraries by their versioned file names, such as ``libcuda.so.``, and the ``nvidia-persistenced`` socket. +The upgrade removes the libraries of the previous version, and the socket does not exist, so containers that are created from these specifications fail to start. .. rubric:: Action :class: h4 @@ -753,6 +757,25 @@ If a module is still in use, the unload fails and you must reboot the node. $ sudo modprobe nvidia +After you reload the modules, restart the NVIDIA Container Toolkit and NVIDIA Device Plugin pods on the node so that they regenerate the CDI specifications. +You do not need this step after a reboot, because the pods restart with the node. + +.. code-block:: console + + $ kubectl delete pod -n gpu-operator -l app=nvidia-container-toolkit-daemonset \ + --field-selector spec.nodeName= + +.. code-block:: console + + $ kubectl delete pod -n gpu-operator -l app=nvidia-device-plugin-daemonset \ + --field-selector spec.nodeName= + +To confirm that the CDI specifications match the loaded driver, compare the library versions in the specifications on the node with the version in ``/proc/driver/nvidia/version``: + +.. code-block:: console + + $ grep -oh 'libcuda.so.[0-9.]*[0-9]' /var/run/cdi/* | sort | uniq -c + To prevent the issue, exclude the driver packages from unattended upgrades. Upgrade the driver during a planned maintenance window and reboot the node afterward. Use one of the following methods: