[fjvbn2003]

Adds a section to gpu-operator/troubleshooting.rst for clusters that use a pre-installed GPU driver on Ubuntu (driver.enabled=false).

On these nodes, the unattended-upgrades package can upgrade the NVIDIA driver packages in the background. The user-space libraries and utilities move to the new version, but the loaded kernel module stays at the old version until the node is rebooted. After that:

• pods that request GPUs fail to start, and the pod events include an error that ends with nvidia-persistenced/socket: no such file or directory;
• nvidia-smi on the host fails with Failed to initialize NVML: Driver/library version mismatch.

The pod error does not point at the driver, so it is easy to look for the cause in the GPU Operator or the container toolkit first. We hit this on a kubeadm cluster with pre-installed drivers on Ubuntu after an unattended upgrade of the driver packages, and recovered by rebooting the node.

The new section uses the same Issue / Observation / Root Cause / Action layout as the other entries. It covers:

• how to confirm the mismatch (/proc/driver/nvidia/version against the installed package versions, and the unattended-upgrades log);
• recovery by draining and rebooting the node, or by reloading the kernel modules when a reboot is not possible;
• prevention with apt-mark hold or Unattended-Upgrade::Package-Blacklist.

The existing footnote in platform-support.rst already recommends disabling unattended kernel upgrades. I kept this PR to the troubleshooting page. I can extend that footnote to the driver packages in a follow-up if you prefer.

This change was drafted with AI assistance and reviewed by me.

🤖 Generated with Claude Code

[fjvbn2003]

I added one more piece that we measured after opening this PR. A second node in the same cluster hit the same unattended upgrade (595.91.07 → 595.99.02) and was not rebooted. The CDI specs on that node still date from the pod start before the upgrade:

• /var/run/cdi/k8s.device-plugin.nvidia.com-gpu.json: 35 of 59 host paths no longer exist (all *.so.595.91.07 libraries, the 595.91.07 GSP firmware and /run/nvidia-persistenced/socket);
• /var/run/cdi/management.nvidia.com-gpu.yaml: 34 of 46 host paths no longer exist.

On every GPU node the modification time of both files equals the start time of the toolkit and device plugin pods, and the nodes that were rebooted after the upgrade have specs with 595.99.02. So after a reboot the specs are regenerated, but if a reader reloads the modules instead, the stale specs remain until those pods restart. The new paragraph covers that.

I used an AI assistant (Claude) to help investigate and write this change; I reviewed and verified it myself.