Gate dcgm pods on the driver module and probe hostengine GPU visibility

[gengwg]

Description

Fixes #2854.

The nvidia-dcgm hostengine can start while the nvidia kernel module is not loaded (node boot racing an operator rollout, an in-place nvidia-driver-ctr restart, or an operator rollout that recreates the dcgm pods while the driver container restarts). The toolkit's host runtime wrapper then execs plain runc with no injection, nv-hostengine starts with no NVML, and nothing ever restarts it. Both nvidia-dcgm and nvidia-dcgm-exporter stay 1/1 Running while exporting zero DCGM_FI_* series for the node. Details and logs in the issue.

Two small changes to the assets:

1. Init gate also waits for the driver module. toolkit-validation in state-dcgm and state-dcgm-exporter only checked for /run/nvidia/validations/toolkit-ready, which survives an in-place driver container restart, so the gate passed while the modules were unloaded. It now also requires grep -q '^nvidia ' /proc/modules. /proc/modules is not namespaced, the init container is already privileged, and the validator uses the same lsmod | grep pattern for nvidia_fs and gdrdrv.

2. Hostengine liveness checks GPU visibility, not just the port. The tcpSocket probe on 5555 passes on a deaf hostengine because it still listens. The probe is now dcgmi discovery -l | grep -qE '^[1-9][0-9]* GPUs? found' every 60s, three failures before restart, so a hostengine that came up without NVML gets restarted by kubelet once the driver is back. The readiness probe is unchanged.

Behaviour worth stating: with the new liveness probe, a node that genuinely loses every GPU will see its hostengine restart every ~3 minutes instead of sitting silently at 0 GPUs. That is intended; it is visible where the previous state was not. Nodes with at least one GPU visible are unaffected. When the hostengine restarts, dcgm-exporter hits DCGM_ST_CONNECTION_NOT_VALID, which it already treats as fatal and exits, so it reconnects on its own restart.

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint) - not run locally; YAML-only change under assets/
• Generated assets in-sync (make validate-generated-assets) - no generated files touched
• Go mod artifacts in-sync (make validate-modules) - go.mod untouched
• Test cases are added for new code paths - no Go code paths changed; go test ./controllers/... passes with the modified assets

Testing

• Probe command against the real nvcr.io/nvidia/cloud-native/dcgm:4.4.2-1-ubuntu22.04 image: with GPUs present dcgmi discovery -l prints 8 GPUs found. and the probe passes; started on a host with no GPU the probe fails as intended. sh, grep and dcgmi are all in the image.
• Init check against the image used for toolkit-validation (nvcr.io/nvidia/gpu-operator:v25.10.0): sh and grep present, /proc/modules readable.
• go test ./controllers/... passes with the modified assets.
• Not end-to-end tested with a rebuilt operator image on a live cluster; the failure needs a driver restart to reproduce and I do not have a cluster I can do that on. The mechanism, logs and the manual recovery (delete hostengine, then exporter) were verified on production nodes and are in the issue.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: 83a07265-74ca-4fdc-8c29-1942a4811943

📥 Commits

Reviewing files that changed from the base of the PR and between df32e7e and 119d901.

📒 Files selected for processing (2)

• assets/state-dcgm-exporter/0800_daemonset.yaml
• assets/state-dcgm/0400_dcgm.yml

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

📝 Walkthrough

Walkthrough

The DCGM and DCGM exporter init containers now wait for the toolkit validation marker and the loaded nvidia kernel module. The DCGM liveness probe now runs dcgmi discovery -l and requires GPU discovery. The probe uses explicit delay, period, timeout, and failure threshold settings.

Merge Risk: ⚪ Minimal · up to 119d9

DCGM and its exporter now wait for the NVIDIA driver module before startup, and DCGM restarts when it cannot discover a GPU. No concrete current-head merge-blocking risk remains.

---

_{Comment @coderabbitai help to get the list of available commands.}