Skip to content

Run validation as part of vgpu device manager init container - #2857

Open
kvalliyurnatt wants to merge 1 commit into
NVIDIA:mainfrom
kvalliyurnatt:fix_vgpu_device_manager_init
Open

Run validation as part of vgpu device manager init container#2857
kvalliyurnatt wants to merge 1 commit into
NVIDIA:mainfrom
kvalliyurnatt:fix_vgpu_device_manager_init

Conversation

@kvalliyurnatt

@kvalliyurnatt kvalliyurnatt commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Description

Currently the vgpu device manager init container waits for files written by the sandbox validator to be present, this introduces a circular dependency, since the vgpu device manager while updating a vgpu config disables the sandbox validator from the node, if the vgpu device manager pod restarts before it can enable the sandbox validator back then we are forever stuck in a deadlock. By running the validation in the init container directly, we break the circular dependency.

I will create another PR in the vgpu device manager to make it more robust in container restart scenarios, this only addresses Pod restarts.

partialy fixes: #2845

Checklist

  • No secrets, sensitive information, or unrelated changes
  • Lint checks passing (make lint)
  • Generated assets in-sync (make validate-generated-assets)
  • Go mod artifacts in-sync (make validate-modules)
  • Test cases are added for new code paths

Testing

Added unit test, will test the changes on an actual machine

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The vGPU device manager validation init container now runs nvidia-validator instead of polling readiness files. It receives component, node, workload, and validator environment settings. It mounts the host root, NVIDIA driver directory, and validation directory with the required security configuration. Controller and transformation tests now verify the generated init container and its configuration.

Merge Risk: 🟡 Moderate · up to f4572

Direct validation fixes the restart deadlock, but configured environment overrides can prevent the vGPU DaemonSet from starting or alter validation, and writable host-driver access increases compromise impact. These issues should be addressed before merge.


Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (2)
controllers/object_controls_test.go-1308-1310 (1)

1308-1310: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Assert the new driver propagation and SELinux configuration.

The test checks only the driver mount path. It does not check its HostToContainer propagation or the new SecurityContext.SELinuxOptions.Level. Removing either manifest setting still passes this test.

Add assertions for both settings.

As per path instructions, flag assertions that would still pass if the behavior under test were broken.

Source: Path instructions

controllers/transforms_test.go-2344-2344 (1)

2344-2344: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Use an independent expected workload value.

This expected value uses defaultGPUWorkloadConfig, which is also the value used by the transformation. A regression in that default mapping keeps this test passing while the validator receives the wrong workload configuration.

Assert the intended workload value directly, or use an independent validator contract constant.

As per path instructions, flag tests that mirror the implementation instead of checking behavior.

Source: Path instructions


ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: 645f2c5e-430d-4e6c-aee6-03294b4cd0b0

📥 Commits

Reviewing files that changed from the base of the PR and between df32e7e and f4572f8.

📒 Files selected for processing (4)
  • assets/state-vgpu-device-manager/0600_daemonset.yaml
  • controllers/object_controls.go
  • controllers/object_controls_test.go
  • controllers/transforms_test.go

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Comment thread assets/state-vgpu-device-manager/0600_daemonset.yaml
Comment thread controllers/object_controls.go
Signed-off-by: Karthikeyan Valliyurnatt <kvalliyurnat@nvidia.com>

address review comments

Signed-off-by: Karthikeyan Valliyurnatt <kvalliyurnat@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Vgpu paused label deadlock issue

1 participant