From d80ca698b843d9552fac7b94772fec822a30ad65 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Wed, 7 Oct 2026 13:42:41 +0000 Subject: [PATCH 1/5] feat: GPU Op 26.7.2 Signed-off-by: Mike McKiernan Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Signed-off-by: Mike McKiernan --- gpu-operator/life-cycle-policy.rst | 38 ++++++++++++++---------- gpu-operator/release-notes.rst | 47 ++++++++++++++++++++++++++++++ repo.toml | 2 +- 3 files changed, 70 insertions(+), 17 deletions(-) diff --git a/gpu-operator/life-cycle-policy.rst b/gpu-operator/life-cycle-policy.rst index 73f26e11e..33b6ff1c9 100644 --- a/gpu-operator/life-cycle-policy.rst +++ b/gpu-operator/life-cycle-policy.rst @@ -88,10 +88,11 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. :header-rows: 2 * - :rspan:`1` Component - - :cspan:`1` GPU Operator Version + - :cspan:`2` GPU Operator Version * - v26.7.0 - v26.7.1 + - v26.7.2 * - NVIDIA GPU Driver - | `610.57.04 `_ @@ -101,66 +102,71 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. | `610.57.04 `_ | `595.91.07 `_ (**R**, **D**) | `580.173.02 `_ + - | `615.71.09 `_ + | `610.57.04 `_ + | `595.91.07 `_ (**R**, **D**) + | `580.173.02 `_ * - NVIDIA Driver Manager for Kubernetes - `v0.12.0 `__ - - `v0.12.1 `__ + - :cspan:`1` `v0.12.1 `__ * - NVIDIA Container Toolkit - `1.20.0 `__ - - `1.20.1 `__ + - :cspan:`1` `1.20.1 `__ * - NVIDIA Device Plugin for Kubernetes - `0.20.0 `__ - - `0.20.1 `__ + - :cspan:`1` `0.20.1 `__ * - DRA Driver for NVIDIA GPUs - - :cspan:`1` `v${dra_version} `__ + - :cspan:`2` `v${dra_version} `__ * - DCGM Exporter - `v4.6.0-4.8.3 `__ - - `v4.6.1-4.8.4 `__ + - :cspan:`1` `v4.6.1-4.8.4 `__ * - Node Feature Discovery - - :cspan:`1` `v0.19.0 `__ + - :cspan:`2` `v0.19.0 `__ * - | NVIDIA GPU Feature Discovery | for Kubernetes - `0.20.0 `__ - - `0.20.1 `__ + - :cspan:`1` `0.20.1 `__ * - NVIDIA MIG Manager for Kubernetes - `0.15.0 `__ - - `0.15.1 `__ + - :cspan:`1` `0.15.1 `__ * - DCGM - `4.6.0-1 `__ - - `4.6.1-1 `__ + - :cspan:`1` `4.6.1-1 `__ * - Validator for NVIDIA GPU Operator - v26.7.0 - v26.7.1 + - v26.7.2 * - NVIDIA KubeVirt GPU Device Plugin - - :cspan:`1` `v1.6.0 `__ + - :cspan:`2` `v1.6.0 `__ * - NVIDIA vGPU Device Manager - `v0.5.0 `__ - - `v0.5.1 `__ + - :cspan:`1` `v0.5.1 `__ * - NVIDIA GDS Driver |gds|_ - - :cspan:`1` `2.29.4 `__ + - :cspan:`2` `2.29.4 `__ * - | NVIDIA Confidential Computing | Manager for Kubernetes - `v0.4.3 `__ - - `v0.4.4 `__ + - :cspan:`1` `v0.4.4 `__ * - NVIDIA GDRCopy Driver - - :cspan:`1` `v2.6 `__ + - :cspan:`2` `v2.6 `__ * - NVIDIA Kata Sandbox Device Plugin - - :cspan:`1` `v0.0.5 `__ + - :cspan:`2` `v0.0.5 `__ .. _gds-open-kernel: diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index e416bb22e..06ee018b1 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -33,6 +33,53 @@ Refer to the :ref:`GPU Operator Component Matrix` for a list of software compone ---- +.. _v26.7.2: + +26.7.2 +====== + +Improvements +------------ + +* The NVIDIA DCGM and NVIDIA DCGM Exporter daemon sets, pods, and services now include the ``app.kubernetes.io/component`` label with the value ``nvidia-dcgm`` or ``nvidia-dcgm-exporter``. + The label applies to deployments managed by the ``ClusterPolicy`` resource and by the ``GPUCluster`` resource. + If you specify the ``app.kubernetes.io/component`` label in the ``daemonsets.labels`` field, the Operator does not apply your value to operands that already have the label, such as NVIDIA DCGM, NVIDIA DCGM Exporter, and the NVIDIA GPU Driver. + The Operator logs a message when it skips a conflicting value. + (`GPU Operator PR #3017 `__) + +Fixed Issues +------------ + +* Fixed an issue on Red Hat OpenShift Container Platform where workloads that request MPS sharing remained in the ``ContainerCreating`` state when you install the DRA Driver for NVIDIA GPUs with the ``GPUCluster`` resource and enable the ``MPSSupport`` feature gate. + The MPS control daemon now runs with the DRA driver service account, and the ``nvidia-dra-driver`` SCC now allows host PID access. + (`GPU Operator PR #3008 `__) + +* Fixed an issue on Red Hat OpenShift Container Platform where ComputeDomain daemon pods failed to start when you install the DRA Driver for NVIDIA GPUs with the ``GPUCluster`` resource and use ComputeDomains. + The ComputeDomain daemon service account can now use the ``anyuid`` SCC, and the ComputeDomain daemon role can now delete ``ComputeDomainClique`` resources. + (`GPU Operator PR #3011 `__, `Issue #3010 `__) + +* Fixed an issue where changing a node label used in an ``NVIDIADriver`` node selector did not always reassign the node to the matching NVIDIA driver resource. + The Operator now reconciles driver ownership when any label in any NVIDIA driver node selector is added, changed, or removed. + (`GPU Operator PR #3003 `__, `Issue #3004 `__) + +* Fixed an issue where the MPS control daemon did not reload after an MPS sharing configuration change. + The configuration manager now signals the ``mps-control-daemon`` process by the process name that the container starts. + (`GPU Operator PR #2974 `__) + +* Fixed an issue where the Operator could not update recurring Kubernetes events, such as ``GPUDriverUpgrade`` events. + The Operator cluster role now includes the ``patch`` verb for events. + (`GPU Operator PR #2995 `__) + +* Fixed an issue where OLM installations on Arm64 GPU nodes failed to start NVIDIA vGPU Device Manager with ``Exec format error``. + The OLM bundle now pins the multi-architecture image index digest for NVIDIA vGPU Device Manager v0.5.1. + (`GPU Operator PR #3013 `__, `Issue #3012 `__) + +* Fixed the ``GPUCluster`` example in the OLM bundle, which specified an outdated DRA Driver for NVIDIA GPUs image. + The example now uses the default DRA driver image that the Operator configures. + (`GPU Operator PR #2960 `__) + +---- + .. _v26.7.1: 26.7.1 diff --git a/repo.toml b/repo.toml index 116c0da3a..4ca706998 100644 --- a/repo.toml +++ b/repo.toml @@ -176,7 +176,7 @@ docs_root = "${root}/gpu-operator" project = "gpu-operator" name = "NVIDIA GPU Operator" version = "26.7" # Update repo_docs.projects.openshift.version to match latest patch version maj.min.patch -source_substitutions = { minor_version = "26.7", version = "v26.7.1", recommended = "595.91.07", dra_version = "0.5.0", kata_version = "4.0.0" } +source_substitutions = { minor_version = "26.7", version = "v26.7.2", recommended = "595.91.07", dra_version = "0.5.0", kata_version = "4.0.0" } copyright_start = 2020 sphinx_exclude_patterns = [ "life-cycle-policy.rst", From 1d0915d6774db03bd16e3f0cc903a2362810b7f4 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Mon, 12 Oct 2026 12:25:50 -0400 Subject: [PATCH 2/5] chore: Describe driver coord w/ Net Op Signed-off-by: Mike McKiernan --- gpu-operator/gpu-operator-rdma.rst | 97 ++++++++++++++- gpu-operator/nvidia-driver-configuration.rst | 4 + gpu-operator/release-notes.rst | 7 ++ openshift/install-gpu-ocp.rst | 119 +++++++++++++++++-- 4 files changed, 214 insertions(+), 13 deletions(-) diff --git a/gpu-operator/gpu-operator-rdma.rst b/gpu-operator/gpu-operator-rdma.rst index 2d3893f00..30fcc2266 100644 --- a/gpu-operator/gpu-operator-rdma.rst +++ b/gpu-operator/gpu-operator-rdma.rst @@ -32,8 +32,8 @@ NVIDIA recommends using the DMA-BUF rather than using the ``nvidia-peermem`` ker The Operator uses GDS driver version 2.17.5 or newer. This version and higher is only supported with the NVIDIA Open GPU Kernel module driver. -In GPU Operator v25.3.0 and later, the ``driver.kernelModuleType`` default is ``auto``, for the supported driver versions. -This configuration allows the GPU Operator to choose the recommended driver kernel module type depending on the driver branch and the GPU devices available. +In GPU Operator v25.3.0 and later, the ``driver.kernelModuleType`` default is ``auto``, for the supported driver versions. +This configuration allows the GPU Operator to choose the recommended driver kernel module type depending on the driver branch and the GPU devices available. Newer driver versions will use the open kernel module by default, however to make sure you are using the open kernel module, include ``--set driver.kernelModuleType=open`` command-line argument in your helm Operator install command. In conjunction with the Network Operator, the GPU Operator can be used to @@ -143,6 +143,99 @@ To use DMA-BUF and network device drivers that are installed on the host: To use the legacy ``nvidia-peermem`` kernel module instead of DMA-BUF, add ``--set driver.rdma.enabled=true`` to either of the preceding commands. Add ``--set driver.kernelModuleType=open`` if you are using a driver version from a branch earlier than R570. +.. _doca-ofed-driver-coordination: + +Special Considerations for Network Operator +=========================================== + +If you use GPU Operator and Network Operator to manage the GPU and DOCA-OFED drivers on the same nodes, +the GPU driver software must stop before a DOCA-OFED update. +Network Operator manages this sequence by using the ``network.nvidia.com/operator.mofed.wait`` node label. +GPU Operator uses this label in the NVIDIADriver node selector. +When Network Operator sets the label to ``true``, the node no longer matches the selector, and GPU Operator stops the GPU driver pod. +When the label returns to ``false``, GPU Operator restores driver ownership and schedules the GPU driver pod on the node. + +The sequence relies on not configuring a default NVIDIA driver resource. +The following procedure configures a non-default NVIDIA driver resource to select nodes only when the label is ``"false"``. + +Before You Begin +---------------- + +* Use GPU Operator v26.7.2 or later. +* Configure Network Operator to manage the DOCA-OFED drivers. +* For an existing installation that uses cluster policy driver management, complete + :ref:`migrating-from-cluster-policy-driver-management` before applying this configuration. + +Configure Driver Coordination +----------------------------- + +#. Install GPU Operator with driver coordination enabled: + + .. code-block:: console + + $ helm install gpu-operator nvidia/gpu-operator \ + -n gpu-operator --create-namespace \ + --version=v26.7.2 \ + --set driver.nvidiaDriverCRD.enabled=true \ + --set driver.nvidiaDriverCRD.deployDefaultCR=false \ + --set driver.rdma.enabled=false + + These arguments enable driver custom resource management, disable chart creation of a default driver resource, and use DMA-BUF instead of the legacy ``nvidia-peermem`` kernel module. + For an existing installation, use the same three ``--set`` arguments with your Helm upgrade command after completing any required driver management migration. + +#. Create a file, such as ``network-coordinated-driver.yaml``, with contents like the following example: + + .. code-block:: yaml + + apiVersion: nvidia.com/v1alpha1 + kind: NVIDIADriver + metadata: + name: demo-network-coordinated-driver + spec: + default: false + repository: nvcr.io/nvidia + image: driver + version: "${recommended}" + nodeSelector: + network.nvidia.com/operator.mofed.wait: "false" + + If you already manage these nodes with an ``NVIDIADriver`` resource, add the coordination selector and set ``default: false`` on that resource instead. + +#. Apply the driver manifest: + + .. code-block:: console + + $ kubectl apply -f network-coordinated-driver.yaml + +#. Check for a default driver resource: + + .. code-block:: console + + $ kubectl get nvidiadriver -o custom-columns=NAME:.metadata.name,DEFAULT:.spec.default + + Ensure no resource shows ``DEFAULT`` set to ``true``. + +Verify Driver Recovery +---------------------- + +#. After the DOCA-OFED update, check the label on the GPU nodes: + + .. code-block:: console + + $ kubectl get nodes -l nvidia.com/gpu.present=true -L network.nvidia.com/operator.mofed.wait + + During the update, ``"true"`` excludes the node from the driver selector so that the GPU driver pod can stop. + After the update, ``"false"`` makes the node match the selector so that GPU Operator can restore driver ownership and scheduling. + A missing label does not match the selector. + +#. Check that GPU driver pods become ready: + + .. code-block:: console + + $ kubectl get pods -n gpu-operator -o wide + + Ensure that the GPU driver pods show ``Running``. + Verifying the Installation of GPUDirect with RDMA ================================================= diff --git a/gpu-operator/nvidia-driver-configuration.rst b/gpu-operator/nvidia-driver-configuration.rst index a5088f80c..4f75370ae 100644 --- a/gpu-operator/nvidia-driver-configuration.rst +++ b/gpu-operator/nvidia-driver-configuration.rst @@ -133,6 +133,10 @@ This setting can help you meet any of the following goals: that selects one of the same nodes. +For clusters with that use Network Operator to manage the DOCA-OFED driver, refer to +:ref:`doca-ofed-driver-coordination` for information about coordinating installation of the GPU driver. + + Feature Compatibility ===================== diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index 06ee018b1..db97992a2 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -78,6 +78,13 @@ Fixed Issues The example now uses the default DRA driver image that the Operator configures. (`GPU Operator PR #2960 `__) +Documentation Improvements +-------------------------- + +* Added documentation to describe how to coordinate the installation of GPU and DOCA-OFED drivers managed by GPU Operator and Network Operator. + For configuration steps, refer to :ref:`doca-ofed-driver-coordination` for Kubernetes or + :ref:`openshift-doca-ofed-driver-coordination` for OpenShift. + ---- .. _v26.7.1: diff --git a/openshift/install-gpu-ocp.rst b/openshift/install-gpu-ocp.rst index 5902f2deb..ad66a5007 100644 --- a/openshift/install-gpu-ocp.rst +++ b/openshift/install-gpu-ocp.rst @@ -139,7 +139,7 @@ As a cluster administrator, you can install the **NVIDIA GPU Operator** using th subscription.operators.coreos.com/gpu-operator-certified created -#. Optional: Log in to the web console and navigate to the **Ecosystem** > **Installed Operators** and select **NVIDIA GPU Operator**. +#. Optional: Log in to the web console and navigate to the **Ecosystem** > **Installed Operators** and select **NVIDIA GPU Operator**. In versions before 4.20, navigate to **Operators** > **Installed Operators** and select **NVIDIA GPU Operator**. In the ``Project: nvidia-gpu-operator`` the following is displayed: @@ -183,6 +183,104 @@ As a cluster administrator, you can install the **NVIDIA GPU Operator** using th .. image:: graphics/cluster_policy_succeed.png +.. _openshift-doca-ofed-driver-coordination: + +******************************************* +Special Considerations for Network Operator +******************************************* + +If you use GPU Operator and Network Operator to manage the GPU and DOCA-OFED drivers on the same nodes, +the GPU driver software must stop before a DOCA-OFED update. +Network Operator manages this sequence by using the ``network.nvidia.com/operator.mofed.wait`` node label. +GPU Operator uses this label in the NVIDIADriver node selector. +When Network Operator sets the label to ``true``, the node no longer matches the selector, and GPU Operator stops the GPU driver pod. +When the label returns to ``false``, GPU Operator restores driver ownership and schedules the GPU driver pod on the node. + +The sequence relies on not configuring a default NVIDIA driver resource. +The following procedure configures a non-default NVIDIA driver resource to select nodes only when the label is ``"false"``. + +Before You Begin +================ + +* Use GPU Operator v26.7.2 or later. +* Configure Network Operator to manage the DOCA-OFED drivers. +* Use a deployment that supports DMA-BUF; the following configuration leaves legacy ``nvidia-peermem`` support disabled. +* Log in with cluster administrator privileges and stop GPU workloads before changing driver management. + +Configure Driver Coordination +============================= + +#. List the cluster policy resources to find the name of your policy: + + .. code-block:: console + + $ oc get clusterpolicy + +#. Edit the cluster policy resource: + + .. code-block:: console + + $ oc edit clusterpolicy + + Set the following fields, retaining the other settings, and save the resource: + + .. code-block:: yaml + + spec: + driver: + useNvidiaDriverCRD: true + rdma: + enabled: false + +#. In the web console, open **Installed Operators**, select **NVIDIA GPU Operator**, and create a non-default ``NVIDIADriver`` resource. + + In the YAML configuration, set the following fields along with the driver settings required by your deployment: + + .. code-block:: yaml + + apiVersion: nvidia.com/v1alpha1 + kind: NVIDIADriver + metadata: + name: demo-network-coordinated-driver + spec: + default: false + repository: nvcr.io/nvidia + image: driver + version: "${recommended}" + nodeSelector: + network.nvidia.com/operator.mofed.wait: "false" + + Save the resource. + +#. Check for a default driver resource: + + .. code-block:: console + + $ oc get nvidiadriver -o custom-columns=NAME:.metadata.name,DEFAULT:.spec.default + + Ensure no resource has ``DEFAULT`` set to ``true``. + +Verify Driver Recovery +====================== + +#. After the DOCA-OFED update, check the label on the GPU nodes: + + .. code-block:: console + + $ oc get nodes -l nvidia.com/gpu.present=true -L network.nvidia.com/operator.mofed.wait + + During the update, ``"true"`` excludes the node from the driver selector so that the GPU driver pod can stop. + After the update, ``"false"`` makes the node match the selector so that GPU Operator can restore driver ownership and scheduling. + A missing label does not match the selector. + +#. Check that GPU driver pods become ready: + + .. code-block:: console + + $ oc get pods -n nvidia-gpu-operator -o wide + + Ensure that the GPU driver pods show ``Running``. + .. _create-cluster-policy: ********************************* @@ -199,7 +297,7 @@ when using **NVIDIA vGPU**. Refer to the appropriate sections below. .. _create-cluster-policy-web-console: Create the cluster policy using the web console ------------------------------------------------ +=============================================== #. In the OpenShift Container Platform web console, from the side menu, select **Ecosystem** > **Installed Operators** (for versions before 4.20, look for **Operators** > **Installed Operators**), and click **NVIDIA GPU Operator**. @@ -224,14 +322,13 @@ Create the cluster policy using the web console .. _verify-gpu-operator-install-ocp: Create the cluster policy using the CLI ---------------------------------------- - +======================================= #. Create the ClusterPolicy: .. code-block:: console - $ oc get csv -n nvidia-gpu-operator $STARTING_CSV -o jsonpath='{.metadata.annotations.alm-examples}' | jq -r 'map(select(.kind == "ClusterPolicy")) | .[0]' > clusterpolicy.json + $ oc get csv -n nvidia-gpu-operator $STARTING_CSV -o jsonpath='{.metadata.annotations.alm-examples}' | jq -r 'map(select(.kind == "ClusterPolicy")) | .[0]' > clusterpolicy.json .. note:: $STARTING_CSV is the value of the ``startingCSV`` field in the ``Subscription`` CR sample created in the :ref:`install-gpu-ocp` section. @@ -258,7 +355,7 @@ Create the ClusterPolicy instance with NVIDIA vGPU *************************************************************************** Prerequisites --------------- +============== Before creating the ClusterPolicy for NVIDIA vGPU, ensure the following: @@ -288,7 +385,7 @@ Before creating the ClusterPolicy for NVIDIA vGPU, ensure the following: -n nvidia-gpu-operator Create the cluster policy using the web console ------------------------------------------------ +=============================================== #. In the OpenShift Container Platform web console, from the side menu, select **Ecosystem** > **Installed Operators** and select **NVIDIA GPU Operator**. In versions before 4.20, navigate to **Operators** > **Installed Operators** and select **NVIDIA GPU Operator**. @@ -296,8 +393,8 @@ Create the cluster policy using the web console #. Select the **ClusterPolicy** tab, then click **Create ClusterPolicy**. The platform assigns the default name *gpu-cluster-policy*. #. Under the **NVIDIA GPU/vGPU Driver config** section fill in the following information - - * Specify the ``secretName`` as the name of the licensing ``Secret`` created during the prerequisite steps for NVIDIA vGPU (for example, ``licensing-config``). + + * Specify the ``secretName`` as the name of the licensing ``Secret`` created during the prerequisite steps for NVIDIA vGPU (for example, ``licensing-config``). .. note:: @@ -319,13 +416,13 @@ Create the cluster policy using the web console Create the cluster policy using the CLI ---------------------------------------- +======================================= #. Create the ClusterPolicy: .. code-block:: console - $ oc get csv -n nvidia-gpu-operator $STARTING_CSV -o jsonpath='{.metadata.annotations.alm-examples}' | jq -r 'map(select(.kind == "ClusterPolicy")) | .[0]') > clusterpolicy.json + $ oc get csv -n nvidia-gpu-operator $STARTING_CSV -o jsonpath='{.metadata.annotations.alm-examples}' | jq -r 'map(select(.kind == "ClusterPolicy")) | .[0]') > clusterpolicy.json .. note:: $STARTING_CSV is the value of the ``startingCSV`` field in the ``Subscription`` CR sample created in the :ref:`install-gpu-ocp` section. From bda74d00746e23023efd1540fb1650db1edf2411 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Mon, 12 Oct 2026 13:23:08 -0400 Subject: [PATCH 3/5] feat: Optional fields in NVD Signed-off-by: Mike McKiernan --- gpu-operator/nvidia-driver-configuration.rst | 4 ++-- gpu-operator/release-notes.rst | 4 ++++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/gpu-operator/nvidia-driver-configuration.rst b/gpu-operator/nvidia-driver-configuration.rst index 4f75370ae..3fd564b0e 100644 --- a/gpu-operator/nvidia-driver-configuration.rst +++ b/gpu-operator/nvidia-driver-configuration.rst @@ -314,13 +314,13 @@ The following table describes some of the fields in the custom resource. - None * - ``default`` - - Specifies whether the resource is the fallback driver configuration for GPU nodes that do not + - Optional. Specifies whether the resource is the fallback driver configuration for GPU nodes that do not match a non-default resource. Only one resource can be the default, and a default resource cannot specify ``nodeSelector``. - ``false`` * - ``driverType`` - - Specifies one of the following: + - Optional. Specifies one of the following: - ``gpu`` to use the NVIDIA data-center GPU driver. - ``vgpu`` to use the NVIDIA vGPU guest driver. diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index db97992a2..6746de477 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -78,6 +78,10 @@ Fixed Issues The example now uses the default DRA driver image that the Operator configures. (`GPU Operator PR #2960 `__) +* Fixed an issue where Operator Lifecycle Manager (OLM) upgrades could fail with ``spec.default: Required value`` when validating existing ``NVIDIADriver`` resources. + The ``spec.default`` and ``spec.driverType`` fields are now optional, with defaults of ``false`` and ``gpu``, respectively. + (`GPU Operator PR #3046 `__) + Documentation Improvements -------------------------- From da678009858e1983e4c09bf81642f12d73ad0d22 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Tue, 13 Oct 2026 11:01:30 -0400 Subject: [PATCH 4/5] feat: DRA driver v0.5.1 Signed-off-by: Mike McKiernan --- gpu-operator/life-cycle-policy.rst | 3 ++- gpu-operator/release-notes.rst | 7 +++++++ repo.toml | 2 +- 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/gpu-operator/life-cycle-policy.rst b/gpu-operator/life-cycle-policy.rst index 33b6ff1c9..4744f24df 100644 --- a/gpu-operator/life-cycle-policy.rst +++ b/gpu-operator/life-cycle-policy.rst @@ -120,7 +120,8 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. - :cspan:`1` `0.20.1 `__ * - DRA Driver for NVIDIA GPUs - - :cspan:`2` `v${dra_version} `__ + - :cspan:`1` `v0.5.0 `__ + - `v${dra_version} `__ * - DCGM Exporter - `v4.6.0-4.8.3 `__ diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index 6746de477..b81ad2b82 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -38,6 +38,13 @@ Refer to the :ref:`GPU Operator Component Matrix` for a list of software compone 26.7.2 ====== +New Features +------------ + +* Updated software component versions: + + - DRA Driver for NVIDIA GPUs v0.5.1 + Improvements ------------ diff --git a/repo.toml b/repo.toml index 4ca706998..9761da848 100644 --- a/repo.toml +++ b/repo.toml @@ -176,7 +176,7 @@ docs_root = "${root}/gpu-operator" project = "gpu-operator" name = "NVIDIA GPU Operator" version = "26.7" # Update repo_docs.projects.openshift.version to match latest patch version maj.min.patch -source_substitutions = { minor_version = "26.7", version = "v26.7.2", recommended = "595.91.07", dra_version = "0.5.0", kata_version = "4.0.0" } +source_substitutions = { minor_version = "26.7", version = "v26.7.2", recommended = "595.91.07", dra_version = "0.5.1", kata_version = "4.0.0" } copyright_start = 2020 sphinx_exclude_patterns = [ "life-cycle-policy.rst", From 2bd263b281f7da380ca2b2152219420390d7c68a Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Tue, 13 Oct 2026 17:09:42 -0400 Subject: [PATCH 5/5] fix: Skip workaround for initial release Signed-off-by: Mike McKiernan --- gpu-operator/gpu-operator-rdma.rst | 93 ------------------- gpu-operator/nvidia-driver-configuration.rst | 4 - gpu-operator/release-notes.rst | 10 -- openshift/install-gpu-ocp.rst | 98 -------------------- 4 files changed, 205 deletions(-) diff --git a/gpu-operator/gpu-operator-rdma.rst b/gpu-operator/gpu-operator-rdma.rst index 30fcc2266..fb883a5a3 100644 --- a/gpu-operator/gpu-operator-rdma.rst +++ b/gpu-operator/gpu-operator-rdma.rst @@ -143,99 +143,6 @@ To use DMA-BUF and network device drivers that are installed on the host: To use the legacy ``nvidia-peermem`` kernel module instead of DMA-BUF, add ``--set driver.rdma.enabled=true`` to either of the preceding commands. Add ``--set driver.kernelModuleType=open`` if you are using a driver version from a branch earlier than R570. -.. _doca-ofed-driver-coordination: - -Special Considerations for Network Operator -=========================================== - -If you use GPU Operator and Network Operator to manage the GPU and DOCA-OFED drivers on the same nodes, -the GPU driver software must stop before a DOCA-OFED update. -Network Operator manages this sequence by using the ``network.nvidia.com/operator.mofed.wait`` node label. -GPU Operator uses this label in the NVIDIADriver node selector. -When Network Operator sets the label to ``true``, the node no longer matches the selector, and GPU Operator stops the GPU driver pod. -When the label returns to ``false``, GPU Operator restores driver ownership and schedules the GPU driver pod on the node. - -The sequence relies on not configuring a default NVIDIA driver resource. -The following procedure configures a non-default NVIDIA driver resource to select nodes only when the label is ``"false"``. - -Before You Begin ----------------- - -* Use GPU Operator v26.7.2 or later. -* Configure Network Operator to manage the DOCA-OFED drivers. -* For an existing installation that uses cluster policy driver management, complete - :ref:`migrating-from-cluster-policy-driver-management` before applying this configuration. - -Configure Driver Coordination ------------------------------ - -#. Install GPU Operator with driver coordination enabled: - - .. code-block:: console - - $ helm install gpu-operator nvidia/gpu-operator \ - -n gpu-operator --create-namespace \ - --version=v26.7.2 \ - --set driver.nvidiaDriverCRD.enabled=true \ - --set driver.nvidiaDriverCRD.deployDefaultCR=false \ - --set driver.rdma.enabled=false - - These arguments enable driver custom resource management, disable chart creation of a default driver resource, and use DMA-BUF instead of the legacy ``nvidia-peermem`` kernel module. - For an existing installation, use the same three ``--set`` arguments with your Helm upgrade command after completing any required driver management migration. - -#. Create a file, such as ``network-coordinated-driver.yaml``, with contents like the following example: - - .. code-block:: yaml - - apiVersion: nvidia.com/v1alpha1 - kind: NVIDIADriver - metadata: - name: demo-network-coordinated-driver - spec: - default: false - repository: nvcr.io/nvidia - image: driver - version: "${recommended}" - nodeSelector: - network.nvidia.com/operator.mofed.wait: "false" - - If you already manage these nodes with an ``NVIDIADriver`` resource, add the coordination selector and set ``default: false`` on that resource instead. - -#. Apply the driver manifest: - - .. code-block:: console - - $ kubectl apply -f network-coordinated-driver.yaml - -#. Check for a default driver resource: - - .. code-block:: console - - $ kubectl get nvidiadriver -o custom-columns=NAME:.metadata.name,DEFAULT:.spec.default - - Ensure no resource shows ``DEFAULT`` set to ``true``. - -Verify Driver Recovery ----------------------- - -#. After the DOCA-OFED update, check the label on the GPU nodes: - - .. code-block:: console - - $ kubectl get nodes -l nvidia.com/gpu.present=true -L network.nvidia.com/operator.mofed.wait - - During the update, ``"true"`` excludes the node from the driver selector so that the GPU driver pod can stop. - After the update, ``"false"`` makes the node match the selector so that GPU Operator can restore driver ownership and scheduling. - A missing label does not match the selector. - -#. Check that GPU driver pods become ready: - - .. code-block:: console - - $ kubectl get pods -n gpu-operator -o wide - - Ensure that the GPU driver pods show ``Running``. - Verifying the Installation of GPUDirect with RDMA ================================================= diff --git a/gpu-operator/nvidia-driver-configuration.rst b/gpu-operator/nvidia-driver-configuration.rst index 3fd564b0e..5121e7152 100644 --- a/gpu-operator/nvidia-driver-configuration.rst +++ b/gpu-operator/nvidia-driver-configuration.rst @@ -133,10 +133,6 @@ This setting can help you meet any of the following goals: that selects one of the same nodes. -For clusters with that use Network Operator to manage the DOCA-OFED driver, refer to -:ref:`doca-ofed-driver-coordination` for information about coordinating installation of the GPU driver. - - Feature Compatibility ===================== diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index b81ad2b82..a82afb21e 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -81,20 +81,10 @@ Fixed Issues The OLM bundle now pins the multi-architecture image index digest for NVIDIA vGPU Device Manager v0.5.1. (`GPU Operator PR #3013 `__, `Issue #3012 `__) -* Fixed the ``GPUCluster`` example in the OLM bundle, which specified an outdated DRA Driver for NVIDIA GPUs image. - The example now uses the default DRA driver image that the Operator configures. - (`GPU Operator PR #2960 `__) - * Fixed an issue where Operator Lifecycle Manager (OLM) upgrades could fail with ``spec.default: Required value`` when validating existing ``NVIDIADriver`` resources. The ``spec.default`` and ``spec.driverType`` fields are now optional, with defaults of ``false`` and ``gpu``, respectively. (`GPU Operator PR #3046 `__) -Documentation Improvements --------------------------- - -* Added documentation to describe how to coordinate the installation of GPU and DOCA-OFED drivers managed by GPU Operator and Network Operator. - For configuration steps, refer to :ref:`doca-ofed-driver-coordination` for Kubernetes or - :ref:`openshift-doca-ofed-driver-coordination` for OpenShift. ---- diff --git a/openshift/install-gpu-ocp.rst b/openshift/install-gpu-ocp.rst index ad66a5007..2cdc1e661 100644 --- a/openshift/install-gpu-ocp.rst +++ b/openshift/install-gpu-ocp.rst @@ -183,104 +183,6 @@ As a cluster administrator, you can install the **NVIDIA GPU Operator** using th .. image:: graphics/cluster_policy_succeed.png -.. _openshift-doca-ofed-driver-coordination: - -******************************************* -Special Considerations for Network Operator -******************************************* - -If you use GPU Operator and Network Operator to manage the GPU and DOCA-OFED drivers on the same nodes, -the GPU driver software must stop before a DOCA-OFED update. -Network Operator manages this sequence by using the ``network.nvidia.com/operator.mofed.wait`` node label. -GPU Operator uses this label in the NVIDIADriver node selector. -When Network Operator sets the label to ``true``, the node no longer matches the selector, and GPU Operator stops the GPU driver pod. -When the label returns to ``false``, GPU Operator restores driver ownership and schedules the GPU driver pod on the node. - -The sequence relies on not configuring a default NVIDIA driver resource. -The following procedure configures a non-default NVIDIA driver resource to select nodes only when the label is ``"false"``. - -Before You Begin -================ - -* Use GPU Operator v26.7.2 or later. -* Configure Network Operator to manage the DOCA-OFED drivers. -* Use a deployment that supports DMA-BUF; the following configuration leaves legacy ``nvidia-peermem`` support disabled. -* Log in with cluster administrator privileges and stop GPU workloads before changing driver management. - -Configure Driver Coordination -============================= - -#. List the cluster policy resources to find the name of your policy: - - .. code-block:: console - - $ oc get clusterpolicy - -#. Edit the cluster policy resource: - - .. code-block:: console - - $ oc edit clusterpolicy - - Set the following fields, retaining the other settings, and save the resource: - - .. code-block:: yaml - - spec: - driver: - useNvidiaDriverCRD: true - rdma: - enabled: false - -#. In the web console, open **Installed Operators**, select **NVIDIA GPU Operator**, and create a non-default ``NVIDIADriver`` resource. - - In the YAML configuration, set the following fields along with the driver settings required by your deployment: - - .. code-block:: yaml - - apiVersion: nvidia.com/v1alpha1 - kind: NVIDIADriver - metadata: - name: demo-network-coordinated-driver - spec: - default: false - repository: nvcr.io/nvidia - image: driver - version: "${recommended}" - nodeSelector: - network.nvidia.com/operator.mofed.wait: "false" - - Save the resource. - -#. Check for a default driver resource: - - .. code-block:: console - - $ oc get nvidiadriver -o custom-columns=NAME:.metadata.name,DEFAULT:.spec.default - - Ensure no resource has ``DEFAULT`` set to ``true``. - -Verify Driver Recovery -====================== - -#. After the DOCA-OFED update, check the label on the GPU nodes: - - .. code-block:: console - - $ oc get nodes -l nvidia.com/gpu.present=true -L network.nvidia.com/operator.mofed.wait - - During the update, ``"true"`` excludes the node from the driver selector so that the GPU driver pod can stop. - After the update, ``"false"`` makes the node match the selector so that GPU Operator can restore driver ownership and scheduling. - A missing label does not match the selector. - -#. Check that GPU driver pods become ready: - - .. code-block:: console - - $ oc get pods -n nvidia-gpu-operator -o wide - - Ensure that the GPU driver pods show ``Running``. - .. _create-cluster-policy: *********************************