diff --git a/assets/images/hosts_create.png b/assets/images/hosts_and_clusters/hosts_create.png similarity index 100% rename from assets/images/hosts_create.png rename to assets/images/hosts_and_clusters/hosts_create.png diff --git a/assets/images/hosts_fireedge.png b/assets/images/hosts_and_clusters/hosts_fireedge.png similarity index 100% rename from assets/images/hosts_fireedge.png rename to assets/images/hosts_and_clusters/hosts_fireedge.png diff --git a/assets/images/sunstone_cluster_create.png b/assets/images/hosts_and_clusters/sunstone_cluster_create.png similarity index 100% rename from assets/images/sunstone_cluster_create.png rename to assets/images/hosts_and_clusters/sunstone_cluster_create.png diff --git a/assets/images/sunstone_cluster_dashboard.png b/assets/images/hosts_and_clusters/sunstone_cluster_dashboard.png similarity index 100% rename from assets/images/sunstone_cluster_dashboard.png rename to assets/images/hosts_and_clusters/sunstone_cluster_dashboard.png diff --git a/assets/images/sunstone_cluster_details.png b/assets/images/hosts_and_clusters/sunstone_cluster_details.png similarity index 100% rename from assets/images/sunstone_cluster_details.png rename to assets/images/hosts_and_clusters/sunstone_cluster_details.png diff --git a/assets/images/sunstone_cluster_evc_attributes.png b/assets/images/hosts_and_clusters/sunstone_cluster_evc_attributes.png similarity index 100% rename from assets/images/sunstone_cluster_evc_attributes.png rename to assets/images/hosts_and_clusters/sunstone_cluster_evc_attributes.png diff --git a/assets/images/sunstone_cluster_evc_update.png b/assets/images/hosts_and_clusters/sunstone_cluster_evc_update.png similarity index 100% rename from assets/images/sunstone_cluster_evc_update.png rename to assets/images/hosts_and_clusters/sunstone_cluster_evc_update.png diff --git a/content/getting_started/try_opennebula/opennebula_sandbox_deployment/deploy_opennebula_onprem_with_poc_iso.md b/content/getting_started/try_opennebula/opennebula_sandbox_deployment/deploy_opennebula_onprem_with_poc_iso.md index d2e224aa7..f3de4e483 100644 --- a/content/getting_started/try_opennebula/opennebula_sandbox_deployment/deploy_opennebula_onprem_with_poc_iso.md +++ b/content/getting_started/try_opennebula/opennebula_sandbox_deployment/deploy_opennebula_onprem_with_poc_iso.md @@ -496,7 +496,7 @@ The filesystem `/` now will be 10GiB bigger with no loss of service. ## GPU Configuration -If the OpenNebula evaluation involves GPU management, GPU should be configured in pass-through mode. For the detailed process check [this guide from the official documentation]({{% relref "/product/cluster_configuration/hosts_and_clusters/nvidia_gpu_passthrough" %}}). Overall, a GPU configuration in OpenNebula consists from 2 main stages: +If the OpenNebula evaluation involves GPU management, GPU should be configured in pass-through mode. For the detailed process check [this guide from the official documentation]({{% relref "/product/cluster_configuration/pci_passthrough_sriov/nvidia_gpu_passthrough" %}}). Overall, a GPU configuration in OpenNebula consists from 2 main stages: - Host preparation and driver configuration - OpenNebula settings for PCI pass-through devices diff --git a/content/getting_started/understand_opennebula/cloud_architecture_and_design/cloud_architecture_design.md b/content/getting_started/understand_opennebula/cloud_architecture_and_design/cloud_architecture_design.md index 321558010..cee8858ee 100644 --- a/content/getting_started/understand_opennebula/cloud_architecture_and_design/cloud_architecture_design.md +++ b/content/getting_started/understand_opennebula/cloud_architecture_and_design/cloud_architecture_design.md @@ -65,10 +65,10 @@ After selecting the hypervisor(s) for your cloud, you are ready to **add the vir Now you should have an OpenNebula cloud up and running with at least one virtualization node. The next step is to configure OpenNebula to work with your infrastructure. OpenNebula directly manages the hypervisor, networking and storage platforms; and you may need additional configuration: -- **Networking setup** with [802.1Q VLANs]({{% relref "../../../product/cluster_configuration/networking_system/vlan#hm-vlan" %}}), [Open vSwitch]({{% relref "../../../product/cluster_configuration/networking_system/openvswitch#openvswitch" %}}) or [VXLAN]({{% relref "../../../product/cluster_configuration/networking_system/vxlan#vxlan" %}}). -- **Storage setup** with [NFS/NAS datastore]({{% relref "../../../product/cluster_configuration/storage_system/nas_ds#nas-ds" %}}), [Local Storage datastore]({{% relref "../../../product/cluster_configuration/storage_system/local_ds#local-ds" %}}), [SAN datastore]({{% relref "../../../product/cluster_configuration/lvm" %}}), [Ceph]({{% relref "../../../product/cluster_configuration/storage_system/ceph_ds#ceph-ds" %}}), [Dev]({{% relref "../../../product/cluster_configuration/storage_system/dev_ds#dev-ds" %}}), or [iSCSI]({{% relref "../../../product/cluster_configuration/san_storage/iscsi_ds#iscsi-ds" %}}) datastore. -- **Host setup** with the configuration options for the [KVM hosts]({{% relref "../../../product/operation_references/hypervisor_configuration/kvm_driver#kvmg" %}}), [LXC hosts]({{% relref "../../../product/operation_references/hypervisor_configuration/lxc_driver#lxdmg" %}}), [Monitoring subsystem]({{% relref "../../../product/cloud_system_administration/resource_monitoring" %}}), [Virtual Machine HA]({{% relref "../../../product/control_plane_configuration/high_availability/vm_ha#ftguide" %}}) or [PCI Passthrough]({{% relref "../../../product/cluster_configuration/hosts_and_clusters/pci_passthrough#kvm-pci-passthrough" %}}). -- **Authentication setup**, OpenNebula includes by default an internal **user/password authentication system**, but it can also use an external authentication driver such as [SSH]({{% relref "../../../product/cloud_system_administration/authentication_configuration/ssh#ssh-auth" %}}), [x509]({{% relref "../../../product/cloud_system_administration/authentication_configuration/x509#x509-auth" %}}), [LDAP]({{% relref "../../../product/cloud_system_administration/authentication_configuration/ldap#ldap" %}}) or [Active Directory]({{% relref "../../../product/cloud_system_administration/authentication_configuration/ldap#ldap" %}}). +- **Networking setup** with [802.1Q VLANs]({{% relref "product/cluster_configuration/networking_system/vlan#hm-vlan" %}}), [Open vSwitch]({{% relref "product/cluster_configuration/networking_system/openvswitch#openvswitch" %}}) or [VXLAN]({{% relref "../../../product/cluster_configuration/networking_system/vxlan#vxlan" %}}). +- **Storage setup** with [NFS/NAS datastore]({{% relref "product/cluster_configuration/storage_system/nas_ds#nas-ds" %}}), [Local Storage datastore]({{% relref "product/cluster_configuration/storage_system/local_ds#local-ds" %}}), [SAN datastore]({{% relref "../../../product/cluster_configuration/lvm" %}}), [Ceph]({{% relref "product/cluster_configuration/storage_system/ceph_ds#ceph-ds" %}}), [Dev]({{% relref "product/cluster_configuration/storage_system/dev_ds#dev-ds" %}}), or [iSCSI]({{% relref "product/cluster_configuration/san_storage/iscsi_ds#iscsi-ds" %}}) datastore. +- **Host setup** with the configuration options for the [KVM hosts]({{% relref "product/operation_references/hypervisor_configuration/kvm_driver#kvmg" %}}), [LXC hosts]({{% relref "product/operation_references/hypervisor_configuration/lxc_driver#lxdmg" %}}), [Monitoring subsystem]({{% relref "product/cloud_system_administration/resource_monitoring" %}}), [Virtual Machine HA]({{% relref "../../../product/control_plane_configuration/high_availability/vm_ha#ftguide" %}}) or [PCI Passthrough]({{% relref "product/cluster_configuration/pci_passthrough_sriov/pci_passthrough#kvm-pci-passthrough" %}}). +- **Authentication setup**, OpenNebula includes by default an internal **user/password authentication system**, but it can also use an external authentication driver such as [SSH]({{% relref "product/cloud_system_administration/authentication_configuration/ssh#ssh-auth" %}}), [x509]({{% relref "product/cloud_system_administration/authentication_configuration/x509#x509-auth" %}}), [LDAP]({{% relref "product/cloud_system_administration/authentication_configuration/ldap#ldap" %}}) or [Active Directory]({{% relref "product/cloud_system_administration/authentication_configuration/ldap#ldap" %}}). ### 3.4. Configure Cloud Services diff --git a/content/product/cluster_configuration/hosts_and_clusters/cluster_guide.md b/content/product/cluster_configuration/hosts_and_clusters/cluster_guide.md index b057089e0..5c25c1f2e 100644 --- a/content/product/cluster_configuration/hosts_and_clusters/cluster_guide.md +++ b/content/product/cluster_configuration/hosts_and_clusters/cluster_guide.md @@ -1,5 +1,5 @@ --- -title: "Clusters" +title: "Cluster Management" date: "2025-02-17" description: categories: @@ -12,14 +12,14 @@ weight: "3" -Clusters group together Hosts, datastores, and virtual networks that are configured to work together. A cluster is used to: +Clusters group together Hosts, datastores, and virtual networks that are configured to work together. A Cluster is used to: > * Ensure that VMs use resources that are compatible. > * Assign resources to user groups by creating Virtual Private Clouds. -Clusters should contain homogeneous resources. Note that some operations like live migrations are restricted to Hosts in the same cluster. +Clusters should contain homogeneous resources. Note that some operations like live migrations are restricted to Hosts in the same Cluster. -The requirements for live migrating VMs between Hosts of the same cluster are that no differences occur in the following areas of the hypervisors: +The requirements for live migrating VMs between Hosts of the same Cluster are that no differences occur in the following areas of the hypervisors: * CPU model * Firmware settings @@ -32,20 +32,20 @@ The requirements for live migrating VMs between Hosts of the same cluster are th ## Cluster Management -Clusters are managed with the `onecluster` command. To create new clusters, use `onecluster create `. Existing clusters can be inspected with the `onecluster list` and `show` commands. +Clusters are managed with the `onecluster` command. To create new Clusters, use `onecluster create `. Existing Clusters can be inspected with the `onecluster list` and `show` commands. -```default -$ onecluster list +```shell +onecluster list ID NAME HOSTS NETS DATASTORES -$ onecluster create production +onecluster create production ID: 100 -$ onecluster list +onecluster list ID NAME HOSTS NETS DATASTORES 100 production 0 0 0 -$ onecluster show production +onecluster show production CLUSTER 100 INFORMATION ID : 100 NAME : production @@ -59,24 +59,24 @@ DATASTORES ### Add Hosts to a Cluster -Every Host must belong to a cluster, so if no cluster is specified it will be assigned to the `default` cluster by default. Hosts can be created directly in a different cluster by using the `--cluster` option of `onehost create`, or be added at any time with the command `onecluster addhost`. Hosts can be in **only one cluster** at a time. +Every Host must belong to a Cluster, so if no Cluster is specified it will be assigned to the `default` Cluster by default. Hosts can be created directly in a different Cluster by using the `--cluster` option of `onehost create`, or be added at any time with the command `onecluster addhost`. Hosts can be in **only one Cluster** at a time. -To delete a Host from a cluster, the command `onecluster delhost` must be used. +To delete a Host from a Cluster, the command `onecluster delhost` must be used. -In the following example, we will add Host 0 to the cluster we created before. You will notice that the `onecluster show` command will list the Host ID 0 as part of the cluster. +In the following example, we will add Host 0 to the Cluster we created before. You will notice that the `onecluster show` command will list the Host ID 0 as part of the Cluster. -```default -$ onehost list +```shell +onehost list ID NAME CLUSTER RVM TCPU FCPU ACPU TMEM FMEM AMEM STAT 0 host01 - 7 400 290 400 3.7G 2.2G 3.7G on -$ onecluster addhost production host01 +onecluster addhost production host01 -$ onehost list +onehost list ID NAME CLUSTER RVM TCPU FCPU ACPU TMEM FMEM AMEM STAT 0 host01 producti 7 400 290 400 3.7G 2.2G 3.7G on -$ onecluster show production +onecluster show production CLUSTER 100 INFORMATION ID : 100 NAME : production @@ -91,22 +91,22 @@ DATASTORES ### Add Resources to Clusters -Datastores and virtual networks can be added to multiple clusters. This means that any Host in those clusters is properly configured to run VMs using images from those datastores, or is using leases from those virtual networks. +Datastores and virtual networks can be added to multiple Clusters. This means that any Host in those Clusters is properly configured to run VMs using images from those datastores, or is using leases from those virtual networks. -For instance, if you have several Hosts configured to use a given Open vSwitch network, you would group them in the same cluster. The [Scheduler]({{% relref "../../cloud_system_administration/scheduler" %}}) will know that VMs using these resources can be deployed in any of the Hosts of the cluster. +For instance, if you have several Hosts configured to use a given Open vSwitch network, you would group them in the same Cluster. The [Scheduler]({{% relref "product/cloud_system_administration/scheduler" %}}) will know that VMs using these resources can be deployed in any of the Hosts of the Cluster. These operations can be done with the `onecluster` `addvnet/delvnet` and `adddatastore/deldatastore`, respectively: -```default -$ onecluster addvnet production priv-ovswitch +```shell +onecluster addvnet production priv-ovswitch -$ onecluster adddatastore production iscsi +onecluster adddatastore production iscsi -$ onecluster list +onecluster list ID NAME HOSTS NETS DATASTORES 100 production 1 1 1 -$ onecluster show 100 +onecluster show 100 CLUSTER 100 INFORMATION ID : 100 NAME : production @@ -125,79 +125,83 @@ DATASTORES ### The System Datastore for a Cluster -In order to create a complete environment where the scheduler can deploy VMs, your clusters need to have at least one System Datastore. +In order to create a complete environment where the scheduler can deploy VMs, your Clusters need to have at least one System Datastore. You can add the default System Datastore (ID: 0) or create a new one to improve its performance (e.g., balance VM I/O between different servers) or to use different System Datastore types (e.g., `shared` and `local`). -To use a specific System Datastore with your cluster instead of the default one, just create it and associate it just like any other datastore (`onecluster adddatastore`). +To use a specific System Datastore with your Cluster instead of the default one, just create it and associate it just like any other datastore (`onecluster adddatastore`). ## Managing Clusters in Sunstone -The [Sunstone UI interface]({{% relref "../../control_plane_configuration/graphical_user_interface/fireedge_sunstone#fireedge-sunstone" %}}) offers an easy way to manage clusters and the resources within them. You will find the cluster sub-menu under the infrastructure menu. From there you will be able to: +The [Sunstone UI interface]({{% relref "../../control_plane_configuration/graphical_user_interface/fireedge_sunstone#fireedge-sunstone" %}}) offers an easy way to manage Clusters and the resources within them. You will find the Cluster sub-menu under the infrastructure menu. From there you will be able to: -- Create new clusters selecting the resources you want to include in this cluster. +- Create new Clusters selecting the resources you want to include in this Cluster. -![create_cluster](/images/sunstone_cluster_create.png) +{{< image path="/images/hosts_and_clusters/sunstone_cluster_create.png" alt="Sunstone login" align="center" width="90%" mb="20px" +>}} -- See the list of current clusters, from which you can update or delete existing ones. +- See the list of current Clusters, from which you can update or delete existing ones. -![dashboard_cluster](/images/sunstone_cluster_dashboard.png) +{{< image path="/images/hosts_and_clusters/sunstone_cluster_dashboard.png" alt="Sunstone login" align="center" width="90%" mb="20px" +>}} +- See Cluster details and update overcommitment. -- See cluster details and update overcommitment. - -![details_cluster](/images/sunstone_cluster_details.png) +{{< image path="/images/hosts_and_clusters/sunstone_cluster_details.png" alt="Sunstone login" align="center" width="90%" mb="20px" +>}} ## Enhanced VM Compatibility (EVC) -The Enhanced VM Compatibility (EVC) feature facilitates the management of heterogeneous OpenNebula clusters by masking host CPU capabilities to enforce a unified base model. Using a lowest-common-denominator approach ensures CPU compatibility across hosts and enables seamless live migration of Virtual Machines between hosts with different processor generations. +The Enhanced VM Compatibility (EVC) feature facilitates the management of heterogeneous OpenNebula Clusters by masking Host CPU capabilities to enforce a unified base model. Using a lowest-common-denominator approach ensures CPU compatibility across Hosts and enables seamless live migration of Virtual Machines between Hosts with different processor generations. -EVC is configured at the cluster level. This simplifies management and improves scalability by allowing administrators to add newer hardware to existing clusters without preventing VM migration due to CPU differences. +EVC is configured at the Cluster level. This simplifies management and improves scalability by allowing administrators to add newer hardware to existing Clusters without preventing VM migration due to CPU differences. ### Using EVC with the CLI -1. Inspect the cluster to view the current template and attributes: +1. Inspect the Cluster to view the current template and attributes: -```bash -$ onecluster show default +```shell +onecluster show default ``` -Look for the `CLUSTER TEMPLATE` section. If `EVC_MODE` is not present, it has not been configured for the cluster. +Look for the `CLUSTER TEMPLATE` section. If `EVC_MODE` is not present, it has not been configured for the Cluster. -2. Set the `EVC_MODE` attribute on a cluster using `onecluster update`. For example, to set a Sandy Bridge baseline: +2. Set the `EVC_MODE` attribute on a Cluster using `onecluster update`. For example, to set a Sandy Bridge baseline: -```bash -$ onecluster update default" +```shell +onecluster update default" ``` Then add the `EVC_MODE` attribute to the list: -```bash +```shell ... EVC_MODE="sandybridge" ... ``` -The exact CPU model string depends on the hypervisor's supported CPU map. You can view the supported cpu models of a given host with the following command under the `KVM_CPU_MODELS` key: +The exact CPU model string depends on the hypervisor's supported CPU map. You can view the supported cpu models of a given Host with the following command under the `KVM_CPU_MODELS` key: -```bash -$ onehost show -j +```shell +onehost show -j ``` -Make sure to select a cpu model available in all hosts in the cluster, otherwise you may fail to deploy VMs on unsupported hosts. +Make sure to select a cpu model available in all Hosts in the Cluster, otherwise you may fail to deploy VMs on unsupported Hosts. -3. To revert or remove EVC, update the cluster template to remove the `EVC_MODE` attribute (for example by setting it to an empty string or re-applying a template without the attribute). +3. To revert or remove EVC, update the Cluster template to remove the `EVC_MODE` attribute (for example by setting it to an empty string or re-applying a template without the attribute). ### Using EVC with Sunstone The Fireedge / Sunstone web UI provides a convenient way to enable and change EVC without editing templates directly. -1. Open the Infrastructure → Clusters view and select the cluster you want to configure. +1. Open the Infrastructure → Clusters view and select the Cluster you want to configure. 2. Click the Update button and go to the Select Hosts tabs, there you will see the EVC Mode section. To enable EVC, choose a model from the droplist. Afterwards you can click Finish. -![Update cluster EVC mode](/images/sunstone_cluster_evc_update.png) +{{< image path="/images/hosts_and_clusters/sunstone_cluster_evc_update.png" alt="Update cluster EVC mode" align="center" width="90%" mb="20px" +>}} -3. Once the EVC mode is set, you can see it listed in the cluster attributes the same way as if the CLI had been used. +3. Once the EVC mode is set, you can see it listed in the Cluster attributes the same way as if the CLI had been used. -![EVC set in cluster attributes](/images/sunstone_cluster_evc_attributes.png) +{{< image path="/images/hosts_and_clusters/sunstone_cluster_evc_attributes.png" alt="EVC set in cluster attributes" align="center" width="90%" mb="20px" +>}} \ No newline at end of file diff --git a/content/product/cluster_configuration/hosts_and_clusters/hosts.md b/content/product/cluster_configuration/hosts_and_clusters/hosts.md index 5dd5f6251..3dc0ce3a0 100644 --- a/content/product/cluster_configuration/hosts_and_clusters/hosts.md +++ b/content/product/cluster_configuration/hosts_and_clusters/hosts.md @@ -1,5 +1,5 @@ --- -title: "Hosts" +title: "Host Management" date: "2025-02-17" description: categories: @@ -12,9 +12,9 @@ weight: "2" - +In order to use a physical or virtual server as a Host, you must first prepare the server with a hypervisor and configure it to be added to OpenNebula as a Host, refer to the guides to prepare a [KVM node]({{% relref "software/installation_process/cluster_installation/kvm_node_installation/" %}}) or an [LXC node]({{% relref "software/installation_process/cluster_installation/lxc_node_installation/" %}}). -In order to use your existing physical nodes, you have to add them to OpenNebula as Hosts. To add a Host only its hostname and type is needed. +Once you have prepared your physical or virtual nodes, you can add them to OpenNebula as Hosts. To add a Host only its hostname and type is needed. {{< alert title="Warning" type="warning" >}} Before adding a Linux Host check that you can SSH to it without being prompted for a password.{{< /alert >}} @@ -25,8 +25,8 @@ Hosts are the servers managed by OpenNebula responsible for running VMs. To use Creating a Host: -```default -$ onehost create host01 --im kvm --vm kvm +```shell +onehost create host01 --im kvm --vm kvm ID: 0 ``` @@ -40,17 +40,17 @@ In the examples included in this guide we’ll use KVM as the hypervisor. Note t To remove a Host you can either specify it by ID or by name. The following commands are equivalent: -```default -$ onehost delete host01 -$ onehost delete 0 +```shell +onehost delete host01 +onehost delete 0 ``` ## Showing and Listing Hosts To display information about a single Host, use the `show` command: -```default -$ onehost show server +```shell +onehost show server HOST 0 INFORMATION ID : 0 @@ -131,8 +131,8 @@ The information of a Host contains: To see a list of all the Hosts: -```default -$ onehost list +```shell +onehost list ID NAME CLUSTER RVM ALLOCATED_CPU ALLOCATED_MEM STAT 0 server server 1 100 / 400 (25%) 1024M / 7.3G (13%) on 1 kvm1 kvm 0 - - off @@ -204,20 +204,20 @@ In order to manage the life cycle of a Host it can be set to different operation The `onehost` tool provides commands to set the operation mode of a Host: `disable`, `offline`, and `enable`, for example: -```default -$ onehost disable 0 +```shell +onehost disable 0 ``` To re-enable the Host, use the `enable` command: -```default -$ onehost enable 0 +```shell +onehost enable 0 ``` Similarly, to take the Host offline: -```default -$ onehost offline 0 +```shell +onehost offline 0 ``` {{< alert title="Note" type="info" >}} @@ -227,13 +227,13 @@ Apart from the commands above, the `onehost` tool also provides some commands th You can use `forceupdate` subcommand to reset the monitoring process on the Host: -```default -$ onehost forceupdate 0 +```shell +onehost forceupdate 0 ``` The `flush` command will migrate all the active VMs in the specified Host to another server with enough capacity. At the same time, the specified Host will be disabled so no more Virtual Machines are deployed in it. This command is useful to clean a Host of active VMs. The migration process can be done by a `resched` action or by a recover `delete-recreate` action; it can be configured in `/etc/one/cli/onehost.yaml` by setting the field `default_actions\flush` to `delete-recreate` or to `resched`. Here is an example: -```default +```shell :default_actions: - :flush: delete-recreate ``` @@ -246,8 +246,8 @@ You can add custom attributes either by [creating a probe in the host]({{% relre For example, to label a Host as *production* we can add a custom tag *TYPE*: -```default -$ onehost update +```shell +onehost update ... TYPE="production" ``` @@ -260,8 +260,8 @@ This tag can be used at a later time for scheduling purposes, [see more details When OpenNebula monitors a Host it copies driver files to `/var/tmp/one`. When these files are updated they need to be copied again to the Hosts with the `sync` command. To keep track of the probes version there’s a file in `/var/lib/one/remotes/VERSION`. By default this holds the OpenNebula version (e.g., ‘7.0.0’). This version can be seen in the Hosts by using `onehost show `: -```default -$ onehost show 0 +```shell +onehost show 0 HOST 0 INFORMATION ID : 0 [...] @@ -274,15 +274,15 @@ The command `onehost sync` only updates the Hosts with `VERSION` lower than the In case you want to force an upgrade, that is, without any `VERSION` checking, you can do it by using the `--force` option: -```default -$ onehost sync --force +```shell +onehost sync --force ``` -You can also select which Hosts you want to upgrade by naming them or selecting a cluster: +You can also select which Hosts you want to upgrade by naming them or selecting a Cluster: -```default -$ onehost sync host01,host02,host03 -$ onehost sync -c myCluster +```shell +onehost sync host01,host02,host03 +onehost sync -c myCluster ``` @@ -332,8 +332,8 @@ The cache is cleared when the monitoring daemon restarts. The monitoring mechanism in OpenNebula reports all VMs found in a hypervisor, even those not launched through OpenNebula. These VMs are referred to as Wild VMs. The Wild VMs can be spotted through the `onehost show` command: -```default -$ onehost show 3 +```shell +onehost show 3 HOST 3 INFORMATION ID : 3 NAME : MyAWSHost @@ -353,8 +353,10 @@ Wild VMs’ support and limitations may differ depending on the virtualization d You can also manage your Hosts using [Sunstone UI Interface]({{% relref "../../control_plane_configuration/graphical_user_interface/fireedge_sunstone#fireedge-sunstone" %}}). Select the Host tab and there you will be able to create, enable, disable, delete, and see information about your Hosts in a user-friendly way. -![image1](/images/hosts_fireedge.png) +{{< image path="/images/hosts_and_clusters/hosts_fireedge.png" alt="Sunstone login" align="center" width="90%" mb="20px" +>}} -- Create new hosts +- Create new Hosts: -![image2](/images/hosts_create.png) +{{< image path="/images/hosts_and_clusters/hosts_create.png" alt="Sunstone login" align="center" width="90%" mb="20px" +>}} diff --git a/content/product/cluster_configuration/hosts_and_clusters/numa.md b/content/product/cluster_configuration/hosts_and_clusters/numa.md index 6d6531cea..893f26661 100644 --- a/content/product/cluster_configuration/hosts_and_clusters/numa.md +++ b/content/product/cluster_configuration/hosts_and_clusters/numa.md @@ -16,9 +16,9 @@ weight: "4" In this guide you’ll learn to set up OpenNebula to control how VM resources are mapped onto the hypervisor ones. These settings will help you to fine tune the performance of VMs. We will use the following concepts: -* **Cores, Threads, and Sockets**. A computer processor is connected to the motherboard through a *socket*. A processor can pack one or more *cores*, each one implements a separate processing unit that shares some cache levels, memory, and I/O ports. CPU cores’ performance can be improved by the use of hardware *multi-threading* (SMT) that permits multiple execution flows to run simultaneously on a single core. -* **NUMA**. Multi-processor servers are usually arranged in nodes or cells. Each *NUMA node* holds a fraction of the overall system memory. In this configuration, a processor accesses memory and I/O ports local to its node faster than those of non-local ones. -* **Hugepages**. Systems with big physical memory also use a large number of virtual memory pages. This big number makes the use of virtual-to-physical translation caches inefficient. Hugepages reduces the number of virtual pages in the system and optimizes the virtual memory subsystem. +* **Cores, Threads, and Sockets**: A computer processor is connected to the motherboard through a *socket*. A processor can pack one or more *cores*, each one implements a separate processing unit that shares some cache levels, memory, and I/O ports. CPU cores’ performance can be improved by the use of hardware *multi-threading* (SMT) that permits multiple execution flows to run simultaneously on a single core. +* **NUMA**: Multi-processor servers are usually arranged in nodes or cells. Each *NUMA node* holds a fraction of the overall system memory. In this configuration, a processor accesses memory and I/O ports local to its node faster than those of non-local ones. +* **Hugepages**: Systems with big physical memory also use a large number of virtual memory pages. This big number makes the use of virtual-to-physical translation caches inefficient. Hugepages reduces the number of virtual pages in the system and optimizes the virtual memory subsystem. In OpenNebula the virtual topology of a VM is defined by the number of sockets, cores, and threads. We assume that a NUMA node or cell is equivalent to a socket and they will be used interchangeably in this guide. diff --git a/content/product/cluster_configuration/hosts_and_clusters/nvidia_gpu_passthrough.md b/content/product/cluster_configuration/hosts_and_clusters/nvidia_gpu_passthrough.md deleted file mode 100644 index 47c405150..000000000 --- a/content/product/cluster_configuration/hosts_and_clusters/nvidia_gpu_passthrough.md +++ /dev/null @@ -1,278 +0,0 @@ ---- -title: "NVIDIA GPU Passthrough" -date: "2025-10-16" -description: -categories: -pageintoc: "58" -tags: ['AI','NVIDIA'] -weight: "7" ---- - - - - - -Here you will find detailed instructions for configuring PCI passthrough for high-performance NVIDIA® GPUs on x86_64 hypervisors. The procedures described here have been validated with NVIDIA H100 GPUs but can be adapted for other similar high-performance NVIDIA GPUs. This allows Virtual Machines to get exclusive access to the GPU, which is recommended for AI/ML workloads. It builds upon the concepts explained in the general [PCI Passthrough]({{% relref "pci_passthrough.md" %}}) guide. This guide is not applicable to ARM-based systems. - -{{< alert title="Note" type="info" >}} -As an alternative to PCI passthrough, OpenNebula also supports NVIDIA vGPU, which allows a single physical GPU to be partitioned and shared among multiple VMs. While PCI passthrough is the recommended approach for demanding AI/ML workloads, especially model training, vGPU can be a suitable option for less intensive tasks like model inference or development environments. For more information, see the [NVIDIA vGPU guide]({{% relref "vgpu.md" %}}). -{{< /alert >}} - -## Hypervisor Configuration - -The following configurations must be performed on the KVM hypervisors that will host the GPUs. - -### Enabling IOMMU - -The IOMMU (Input-Output Memory Management Unit) is a hardware feature that allows the hypervisor to isolate I/O devices and is a prerequisite for PCI passthrough. To enable it, you need to add kernel parameters to the hypervisor's boot configuration. - -1. Edit the GRUB configuration file at `/etc/default/grub` and add the appropriate parameter to the `GRUB_CMDLINE_LINUX_DEFAULT` line: - -* **For Intel CPUs**: `intel_iommu=on iommu=pt` -* **For AMD CPUs**: `amd_iommu=on iommu=pt` - -Example for an Intel-based hypervisor: - -```default -GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt" -``` - -2. After saving the file, update the GRUB configuration and reboot the hypervisor: - -```default -# update-grub -# reboot -``` - -3. After rebooting, verify that IOMMU is enabled by checking the kernel messages: - -```default -# dmesg | grep -i iommu -``` - -Alternatively, to confirm that IOMMU is active proceed to check that the `/sys/kernel/iommu_groups/` directory exists and is populated with subdirectories. An empty directory might indicate that IOMMU is not correctly enabled in the kernel or BIOS. - -### VFIO Driver Binding - -Once IOMMU is enabled, the GPU must be unbound from the default host driver (e.g., `nouveau`) and bound to the `vfio-pci` driver, which is designed for PCI passthrough. - -1. Install `driverctl` utility: - - ```default - # apt install driverctl - ``` - -2. Ensure `vfio-pci` module is loaded on boot: - - ```default - # echo "vfio-pci" | sudo tee /etc/modules-load.d/vfio-pci.conf - # modprobe vfio-pci - ``` - -3. Identify the GPU's PCI address: - - ```default - # lspci -D | grep -i nvidia - 0000:e1:00.0 3D controller: NVIDIA Corporation GH100 [H100 PCIe] (rev a1) - ``` - -4. Set the driver override: - Use the PCI address from the previous step to set an override for the device to use the `vfio-pci` driver. - - ```default - # driverctl set-override 0000:e1:00.0 vfio-pci - ``` - -5. Verify the driver binding: - Check that the GPU is now using the `vfio-pci` driver. - - ```default - # lspci -Dnns e1:00.0 -k - Kernel driver in use: vfio-pci - ``` - -### VFIO Device Ownership - -For OpenNebula to manage the GPU, the VFIO device files in `/dev/vfio/` must be owned by the `root:kvm` user and group. This is achieved by creating a `udev` rule. - -1. Identify the IOMMU group for your GPU using its PCI address: - - ```default - # find /sys/kernel/iommu_groups/ -type l | grep e1:00.0 - /sys/kernel/iommu_groups/85/devices/0000:e1:00.0 - ``` - In this example, the IOMMU group is `85`. - -2. Create a `udev` rule: - Create the file `/etc/udev/rules.d/99-vfio.rules` with the following content: - - ```default - SUBSYSTEM=="vfio", GROUP="kvm", MODE="0666" - ``` - -3. Reload `udev` rules: - - ```default - # udevadm control --reload - # udevadm trigger - ``` - -4. Verify ownership: - Check the ownership of the device file corresponding to your GPU's IOMMU group. - - ```default - # ls -la /dev/vfio/ - crw-rw-rw- 1 root kvm 509, 0 Oct 16 10:00 85 - ``` - -## OpenNebula Configuration - -### Monitoring PCI Devices - -To make the GPUs available in OpenNebula, configure the PCI probe on the front-end node to monitor NVIDIA devices. - -1. Edit the PCI probe configuration file at `/var/lib/one/remotes/etc/im/kvm-probes.d/pci.conf`. -2. Add a filter for NVIDIA devices: - - ```default - :filter: '10de:*' - ``` - -3. Synchronize the hosts from the Front-end to apply the new configuration: - - ```default - # su - oneadmin - $ onehost sync -f - ``` - -After a few moments, you can check if the GPU is being monitored correctly by showing the host information (`onehost show `). The GPU should appear in the `PCI DEVICES` section. - -By default, the host system monitoring probe may take up to **10 minutes** to detect new PCI devices. To run the host monitoring probe immediately, you can force it with: -```default -$ onehost forceupdate -``` - -## VM Template Best Practices for AI Workloads - -To achieve proper GPU operation inside the VM, including optimal performance for AI/ML workloads, configure the VM Template with the following settings: - -### Firmware (UEFI) - -Modern high-performance GPUs like the H100 benefit from Resizable BAR, which requires UEFI booting. OpenNebula will automatically select the appropriate UEFI firmware file. If you need to use a specific OVMF file, you can provide its full path in the `FIRMWARE` attribute instead of `\"UEFI\"`. - -* **Attribute**: `FIRMWARE` -* **Section**: `OS` -* **Value**: `"UEFI"` - -```default -OS = [ - ARCH = "x86_64", - FIRMWARE = "UEFI" -] -``` - -### Machine Type (q35) - -The `q35` machine type provides a modern PCIe-based chipset, improving compatibility and performance for PCIe devices like GPUs. Additionally, if the q35 machine type is used, OpenNebula replicates the hypervisor PCI topology into the guest VM. This gives AI frameworks visibility into the PCI layout, allowing them to optimize data transfers and reduce latency. - -* **Attribute**: `MACHINE` -* **Section**: `OS` -* **Value**: `"q35"` - -```default -OS = [ - ..., - MACHINE = "q35" -] -``` - -### CPU Model (Host Passthrough) - -Exposing the host CPU exact model and feature set to the VM results in better hardware support and performance. Having the full list of CPU features is a hard requirement for modern high-performance GPUs like the NVIDIA H100, because its drivers require access to advanced CPU instruction sets to initialize and function correctly. - -* **Attribute**: `CPU_MODEL` -* **Value**: `MODEL="host-passthrough"` - -```default -CPU_MODEL = [ - MODEL = "host-passthrough" -] -``` - -### NUMA and CPU Pinning - -CPU pinning is crucial for performance as it dedicates physical CPU cores to the VM, preventing them from being shared with other VMs. When a `PIN_POLICY` is defined, the OpenNebula scheduler automatically places the VM on a NUMA node that is physically close to the requested GPU and pins the vCPUs to the cores of that node, minimizing latency. For a more in-depth explanation of this topic, refer to the [Virtual Topology and CPU Pinning]({{% relref "numa.md" %}}) guide. - -For VM pinning to work, you must also enable the pinning policy on the Host. You can do this by editing the Host's template and setting `PIN_POLICY="PINNED"`, or by selecting the `PINNED` policy in the Host's `NUMA` tab in Sunstone. - -* **Attribute**: `TOPOLOGY` -* **Values**: `PIN_POLICY`, `CORES`, `SOCKETS` - -```default -TOPOLOGY = [ - PIN_POLICY = "CORE", - CORES = "16", - SOCKETS = "1" -] -``` - -### PCI Device Passthrough - -Finally, assign the GPU to the VM using the `PCI` attribute. You can select a specific GPU by its address. Once the VM is running, verify that the guest OS recognizes the device by executing the `lspci` command inside the VM. - -* **Attribute**: `PCI` -* **Value**: `SHORT_ADDRESS=""` - -```default -PCI = [ - SHORT_ADDRESS = "e1:00.0" -] -``` - -## Sample VM Template - -Here is a sample VM Template that incorporates all the recommended attributes for an AI workload VM with an NVIDIA H100 GPU. This template is a reference. To meet your specific requirements, specify `MEMORY`, `VCPU`, `IMAGE_ID`, and other parameters. - -```default -CONTEXT=[ - NETWORK="YES", - SSH_PUBLIC_KEY="$USER[SSH_PUBLIC_KEY]" ] - -CPU="1" - -CPU_MODEL=[ - MODEL="host-passthrough" ] - -DISK=[ - IMAGE_ID="163" ] - -GRAPHICS=[ - LISTEN="0.0.0.0", - TYPE="vnc" ] - -HYPERVISOR="kvm" - -LOGO="images/logos/ubuntu.png" - -LXD_SECURITY_PRIVILEGED="true" - -MEMORY="768" - -OS=[ - ARCH="x86_64", - FIRMWARE="UEFI", - MACHINE="q35" ] - -PCI=[ - SHORT_ADDRESS="e1:00.0" ] - -SCHED_REQUIREMENTS="HYPERVISOR=kvm" - -TOPOLOGY=[ - CORES="4", - PIN_POLICY="CORE", - SOCKETS="1", - THREADS="1" ] - -VCPU="4" -``` \ No newline at end of file diff --git a/content/product/cluster_configuration/hosts_and_clusters/overview.md b/content/product/cluster_configuration/hosts_and_clusters/overview.md index 3808ea566..fe711c94b 100644 --- a/content/product/cluster_configuration/hosts_and_clusters/overview.md +++ b/content/product/cluster_configuration/hosts_and_clusters/overview.md @@ -12,16 +12,28 @@ weight: "1" -A **Host** is a server that has the ability to run Virtual Machines and that is connected to OpenNebula’s Front-end server. To learn how to prepare the Hosts, you can read the [Installation]({{% relref "../../../software/installation_process" %}}) guide. Hosts are usually grouped in **Clusters**. +A **Host** is a server (physical or virtual) that provides compute capacity for cloud workloads, such as Virtual Machines or Kubernetes Clusters. Hosts must be connected to and controlled by one or more OpenNebula Front-end servers. To learn how to prepare Hosts, read the [Cluster section of the installation guides]({{% relref "software/installation_process/cluster_installation/" %}}). A **Cluster** is a logical grouping of OpenNebula resources including Hosts, Virtual Networks and datastores. Hosts are usually logically grouped within **Clusters**, but may exist independently of a Cluster object. ## How Should I Read This Chapter In this chapter there are four guides describing these objects. -* **Host Management**: Host management is achieved through the `onehost` CLI command or through the Sunstone GUI. You can read about Host Management in more detail in the [Hosts]({{% relref "hosts#hosts-guide" %}}) guide. -* **Cluster Management**: Hosts can be grouped in clusters. These clusters are managed with the `onecluster` CLI command or through the Sunstone GUI. You can read about Cluster Management in more detail in the [Clusters]({{% relref "cluster_guide#cluster-guide" %}}) guide. +* **Host Management**: Host management is achieved through the `onehost` CLI command or through the **Infrastructure** section of the Sunstone GUI. You can read about Host Management in more detail in the [Host Management]({{% relref "hosts#hosts-guide" %}}) guide. +* **Cluster Management**: Hosts can be grouped in Clusters. These Clusters are managed with the `onecluster` CLI command or through the **Infrastructure** section of the Sunstone GUI. You can read about Cluster Management in more detail in the [Cluster Management]({{% relref "cluster_guide#cluster-guide" %}}) guide. -You should read all the guides in this chapter to familiarize yourself with these objects. For small and homogeneous clouds you may not need to create new clusters. +You should read all the guides in this chapter to familiarize yourself with the details of Host and Cluster objects. For small and homogeneous clouds you may not need to create new Clusters. + +### PCI Passthrough and SR-IOV + +If you intend to use specialized PCI hardware, such as GPUs or PCI network interfaces, it is necessary to complete additional configuration to ensure that workloads deployed within OpenNebula Hosts or Clusters have proper access to the devices. Refer to the [PCI Passthrough and SR-IOV]({{% relref "product/cluster_configuration/pci_passthrough_sriov/" %}}) section for more details. + +### Networking + +Virtual Machines deployed by OpenNebula on Hosts and Clusters within a cloud deployment must be connected to a Virtual Network to facilitate communication within the cloud and to the internet. Refer to the [Networking System Section]({{% relref "product/cluster_configuration/networking_system/" %}}) for details on configuring Virtual Networks within an OpenNebula cloud. + +### Storage and Backups + +OpenNebula supports multiple options for cloud storage, including Ceph storage and SAN storage and backup solutions from multiple vendors. Refer to the [Storage System Guides]({{% relref "product/cluster_configuration/storage_system/" %}}) and the [Backup System Section]({{% relref "product/cluster_configuration/backup_system/" %}}) for further details. ## Hypervisor Compatibility diff --git a/content/product/cluster_configuration/networking_system/bridged.md b/content/product/cluster_configuration/networking_system/bridged.md index 827877092..343c9faf9 100644 --- a/content/product/cluster_configuration/networking_system/bridged.md +++ b/content/product/cluster_configuration/networking_system/bridged.md @@ -5,7 +5,7 @@ description: categories: pageintoc: "62" tags: -weight: "4" +weight: "3" --- diff --git a/content/product/cluster_configuration/networking_system/infiniband.md b/content/product/cluster_configuration/networking_system/infiniband.md new file mode 100644 index 000000000..59ccd9ffa --- /dev/null +++ b/content/product/cluster_configuration/networking_system/infiniband.md @@ -0,0 +1,113 @@ +--- +title: "InfiniBand" +date: "2026-07-10" +categories: ["networking"] +pageintoc: "64" +tags: ["networking"] +weight: "7" +--- + +InfiniBand is a high-performance networking technology designed for environments where extremely low latency, high throughput, and efficient CPU utilization are critical. It is widely used in High Performance Computing (HPC), AI training and inference Clusters, and other data-intensive workloads that require significantly higher performance than conventional Ethernet networks. InfiniBand achieves this through technologies such as Remote Direct Memory Access (RDMA), allowing applications to exchange data directly between systems with minimal CPU involvement. + +OpenNebula supports deployment on infrastructure equipped with InfiniBand fabrics, enabling virtualized and bare-metal workloads to take advantage of high-speed interconnects. Depending on the workload requirements, InfiniBand can be used in several different ways: + +* As an IP network using IP over InfiniBand (IPoIB). +* By exposing InfiniBand adapters directly to Virtual Machines using PCI Passthrough or SR-IOV Virtual Functions (VFs). +* For storage and Cluster communication over high-speed fabrics. +* For applications that use RDMA directly. + +## Typical Use Cases + +InfiniBand is most commonly deployed for workloads such as: + +* AI and machine learning training Clusters +* High Performance Computing (HPC) +* Scientific simulations +* High-performance distributed storage +* MPI-based parallel applications +* GPU Clusters requiring low-latency node-to-node communication + +These environments benefit from InfiniBand's low latency, high bandwidth, and efficient RDMA transport. + +## Deployment Models + +OpenNebula does not implement a dedicated InfiniBand networking driver. Instead, InfiniBand is integrated using the standard Linux networking stack and OpenNebula's existing networking and PCI device management capabilities. + +## IP over InfiniBand (IPoIB) + +The simplest deployment method is IP over InfiniBand (IPoIB), where the InfiniBand fabric behaves as a conventional IP network. + +Once the Host operating system has configured the InfiniBand interfaces, they appear as standard Linux network interfaces and can be used throughout OpenNebula just like Ethernet interfaces. + +Typical uses include: + +* Front-end and back-end Cluster communication +* Storage traffic +* Virtual Network bridges +* VXLAN or VLAN transport over the InfiniBand fabric (where supported by the underlying infrastructure) + +From OpenNebula's perspective, no special configuration is required beyond using the appropriate bridge or physical interface when defining Virtual Networks. + +## PCI Passthrough + +Applications that require native InfiniBand capabilities or RDMA typically require direct access to the InfiniBand Host Channel Adapter (HCA). + +OpenNebula supports assigning InfiniBand adapters directly to Virtual Machines using PCI Passthrough. This provides the guest operating system with direct hardware access, allowing it to install the appropriate InfiniBand drivers and use the adapter as if running on bare metal. + +Configuration follows the standard PCI Passthrough workflow described in the PCI Passthrough documentation. + +## SR-IOV Virtual Functions + +Many modern InfiniBand adapters support Single Root I/O Virtualization (SR-IOV), allowing multiple Virtual Functions (VFs) to be created from a single physical adapter. + +OpenNebula can schedule and assign these Virtual Functions directly to Virtual Machines, providing: + +* Near-native networking performance +* RDMA support +* Lower overhead than emulated virtual NICs +* Efficient sharing of high-performance network adapters + +When using SR-IOV network devices, OpenNebula can configure attributes such as: + +* MAC address +* VLAN ID +* Spoof checking +* Trust mode + +depending on whether the adapter operates in Legacy or Switchdev mode. + +## Host Requirements + +Before deploying workloads using InfiniBand, ensure that each Host is correctly configured by the operating system. + +Typical requirements include: + +* Supported InfiniBand Host Channel Adapters (HCAs) +* Vendor drivers and firmware +* RDMA software stack (for example, RDMA Core or vendor-provided drivers) +* A functioning InfiniBand fabric +* An operational Subnet Manager (such as OpenSM), which is required for InfiniBand fabrics to function correctly +* Appropriate kernel modules and IPoIB configuration if using IP networking over InfiniBand + +OpenNebula assumes that the InfiniBand infrastructure has already been configured and validated by the operating system. + +## Guest Requirements + +If Virtual Machines require native InfiniBand functionality, the guest operating system must include: + +* InfiniBand drivers +* RDMA libraries (when required) +* Appropriate user-space software for the intended workload + +Guests using PCI Passthrough or SR-IOV interact directly with the assigned hardware and are responsible for managing the device. + +### Considerations + +When planning an InfiniBand deployment, consider the following: + +PCI Passthrough and SR-IOV provide the highest performance but may reduce Virtual Machine mobility, including live migration support, depending on the hardware and deployment configuration. + +IPoIB offers excellent compatibility with existing networking tools but does not expose all RDMA capabilities to applications. +Applications must be specifically designed or configured to use RDMA in order to benefit from InfiniBand's lowest latency communication. + +Network configuration, firmware management, and Subnet Manager deployment remain the responsibility of the underlying operating system and network infrastructure. \ No newline at end of file diff --git a/content/product/cluster_configuration/networking_system/nvidia_bluefield_dpu.md b/content/product/cluster_configuration/networking_system/nvidia_bluefield_dpu.md new file mode 100644 index 000000000..a1b936ac8 --- /dev/null +++ b/content/product/cluster_configuration/networking_system/nvidia_bluefield_dpu.md @@ -0,0 +1,11 @@ +--- +title: "NVIDIA BlueField DPU (AE)" +linkTitle: "NVIDIA BlueField DPU (AE)" +date: "2026-06-30" +categories: ["networking"] +pageintoc: "64" +tags: ["networking", "hpc", "ai", "nvidia", "spectrum-x", "evpn"] +weight: "9" +--- + +Work in progress. \ No newline at end of file diff --git a/content/product/cluster_configuration/networking_system/openvswitch.md b/content/product/cluster_configuration/networking_system/openvswitch.md index 257f1df62..83bbd78f6 100644 --- a/content/product/cluster_configuration/networking_system/openvswitch.md +++ b/content/product/cluster_configuration/networking_system/openvswitch.md @@ -6,7 +6,7 @@ description: categories: pageintoc: "65" tags: -weight: "7" +weight: "6" --- diff --git a/content/product/cluster_configuration/networking_system/overview.md b/content/product/cluster_configuration/networking_system/overview.md index 91d8c96cb..ffa01cc40 100644 --- a/content/product/cluster_configuration/networking_system/overview.md +++ b/content/product/cluster_configuration/networking_system/overview.md @@ -16,17 +16,29 @@ When a new Virtual Machine is launched, OpenNebula will connect its virtual netw OpenNebula supports the following networking modes: -* [Bridged]({{% relref "bridged#bridged" %}}). The VM NIC is added to a Linux bridge on the Host. This mode can be configured to use Security Groups and network isolation. -* [802.1Q VLAN]({{% relref "vlan#hm-vlan" %}}). The VM NIC is added to a Linux bridge on the Host and the Virtual Network is configured to handle 802.1Q VLAN isolation. -* [VXLAN]({{% relref "vxlan#vxlan" %}}). The VM NIC is added to a Linux bridge on the Host and the Virtual Network implements isolation using the VXLAN encapsulation. -* [Open vSwitch]({{% relref "openvswitch#openvswitch" %}}). The VM NIC is added to a Open vSwitch bridge on the Host and the Virtual Network optionally handles 802.1Q VLAN isolation. -* [Open vSwitch on VXLAN]({{% relref "openvswitch#openvswitch-vxlan" %}}). The VM NIC is added to a Open vSwitch bridge on the Host and the Virtual Network is configured to provide both isolation with VXLAN encapsulation and optionally 802.1Q VLAN. +* [Bridged]({{% relref "bridged#bridged" %}}): The VM NIC is added to a Linux bridge on the Host. This mode can be configured to use Security Groups and network isolation. +* [802.1Q VLAN]({{% relref "vlan#hm-vlan" %}}): The VM NIC is added to a Linux bridge on the Host and the Virtual Network is configured to handle 802.1Q VLAN isolation. +* [VXLAN]({{% relref "vxlan#vxlan" %}}): The VM NIC is added to a Linux bridge on the Host and the Virtual Network implements isolation using the VXLAN encapsulation. +* [Open vSwitch]({{% relref "openvswitch#openvswitch" %}}): The VM NIC is added to a Open vSwitch bridge on the Host and the Virtual Network optionally handles 802.1Q VLAN isolation. +* [Open vSwitch on VXLAN]({{% relref "openvswitch#openvswitch-vxlan" %}}): The VM NIC is added to a Open vSwitch bridge on the Host and the Virtual Network is configured to provide both isolation with VXLAN encapsulation and optionally 802.1Q VLAN. The attribute `VN_MAD` of a Virtual Network determines which of the above networking modes is used. {{< alert title="Note" type="info" >}} Security Groups are not supported in the Open vSwitch modes.{{< /alert >}} +## Accelerated Networking + +OpenNebula supports accelerated networking and interconnect technologies for AI and HPC environments. These technologies complement the standard Virtual Network modes by providing high-bandwidth, low-latency communication, direct device access, hardware offload, and optimized communication between GPUs and compute nodes. + +* [InfiniBand]({{% relref "product/cluster_configuration/networking_system/infiniband/" %}}): Provides a high-throughput, low-latency network fabric for distributed AI and HPC workloads. InfiniBand devices can be exposed to Virtual Machines through mechanisms such as PCI passthrough or SR-IOV. + +* [NVIDIA Spectrum-X]({{% relref "product/cluster_configuration/networking_system/spectrumx/" %}}): Provides an accelerated Ethernet fabric optimized for AI workloads, combining NVIDIA Spectrum switches and high-performance network adapters to improve communication between compute nodes. + +* [NVIDIA BlueField DPU]({{% relref "product/cluster_configuration/networking_system/nvidia_bluefield_dpu/" %}}): Offloads networking, security, encryption, and infrastructure services from the Host CPU. BlueField devices can be integrated into the infrastructure to provide accelerated and isolated data paths for tenant workloads. + +* [NVIDIA Fabric Manager]({{% relref "product/cluster_configuration/pci_passthrough_sriov/one_fabricmanager/" %}}): Manages and monitors the NVSwitch fabric used for high-speed communication between multiple NVIDIA GPUs within a Host. It initializes the fabric and coordinates GPU connectivity for supported multi-GPU systems. + ## How Should I Read This Chapter Before reading this Chapter make sure you are familiar with the [Open Cloud Storage]({{% relref "../storage_system/overview#storage" %}}). It’s necessary to be aware of requirements for your selected storage solution in order to be able to design the network architecture of your hypervisor nodes. diff --git a/content/product/cluster_configuration/networking_system/vlan.md b/content/product/cluster_configuration/networking_system/vlan.md index f98dff442..778b15210 100644 --- a/content/product/cluster_configuration/networking_system/vlan.md +++ b/content/product/cluster_configuration/networking_system/vlan.md @@ -6,7 +6,7 @@ description: categories: pageintoc: "63" tags: -weight: "5" +weight: "4" --- diff --git a/content/product/cluster_configuration/networking_system/vxlan.md b/content/product/cluster_configuration/networking_system/vxlan.md index b1eff7f8b..c07f7507a 100644 --- a/content/product/cluster_configuration/networking_system/vxlan.md +++ b/content/product/cluster_configuration/networking_system/vxlan.md @@ -6,7 +6,7 @@ description: categories: pageintoc: "64" tags: -weight: "6" +weight: "5" --- diff --git a/content/product/cluster_configuration/pci_passthrough_sriov/_index.md b/content/product/cluster_configuration/pci_passthrough_sriov/_index.md new file mode 100644 index 000000000..cfb92749d --- /dev/null +++ b/content/product/cluster_configuration/pci_passthrough_sriov/_index.md @@ -0,0 +1,9 @@ +--- +title: "PCI Passthrough and SR-IOV" +date: "2026-06-30" +description: "Enable VMs to access PCI devices directly, including GPUs, high-performance network adapters and storage controllers." +categories: +pageintoc: "58" +tags: +weight: "2" +--- \ No newline at end of file diff --git a/content/product/cluster_configuration/pci_passthrough_sriov/amd_gpu_passthrough.md b/content/product/cluster_configuration/pci_passthrough_sriov/amd_gpu_passthrough.md new file mode 100644 index 000000000..3d4ecce17 --- /dev/null +++ b/content/product/cluster_configuration/pci_passthrough_sriov/amd_gpu_passthrough.md @@ -0,0 +1,12 @@ +--- +title: "AMD GPU Passthrough" +linkTitle: "AMD GPUs" +date: "2026-06-30" +description: +categories: +pageintoc: "58" +tags: ['AI','NVIDIA'] +weight: "6" +--- + +Work in progress. \ No newline at end of file diff --git a/content/product/cluster_configuration/pci_passthrough_sriov/axelera_gpu_passthrough.md b/content/product/cluster_configuration/pci_passthrough_sriov/axelera_gpu_passthrough.md new file mode 100644 index 000000000..c3ad65ae3 --- /dev/null +++ b/content/product/cluster_configuration/pci_passthrough_sriov/axelera_gpu_passthrough.md @@ -0,0 +1,10 @@ +--- +title: "Axelera GPU Passthrough" +linkTitle: "Axelera GPUs" +date: "2025-10-16" +description: +categories: +pageintoc: "58" +tags: ['AI','NVIDIA'] +weight: "7" +--- \ No newline at end of file diff --git a/content/product/cluster_configuration/pci_passthrough_sriov/host_configuration.md b/content/product/cluster_configuration/pci_passthrough_sriov/host_configuration.md new file mode 100644 index 000000000..e40850d38 --- /dev/null +++ b/content/product/cluster_configuration/pci_passthrough_sriov/host_configuration.md @@ -0,0 +1,441 @@ +--- +title: "Host Configuration" +date: "2026-06-30" +description: +categories: +pageintoc: "58" +tags: ['AI','NVIDIA'] +weight: "2" +--- + +This guide describes the common host configuration required to enable PCI passthrough in OpenNebula. These requirements apply independently of the type of PCI device being assigned to virtual machines. Device-specific configuration is covered in the corresponding guides. +The guide assumes a Linux host running the OpenNebula KVM hypervisor. While the examples use common Linux utilities and a GRUB-based bootloader, equivalent procedures can be used with other Linux distributions and bootloaders. + +## Prerequisites + +Before configuring PCI passthrough, verify that the host satisfies the following requirements: +* Hardware-assisted virtualization enabled in the system firmware. +* An IOMMU implementation supported by the processor: + * Intel VT-d + * AMD-Vi + * Arm SMMUv3 +* OpenNebula host packages installed, including KVM, QEMU, and libvirt support. + +--- + +## Step 1. Verify Virtualization Support + +Before enabling PCI passthrough, verify that the processor supports hardware virtualization. + +Display the processor capabilities: + +```shell +lscpu +``` + +Look for the **Virtualization** field: + +```default +Virtualization: VT-x +``` + +or + +```default +Virtualization: AMD-V +``` + +On ARM systems, verify that virtualization extensions are available according to the platform documentation. + +## Step 2. Enable the IOMMU + +PCI passthrough requires the Input-Output Memory Management Unit (IOMMU) to isolate DMA accesses performed by assigned devices. + +### Enable the IOMMU in the Firmware + +Access your system's BIOS or UEFI setup and enable the corresponding IOMMU option. Typical firmware options include: + +* Intel VT-d +* AMD IOMMU +* Arm SMMU + +Save the changes and reboot the system. + +### Enable the IOMMU in the Kernel + +{{< tabpane text=true right=false >}} +{{% tab header="**Arch**:" disabled=true /%}} +{{% tab header="Intel"%}} +```shell +intel_iommu=on iommu=pt +``` +{{% /tab %}} +{{% tab header="AMD"%}} +```shell +amd_iommu=on iommu=pt +``` +{{% /tab %}} +{{% tab header="ARM"%}} +Most Arm platforms enable the SMMU automatically. Refer to the platform documentation if additional kernel parameters are required. + +After modifying the kernel command line, regenerate the bootloader configuration using the appropriate command for your Linux distribution and reboot the host. +{{% /tab %}} +{{< /tabpane >}} + +### Verification + +Display the kernel command line: + +```shell +cat /proc/cmdline +``` + +Verify that the IOMMU has been initialized: + +```shell +dmesg | grep -Ei "iommu|dmar|smmu" +``` + +Typical output: + +```default +DMAR: IOMMU enabled +``` + +or: + +```default +AMD-Vi: Initialized +``` + +or: + +```default +arm-smmu-v3 ... +``` + +## Step 3. Configure VFIO Device Binding + +PCI devices assigned to virtual machines must be managed by a VFIO driver instead of their native host driver. OpenNebula supports any mechanism that binds devices to VFIO. The recommended approach is to use the **driverctl** utility, which provides persistent per-device driver overrides using the Linux kernel `driver_override` interface. + +For Linux Containers, the PCI device driver shouldn’t be changed. Skip this section entirely if using passthrough in Linux Containers. + +{{< alert title="Note" type="primary" >}}This step is not needed for NVIDIA GPUs in MIG or vGPU mode. In these cases you can continue to [Step 4](#step-4-configure-huge-pages-optional).{{< /alert >}} + +### Install driverctl + +Install the driverctl package using your Linux distribution package manager. + +### Load the VFIO Modules + +Load the required kernel modules: + +```shell +modprobe vfio +modprobe vfio-pci +``` + +To load them automatically after reboot, create: + +```default +/etc/modules-load.d/vfio.conf +``` + +With contents: + +```default +vfio +vfio-pci +``` + +### Identify the PCI Device + +List the available PCI devices: + +```shell +lspci -nn +``` + +Example output: + +```default +43:00.0 3D controller: NVIDIA Corporation GH100 ... +43:00.1 Audio device: NVIDIA Corporation ... +``` + +Display the current driver: + +```shell +lspci -nnk -s 43:00.0 +``` + +Example output: + +```default +Kernel driver in use: nvidia +``` + +### Bind the Device + +Assign the device to the `vfio-pci` driver: + +```shell +driverctl set-override 0000:43:00.0 vfio-pci +``` + +Many PCI devices expose multiple functions. For example, GPUs often expose both a graphics controller and an audio controller. Repeat the command for every PCI function that will be assigned to the virtual machine. The override is persistent and is automatically restored after reboot. + +{{< alert title="Note" type="primary" >}} +Once a device is bound to a VFIO driver, it is no longer available to the host operating system until the override is removed. +{{< /alert >}} + +### Verification + +Verify that the device is managed by VFIO: + +```shell +lspci -nnk -s 43:00.0 +``` + +Expected output: + +```default +Kernel driver in use: vfio-pci +``` + +Verify that libvirt can detect the device: + +```shell +virsh nodedev-list | grep pci +``` + +List the configured driver overrides: + +```shell +driverctl list-overrides +``` + +### Removing an Override + +To restore the default host driver: + +```shell +driverctl unset-override 0000:43:00.0 +``` + +{{< alert title="Note" type="primary" >}} +Some hardware platforms require a platform-specific VFIO driver instead of `vfio-pci`. For example, NVIDIA Grace systems use the `nvgrace_gpu_vfio_pci` driver. These platform-specific procedures are described in the corresponding device guides. +{{< /alert >}} + +### VFIO Device Ownership + +For OpenNebula to manage the VFIO device files in `/dev/vfio/` must be owned by the root:kvm user and group. This is achieved by creating a `udev` rule. + +1. **Identify the IOMMU group for your GPU using its PCI address**: + + ```shell + find /sys/kernel/iommu_groups/ -type l | grep e1:00.0 + + /sys/kernel/iommu_groups/85/devices/0000:e1:00.0 + ``` + + In this example, the IOMMU group is 85 + +2. **Create a udev rule**: Create the file /etc/udev/rules.d/99-vfio.rules with the following content: + + ```default + SUBSYSTEM=="vfio", GROUP="kvm", MODE="0666" + ``` + +3. **Reload udev rules**: + + ```shell + udevadm control --reload + udevadm trigger + ``` + +4. **Verify ownership**: Check the ownership of the device file corresponding to your GPU’s IOMMU group. + + ```shell + ls -la /dev/vfio/ + crw-rw-rw- 1 root kvm 509, 0 Oct 16 10:00 85 + ``` + +## Step 4. Configure Huge Pages (Optional) + +Huge Pages reduce memory translation overhead and are recommended for memory-intensive workloads such as GPUs and high-performance networking. + +Edit the bootloader configuration and append the appropriate Huge Page parameters to the kernel command line. + +Example: + +```default +hugepages=1024 default_hugepagesz=1G hugepagesz=1G +``` + +The appropriate Huge Page size depends on the processor architecture and workload. On x86 platforms, common Huge Page sizes are 2 MB and 1 GB. + +Regenerate the bootloader configuration using the appropriate command for your Linux distribution and reboot the host. + +**Notes**: + +* Ensure sufficient free memory is available before reserving Huge Pages. +* The selected Huge Page size must be supported by both the processor and the kernel. +* Display the supported Huge Page size: + ```shell + grep Hugepagesize /proc/meminfo + ``` + +OpenNebula allows virtual machines to request Huge Pages directly from the VM Template. + +### Verification + +Display the current Huge Page configuration: + +```shell +grep Huge /proc/meminfo +``` + +## Step 5. Configure PCI Monitoring + +OpenNebula discovers PCI devices through the host monitoring subsystem. PCI monitoring is disabled by default to avoid exposing unnecessary hardware resources. + +Before PCI devices can be assigned to virtual machines, configure the monitoring subsystem to discover the devices intended for passthrough. + +### Configure System-wide PCI Monitoring + +The default monitoring configuration is stored on the Front-end: + +```default +/var/lib/one/remotes/etc/im/kvm-probes.d/pci.conf +``` + +The configuration supports three filtering mechanisms: + +* **filter** — Select devices using PCI vendor, device and class identifiers. +* **short_address** — Restrict discovery to specific PCI addresses. +* **device_name** — Select devices matching one or more regular expressions. + +The filters are applied sequentially. + +Example: + +```default +filter: + + - "10de:*" + +short_address: [] + +device_name: [] +``` + +To monitor a specific PCI device: + +```default +filter: + + - "10de:*" + +short_address: + + - "43:00.0" + +device_name: [] +``` + +To discover SR-IOV Virtual Functions: + +```default +filter: + + - "*:*" + +device_name: + + - "Virtual Function" +``` + +### Override Monitoring at Cluster or Host Level + +The system-wide configuration can be overridden for individual clusters or hosts using the corresponding template attributes: + +| Parameter | Description | +|-----------|-------------| +| PCI_FILTER | (List) Filters by PCI vendor:device:class patterns (same as for lspci) | +| PCI_SHORT_ADDRESS | (List) Filters by short PCI address bus:device.function | +| PCI_DEVICE_NAME | (List) Filters by the PCI device name reported by the Host | + +This allows different clusters to expose different classes of PCI devices while sharing the same frontend configuration. + +{{< alert title="Recommendation" type="primary" >}} +Configure the narrowest possible filters to expose only devices intended for passthrough.{{< /alert >}} + +### Synchronize the Monitoring Probes + +After modifying `pci.conf`, synchronize the updated monitoring probes to all hosts (run as oneadmin): + +```shell +onehost sync --force +``` + +Then trigger a new monitoring cycle: + +```shell +onehost flush +``` + +Alternatively, wait for the next scheduled monitoring cycle. + +## Step 6. Verify PCI Device Discovery + +After the monitoring cycle completes, verify that OpenNebula has discovered the expected PCI devices. +Display the monitored devices: + +```shell +onehost show +``` + +Alternatively, inspect the PCI Devices tab from the Sunstone interface. + +Verify that: + +* The expected PCI devices are listed. +* Vendor and device identifiers are correct. +* Driver information is correct. +* NUMA information is reported. +* SR-IOV Virtual Functions are detected, when applicable. +* Device availability matches the host configuration. + +If a device does not appear: + +* Verify that it matches the configured monitoring filters. +* Verify that pci.conf has been synchronized to the hosts. +* Verify that the host has completed a monitoring cycle. +* Verify that the device is visible using lspci. +* Verify that the device is bound to the appropriate VFIO driver. +* Verify that the OpenNebula monitoring service is running. + +## Configuration Checklist + +Before deploying virtual machines using PCI passthrough, verify that the following requirements have been completed: +* ✔ Hardware virtualization enabled +* ✔ IOMMU enabled +* ✔ VFIO modules loaded +* ✔ PCI devices bound to VFIO +* ✔ Huge Pages configured (if required) +* ✔ SR-IOV configured (if required) +* ✔ PCI monitoring configured +* ✔ PCI devices discovered by OpenNebula + +Once these checks are complete, the host is ready to assign PCI devices to virtual machines. + +## Next Steps + +After completing the host configuration: + +* Continue with the [Network Interfaces Guide]({{% relref "product/cluster_configuration/pci_passthrough_sriov/network_interfaces/" %}}) to configure PCI passthrough or SR-IOV network devices. +* Continue with the [NVIDIA GPUs Guide]({{% relref "product/cluster_configuration/pci_passthrough_sriov/nvidia_gpu_passthrough/" %}}) for NVIDIA GPU passthrough, mediated devices (vGPU), and platform-specific GPU configuration. +* Continue with the [NVIDIA Fabric Manager Guide]({{% relref "product/cluster_configuration/pci_passthrough_sriov/one_fabricmanager/" %}}) when deploying supported NVSwitch-based systems. + + diff --git a/content/product/cluster_configuration/pci_passthrough_sriov/network_interfaces.md b/content/product/cluster_configuration/pci_passthrough_sriov/network_interfaces.md new file mode 100644 index 000000000..71ae87b50 --- /dev/null +++ b/content/product/cluster_configuration/pci_passthrough_sriov/network_interfaces.md @@ -0,0 +1,519 @@ +--- +title: "Network Interfaces" +date: "2026-06-30" +description: +categories: +pageintoc: "58" +tags: ['AI','NVIDIA'] +weight: "3" +--- + +This guide describes how to use PCI network devices as OpenNebula network interfaces. + +OpenNebula extends the generic PCI passthrough mechanism by integrating PCI network devices with the Virtual Network subsystem. This allows virtual machines to benefit from the performance of direct device assignment while preserving the OpenNebula networking workflow, including IP address management, MAC address allocation, scheduling, and contextualization. + +General host preparation, including IOMMU, VFIO, Huge Pages, SR-IOV configuration, and PCI monitoring, is described in the [Host Configuration]({{% relref "product/cluster_configuration/pci_passthrough_sriov/host_configuration" %}}) guide. + +## Overview + +OpenNebula supports assigning both Physical Functions (PFs) and Single Root I/O Virtualization (SR-IOV) Virtual Functions (VFs) as network interfaces. + +Unlike generic PCI devices, network interfaces can be fully integrated with OpenNebula Virtual Networks. During deployment, OpenNebula: + +* Selects a PCI device matching the requested constraints. +* Allocates networking resources from the selected Virtual Network. +* Assigns MAC and IP addresses. +* Configures VLAN parameters when required. +* Contextualizes the guest operating system. + +This provides the performance benefits of PCI passthrough while preserving the operational model of OpenNebula networking. + +## Physical Functions and Virtual Functions + +Modern network adapters commonly implement Single Root I/O Virtualization (SR-IOV), allowing a single Physical Function (PF) to expose multiple lightweight Virtual Functions (VFs): +* **Physical Functions**: A Physical Function represents the complete PCI device. Assigning a PF gives a virtual machine exclusive access to the network adapter and all of its capabilities. +* **Virtual Functions**: A Virtual Function is an independent PCI function created by the Physical Function. Each VF can be assigned independently to a virtual machine while sharing the underlying hardware resources. + +Virtual Functions are typically used for cloud and NFV deployments because they provide excellent performance while allowing multiple virtual machines to share the same physical adapter. + +### Legacy and Switchdev Modes + +OpenNebula supports both SR-IOV operating modes. + +* **Legacy Mode**: In Legacy mode, OpenNebula programs the Virtual Function directly. The following attributes are supported: + * MAC + * VLAN_ID + * TRUST + * SPOOFCHK +
+ + These attributes are applied directly by the network adapter. + +* **Switchdev Mode**: In Switchdev mode, VF parameters are controlled by Host-side representor interfaces. These representor interfaces can be attached to a virtual switch to establish port-level control. + + In this mode, only the MAC address is applied directly to the VF interface. All other control parameters are managed by the virtual switch driver associated with the Virtual Network. + + OpenNebula automatically configures the representor interface during deployment. As of now, only Open vSwitch is supported for Switchdev mode. + +## Host Configuration + +If the PCI device supports Single Root I/O Virtualization (SR-IOV), Virtual Functions (VFs) can be created and assigned independently to virtual machines. Determine the maximum number of supported Virtual Functions: + +```shell +cat /sys/bus/pci/devices//sriov_totalvfs +``` + +Enable the desired number of Virtual Functions: + +```shell +echo 8 > /sys/bus/pci/devices//sriov_numvfs +``` + +{{< alert title="Note" type="primary" >}} +The configured number of Virtual Functions is typically reset after reboot. Refer to your Linux distribution or hardware vendor documentation to configure persistent SR-IOV devices.{{< /alert >}} + +### Verification + +Verify that the Virtual Functions have been created: + +```shell +lspci +``` + +or + +```shell +ip link +``` + +depending on the device type. + +Additional SR-IOV configuration for network adapters is described in the **Network Interfaces** guide. + +## Host Configuration for DPDK Interfaces + +In cases where high network performance is needed but passing through PCI network interfaces to a VM is not possible, Open vSwitch can run with an accelerated datapath by using DPDK userspace libraries. In this mode, the packet processing responsibility is taken away from the kernel and dedicated resources are assigned to the packet processing. This includes CPU threads and huge pages. + +The virtual network interfaces of the VM are not passed through, but the uplink backing the virtual switch normally is passed through to OVS, with the use of vfio-pci. In this section there is the host configuration required to set up DPDK. + +### Installation + +You need a version of OVS that has been compiled with DPDK support. Linux distributions handle the distribution of this version quite differently. + +* **Ubuntu and Debian**: Use the openvswitch-switch-dpdk package. The openvswitch-switch package lacks DPDK libraries. +* **Red Hat**: + * Activate the fast-datapath-for-rhel--x86_64-rpms repo in subscription manager. + * Install any of the openvswitch packages. Red Hat distributes multiple versions, each with its own package. +* **Alma**: + * Install centos-release-nfv-openvswitch to activate the required repo. + * Install any of the openvswitch packages. + +Verify DPDK support by checking the OVS version: + +```shell +ovs-vswitchd --version + ovs-vswitchd (Open vSwitch) 3.5.3-6.el9s + DPDK 24.11.3 +``` + +### Resource Allocation + +It is important to have a clear picture of which resources are going to be exclusively dedicated to DPDK. These resources are effectively a payload, since they are not available to the rest of the system. + +### Network Interfaces + +Identify which network interfaces are going to be used as DPDK interfaces. When the system has multiple NUMA nodes, it is important to consider the node placement of these interfaces. + +To identify which node the interfaces belong to: + +```shell +cat /sys/bus/pci/devices/0000\:01\:00.0/numa_nod +``` + +### Memory + +Each Network Interface in DPDK requires a certain amount of memory in terms of hugepages. This amount of memory depends on the MTU of the interface and the NUMA node location of the interface. Whenever possible, use 1GB hugepages. + +A good rule of thumb is to use 1GB per 1500 MTU NIC and 3GB per 9000+ MTU NIC. For a detailed memory calculation refer to the [memory model documentation](https://docs.openvswitch.org/en/stable/topics/dpdk/memory/) in the OpenvSwitch doc. If there are NUMA nodes without a network interface, 1GB of memory should be allocated to said NUMA node. + +VMs with their virtual network interfaces backed by a DPDK capable bridge need hugepages to run as well. Therefore, more huge pages are required to consume this feature. + +Refer to the Huge Pages section to configure the required amount of pages. Consider that huge pages exist on a per node basis. If the node location is unspecified when requesting hugepages, the pages will be requested evenly. Make sure enough is available per node. + +VMs backed by a DPDK bridge also require hugepages. On top of the hugepages required for the OVS daemon, consider also pages for running VMs. + +Every memory dedicated to hugepages is effectively memory that is no longer available for running regular non huge page backed VMs. Since dynamic huge page allocation is not guaranteed, especially with 1GB huge pages, it is important to pre-allocate in the kernel command line a reasonable number, based on a typical workload. Alternatively, 2MB huge pages could be used since they are much easier to dynamically allocate. + +### CPU + +The DPDK PMD driver runs continuous polling threads to process network packets. These threads are assigned to dedicated CPUs from the operating system. By default, one thread per interface in a NUMA node is used. More threads means faster polling. + +After the configuration, regardless of whether there is traffic or not in the switch, the polling processes will be using these threads at 100% usage, so it is effectively removed from the system and the linux scheduler will not use those processes normally. + +For added security, you can use the isolcpus kernel parameter to declare those threads as not available to the linux scheduler, however the linux scheduler is mature enough to not need this. The important reservation comes in the OpenNebula scheduler. + +You must set the ISOCLPUS host parameter to prevent a case where the scheduler could pin VCPUs from VMs to those threads. + +### Deamon Configuration + +After having a clear picture of the resource allocation, it’s time to tell OVS about how to use them. + +First, make sure that the daemon is enabled: + +```shell +systemctl start openvswitch +systemctl enable openvswitch +``` + +Then initialize DPDK and signal the huge table mountpount: + +```shell +ovs-vsctl set Open_vSwitch . other_config:dpdk-init=true +ovs-vsctl set Open_vSwitch . other_config:dpdk-hugepage-dir="/dev/hugepages" +``` + +This mountpoint should be automatically managed by a system where libvirt is installed. It is an interface to consume the pages of the size specified in that mountpoint: + +```shell +mount | grep -i huge + hugetlbfs on /dev/hugepages type hugetlbfs + (rw,relatime,seclabel,pagesize=1024M) +``` +To assign the previously reserved huge pages to DPDK use the` other_config:dpdk-socket-mem` parameter. This is a comma separated list of memory, in MB, to allocate per node, which will then be backed by the huge pages available at `other_config:dpdk-hugepage-dir`. + +### Examples + +A system has 2 NUMA nodes. Two network interfaces are going to be used as DPDK interfaces. Each interface is in a different NUMA node. The interfaces will have 1500 MTU. 1GB hugepages are used. 2 pages are required in each node: + +```shell +ovs-vsctl set Open_vSwitch . other_config:dpdk-socket-mem=1024,1024 +``` + +A system has 1 NUMA node. Two network interfaces are going to be used as DPDK interfaces. The interfaces will have 9000 MTU. 1GB hugepages are used. 6 pages are required: + +```shell +ovs-vsctl set Open_vSwitch . other_config:dpdk-socket-mem=6144 +``` + +A system has 4 NUMA nodes. Two network interfaces are going to be used as DPDK interfaces. Interfaces are located in node0 and node1. The interfaces will have 9000 MTU. 1GB hugepages are used. 3 pages are required in node0 and node1 each.1 page is required in node2 and 3 each. + +```shell +ovs-vsctl set Open_vSwitch . other_config:dpdk-socket-mem=3072,3072,1024,1024 +``` + +### PMD Threads + +The CPU threads where OVS will pin the PMD threads are signaled with the parameter `other_config:pmd-cpu-mask`. It is a hex bitmask. Each bit represents a specific logical CPU ID. At least one CPU thread per node is needed. When assigning multiple threads per node, it is recommended to pick threads sharing the same physical CPU core. + +To generate the value of the bitmask: + +* Inspect the CPUs available in the Operating System. To get a full picture of this, run the command lscpu --all -p=CPU,CORE,NODE. +* This will yield a list of logical CPU threads ids, their respective parent logical core id and the NUMA node they belong to. +* Create a comma separated list of the CPU IDs going to be used according to recommendations. +* Create a binary bitmask with each desired CPU thread ID set to 1. +* Convert to hex. + +Example: + +```shell +lscpu --all -p=CPU,CORE,NODE + # The following is the parsable format, which can be fed to other + # programs. Each different item in every column has an unique ID + # starting usually from zero. + # CPU,Core,Node + 0,0,0 # pick + 1,1,1 # pick + 2,2,0 + ... +``` + +Core 0 in node 0 holds threads 0 and 64. Core 1 in node 1 holds threads 1 and 65. The string `0,1,64,65` should be used as the ISOLCPUS parameter in the hypervisor node configuration. + +```shell +echo "ISOLCPUS=\"0,1,64,65\"" > /tmp/dpdk-host.tpl +onehost update sm15 -a /tmp/dpdk-host.tpl +``` + +The subsequent mask is then applied: + +```shell +ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x30000000000000003 +``` + +Restart the openvswitch daemon so it loads all of the DPDK related configurations: + +```shell +systemctl restart openvswitch +``` + +### Verification + +Check OVS is properly configured. Make sure that DPDK is initialized and the configuration values we established for the OVS daemon are loaded: + +```shell +ovs-vsctl get Open_vSwitch . dpdk_initialized +grep DPDK /var/log/openvswitch/ovs-vswitchd.log +ovs-vsctl get Open_vSwitch . other_config:dpdk-socket-mem +ovs-vsctl get Open_vSwitch . other_config:pmd-cpu-mask +ovs-vsctl get Open_vSwitch . other_config:dpdk-hugepage-dir +``` + +### Bridge Creation + +The OVS DPDK bridge must be pre-created before being used in OpenNebula. Besides some extra configuration, most of the normal OVS bridge related configurations apply. There are two elements that distinguish a DPDK bridge from a regular bridge: accelerated datapath and the use of DPDK interfaces. + +A DPDK bridge is created as follows: + +```shell +ovs-vsctl --may-exist add-br 'ovsbr0' -- set Bridge 'ovsbr0' 'datapath_type=netdev' +``` + +Then the DPDK interfaces must be assigned to that bridge. Normally, those interfaces need to have their kernel driver unbound and use instead the `vfio-pci` driver used for passthrough. However, there are cases where the kernel driver must not be unbound. This is the case for the `MT27710 Family [ConnectX-4 Lx] 1015` Mellanox network interfaces using the `mlx5_core` driver. + +Refer to the PCI Device VFIO binding section for this procedure. Alternatively, use the utility `dpdk-devbind.py` provided by the DPDK related packages. It allows to rebind interfaces as well, without the use of `driverctl` and it also provides a summary of every PCI device related property. + +```shell +dpdk-devbind.py --status-dev net + + Network devices using DPDK-compatible driver + ============================================ + 0000:01:00.0 'BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller 16d8' numa_node=0 drv=vfio-pci unused=bnxt_en + 0000:01:00.1 'BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller 16d8' numa_node=0 drv=vfio-pci unused=bnxt_en + 0000:81:00.2 'MT27710 Family [ConnectX-4 Lx Virtual Function] 1016' numa_node=0 drv=vfio-pci unused=mlx5_core + ... +``` + +After the interface is ready, attach it to the bridge: + +```shell +ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 \ +type=dpdk options:dpdk-devargs= # replace with the network interface PCI device address +``` + +In this case `dpdk0` was the name used for the interface and the port. This name is not important as the interface is referenced by its PCI address. vfio-pci bound interfaces do not even have names. What’s important is that the said name must be unique per interface. + +If using multiple interfaces in bond, first create the bond normally, then attach the interfaces to the bond port. For example: + +```shell +ovs-vsctl add-bond 'ovsbr0' 'bond0' 'dpdk0' 'dpdk1' 'bond_mode=balance-slb' \ + -- set Interface 'vmnic0' 'type=dpdk' \ + -- set Interface 'vmnic0' 'options:dpdk-devargs=0000:01:00.0' \ + -- set Interface 'vmnic0' 'mtu_request=9126' \ + -- set Interface 'vmnic1' 'type=dpdk' \ + -- set Interface 'vmnic1' 'options:dpdk-devargs=0000:01:00.1' \ + -- set Interface 'vmnic1' 'mtu_request=9126' \ +``` + +After binding interfaces you should now start seeing the PMD threads at 100% CPU usage. This is expected and will happen regardless of whether that bridge is serving traffic to VMs or not. + +If the DPDK interface holds an IP address which is needed, this IP address must be migrated to the OVS bridge internal interface. It has the same name as the bridge. This interface can be configured like regular system interfaces. We recommend setting all of the L2 related configuration in OVS directly and only manage the IP related configuration outside of OVS, you can use manual commands or the system network renderer for this. + +If the interface is a management interface then you must ensure a proper IP migration via a script since binding the network interface and submitting it to a bridge makes it lose connectivity. + +### Security Configuration + +Depending on your distro, it might be required to tune Selinux or AppArmor to allow proper OVS-QEMU interaction. + +When a VM is created, qemu creates a UNIX socket for each network interface backed by a DPDK capable bridge. OVS then attempts to connect to these sockets. Some operations involving VM states, like migrations and power cycles require qemu also to perform an unlink operation in the socket. + +Selinux or AppArmor can, at any given time block any of these operations. This is affected also by how the Operating System distributes the OVS binary. In the case of the Red Hat familiy, OVS runs with a dedicated user. In the case of the Debian family, OVS runs as root. This might cause issues with directory based permissions, depending on the location of the socket. These sockets, starting from OpenNebula 7.2.1+ are located at `/var/run/one/vhost-socks/`. + +Below is a reference of the configuration required for Selinux systems. If errors persist after creating VMs, check the system logs, especially the audit logs, for operations being blocked if you still see permission denied errors but permissions look correct: + +```shell +sudo chcon -t virt_var_run_t /var/run/one/vhost-socks/ + +TDIR=$(mktemp -d) +cat << 'EOF' > "$TDIR/dpdk_virt.te" +module dpdk_virt 1.0; +require { + type openvswitch_t; + type virt_var_run_t; + class dir { search getattr read open }; + class sock_file { write getattr append }; +} +allow openvswitch_t virt_var_run_t:dir { search getattr read open }; +allow openvswitch_t virt_var_run_t:sock_file { write getattr append }; +EOF + +checkmodule -M -m -o "$TDIR/dpdk_virt.mod" "$TDIR/dpdk_virt.te" +semodule_package -o "$TDIR/dpdk_virt.pp" -m "$TDIR/dpdk_virt.mod" +sudo semodule -i "$TDIR/dpdk_virt.pp" +rm -rf "$TDIR" +sudo setfacl -m u:openvswitch:rx /var/run/one/ +sudo setfacl -m u:openvswitch:rx /var/run/one/vhost-socks/ +sudo setfacl -d -m u:openvswitch:rwX /var/run/one/vhost-socks/ +``` + +Refer to the OpenvSwitch with DPDK section for the Virtual Network configuration and reference VM Templates. + +### Security Configuration + +You can use the [openvswitch role](https://github.com/OpenNebula/one-deploy/blob/dbbec90d80a0a6a7e598b9a55169b0050a8c7c9f/roles/openvswitch/README.md#L16) to automate all of the DPDK related configuration and standalone OVS complex configurations as well. + +## Using PCI Devices as Network Interfaces + +To use a PCI device as a network interface, set the `TYPE` attribute of the `PCI` element to `NIC`. Example: + +```default +PCI = [ + TYPE = "NIC", + NETWORK = "SRIOV-NET" +] +``` + +Unlike generic PCI passthrough, a PCI device configured as a network interface becomes part of the OpenNebula networking subsystem. + +During deployment, OpenNebula: + +1. Selects a compatible PCI device. +2. Allocates networking resources from the specified Virtual Network. +3. Configures the PCI device. +4. Generates the corresponding NIC context information. +5. Contextualizes the guest operating system. + +As a result, the guest receives a fully configured network interface without additional manual configuration. + +## Selecting PCI Devices + +PCI devices may be selected explicitly or automatically. + +### Automatic Selection + +Automatic selection is the recommended approach. Instead of specifying a PCI address, define the required device characteristics. + +Example: + +``` +PCI = [ + TYPE = "NIC", + NETWORK = "SRIOV-NET", + + CLASS = "0200", + VENDOR = "15b3" +] +``` + +The scheduler automatically selects an available PCI device satisfying the specified constraints. Automatic selection improves workload portability across hosts. + +### Explicit Device Selection + +Specific devices may also be selected using their PCI address. + +Example: + +``` +PCI = [ + TYPE = "NIC", + DEVICE = "0000:81:00.4" +] +``` + +Explicit selection should generally be reserved for specialized deployments where the VM needs to be attached to specific networks. + +## Virtual Network Integration + +Unlike generic PCI passthrough, PCI network interfaces participate fully in OpenNebula Virtual Networks. The selected Virtual Network provides: + +* MAC address allocation +* IPv4 allocation +* IPv6 allocation +* VLAN configuration +* Security Groups +* Address management + +This allows PCI passthrough interfaces to behave consistently with virtual network interfaces from the administrator's perspective. + +## Contextualization + +When the Context package is installed inside the guest operating system, OpenNebula automatically configures the assigned PCI interface. The guest receives: + +* MAC address +* IPv4 address +* IPv6 address +* Network mask +* Gateway +* DNS configuration +* Hostname + +No manual network configuration is required inside the guest. + +## Supported PCI Attributes + +PCI network interfaces support the standard PCI attributes together with networking-specific attributes. + +Common PCI attributes include: + +* `DEVICE` +* `CLASS` +* `VENDOR` +* `TYPE` + +Network-specific attributes include: + +* `NETWORK` +* `NETWORK_UNAME` +* `MAC` +* `IP` +* `IP6` +* `VLAN_ID` +* `TRUST` +* `SPOOFCHK` + +Refer to the Virtual Machine Template reference for a complete description of each attribute. + +## Supported PCI Attributes + +PCI network interfaces support the standard PCI attributes together with networking-specific attributes. + +Common PCI attributes include: + +* `DEVICE` +* `CLASS` +* `VENDOR` +* `TYPE` + +Network-specific attributes include: + +* `NETWORK` +* `NETWORK_UNAME` +* `MAC` +* `IP` +* `IP6` +* `VLAN_ID` +* `TRUST` +* `SPOOFCHK` + +Refer to the Virtual Machine Template reference for a complete description of each attribute. + +## Guest Verification + +After deployment, verify that the guest detects the assigned network adapter. + +Display the PCI devices: + +```shell +lspci +``` + +Display the network interfaces: + +```shell +ip link +``` + +The interface should appear as a native PCI network adapter and be configured automatically through contextualization. + +## Best Practices + +* Prefer automatic PCI device selection whenever possible. +* Use Virtual Functions instead of Physical Functions for cloud deployments. +* Reserve Physical Functions for workloads requiring exclusive device ownership. +* Configure PCI monitoring to expose only passthrough devices. +* Install the OpenNebula Context package in guest operating systems. +* Use NUMA-aware placement for latency-sensitive workloads. +* Prefer Switchdev mode when integrating with Open vSwitch. + +## Next Steps + +If you have not yet configured the host for PCI passthrough, complete the [Host Configuration Guide]({{% relref "product/cluster_configuration/pci_passthrough_sriov/host_configuration/" %}}). For NVIDIA GPUs, continue with the [NVIDIA GPU Guide]({{% relref "product/cluster_configuration/pci_passthrough_sriov/nvidia_gpu_passthrough/" %}}). \ No newline at end of file diff --git a/content/product/cluster_configuration/pci_passthrough_sriov/nvidia_gpu_passthrough.md b/content/product/cluster_configuration/pci_passthrough_sriov/nvidia_gpu_passthrough.md new file mode 100644 index 000000000..0320f03c1 --- /dev/null +++ b/content/product/cluster_configuration/pci_passthrough_sriov/nvidia_gpu_passthrough.md @@ -0,0 +1,772 @@ +--- +title: "NVIDIA GPU Passthrough" +linkTitle: "NVIDIA GPU (vGPU, MIG, Passthrough)" +date: "2025-10-16" +description: +categories: +pageintoc: "58" +tags: ['AI','NVIDIA'] +weight: "5" +--- + +## Overview + +This guide describes how to deploy NVIDIA GPUs using PCI passthrough technologies in OpenNebula. OpenNebula supports multiple NVIDIA GPU deployment models, including conventional PCI passthrough, NVIDIA vGPU, MIG-backed vGPU, and NVIDIA Grace platform virtualization. While these deployment models share the same scheduling and deployment workflow, each requires a different Host configuration. + +The generic PCI passthrough configuration, including IOMMU, VFIO, Huge Pages, PCI monitoring, and device discovery, is described in the [Host Configuration Guide]({{% relref "product/cluster_configuration/pci_passthrough_sriov/host_configuration/" %}}). This guide focuses exclusively on the NVIDIA-specific configuration and deployment procedures for each supported deployment model. + +## Deployment Models + +NVIDIA GPUs can be deployed in OpenNebula using one of the following deployment models. + +| Deployment Model | Typical Platforms | VFIO Driver | Additional Configuration | +| ----- | ----- | ----- | ----- | +| Conventional PCI Passthrough | H100 PCIe, H100 SXM, H200, L40, L40S, RTX, A100 | `vfio-pci` | Standard PCI passthrough | +| NVIDIA vGPU / MIG-backed vGPU | A100, H100, H200, B200 | NVIDIA vGPU stack | NVIDIA vGPU or MIG-backed vGPU configuration | +| Grace Platform Virtualization | GH200, GB200 | `nvgrace_gpu_vfio_pci` | Grace-specific virtualization configuration | + +For **conventional PCI passthrough**, the NVIDIA driver is **not required** on the Host. The GPU is owned exclusively by the guest operating system, where the NVIDIA driver is installed after deployment. + +For **NVIDIA vGPU** and **MIG-backed vGPU**, the NVIDIA driver is required on the Host because the Host manages GPU partitioning and virtual GPU creation before assigning GPU resources to Virtual Machines. + +--- + +## Conventional PCI Passthrough + +## Overview + +Conventional PCI passthrough assigns one or more physical NVIDIA GPUs directly to a Virtual Machine, providing the guest operating system with exclusive access to the assigned devices. In this deployment model, the Host does not use the GPUs, which are instead bound to the `vfio-pci` driver and managed entirely by the guest operating system. + +This deployment model is recommended for workloads requiring direct access to the GPU, such as AI training and inference, high-performance computing (HPC), scientific computing, visualization, and CUDA development. It is supported by NVIDIA PCIe and SXM GPUs, including NVIDIA H100, H200, L40, L40S, RTX, and A100 GPUs when Multi-Instance GPU (MIG) is not enabled. + +## Requirements + +Before deploying NVIDIA GPUs using conventional PCI passthrough, complete the generic PCI passthrough configuration described in the [Host Configuration Guide]({{% relref "product/cluster_configuration/pci_passthrough_sriov/host_configuration/" %}}). + +Verify that the Host satisfies the following requirements: + +* Hardware-assisted virtualization and the IOMMU are enabled. +* The GPU is bound to the `vfio-pci` driver. +* PCI monitoring is configured to discover NVIDIA GPUs. +* The GPUs have been discovered by OpenNebula. + +Verify that the GPU is managed by the `vfio-pci` driver: + +```shell +lspci -nnk -d 10de: +``` + +Example output: + +``` +81:00.0 3D controller: NVIDIA Corporation H100 PCIe + Kernel driver in use: vfio-pci + +81:00.1 Audio device: NVIDIA Corporation Device xxxx + Kernel driver in use: vfio-pci +``` + +Verify that the GPUs have been discovered by OpenNebula: + +```shell +onehost show +``` + +The GPUs should appear in the **PCI Devices** section and be available for scheduling. + +{{< alert title="Note" type="primary" >}} +Unlike NVIDIA vGPU or MIG-backed vGPU deployments, the NVIDIA driver is not required on the Host for conventional PCI passthrough. The GPU is owned exclusively by the guest operating system, where the NVIDIA driver is installed after deployment.{{< /alert >}} + +--- + +## Host Configuration + +This section describes the NVIDIA-specific configuration required after completing the generic PCI passthrough configuration. If the NVIDIA driver is installed on the Host, prevent it from binding to GPUs intended for passthrough by blacklisting the NVIDIA kernel modules. + +Create the following configuration file: + +```default +/etc/modprobe.d/blacklist-nvidia.conf +``` + +Example contents: + +```default +blacklist nouveau +blacklist nvidia +blacklist nvidia_drm +blacklist nvidia_modeset +blacklist nvidia_uvm +``` + +Regenerate the initramfs using the appropriate command for your Linux distribution and reboot the Host. + +{{< alert title="Note" type="primary" >}} +This step is only required if the NVIDIA driver is installed on the Host. Hosts dedicated to GPU PCI passthrough typically do not require the NVIDIA driver.{{< /alert >}} + +### Verify the GPU Driver + +After completing the generic PCI passthrough configuration, verify that all GPU PCI functions are managed by the `vfio-pci` driver. + +```shell +lspci -nnk -d 10de: +``` + +Example output: + +``` +81:00.0 3D controller: NVIDIA Corporation H100 PCIe + Kernel driver in use: vfio-pci + +81:00.1 Audio device: NVIDIA Corporation Device xxxx + Kernel driver in use: vfio-pci +``` + +If any GPU function is still managed by the NVIDIA or Nouveau driver, review the VFIO configuration described in the [Host Configuration Guide]({{% relref "product/cluster_configuration/pci_passthrough_sriov/host_configuration/" %}}). + +{{< alert title="Note" type="primary" >}} +NVIDIA GPUs may expose multiple PCI functions, such as audio or USB controllers. All functions belonging to the GPU should be assigned together to the same Virtual Machine.{{< /alert >}} + +### Verify GPU Discovery + +After the next monitoring cycle, verify that OpenNebula has discovered the GPUs. + +```shell +onehost show +``` + +Example output: + +```default +PCI DEVICES + + VM ADDR TYPE NAME + e1:00.0 10de:2321:0302 NVIDIA Corporation GH100 [H100L 94GB] +``` + +The GPUs should appear in the **PCI Devices** section and be available for scheduling. + +Alternatively, the detected PCI devices can be inspected from the **PCI Devices** tab in Sunstone. + +## Deploying a Virtual Machine + +After the Host has been configured and the GPUs have been discovered by OpenNebula, they can be assigned to Virtual Machines using the standard `PCI` attribute in the Virtual Machine Template. + +OpenNebula supports both automatic and explicit GPU selection. Automatic selection is recommended because it allows the scheduler to place workloads on any compatible Host, improving workload portability and resource utilization. + +### Automatic GPU Selection + +The following example requests any available NVIDIA GPU: + +``` +PCI = [ + CLASS = "0302", + VENDOR = "10de" +] +``` + +During deployment, OpenNebula selects a compatible GPU that matches the requested attributes and assigns it to the Virtual Machine. + +Additional `PCI` attributes, such as `DEVICE` or `SHORT_ADDRESS`, can be used to further restrict device selection when required. + +### Selecting a Specific GPU + +For topology-sensitive deployments, benchmarking, or debugging, a specific GPU can be requested using its PCI address. + +For example: + +``` +PCI = [ + SHORT_ADDRESS = "81:00.0" +] +``` + +Explicit device selection should be reserved for specialized deployments. In most environments, automatic GPU selection provides greater flexibility and simplifies workload scheduling. + +### Assigning Multiple GPUs + +Multiple GPUs can be assigned to the same Virtual Machine by including multiple `PCI` sections. + +The following example requests two NVIDIA GPUs: + +``` +PCI = [ + CLASS = "0302", + VENDOR = "10de" +] + +PCI = [ + CLASS = "0302", + VENDOR = "10de" +] +``` + +Each `PCI` section requests one compatible GPU. During deployment, OpenNebula assigns the requested number of devices to the Virtual Machine. + +--- + +## Example H100 Virtual Machine Template + +The following example shows a Virtual Machine Template for an NVIDIA H100 SXM system with eight GPUs assigned through PCI passthrough. + +```default +CONTEXT = [ + NETWORK = "YES", + SSH_PUBLIC_KEY = "$USER[SSH_PUBLIC_KEY]" +] + +CPU = "208" +VCPU = "208" +MEMORY = "716800" + +CPU_MODEL = [ + MODEL = "host-passthrough" +] + +TOPOLOGY = [ + CORES = "52", + SOCKETS = "2", + THREADS = "2", + PIN_POLICY = "THREAD" +] + +OS = [ + ARCH = "x86_64", + FIRMWARE = "/usr/share/OVMF/OVMF_CODE_4M.fd", + FIRMWARE_SECURE = "NO", + MACHINE = "pc-q35-jammy" +] + +DISK = [ + IMAGE = "ubuntu-24.04-gpu", + SIZE = "1572864" +] + +NIC = [ + NETWORK = "admin_net" +] + +GRAPHICS = [ + LISTEN = "0.0.0.0", + TYPE = "vnc" +] + +FEATURES = [ + PCIHOLE64 = "2147483648" +] + +PCI = [ SHORT_ADDRESS = "1b:00.0" ] +PCI = [ SHORT_ADDRESS = "43:00.0" ] +PCI = [ SHORT_ADDRESS = "52:00.0" ] +PCI = [ SHORT_ADDRESS = "df:00.0" ] +``` + +The `PCI` entries assign the eight H100 GPUs to the Virtual Machine. + + +## Guest Configuration + +After the Virtual Machine has been deployed, verify that the GPUs are visible inside the guest: + +```shell +lspci | grep -i nvidia +``` + +Example output: + +```default +23:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] +24:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] +... +46:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB] +``` + +Install the appropriate NVIDIA driver inside the guest operating system. The selected driver version must support the assigned GPU model and be compatible with the required CUDA software stack. + +After installing the driver, verify that all assigned GPUs are detected: + +```shell +nvidia-smi +``` + +Use the output provided in the NVIDIA qualification guide for your platform to verify that all assigned GPUs are present and that no driver or hardware errors are reported. Avoid using placeholder outputs, as driver versions and formatting vary across NVIDIA software releases. + +## NVIDIA vGPU and MIG-backed vGPU + +### Overview + +NVIDIA vGPU enables multiple Virtual Machines to share the resources of a physical GPU by exposing one or more virtual GPU (vGPU) profiles that can be independently assigned to guest operating systems. + +Depending on the GPU generation, vGPU profiles can be created using one of two deployment models: + +* **NVIDIA vGPU**, where virtual GPU profiles are created directly by the NVIDIA AI Enterprise software stack. +* **MIG-backed vGPU**, where the GPU is first partitioned using NVIDIA Multi-Instance GPU (MIG), and one or more vGPU profiles are created from the resulting GPU Instances. + +Regardless of how the profiles are created, OpenNebula automatically discovers the available vGPU profiles during Host monitoring and schedules them as GPU resources that can be assigned to Virtual Machines. + +Before deploying NVIDIA vGPU devices, complete the generic PCI passthrough configuration described in the [Host Configuration Guide]({{% relref "product/cluster_configuration/pci_passthrough_sriov/host_configuration/" %}}). + +--- + +## Requirements + +Before continuing, verify that the Host satisfies the following requirements: + +* A supported NVIDIA GPU is installed. +* The NVIDIA Host driver is installed and operational. +* PCI monitoring is configured. +* The GPU has been discovered by OpenNebula. + +Verify that the NVIDIA driver is correctly installed: + +```shell +nvidia-smi +``` + +Verify that the GPU has been discovered by OpenNebula: + +```shell +onehost show +``` + +The GPU should appear in the **PCI Devices** section. + +### Host Configuration + +Depending on the GPU model, configure either **NVIDIA vGPU** or **MIG-backed vGPU**. + +#### Configure NVIDIA vGPU + +Conventional NVIDIA vGPU is supported on GPUs that expose vGPU profiles directly through the NVIDIA AI Enterprise software stack. + +Install the NVIDIA AI Enterprise software stack and the NVIDIA vGPU Manager according to the NVIDIA documentation. + +After installation, verify that the NVIDIA driver is operational: + +```shell +nvidia-smi +``` + +Display the supported vGPU profiles: + +```shell +nvidia-smi vgpu -s -v +``` + +The command lists all vGPU profiles supported by the GPU. + +After the next monitoring cycle, OpenNebula automatically discovers the available vGPU profiles. + +## Configure MIG-backed vGPU + +Hopper and newer GPUs support creating vGPU profiles from GPU partitions created using NVIDIA Multi-Instance GPU (MIG). + +### Enable MIG Mode + +Display the current MIG configuration: + +```shell +nvidia-smi -i 0 -q | grep MIG -A2 +``` + +Enable MIG mode: + +```shell +sudo nvidia-smi -i 0 -mig 1 +``` + +Verify that MIG mode has been enabled: + +```shell +nvidia-smi -i 0 -q | grep MIG -A2 +``` + +Example output: + +```default +MIG Mode + Current : Enabled + Pending : Enabled +``` + +Some GPU models require a reboot before MIG mode becomes active. + +### Display Available GPU Instance Profiles + +Display the supported GPU Instance profiles: + +```shell +nvidia-smi mig -lgip +``` + +Example output: + +```shell ++-----------------------------------------------------------------------------+ +| GPU instance profiles: | +| GPU Name ID Instances Memory SM DEC ENC OFA | +| Profile Free/Total GiB CE | +|=============================================================================| +| 0 MIG 1g.10gb 19 7/7 9.62 16 0 0 0 | +| 0 MIG 2g.20gb 14 3/3 19.50 32 1 0 0 | +| 0 MIG 3g.40gb 9 2/2 39.25 60 2 0 0 | ++-----------------------------------------------------------------------------+ +``` + +Display the supported GPU Instance placements: + +```shell +nvidia-smi mig -lgipp +``` + +### Create GPU and Compute Instances + +Create the desired GPU Instances. + +For example, the following command creates four **1g.10gb** GPU Instances together with their default Compute Instances: + +```shell +sudo nvidia-smi mig -cgi 19,19,19,19 -C +``` + +Verify the resulting MIG devices: + +```shell +nvidia-smi -L +``` + +Example output: + +``` +GPU 0: NVIDIA H100 PCIe + + MIG 1g.10gb Device 0: (UUID: MIG-xxxxxxxx) + MIG 1g.10gb Device 1: (UUID: MIG-xxxxxxxx) + MIG 1g.10gb Device 2: (UUID: MIG-xxxxxxxx) + MIG 1g.10gb Device 3: (UUID: MIG-xxxxxxxx) +``` + +### Configure vGPU Profiles + +After creating the GPU and Compute Instances, configure the required NVIDIA vGPU profiles according to the NVIDIA AI Enterprise documentation. + +Verify the available profiles: + +```shell +nvidia-smi vgpu -s -v +``` + +### Refresh OpenNebula Monitoring + +Once the GPU configuration has been completed, refresh Host monitoring: + +```shell +onehost forceupdate +``` + +Verify that the configured vGPU profiles have been discovered: + +```shell +onehost show +``` + +The configured profiles should appear in the **PCI Devices** section and be available for scheduling. + +--- + +### Deploying a Virtual Machine + +Virtual machines request NVIDIA vGPU resources by selecting one of the vGPU profiles discovered during Host monitoring. + +The available profiles can be inspected from the **PCI Devices** tab in Sunstone or using: + +```shell +onehost show +``` + +Each discovered profile reports the profile name together with the number of available instances. + +--- + +**Example Virtual Machine Template** + +Virtual machines request NVIDIA vGPU resources using the `PCI` attribute. The `PROFILE` attribute must match one of the profiles discovered during Host monitoring. + +The following example requests an NVIDIA L40S-2B vGPU profile: + +``` +PCI = [ + CLASS = "0302", + DEVICE = "26b9", + PROFILE = "1146 (NVIDIA L40S-2B)", + VENDOR = "10de" +] +``` + +When the Virtual Machine is deployed, OpenNebula allocates a compatible GPU exposing the requested profile. + +For MIG-backed vGPU deployments, the deployment workflow is identical. The available profiles are generated from the configured GPU Instances and automatically discovered during Host monitoring. + +### Guest Configuration + +After the Virtual Machine has been deployed, install the NVIDIA guest driver corresponding to the selected NVIDIA vGPU software release. + +Refer to the NVIDIA AI Enterprise documentation or your Linux distribution for instructions on installing the appropriate guest driver. + +After the installation completes, verify that the assigned virtual GPU is detected: + +```shell +nvidia-smi +``` + +Use the output provided by the NVIDIA vGPU software stack for your selected profile to verify that: + +* the assigned virtual GPU is detected correctly; +* the reported GPU name matches the selected vGPU profile; +* no driver or hardware errors are reported. + +The guest configuration is identical for conventional NVIDIA vGPU and MIG-backed vGPU deployments. Once deployed, the guest operating system interacts with the assigned virtual GPU transparently, regardless of how the profile was created. + +--- + +## NVIDIA Grace Platforms (AE) + +### Overview + +NVIDIA Grace Hopper and Grace Blackwell platforms are based on the Arm architecture and implement a virtualization model that differs from conventional x86 PCI passthrough systems. In addition to assigning GPUs to Virtual Machines, Grace platforms require a specific PCI topology based on dedicated PCI Root Complexes, SMMUv3 devices, IOMMUFD, and ACPI Generic Initiator NUMA nodes. + +OpenNebula automates the creation of this virtual hardware topology during deployment. From the administrator's perspective, GPU assignment uses the standard `PCI` attribute while OpenNebula generates the libvirt configuration required by the Grace virtualization architecture. + +### Requirements + +Before configuring Grace platform virtualization, ensure that the Host has already been configured following the [Host Configuration Guide]({{% relref "product/cluster_configuration/pci_passthrough_sriov/host_configuration/" %}}). + +In addition, verify that the Host satisfies the following Grace-specific requirements: + +* A Linux kernel supporting NVIDIA Grace virtualization. +* NVIDIA QEMU and libvirt packages with Grace virtualization support. +* QEMU configured with `CAP_IPC_LOCK`. +* GPUs bound to the `nvgrace_gpu_vfio_pci` driver. + +The installation and configuration of the virtualization software stack is outside the scope of this guide. Refer to the NVIDIA Grace Virtualization documentation for platform-specific installation instructions. + +### Host Configuration + +Only the Grace-specific configuration required after completing the generic Host configuration is described in this section. + +#### Configure QEMU Memory Locking + +Grace platforms require QEMU to lock large GPU memory mappings. + +Grant the `CAP_IPC_LOCK` capability to the QEMU binary: + +```shell +sudo setcap cap_ipc_lock=ep /usr/bin/qemu-system-aarch64 +``` + +Verify the capability: + +```shell +getcap /usr/bin/qemu-system-aarch64 +``` + +The capability must be restored whenever the QEMU binary is replaced during a package upgrade. + +#### Bind GPUs to the Grace VFIO Driver + +Load the Grace VFIO driver: + +```shell +sudo modprobe nvgrace-gpu-vfio-pci +``` + +Register the NVIDIA GB200 device ID with the driver: + +```shell +echo "10de 2941" | sudo tee /sys/bus/pci/drivers/nvgrace_gpu_vfio_pci/new_id +``` + +Verify that the GPUs are bound to the `nvgrace_gpu_vfio_pci` driver: + +```shell +lspci -nnkk | grep -i 3D -A2 +``` + +Example output: + +```default +0008:01:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2941] (rev a1) + Subsystem: NVIDIA Corporation Device [10de:2046] + Kernel driver in use: nvgrace_gpu_vfio_pci +-- +0009:01:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2941] (rev a1) + Subsystem: NVIDIA Corporation Device [10de:2046] + Kernel driver in use: nvgrace_gpu_vfio_pci +-- +0019:01:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2046] + Kernel driver in use: nvgrace_gpu_vfio_pci +``` +#### Deploying a Virtual Machine + +Grace platform Virtual Machines are deployed using the standard OpenNebula `PCI` attribute together with the CPU and NUMA configuration required by the workload. + +The following example shows the relevant sections of a Virtual Machine Template configured with two Grace CPU NUMA nodes and four NVIDIA GB200 GPUs. + +```default +TOPOLOGY = [ + SOCKETS = "2", + CORES = "72", + THREADS = "1", + PIN_POLICY = "THREAD", + MEMORY_ACCESS = "shared", + HUGEPAGE_SIZE = "512" +] + +NUMA_NODE = [ + MEMORY = "192000", + TOTAL_CPUS = "72", + DISTANCE = "2-17:254" +] + +NUMA_NODE = [ + MEMORY = "192000", + TOTAL_CPUS = "72", + DISTANCE = "2-17:254" +] + +PCI=[ + ACPI_NODES="2-9", + CLASS="0302", + DEVICE="2941", + IOMMU="YES", + IOMMUFD="YES", + ROOT="dedicated", + VENDOR="10de" +] + +PCI=[ + ACPI_NODES="10-17", + CLASS="0302", + ... + +FEATURES=[ + ACPI="yes", + APIC="yes", + GIC="3", + PCIHOLE64="4294967296", + RAS="yes" +] + +IOMMU=[ + MODE="device", + MODEL="smmuv3", + OPTIONS="accel=on ats=on ril=off pasid=on oas=48" +] +``` + +The `PCI` attributes request two NVIDIA GB200 GPUs using dedicated PCI Root Complexes and dedicated IOMMU contexts. + +During deployment, OpenNebula automatically generates the Grace virtualization topology required by the guest, including: + +* PCI Root Complexes and PCI Root Ports. +* Virtual SMMUv3 devices. +* IOMMUFD objects. +* ACPI Generic Initiator NUMA nodes. +* Guest NUMA topology. +* NUMA distance matrices. + +This topology follows the NVIDIA Grace virtualization architecture and preserves CPU and GPU NUMA locality. + +{{< alert title="Note" type="primary" >}} +When using automatic GPU selection, OpenNebula computes the required `ACPI_NODES` assignment automatically. Explicit `ACPI_NODES` values are only required when selecting specific devices and defining a fixed virtual topology.{{< /alert >}} + +### Guest Configuration + +Install the NVIDIA data center driver inside the guest operating system following the NVIDIA documentation. + +After installation, the GPUs are exposed as native Grace devices and CUDA applications require no additional configuration. + +A correctly configured guest exposes: + +* The configured CPU NUMA nodes. +* Eight ACPI Generic Initiator NUMA nodes for each assigned GPU. +* Preserved NVLink connectivity. +* GPU affinity matching the physical Grace topology. + +The NUMA topology can be inspected using: + +```shell +numactl -H +``` + +Example output: + +``` +available: 32 nodes (0-4,6-7,9-33) + +node 0 cpus: 0-71 +node 0 size: 191674 MB + +node 1 cpus: 72-143 +node 1 size: 191741 MB + +... + +node distances: +node 0 1 2 3 4 ... +0: 10 20 254 254 254 ... +1: 20 10 254 254 254 ... +2: 254 254 10 20 20 ... +... +``` + +This topology reflects the virtual hardware generated by OpenNebula. NUMA nodes 0 and 1 correspond to the Grace CPU memory domains, while the remaining nodes represent the ACPI Generic Initiator nodes associated with the assigned GPUs. + +Verify that the GPUs are visible inside the guest: + +```shell +nvidia-smi +``` + +Example output: + +```default ++-----------------------------------------------------------------+ +| NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 | +| CUDA Version: 13.2 | ++-----------------------------------------------------------------+ +| GPU Name Persistence-M | Bus-Id Disp.A | Vol ECC | +| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU Compute | +| | | MIG M | +|=============================+=====================+=============| +| 0 NVIDIA GB200 Off | 00000000:A1:00.0 Off | 0 | +| N/A 38C P0 163W/1200W | 0MiB / 189471MiB | 0% Def | +| | | Disabled | ++-----------------------------+------------------------+----------+ +| 3 NVIDIA GB200 Off | 00000000:B9:00.0 Off | 0 | +| N/A 38C P0 221W/1200W | 0MiB / 189471MiB | 0% Def | +| | | Disabled | ++-----------------------------+------------------------+----------+ +``` + +Inspect the GPU topology: + +```shell +nvidia-smi topo -m +``` + +Example output: + +```default + GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity GPU NUMA ID +GPU0 X NV18 NV18 NV18 0-71 0 4 +GPU1 NV18 X NV18 NV18 0-71 0 3 +GPU2 NV18 NV18 X NV18 72-143 1 9 +GPU3 NV18 NV18 NV18 X 72-143 1 2 +``` + +This output confirms that: + +* The GPUs are correctly detected inside the guest +* NVLink connectivity is preserved +* GPU affinity matches the corresponding CPU NUMA node and the virtual topology generated by OpenNebula reproduces the physical Grace platform topology. diff --git a/content/product/cluster_configuration/pci_passthrough_sriov/nvidia_grace_platform.md b/content/product/cluster_configuration/pci_passthrough_sriov/nvidia_grace_platform.md new file mode 100644 index 000000000..9d884a998 --- /dev/null +++ b/content/product/cluster_configuration/pci_passthrough_sriov/nvidia_grace_platform.md @@ -0,0 +1,264 @@ +--- +title: "NVIDIA Grace Platform Virtualization (AE)" +linkTitle: "NVIDIA Grace Platforms (AE)" +date: "2026-07-09" +description: +categories: +pageintoc: "58" +tags: ['AI','NVIDIA'] +weight: "9" +toc_hide: true +--- + +## Overview + +NVIDIA Grace Hopper and Grace Blackwell platforms implement a virtualization architecture that differs from conventional PCI passthrough systems. In addition to assigning GPUs to virtual machines, Grace platforms require a specific PCI topology based on dedicated PCI Root Complexes, SMMUv3 devices, IOMMUFD, and ACPI Generic Initiator NUMA nodes. + +OpenNebula automates the creation of this virtual hardware topology during deployment. From the administrator's perspective, GPU assignment uses the standard `PCI` attribute while OpenNebula generates the libvirt configuration required by the Grace virtualization architecture. + +### Requirements + +Before configuring Grace platform virtualization, ensure that the Host has already been configured following the [Host Configuration Guide]({{% relref "product/cluster_configuration/pci_passthrough_sriov/host_configuration/" %}}). + +In addition, verify that the Host satisfies the following Grace-specific requirements: + +* A Linux kernel supporting NVIDIA Grace virtualization. +* NVIDIA QEMU and libvirt packages with Grace virtualization support. +* QEMU configured with `CAP_IPC_LOCK`. +* GPUs bound to the `nvgrace_gpu_vfio_pci` driver. + +The installation and configuration of the virtualization software stack is outside the scope of this guide. Refer to the NVIDIA Grace Virtualization documentation for platform-specific installation instructions. + +### Host Configuration + +Only the Grace-specific configuration required after completing the generic Host configuration is described in this section. + +### Configure QEMU Memory Locking + +Grace platforms require QEMU to lock large GPU memory mappings. + +Grant the `CAP_IPC_LOCK` capability to the QEMU binary: + +```shell +sudo setcap cap_ipc_lock=ep /usr/bin/qemu-system-aarch64 +``` + +Verify the capability: + +```shell +getcap /usr/bin/qemu-system-aarch64 +``` + +The capability must be restored whenever the QEMU binary is replaced during a package upgrade. + +### Bind GPUs to the Grace VFIO Driver + +Load the Grace VFIO driver: + +```shell +sudo modprobe nvgrace-gpu-vfio-pci +``` + +Register the NVIDIA GB200 device ID with the driver: + +```shell +echo "10de 2941" | sudo tee /sys/bus/pci/drivers/nvgrace_gpu_vfio_pci/new_id +``` + +Verify that the GPUs are bound to the `nvgrace_gpu_vfio_pci` driver: + +```shell +lspci -nnkk | grep -i 3D -A2 +``` + +Example output: + +```default +0008:01:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2941] (rev a1) + Subsystem: NVIDIA Corporation Device [10de:2046] + Kernel driver in use: nvgrace_gpu_vfio_pci +-- +0009:01:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2941] (rev a1) + Subsystem: NVIDIA Corporation Device [10de:2046] + Kernel driver in use: nvgrace_gpu_vfio_pci +-- +0018:01:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2941] (rev a1) + Subsystem: NVIDIA Corporation Device [10de:2046] + Kernel driver in use: nvgrace_gpu_vfio_pci +-- +0019:01:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2046] + Kernel driver in use: nvgrace_gpu_vfio_pci +``` +### Deploying a Virtual Machine + +Grace platform virtual machines are deployed using the standard OpenNebula `PCI` attribute together with the CPU and NUMA configuration required by the workload. + +The following example shows the relevant sections of a Virtual Machine Template configured with two Grace CPU NUMA nodes and four NVIDIA GB200 GPUs. + +``` +TOPOLOGY = [ + SOCKETS = "2", + CORES = "72", + THREADS = "1", + PIN_POLICY = "THREAD", + MEMORY_ACCESS = "shared", + HUGEPAGE_SIZE = "512" +] + +NUMA_NODE = [ + MEMORY = "192000", + TOTAL_CPUS = "72", + DISTANCE = "2-17:254" +] + +NUMA_NODE = [ + MEMORY = "192000", + TOTAL_CPUS = "72", + DISTANCE = "2-17:254" +] + +PCI=[ + ACPI_NODES="2-9", + CLASS="0302", + DEVICE="2941", + IOMMU="YES", + IOMMUFD="YES", + ROOT="dedicated", + VENDOR="10de" +] + +PCI=[ + ACPI_NODES="10-17", + CLASS="0302", + ... + + +FEATURES=[ + ACPI="yes", + APIC="yes", + GIC="3", + PCIHOLE64="4294967296", + RAS="yes" +] + +IOMMU=[ + MODE="device", + MODEL="smmuv3", + OPTIONS="accel=on ats=on ril=off pasid=on oas=48" +] +``` + +The `PCI` attributes request two NVIDIA GB200 GPUs using dedicated PCI Root Complexes and dedicated IOMMU contexts. + +During deployment, OpenNebula automatically generates the Grace virtualization topology required by the guest, including: + +* PCI Root Complexes and PCI Root Ports. +* Virtual SMMUv3 devices. +* IOMMUFD objects. +* ACPI Generic Initiator NUMA nodes. +* Guest NUMA topology. +* NUMA distance matrices. + +This topology follows the NVIDIA Grace virtualization architecture and preserves CPU and GPU NUMA locality. + +{{< alert title="Note" type="primary" >}} +When using automatic GPU selection, OpenNebula computes the required `ACPI_NODES` assignment automatically. Explicit `ACPI_NODES` values are only required when selecting specific devices and defining a fixed virtual topology.{{< /alert >}} + +### Guest Configuration + +Install the NVIDIA data center driver inside the guest operating system following the NVIDIA documentation. + +After installation, the GPUs are exposed as native Grace devices and CUDA applications require no additional configuration. + +A correctly configured guest exposes: + +* The configured CPU NUMA nodes. +* Eight ACPI Generic Initiator NUMA nodes for each assigned GPU. +* Preserved NVLink connectivity. +* GPU affinity matching the physical Grace topology. + +The NUMA topology can be inspected using: + +```shell +numactl -H +``` + +Example output: + +``` +available: 32 nodes (0-4,6-7,9-33) + +node 0 cpus: 0-71 +node 0 size: 191674 MB + +node 1 cpus: 72-143 +node 1 size: 191741 MB + +... + +node distances: +node 0 1 2 3 4 ... +0: 10 20 254 254 254 ... +1: 20 10 254 254 254 ... +2: 254 254 10 20 20 ... +... +``` + +This topology reflects the virtual hardware generated by OpenNebula. NUMA nodes 0 and 1 correspond to the Grace CPU memory domains, while the remaining nodes represent the ACPI Generic Initiator nodes associated with the assigned GPUs. + +Verify that the GPUs are visible inside the guest: + +```shell +nvidia-smi +``` + +Example output: + +```default ++-----------------------------------------------------------------+ +| NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 | +| CUDA Version: 13.2 | ++-----------------------------------------------------------------+ +| GPU Name Persistence-M | Bus-Id Disp.A | Vol ECC | +| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU Compute | +| | | MIG M | +|=============================+=====================+=============| +| 0 NVIDIA GB200 Off | 00000000:A1:00.0 Off | 0 | +| N/A 38C P0 163W/1200W | 0MiB / 189471MiB | 0% Def | +| | | Disabled | ++-----------------------------+------------------------+----------+ +| 1 NVIDIA GB200 Off | 00000000:A9:00.0 Off | 0 | +| N/A 38C P0 164W/1200W | 0MiB / 189471MiB | 0% Def | +| | | Disabled | ++-----------------------------+------------------------+----------+ +| 2 NVIDIA GB200 Off | 00000000:B1:00.0 Off | 0 | +| N/A 38C P0 206W/1200W | 0MiB / 189471MiB | 0% Def | +| | | Disabled | ++-----------------------------+------------------------+----------+ +| 3 NVIDIA GB200 Off | 00000000:B9:00.0 Off | 0 | +| N/A 38C P0 221W/1200W | 0MiB / 189471MiB | 0% Def | +| | | Disabled | ++-----------------------------+------------------------+----------+ +``` + +Inspect the GPU topology: + +```shell +nvidia-smi topo -m +``` + +Example output: + +```default + GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity GPU NUMA ID +GPU0 X NV18 NV18 NV18 0-71 0 4 +GPU1 NV18 X NV18 NV18 0-71 0 3 +GPU2 NV18 NV18 X NV18 72-143 1 9 +GPU3 NV18 NV18 NV18 X 72-143 1 2 +``` + +This output confirms that: + +* The GPUs are correctly detected inside the guest +* NVLink connectivity is preserved +* GPU affinity matches the corresponding CPU NUMA node and the virtual topology generated by OpenNebula reproduces the physical Grace platform topology. diff --git a/content/product/cluster_configuration/pci_passthrough_sriov/nvidia_vswitch.md b/content/product/cluster_configuration/pci_passthrough_sriov/nvidia_vswitch.md new file mode 100644 index 000000000..a118adee3 --- /dev/null +++ b/content/product/cluster_configuration/pci_passthrough_sriov/nvidia_vswitch.md @@ -0,0 +1,10 @@ +--- +title: "NVIDIA NVSwitch (AE)" +linkTitle: "NVIDIA NVSwitch (AE) " +date: "2026-07-09" +description: +categories: +pageintoc: "58" +tags: ['AI','NVIDIA'] +weight: "8" +--- \ No newline at end of file diff --git a/content/product/cluster_configuration/hosts_and_clusters/one_fabricmanager.md b/content/product/cluster_configuration/pci_passthrough_sriov/one_fabricmanager.md similarity index 97% rename from content/product/cluster_configuration/hosts_and_clusters/one_fabricmanager.md rename to content/product/cluster_configuration/pci_passthrough_sriov/one_fabricmanager.md index fe87f52bd..41b01a689 100644 --- a/content/product/cluster_configuration/hosts_and_clusters/one_fabricmanager.md +++ b/content/product/cluster_configuration/pci_passthrough_sriov/one_fabricmanager.md @@ -1,7 +1,7 @@ --- title: OpenNebula NVIDIA Fabric Manager (AE) linkTitle: NVIDIA Fabric Manager (AE) -weight: 8 +weight: 10 tags: ['AI','NVIDIA'] --- @@ -59,7 +59,7 @@ To begin the installation and configuration, verify these prerequisites: * KVM Node Package: ensures that the `opennebula-kvm-node` EE package is installed on every KVM host that contains NVSwitch devices. This package contains the `opennebula-fabricmanager` service. -* NVSwitch PCI passthrough Setup: the NVSwitch devices must be prepared for PCI passthrough using the vfio-pci driver. Configure these devices at OpenNebula deployment time using one-deploy; check [here]({{% relref "/product/cluster_configuration/hosts_and_clusters/pci_passthrough#vfio-device-binding" %}}) for instructions. If this is not done during deployment, it is possible to manually configure the NVSwitches to use the virtio-pci driver by following the "Hypervisor Configuration" section from [NVIDIA GPU Passthrough]({{% relref "./nvidia_gpu_passthrough" %}}). +* NVSwitch PCI passthrough Setup: the NVSwitch devices must be prepared for PCI passthrough using the vfio-pci driver. Configure these devices at OpenNebula deployment time using one-deploy; check [here]({{% relref "product/cluster_configuration/pci_passthrough_sriov/pci_passthrough#vfio-device-binding" %}}) for instructions. If this is not done during deployment, it is possible to manually configure the NVSwitches to use the virtio-pci driver by following the "Hypervisor Configuration" section from [NVIDIA GPU Passthrough]({{% relref "nvidia_gpu_passthrough" %}}). Once you have validated the prerequisites, start OpenNebula FabricManager service: the `opennebula-fabricmanager` service on the host is disabled by default, as it is designed to be started and stopped on demand or managed by you as the OpenNebula administrator. diff --git a/content/product/cluster_configuration/pci_passthrough_sriov/overview.md b/content/product/cluster_configuration/pci_passthrough_sriov/overview.md new file mode 100644 index 000000000..2470b80f5 --- /dev/null +++ b/content/product/cluster_configuration/pci_passthrough_sriov/overview.md @@ -0,0 +1,78 @@ +--- +title: "Overview" +date: "2026-06-30" +description: +categories: +pageintoc: "52" +tags: +weight: "1" +--- + + +Overview +PCI passthrough allows virtual machines to access physical PCI devices directly, bypassing device emulation. This approach enables workloads to use the full capabilities of specialized hardware such as GPUs, high-performance network adapters, storage controllers, and other accelerators while maintaining the flexibility of virtualization. + +This guide introduces the PCI passthrough architecture in OpenNebula and provides an overview of the supported device types. Detailed configuration procedures are covered in the following guides. + +## PCI Passthrough Architecture + +A PCI passthrough deployment relies on coordinated hardware and software support. Modern processors provide an Input-Output Memory Management Unit (IOMMU), such as Intel VT-d, AMD-Vi, or Arm SMMUv3, to safely map device access to guest memory. The Linux kernel exposes this capability through the VFIO framework, allowing libvirt and QEMU to securely assign physical PCI devices directly to virtual machines. + +OpenNebula manages this process by discovering available PCI devices on each Host, exposing them as Host resources that can be scheduled for virtual machines, selecting suitable Hosts during scheduling, and generating the appropriate virtualization configuration as part of its integrated scheduling, monitoring, and virtual machine management infrastructure. + +OpenNebula abstracts this complexity and allows administrators to manage PCI devices using standard Host and virtual machine templates. + +## Supported Device Types + +OpenNebula supports assigning a wide variety of PCI devices to virtual machines. PCI passthrough is a generic mechanism that can be applied to many types of hardware. However, two categories are of particular relevance and OpenNebula provides enhanced integration and configuration capabilities: + +* **Graphics Processing Units (GPUs)**: GPU passthrough enables virtual machines to use physical GPUs for compute-intensive workloads such as Artificial Intelligence (AI), High Performance Computing (HPC), visualization, and graphics acceleration. OpenNebula supports NVIDIA GPUs through both full PCI passthrough and mediated devices (vGPU), depending on the GPU model and virtualization technology. +* **Network Interfaces**: PCI passthrough can be used with high-performance network adapters to provide direct access to physical networking hardware from virtual machines. OpenNebula supports assigning both Physical Functions (PFs) and Single Root I/O Virtualization (SR-IOV) Virtual Functions (VFs), while maintaining centralized management through OpenNebula. + +## PCI Device Lifecycle in OpenNebula + +OpenNebula manages PCI devices throughout their lifecycle, allowing administrators to manage PCI devices using the same workflow as other OpenNebula resources: + +1. **Host Discovery**: The monitoring subsystem automatically discovers PCI devices available on each virtualization Host and collects information such as vendor, device identifiers, class, NUMA locality, and virtualization capabilities. +2. **Device Scheduling**: PCI devices become schedulable Host resources. During deployment, the scheduler selects a Host satisfying both the virtual machine requirements and the requested PCI devices. +3. **Virtual Machine Deployment**: When a virtual machine is deployed, OpenNebula generates the corresponding libvirt domain configuration, attaches the requested PCI devices, configures the required virtual PCI topology, and enables additional virtualization features when required by the hardware platform. + * The PCI device can also be configured in the VM post VM deployment. Network interfaces can be hot-plugged. +4. **Device Release**: When the virtual machine is terminated, the PCI devices are automatically released and become available for future deployments. + +## Supported Virtualization Features + +OpenNebula supports several PCI virtualization technologies, including: + +* Direct PCI passthrough using VFIO +* SR-IOV Virtual Functions (VFs) in switchdev and legacy mode +* NVIDIA mediated devices (vGPU) and MIG + +Depending on the hardware platform, OpenNebula also supports additional virtualization capabilities, including: + +* IOMMUFD-based device assignment +* NUMA-aware device placement +* Huge Page-backed memory +* Advanced PCI topology generation +* Platform-specific features such as Arm SMMUv3, ACPI Generic Initiator nodes, and dedicated IOMMU devices for NVIDIA Grace platforms + +Not all features are applicable to every device type or hardware platform. Refer to the device-specific guides for detailed compatibility information. + +## Documentation Structure + +The PCI passthrough documentation is organized into the following guides: + +| **Guide** | **Description** | +|-------|-------------| +| [Host Configuration]({{% relref "product/cluster_configuration/pci_passthrough_sriov/host_configuration/" %}}) | Configure virtualization Hosts for PCI passthrough, including IOMMU, VFIO, Huge Pages, SR-IOV, and other common Host requirements. | +| [Network Interfaces]({{% relref "product/cluster_configuration/pci_passthrough_sriov/network_interfaces/" %}}) | Configure PCI passthrough and SR-IOV for network adapters. | +| [Device Passthrough]({{% relref "product/cluster_configuration/pci_passthrough_sriov/pci_passthrough/" %}}) | Configure the passthrough configuration of a given device. | +| [NVIDIA GPUs]({{% relref "product/cluster_configuration/pci_passthrough_sriov/nvidia_gpu_passthrough/" %}}) | Configure NVIDIA GPUs for passthrough and vGPU deployments, including platform-specific features such as Grace and Grace Blackwell. | +| [AMD GPUs]({{% relref "product/cluster_configuration/pci_passthrough_sriov/amd_gpu_passthrough/" %}}) | Configure AMD GPUs for passthrough. | +| [Axelera GPUs]({{% relref "product/cluster_configuration/pci_passthrough_sriov/axelera_gpu_passthrough/" %}}) | Configure Axelera GPUs for passthrough. | +| [NVIDIA Fabric Manager]({{% relref "product/cluster_configuration/pci_passthrough_sriov/one_fabricmanager/" %}}) | Configure NVIDIA Fabric Manager for supported GPU platforms. | + +## Next Steps + +Before assigning PCI devices to Virtual Machines, configure the virtualization Hosts by following the [Host Configuration guide]({{% relref "product/cluster_configuration/pci_passthrough_sriov/host_configuration/" %}}). + +Additional configuration required for specific device types is described in the corresponding device-specific guides. diff --git a/content/product/cluster_configuration/hosts_and_clusters/pci_passthrough.md b/content/product/cluster_configuration/pci_passthrough_sriov/pci_passthrough.md similarity index 99% rename from content/product/cluster_configuration/hosts_and_clusters/pci_passthrough.md rename to content/product/cluster_configuration/pci_passthrough_sriov/pci_passthrough.md index c894deb3f..a0042de06 100644 --- a/content/product/cluster_configuration/hosts_and_clusters/pci_passthrough.md +++ b/content/product/cluster_configuration/pci_passthrough_sriov/pci_passthrough.md @@ -5,7 +5,7 @@ description: categories: pageintoc: "56" tags: -weight: "5" +weight: "4" --- It is possible to discover PCI devices in the Hosts and directly assign them to Virtual Machines in the KVM hypervisor. diff --git a/content/product/cluster_configuration/hosts_and_clusters/vgpu.md b/content/product/cluster_configuration/pci_passthrough_sriov/vgpu.md similarity index 99% rename from content/product/cluster_configuration/hosts_and_clusters/vgpu.md rename to content/product/cluster_configuration/pci_passthrough_sriov/vgpu.md index cc1f60f45..a12b4c0f9 100644 --- a/content/product/cluster_configuration/hosts_and_clusters/vgpu.md +++ b/content/product/cluster_configuration/pci_passthrough_sriov/vgpu.md @@ -6,7 +6,8 @@ description: categories: pageintoc: "57" tags: ['AI','NVIDIA'] -weight: "6" +weight: "4" +toc_hide: true --- diff --git a/content/product/operation_references/hypervisor_configuration/lxc_driver.md b/content/product/operation_references/hypervisor_configuration/lxc_driver.md index 8f3608ebf..ce7142d79 100644 --- a/content/product/operation_references/hypervisor_configuration/lxc_driver.md +++ b/content/product/operation_references/hypervisor_configuration/lxc_driver.md @@ -73,7 +73,7 @@ Some of the VM actions available in the OpenNebula API are not implemented yet f ### PCI Passthrough -You can perform NIC PCI Passthrough on containers. For this you simply need the [PCI NICs to be monitored in the LXC hosts]({{% relref "../../cluster_configuration/hosts_and_clusters/pci_passthrough.md#driver-configuration" %}}). The configuration file for the PCI device detection is `/var/lib/one/remotes/etc/im/lxc-probes.d/pci.conf`. Once monitoring is successful, you can [use this PCI NIC]({{% relref "../../cluster_configuration/hosts_and_clusters/pci_passthrough.md#usage" %}}) +You can perform NIC PCI Passthrough on containers. For this you simply need the [PCI NICs to be monitored in the LXC hosts]({{% relref "../../cluster_configuration/pci_passthrough_sriov/pci_passthrough.md#driver-configuration" %}}). The configuration file for the PCI device detection is `/var/lib/one/remotes/etc/im/lxc-probes.d/pci.conf`. Once monitoring is successful, you can [use this PCI NIC]({{% relref "../../cluster_configuration/pci_passthrough_sriov/pci_passthrough.md#usage" %}}) by declaring in it on the VM Template or hot plugging it. Example PCI device monitoring diff --git a/content/product/virtual_machines_operation/guest_operating_systems/windows_best_practice.md b/content/product/virtual_machines_operation/guest_operating_systems/windows_best_practice.md index 15d812d53..1ab9ed646 100644 --- a/content/product/virtual_machines_operation/guest_operating_systems/windows_best_practice.md +++ b/content/product/virtual_machines_operation/guest_operating_systems/windows_best_practice.md @@ -133,7 +133,7 @@ Under the Inputs section select a Tablet type on USB bus, then click Add. This w Defining a Virtio model display device at a higher resolution can be useful here as well; this can allow higher resolutions in the desktop. -If you are using non-networking PCI Passthrough devices, this is the place to add them too, such as GPUs. See the [PCI Passthrough Guide]({{% relref "../../cluster_configuration/hosts_and_clusters/pci_passthrough#kvm-pci-passthrough" %}}). +If you are using non-networking PCI Passthrough devices, this is the place to add them too, such as GPUs. See the [PCI Passthrough Guide]({{% relref "../../cluster_configuration/pci_passthrough_sriov/pci_passthrough#kvm-pci-passthrough" %}}). #### Tags diff --git a/content/software/release_information/release_notes/platform_notes.md b/content/software/release_information/release_notes/platform_notes.md index 02377666a..7a0a3b581 100644 --- a/content/software/release_information/release_notes/platform_notes.md +++ b/content/software/release_information/release_notes/platform_notes.md @@ -73,22 +73,22 @@ More information: [one-apps wiki](https://github.com/OpenNebula/one-apps/wiki) | **Component** | **Version** | **More information** | |---------------------|--------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------| -| 8021q kernel module | Version included in the Linux distribution | [802.1Q VLAN]({{% relref "../../../product/cluster_configuration/networking_system/vlan#hm-vlan" %}}) | -| Open vSwitch | Version included in the Linux distribution | [Open vSwitch]({{% relref "../../../product/cluster_configuration/networking_system/openvswitch#openvswitch" %}}) | -| iproute2 | Version included in the Linux distribution | [VXLAN]({{% relref "../../../product/cluster_configuration/networking_system/vxlan#vxlan" %}}) | -| Spectrum-X | N/A | [Spectrum-X documentation]({{% relref "../../../product/cluster_configuration/networking_system/spectrumx" %}}) | -| InfiniBand | N/A | [PCI Passthrough]({{% relref "product/cluster_configuration/hosts_and_clusters/pci_passthrough/" %}}) | +| 8021q kernel module | Version included in the Linux distribution | [802.1Q VLAN]({{% relref "product/cluster_configuration/networking_system/vlan#hm-vlan" %}}) | +| Open vSwitch | Version included in the Linux distribution | [Open vSwitch]({{% relref "product/cluster_configuration/networking_system/openvswitch#openvswitch" %}}) | +| iproute2 | Version included in the Linux distribution | [VXLAN]({{% relref "product/cluster_configuration/networking_system/vxlan#vxlan" %}}) | +| Spectrum-X | N/A | [Spectrum-X documentation]({{% relref "product/cluster_configuration/networking_system/spectrumx" %}}) | +| InfiniBand | N/A | [PCI Passthrough]({{% relref "product/cluster_configuration/pci_passthrough_sriov/pci_passthrough/" %}}) | ### Storage Infrastructure | **Component** | **Version** | **More information** | |-----------|--------------------------------------------|--------------------------------------------------------------------------------------------------------------| -| iSCSI | Version included in the Linux distribution | [LVM Drivers]({{% relref "../../../product/cluster_configuration/lvm" %}}) | -| LVM2 | Version included in the Linux distribution | [LVM Drivers]({{% relref "../../../product/cluster_configuration/lvm" %}}) | -| Ceph | Reef v18.2.x
Squid v19.2.x | [The Ceph Datastore]({{% relref "../../../product/cluster_configuration/storage_system/ceph_ds#ceph-ds" %}}) | -| NetApp | ONTAP 9.16.1P1. | [NetApp ONTAP Drivers]({{% relref "../../../product/cluster_configuration/san_storage/netapp" %}}) | +| iSCSI | Version included in the Linux distribution | [LVM Drivers]({{% relref "product/cluster_configuration/lvm" %}}) | +| LVM2 | Version included in the Linux distribution | [LVM Drivers]({{% relref "product/cluster_configuration/lvm" %}}) | +| Ceph | Reef v18.2.x
Squid v19.2.x | [The Ceph Datastore]({{% relref "product/cluster_configuration/storage_system/ceph_ds#ceph-ds" %}}) | +| NetApp | ONTAP 9.16.1P1. | [NetApp ONTAP Drivers]({{% relref "product/cluster_configuration/san_storage/netapp" %}}) | | Everpure | 6.7.2 | [Everpure SAN Datastore]({{% relref "product/cluster_configuration/san_storage/everpure" %}}) | -| LVM-thin | NetApp ONTAP 9.16.1P1 | [LVM Thin]({{% relref "../../../product/cluster_configuration/lvm" %}}) | +| LVM-thin | NetApp ONTAP 9.16.1P1 | [LVM Thin]({{% relref "product/cluster_configuration/lvm" %}}) | ### Accelerated Infrastructure diff --git a/content/software/release_information/release_notes/whats_new.md b/content/software/release_information/release_notes/whats_new.md index 0922010e8..e5ad81aea 100644 --- a/content/software/release_information/release_notes/whats_new.md +++ b/content/software/release_information/release_notes/whats_new.md @@ -83,7 +83,7 @@ Thank you to our incredible community and partners for your continued support in ## Networking * [VLAN Rules]({{% relref "manage_vnets#group-vlan-rules" %}}) enable cloud administrators to delegate VLAN management to tenants in multi-tenant clouds, enabling tenant self-provisioning of Virtual Networks. -* Support for [SR-IOV capable PCI network interfaces in Switchdev mode]({{% relref "product/cluster_configuration/hosts_and_clusters/pci_passthrough/#usage-as-network-interfaces" %}}) with Open vSwitch. +* Support for [SR-IOV capable PCI network interfaces in Switchdev mode]({{% relref "product/cluster_configuration/pci_passthrough_sriov/pci_passthrough/#usage-as-network-interfaces" %}}) with Open vSwitch. ## Packaging diff --git a/content/solutions/ai_factory_blueprints/containerized_ai_execution/ai_ready_k8s.md b/content/solutions/ai_factory_blueprints/containerized_ai_execution/ai_ready_k8s.md index 98176d9d4..38ea9fe98 100644 --- a/content/solutions/ai_factory_blueprints/containerized_ai_execution/ai_ready_k8s.md +++ b/content/solutions/ai_factory_blueprints/containerized_ai_execution/ai_ready_k8s.md @@ -286,7 +286,7 @@ The CAPI appliance takes some minutes to be in “Ready” status. Once the appl EOF ``` - The number of GPU devices mounted in each worker node depends on the definition of the `PCI` attribute in the worker nodes template (note that the attributes of this map could change depending on the GPU model). In our case, we are deploying 2 worker nodes, with 1 GPU attached to each one. In case you only have available a single GPU card, change the number or `WORKER_MACHINE_COUNT` to 1. More information on [NVIDIA GPU Passthrough]({{% relref "product/cluster_configuration/hosts_and_clusters/nvidia_gpu_passthrough.md#pci-device-passthrough" %}}) section. In this example, we will attach a single NVIDIA GPU card to each worker nodes. + The number of GPU devices mounted in each worker node depends on the definition of the `PCI` attribute in the worker nodes template (note that the attributes of this map could change depending on the GPU model). In our case, we are deploying 2 worker nodes, with 1 GPU attached to each one. In case you only have available a single GPU card, change the number or `WORKER_MACHINE_COUNT` to 1. More information on [NVIDIA GPU Passthrough]({{% relref "product/cluster_configuration/pci_passthrough_sriov/nvidia_gpu_passthrough.md#pci-device-passthrough" %}}) section. In this example, we will attach a single NVIDIA GPU card to each worker nodes. 5. Once the `values.yaml` file is available, you can proceed to deploy the workload Cluster with Helm. First, add the helm chart repo for CAPONE and apply the helm chart referencing the values file: diff --git a/content/solutions/ai_factory_blueprints/deployment/cd_on-premises.md b/content/solutions/ai_factory_blueprints/deployment/cd_on-premises.md index 21b485412..1c9998b4f 100644 --- a/content/solutions/ai_factory_blueprints/deployment/cd_on-premises.md +++ b/content/solutions/ai_factory_blueprints/deployment/cd_on-premises.md @@ -41,7 +41,7 @@ If IOMMU is not active, add the appropriate parameter to the kernel's boot comma * For Intel CPUs: `intel_iommu=on` * For AMD CPUs: `amd_iommu=on` -For a detailed guide on how to perform this kernel configuration, refer to the [NVIDIA GPU Passthrough documentation]({{% relref "product/cluster_configuration/hosts_and_clusters/nvidia_gpu_passthrough.md" %}}). +For a detailed guide on how to perform this kernel configuration, refer to the [NVIDIA GPU Passthrough documentation]({{% relref "product/cluster_configuration/pci_passthrough_sriov/nvidia_gpu_passthrough.md" %}}). ### Hypervisor Preparation diff --git a/content/solutions/ai_factory_blueprints/overview.md b/content/solutions/ai_factory_blueprints/overview.md index 4c6097d4b..66544f340 100644 --- a/content/solutions/ai_factory_blueprints/overview.md +++ b/content/solutions/ai_factory_blueprints/overview.md @@ -23,7 +23,7 @@ The purpose of the AI Factory collection is to provide a step-by-step process fo Configuring, deploying and validating a high-performance AI infrastructure using OpenNebula involves these steps: -1. Familiarize yourself with **Architecture and Specifications**. We recommend consulting the [guide on GPU PCI-passthrough]({{% relref "product/cluster_configuration/hosts_and_clusters/nvidia_gpu_passthrough" %}}) for details relating to your GPU hardware and IOMMU. +1. Familiarize yourself with **Architecture and Specifications**. We recommend consulting the [guide on GPU PCI-passthrough]({{% relref "product/cluster_configuration/pci_passthrough_sriov/nvidia_gpu_passthrough" %}}) for details relating to your GPU hardware and IOMMU.
2. Deploy and configure your AI Factory with one of these alternatives: