Repository navigation
fix: Add known issue workaround - #523
Conversation
Signed-off-by: Mike McKiernan <mmckiernan@nvidia.com>
| NVIDIA recommends the following alternatives before using this workaround: | ||
| - Install the DOCA-OFED driver on the host and not use a containerized driver. | ||
| - Use the _inbox_ drivers provided by the operating system. | ||
| - If neither of those alternatives are viable, use this workaround. |
There was a problem hiding this comment.
@tariq1890, is using the cluster policy an additional alternative?
There was a problem hiding this comment.
I am not sure I understand the question, can you clarify?
There was a problem hiding this comment.
I meant if the customer could use cluster policy instead of an NVIDIA driver custom resource, does that somehow dodge the race condition? (Maybe not after thinking about that for a second time.) But perhaps more importantly, is the list missing any actions that a customer could take to dodge the race?
There was a problem hiding this comment.
Yes, the first two options allow for sticking with the ClusterPolicy
Documentation preview |
| Known Issues | ||
| ------------ | ||
|
|
||
| * When GPU Operator and Network Operator manage containerized drivers on the same nodes, GPU driver pods can remain running during a DOCA-OFED update. |
There was a problem hiding this comment.
@mikemckiernan So the race applies more to GPU nodes like HGX B200/B300; machines with NVLink5 or later. For these nodes, the gpu driver container starts up the nvlsm (NVLink Subnet Manager) along with the FabricManager. It's the nvlsm service holding a reference to the ib_umad module that is at the root of this driver deadlock issue. The running nvlsm service is the reason that MOFED-DOCA driver container is blocked from proceeding with its installation process.
| This reference prevents the DOCA-OFED driver container from proceeding with installation. | ||
|
|
||
| NVIDIA recommends the following alternatives before using this workaround: | ||
| - Install the DOCA-OFED driver on the host and not use a containerized driver. |
There was a problem hiding this comment.
Should we add in parenthesis that this is not applicable to OpenShift?
Signed-off-by: Mike McKiernan <mmckiernan@nvidia.com>
d624727 to
2cd4028
Compare
2 participants
No description provided.