Whether a host is healthy or not is currently only exposed to site admins via the machines health status.
The tenant does not have any insight into it.
There are a few cases where the tenant should be aware that the host has a health problem: E.g.
- site-admins want to notify the tenant to evacuate the host so that it can be repaired.
- the tenant/user runs a job scheduler that makes decisions based on external node health signals. E.g.
- SLURM proglue checks could decide whether to run a job on a host depending on NICo reported health.
- Tools like NVSentinel could remove a node automatically from a K8S cluster if reported unhealthy by NICo.
That information should be available via
- Instance gRPC API
- Instance REST API
- Instance Metadata service
A core requirement for the implementation is that not all health alerts on the machine are forwarded, but just a subset. E.g. a tenant should not need to worry if a single network link on a DPU is down as long as the 2nd link is up.
In order to fulfill this case, we already included the field tenant_message in HealthProbeAlert. It was meant to store a message that could be forwarded as is from the machine to the instance API. However we will still need to design further how the health notifications to tenants should look like. E.g.
- Should it be just a single message?
- Do we also need stable probe IDs, classifications, ec?
- How is tenant health status derived from machine health?
Whether a host is healthy or not is currently only exposed to site admins via the machines
healthstatus.The tenant does not have any insight into it.
There are a few cases where the tenant should be aware that the host has a health problem: E.g.
That information should be available via
A core requirement for the implementation is that not all health alerts on the machine are forwarded, but just a subset. E.g. a tenant should not need to worry if a single network link on a DPU is down as long as the 2nd link is up.
In order to fulfill this case, we already included the field
tenant_messageinHealthProbeAlert. It was meant to store a message that could be forwarded as is from the machine to the instance API. However we will still need to design further how the health notifications to tenants should look like. E.g.