diff --git a/ONBOARDING.md b/ONBOARDING.md index b94533e9..4ac177de 100644 --- a/ONBOARDING.md +++ b/ONBOARDING.md @@ -34,7 +34,7 @@ Every 30 s the plugin re-publishes the node annotation; every `-pulse` seconds i Notes: -- A request with only `gpumem` still books 100 percent of the cores, so only one such pod runs at a time (HAMi #3192 adds `defaultCores` and `defaultMemory` settings). +- A request with only `gpumem` still books 100 percent of the cores, so only one such pod runs at a time (HAMi #3192, still open, adds `defaultCores` and `defaultMemory` settings). - Optional pod annotations the HAMi scheduler honors: `amd.com/use-gputype`, `amd.com/nouse-gputype`, `amd.com/use-gpu-uuid`, `amd.com/nouse-gpu-uuid`, `amd.com/numa-bind`, `hami.io/gpu-scheduler-policy`. - With `-resource_naming_strategy=mixed`, partitioned Instinct GPUs get names like `amd.com/cpx_nps4`. - Do not put whole-GPU pods and slices on one card when VRAM isolation matters. @@ -51,7 +51,7 @@ The node annotation `hami.io/node-amd-register` is a JSON list, one entry per GP | `devmem`, `devcore` | VRAM in MiB (times `-device_memory_scaling`), active CUs | | `custominfo` | `pciBDF`, `cuPerWGP` (2 on RDNA), `computeQueues`, `xgmiPeers` (BDFs reachable over XGMI), partition profiles | -Other annotations: `hami.io/amd-operating-mode` and `hami.io/amd-compute-partition` (node, per-node overrides of `OPERATING_MODE` and `COMPUTE_PARTITION`), `hami.io/device-cordon` (node, HAMi skips the listed GPUs). +Other annotations: `hami.io/amd-operating-mode` and `hami.io/amd-compute-partition` (node, per-node overrides of `OPERATING_MODE` and `COMPUTE_PARTITION`), `hami.io/device-cordon` (node, HAMi skips the listed GPUs; needs HAMi #3192, still open). Extras: Events `AMDGPUUnhealthy`, `AMDGPUMissing`, `AMDGPURecovered`, `AMDGPUModeChangedInUse`; with `-report_node_capacity` the node also gets `amd.com/gpumem` and `amd.com/gpucores` capacity. When no GPU is left to serve, the annotation is cleared. diff --git a/STATUS.md b/STATUS.md index dfd74431..98f3c17c 100644 --- a/STATUS.md +++ b/STATUS.md @@ -4,7 +4,7 @@ What was checked, what was found and what is left, in plain words. Read [ONBOARD ## In one minute -- We re-ran **every merged pull request** of this repository (131, plus #189 found afterwards) and of [amd-hami-core](https://github.com/Project-HAMi/amd-hami-core) (19) on a real AMD GPU, and the AMD pull requests of [HAMi](https://github.com/Project-HAMi/HAMi) (all closed or merged). +- We re-ran **every merged pull request** of this repository (131, plus #189 found afterwards) and of [amd-hami-core](https://github.com/Project-HAMi/amd-hami-core) (19) on a real AMD GPU, and the AMD pull requests of [HAMi](https://github.com/Project-HAMi/HAMi) (22 closed, 1 open). - Each pull request has one line below: the result and what is still not verified. "Nothing" means it ran fully on the real GPU. - What is left: the checks that need hardware this machine did not have (end of this file). @@ -84,7 +84,7 @@ These are in the HAMi repository. "Not run on a GPU" means the change was read a | #3188 | docs(amd): add the AMD GPU guide and examples | Closed, not merged. Re-opened as #3213, which was also closed. All 8 example manifests were applied on the real GPU: the whole-GPU, cores, memory and memory percentage ones ran, the type and UUID ones ran or stayed Pending (CardTypeMismatch, CardUuidMismatch) as the comments say. | The guide text itself was only read. | | #3190 | feat(amd): add amd.com/priority as AMD_TASK_PRIORITY | Closed, not merged: the maintainer asked for the environment variable only, which already works, so the resource form was dropped. Before closing, on a real node the Pod got the variable and a bad value was rejected. | Effect of the priority under HAMi slices was not re-measured. | | #3191 | feat(amd): make the topology-aware policy choose the most XGMI-linked GPUs | Merged (includes link-bind). Same fake multi-GPU run: topology-aware chose the linked pair, the same-NUMA pair without links, and the linked NUMA-1 pair together with numa-bind. | Real XGMI hardware and a real multi-GPU node. | -| #3192 | feat(amd): default the GPU count, memory and cores, and honor device cordon | Merged. Unit tests pass. On a real node defaults and cordon worked, four Pods ran with disjoint masks. Two review findings were fixed: a memory request without a limit is read, and an explicit 0 memory is kept as a whole GPU. | A Pod created right after the cordon can still be placed. | +| #3192 | feat(amd): default the GPU count, memory and cores, and honor device cordon | Open, waits for a review. Unit tests pass. On a real node defaults and cordon worked, four Pods ran with disjoint masks. Two review findings were fixed: a memory request without a limit is read, and an explicit 0 memory is kept as a whole GPU. | A Pod created right after the cordon can still be placed. | | #3193 | test(e2e): add an AMD GPU end-to-end suite | Merged. On a real node 9 specs passed, 0 failed, 2 skipped (re-run on current master). | The two skipped specs and the workflow run were not checked. | | #3194 | feat(amd): set a configurable runtimeClassName on AMD GPU Pods | Merged. Real Pods: a GPU Pod gets the class, an explicit class is kept, a Pod without a GPU is left alone. | Helm rendering and a missing RuntimeClass are untested. | | #3212 | fix(amd): read a registration without a type as the AMD type | Merged. Found by re-running #2815: a GPU registered without a type never matched, so its Pods stayed Pending. The new test fails without the change; with it a real Pod is scheduled and runs. | Nothing |