Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ONBOARDING.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Every 30 s the plugin re-publishes the node annotation; every `-pulse` seconds i

Notes:

- A request with only `gpumem` still books 100 percent of the cores, so only one such pod runs at a time (HAMi #3192 adds `defaultCores` and `defaultMemory` settings).
- A request with only `gpumem` still books 100 percent of the cores, so only one such pod runs at a time (HAMi #3192, still open, adds `defaultCores` and `defaultMemory` settings).
- Optional pod annotations the HAMi scheduler honors: `amd.com/use-gputype`, `amd.com/nouse-gputype`, `amd.com/use-gpu-uuid`, `amd.com/nouse-gpu-uuid`, `amd.com/numa-bind`, `hami.io/gpu-scheduler-policy`.
- With `-resource_naming_strategy=mixed`, partitioned Instinct GPUs get names like `amd.com/cpx_nps4`.
- Do not put whole-GPU pods and slices on one card when VRAM isolation matters.
Expand All @@ -51,7 +51,7 @@ The node annotation `hami.io/node-amd-register` is a JSON list, one entry per GP
| `devmem`, `devcore` | VRAM in MiB (times `-device_memory_scaling`), active CUs |
| `custominfo` | `pciBDF`, `cuPerWGP` (2 on RDNA), `computeQueues`, `xgmiPeers` (BDFs reachable over XGMI), partition profiles |

Other annotations: `hami.io/amd-operating-mode` and `hami.io/amd-compute-partition` (node, per-node overrides of `OPERATING_MODE` and `COMPUTE_PARTITION`), `hami.io/device-cordon` (node, HAMi skips the listed GPUs).
Other annotations: `hami.io/amd-operating-mode` and `hami.io/amd-compute-partition` (node, per-node overrides of `OPERATING_MODE` and `COMPUTE_PARTITION`), `hami.io/device-cordon` (node, HAMi skips the listed GPUs; needs HAMi #3192, still open).

Extras: Events `AMDGPUUnhealthy`, `AMDGPUMissing`, `AMDGPURecovered`, `AMDGPUModeChangedInUse`; with `-report_node_capacity` the node also gets `amd.com/gpumem` and `amd.com/gpucores` capacity. When no GPU is left to serve, the annotation is cleared.

Expand Down
4 changes: 2 additions & 2 deletions STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ What was checked, what was found and what is left, in plain words. Read [ONBOARD

## In one minute

- We re-ran **every merged pull request** of this repository (131, plus #189 found afterwards) and of [amd-hami-core](https://github.com/Project-HAMi/amd-hami-core) (19) on a real AMD GPU, and the AMD pull requests of [HAMi](https://github.com/Project-HAMi/HAMi) (all closed or merged).
- We re-ran **every merged pull request** of this repository (131, plus #189 found afterwards) and of [amd-hami-core](https://github.com/Project-HAMi/amd-hami-core) (19) on a real AMD GPU, and the AMD pull requests of [HAMi](https://github.com/Project-HAMi/HAMi) (22 closed, 1 open).
- Each pull request has one line below: the result and what is still not verified. "Nothing" means it ran fully on the real GPU.
- What is left: the checks that need hardware this machine did not have (end of this file).

Expand Down Expand Up @@ -84,7 +84,7 @@ These are in the HAMi repository. "Not run on a GPU" means the change was read a
| #3188 | docs(amd): add the AMD GPU guide and examples | Closed, not merged. Re-opened as #3213, which was also closed. All 8 example manifests were applied on the real GPU: the whole-GPU, cores, memory and memory percentage ones ran, the type and UUID ones ran or stayed Pending (CardTypeMismatch, CardUuidMismatch) as the comments say. | The guide text itself was only read. |
| #3190 | feat(amd): add amd.com/priority as AMD_TASK_PRIORITY | Closed, not merged: the maintainer asked for the environment variable only, which already works, so the resource form was dropped. Before closing, on a real node the Pod got the variable and a bad value was rejected. | Effect of the priority under HAMi slices was not re-measured. |
| #3191 | feat(amd): make the topology-aware policy choose the most XGMI-linked GPUs | Merged (includes link-bind). Same fake multi-GPU run: topology-aware chose the linked pair, the same-NUMA pair without links, and the linked NUMA-1 pair together with numa-bind. | Real XGMI hardware and a real multi-GPU node. |
| #3192 | feat(amd): default the GPU count, memory and cores, and honor device cordon | Merged. Unit tests pass. On a real node defaults and cordon worked, four Pods ran with disjoint masks. Two review findings were fixed: a memory request without a limit is read, and an explicit 0 memory is kept as a whole GPU. | A Pod created right after the cordon can still be placed. |
| #3192 | feat(amd): default the GPU count, memory and cores, and honor device cordon | Open, waits for a review. Unit tests pass. On a real node defaults and cordon worked, four Pods ran with disjoint masks. Two review findings were fixed: a memory request without a limit is read, and an explicit 0 memory is kept as a whole GPU. | A Pod created right after the cordon can still be placed. |
| #3193 | test(e2e): add an AMD GPU end-to-end suite | Merged. On a real node 9 specs passed, 0 failed, 2 skipped (re-run on current master). | The two skipped specs and the workflow run were not checked. |
| #3194 | feat(amd): set a configurable runtimeClassName on AMD GPU Pods | Merged. Real Pods: a GPU Pod gets the class, an explicit class is kept, a Pod without a GPU is left alone. | Helm rendering and a missing RuntimeClass are untested. |
| #3212 | fix(amd): read a registration without a type as the AMD type | Merged. Found by re-running #2815: a GPU registered without a type never matched, so its Pods stayed Pending. The new test fails without the change; with it a real Pod is scheduled and runs. | Nothing |
Expand Down
Loading