From 0902e7862cc5666466642d0bbb1d89a0ee253d6c Mon Sep 17 00:00:00 2001 From: simontesar Date: Tue, 8 Sep 2026 12:16:52 +0200 Subject: [PATCH 01/10] feat: first version of capability matrix Signed-off-by: simontesar --- capability-matrix.md | 124 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 capability-matrix.md diff --git a/capability-matrix.md b/capability-matrix.md new file mode 100644 index 0000000..ca564f0 --- /dev/null +++ b/capability-matrix.md @@ -0,0 +1,124 @@ +# Capability Matrix + +This document specifies sets of capabilities a BMC needs to support to function with the metal-operator. Right now there is essentially one capability per method of the (`bmc.BMC`)[https://github.com/ironcore-dev/metal-operator/blob/f5f9b8121c3180ee40f3c672476ab14f7109562f/bmc/bmc.go#L192]-interface of the metal-operator. + +## Description of columns in capability tables + +| Column | Description | +|---|---| +| **Capability id** | A key identifying the capability. | +| **`BMC` method** | The method of the `BMC` interface in the metal-operator the row stands for. | +| **Level** | How important the capability is. | +| **Redfish** | The Redfish path/action the BMC must expose for the capability to work. | +| **Redfish request body** | An example request payload for write operations. | +| **test** | The Chainsaw test under `tests/` that exercises this today (via metal-operator CRDs). | + +## Conformance levels + +| Level | Description | +|---|---| +| `MUST` | Core functions like `BMC`- `Server`- and `ServerClaim`-reconciliatio or power management cannot complete without it. | +| `SHOULD` | A first-class workflow like `BIOSSettings`- and `ServerMaintenance`-reconciliation or persistent boot order and locator LED depends on it.| +| `OPTIONAL` | Opt-in features like firmware upgrade, account management, event subscriptions and BMC settings. | + +--- + +## PowerController - power on/off/cycle + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | +|---|---|:---:|---|---|---| +| `power.on` | `PowerOn` | MUST | `POST` `#ComputerSystem.Reset` | `{"ResetType": "On"}` | `tests/03-power-annotation` | +| `power.off-graceful` | `PowerOff` | MUST | `POST` `#ComputerSystem.Reset` | `{"ResetType": "GracefulShutdown"}` | `tests/03-power-annotation` | +| `power.off-force` | `ForcePowerOff` | MUST | `POST` `#ComputerSystem.Reset` | `{"ResetType": "ForceOff"}` | `tests/03-power-annotation` | +| `power.wait-state` | `WaitForServerPowerState` | MUST | `GET` `ComputerSystem.PowerState` | - | `tests/03-power-annotation` | + +## BootController - boot order & one-time override + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | +|---|---|:---:|---|---|---| +| `boot.override-pxe` | `SetBootOverride` | MUST | `PATCH` `ComputerSystem` | `{"Boot": {"BootSourceOverrideEnabled": "Once", "BootSourceOverrideTarget": "Pxe", "BootSourceOverrideMode": "UEFI"}}` | `tests/02-discovery` | +| `boot.order-get` | `GetBootOrder` | SHOULD | `GET` `ComputerSystem.Boot.BootOrder` | - | `tests/09-persistent-boot-order` | +| `boot.order-set` | `SetBootOrder` | SHOULD | `PATCH` `ComputerSystem` | `{"Boot": {"BootOrder": ["", "…"], "BootSourceOverrideEnabled": "Continuous", "BootSourceOverrideTarget": "None"}}` | `tests/09-persistent-boot-order` | + +## SystemInspector - inventory + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | +|---|---|:---:|---|---|---| +| `inventory.systems` | `GetSystems` | MUST | `GET` `/redfish/v1/Systems` collection; members need `UUID` and `Boot.BootSourceOverrideTarget` | - | `tests/02-discovery` | +| `inventory.system-info` | `GetSystemInfo` | MUST | `GET` `ComputerSystem` - `Manufacturer`, `Model`, `SerialNumber`, `SKU`, `PowerState`, `Status`, `MemorySummary.TotalSystemMemoryGiB`, `BiosVersion`, `IndicatorLED` | - | `tests/01-bmc-registration`, `tests/02-discovery` | +| `inventory.processors` | `GetProcessors` | SHOULD | `GET` `ComputerSystem/Processors` collection (`Processor`) | - | `tests/02-discovery` | +| `inventory.storages` | `GetStorages` | SHOULD | `GET` `ComputerSystem/Storage` (`Storage`, `Drives`, `Volumes`); fallback `GET` `ComputerSystem/SimpleStorage` | - | `tests/02-discovery` | + +## BIOSManager - BIOS version & attributes + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | +|---|---|:---:|---|---|---| +| `bios.version` | `GetBiosVersion` | MUST | `GET` `ComputerSystem.BiosVersion` | - | `tests/01-bmc-registration` | +| `bios.attr-get` | `GetBiosAttributeValues` | SHOULD | `GET` `ComputerSystem/Bios.Attributes` + `GET` `/redfish/v1/Registries` | - | `tests/06-biossettings-noreboot` | +| `bios.attr-pending` | `GetBiosPendingAttributeValues` | SHOULD | `GET` `Bios` `@Redfish.Settings` + `GET` settings object `.Attributes` | - | `tests/07-biossettings-reboot` | +| `bios.attr-set-on-reset` | `SetBiosAttributesOnReset` | SHOULD | `PATCH` the `Bios` `@Redfish.Settings` settings object| `{"Attributes": {"": "", …}, "@Redfish.SettingsApplyTime": {"ApplyTime": "OnReset"}}` | `tests/06-biossettings-noreboot`, `tests/07-biossettings-reboot` | +| `bios.attr-check` | `CheckBiosAttributes` | SHOULD | `GET` `/redfish/v1/Registries` | - | `tests/06-biossettings-noreboot` | + +## BMCSettingsManager - BMC version & Manager attributes + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | +|---|---|:---:|---|---|---| +| `bmc.version` | `GetBMCVersion` | MUST | `GET` `Manager.FirmwareVersion` | - | `tests/08-bmcsettings` | +| `bmc-settings.attr-get` | `GetBMCAttributeValues` | OPTIONAL | Manager attribute resource / `Manager` `@Redfish.Settings` - vendor-specific implementation | - | `tests/08-bmcsettings` | +| `bmc-settings.attr-pending` | `GetBMCPendingAttributeValues` | OPTIONAL | `GET` `Manager` `@Redfish.Settings` | - | `tests/08-bmcsettings` | +| `bmc-settings.set-immediate` | `SetBMCAttributesImmediately` | OPTIONAL | `PATCH` the `Manager` settings object, vendor-specific implementation | `{"Attributes": {"": "", …}, "@Redfish.SettingsApplyTime": {"ApplyTime": "Immediate"}}` | `tests/08-bmcsettings` | +| `bmc-settings.attr-check` | `CheckBMCAttributes` | OPTIONAL | `GET` `/redfish/v1/Registries` | - | `tests/08-bmcsettings` | + +## FirmwareUpdater - BIOS/BMC firmware upgrade + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | +|---|---|:---:|---|---|---| +| `firmware.bios.upgrade` | `UpgradeBiosVersion` | OPTIONAL | `POST` `#UpdateService.SimpleUpdate` | `{"ImageURI": "", "TransferProtocol": "", "Targets": [""], "@Redfish.OperationApplyTime": "Immediate"}` | uncovered | +| `firmware.bios.task` | `GetBiosUpgradeTask` | OPTIONAL | `GET` task monitor / `TaskService` `Task` (`TaskState`, `TaskStatus`) | - | uncovered | +| `firmware.bmc.upgrade` | `UpgradeBMCVersion` | OPTIONAL | `POST` `#UpdateService.SimpleUpdate` | `{"ImageURI": "", "TransferProtocol": "", "Targets": [""], "@Redfish.OperationApplyTime": "Immediate"}` | uncovered | +| `firmware.bmc.task` | `GetBMCUpgradeTask` | OPTIONAL | `GET` task monitor / `Task` | - | uncovered | +| `firmware.pending-check` | `CheckBMCPendingComponentUpgrade` | OPTIONAL | `GET` `UpdateService/FirmwareInventory` | - | uncovered | + +## ManagerController - the BMC's own Manager resource + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | +|---|---|:---:|---|---|---| +| `manager.get` | `GetManager` | MUST | `GET` `/redfish/v1/Managers` collection; match `Manager.UUID` | - | `tests/01-bmc-registration` | +| `manager.discover` | `DiscoverManager` | SHOULD | `GET` `/redfish/v1/Managers`; `Manager.GraphicalConsole` (`MaxConcurrentSessions`, `ConnectTypesSupported`) | - | uncovered | +| `manager.reset` | `ResetManager` | SHOULD | `POST` `#Manager.Reset` - value must be in `Manager.SupportedResetTypes` | `{"ResetType": "GracefulRestart"}` | `tests/04-bmc-reset` | + +## AccountManager - BMC user accounts + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | +|---|---|:---:|---|---|---| +| `account.create-update` | `CreateOrUpdateAccount` | OPTIONAL | `POST` `AccountService/Accounts`; fallback `PATCH` an existing / empty slot with the same fields | `{"UserName": "", "Password": "", "RoleId": "", "Enabled": true}` | uncovered | +| `account.delete` | `DeleteAccount` | OPTIONAL | `DELETE` the `ManagerAccount` URI or fallback to `PATCH`ing the slot | `{"UserName": "", "Enabled": false}` for PATCH fallback | uncovered | +| `account.list` | `GetAccounts` | OPTIONAL | `GET` `AccountService/Accounts` collection | - | uncovered | +| `account.service` | `GetAccountService` | OPTIONAL | `GET` `/redfish/v1/AccountService` | - | uncovered | + +## IndicatorController - locator LED + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | +|---|---|:---:|---|---|---| +| `indicator.set-led` | `SetIndicatorLED` | SHOULD | `PATCH` `ComputerSystem` | `{"IndicatorLED": "Lit"}` | `tests/05-indicator-led` | + +## EventSubscriber - Redfish event subscriptions + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | +|---|---|:---:|---|---|---| +| `events.subscribe` | `CreateEventSubscription` | OPTIONAL | `GET` `EventService` (`ServiceEnabled`), then `POST` `EventService/Subscriptions`; read the new URI from the `Location` header | `{"Destination": "", "Protocol": "Redfish", "EventFormatType": "Event", "Context": "metal-operator"}` | uncovered | +| `events.unsubscribe` | `DeleteEventSubscription` | OPTIONAL | `DELETE` the `EventDestination` URI | - | uncovered | + +# Coverage summary + +| Chainsaw test | Capabilities touched | +|---|---| +| `tests/01-bmc-registration` | `manager.get`, `inventory.system-info`, `bios.version`, `bmc.version` | +| `tests/02-discovery` | `inventory.systems`, `inventory.system-info`, `inventory.processors`, `inventory.storages`, `power.on`, `boot.override-pxe` | +| `tests/03-power-annotation` | `power.on`, `power.off-graceful`, `power.off-force`, `power.reset`, `power.wait-state` | +| `tests/04-bmc-reset` | `manager.reset` | +| `tests/05-indicator-led` | `indicator.set-led` | +| `tests/06-biossettings-noreboot` | `bios.attr-get`, `bios.attr-set-on-reset`, `bios.attr-check` | +| `tests/07-biossettings-reboot` | `bios.attr-get`, `bios.attr-pending`, `bios.attr-set-on-reset`, `power.reset` | +| `tests/08-bmcsettings` | `bmc.version`, `bmc-settings.attr-get`, `bmc-settings.attr-pending`, `bmc-settings.set-immediate`, `bmc-settings.attr-check` | +| `tests/09-persistent-boot-order` | `boot.order-get`, `boot.order-set` | From 7b45fada9300d75eb5d949a5e29383dce15d027a Mon Sep 17 00:00:00 2001 From: simontesar Date: Wed, 9 Sep 2026 05:21:18 +0200 Subject: [PATCH 02/10] fix: typo in matrix Signed-off-by: simontesar --- capability-matrix.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/capability-matrix.md b/capability-matrix.md index ca564f0..94112cb 100644 --- a/capability-matrix.md +++ b/capability-matrix.md @@ -1,6 +1,6 @@ # Capability Matrix -This document specifies sets of capabilities a BMC needs to support to function with the metal-operator. Right now there is essentially one capability per method of the (`bmc.BMC`)[https://github.com/ironcore-dev/metal-operator/blob/f5f9b8121c3180ee40f3c672476ab14f7109562f/bmc/bmc.go#L192]-interface of the metal-operator. +This document specifies sets of capabilities a BMC needs to support to function with the metal-operator. Right now there is essentially one capability per method of the [`bmc.BMC`](https://github.com/ironcore-dev/metal-operator/blob/f5f9b8121c3180ee40f3c672476ab14f7109562f/bmc/bmc.go#L192)-interface of the metal-operator. ## Description of columns in capability tables From ffdb281b6604f2eca449c0736f290f88b1513444 Mon Sep 17 00:00:00 2001 From: simontesar Date: Wed, 9 Sep 2026 11:06:41 +0200 Subject: [PATCH 03/10] feat: update matrix to not include coverage Signed-off-by: simontesar --- capability-matrix.md | 132 +++++++++++++++++++------------------------ 1 file changed, 59 insertions(+), 73 deletions(-) diff --git a/capability-matrix.md b/capability-matrix.md index 94112cb..d1062ba 100644 --- a/capability-matrix.md +++ b/capability-matrix.md @@ -11,114 +11,100 @@ This document specifies sets of capabilities a BMC needs to support to function | **Level** | How important the capability is. | | **Redfish** | The Redfish path/action the BMC must expose for the capability to work. | | **Redfish request body** | An example request payload for write operations. | -| **test** | The Chainsaw test under `tests/` that exercises this today (via metal-operator CRDs). | ## Conformance levels | Level | Description | |---|---| -| `MUST` | Core functions like `BMC`- `Server`- and `ServerClaim`-reconciliatio or power management cannot complete without it. | -| `SHOULD` | A first-class workflow like `BIOSSettings`- and `ServerMaintenance`-reconciliation or persistent boot order and locator LED depends on it.| +| `MUST` | Core functions like `BMC`-, `Server`- and `ServerClaim`-reconciliation or power management cannot complete without it. | +| `SHOULD` | A first-class workflow like `BIOSSettings`- and `ServerMaintenance`-reconciliation or persistent boot order and locator LED depends on it. | | `OPTIONAL` | Opt-in features like firmware upgrade, account management, event subscriptions and BMC settings. | --- ## PowerController - power on/off/cycle -| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | -|---|---|:---:|---|---|---| -| `power.on` | `PowerOn` | MUST | `POST` `#ComputerSystem.Reset` | `{"ResetType": "On"}` | `tests/03-power-annotation` | -| `power.off-graceful` | `PowerOff` | MUST | `POST` `#ComputerSystem.Reset` | `{"ResetType": "GracefulShutdown"}` | `tests/03-power-annotation` | -| `power.off-force` | `ForcePowerOff` | MUST | `POST` `#ComputerSystem.Reset` | `{"ResetType": "ForceOff"}` | `tests/03-power-annotation` | -| `power.wait-state` | `WaitForServerPowerState` | MUST | `GET` `ComputerSystem.PowerState` | - | `tests/03-power-annotation` | +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `power.on` | `PowerOn` | MUST | `POST` `#ComputerSystem.Reset` | `{"ResetType": "On"}` | +| `power.off-graceful` | `PowerOff` | MUST | `POST` `#ComputerSystem.Reset` | `{"ResetType": "GracefulShutdown"}` | +| `power.off-force` | `ForcePowerOff` | MUST | `POST` `#ComputerSystem.Reset` | `{"ResetType": "ForceOff"}` | +| `power.reset` | `Reset` | MUST | `POST` `#ComputerSystem.Reset` - reset type from `AnnotationToRedfishMapping` | `{"ResetType": "GracefulRestart" \| "ForceRestart" \| "PowerCycle" \| "ForceOff" \| "ForceOn"}` | +| `power.wait-state` | `WaitForServerPowerState` | MUST | `GET` `ComputerSystem.PowerState` | - | ## BootController - boot order & one-time override -| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | -|---|---|:---:|---|---|---| -| `boot.override-pxe` | `SetBootOverride` | MUST | `PATCH` `ComputerSystem` | `{"Boot": {"BootSourceOverrideEnabled": "Once", "BootSourceOverrideTarget": "Pxe", "BootSourceOverrideMode": "UEFI"}}` | `tests/02-discovery` | -| `boot.order-get` | `GetBootOrder` | SHOULD | `GET` `ComputerSystem.Boot.BootOrder` | - | `tests/09-persistent-boot-order` | -| `boot.order-set` | `SetBootOrder` | SHOULD | `PATCH` `ComputerSystem` | `{"Boot": {"BootOrder": ["", "…"], "BootSourceOverrideEnabled": "Continuous", "BootSourceOverrideTarget": "None"}}` | `tests/09-persistent-boot-order` | +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `boot.override-pxe` | `SetBootOverride` | MUST | `PATCH` `ComputerSystem` | `{"Boot": {"BootSourceOverrideEnabled": "Once", "BootSourceOverrideTarget": "Pxe", "BootSourceOverrideMode": "UEFI"}}` | +| `boot.order-get` | `GetBootOrder` | SHOULD | `GET` `ComputerSystem.Boot.BootOrder` | - | +| `boot.order-set` | `SetBootOrder` | SHOULD | `PATCH` `ComputerSystem` | `{"Boot": {"BootOrder": ["", "…"], "BootSourceOverrideEnabled": "Continuous", "BootSourceOverrideTarget": "None"}}` | ## SystemInspector - inventory -| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | -|---|---|:---:|---|---|---| -| `inventory.systems` | `GetSystems` | MUST | `GET` `/redfish/v1/Systems` collection; members need `UUID` and `Boot.BootSourceOverrideTarget` | - | `tests/02-discovery` | -| `inventory.system-info` | `GetSystemInfo` | MUST | `GET` `ComputerSystem` - `Manufacturer`, `Model`, `SerialNumber`, `SKU`, `PowerState`, `Status`, `MemorySummary.TotalSystemMemoryGiB`, `BiosVersion`, `IndicatorLED` | - | `tests/01-bmc-registration`, `tests/02-discovery` | -| `inventory.processors` | `GetProcessors` | SHOULD | `GET` `ComputerSystem/Processors` collection (`Processor`) | - | `tests/02-discovery` | -| `inventory.storages` | `GetStorages` | SHOULD | `GET` `ComputerSystem/Storage` (`Storage`, `Drives`, `Volumes`); fallback `GET` `ComputerSystem/SimpleStorage` | - | `tests/02-discovery` | +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `inventory.systems` | `GetSystems` | MUST | `GET` `/redfish/v1/Systems` collection; members need `UUID` and `Boot.BootSourceOverrideTarget` | - | +| `inventory.system-info` | `GetSystemInfo` | MUST | `GET` `ComputerSystem` - `Manufacturer`, `Model`, `SerialNumber`, `SKU`, `PowerState`, `Status`, `MemorySummary.TotalSystemMemoryGiB`, `BiosVersion`, `IndicatorLED` | - | +| `inventory.processors` | `GetProcessors` | SHOULD | `GET` `ComputerSystem/Processors` collection (`Processor`) | - | +| `inventory.storages` | `GetStorages` | SHOULD | `GET` `ComputerSystem/Storage` (`Storage`, `Drives`, `Volumes`); fallback `GET` `ComputerSystem/SimpleStorage` | - | ## BIOSManager - BIOS version & attributes -| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | -|---|---|:---:|---|---|---| -| `bios.version` | `GetBiosVersion` | MUST | `GET` `ComputerSystem.BiosVersion` | - | `tests/01-bmc-registration` | -| `bios.attr-get` | `GetBiosAttributeValues` | SHOULD | `GET` `ComputerSystem/Bios.Attributes` + `GET` `/redfish/v1/Registries` | - | `tests/06-biossettings-noreboot` | -| `bios.attr-pending` | `GetBiosPendingAttributeValues` | SHOULD | `GET` `Bios` `@Redfish.Settings` + `GET` settings object `.Attributes` | - | `tests/07-biossettings-reboot` | -| `bios.attr-set-on-reset` | `SetBiosAttributesOnReset` | SHOULD | `PATCH` the `Bios` `@Redfish.Settings` settings object| `{"Attributes": {"": "", …}, "@Redfish.SettingsApplyTime": {"ApplyTime": "OnReset"}}` | `tests/06-biossettings-noreboot`, `tests/07-biossettings-reboot` | -| `bios.attr-check` | `CheckBiosAttributes` | SHOULD | `GET` `/redfish/v1/Registries` | - | `tests/06-biossettings-noreboot` | +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `bios.version` | `GetBiosVersion` | MUST | `GET` `ComputerSystem.BiosVersion` | - | +| `bios.attr-get` | `GetBiosAttributeValues` | SHOULD | `GET` `ComputerSystem/Bios.Attributes` + `GET` `/redfish/v1/Registries` | - | +| `bios.attr-pending` | `GetBiosPendingAttributeValues` | SHOULD | `GET` `Bios` `@Redfish.Settings` + `GET` settings object `.Attributes` | - | +| `bios.attr-set-on-reset` | `SetBiosAttributesOnReset` | SHOULD | `PATCH` the `Bios` `@Redfish.Settings` settings object | `{"Attributes": {"": "", …}, "@Redfish.SettingsApplyTime": {"ApplyTime": "OnReset"}}` | +| `bios.attr-check` | `CheckBiosAttributes` | SHOULD | `GET` `/redfish/v1/Registries` | - | ## BMCSettingsManager - BMC version & Manager attributes -| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | -|---|---|:---:|---|---|---| -| `bmc.version` | `GetBMCVersion` | MUST | `GET` `Manager.FirmwareVersion` | - | `tests/08-bmcsettings` | -| `bmc-settings.attr-get` | `GetBMCAttributeValues` | OPTIONAL | Manager attribute resource / `Manager` `@Redfish.Settings` - vendor-specific implementation | - | `tests/08-bmcsettings` | -| `bmc-settings.attr-pending` | `GetBMCPendingAttributeValues` | OPTIONAL | `GET` `Manager` `@Redfish.Settings` | - | `tests/08-bmcsettings` | -| `bmc-settings.set-immediate` | `SetBMCAttributesImmediately` | OPTIONAL | `PATCH` the `Manager` settings object, vendor-specific implementation | `{"Attributes": {"": "", …}, "@Redfish.SettingsApplyTime": {"ApplyTime": "Immediate"}}` | `tests/08-bmcsettings` | -| `bmc-settings.attr-check` | `CheckBMCAttributes` | OPTIONAL | `GET` `/redfish/v1/Registries` | - | `tests/08-bmcsettings` | +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `bmc.version` | `GetBMCVersion` | MUST | `GET` `Manager.FirmwareVersion` | - | +| `bmc-settings.attr-get` | `GetBMCAttributeValues` | OPTIONAL | Manager attribute resource / `Manager` `@Redfish.Settings` - vendor-specific implementation | - | +| `bmc-settings.attr-pending` | `GetBMCPendingAttributeValues` | OPTIONAL | `GET` `Manager` `@Redfish.Settings` | - | +| `bmc-settings.set-immediate` | `SetBMCAttributesImmediately` | OPTIONAL | `PATCH` the `Manager` settings object, vendor-specific implementation | `{"Attributes": {"": "", …}, "@Redfish.SettingsApplyTime": {"ApplyTime": "Immediate"}}` | +| `bmc-settings.attr-check` | `CheckBMCAttributes` | OPTIONAL | `GET` `/redfish/v1/Registries` | - | ## FirmwareUpdater - BIOS/BMC firmware upgrade -| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | -|---|---|:---:|---|---|---| -| `firmware.bios.upgrade` | `UpgradeBiosVersion` | OPTIONAL | `POST` `#UpdateService.SimpleUpdate` | `{"ImageURI": "", "TransferProtocol": "", "Targets": [""], "@Redfish.OperationApplyTime": "Immediate"}` | uncovered | -| `firmware.bios.task` | `GetBiosUpgradeTask` | OPTIONAL | `GET` task monitor / `TaskService` `Task` (`TaskState`, `TaskStatus`) | - | uncovered | -| `firmware.bmc.upgrade` | `UpgradeBMCVersion` | OPTIONAL | `POST` `#UpdateService.SimpleUpdate` | `{"ImageURI": "", "TransferProtocol": "", "Targets": [""], "@Redfish.OperationApplyTime": "Immediate"}` | uncovered | -| `firmware.bmc.task` | `GetBMCUpgradeTask` | OPTIONAL | `GET` task monitor / `Task` | - | uncovered | -| `firmware.pending-check` | `CheckBMCPendingComponentUpgrade` | OPTIONAL | `GET` `UpdateService/FirmwareInventory` | - | uncovered | +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `firmware.bios.upgrade` | `UpgradeBiosVersion` | OPTIONAL | `POST` `#UpdateService.SimpleUpdate` | `{"ImageURI": "", "TransferProtocol": "", "Targets": [""], "@Redfish.OperationApplyTime": "Immediate"}` | +| `firmware.bios.task` | `GetBiosUpgradeTask` | OPTIONAL | `GET` task monitor / `TaskService` `Task` (`TaskState`, `TaskStatus`) | - | +| `firmware.bmc.upgrade` | `UpgradeBMCVersion` | OPTIONAL | `POST` `#UpdateService.SimpleUpdate` | `{"ImageURI": "", "TransferProtocol": "", "Targets": [""], "@Redfish.OperationApplyTime": "Immediate"}` | +| `firmware.bmc.task` | `GetBMCUpgradeTask` | OPTIONAL | `GET` task monitor / `Task` | - | +| `firmware.pending-check` | `CheckBMCPendingComponentUpgrade` | OPTIONAL | `GET` `UpdateService/FirmwareInventory` | - | ## ManagerController - the BMC's own Manager resource -| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | -|---|---|:---:|---|---|---| -| `manager.get` | `GetManager` | MUST | `GET` `/redfish/v1/Managers` collection; match `Manager.UUID` | - | `tests/01-bmc-registration` | -| `manager.discover` | `DiscoverManager` | SHOULD | `GET` `/redfish/v1/Managers`; `Manager.GraphicalConsole` (`MaxConcurrentSessions`, `ConnectTypesSupported`) | - | uncovered | -| `manager.reset` | `ResetManager` | SHOULD | `POST` `#Manager.Reset` - value must be in `Manager.SupportedResetTypes` | `{"ResetType": "GracefulRestart"}` | `tests/04-bmc-reset` | +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `manager.get` | `GetManager` | MUST | `GET` `/redfish/v1/Managers` collection; match `Manager.UUID` | - | +| `manager.discover` | `DiscoverManager` | SHOULD | `GET` `/redfish/v1/Managers`; `Manager.GraphicalConsole` (`MaxConcurrentSessions`, `ConnectTypesSupported`) | - | +| `manager.reset` | `ResetManager` | SHOULD | `POST` `#Manager.Reset` - value must be in `Manager.SupportedResetTypes` | `{"ResetType": "GracefulRestart"}` | ## AccountManager - BMC user accounts -| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | -|---|---|:---:|---|---|---| -| `account.create-update` | `CreateOrUpdateAccount` | OPTIONAL | `POST` `AccountService/Accounts`; fallback `PATCH` an existing / empty slot with the same fields | `{"UserName": "", "Password": "", "RoleId": "", "Enabled": true}` | uncovered | -| `account.delete` | `DeleteAccount` | OPTIONAL | `DELETE` the `ManagerAccount` URI or fallback to `PATCH`ing the slot | `{"UserName": "", "Enabled": false}` for PATCH fallback | uncovered | -| `account.list` | `GetAccounts` | OPTIONAL | `GET` `AccountService/Accounts` collection | - | uncovered | -| `account.service` | `GetAccountService` | OPTIONAL | `GET` `/redfish/v1/AccountService` | - | uncovered | +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `account.create-update` | `CreateOrUpdateAccount` | OPTIONAL | `POST` `AccountService/Accounts`; fallback `PATCH` an existing / empty slot with the same fields | `{"UserName": "", "Password": "", "RoleId": "", "Enabled": true}` | +| `account.delete` | `DeleteAccount` | OPTIONAL | `DELETE` the `ManagerAccount` URI or fallback to `PATCH`ing the slot | `{"UserName": "", "Enabled": false}` for PATCH fallback | +| `account.list` | `GetAccounts` | OPTIONAL | `GET` `AccountService/Accounts` collection | - | +| `account.service` | `GetAccountService` | OPTIONAL | `GET` `/redfish/v1/AccountService` | - | ## IndicatorController - locator LED -| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | -|---|---|:---:|---|---|---| -| `indicator.set-led` | `SetIndicatorLED` | SHOULD | `PATCH` `ComputerSystem` | `{"IndicatorLED": "Lit"}` | `tests/05-indicator-led` | +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `indicator.set-led` | `SetIndicatorLED` | SHOULD | `PATCH` `ComputerSystem` | `{"IndicatorLED": "Lit"}` | ## EventSubscriber - Redfish event subscriptions -| Capability id | `BMC` method | Level | Redfish API | Redfish request body | test | -|---|---|:---:|---|---|---| -| `events.subscribe` | `CreateEventSubscription` | OPTIONAL | `GET` `EventService` (`ServiceEnabled`), then `POST` `EventService/Subscriptions`; read the new URI from the `Location` header | `{"Destination": "", "Protocol": "Redfish", "EventFormatType": "Event", "Context": "metal-operator"}` | uncovered | -| `events.unsubscribe` | `DeleteEventSubscription` | OPTIONAL | `DELETE` the `EventDestination` URI | - | uncovered | - -# Coverage summary - -| Chainsaw test | Capabilities touched | -|---|---| -| `tests/01-bmc-registration` | `manager.get`, `inventory.system-info`, `bios.version`, `bmc.version` | -| `tests/02-discovery` | `inventory.systems`, `inventory.system-info`, `inventory.processors`, `inventory.storages`, `power.on`, `boot.override-pxe` | -| `tests/03-power-annotation` | `power.on`, `power.off-graceful`, `power.off-force`, `power.reset`, `power.wait-state` | -| `tests/04-bmc-reset` | `manager.reset` | -| `tests/05-indicator-led` | `indicator.set-led` | -| `tests/06-biossettings-noreboot` | `bios.attr-get`, `bios.attr-set-on-reset`, `bios.attr-check` | -| `tests/07-biossettings-reboot` | `bios.attr-get`, `bios.attr-pending`, `bios.attr-set-on-reset`, `power.reset` | -| `tests/08-bmcsettings` | `bmc.version`, `bmc-settings.attr-get`, `bmc-settings.attr-pending`, `bmc-settings.set-immediate`, `bmc-settings.attr-check` | -| `tests/09-persistent-boot-order` | `boot.order-get`, `boot.order-set` | +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `events.subscribe` | `CreateEventSubscription` | OPTIONAL | `GET` `EventService` (`ServiceEnabled`), then `POST` `EventService/Subscriptions`; read the new URI from the `Location` header | `{"Destination": "", "Protocol": "Redfish", "EventFormatType": "Event", "Context": "metal-operator"}` | +| `events.unsubscribe` | `DeleteEventSubscription` | OPTIONAL | `DELETE` the `EventDestination` URI | - | From e0bfd2541f9d4a4c673830850fffedda424cdab1 Mon Sep 17 00:00:00 2001 From: simontesar Date: Wed, 9 Sep 2026 11:07:25 +0200 Subject: [PATCH 04/10] feat: add coverage document Signed-off-by: simontesar --- capability-matrix.md | 2 ++ coverage.md | 74 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+) create mode 100644 coverage.md diff --git a/capability-matrix.md b/capability-matrix.md index d1062ba..47ab747 100644 --- a/capability-matrix.md +++ b/capability-matrix.md @@ -2,6 +2,8 @@ This document specifies sets of capabilities a BMC needs to support to function with the metal-operator. Right now there is essentially one capability per method of the [`bmc.BMC`](https://github.com/ironcore-dev/metal-operator/blob/f5f9b8121c3180ee40f3c672476ab14f7109562f/bmc/bmc.go#L192)-interface of the metal-operator. +Which Chainsaw tests cover each capability is tracked in the [coverage document](coverage.md). + ## Description of columns in capability tables | Column | Description | diff --git a/coverage.md b/coverage.md new file mode 100644 index 0000000..42d4635 --- /dev/null +++ b/coverage.md @@ -0,0 +1,74 @@ +# Test Coverage + +This document summarises what tests cover which capabilities in the [capability-matrix](capability-matrix.md). + +* A capability is **covered** when running the test makes the operator invoke the method. +* A capability is verified when the test asserts that it has produced expected values. + +## Common coverage + +Some capabilities are covered by multiple tests implicitly because they share common workflows. +These rows/capabilities include the following placeholders in their `test` field: + +| Placeholder | Description | Capability ids | +|---|---|---| +| `BMC Registration` | Test creates a `BMC` that results in a `Server` resource. | `manager.get`, `inventory.systems`, `inventory.system-info` | +| `Discovery` | Test waits for the `Server` to reach the `Available` state. | `power.on`, `power.off-graceful`, `power.wait-state`, `boot.override-pxe`, `bios.version`, `inventory.processors`, `inventory.storages` | + +--- + +## Coverage by test + +| Chainsaw test | Capability ids covered | +|---|---| +| `tests/01-bmc-registration` | `BMC Registration` | +| `tests/02-discovery` | `BMC Registration`, `Discovery` | +| `tests/03-power-annotation` | `BMC Registration`, `Discovery`, `power.reset` | +| `tests/04-bmc-reset` | `BMC Registration`, `manager.reset` | +| `tests/05-indicator-led` | `BMC Registration`, `Discovery`, `indicator.set-led` | +| `tests/06-biossettings-noreboot` | `BMC Registration`, `Discovery`, `bios.attr-get`, `bios.attr-pending`, `bios.attr-set-on-reset`, `bios.attr-check` | +| `tests/07-biossettings-reboot` | `BMC Registration`, `Discovery`, `bios.attr-get`, `bios.attr-pending`, `bios.attr-set-on-reset`, `bios.attr-check` | +| `tests/08-bmcsettings` | `BMC Registration`, `Discovery`, `bmc-settings.attr-get`, `bmc-settings.attr-pending`, `bmc-settings.set-immediate`, `bmc-settings.attr-check` | +| `tests/09-persistent-boot-order` | `BMC Registration`, `Discovery`, `boot.order-get`, `boot.order-set`, `power.reset` | + +## Coverage by capability + +| Capability id | test | verified | +|---|---|:---:| +| `power.on` | `Discovery` | yes | +| `power.off-graceful` | `Discovery` | yes | +| `power.off-force` | uncovered | - | +| `power.reset` | `tests/03-power-annotation`, `tests/09-persistent-boot-order` | no | +| `power.wait-state` | `Discovery` | yes | +| `boot.override-pxe` | `Discovery` | yes | +| `boot.order-get` | `tests/09-persistent-boot-order` | yes | +| `boot.order-set` | `tests/09-persistent-boot-order` | yes | +| `inventory.systems` | `BMC Registration` | yes | +| `inventory.system-info` | `BMC Registration` | yes | +| `inventory.processors` | `Discovery` | yes | +| `inventory.storages` | `Discovery` | no | +| `bios.version` | `Discovery`, `tests/06-biossettings-noreboot`, `tests/07-biossettings-reboot` | yes | +| `bios.attr-get` | `tests/06-biossettings-noreboot`, `tests/07-biossettings-reboot` | yes | +| `bios.attr-pending` | `tests/06-biossettings-noreboot`, `tests/07-biossettings-reboot` | yes | +| `bios.attr-set-on-reset` | `tests/06-biossettings-noreboot`, `tests/07-biossettings-reboot` | yes | +| `bios.attr-check` | `tests/06-biossettings-noreboot`, `tests/07-biossettings-reboot` | yes | +| `bmc.version` | uncovered | - | +| `bmc-settings.attr-get` | `tests/08-bmcsettings` | yes | +| `bmc-settings.attr-pending` | `tests/08-bmcsettings` | no | +| `bmc-settings.set-immediate` | `tests/08-bmcsettings` | yes | +| `bmc-settings.attr-check` | `tests/08-bmcsettings` | no | +| `firmware.bios.upgrade` | uncovered | - | +| `firmware.bios.task` | uncovered | - | +| `firmware.bmc.upgrade` | uncovered | - | +| `firmware.bmc.task` | uncovered | - | +| `firmware.pending-check` | uncovered | - | +| `manager.get` | `BMC Registration` | yes | +| `manager.discover` | uncovered | - | +| `manager.reset` | `tests/04-bmc-reset` | no | +| `account.create-update` | uncovered | - | +| `account.delete` | uncovered | - | +| `account.list` | uncovered | - | +| `account.service` | uncovered | - | +| `indicator.set-led` | `tests/05-indicator-led` | yes | +| `events.subscribe` | uncovered | - | +| `events.unsubscribe` | uncovered | - | From bcc26e48e2ddac68286fbc43828ca0d9f4c18a2d Mon Sep 17 00:00:00 2001 From: simontesar Date: Thu, 10 Sep 2026 07:00:47 +0200 Subject: [PATCH 05/10] feat: split off to environments.md and update general usage Signed-off-by: simontesar --- README.md | 39 ++++++++++----------------------------- environments.md | 26 ++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 29 deletions(-) create mode 100644 environments.md diff --git a/README.md b/README.md index 45e5086..5bd381a 100644 --- a/README.md +++ b/README.md @@ -1,23 +1,28 @@ # metal-operator test framework +This repository aims to provide a set of capabilities a BMC-implementation needs to support to be useable with the [metal-operator](https://github.com/ironcore-dev/metal-operator), and a set of tests to verify these capabilities. Its goal is to enable BMC vendors or users to run these tests against a BMC with minimal effort and more easily determine what capabilities their BMC may be missing or incorrectly implement. + ## Test suite The `tests` directory contains a suite of tests based on [chainsaw](https://kyverno.github.io/chainsaw/latest/). Every test case creates k8s resources in steps and asserts their status before proceeding to the next steps and implements common metal-operator workflows. Tests are independent from the infrastructure they run on and respect `KUBECONFIG`. ### Requirements * [chainsaw](https://kyverno.github.io/chainsaw/latest/) -* A metal-operator installation and BMC to run tests against. This repository usually uses the locally virtualised [metal-lab](https://github.com/simontesar/metal-lab). +* A metal-operator installation and BMC to run tests against. This repository usually uses the locally virtualised [metal-lab](https://github.com/simontesar/metal-lab) to develop or verify functionality of the actual tests. You can use the lab setup as a reference. ### Usage -The server to run a test against is configured by passing a values file to chainsaw. The default file is `infra/kind/values-basic-go.yaml` that points to a redfish mock setup in the `kind` environment and can be overridden via `VALUES`. +The server to run a test against is configured by passing a values file to chainsaw. The default file is `infra/kind/values-basic-go.yaml` that points to a redfish mock setup in the [`kind` environment](environments.md) and should be overridden via `VALUES`. ```bash -make test # Run all tests -make test/01-bmc-registration # Run a specific test +# Examples +make test # Run all tests +make test/01-bmc-registration # Run a specific test make test/03-power-annotation VALUES=/path/to/metal-lab/values-containerlab-node1.yaml # Run against a specific BMC. ``` +The [`metal-lab`](https://github.com/simontesar/metal-lab) repository provides a standalone containerlab-based environment and ships its own `VALUES` `values-containerlab-node1.yaml` / `values-containerlab-node2.yaml`. Clone the metal-lab repository and refer to its `README` to try the suite of tests without your own BMC. + ### Bring your own BMC -To run tests against your own BMC, copy values into a new file: +To run tests against your own BMC, copy these values into a new file: ```yaml bmcIP: "172.16.100.11" bmcPort: 443 @@ -40,27 +45,3 @@ Adjust the credentials and expectations to their respective values and point the ```bash make make test/02-discovery VALUES=/path/to/new/file.yaml ``` - -### Predefined values -A set of predefined values that point to BMCs deployed via this repository exist in their respective environment's directories: -- `infra/kind/values-basic-go.yaml` -- `infra/kind/values-contoso-go.yaml` - -The `metal-lab` repository (a standalone containerlab-based environment, see below) ships its own equivalent `values-containerlab-node1.yaml` / `values-containerlab-node2.yaml`. - -## Supporting Environments -This repository contains virtualised or containerised infrastructure environments that mock or emulate physical BMC/server nodes. Refer to the `make help` target in every environment's subdirectory for usage. - -### KIND environment -Manages a [kind](https://kind.sigs.k8s.io/) cluster to run the metal-operator and its dependencies. To simulate BMCs and Servers, it runs a Go-based Redfish Mock Server (modified version of the metal-operator's `bmc/mock/main.go`) that supports using system-specific redfish client mock data like the [DMTF mockup server](https://github.com/DMTF/Redfish-Mockup-Server) but has support for dynamic functions like simulating reboots. The server runs once per client data, i.e. BMC. -To simulate booting Servers to run the `metalprobe` tool, a custom `boot-operator`-like implementation runs the metalprobe agent once per discovered `ServerBootConfiguration` and reports back bogus data. This works fine for simple tests. - -### Vagrant environment -Manages a [vagrant](https://developer.hashicorp.com/vagrant)-based environment that provides a network setup as close to a physical environment as possible. Its primary focus is to provide a reference setup for a physical lab. It supports development for scripts, ansible playbooks etc. that can be used to setup actual infrastructure. It does not support running tests against its nodes. **Its current state is a work in progress. It can probably be replaced by a containerlab-based setup.** - -## Caveats -### Dependencies on forks -#### Metal-operator -The `kind`-environment depends on two services that currently live in a [fork of the metal-operator](https://github.com/simontesar/metal-operator): -* A Go implementation of a [redfish mock server](https://github.com/simontesar/metal-operator/blob/dell/bmc/mock/main.go) that is included in the upstream metal-operator for testing but modified in the fork to support some dynamic features like `lastResetTime` and replaceable client mock data to be able to test against mocks of specific BMC models. -* The `metalprobe-mock-controller` that watches `ServerBootConfigs` for BMC mocks and runs a `metalprobe` agent for every instance. It simulates a server controlled by the BMC mock booting up and running `metalprobe` to report to the metal-operator's registry. The reason it lives in the fork is that the `probe`-package of the metal-operator is internal. diff --git a/environments.md b/environments.md new file mode 100644 index 0000000..be975b8 --- /dev/null +++ b/environments.md @@ -0,0 +1,26 @@ +# Environments + +This repository contains virtualised or containerised infrastructure environments that mock or emulate physical BMC/server nodes. They don't relate to running tests against a custom BMC. + + Refer to the `make help` target in every environment's subdirectory for usage. + +## Supporting Environments + +### KIND environment +Manages a [kind](https://kind.sigs.k8s.io/) cluster to run the metal-operator and its dependencies. To simulate BMCs and Servers, it runs a Go-based Redfish Mock Server (modified version of the metal-operator's `bmc/mock/main.go`) that supports using system-specific redfish client mock data like the [DMTF mockup server](https://github.com/DMTF/Redfish-Mockup-Server) but has support for dynamic functions like simulating reboots. The server runs once per client data, i.e. BMC. +To simulate booting Servers to run the `metalprobe` tool, a custom `boot-operator`-like implementation runs the metalprobe agent once per discovered `ServerBootConfiguration` and reports back bogus data. This works fine for simple tests. + +### Vagrant environment +Manages a [vagrant](https://developer.hashicorp.com/vagrant)-based environment that provides a network setup as close to a physical environment as possible. Its primary focus is to provide a reference setup for a physical lab. It supports development for scripts, ansible playbooks etc. that can be used to setup actual infrastructure. It does not support running tests against its nodes. **Its current state is a work in progress. It can probably be replaced by a containerlab-based setup.** + +## Predefined values +A set of predefined values that point to BMCs deployed via this repository exist in their respective environment's directories: +- `infra/kind/values-basic-go.yaml` +- `infra/kind/values-contoso-go.yaml` + +## Caveats +### Dependencies on forks +#### Metal-operator +The `kind`-environment depends on two services that currently live in a [fork of the metal-operator](https://github.com/simontesar/metal-operator): +* A Go implementation of a [redfish mock server](https://github.com/simontesar/metal-operator/blob/dell/bmc/mock/main.go) that is included in the upstream metal-operator for testing but modified in the fork to support some dynamic features like `lastResetTime` and replaceable client mock data to be able to test against mocks of specific BMC models. +* The `metalprobe-mock-controller` that watches `ServerBootConfigs` for BMC mocks and runs a `metalprobe` agent for every instance. It simulates a server controlled by the BMC mock booting up and running `metalprobe` to report to the metal-operator's registry. The reason it lives in the fork is that the `probe`-package of the metal-operator is internal. From c7881beb8d0ce39348b10148dd73d92e8b63f3f4 Mon Sep 17 00:00:00 2001 From: simontesar Date: Thu, 10 Sep 2026 07:03:26 +0200 Subject: [PATCH 06/10] feat: add coverage document Signed-off-by: simontesar --- README.md | 3 +++ coverage.md | 27 +++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/README.md b/README.md index 5bd381a..08bf402 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,9 @@ This repository aims to provide a set of capabilities a BMC-implementation needs to support to be useable with the [metal-operator](https://github.com/ironcore-dev/metal-operator), and a set of tests to verify these capabilities. Its goal is to enable BMC vendors or users to run these tests against a BMC with minimal effort and more easily determine what capabilities their BMC may be missing or incorrectly implement. +## Capabilities +Refer to [capability matrix](capability-matrix.md) for a list of capabilities and their descriptions. Refer to the [coverage document](coverage.md) for what tests cover which capabilities. Determine what your BMC might not implement by referring to a failed test. + ## Test suite The `tests` directory contains a suite of tests based on [chainsaw](https://kyverno.github.io/chainsaw/latest/). Every test case creates k8s resources in steps and asserts their status before proceeding to the next steps and implements common metal-operator workflows. Tests are independent from the infrastructure they run on and respect `KUBECONFIG`. diff --git a/coverage.md b/coverage.md index 42d4635..c99a848 100644 --- a/coverage.md +++ b/coverage.md @@ -72,3 +72,30 @@ These rows/capabilities include the following placeholders in their `test` field | `indicator.set-led` | `tests/05-indicator-led` | yes | | `events.subscribe` | uncovered | - | | `events.unsubscribe` | uncovered | - | + +**37 capabilities: 23 covered (18 verified, 5 not), 14 uncovered.** + +## Uncovered capabilities + +Nothing here is uncovered merely because its result goes unasserted - in every case no test causes +the method to be called at all. + +| Capability | `BMC` method | Why it is not reached | What would cover it | +|---|---|---|---| +| `power.off-force` | `ForcePowerOff` | needs a graceful shutdown to time out with `--enforce-power-off` set | a mock BMC that ignores `GracefulShutdown` | +| `bmc.version` | `GetBMCVersion` | only called by `BMCVersionReconciler` | a test that creates a `BMCVersion` CR | +| `manager.discover` | `DiscoverManager` | only called by `EndpointReconciler`; registration is IP-based | add `macAddress` to `tests/shared/bmc.yaml` so an `Endpoint` is created | +| `firmware.*` (5 rows) | `FirmwareUpdater` | no `BIOSVersion` / `BMCVersion` CR exists | those CRs plus a reachable firmware image | +| `account.*` (4 rows) | `AccountManager` | no `BMCUser` CR exists | a test that creates a `BMCUser` | +| `events.*` (2 rows) | `EventSubscriber` | operator is not started with `--event-url` | set the flag and assert `status.eventsSubscriptionLink` | + +## Covered but not verified + +Reached by a test, but the test would still pass against a BMC that accepts the call and does nothing. + +| Capability | test | Gap | +|---|---|---| +| `power.reset` | 03, 09 | asserts only annotation removal; the `ServerClaim` independently guarantees the asserted power state | +| `manager.reset` | 04 | asserts `lastResetTime != null`, not that it advanced | +| `inventory.storages` | `Discovery` | storage inventory is read but never asserted | +| `bmc-settings.attr-pending`, `bmc-settings.attr-check` | 08 | no-ops on HPE/Lenovo, error on the base client; only Dell exercises real Redfish | From 8d1160623edfe63620facfcc71d8bf1e46529db3 Mon Sep 17 00:00:00 2001 From: simontesar Date: Thu, 10 Sep 2026 08:13:00 +0200 Subject: [PATCH 07/10] feat: matrix: link to interfaces Signed-off-by: simontesar --- capability-matrix.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/capability-matrix.md b/capability-matrix.md index 47ab747..c15170c 100644 --- a/capability-matrix.md +++ b/capability-matrix.md @@ -23,8 +23,9 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `OPTIONAL` | Opt-in features like firmware upgrade, account management, event subscriptions and BMC settings. | --- +# Interfaces -## PowerController - power on/off/cycle +## [PowerController](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L44) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -34,7 +35,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `power.reset` | `Reset` | MUST | `POST` `#ComputerSystem.Reset` - reset type from `AnnotationToRedfishMapping` | `{"ResetType": "GracefulRestart" \| "ForceRestart" \| "PowerCycle" \| "ForceOff" \| "ForceOn"}` | | `power.wait-state` | `WaitForServerPowerState` | MUST | `GET` `ComputerSystem.PowerState` | - | -## BootController - boot order & one-time override +## [BootController](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L62) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -42,7 +43,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `boot.order-get` | `GetBootOrder` | SHOULD | `GET` `ComputerSystem.Boot.BootOrder` | - | | `boot.order-set` | `SetBootOrder` | SHOULD | `PATCH` `ComputerSystem` | `{"Boot": {"BootOrder": ["", "…"], "BootSourceOverrideEnabled": "Continuous", "BootSourceOverrideTarget": "None"}}` | -## SystemInspector - inventory +## [SystemInspector](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L76) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -51,7 +52,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `inventory.processors` | `GetProcessors` | SHOULD | `GET` `ComputerSystem/Processors` collection (`Processor`) | - | | `inventory.storages` | `GetStorages` | SHOULD | `GET` `ComputerSystem/Storage` (`Storage`, `Drives`, `Volumes`); fallback `GET` `ComputerSystem/SimpleStorage` | - | -## BIOSManager - BIOS version & attributes +## [BIOSManager](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L91) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -61,7 +62,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `bios.attr-set-on-reset` | `SetBiosAttributesOnReset` | SHOULD | `PATCH` the `Bios` `@Redfish.Settings` settings object | `{"Attributes": {"": "", …}, "@Redfish.SettingsApplyTime": {"ApplyTime": "OnReset"}}` | | `bios.attr-check` | `CheckBiosAttributes` | SHOULD | `GET` `/redfish/v1/Registries` | - | -## BMCSettingsManager - BMC version & Manager attributes +## [BMCSettingsManager](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L109) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -71,7 +72,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `bmc-settings.set-immediate` | `SetBMCAttributesImmediately` | OPTIONAL | `PATCH` the `Manager` settings object, vendor-specific implementation | `{"Attributes": {"": "", …}, "@Redfish.SettingsApplyTime": {"ApplyTime": "Immediate"}}` | | `bmc-settings.attr-check` | `CheckBMCAttributes` | OPTIONAL | `GET` `/redfish/v1/Registries` | - | -## FirmwareUpdater - BIOS/BMC firmware upgrade +## [FirmwareUpdater](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L128) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -81,7 +82,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `firmware.bmc.task` | `GetBMCUpgradeTask` | OPTIONAL | `GET` task monitor / `Task` | - | | `firmware.pending-check` | `CheckBMCPendingComponentUpgrade` | OPTIONAL | `GET` `UpdateService/FirmwareInventory` | - | -## ManagerController - the BMC's own Manager resource +## [ManagerController](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L147) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -89,7 +90,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `manager.discover` | `DiscoverManager` | SHOULD | `GET` `/redfish/v1/Managers`; `Manager.GraphicalConsole` (`MaxConcurrentSessions`, `ConnectTypesSupported`) | - | | `manager.reset` | `ResetManager` | SHOULD | `POST` `#Manager.Reset` - value must be in `Manager.SupportedResetTypes` | `{"ResetType": "GracefulRestart"}` | -## AccountManager - BMC user accounts +## [AccountManager](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L159) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -98,13 +99,13 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `account.list` | `GetAccounts` | OPTIONAL | `GET` `AccountService/Accounts` collection | - | | `account.service` | `GetAccountService` | OPTIONAL | `GET` `/redfish/v1/AccountService` | - | -## IndicatorController - locator LED +## [IndicatorController](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L174) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| | `indicator.set-led` | `SetIndicatorLED` | SHOULD | `PATCH` `ComputerSystem` | `{"IndicatorLED": "Lit"}` | -## EventSubscriber - Redfish event subscriptions +## [EventSubscriber](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L180) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| From b713d6a4107e546eaeded80b8386803a5a092d8c Mon Sep 17 00:00:00 2001 From: simontesar Date: Thu, 10 Sep 2026 08:23:45 +0200 Subject: [PATCH 08/10] feat: matrix: link to v0.7.1 Signed-off-by: simontesar --- capability-matrix.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/capability-matrix.md b/capability-matrix.md index c15170c..5762b50 100644 --- a/capability-matrix.md +++ b/capability-matrix.md @@ -1,6 +1,6 @@ # Capability Matrix -This document specifies sets of capabilities a BMC needs to support to function with the metal-operator. Right now there is essentially one capability per method of the [`bmc.BMC`](https://github.com/ironcore-dev/metal-operator/blob/f5f9b8121c3180ee40f3c672476ab14f7109562f/bmc/bmc.go#L192)-interface of the metal-operator. +This document specifies sets of capabilities a BMC needs to support to function with the metal-operator. Right now there is essentially one capability per method of the [`bmc.BMC`](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L191)-interface of the metal-operator. Which Chainsaw tests cover each capability is tracked in the [coverage document](coverage.md). @@ -25,7 +25,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] --- # Interfaces -## [PowerController](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L44) +## [PowerController](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L43) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -35,7 +35,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `power.reset` | `Reset` | MUST | `POST` `#ComputerSystem.Reset` - reset type from `AnnotationToRedfishMapping` | `{"ResetType": "GracefulRestart" \| "ForceRestart" \| "PowerCycle" \| "ForceOff" \| "ForceOn"}` | | `power.wait-state` | `WaitForServerPowerState` | MUST | `GET` `ComputerSystem.PowerState` | - | -## [BootController](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L62) +## [BootController](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L61) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -43,7 +43,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `boot.order-get` | `GetBootOrder` | SHOULD | `GET` `ComputerSystem.Boot.BootOrder` | - | | `boot.order-set` | `SetBootOrder` | SHOULD | `PATCH` `ComputerSystem` | `{"Boot": {"BootOrder": ["", "…"], "BootSourceOverrideEnabled": "Continuous", "BootSourceOverrideTarget": "None"}}` | -## [SystemInspector](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L76) +## [SystemInspector](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L75) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -52,7 +52,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `inventory.processors` | `GetProcessors` | SHOULD | `GET` `ComputerSystem/Processors` collection (`Processor`) | - | | `inventory.storages` | `GetStorages` | SHOULD | `GET` `ComputerSystem/Storage` (`Storage`, `Drives`, `Volumes`); fallback `GET` `ComputerSystem/SimpleStorage` | - | -## [BIOSManager](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L91) +## [BIOSManager](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L90) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -62,7 +62,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `bios.attr-set-on-reset` | `SetBiosAttributesOnReset` | SHOULD | `PATCH` the `Bios` `@Redfish.Settings` settings object | `{"Attributes": {"": "", …}, "@Redfish.SettingsApplyTime": {"ApplyTime": "OnReset"}}` | | `bios.attr-check` | `CheckBiosAttributes` | SHOULD | `GET` `/redfish/v1/Registries` | - | -## [BMCSettingsManager](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L109) +## [BMCSettingsManager](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L108) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -72,7 +72,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `bmc-settings.set-immediate` | `SetBMCAttributesImmediately` | OPTIONAL | `PATCH` the `Manager` settings object, vendor-specific implementation | `{"Attributes": {"": "", …}, "@Redfish.SettingsApplyTime": {"ApplyTime": "Immediate"}}` | | `bmc-settings.attr-check` | `CheckBMCAttributes` | OPTIONAL | `GET` `/redfish/v1/Registries` | - | -## [FirmwareUpdater](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L128) +## [FirmwareUpdater](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L127) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -82,7 +82,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `firmware.bmc.task` | `GetBMCUpgradeTask` | OPTIONAL | `GET` task monitor / `Task` | - | | `firmware.pending-check` | `CheckBMCPendingComponentUpgrade` | OPTIONAL | `GET` `UpdateService/FirmwareInventory` | - | -## [ManagerController](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L147) +## [ManagerController](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L146) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -90,7 +90,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `manager.discover` | `DiscoverManager` | SHOULD | `GET` `/redfish/v1/Managers`; `Manager.GraphicalConsole` (`MaxConcurrentSessions`, `ConnectTypesSupported`) | - | | `manager.reset` | `ResetManager` | SHOULD | `POST` `#Manager.Reset` - value must be in `Manager.SupportedResetTypes` | `{"ResetType": "GracefulRestart"}` | -## [AccountManager](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L159) +## [AccountManager](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L158) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| @@ -99,13 +99,13 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | `account.list` | `GetAccounts` | OPTIONAL | `GET` `AccountService/Accounts` collection | - | | `account.service` | `GetAccountService` | OPTIONAL | `GET` `/redfish/v1/AccountService` | - | -## [IndicatorController](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L174) +## [IndicatorController](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L173) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| | `indicator.set-led` | `SetIndicatorLED` | SHOULD | `PATCH` `ComputerSystem` | `{"IndicatorLED": "Lit"}` | -## [EventSubscriber](https://github.com/ironcore-dev/metal-operator/blob/bd0873d4ca3803fbb78232b85794d7ccc8e9d5e6/bmc/bmc.go#L180) +## [EventSubscriber](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L179) | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| From 135e0b06fdd7d45cb6470c398a3d9dc8ec49ce77 Mon Sep 17 00:00:00 2001 From: simontesar Date: Thu, 10 Sep 2026 11:38:25 +0200 Subject: [PATCH 09/10] fix: remove note part in coverage Signed-off-by: simontesar --- coverage.md | 26 -------------------------- 1 file changed, 26 deletions(-) diff --git a/coverage.md b/coverage.md index c99a848..a08e062 100644 --- a/coverage.md +++ b/coverage.md @@ -73,29 +73,3 @@ These rows/capabilities include the following placeholders in their `test` field | `events.subscribe` | uncovered | - | | `events.unsubscribe` | uncovered | - | -**37 capabilities: 23 covered (18 verified, 5 not), 14 uncovered.** - -## Uncovered capabilities - -Nothing here is uncovered merely because its result goes unasserted - in every case no test causes -the method to be called at all. - -| Capability | `BMC` method | Why it is not reached | What would cover it | -|---|---|---|---| -| `power.off-force` | `ForcePowerOff` | needs a graceful shutdown to time out with `--enforce-power-off` set | a mock BMC that ignores `GracefulShutdown` | -| `bmc.version` | `GetBMCVersion` | only called by `BMCVersionReconciler` | a test that creates a `BMCVersion` CR | -| `manager.discover` | `DiscoverManager` | only called by `EndpointReconciler`; registration is IP-based | add `macAddress` to `tests/shared/bmc.yaml` so an `Endpoint` is created | -| `firmware.*` (5 rows) | `FirmwareUpdater` | no `BIOSVersion` / `BMCVersion` CR exists | those CRs plus a reachable firmware image | -| `account.*` (4 rows) | `AccountManager` | no `BMCUser` CR exists | a test that creates a `BMCUser` | -| `events.*` (2 rows) | `EventSubscriber` | operator is not started with `--event-url` | set the flag and assert `status.eventsSubscriptionLink` | - -## Covered but not verified - -Reached by a test, but the test would still pass against a BMC that accepts the call and does nothing. - -| Capability | test | Gap | -|---|---|---| -| `power.reset` | 03, 09 | asserts only annotation removal; the `ServerClaim` independently guarantees the asserted power state | -| `manager.reset` | 04 | asserts `lastResetTime != null`, not that it advanced | -| `inventory.storages` | `Discovery` | storage inventory is read but never asserted | -| `bmc-settings.attr-pending`, `bmc-settings.attr-check` | 08 | no-ops on HPE/Lenovo, error on the base client; only Dell exercises real Redfish | From 130df0ad8d4ec73efc873519a52dc7fd8b16a373 Mon Sep 17 00:00:00 2001 From: simontesar Date: Thu, 10 Sep 2026 11:39:06 +0200 Subject: [PATCH 10/10] fix: typos and grammar Signed-off-by: simontesar --- README.md | 14 +++++++------- capability-matrix.md | 4 ++-- coverage.md | 4 ++-- environments.md | 12 ++++++------ 4 files changed, 17 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 08bf402..8e98f7c 100644 --- a/README.md +++ b/README.md @@ -1,25 +1,25 @@ # metal-operator test framework -This repository aims to provide a set of capabilities a BMC-implementation needs to support to be useable with the [metal-operator](https://github.com/ironcore-dev/metal-operator), and a set of tests to verify these capabilities. Its goal is to enable BMC vendors or users to run these tests against a BMC with minimal effort and more easily determine what capabilities their BMC may be missing or incorrectly implement. +This repository aims to provide a set of capabilities a BMC-implementation needs to support to be usable with the [metal-operator](https://github.com/ironcore-dev/metal-operator), and a set of tests to verify these capabilities. Its goal is to enable BMC vendors or users to run these tests against a BMC with minimal effort and more easily determine what capabilities their BMC may be missing or implementing incorrectly. ## Capabilities Refer to [capability matrix](capability-matrix.md) for a list of capabilities and their descriptions. Refer to the [coverage document](coverage.md) for what tests cover which capabilities. Determine what your BMC might not implement by referring to a failed test. ## Test suite -The `tests` directory contains a suite of tests based on [chainsaw](https://kyverno.github.io/chainsaw/latest/). Every test case creates k8s resources in steps and asserts their status before proceeding to the next steps and implements common metal-operator workflows. Tests are independent from the infrastructure they run on and respect `KUBECONFIG`. +The `tests` directory contains a suite of tests based on [chainsaw](https://kyverno.github.io/chainsaw/latest/). Every test case creates k8s resources in steps and asserts their status before proceeding. Tests are independent of from the infrastructure they run on and respect `KUBECONFIG`. ### Requirements * [chainsaw](https://kyverno.github.io/chainsaw/latest/) * A metal-operator installation and BMC to run tests against. This repository usually uses the locally virtualised [metal-lab](https://github.com/simontesar/metal-lab) to develop or verify functionality of the actual tests. You can use the lab setup as a reference. ### Usage -The server to run a test against is configured by passing a values file to chainsaw. The default file is `infra/kind/values-basic-go.yaml` that points to a redfish mock setup in the [`kind` environment](environments.md) and should be overridden via `VALUES`. +The server to run a test against is configured by passing a values file to chainsaw. The default file is `infra/kind/values-basic-go.yaml` which points to a Redfish mock setup in the [`kind` environment](environments.md) and should be overridden via `VALUES`. ```bash -# Examples +# Examples make test # Run all tests make test/01-bmc-registration # Run a specific test -make test/03-power-annotation VALUES=/path/to/metal-lab/values-containerlab-node1.yaml # Run against a specific BMC. +make test/03-power-annotation VALUES=/path/to/metal-lab/values-containerlab-node1.yaml # Run against a specific BMC ``` The [`metal-lab`](https://github.com/simontesar/metal-lab) repository provides a standalone containerlab-based environment and ships its own `VALUES` `values-containerlab-node1.yaml` / `values-containerlab-node2.yaml`. Clone the metal-lab repository and refer to its `README` to try the suite of tests without your own BMC. @@ -43,8 +43,8 @@ bmcSettingKey: "EmailAlert.1.Address" bmcSettingValue: "alerts@example.com" ``` -Adjust the credentials and expectations to their respective values and point the tests to it: +Adjust the credentials and expectations to your respective values and point the tests to it: ```bash -make make test/02-discovery VALUES=/path/to/new/file.yaml +make test/02-discovery VALUES=/path/to/new/file.yaml ``` diff --git a/capability-matrix.md b/capability-matrix.md index 5762b50..0cb6996 100644 --- a/capability-matrix.md +++ b/capability-matrix.md @@ -11,7 +11,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | **Capability id** | A key identifying the capability. | | **`BMC` method** | The method of the `BMC` interface in the metal-operator the row stands for. | | **Level** | How important the capability is. | -| **Redfish** | The Redfish path/action the BMC must expose for the capability to work. | +| **Redfish API** | The Redfish path/action the BMC must expose for the capability to work. | | **Redfish request body** | An example request payload for write operations. | ## Conformance levels @@ -95,7 +95,7 @@ Which Chainsaw tests cover each capability is tracked in the [coverage document] | Capability id | `BMC` method | Level | Redfish API | Redfish request body | |---|---|:---:|---|---| | `account.create-update` | `CreateOrUpdateAccount` | OPTIONAL | `POST` `AccountService/Accounts`; fallback `PATCH` an existing / empty slot with the same fields | `{"UserName": "", "Password": "", "RoleId": "", "Enabled": true}` | -| `account.delete` | `DeleteAccount` | OPTIONAL | `DELETE` the `ManagerAccount` URI or fallback to `PATCH`ing the slot | `{"UserName": "", "Enabled": false}` for PATCH fallback | +| `account.delete` | `DeleteAccount` | OPTIONAL | `DELETE` the `ManagerAccount` URI or fall back to `PATCH`ing the slot | `{"UserName": "", "Enabled": false}` for PATCH fallback | | `account.list` | `GetAccounts` | OPTIONAL | `GET` `AccountService/Accounts` collection | - | | `account.service` | `GetAccountService` | OPTIONAL | `GET` `/redfish/v1/AccountService` | - | diff --git a/coverage.md b/coverage.md index a08e062..43ebe43 100644 --- a/coverage.md +++ b/coverage.md @@ -3,7 +3,7 @@ This document summarises what tests cover which capabilities in the [capability-matrix](capability-matrix.md). * A capability is **covered** when running the test makes the operator invoke the method. -* A capability is verified when the test asserts that it has produced expected values. +* A capability is **verified** when the test asserts that it has produced expected values. ## Common coverage @@ -33,7 +33,7 @@ These rows/capabilities include the following placeholders in their `test` field ## Coverage by capability -| Capability id | test | verified | +| Capability ID | Test | Verified | |---|---|:---:| | `power.on` | `Discovery` | yes | | `power.off-graceful` | `Discovery` | yes | diff --git a/environments.md b/environments.md index be975b8..30ca40d 100644 --- a/environments.md +++ b/environments.md @@ -2,19 +2,19 @@ This repository contains virtualised or containerised infrastructure environments that mock or emulate physical BMC/server nodes. They don't relate to running tests against a custom BMC. - Refer to the `make help` target in every environment's subdirectory for usage. +Refer to the `make help` target in every environment's subdirectory for usage. ## Supporting Environments ### KIND environment -Manages a [kind](https://kind.sigs.k8s.io/) cluster to run the metal-operator and its dependencies. To simulate BMCs and Servers, it runs a Go-based Redfish Mock Server (modified version of the metal-operator's `bmc/mock/main.go`) that supports using system-specific redfish client mock data like the [DMTF mockup server](https://github.com/DMTF/Redfish-Mockup-Server) but has support for dynamic functions like simulating reboots. The server runs once per client data, i.e. BMC. -To simulate booting Servers to run the `metalprobe` tool, a custom `boot-operator`-like implementation runs the metalprobe agent once per discovered `ServerBootConfiguration` and reports back bogus data. This works fine for simple tests. +Manages a [kind](https://kind.sigs.k8s.io/) cluster to run the metal-operator and its dependencies. To simulate BMCs and Servers, it runs a Go-based Redfish mock server (a modified version of the metal-operator's `bmc/mock/main.go`) that supports using system-specific Redfish client mock data like the [DMTF mockup server](https://github.com/DMTF/Redfish-Mockup-Server) but additionally implements dynamic functions like simulating reboots. The server runs once per client data set, i.e. BMC. +To simulate booting Servers to run the `metalprobe` tool, a custom `boot-operator`-like implementation runs the `metalprobe` agent once per discovered `ServerBootConfiguration` and reports back bogus data. This works fine for simple tests. ### Vagrant environment -Manages a [vagrant](https://developer.hashicorp.com/vagrant)-based environment that provides a network setup as close to a physical environment as possible. Its primary focus is to provide a reference setup for a physical lab. It supports development for scripts, ansible playbooks etc. that can be used to setup actual infrastructure. It does not support running tests against its nodes. **Its current state is a work in progress. It can probably be replaced by a containerlab-based setup.** +Manages a [vagrant](https://developer.hashicorp.com/vagrant)-based environment that provides a network setup as close to a physical environment as possible. Its primary focus is to provide a reference setup for a physical lab. It supports development for scripts, Ansible playbooks etc. that can be used to set up actual infrastructure. It does not support running tests against its nodes. **Its current state is a work in progress. It can probably be replaced by a containerlab-based setup.** ## Predefined values -A set of predefined values that point to BMCs deployed via this repository exist in their respective environment's directories: +Two predefined value files that point to the DMTF- or Go-based mock exist for the `kind`-environment: - `infra/kind/values-basic-go.yaml` - `infra/kind/values-contoso-go.yaml` @@ -23,4 +23,4 @@ A set of predefined values that point to BMCs deployed via this repository exist #### Metal-operator The `kind`-environment depends on two services that currently live in a [fork of the metal-operator](https://github.com/simontesar/metal-operator): * A Go implementation of a [redfish mock server](https://github.com/simontesar/metal-operator/blob/dell/bmc/mock/main.go) that is included in the upstream metal-operator for testing but modified in the fork to support some dynamic features like `lastResetTime` and replaceable client mock data to be able to test against mocks of specific BMC models. -* The `metalprobe-mock-controller` that watches `ServerBootConfigs` for BMC mocks and runs a `metalprobe` agent for every instance. It simulates a server controlled by the BMC mock booting up and running `metalprobe` to report to the metal-operator's registry. The reason it lives in the fork is that the `probe`-package of the metal-operator is internal. +* The `metalprobe-mock-controller` that watches `ServerBootConfigurations` for BMC mocks and runs a `metalprobe` agent for every instance. It simulates a server controlled by the BMC mock booting up and running `metalprobe` to report to the metal-operator's registry. The reason it lives in the fork is that the `probe`-package of the metal-operator is internal.