diff --git a/README.md b/README.md index 45e5086..8e98f7c 100644 --- a/README.md +++ b/README.md @@ -1,23 +1,31 @@ # metal-operator test framework +This repository aims to provide a set of capabilities a BMC-implementation needs to support to be usable with the [metal-operator](https://github.com/ironcore-dev/metal-operator), and a set of tests to verify these capabilities. Its goal is to enable BMC vendors or users to run these tests against a BMC with minimal effort and more easily determine what capabilities their BMC may be missing or implementing incorrectly. + +## Capabilities +Refer to [capability matrix](capability-matrix.md) for a list of capabilities and their descriptions. Refer to the [coverage document](coverage.md) for what tests cover which capabilities. Determine what your BMC might not implement by referring to a failed test. + ## Test suite -The `tests` directory contains a suite of tests based on [chainsaw](https://kyverno.github.io/chainsaw/latest/). Every test case creates k8s resources in steps and asserts their status before proceeding to the next steps and implements common metal-operator workflows. Tests are independent from the infrastructure they run on and respect `KUBECONFIG`. +The `tests` directory contains a suite of tests based on [chainsaw](https://kyverno.github.io/chainsaw/latest/). Every test case creates k8s resources in steps and asserts their status before proceeding. Tests are independent of from the infrastructure they run on and respect `KUBECONFIG`. ### Requirements * [chainsaw](https://kyverno.github.io/chainsaw/latest/) -* A metal-operator installation and BMC to run tests against. This repository usually uses the locally virtualised [metal-lab](https://github.com/simontesar/metal-lab). +* A metal-operator installation and BMC to run tests against. This repository usually uses the locally virtualised [metal-lab](https://github.com/simontesar/metal-lab) to develop or verify functionality of the actual tests. You can use the lab setup as a reference. ### Usage -The server to run a test against is configured by passing a values file to chainsaw. The default file is `infra/kind/values-basic-go.yaml` that points to a redfish mock setup in the `kind` environment and can be overridden via `VALUES`. +The server to run a test against is configured by passing a values file to chainsaw. The default file is `infra/kind/values-basic-go.yaml` which points to a Redfish mock setup in the [`kind` environment](environments.md) and should be overridden via `VALUES`. ```bash -make test # Run all tests -make test/01-bmc-registration # Run a specific test -make test/03-power-annotation VALUES=/path/to/metal-lab/values-containerlab-node1.yaml # Run against a specific BMC. +# Examples +make test # Run all tests +make test/01-bmc-registration # Run a specific test +make test/03-power-annotation VALUES=/path/to/metal-lab/values-containerlab-node1.yaml # Run against a specific BMC ``` +The [`metal-lab`](https://github.com/simontesar/metal-lab) repository provides a standalone containerlab-based environment and ships its own `VALUES` `values-containerlab-node1.yaml` / `values-containerlab-node2.yaml`. Clone the metal-lab repository and refer to its `README` to try the suite of tests without your own BMC. + ### Bring your own BMC -To run tests against your own BMC, copy values into a new file: +To run tests against your own BMC, copy these values into a new file: ```yaml bmcIP: "172.16.100.11" bmcPort: 443 @@ -35,32 +43,8 @@ bmcSettingKey: "EmailAlert.1.Address" bmcSettingValue: "alerts@example.com" ``` -Adjust the credentials and expectations to their respective values and point the tests to it: +Adjust the credentials and expectations to your respective values and point the tests to it: ```bash -make make test/02-discovery VALUES=/path/to/new/file.yaml +make test/02-discovery VALUES=/path/to/new/file.yaml ``` - -### Predefined values -A set of predefined values that point to BMCs deployed via this repository exist in their respective environment's directories: -- `infra/kind/values-basic-go.yaml` -- `infra/kind/values-contoso-go.yaml` - -The `metal-lab` repository (a standalone containerlab-based environment, see below) ships its own equivalent `values-containerlab-node1.yaml` / `values-containerlab-node2.yaml`. - -## Supporting Environments -This repository contains virtualised or containerised infrastructure environments that mock or emulate physical BMC/server nodes. Refer to the `make help` target in every environment's subdirectory for usage. - -### KIND environment -Manages a [kind](https://kind.sigs.k8s.io/) cluster to run the metal-operator and its dependencies. To simulate BMCs and Servers, it runs a Go-based Redfish Mock Server (modified version of the metal-operator's `bmc/mock/main.go`) that supports using system-specific redfish client mock data like the [DMTF mockup server](https://github.com/DMTF/Redfish-Mockup-Server) but has support for dynamic functions like simulating reboots. The server runs once per client data, i.e. BMC. -To simulate booting Servers to run the `metalprobe` tool, a custom `boot-operator`-like implementation runs the metalprobe agent once per discovered `ServerBootConfiguration` and reports back bogus data. This works fine for simple tests. - -### Vagrant environment -Manages a [vagrant](https://developer.hashicorp.com/vagrant)-based environment that provides a network setup as close to a physical environment as possible. Its primary focus is to provide a reference setup for a physical lab. It supports development for scripts, ansible playbooks etc. that can be used to setup actual infrastructure. It does not support running tests against its nodes. **Its current state is a work in progress. It can probably be replaced by a containerlab-based setup.** - -## Caveats -### Dependencies on forks -#### Metal-operator -The `kind`-environment depends on two services that currently live in a [fork of the metal-operator](https://github.com/simontesar/metal-operator): -* A Go implementation of a [redfish mock server](https://github.com/simontesar/metal-operator/blob/dell/bmc/mock/main.go) that is included in the upstream metal-operator for testing but modified in the fork to support some dynamic features like `lastResetTime` and replaceable client mock data to be able to test against mocks of specific BMC models. -* The `metalprobe-mock-controller` that watches `ServerBootConfigs` for BMC mocks and runs a `metalprobe` agent for every instance. It simulates a server controlled by the BMC mock booting up and running `metalprobe` to report to the metal-operator's registry. The reason it lives in the fork is that the `probe`-package of the metal-operator is internal. diff --git a/capability-matrix.md b/capability-matrix.md new file mode 100644 index 0000000..0cb6996 --- /dev/null +++ b/capability-matrix.md @@ -0,0 +1,113 @@ +# Capability Matrix + +This document specifies sets of capabilities a BMC needs to support to function with the metal-operator. Right now there is essentially one capability per method of the [`bmc.BMC`](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L191)-interface of the metal-operator. + +Which Chainsaw tests cover each capability is tracked in the [coverage document](coverage.md). + +## Description of columns in capability tables + +| Column | Description | +|---|---| +| **Capability id** | A key identifying the capability. | +| **`BMC` method** | The method of the `BMC` interface in the metal-operator the row stands for. | +| **Level** | How important the capability is. | +| **Redfish API** | The Redfish path/action the BMC must expose for the capability to work. | +| **Redfish request body** | An example request payload for write operations. | + +## Conformance levels + +| Level | Description | +|---|---| +| `MUST` | Core functions like `BMC`-, `Server`- and `ServerClaim`-reconciliation or power management cannot complete without it. | +| `SHOULD` | A first-class workflow like `BIOSSettings`- and `ServerMaintenance`-reconciliation or persistent boot order and locator LED depends on it. | +| `OPTIONAL` | Opt-in features like firmware upgrade, account management, event subscriptions and BMC settings. | + +--- +# Interfaces + +## [PowerController](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L43) + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `power.on` | `PowerOn` | MUST | `POST` `#ComputerSystem.Reset` | `{"ResetType": "On"}` | +| `power.off-graceful` | `PowerOff` | MUST | `POST` `#ComputerSystem.Reset` | `{"ResetType": "GracefulShutdown"}` | +| `power.off-force` | `ForcePowerOff` | MUST | `POST` `#ComputerSystem.Reset` | `{"ResetType": "ForceOff"}` | +| `power.reset` | `Reset` | MUST | `POST` `#ComputerSystem.Reset` - reset type from `AnnotationToRedfishMapping` | `{"ResetType": "GracefulRestart" \| "ForceRestart" \| "PowerCycle" \| "ForceOff" \| "ForceOn"}` | +| `power.wait-state` | `WaitForServerPowerState` | MUST | `GET` `ComputerSystem.PowerState` | - | + +## [BootController](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L61) + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `boot.override-pxe` | `SetBootOverride` | MUST | `PATCH` `ComputerSystem` | `{"Boot": {"BootSourceOverrideEnabled": "Once", "BootSourceOverrideTarget": "Pxe", "BootSourceOverrideMode": "UEFI"}}` | +| `boot.order-get` | `GetBootOrder` | SHOULD | `GET` `ComputerSystem.Boot.BootOrder` | - | +| `boot.order-set` | `SetBootOrder` | SHOULD | `PATCH` `ComputerSystem` | `{"Boot": {"BootOrder": ["", "…"], "BootSourceOverrideEnabled": "Continuous", "BootSourceOverrideTarget": "None"}}` | + +## [SystemInspector](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L75) + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `inventory.systems` | `GetSystems` | MUST | `GET` `/redfish/v1/Systems` collection; members need `UUID` and `Boot.BootSourceOverrideTarget` | - | +| `inventory.system-info` | `GetSystemInfo` | MUST | `GET` `ComputerSystem` - `Manufacturer`, `Model`, `SerialNumber`, `SKU`, `PowerState`, `Status`, `MemorySummary.TotalSystemMemoryGiB`, `BiosVersion`, `IndicatorLED` | - | +| `inventory.processors` | `GetProcessors` | SHOULD | `GET` `ComputerSystem/Processors` collection (`Processor`) | - | +| `inventory.storages` | `GetStorages` | SHOULD | `GET` `ComputerSystem/Storage` (`Storage`, `Drives`, `Volumes`); fallback `GET` `ComputerSystem/SimpleStorage` | - | + +## [BIOSManager](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L90) + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `bios.version` | `GetBiosVersion` | MUST | `GET` `ComputerSystem.BiosVersion` | - | +| `bios.attr-get` | `GetBiosAttributeValues` | SHOULD | `GET` `ComputerSystem/Bios.Attributes` + `GET` `/redfish/v1/Registries` | - | +| `bios.attr-pending` | `GetBiosPendingAttributeValues` | SHOULD | `GET` `Bios` `@Redfish.Settings` + `GET` settings object `.Attributes` | - | +| `bios.attr-set-on-reset` | `SetBiosAttributesOnReset` | SHOULD | `PATCH` the `Bios` `@Redfish.Settings` settings object | `{"Attributes": {"": "", …}, "@Redfish.SettingsApplyTime": {"ApplyTime": "OnReset"}}` | +| `bios.attr-check` | `CheckBiosAttributes` | SHOULD | `GET` `/redfish/v1/Registries` | - | + +## [BMCSettingsManager](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L108) + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `bmc.version` | `GetBMCVersion` | MUST | `GET` `Manager.FirmwareVersion` | - | +| `bmc-settings.attr-get` | `GetBMCAttributeValues` | OPTIONAL | Manager attribute resource / `Manager` `@Redfish.Settings` - vendor-specific implementation | - | +| `bmc-settings.attr-pending` | `GetBMCPendingAttributeValues` | OPTIONAL | `GET` `Manager` `@Redfish.Settings` | - | +| `bmc-settings.set-immediate` | `SetBMCAttributesImmediately` | OPTIONAL | `PATCH` the `Manager` settings object, vendor-specific implementation | `{"Attributes": {"": "", …}, "@Redfish.SettingsApplyTime": {"ApplyTime": "Immediate"}}` | +| `bmc-settings.attr-check` | `CheckBMCAttributes` | OPTIONAL | `GET` `/redfish/v1/Registries` | - | + +## [FirmwareUpdater](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L127) + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `firmware.bios.upgrade` | `UpgradeBiosVersion` | OPTIONAL | `POST` `#UpdateService.SimpleUpdate` | `{"ImageURI": "", "TransferProtocol": "", "Targets": [""], "@Redfish.OperationApplyTime": "Immediate"}` | +| `firmware.bios.task` | `GetBiosUpgradeTask` | OPTIONAL | `GET` task monitor / `TaskService` `Task` (`TaskState`, `TaskStatus`) | - | +| `firmware.bmc.upgrade` | `UpgradeBMCVersion` | OPTIONAL | `POST` `#UpdateService.SimpleUpdate` | `{"ImageURI": "", "TransferProtocol": "", "Targets": [""], "@Redfish.OperationApplyTime": "Immediate"}` | +| `firmware.bmc.task` | `GetBMCUpgradeTask` | OPTIONAL | `GET` task monitor / `Task` | - | +| `firmware.pending-check` | `CheckBMCPendingComponentUpgrade` | OPTIONAL | `GET` `UpdateService/FirmwareInventory` | - | + +## [ManagerController](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L146) + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `manager.get` | `GetManager` | MUST | `GET` `/redfish/v1/Managers` collection; match `Manager.UUID` | - | +| `manager.discover` | `DiscoverManager` | SHOULD | `GET` `/redfish/v1/Managers`; `Manager.GraphicalConsole` (`MaxConcurrentSessions`, `ConnectTypesSupported`) | - | +| `manager.reset` | `ResetManager` | SHOULD | `POST` `#Manager.Reset` - value must be in `Manager.SupportedResetTypes` | `{"ResetType": "GracefulRestart"}` | + +## [AccountManager](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L158) + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `account.create-update` | `CreateOrUpdateAccount` | OPTIONAL | `POST` `AccountService/Accounts`; fallback `PATCH` an existing / empty slot with the same fields | `{"UserName": "", "Password": "", "RoleId": "", "Enabled": true}` | +| `account.delete` | `DeleteAccount` | OPTIONAL | `DELETE` the `ManagerAccount` URI or fall back to `PATCH`ing the slot | `{"UserName": "", "Enabled": false}` for PATCH fallback | +| `account.list` | `GetAccounts` | OPTIONAL | `GET` `AccountService/Accounts` collection | - | +| `account.service` | `GetAccountService` | OPTIONAL | `GET` `/redfish/v1/AccountService` | - | + +## [IndicatorController](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L173) + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `indicator.set-led` | `SetIndicatorLED` | SHOULD | `PATCH` `ComputerSystem` | `{"IndicatorLED": "Lit"}` | + +## [EventSubscriber](https://github.com/ironcore-dev/metal-operator/blob/4d2a4eb1a372c9a01602a5bda26f6c9489393918/bmc/bmc.go#L179) + +| Capability id | `BMC` method | Level | Redfish API | Redfish request body | +|---|---|:---:|---|---| +| `events.subscribe` | `CreateEventSubscription` | OPTIONAL | `GET` `EventService` (`ServiceEnabled`), then `POST` `EventService/Subscriptions`; read the new URI from the `Location` header | `{"Destination": "", "Protocol": "Redfish", "EventFormatType": "Event", "Context": "metal-operator"}` | +| `events.unsubscribe` | `DeleteEventSubscription` | OPTIONAL | `DELETE` the `EventDestination` URI | - | diff --git a/coverage.md b/coverage.md new file mode 100644 index 0000000..43ebe43 --- /dev/null +++ b/coverage.md @@ -0,0 +1,75 @@ +# Test Coverage + +This document summarises what tests cover which capabilities in the [capability-matrix](capability-matrix.md). + +* A capability is **covered** when running the test makes the operator invoke the method. +* A capability is **verified** when the test asserts that it has produced expected values. + +## Common coverage + +Some capabilities are covered by multiple tests implicitly because they share common workflows. +These rows/capabilities include the following placeholders in their `test` field: + +| Placeholder | Description | Capability ids | +|---|---|---| +| `BMC Registration` | Test creates a `BMC` that results in a `Server` resource. | `manager.get`, `inventory.systems`, `inventory.system-info` | +| `Discovery` | Test waits for the `Server` to reach the `Available` state. | `power.on`, `power.off-graceful`, `power.wait-state`, `boot.override-pxe`, `bios.version`, `inventory.processors`, `inventory.storages` | + +--- + +## Coverage by test + +| Chainsaw test | Capability ids covered | +|---|---| +| `tests/01-bmc-registration` | `BMC Registration` | +| `tests/02-discovery` | `BMC Registration`, `Discovery` | +| `tests/03-power-annotation` | `BMC Registration`, `Discovery`, `power.reset` | +| `tests/04-bmc-reset` | `BMC Registration`, `manager.reset` | +| `tests/05-indicator-led` | `BMC Registration`, `Discovery`, `indicator.set-led` | +| `tests/06-biossettings-noreboot` | `BMC Registration`, `Discovery`, `bios.attr-get`, `bios.attr-pending`, `bios.attr-set-on-reset`, `bios.attr-check` | +| `tests/07-biossettings-reboot` | `BMC Registration`, `Discovery`, `bios.attr-get`, `bios.attr-pending`, `bios.attr-set-on-reset`, `bios.attr-check` | +| `tests/08-bmcsettings` | `BMC Registration`, `Discovery`, `bmc-settings.attr-get`, `bmc-settings.attr-pending`, `bmc-settings.set-immediate`, `bmc-settings.attr-check` | +| `tests/09-persistent-boot-order` | `BMC Registration`, `Discovery`, `boot.order-get`, `boot.order-set`, `power.reset` | + +## Coverage by capability + +| Capability ID | Test | Verified | +|---|---|:---:| +| `power.on` | `Discovery` | yes | +| `power.off-graceful` | `Discovery` | yes | +| `power.off-force` | uncovered | - | +| `power.reset` | `tests/03-power-annotation`, `tests/09-persistent-boot-order` | no | +| `power.wait-state` | `Discovery` | yes | +| `boot.override-pxe` | `Discovery` | yes | +| `boot.order-get` | `tests/09-persistent-boot-order` | yes | +| `boot.order-set` | `tests/09-persistent-boot-order` | yes | +| `inventory.systems` | `BMC Registration` | yes | +| `inventory.system-info` | `BMC Registration` | yes | +| `inventory.processors` | `Discovery` | yes | +| `inventory.storages` | `Discovery` | no | +| `bios.version` | `Discovery`, `tests/06-biossettings-noreboot`, `tests/07-biossettings-reboot` | yes | +| `bios.attr-get` | `tests/06-biossettings-noreboot`, `tests/07-biossettings-reboot` | yes | +| `bios.attr-pending` | `tests/06-biossettings-noreboot`, `tests/07-biossettings-reboot` | yes | +| `bios.attr-set-on-reset` | `tests/06-biossettings-noreboot`, `tests/07-biossettings-reboot` | yes | +| `bios.attr-check` | `tests/06-biossettings-noreboot`, `tests/07-biossettings-reboot` | yes | +| `bmc.version` | uncovered | - | +| `bmc-settings.attr-get` | `tests/08-bmcsettings` | yes | +| `bmc-settings.attr-pending` | `tests/08-bmcsettings` | no | +| `bmc-settings.set-immediate` | `tests/08-bmcsettings` | yes | +| `bmc-settings.attr-check` | `tests/08-bmcsettings` | no | +| `firmware.bios.upgrade` | uncovered | - | +| `firmware.bios.task` | uncovered | - | +| `firmware.bmc.upgrade` | uncovered | - | +| `firmware.bmc.task` | uncovered | - | +| `firmware.pending-check` | uncovered | - | +| `manager.get` | `BMC Registration` | yes | +| `manager.discover` | uncovered | - | +| `manager.reset` | `tests/04-bmc-reset` | no | +| `account.create-update` | uncovered | - | +| `account.delete` | uncovered | - | +| `account.list` | uncovered | - | +| `account.service` | uncovered | - | +| `indicator.set-led` | `tests/05-indicator-led` | yes | +| `events.subscribe` | uncovered | - | +| `events.unsubscribe` | uncovered | - | + diff --git a/environments.md b/environments.md new file mode 100644 index 0000000..30ca40d --- /dev/null +++ b/environments.md @@ -0,0 +1,26 @@ +# Environments + +This repository contains virtualised or containerised infrastructure environments that mock or emulate physical BMC/server nodes. They don't relate to running tests against a custom BMC. + +Refer to the `make help` target in every environment's subdirectory for usage. + +## Supporting Environments + +### KIND environment +Manages a [kind](https://kind.sigs.k8s.io/) cluster to run the metal-operator and its dependencies. To simulate BMCs and Servers, it runs a Go-based Redfish mock server (a modified version of the metal-operator's `bmc/mock/main.go`) that supports using system-specific Redfish client mock data like the [DMTF mockup server](https://github.com/DMTF/Redfish-Mockup-Server) but additionally implements dynamic functions like simulating reboots. The server runs once per client data set, i.e. BMC. +To simulate booting Servers to run the `metalprobe` tool, a custom `boot-operator`-like implementation runs the `metalprobe` agent once per discovered `ServerBootConfiguration` and reports back bogus data. This works fine for simple tests. + +### Vagrant environment +Manages a [vagrant](https://developer.hashicorp.com/vagrant)-based environment that provides a network setup as close to a physical environment as possible. Its primary focus is to provide a reference setup for a physical lab. It supports development for scripts, Ansible playbooks etc. that can be used to set up actual infrastructure. It does not support running tests against its nodes. **Its current state is a work in progress. It can probably be replaced by a containerlab-based setup.** + +## Predefined values +Two predefined value files that point to the DMTF- or Go-based mock exist for the `kind`-environment: +- `infra/kind/values-basic-go.yaml` +- `infra/kind/values-contoso-go.yaml` + +## Caveats +### Dependencies on forks +#### Metal-operator +The `kind`-environment depends on two services that currently live in a [fork of the metal-operator](https://github.com/simontesar/metal-operator): +* A Go implementation of a [redfish mock server](https://github.com/simontesar/metal-operator/blob/dell/bmc/mock/main.go) that is included in the upstream metal-operator for testing but modified in the fork to support some dynamic features like `lastResetTime` and replaceable client mock data to be able to test against mocks of specific BMC models. +* The `metalprobe-mock-controller` that watches `ServerBootConfigurations` for BMC mocks and runs a `metalprobe` agent for every instance. It simulates a server controlled by the BMC mock booting up and running `metalprobe` to report to the metal-operator's registry. The reason it lives in the fork is that the `probe`-package of the metal-operator is internal.