From 3600aac830c935cf17f4f3609a55e7de521860f9 Mon Sep 17 00:00:00 2001 From: "Christoph Engelbert (noctarius)" Date: Sat, 22 Aug 2026 14:26:38 +0200 Subject: [PATCH 1/3] docs: split the operator cluster operations page into topics operator-cluster-operations.md covered six cluster actions, six node actions, a migration, and a five-phase drain on one page, at a depth that fit none of them. Split it into five pages and document each against the operator implementation: - cluster-actions.md: the spec.action mechanism, including that an action only counts as complete when observedGeneration matches, and that a set action suspends the periodic status sync - rolling-restart.md: the node-recycle action with its five phases, the idempotency skips, and resume-after-restart behavior - storage-node-actions.md: shutdown, restart, suspend, resume, the activeOpsRef lock, and how to find a StorageNode by name - migrating-a-storage-node.md: the four sub-phases (the page and the operator reference both claimed three), the DNS precondition in Preparing, and what Promoting re-points - removing-a-storage-node.md: the drain sub-phases, the pause on an unhealthy cluster, and the resume-on-failure path Pinned-volume behavior is written to match drainValidate: any pinned volume blocks the drain. Cross-links added from volume-migration.md and node-drain-coordination.md, which is where some of this content had been stranded. Co-Authored-By: Claude Opus 5 (1M context) --- docs/kubernetes/operations/cluster-actions.md | 176 +++++++++++++ .../operations/migrating-a-storage-node.md | 139 ++++++++++ .../operations/node-drain-coordination.md | 4 + .../operations/operator-cluster-operations.md | 240 ------------------ .../operations/removing-a-storage-node.md | 164 ++++++++++++ docs/kubernetes/operations/rolling-restart.md | 108 ++++++++ .../operations/storage-node-actions.md | 146 +++++++++++ .../kubernetes/operations/volume-migration.md | 3 +- 8 files changed, 739 insertions(+), 241 deletions(-) create mode 100644 docs/kubernetes/operations/cluster-actions.md create mode 100644 docs/kubernetes/operations/migrating-a-storage-node.md delete mode 100644 docs/kubernetes/operations/operator-cluster-operations.md create mode 100644 docs/kubernetes/operations/removing-a-storage-node.md create mode 100644 docs/kubernetes/operations/rolling-restart.md create mode 100644 docs/kubernetes/operations/storage-node-actions.md diff --git a/docs/kubernetes/operations/cluster-actions.md b/docs/kubernetes/operations/cluster-actions.md new file mode 100644 index 00000000..7ccf632f --- /dev/null +++ b/docs/kubernetes/operations/cluster-actions.md @@ -0,0 +1,176 @@ +--- +title: "Storage Cluster Actions" +description: "Trigger cluster-wide lifecycle operations on a simplyblock storage cluster through the action field of the StorageCluster resource and track their outcome." +weight: 10700 +--- + +Cluster-wide lifecycle operations are requested declaratively on Kubernetes. Setting `spec.action` on a +`StorageCluster` resource makes the Simplyblock Operator call the corresponding backend API, poll until the cluster +reaches the expected state, and record the outcome in `status.actionStatus`. The CLI is not involved. + +Only one action can be requested at a time, since `spec.action` holds a single value. + +## Requesting an Action + +An action is requested by patching the field. The example below shuts the cluster down. + +```bash title="Requesting a cluster action" +kubectl patch storagecluster simplyblock-cluster -n simplyblock \ + --type=merge -p '{"spec": {"action": "shutdown"}}' +``` + +| Action | Effect | Expected cluster status | +|----------------|------------------------------------------------------------------------|-------------------------| +| `activate` | Activates a cluster whose nodes have joined but which is not yet live. | `active` | +| `expand` | Finalizes a cluster expansion after new storage nodes came online. | `active` | +| `shutdown` | Shuts the whole cluster down. | `suspended` | +| `start` | Starts a previously shut down cluster. | `active` | +| `restart` | Runs a shutdown followed by a start. | `active` | +| `node-recycle` | Restarts every storage node of the cluster, one after another. | `active` | + +Any other value is rejected by the CRD schema. The `node-recycle` action has its own page, see +[Rolling Restart](rolling-restart.md). + +## How an Action Is Executed + +Every action follows the same pattern. The operator records the action in `status.actionStatus` with the state +`running`, sends the backend request once, and then polls the cluster until the expected status is reached. The first +poll follows five seconds after the request, and further polls follow every ten seconds. + +The `status.actionStatus.triggered` flag marks that the request has already been sent, so a requeue or an operator +restart never sends it twice. A failed request moves the state to `failed` and writes the reason into +`status.actionStatus.message`. A failed action is not retried automatically. + +| Field | Description | +|----------------------|--------------------------------------------------------------------------| +| `action` | The action this status belongs to. | +| `state` | `running` while the action is in progress, then `success` or `failed`. | +| `message` | The result, the failure reason, or the current sub-phase of a `restart`. | +| `triggered` | Whether the backend request has already been sent. | +| `observedGeneration` | The `metadata.generation` of the spec this action was started for. | +| `updatedAt` | The time of the last status transition. | + +## Re-Running and Clearing an Action + +An action counts as complete when its state is `success` **and** its `observedGeneration` matches the current +`metadata.generation` of the resource. Patching `spec.action` with the value it already holds does not change the +generation, so nothing happens. Re-running the same action therefore takes two patches: the field is cleared first +and set again afterward. + +```bash title="Re-running the same action" +kubectl patch storagecluster simplyblock-cluster -n simplyblock \ + --type=merge -p '{"spec": {"action": ""}}' +kubectl patch storagecluster simplyblock-cluster -n simplyblock \ + --type=merge -p '{"spec": {"action": "restart"}}' +``` + +!!! important + While `spec.action` holds a value, the reconciler serves the action instead of its periodic status sync. Once the + action has succeeded, nothing further happens to the resource, and the remaining `status` fields are no longer + refreshed from the backend. Clearing `spec.action` after a completed action returns the cluster to normal status + reconciliation. + +## Shutdown + +A shutdown suspends the entire cluster. The operator calls the backend shutdown API and polls until the cluster +reports `suspended`. + +```bash title="Shutting down the storage cluster" +kubectl patch storagecluster simplyblock-cluster -n simplyblock \ + --type=merge -p '{"spec": {"action": "shutdown"}}' +``` + +!!! warning + A cluster shutdown takes every volume of the cluster offline. Workloads consuming those volumes lose their storage + for the duration of the shutdown. To take a single storage node out of service instead, see + [Storage Node Actions](storage-node-actions.md). + +## Start + +A start brings a suspended cluster back. The operator calls the backend start API and polls until the cluster reports +`active`. The rebalancing flag reported by the backend is recorded in `status.rebalancing` once the cluster is up. + +```bash title="Starting a suspended storage cluster" +kubectl patch storagecluster simplyblock-cluster -n simplyblock \ + --type=merge -p '{"spec": {"action": "start"}}' +``` + +## Restart + +A restart sequences a shutdown and a start. Both legs are driven by the same action, and the leg currently running is +held in `status.actionStatus.message` as either `shutdown` or `start`. The action succeeds once the cluster is `active` +again. + +```bash title="Restarting the storage cluster" +kubectl patch storagecluster simplyblock-cluster -n simplyblock \ + --type=merge -p '{"spec": {"action": "restart"}}' +``` + +```bash title="Following the leg of a running restart" +kubectl get storagecluster simplyblock-cluster -n simplyblock \ + -o jsonpath='{.status.actionStatus.message}{"\n"}' +``` + +## Activate + +Activation is normally automatic. The operator activates a cluster on its own once every storage node declared in its +`StorageNodeSet` is online and healthy, and the number of those nodes is at least the sum of the data chunks, the +parity chunks, and one. See +[Create a Storage Cluster](../installation/k8s-storage-plane.md#when-does-the-cluster-become-active). + +The `activate` action exists for the case where that did not happen, for example, because nodes came online after the +automatic check had already passed. + +```bash title="Activating a cluster manually" +kubectl patch storagecluster simplyblock-cluster -n simplyblock \ + --type=merge -p '{"spec": {"action": "activate"}}' +``` + +!!! warning + A `StorageCluster` that is deleted while `spec.action` is `activate` has its finalizer removed without the backend + cluster being deleted. The cluster is then left behind on the control plane and has to be removed there. Clear + `spec.action` before deleting the resource. + +## Expand + +An expansion is finalized with the `expand` action, after the new storage nodes have been added and are online. The +operator calls the backend expand API and polls until the cluster returns to `active`. + +```bash title="Finalizing a cluster expansion" +kubectl patch storagecluster simplyblock-cluster -n simplyblock \ + --type=merge -p '{"spec": {"action": "expand"}}' +``` + +Adding the storage nodes themselves is the step before this one, described in +[Expanding a Storage Cluster](scaling/expanding-storage-cluster.md). + +## Monitoring an Action + +The action state is exposed in the resource status. + +```bash title="Reading the current action status" +kubectl get storagecluster simplyblock-cluster -n simplyblock \ + -o jsonpath='{.status.actionStatus}' | jq . +``` + +```plain title="Example output of a running action" +{ + "action": "restart", + "state": "running", + "message": "start", + "observedGeneration": 7, + "triggered": true, + "updatedAt": "2026-08-22T09:14:03Z" +} +``` + +The backend lifecycle status of the cluster is tracked separately from the action. + +```bash title="Reading the backend cluster status" +kubectl get storagecluster simplyblock-cluster -n simplyblock \ + -o jsonpath='{.status.status}{"\n"}' +``` + +```bash title="Streaming live cluster status changes" +kubectl get storagecluster simplyblock-cluster -n simplyblock -w +``` diff --git a/docs/kubernetes/operations/migrating-a-storage-node.md b/docs/kubernetes/operations/migrating-a-storage-node.md new file mode 100644 index 00000000..c0256cc0 --- /dev/null +++ b/docs/kubernetes/operations/migrating-a-storage-node.md @@ -0,0 +1,139 @@ +--- +title: "Migrating a Storage Node" +description: "Relocate a simplyblock storage node onto a different Kubernetes worker with the migrate action, keeping its backend identity, devices, and logical volumes." +weight: 10730 +--- + +The `migrate` action of a `StorageNodeOps` resource moves a storage node onto a different Kubernetes worker without +taking it out of the cluster. The node keeps its backend UUID, its devices, and its logical volume assignments, and no +volume is moved between nodes. What changes is the host the node runs on. + +This is the operation for replacing the hardware under a node, or for vacating a worker that has to be +decommissioned. It is not the operation for moving individual volumes, which is +[Volume Migration](volume-migration.md), and not the operation for taking a node out of the cluster, which is +[Removing a Storage Node](removing-a-storage-node.md). + +!!! important + A storage node must never be removed and re-added to move it to another host. Removing a node without draining it + first makes the logical volumes it owns inaccessible. A migration keeps the node's identity, which is precisely + what preserves those volumes. + +`spec.workerNode` on a `StorageNode` cannot be edited directly either. A validating webhook rejects any change made by +a user, and only the operator re-points the field, as the final step of this operation. + +## Prerequisites + +The target worker has to be part of the storage plane before the migration starts. The operator labels it and waits +for its storage-node pod, but the node itself has to exist and be usable. + +- The target is a Kubernetes node in the cluster and is `Ready`. A target that is missing or not ready fails the + operation immediately. +- The target is not the worker the storage node currently runs on. +- The target has the devices the node expects, either because they carry the same PCIe addresses as on the source + host, or because the additional addresses are declared in `spec.newSsdPcie`. + +## Requesting a Migration + +```bash title="Migrating a storage node to a different worker" +kubectl apply -n simplyblock -f - < Date: Sat, 22 Aug 2026 14:38:34 +0200 Subject: [PATCH 2/3] docs: add the Kubernetes topics missing against non-Kubernetes The non-Kubernetes operations section covered five topics that had no Kubernetes counterpart, three of them backed by CRD fields that exist only on Kubernetes: - failure-domains.md: enableFailureDomains plus the two assignment fields, including that a group index of 0 passes schema validation but is dropped from the request, and that a missing assignment blocks the node-add with a FailureDomainMissing event - cluster-upgrade.md: the chart upgrade for operator, control plane and CSI driver, the three image fields and which consumer reads each, and the rolling restart that carries a new image to running nodes - node-affinity.md: enableNodeAffinity, why it is creation-only, and how it combines with the per-PVC placement annotations - replacing-a-storage-node.md: relocate versus retire-and-replace, and the add-before-remove ordering - path-loss-recovery.md: the node plugin's fabric repair and the opt-in pod restart, including the coordinated restart of pods that share an NVMe subsystem Also records a limitation found while checking the pool gap: capacityLimit, logicalVolumeMaxSize and qos are read when a StoragePool is created and never reconciled, so patching them is accepted and does nothing. Co-Authored-By: Claude Opus 5 (1M context) --- .../installation/k8s-storage-plane.md | 7 + docs/kubernetes/operations/cluster-upgrade.md | 165 ++++++++++++++++ docs/kubernetes/operations/failure-domains.md | 178 ++++++++++++++++++ docs/kubernetes/operations/node-affinity.md | 100 ++++++++++ .../operations/path-loss-recovery.md | 119 ++++++++++++ .../operations/replacing-a-storage-node.md | 123 ++++++++++++ 6 files changed, 692 insertions(+) create mode 100644 docs/kubernetes/operations/cluster-upgrade.md create mode 100644 docs/kubernetes/operations/failure-domains.md create mode 100644 docs/kubernetes/operations/node-affinity.md create mode 100644 docs/kubernetes/operations/path-loss-recovery.md create mode 100644 docs/kubernetes/operations/replacing-a-storage-node.md diff --git a/docs/kubernetes/installation/k8s-storage-plane.md b/docs/kubernetes/installation/k8s-storage-plane.md index 30eef6ed..7c28a1ff 100644 --- a/docs/kubernetes/installation/k8s-storage-plane.md +++ b/docs/kubernetes/installation/k8s-storage-plane.md @@ -211,6 +211,13 @@ for the full parameter mapping. A StorageClass's parameters cannot be changed after creation, so `spec.storageClassParameters` is immutable once the storage pool is created. A new storage pool is required to provision volumes with different defaults. +!!! warning "Pool limits are read once" + `capacityLimit`, `logicalVolumeMaxSize`, and `qos` are sent to the control plane when the pool is created and are + not reconciled afterward. Patching one of them on an existing `StoragePool` is accepted by the API server and has + no effect on the pool, so a different capacity limit or a different set of QoS limits requires a new storage pool. + `allowedNodes` is the exception and is reconciled, see + [Host Authentication and Encryption](../operations/security/authentication-encryption.md#configuring-dhchap-via-the-storagepool-crd). + The StorageClass is automatically removed when the storage pool is deleted. Full details and customization options are available at [Simplyblock Operator: Storage Pool](../../reference/operator/reference.md#storagepool). diff --git a/docs/kubernetes/operations/cluster-upgrade.md b/docs/kubernetes/operations/cluster-upgrade.md new file mode 100644 index 00000000..183cc34b --- /dev/null +++ b/docs/kubernetes/operations/cluster-upgrade.md @@ -0,0 +1,165 @@ +--- +title: "Upgrading a Cluster" +description: "Upgrade the simplyblock operator, control plane, and CSI driver with Helm, then roll the new storage-node image across the storage plane one node at a time." +weight: 10610 +--- + +A simplyblock deployment on Kubernetes upgrades in two parts. The control plane, the operator, and the CSI driver come +from the Helm chart and move together with a chart upgrade. The storage plane runs from container images referenced by +the operator resources, and it is rolled node by node afterward. + +The two parts can be upgraded independently, but a control plane that is newer than its storage planes is the only +combination that is supported during the transition. The control plane is therefore upgraded first, and a control +plane that manages several storage clusters is upgraded before any of them. + +## Upgrade Order + +1. Upgrade the Helm release, which covers the operator, the control plane, and the CSI driver. +2. Wait for the control plane to report itself ready again. +3. Roll the storage-node image across each storage cluster. + +## Upgrading the Control Plane + +The control plane, the operator, and the CSI driver are all rendered by the same chart, so one upgrade moves them. + +```bash title="Upgrading the Helm release" +helm repo update +helm upgrade --install simplyblock -n simplyblock simplyblock/simplyblock-operator \ + --reuse-values +``` + +`--reuse-values` keeps the values the release was installed with. Without it, every value that was set at install time +falls back to the chart default, which silently reverts settings such as the TLS configuration. + +!!! warning + A chart upgrade re-renders every object the chart owns, which discards manual edits to them. A patch that has to + survive an upgrade is reapplied afterward, for example, the credentials mount described in + [FoundationDB Backup and Restore](foundationdb-backup.md). + +### Confirming the Control Plane Is Ready + +The `ControlPlane` resource is a singleton named `simplyblock`, created by the chart. Its phase is driven by the +readiness endpoint of the management API, which the operator polls every 30 seconds. + +```bash title="Checking the control plane phase" +kubectl get controlplane simplyblock -n simplyblock +``` + +```plain title="Example output of the control plane status" +NAME PHASE MESSAGE AGE +simplyblock Ready 14d +``` + +A phase of `Initializing` means the health check is still failing, and the `MESSAGE` column carries the reason. The +storage plane is not touched until the phase is `Ready`. + +```bash title="Waiting for the control plane to become ready" +kubectl wait --for=jsonpath='{.status.phase}'=Ready \ + controlplane/simplyblock -n simplyblock --timeout=10m +``` + +## Upgrading the Storage Plane + +Which image a storage node runs is decided by three fields. All of them accept only the trusted simplyblock +registries, and pinning by digest is recommended. + +| Field | Applies to | Default | +|-----------------------|-------------------------------------------------------|---------------------------| +| `spec.clusterImage` | The storage-node pod of the `StorageNodeSet`. | `ControlPlane.spec.image` | +| `spec.spdkImage` | The SPDK image, sent with the node-add request. | The control plane default | +| `spec.spdkProxyImage` | The SPDK proxy image, sent with the node-add request. | The control plane default | + +A `StorageNodeSet` that leaves `spec.clusterImage` empty inherits the image from the `ControlPlane` resource, which the +chart keeps up to date. On such a set the chart upgrade already changed the image, and the DaemonSet rolls its pods as +a consequence. + +A `StorageNodeSet` that pins `spec.clusterImage` does not follow the chart. Its image is raised explicitly. + +```bash title="Pinning a new storage-node image on a StorageNodeSet" +kubectl patch storagenodeset simplyblock-node -n simplyblock --type=merge \ + -p '{"spec": {"clusterImage": "quay.io/simplyblock-io/simplyblock:26.3.0"}}' +``` + +`spec.spdkImage` and `spec.spdkProxyImage` are read when a storage node is added, so a change to them governs nodes +added from that point on. + +```bash title="Reading the images a StorageNodeSet is configured with" +kubectl get storagenodeset simplyblock-node -n simplyblock \ + -o jsonpath='{.spec.clusterImage}{"\n"}{.spec.spdkImage}{"\n"}{.spec.spdkProxyImage}{"\n"}' +``` + +### Rolling the Change Across the Nodes + +A new image does not reach a running storage node on its own. The node has to be restarted, and the storage-node pod +has to be replaced so that it picks the image up rather than keeping the one it started with. + +Both happen in a [Rolling Restart](rolling-restart.md) with the pod refresh enabled. One node at a time is shut down, +its pod is replaced, the node is restarted, and the cluster rebalances before the next node follows. + +```bash title="Rolling the new image across the storage nodes" +kubectl patch storagecluster simplyblock-cluster -n simplyblock --type=merge \ + -p '{"spec": {"action": "node-recycle", "nodeRecycle": {"refreshSNodeAPI": true}}}' +``` + +```bash title="Following the rollout" +kubectl get storagecluster simplyblock-cluster -n simplyblock \ + -o jsonpath='{.status.nodeRecycleStatus}' | jq . +``` + +The rollout is complete when `status.actionStatus.state` is `success`. The action field is then cleared, so that the +cluster returns to normal status reconciliation, as described in +[Storage Cluster Actions](cluster-actions.md#re-running-and-clearing-an-action). + +```bash title="Clearing the action after the rollout" +kubectl patch storagecluster simplyblock-cluster -n simplyblock \ + --type=merge -p '{"spec": {"action": ""}}' +``` + +### Upgrading a Subset of Nodes First + +A new image can be tried on a few nodes before the whole fleet follows. The per-node configuration of a +`StorageNodeSet` overrides the fleet image for the workers named in it. + +```yaml title="Example of a phased rollout to two workers" +spec: + nodeConfigs: + worker-1.example.com: + spdkImage: quay.io/simplyblock-io/spdk:26.3.0 + worker-2.example.com: + spdkImage: quay.io/simplyblock-io/spdk:26.3.0 +``` + +The overrides are propagated to the `StorageNode` resources of those workers on the next reconcile. Once the sample +has proven itself, the fleet field is raised and the overrides are removed again. + +## Verifying the Result + +The storage nodes are online and healthy after a rollout, and the cluster is no longer rebalancing. + +```bash title="Checking the storage nodes after an upgrade" +kubectl get storagenodes -n simplyblock +``` + +```bash title="Checking that the cluster settled" +kubectl get storagecluster simplyblock-cluster -n simplyblock \ + -o jsonpath='{.status.status}{" rebalancing="}{.status.rebalancing}{"\n"}' +``` + +```bash title="Checking the running storage-node pods" +kubectl get pods -n simplyblock -l app=storage-node \ + -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,IMAGE:.spec.containers[0].image +``` + +## Rolling Back + +A storage-plane image is rolled back the way it was rolled forward: the field is set to the previous reference and the +nodes are recycled again. A Helm release is rolled back with `helm rollback`, which restores the previous chart +version together with the values it was rendered from. + +```bash title="Rolling the Helm release back to the previous revision" +helm rollback simplyblock -n simplyblock +``` + +!!! important + A rollback of the control plane below the version of a storage plane leaves the deployment in the one combination + that is not supported. The storage planes are rolled back first, and the control plane after them. diff --git a/docs/kubernetes/operations/failure-domains.md b/docs/kubernetes/operations/failure-domains.md new file mode 100644 index 00000000..e7718ff2 --- /dev/null +++ b/docs/kubernetes/operations/failure-domains.md @@ -0,0 +1,178 @@ +--- +title: "Managing Failure Domains" +description: "Enable failure-domain mode on a simplyblock cluster and assign Kubernetes workers to fault groups so erasure-coding chunks are spread across racks or zones." +weight: 10600 +--- + +Failure-domain mode groups the storage nodes of a cluster into independent fault groups, so that the control plane +spreads erasure-coding chunks and failover paths across them. A group stands for whatever fails together in the +physical layout: a rack, a power unit, or an availability zone. Losing one group then costs at most one chunk per +stripe. + +The placement contract, the balance rules, and the interaction with the erasure coding scheme are described in +[Failure Domains](../../architecture/concepts/failure-domains.md). This page covers how the mode is turned on and how +workers are assigned to groups through the operator resources. + +## Enabling Failure Domains + +Failure-domain mode is a cluster-level property, set through `spec.enableFailureDomains` on the `StorageCluster`. + +```yaml title="Example of a StorageCluster with failure domains enabled (storage-cluster.yaml)" +apiVersion: storage.simplyblock.io/v1alpha1 +kind: StorageCluster +metadata: + name: simplyblock-cluster + namespace: simplyblock +spec: + fabricType: tcp + maxSubsystemCount: 75 + vcpuCount: 16 + enableFailureDomains: true + stripe: + dataChunks: 2 + parityChunks: 1 +``` + +!!! important + The field is immutable. Failure-domain mode cannot be turned on for a cluster that is already running, and it + cannot be turned off again. A cluster that should use failure domains has to be created with the field set. + +## Assigning Workers to a Domain + +A domain is a non-negative integer, the group index, and every worker of a failure-domain cluster needs one. Two +fields carry the assignment, both on the `StorageNodeSet`. + +| Field | Scope | Description | +|------------------------------------------|----------|-------------------------------------------------------------------------| +| `spec.nodeFailureDomains` | Fleet | Maps a worker name to its group index. | +| `spec.nodeConfigs[worker].failureDomain` | Per node | Group index for one worker. Takes precedence over `nodeFailureDomains`. | + +`spec.nodeFailureDomains` is the readable form for a whole fleet, since the whole topology is visible in one block. + +```yaml title="Example of a StorageNodeSet spread across three failure domains (storage-nodeset.yaml)" +apiVersion: storage.simplyblock.io/v1alpha1 +kind: StorageNodeSet +metadata: + name: simplyblock-node + namespace: simplyblock +spec: + clusterName: simplyblock-cluster + journalManager: + count: 4 + workerNodes: + - worker-1.example.com + - worker-2.example.com + - worker-3.example.com + - worker-4.example.com + - worker-5.example.com + - worker-6.example.com + nodeFailureDomains: + worker-1.example.com: 1 + worker-2.example.com: 1 + worker-3.example.com: 2 + worker-4.example.com: 2 + worker-5.example.com: 3 + worker-6.example.com: 3 +``` + +Workers that share a group index are treated as failing together, so the index has to follow the physical layout. +Two workers in the same rack belong in the same group, and two workers in different racks belong in different ones. + +`nodeConfigs` is the place for an assignment that travels with other per-node settings. + +```yaml title="Example of a failure domain set through the per-node configuration" +spec: + nodeConfigs: + worker-1.example.com: + failureDomain: 1 + spdkSystemMemory: "8G" +``` + +Every key of `nodeConfigs` has to name a worker that is also listed in `spec.workerNodes`, which the CRD enforces. + +!!! warning + Group indexes should be numbered from `1`. The schema accepts `0`, but the value is dropped from the request the + operator sends to the control plane, which leaves the node looking unassigned. A group index of `0` therefore + silently behaves like no assignment at all. + +### Multi-Socket Workers + +Both fields are keyed by the worker name, not by the storage node. A worker that hosts several storage nodes, because +it has more than one NUMA socket or runs more than one node per socket, contributes all of them to the same group. +That is the intended behavior, since a host cannot fail in two places at once, and the balance rules require a host +to stay within one domain. + +## Journal Copies + +A failure-domain cluster needs at least four copies of the high-availability journal, even with a single parity chunk. +The default is three, which would put two copies in one domain on a two-domain cluster, so losing that domain would +break the journal quorum. + +The copy count is raised through the journal manager configuration, as in the `StorageNodeSet` above. + +```yaml title="Example of raising the journal copies for a failure-domain cluster" +spec: + journalManager: + count: 4 +``` + +## Verifying the Assignment + +The effective group of a node is reported back from the control plane in `StorageNode.status.failureDomain`. It is +also a print column, at a lower priority, so it shows with `-o wide`. + +```bash title="Listing the storage nodes with their failure domains" +kubectl get storagenodes -n simplyblock -o wide +``` + +```plain title="Example output of the storage node listing" +NAME WORKER SOCKET NODEIDX FD UUID STATUS HEALTH AGE +simplyblock-node-mejue8 worker-1.example.com 0 0 1 82198a36-fcbb-43e3-949c-0260bf40f0ac online true 43h +simplyblock-node-o6x20i worker-3.example.com 0 0 2 707dd443-5d0e-470f-bdde-92f1238c4b01 online true 43h +simplyblock-node-v92jx7 worker-5.example.com 0 0 3 114899a6-d708-499e-8051-bc9ca9713cf8 online true 43h +``` + +The same value is mirrored per node in the status of the owning `StorageNodeSet`. + +```bash title="Reading the failure domain of every node in a set" +kubectl get storagenodeset simplyblock-node -n simplyblock \ + -o jsonpath='{range .status.nodes[*]}{.hostname}{"\t"}{.failureDomain}{"\n"}{end}' +``` + +A node whose status reports no failure domain has not been assigned one on the backend. On a failure-domain cluster +that means the node was never added, since the operator refuses to add it. + +## A Missing Assignment Blocks the Node + +On a cluster with `enableFailureDomains: true`, a storage node without a group index is not provisioned. The operator +holds the node-add, emits a `FailureDomainMissing` warning on the `StorageNode`, and retries every 60 seconds. The +event names the worker and the field to set, and provisioning continues on its own once the assignment is added. + +```bash title="Checking for nodes blocked on a missing failure domain" +kubectl get events -n simplyblock \ + --field-selector reason=FailureDomainMissing +``` + +```bash title="Assigning a failure domain to a worker of an existing StorageNodeSet" +kubectl patch storagenodeset simplyblock-node -n simplyblock --type=merge \ + -p '{"spec": {"nodeFailureDomains": {"worker-1.example.com": 1}}}' +``` + +## Adding and Removing Nodes + +Once a failure-domain cluster holds data, the control plane admits a topology change only while the domains stay +balanced. The host count per domain may not diverge by more than one, no domain may drop below two hosts, and adding +another storage node on a host that is already a member is always allowed. A change that would violate a rule is +refused before any data moves. + +In practice, this means workers are added in whole rounds. On a balanced cluster one worker can be added to any domain, +and the next worker has to go to a different one. The full rules are in +[Failure Domains: Balance Rules](../../architecture/concepts/failure-domains.md#balance-rules). + +For the mechanics of adding the workers themselves, see +[Expanding a Storage Cluster](scaling/expanding-storage-cluster.md), and for taking one out, +[Removing a Storage Node](removing-a-storage-node.md). + +!!! note + Domain membership does not change on a live node. A worker that has to move to a different domain is drained and + removed, then added again with the new group index. diff --git a/docs/kubernetes/operations/node-affinity.md b/docs/kubernetes/operations/node-affinity.md new file mode 100644 index 00000000..4a6b7670 --- /dev/null +++ b/docs/kubernetes/operations/node-affinity.md @@ -0,0 +1,100 @@ +--- +title: "Configuring Node Affinity" +description: "Enable node affinity on a simplyblock cluster so a volume's data stays local to the storage node that owns it, and combine it with per-PVC placement." +weight: 10620 +--- + +Node affinity, also called data locality, keeps the data of a logical volume on the storage node that owns the volume, +rather than spreading it evenly across the cluster. Reads are then served from the local node, which removes a network +hop from the data path. It is meant for latency-sensitive workloads in hyper-converged and hybrid deployments, where +the storage node and the workload consuming it run on the same worker. + +Fault tolerance is not traded away for it. The parity chunks are still distributed across the other nodes of the +cluster, so a node failure still fails over transparently, and a volume whose local capacity runs out still spills over +onto other nodes. + +!!! info + Node affinity only has an effect in a hyper-converged or hybrid deployment. In a disaggregated deployment the + storage nodes are separate from the workloads, so there is no locality to preserve. + +## Enabling Node Affinity + +Node affinity is a cluster-level property, set through `spec.enableNodeAffinity` on the `StorageCluster`. + +```yaml title="Example of a StorageCluster with node affinity enabled (storage-cluster.yaml)" +apiVersion: storage.simplyblock.io/v1alpha1 +kind: StorageCluster +metadata: + name: simplyblock-cluster + namespace: simplyblock +spec: + fabricType: tcp + maxSubsystemCount: 75 + vcpuCount: 16 + enableNodeAffinity: true + stripe: + dataChunks: 2 + parityChunks: 1 +``` + +!!! important + The field is only read when the cluster is created. Setting it on a cluster that already exists has no effect, and + a cluster that should use node affinity has to be created with it. + +## Placing a Volume on a Storage Node + +Node affinity on its own changes how the data of a volume is laid out. Which storage node a volume belongs to is a +separate decision, made when the volume is provisioned, and it is what makes the locality useful. + +On Kubernetes that decision is driven by PVC annotations: + +- `simplyblock.io/pod-affinity` places a new volume on the storage node that is co-located with the pod consuming it, + which is the combination a hyper-converged deployment usually wants. +- `simplyblock.io/selected-storage-node` pins a volume to a named storage node. + +Both are described in [Automatic Volume Placement](../usage/volume-placement.md). They work whether or not node +affinity is enabled for the cluster, and the difference is what the backend then does with the data. Without node +affinity the volume is owned by that node but its data is spread across the cluster. With node affinity the data +follows the owner. + +```yaml title="Example of a PVC co-located with its consuming pod" +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: my-pvc + annotations: + simplyblock.io/pod-affinity: "true" +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: simplyblock-csi-sc +``` + +## Keeping Locality After a Volume Moves + +A volume that moves to another storage node leaves its data behind, so the locality is broken until the cluster +re-aligns its internal data structures to the new placement. That realignment restores both the fault-tolerance and +the node-affinity guarantees, and the operator triggers it automatically after volumes have moved. + +It applies to every move, whether the volume was migrated manually, relocated by auto-rebalancing, or evacuated from a +node being drained. It is enabled by default and configured under `volumeMigrationSettings.dataRealignment`, described +in [Volume Migration: Data Realignment](volume-migration.md#data-realignment). + +On a cluster with node affinity this is not a detail to leave unattended. A cluster that moves volumes frequently and +has realignment turned off keeps losing locality with every move, and never regains it. + +```bash title="Checking that data realignment is enabled" +kubectl get storagecluster simplyblock-cluster -n simplyblock \ + -o jsonpath='{.spec.volumeMigrationSettings.dataRealignment}' | jq . +``` + +```bash title="Checking whether a realignment is outstanding" +kubectl get storagecluster simplyblock-cluster -n simplyblock \ + -o jsonpath='{"moves="}{.status.volumeMoveGeneration}{" realigned="}{.status.realignedGeneration}{"\n"}' +``` + +A `volumeMoveGeneration` above `realignedGeneration` means volumes have moved since the last realignment, so one is +still pending. diff --git a/docs/kubernetes/operations/path-loss-recovery.md b/docs/kubernetes/operations/path-loss-recovery.md new file mode 100644 index 00000000..051a6c18 --- /dev/null +++ b/docs/kubernetes/operations/path-loss-recovery.md @@ -0,0 +1,119 @@ +--- +title: "Recovering from Path Loss" +description: "How the simplyblock CSI node plugin restores NVMe-oF paths after a storage node outage, and how to opt a workload into an automatic pod restart." +weight: 10820 +--- + +An NVMe-oF path breaks when the storage node serving a volume goes away, during a node restart, a worker reboot, or a +cluster outage. The paths are re-established by the CSI node plugin, which needs no intervention. What sometimes does +need attention is the workload: a process holding a file descriptor across a long path loss can be left with I/O errors +that only a restart clears. + +Plain Linux clients reconnect their volumes by hand, as described in +[Reconnecting Logical Volume](../../non-kubernetes/operations/reconnect-nvme-device.md). On Kubernetes none of that +applies. The node plugin repairs the fabric itself, and the only decision left is whether affected pods should be +restarted automatically. + +## Automatic Path Repair + +Each volume is attached over NVMe-oF with a primary path and one or more failover paths, so the loss of a single +storage node is absorbed by the remaining paths. The node plugin watches for the case where a subsystem is connected +but exports no usable namespace, which is what a stale controller looks like, and repairs it by reattaching the +subsystem. + +This happens during volume staging and at runtime, and it requires no configuration. + +## Automatic Pod Restart + +A path that comes back does not necessarily heal the workload. A filesystem that saw I/O errors can stay in a state +that only a remount clears, and a database that lost its data directory mid-write usually has to start over. + +The node plugin therefore ships a guardian that can restart the affected pods once the storage is healthy again. It is +off by default and opted into per workload. + +```yaml title="Example of a StorageClass whose volumes opt into the automatic restart" +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: simplyblock-autorestart + annotations: + simplyblock.io/auto-restart-on-pathloss: "true" +provisioner: csi.simplyblock.io +``` + +The same key is honored on a Pod, on a PersistentVolumeClaim, and on a StorageClass, as either a label or an +annotation. The first of them that carries the value `true` opts the pod in, which allows a single workload to opt in +without changing the StorageClass every volume shares. + +```bash title="Opting a single workload in through its claim" +kubectl label pvc my-pvc -n simplyblock \ + simplyblock.io/auto-restart-on-pathloss=true +``` + +### When a Pod Is Restarted + +A restart is deliberately slow to trigger, because a pod that is restarted while its storage is still gone comes back +no healthier. Several conditions have to hold at once. + +- The volume has been unusable for at least 30 seconds. +- The cluster the volume belongs to reports itself active again. A volume whose cluster is still down is left alone, + however long it has been broken. +- The pod has an owning controller, so that something recreates it. A bare pod is never deleted, since deleting it + would remove the workload rather than restart it. +- The pod is outside the restart backoff, which is ten minutes per pod. + +The guardian evaluates this every five minutes, and a grace period of 90 seconds after the first broken volume gives +the cluster time to report its own state before any decision is made. A restart is carried out by deleting the pod and +letting its controller recreate it. + +### Volumes Sharing an NVMe Subsystem + +A volume provisioned with `max_namespace_per_subsys` above `1` shares its NVMe-oF subsystem with other volumes, so +tearing its paths down would disconnect volumes belonging to other pods. Such pods are restarted as a group, all at +once, and only when every pod in the group passes the checks above. A single pod that fails them suppresses the whole +group, and an event on that pod names it as the blocker. + +| Event reason | Emitted on | Meaning | +|-----------------------------|------------------|---------------------------------------------------------------------------| +| `AutoRestartSuppressed` | The affected pod | The volume shares a subsystem and a coordinated restart was not possible. | +| `CoordinatedRestartBlocked` | The blocking pod | This pod prevented the restart of its subsystem group. | + +```bash title="Checking for suppressed restarts" +kubectl get events -A --field-selector reason=AutoRestartSuppressed +kubectl get events -A --field-selector reason=CoordinatedRestartBlocked +``` + +The message of a `CoordinatedRestartBlocked` event names what the blocking pod is missing, which is usually the opt-in +on its own controller. + +### Disabling the Restart for One Workload + +A pod that must never be restarted automatically carries the opt-out key, which wins over any opt-in inherited from +its claim or its StorageClass. + +```yaml title="Example of a pod excluded from the automatic restart" +metadata: + labels: + simplyblock.io/guardian-disable: "true" +``` + +## Checking the Paths of a Volume + +The paths of a volume are inspected on the worker that consumes it. The subsystem carries the volume UUID in its NQN. + +```bash title="Listing the NVMe subsystems on a worker" +kubectl debug node/worker-1.example.com -it --image=busybox -- \ + nvme list-subsys +``` + +A subsystem that is connected but exports no namespace is the state the node plugin repairs. A subsystem that is +missing entirely means the volume is not staged on this worker. + +Further diagnosis of the CSI driver is covered in +[Kubernetes CSI Troubleshooting](../../reference/troubleshooting/simplyblock-csi.md). + +## Related Operations + +A path loss that follows from a planned operation is expected, and the operation itself reports its progress. See +[Storage Node Actions](storage-node-actions.md) for a single node, [Rolling Restart](rolling-restart.md) for a whole +cluster, and [Coordinated Worker Node Drain](node-drain-coordination.md) for a worker going down for maintenance. diff --git a/docs/kubernetes/operations/replacing-a-storage-node.md b/docs/kubernetes/operations/replacing-a-storage-node.md new file mode 100644 index 00000000..f8a44ae4 --- /dev/null +++ b/docs/kubernetes/operations/replacing-a-storage-node.md @@ -0,0 +1,123 @@ +--- +title: "Replacing a Storage Node" +description: "Replace the host under a simplyblock storage node, or retire a node and add a replacement, without making the logical volumes it owns inaccessible." +weight: 10750 +--- + +A storage cluster is designed to stay up, so replacing a storage node is an online operation. Which operation it is +depends on what is being replaced. Moving a node onto different hardware keeps the node, while retiring a node and +putting a new one in its place does not. + +!!! danger + A storage node must never be removed and re-added to replace it. Removing a node without draining it first makes + the logical volumes it owns inaccessible. The two procedures below both avoid that, and neither of them deletes a + node that still holds volumes. + +## Choosing the Procedure + +| Situation | Procedure | +|------------------------------------------------------------------------|---------------------------------------------------------------------------| +| The host has to be swapped, and the node should keep its data. | [Relocate the node](#relocating-the-node) | +| The node is being retired, and its capacity is replaced by a new node. | [Retire and replace](#retiring-and-replacing) | +| The worker is only down for maintenance and comes back. | Neither. See [Coordinated Worker Node Drain](node-drain-coordination.md). | + +Relocating is the cheaper of the two by a wide margin. The node keeps its backend identity, its devices, and its +logical volume assignments, so no volume data is copied between nodes. Retiring moves every volume off the node first, +which is a data movement across the cluster. + +## Relocating the Node + +The `migrate` action moves a storage node onto a different Kubernetes worker. The node keeps its UUID and its volumes, +and the cluster rebalances afterward. + +This is the procedure for a host that is being replaced, decommissioned, or taken out of the storage plane while its +storage node lives on. + +1. Bring the replacement worker into the cluster and confirm it is `Ready`. The devices it offers have to match what + the node expects, either at the same PCIe addresses as on the old host or declared through `spec.newSsdPcie`. +2. Create a `StorageNodeOps` resource with `action: migrate` and the new worker as `targetWorkerNode`. +3. Follow the operation to `Succeeded`. + +```bash title="Relocating a storage node onto a replacement worker" +kubectl apply -n simplyblock -f - < Date: Sat, 22 Aug 2026 14:57:21 +0200 Subject: [PATCH 3/3] docs: address the review comments on the new operations pages - path-loss-recovery.md: the kubectl debug example used the busybox image, which ships no nvme binary, so the command could not work. Replaced with `sudo nvme list-subsys` on the worker, matching how reference/troubleshooting/simplyblock-csi.md already documents the same check. - removing-a-storage-node.md: the two blocking-volume bullets are term-definition items and now use the house `**Term:** Definition` form instead of carrying the term mid-sentence. Co-Authored-By: Claude Opus 5 (1M context) --- docs/kubernetes/operations/path-loss-recovery.md | 6 +++--- docs/kubernetes/operations/removing-a-storage-node.md | 9 +++++---- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/kubernetes/operations/path-loss-recovery.md b/docs/kubernetes/operations/path-loss-recovery.md index 051a6c18..51e60452 100644 --- a/docs/kubernetes/operations/path-loss-recovery.md +++ b/docs/kubernetes/operations/path-loss-recovery.md @@ -99,11 +99,11 @@ metadata: ## Checking the Paths of a Volume -The paths of a volume are inspected on the worker that consumes it. The subsystem carries the volume UUID in its NQN. +The paths of a volume are inspected on the worker that consumes it, with `nvme-cli` installed there. The subsystem +carries the volume UUID in its NQN. ```bash title="Listing the NVMe subsystems on a worker" -kubectl debug node/worker-1.example.com -it --image=busybox -- \ - nvme list-subsys +sudo nvme list-subsys ``` A subsystem that is connected but exports no namespace is the state the node plugin repairs. A subsystem that is diff --git a/docs/kubernetes/operations/removing-a-storage-node.md b/docs/kubernetes/operations/removing-a-storage-node.md index f5e6704d..02b015b0 100644 --- a/docs/kubernetes/operations/removing-a-storage-node.md +++ b/docs/kubernetes/operations/removing-a-storage-node.md @@ -57,10 +57,11 @@ unmanaged, for example, because it was created outside Kubernetes. The drain does not start while any volume blocks it: -- A **pinned volume**, meaning a PVC carrying the `simplyblock.io/selected-storage-node` annotation, blocks the drain. - A `PinnedVolumeBlocking` event names how many are affected, and the annotation has to be removed for the drain to - proceed. See [Pinned Volumes](volume-migration.md#pinned-volumes). -- An **unmanaged volume** blocks the drain with an `UnmanagedVolumeBlocking` event and has to be removed by hand. +- **Pinned volume:** A PVC carrying the `simplyblock.io/selected-storage-node` annotation. A `PinnedVolumeBlocking` + event names how many are affected, and the annotation has to be removed for the drain to proceed. See + [Pinned Volumes](volume-migration.md#pinned-volumes). +- **Unmanaged volume:** A volume without a `PersistentVolume`. An `UnmanagedVolumeBlocking` event is emitted, and the + volume has to be removed by hand. Both checks are rechecked every 60 seconds, so a drain that is blocked resumes on its own once the cause is cleared.