From 71a1bbdb6a991f5d6fb5542c4c45de13e097791f Mon Sep 17 00:00:00 2001 From: "Christoph Engelbert (noctarius)" Date: Sat, 22 Aug 2026 10:22:55 +0200 Subject: [PATCH] Updated async replication documentation --- docs/architecture/concepts/replication.md | 89 +-- .../operations/asynchronous-replication.md | 384 ++++++++--- .../operations/asynchronous-replication.md | 2 +- docs/reference/kubernetes/index.md | 4 - docs/reference/operator/reference.md | 648 ++++++++++++------ 5 files changed, 795 insertions(+), 332 deletions(-) diff --git a/docs/architecture/concepts/replication.md b/docs/architecture/concepts/replication.md index 68858ac7..b54ce666 100644 --- a/docs/architecture/concepts/replication.md +++ b/docs/architecture/concepts/replication.md @@ -5,9 +5,9 @@ weight: 30800 --- Simplyblock supports asynchronous replication between clusters for multi-site disaster recovery and data -availability. Replication ensures that snapshots of volumes on a source cluster are continuously transferred to a -remote target cluster, enabling recovery from site-level failures with automatic failover detection and controlled -failback. +availability. Snapshots of the volumes on a source cluster are transferred continuously to a remote target cluster. +After a site-level failure, the volumes are switched over to the target, and the switch is reversed once the source +cluster has been recovered. ## Snapshot Replication @@ -21,12 +21,12 @@ Key characteristics: seconds). - **Incremental:** Snapshots are chained on the target. Each replicated snapshot references its predecessor, enabling efficient copy-on-write storage. -- **Pool or Volume Scope:** Replication can be enabled for specific volumes. All volumes with replication enabled in a - cluster are managed by a single replication relationship. -- **Per-Volume Tracking:** The operator tracks replication status per volume, including last replicated snapshot, - replication count, and timestamps. +- **Per-Volume Scope:** Replication is enabled per volume. Volumes of the same cluster can replicate to different + target clusters, and under different schedules. +- **Per-Volume Tracking:** The replication state of every volume is tracked separately, including the timestamp of its + last replicated snapshot and the direction of its relationship. - **Automatic Task Management:** Each replication cycle creates a background task that handles the data transfer - asynchronously. The operator waits for the previous task to complete before triggering the next cycle. + asynchronously. The next cycle is only triggered once the previous task has completed. Snapshot replication is suitable for disaster recovery scenarios where a recovery point objective (RPO) of minutes is acceptable. It can also be used for local and global CDN-like data distribution processes or for the site migration of @@ -34,15 +34,30 @@ clusters. !!! info Basic remote snapshot replication is available on any platform via CLI/API, but full asynchronous replication - with fail-over and fail-back is only available on Kubernetes. + with failover and failback is only available on Kubernetes. + +## Replication Relationships + +A replication relationship is described by three layers, and each of them is configured separately. + +- **The cluster pair** names the source and the target cluster. It provisions the replication target on the backend + and is reusable, so several schedules can replicate between the same two clusters. +- **The policy** carries the cadence, the snapshot retention, and the mode of a pair. A `failover` policy keeps the + target a read-only standby for disaster recovery, while a `migration` policy prepares a planned cutover to the + target cluster. +- **The slot** exists once per replicated volume. It holds the live state of that volume and the direction of its + relationship, which states whether the local cluster currently serves the volume or holds the replica. + +A volume is enrolled by naming a policy, either for a whole storage class or for a single volume. Both clusters have +to be attached to the same control plane, since the relationship is resolved against the clusters it knows. ## Replication Architecture The replication system involves three components: -1. **Simplyblock Operator** ([Simplyblock Operator](https://github.com/simplyblock/simplyblock-manager){:target="_blank" rel="noopener"}): A Kubernetes - operator that watches the `SnapshotReplication` CRD and orchestrates replication cycles. It detects - failover conditions and manages the failback process. +1. **Simplyblock Operator** ([Simplyblock Operator](https://github.com/simplyblock/simplyblock-operator){:target="_blank" rel="noopener"}): A Kubernetes + operator that reconciles the replication resources into control plane calls. It attaches and detaches volumes, + tracks their state, and carries out the failover and failback operations that are requested of it. 2. **Control Plane** (sbcli): The simplyblock management API handles the actual snapshot creation, data transfer via NVMe-oF connections, and snapshot chain management on both source and target clusters. @@ -52,17 +67,13 @@ The replication system involves three components: ## Failover -Failover is triggered **automatically** when the operator detects that the source cluster is in a failure state: +Failover is never started by the operator on its own. Cluster state alone does not distinguish a lost site from a +transient outage, so the switch is requested explicitly. A request covers a single volume, every volume of one policy, +or every volume replicating to one target cluster. -- The source cluster status is `suspended`, **or** -- All storage nodes in the source cluster are `unreachable`. - -When both conditions are met, the operator initiates a one-time volume switch (`replicate_lvol`) for each -replicated volume, effectively providing access to the full volume on the target cluster via new NVMe-oF paths. -The RPO is based on the latest completed snapshot replication. -The target volumes become primary and begin serving I/O. - -No manual action is required to trigger failover. The conditions are detected by the operator, which acts automatically. +The request results in a one-time volume switch for each affected volume, which provides access to the full volume on +the target cluster via new NVMe-oF paths. The target volumes become primary and begin serving I/O. The RPO is based on +the latest completed snapshot replication. !!! warning After failover, any data written to the source cluster since the last successful snapshot replication will not be @@ -76,31 +87,23 @@ any other site by setting up the replication path toward this new cluster. This as a new replication. Failback refers to the option to replicate the delta accumulated in the target cluster back to the source in case the -source cluster can be recovered at origin (e.g., after temporary outage or maintenance action). - -Failback is triggered **manually** by setting `action: failback` on the `SnapshotReplication` CRD after the -source cluster has been restored. - -The failback process for each volume: +source cluster can be recovered at origin (e.g., after temporary outage or maintenance action). Like a failover, it is +requested explicitly, and the volumes it covers are selected by the scope of the request. -1. **Trigger replication on target:** Create a snapshot on the target and replicate it back to the source to capture - changes made during failover. -2. **Wait for completion:** Poll until the replication task finishes. -3. **Suspend target volume:** Freeze I/O on the target to prevent further changes. -4. **Trigger final replication:** Capture and transfer the last delta since the previous replication. -5. **Wait for completion:** Ensure all data is synchronized. -6. **Delete target volume:** Remove the failover copy from the target cluster. -7. **Resume on source:** Notify the source cluster to resume serving the volume. +Failback runs in two phases per volume: -The failback process supports filtering volumes using `includeVolumeIDs` and `excludeVolumeIDs` for selective failback. +1. **Reverse replication:** A snapshot is taken on the target and replicated back to the source, transferring the bulk + of the changes that accumulated while the target was serving I/O. +2. **Commit:** The volume is frozen, the remaining delta is transferred, and the volume is handed back to the source + cluster, which resumes serving it as primary. !!! note - The two-phase replication (steps 1 and 4) minimizes the I/O freeze window. The first replication transfers the bulk - of changes while the target is still active. The second replication only needs to transfer the small delta - accumulated during the first transfer. + The two phases minimize the I/O freeze window. The first phase transfers the bulk of the changes while the target + is still active. The second phase only needs to transfer the small delta accumulated during the first transfer. ## Kubernetes Integration -In Kubernetes environments, replication is managed through the `SnapshotReplication` CRD. For Kubernetes -deployment and configuration details, see -[Kubernetes Helm Chart Parameters](../../reference/kubernetes/index.md). +On Kubernetes, every layer of a replication relationship is a custom resource, and a failover or a failback is +requested by creating a one-shot operation resource. For those resources, their fields, and the annotation that +enrolls a volume, see +[Asynchronous Replication](../../kubernetes/operations/asynchronous-replication.md). diff --git a/docs/kubernetes/operations/asynchronous-replication.md b/docs/kubernetes/operations/asynchronous-replication.md index 3f8dd9b3..2298bb49 100644 --- a/docs/kubernetes/operations/asynchronous-replication.md +++ b/docs/kubernetes/operations/asynchronous-replication.md @@ -1,151 +1,349 @@ --- title: "Asynchronous Replication" -description: "Configure and operate simplyblock snapshot-based asynchronous replication across clusters with automatic failover and manual failback." +description: "Configure snapshot-based asynchronous replication between simplyblock clusters with the ReplicationPair, ReplicationPolicy, and ReplicationOps resources." weight: 10650 --- -Simplyblock provides a snapshot-based asynchronous replication mechanism that replicates volumes at regular intervals. -For each interval, simplyblock takes a copy-on-write snapshot on the source and replicates it to a volume in a storage -pool on the target storage cluster. +Simplyblock replicates volumes between two storage clusters by transferring copy-on-write snapshots at a fixed +interval. For each interval, a snapshot is taken on the source cluster and replicated into the target cluster, where +the snapshots form an incremental chain. On Kubernetes, replication is declared through custom resources, and every +backend call is issued by the Simplyblock Operator. For the architecture background, see [Replication Concepts](../../architecture/concepts/replication.md). ## Scope and Prerequisites -- Asynchronous replication with automatic failover and controlled failback is a Kubernetes-only feature. -- It is managed by the Simplyblock Operator using the `SnapshotReplication` CRD. -- Source and target storage clusters must be attached to the same simplyblock control plane. -- The two simplyblock clusters (source and target) must have network interconnectivity. -- Both clusters must be activated and have storage nodes online. +Asynchronous replication with controlled failover and failback is a Kubernetes-only feature, managed by the +Simplyblock Operator. + +Both clusters have to be represented by a `StorageCluster` resource in the same namespace as the replication +resources, which means both are attached to the same simplyblock control plane. A cluster is referenced by the name +of its `StorageCluster`, and its UUID has to be reported in `status.uuid` before replication can be configured. +Cross-namespace references are not supported. Both clusters have to be active with their storage nodes online, and +the two clusters need network interconnectivity. !!! note - For multi-site setups (for example, DR / offsite failover), using a distributed control plane is highly recommended. - A typical setup is 2 management nodes on the main site and 3 management nodes on the failover site, so quorum / - consensus can be maintained during a site failure. + For multi-site setups, such as disaster recovery or offsite failover, a distributed control plane is highly + recommended. A typical setup is two management nodes on the main site and three management nodes on the failover + site, so that quorum is maintained during a site failure. -## Enabling Replication on Volumes +## Resource Model -Replication participation is controlled on volumes by setting `replicate: true`. +Replication is split across four resources. The first two are created by an administrator, the third is created by +the operator, and the fourth is created to trigger an operation. -```yaml title="Example enabling replication via a storage class" -apiVersion: storage.k8s.io/v1 -kind: StorageClass +| Resource | Short name | Cardinality | Purpose | +|---------------------|------------|------------------------|---------------------------------------------------------------------------| +| `ReplicationPair` | `relpair` | One per cluster pair | Declares the source and the target cluster of a replication relationship. | +| `ReplicationPolicy` | `repl` | One per schedule | Sets the cadence, the mode, and the snapshot retention of a pair. | +| `ReplicationSlot` | `relslot` | One per replicated PVC | Holds the live per-volume replication state. Created by the operator. | +| `ReplicationOps` | `replops` | One per operation | Triggers a failover or a failback and records its outcome. | + +A pair is reusable. Several policies can reference the same pair to replicate between the same two clusters with +different schedules or retention. + +## Declaring the Cluster Pair + +A `ReplicationPair` names the local cluster and the remote cluster. Creating it provisions the replication target on +the backend, and deleting it tears that target down. + +```yaml title="Example of a ReplicationPair between two clusters (replication-pair.yaml)" +apiVersion: storage.simplyblock.io/v1alpha1 +kind: ReplicationPair metadata: - name: encrypted-volumes -provisioner: csi.simplyblock.io -parameters: - replicate: true - ... other parameters -reclaimPolicy: Delete -volumeBindingMode: WaitForFirstConsumer -allowVolumeExpansion: true + name: site-a-to-site-b + namespace: simplyblock +spec: + sourceCluster: simplyblock-cluster + targetCluster: simplyblock-cluster-dr +``` + +```bash title="Creating the replication pair" +kubectl apply -f replication-pair.yaml +``` + +The pair is usable once `status.ready` is `true`, at which point the UUID of the backend replication target is +recorded in `status.backendTargetID`. + +```bash title="Checking the state of the replication pairs" +kubectl get replicationpair -n simplyblock +``` + +```plain title="Example output of the replication pair listing" +NAME SOURCE TARGET READY AGE +site-a-to-site-b simplyblock-cluster simplyblock-cluster-dr true 2m ``` -## Configuring Source and Target +`spec.targetCluster` is immutable. Replicating to a different cluster requires a new `ReplicationPair`. -Replication also requires a source/target cluster and target storage pool configuration via `SnapshotReplication`. +## Defining a Replication Policy -```yaml title="Example enabling asynchronous replication" +A `ReplicationPolicy` couples a pair to a schedule and a retention rule. It is the resource that volumes are attached +to. + +```yaml title="Example of a ReplicationPolicy for disaster recovery (replication-policy.yaml)" apiVersion: storage.simplyblock.io/v1alpha1 -kind: SnapshotReplication +kind: ReplicationPolicy metadata: - name: simplyblock-snapshot-replication + name: dr-policy namespace: simplyblock spec: - sourceCluster: - targetCluster: - targetPool: - interval: + pairRef: site-a-to-site-b + mode: failover + interval: 5m + snapshotRetention: 3 +``` + +```bash title="Creating the replication policy" +kubectl apply -f replication-policy.yaml +``` + +The policy waits for its pair to become ready before the backend policy is created. Once `status.ready` is `true`, +the backend policy UUID is held in `status.backendPolicyID`, and `status.slotCount` reports how many volumes are +currently attached. + +```bash title="Checking the state of the replication policies" +kubectl get replicationpolicy -n simplyblock +``` + +```plain title="Example output of the replication policy listing" +NAME PAIR MODE INTERVAL READY SLOTS AGE +dr-policy site-a-to-site-b failover 5m true 4 2m +``` + +### Policy Fields + +| Field | Type | Default | Description | +|---------------------|----------|------------|--------------------------------------------------------------------------------------------------------| +| `pairRef` | string | - | Name of the `ReplicationPair` in the same namespace. Required. | +| `mode` | string | `failover` | `failover` keeps the target a read-only standby. `migration` prepares a planned cutover to the target. | +| `interval` | duration | `5m` | How often a replication snapshot is taken. Rounded to whole minutes, with one minute as the minimum. | +| `snapshotRetention` | int | `3` | Minimum number of snapshots retained on the target. The lowest accepted value is `2`. | + +An interval that cannot be parsed as a duration falls back to five minutes. + +## Selecting the Volumes to Replicate + +A volume is opted into replication by the `storage.simplyblock.io/replication-policy` annotation, which names a +`ReplicationPolicy` in the namespace of the PVC. The annotation is read from the PVC and from its StorageClass, and +the annotation on the PVC wins when both carry one. Annotating the StorageClass therefore replicates every volume +provisioned from it, while annotating a single PVC replicates only that volume. + +```yaml title="Example of a StorageClass that replicates every volume it provisions" +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: simplyblock-dr + annotations: + storage.simplyblock.io/replication-policy: dr-policy +provisioner: csi.simplyblock.io +``` + +```bash title="Opting a single PVC into a replication policy" +kubectl annotate pvc my-pvc -n simplyblock \ + storage.simplyblock.io/replication-policy=dr-policy +``` + +The annotation is honored on a new PVC and on an already-bound one. A `ReplicationSlot` is created as soon as the PVC +is `Bound` and the named policy is ready, and it is named `-`. The slot is owned by its PVC, so deleting +the PVC deletes the slot and stops replication for that volume. + +!!! note + The `replicate` StorageClass parameter is unrelated to this mechanism. It is a backend volume property and does + not attach a volume to a `ReplicationPolicy`. See + [Storage Class: Available Parameters](../usage/storage-class.md#available-parameters). + +### Changing or Removing the Policy + +Pointing the annotation at a different policy is carried out as a detach followed by a fresh attach, which means the +new target receives a full copy of the volume. The existing slot is deleted first, and the replacement slot is +created once the detach has completed. + +Removing the annotation stops replication. The replication snapshots are deleted on both the source and the target, +and the slot is removed afterward. + +```bash title="Removing a PVC from replication" +kubectl annotate pvc my-pvc -n simplyblock \ + storage.simplyblock.io/replication-policy- ``` -### SnapshotReplication Spec Fields +## Replication Slots -| Field | Type | Description | -|--------------------|----------|-------------------------------------------------------------------------------------------| -| `sourceCluster` | string | Source simplyblock cluster name. Required. | -| `targetCluster` | string | Target simplyblock cluster name. Required. | -| `targetPool` | string | Target storage pool for replicated volumes. Required. | -| `interval` | int | Interval for creating new replication snapshots. Required. | -| `timeout` | int | Per-task replication timeout. Optional. Defaults to `60` seconds (control plane default). | -| `action` | string | Lifecycle action. Use `failback` to trigger failback after source recovery. | -| `sourcePool` | string | Source storage pool, required for failback workflows. | -| `includeVolumeIDs` | []string | Optional list of volumes to include in replication/failback. | -| `excludeVolumeIDs` | []string | Optional list of volumes to exclude from replication/failback. | +One `ReplicationSlot` exists per replicated volume and carries the state of that volume. The slot records the volume +handle of the source volume in `spec.volumeID`, in the form `::`. -## Replication Cycle and Queue Behavior +```bash title="Listing the replication slots of a namespace" +kubectl get replicationslot -n simplyblock +``` -At each configured interval: +```plain title="Example output of the replication slot listing" +NAME POLICY PVC STATE DIRECTION AGE +dr-policy-my-pvc dr-policy my-pvc replicating source 4m +dr-policy-logs dr-policy logs replicating source 4m +``` -1. A copy-on-write snapshot is taken for the logical volume. -2. A replication task is created and added to the replication queue. -3. Queue tasks are processed one-by-one. +`status.direction` states which side of the relationship the local cluster holds, so a slot reads `source` under +normal replication and `target` after a failover. -`SnapshotReplication.spec.interval` defines how often new snapshots are scheduled. +### Slot States -`SnapshotReplication.spec.timeout` limits the maximum runtime of a replication task. By default, the control plane -uses `60` seconds if not explicitly configured. +| State | Description | +|-------------------|-----------------------------------------------------------------------------------------------| +| `replicating` | Steady state. Snapshots are taken and transferred on the policy's interval. | +| `cutover_pending` | A planned cutover has been prepared on the backend and is awaiting its commit. | +| `cutover_done` | The cutover has completed. | +| `failed_over` | The volume is served by the target cluster. `status.targetNQN` carries the NQN on the target. | +| `detaching` | Replication is being stopped and the replication snapshots are being deleted on both sides. | +| `error` | A backend call failed. `status.message` holds the reason, and the attach is retried. | +| `attaching` | Legacy state, only seen on slots created by earlier operator versions. | -To avoid queue exhaustion from stacked tasks (for example, when replication is slower than snapshot creation, or the -target cluster is unreachable), set `timeout` lower than or up to a maximum of approximately `1.5x` the `interval`. +An attach is synchronous on the backend, so a new slot reaches `replicating` directly rather than passing through an +intermediate state. -## Monitoring Replication Status +## Monitoring Replication -The operator tracks replication progress per volume in the `SnapshotReplication` status. +While a slot is replicating, the operator polls the backend every 60 seconds and records the timestamp of the last +successful snapshot in `status.lastReplicatedAt`. After a failed backend call, the poll backs off to 30 seconds. -```bash -kubectl get snapshotreplication \ - simplyblock-snapshot-replication \ - -n simplyblock -o yaml +```bash title="Reading the replication state of a single volume" +kubectl get replicationslot dr-policy-my-pvc -n simplyblock \ + -o jsonpath='{.status}' | jq . ``` -Status includes per-volume replication details such as the last replicated snapshot, counters, and timestamps. +```bash title="Watching the replication lag across all volumes of a namespace" +kubectl get replicationslot -n simplyblock \ + -o custom-columns=NAME:.metadata.name,STATE:.status.state,LAST:.status.lastReplicatedAt +``` + +A state change that originates on the backend, such as a cutover or an externally triggered failover, is picked up by +the same poll and reflected into the slot. ## Failover -Failover to the target cluster happens automatically when the source cluster becomes unhealthy or unavailable. +A failover is never performed automatically. It is triggered by creating a `ReplicationOps` resource with +`action: failover`, which covers both the unplanned failover of a lost site and the planned cutover of a `migration` +policy. -- The source cluster status is `suspended`, or -- All storage nodes in the source cluster are `unreachable`. +```bash title="Failing over every volume of a policy to the target cluster" +kubectl apply -n simplyblock -f - <` | | `storagenode.create` | Specifies whether to create storage node on Kubernetes worker node. | `false` | | `storagenode.ifname` | Sets the default interface to be used for binding the storage node to host interface. | `eth0` | -| `storagenode.maxLogicalVolumes` | Sets the default maximum number of logical volumes per storage node. | `10` | | `storagenode.maxSnapshots` | Sets the default maximum number of snapshot per storage node. | `10` | -| `storagenode.maxSize` | Sets the max provisioning size of all storage nodes. | `` | -| `storagenode.numPartitions` | Sets the number of partitions to create per device. | `1` | | `storagenode.numDataChunks` | Sets default NDCS value used by storage-node automation. | `1` | | `storagenode.numParityChunks` | Sets default NPCS value used by storage-node automation. | `1` | | `storagenode.isolateCores` | Enables automatic core isolation. | `false` | @@ -126,7 +123,6 @@ For details, see [Securing the Control Plane](../../kubernetes/installation/secu | `storagenode.format4k` | Enables 4K-format handling during storage-node setup. | `false` | | `storagenode.socketsToUse` | Sets the list of sockets to use. | `` | | `storagenode.nodesPerSocket` | Sets the number of nodes to use per socket. | `` | -| `storagenode.coresPercentage` | Sets the percentage of total cores (vCPUs) available to simplyblock storage node services. | `` | | `storagenode.ubuntuHost` | Set to true if the worker node runs Ubuntu and needs the nvme-tcp kernel module installed. | `false` | | `storagenode.enableCpuTopology` | Enables CPU topology configuration on storage nodes. | `false` | | `storagenode.enableDevicePlugin` | Enables NUMA resource device plugin deployment. | `true` | diff --git a/docs/reference/operator/reference.md b/docs/reference/operator/reference.md index 14b426ca..07867591 100644 --- a/docs/reference/operator/reference.md +++ b/docs/reference/operator/reference.md @@ -24,7 +24,10 @@ Package v1alpha1 contains API Schema definitions for the simplyblock v1alpha1 AP - [BackupPolicy](#backuppolicy) - [BackupRestore](#backuprestore) - [ControlPlane](#controlplane) -- [SnapshotReplication](#snapshotreplication) +- [ReplicationOps](#replicationops) +- [ReplicationPair](#replicationpair) +- [ReplicationPolicy](#replicationpolicy) +- [ReplicationSlot](#replicationslot) - [StorageBackup](#storagebackup) - [StorageCluster](#storagecluster) - [StorageClusterOps](#storageclusterops) @@ -361,7 +364,6 @@ status: pvcName: string pvcNamespace: string sourceClusterUUID: string - sourceSwitchedAt: Time startedAt: Time completedAt: Time ``` @@ -436,7 +438,6 @@ pvName: string pvcName: string pvcNamespace: string sourceClusterUUID: string -sourceSwitchedAt: Time startedAt: Time completedAt: Time ``` @@ -455,8 +456,7 @@ completedAt: Time | `pvName` _string_ | PVName is the name of the PersistentVolume created by the controller. | | | | `pvcName` _string_ | PVCName is the name of the PersistentVolumeClaim created from pvcTemplate. | | | | `pvcNamespace` _string_ | PVCNamespace is the namespace of the created PVC. | | | -| `sourceClusterUUID` _string_ | SourceClusterUUID is the UUID of the cluster that originally created the backup.
Copied from the referenced StorageBackup's status.sourceClusterUUID.
When non-empty, the controller performs source-switch before and after the restore. | | | -| `sourceSwitchedAt` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#time-v1-meta)_ | SourceSwitchedAt records when the target cluster was switched to read from the
source cluster's S3 bucket. Cleared once source-switch local completes. | | | +| `sourceClusterUUID` _string_ | SourceClusterUUID is the UUID of the cluster that originally created the backup.
Copied from the referenced StorageBackup's status.sourceClusterUUID. When non-empty
and different from ClusterUUID, the controller resolves that cluster's backup
credentials and sends them with the restore request, since the backup's bucket may
not be this cluster's own. | | | | `startedAt` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#time-v1-meta)_ | StartedAt is when the backend restore task was accepted. | | | | `completedAt` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#time-v1-meta)_ | CompletedAt is when the PVC became bound. | | | @@ -623,12 +623,14 @@ _Example:_ ```yaml enabled: boolean interval: Duration +minMoves: integer ``` | Field | Description | Default | Validation | | --- | --- | --- | --- | | `enabled` _boolean_ | Enabled activates automatic post-migration data realignment for this cluster.
Defaults to true. | | Optional: \{\}
| -| `interval` _[Duration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#duration-v1-meta)_ | Interval is how often the operator checks whether a realignment is pending
(i.e. at least one volume has moved since the last successful realignment) and,
if so, triggers it. Explicit triggers (see the
simplyblock.io/trigger-realignment annotation) bypass this spacing. Defaults to
10m. | | Optional: \{\}
| +| `interval` _[Duration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#duration-v1-meta)_ | Interval is how often the operator checks whether a realignment is pending
(i.e. at least one volume has moved since the last successful realignment) and,
if so, triggers it. Explicit triggers (see the
simplyblock.io/trigger-realignment annotation) bypass this spacing. Defaults to
10m.
Note that this is a floor on the spacing between realignment *requests*, not a
ceiling on how long one takes: a realignment blocks all volume migrations for as
long as the control plane needs, which on a busy cluster has been measured at
tens of minutes. An interval shorter than that means the next realignment is
requested as soon as the previous one finishes and any volume has moved, which is
what MinMoves exists to damp. | | Optional: \{\}
| +| `minMoves` _integer_ | MinMoves is how many volume moves must accumulate before a realignment is
triggered. Defaults to 1: every completed migration schedules a realignment.
Raise it to batch. Because the control plane refuses new migrations while a
realignment runs, a value of 1 makes the two alternate — one migration completes,
a realignment follows and blocks migrations until it is done. On a cluster where
realignment takes tens of minutes that is most of the available time, so a run
that migrates continuously spends the majority of it waiting. A higher value
trades realignment promptness (data structures stay unaligned for longer, so
fault-tolerance and node-affinity guarantees are restored later) for migration
throughput.
Explicit triggers (the simplyblock.io/trigger-realignment annotation) ignore this
threshold, so a drain or node removal still realigns immediately. | | Minimum: 1
Optional: \{\}
| #### DrainOpsSpec @@ -1069,35 +1071,158 @@ nodeMetrics: | `nodeMetrics` _[NodeLoadMetrics](#nodeloadmetrics) array_ | | | | -#### ReplicationError +#### ReplicationOps -ReplicationError stores timestamped error messages +ReplicationOps is a one-shot user-driven CR for imperative replication operations: +failover (planned or unplanned) and failback. The operator drives the backend calls +to completion and records per-volume outcomes in status.results. Only one ReplicationOps +may be active per ReplicationPolicy at a time, enforced via ReplicationPolicy.status.activeOpsRef. + + + + + +_Example:_ + +```yaml +apiVersion: storage.simplyblock.io/v1alpha1 +kind: ReplicationOps +metadata: + name: string +spec: + action: string + scope: string + ref: string + sourceClusterID: string +status: + phase: string + subphase: string + message: string + startedAt: Time + completedAt: Time + results: + - slotRef: string + status: string + detail: string + targetLvolID: string +``` + +| Field | Description | Default | Validation | +| --- | --- | --- | --- | +| `apiVersion` _string_ | `storage.simplyblock.io/v1alpha1` | | | +| `kind` _string_ | `ReplicationOps` | | | +| `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | +| `spec` _[ReplicationOpsSpec](#replicationopsspec)_ | | | | +| `status` _[ReplicationOpsStatus](#replicationopsstatus)_ | | | | + + + + +#### ReplicationOpsResult + + + +ReplicationOpsResult holds the outcome for a single volume in a ReplicationOps. + + + +_Appears in:_ +- [ReplicationOpsStatus](#replicationopsstatus) + +_Example:_ + +```yaml +slotRef: string +status: string +detail: string +targetLvolID: string +``` + +| Field | Description | Default | Validation | +| --- | --- | --- | --- | +| `slotRef` _string_ | SlotRef is the name of the ReplicationSlot CR. | | | +| `status` _string_ | Status is the outcome for this volume. | | Enum: [succeeded skipped failed]
| +| `detail` _string_ | Detail is an optional human-readable note (error message or skip reason). | | Optional: \{\}
| +| `targetLvolID` _string_ | TargetLvolID is the UUID of the volume on the target cluster (failover only). | | Optional: \{\}
| + + + + +#### ReplicationOpsSpec + + + +ReplicationOpsSpec defines the desired state of a ReplicationOps. + + + +_Appears in:_ +- [ReplicationOps](#replicationops) + +_Example:_ + +```yaml +action: string +scope: string +ref: string +sourceClusterID: string +``` + +| Field | Description | Default | Validation | +| --- | --- | --- | --- | +| `action` _string_ | Action is the operation to perform. Immutable. | | Enum: [failover failback]
Required: \{\}
| +| `scope` _string_ | Scope controls which volumes are affected. Immutable.
target: all volumes across every policy that uses the named ReplicationPair.
policy: all volumes managed by the named ReplicationPolicy CR.
volume: a single ReplicationSlot (unplanned per-volume failover). | | Enum: [target policy volume]
Required: \{\}
| +| `ref` _string_ | Ref is the name of the resource identified by Scope:
a ReplicationPair name for scope=target,
a ReplicationPolicy name for scope=policy,
or a ReplicationSlot name for scope=volume. Immutable. | | Required: \{\}
| +| `sourceClusterID` _string_ | SourceClusterID is used for failback only. Omit to recover to the original source. | | Optional: \{\}
| + + +#### ReplicationOpsStatus + + + +ReplicationOpsStatus holds the observed state of a ReplicationOps. _Appears in:_ -- [VolumeReplicationStatus](#volumereplicationstatus) +- [ReplicationOps](#replicationops) _Example:_ ```yaml -timestamp: Time +phase: string +subphase: string message: string +startedAt: Time +completedAt: Time +results: + - slotRef: string + status: string + detail: string + targetLvolID: string ``` | Field | Description | Default | Validation | | --- | --- | --- | --- | -| `timestamp` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#time-v1-meta)_ | | | | -| `message` _string_ | | | | +| `phase` _string_ | Phase is the current lifecycle phase of this operation. | | Enum: [Pending Running Succeeded Failed]
Optional: \{\}
| +| `subphase` _string_ | Subphase describes what the operation is currently doing within the phase
(e.g. "TriggeringFailover", "UpdatingSlotStatuses", "ReleasingLock"). | | Optional: \{\}
| +| `message` _string_ | Message is a human-readable description of the current phase. | | Optional: \{\}
| +| `startedAt` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#time-v1-meta)_ | StartedAt is when the operation began. | | Optional: \{\}
| +| `completedAt` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#time-v1-meta)_ | CompletedAt is when the operation finished (successfully or not). | | Optional: \{\}
| +| `results` _[ReplicationOpsResult](#replicationopsresult) array_ | Results holds a per-volume summary of the operation outcome. | | Optional: \{\}
| -#### SnapshotReplication +#### ReplicationPair -SnapshotReplication is the Schema for the snapshotreplications API +ReplicationPair defines the source and target clusters for a replication relationship. +It is reusable configuration — multiple ReplicationPolicies may reference the same pair +to replicate volumes between the same two clusters with different schedules or retention. +The operator ensures the corresponding backend ReplicationTarget exists and stores its ID +in status.backendTargetID for use by ReplicationPolicy resources. @@ -1107,126 +1232,300 @@ _Example:_ ```yaml apiVersion: storage.simplyblock.io/v1alpha1 -kind: SnapshotReplication +kind: ReplicationPair metadata: name: string spec: sourceCluster: string targetCluster: string - targetPool: string - sourcePool: string - timeout: integer - interval: integer - action: string - includeVolumeIDs: - - string - excludeVolumeIDs: - - string - volumeIDs: - - string status: - configured: boolean - observedFailbackGeneration: integer - volumes: - - volumeID: string - phase: string - lastSnapshotID: string - lastReplicationTime: Time - replicatedCount: integer - errors: - - timestamp: Time - message: string + ready: boolean + backendTargetID: string + message: string conditions: - Condition + activeOpsRef: string ``` | Field | Description | Default | Validation | | --- | --- | --- | --- | | `apiVersion` _string_ | `storage.simplyblock.io/v1alpha1` | | | -| `kind` _string_ | `SnapshotReplication` | | | -| `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | Optional: \{\}
| -| `spec` _[SnapshotReplicationSpec](#snapshotreplicationspec)_ | spec defines the desired state of SnapshotReplication | | Required: \{\}
| -| `status` _[SnapshotReplicationStatus](#snapshotreplicationstatus)_ | status defines the observed state of SnapshotReplication | | Optional: \{\}
| +| `kind` _string_ | `ReplicationPair` | | | +| `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | +| `spec` _[ReplicationPairSpec](#replicationpairspec)_ | | | | +| `status` _[ReplicationPairStatus](#replicationpairstatus)_ | | | | -#### SnapshotReplicationSpec +#### ReplicationPairSpec -SnapshotReplicationSpec defines the desired state of SnapshotReplication +ReplicationPairSpec defines the source and target clusters for a replication relationship. _Appears in:_ -- [SnapshotReplication](#snapshotreplication) +- [ReplicationPair](#replicationpair) _Example:_ ```yaml sourceCluster: string targetCluster: string -targetPool: string -sourcePool: string -timeout: integer -interval: integer -action: string -includeVolumeIDs: - - string -excludeVolumeIDs: - - string -volumeIDs: - - string ``` | Field | Description | Default | Validation | | --- | --- | --- | --- | -| `sourceCluster` _string_ | Source cluster for the snapshots | | | -| `targetCluster` _string_ | Target cluster for replication | | | -| `targetPool` _string_ | Target cluster pool for replication | | | -| `sourcePool` _string_ | required for failback to a fresh source cluster | | | -| `timeout` _integer_ | snapshot replication timeout | | | -| `interval` _integer_ | snapshot replication interval in seconds (default: 300sec) | | | -| `action` _string_ | | | Enum: [failback]
| -| `includeVolumeIDs` _string array_ | Optional: only these volumes are included in failback.
If empty, all volumes are candidates unless excluded below. | | | -| `excludeVolumeIDs` _string array_ | Optional: volumes to exclude from failback. | | | -| `volumeIDs` _string array_ | Optional: list of volumes to replicate. Empty means all volumes | | | +| `sourceCluster` _string_ | SourceCluster is the name of the local StorageCluster (the replication source). | | Required: \{\}
| +| `targetCluster` _string_ | TargetCluster is the name or UUID of the remote cluster (the replication target).
Immutable after creation. | | Required: \{\}
| -#### SnapshotReplicationStatus +#### ReplicationPairStatus -SnapshotReplicationStatus defines the observed state of SnapshotReplication. +ReplicationPairStatus holds the observed state of a ReplicationPair. _Appears in:_ -- [SnapshotReplication](#snapshotreplication) +- [ReplicationPair](#replicationpair) _Example:_ ```yaml -configured: boolean -observedFailbackGeneration: integer -volumes: - - volumeID: string - phase: string - lastSnapshotID: string - lastReplicationTime: Time - replicatedCount: integer - errors: - - timestamp: Time - message: string +ready: boolean +backendTargetID: string +message: string conditions: - Condition +activeOpsRef: string ``` | Field | Description | Default | Validation | | --- | --- | --- | --- | -| `configured` _boolean_ | | | | -| `observedFailbackGeneration` _integer_ | The metadata.generation value for which failback was last processed. | | | -| `volumes` _[VolumeReplicationStatus](#volumereplicationstatus) array_ | Per-volume replication status | | | -| `conditions` _[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#condition-v1-meta) array_ | Conditions provides human-readable status conditions for kubectl get output. | | | +| `ready` _boolean_ | Ready is true when the backend ReplicationTarget has been created and is available. | | Optional: \{\}
| +| `backendTargetID` _string_ | BackendTargetID is the UUID of the backend ReplicationTarget resource. | | Optional: \{\}
| +| `message` _string_ | Message provides a human-readable description of the current state. | | Optional: \{\}
| +| `conditions` _[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#condition-v1-meta) array_ | Conditions holds standard Kubernetes condition types. | | Optional: \{\}
| +| `activeOpsRef` _string_ | ActiveOpsRef is the name of the ReplicationOps currently holding the
target-scope lock on this pair. Only one scope=target ReplicationOps may
be active per pair at a time. | | Optional: \{\}
| + + +#### ReplicationPolicy + + + +ReplicationPolicy defines the replication schedule and retention for volumes replicated +between the clusters defined by a ReplicationPair. +A StorageClass or PVC references a policy via the storage.simplyblock.io/replication-policy +annotation. The operator automatically creates one ReplicationSlot per bound PVC. +Deletion is blocked while any ReplicationSlots reference this policy. + + + + + +_Example:_ + +```yaml +apiVersion: storage.simplyblock.io/v1alpha1 +kind: ReplicationPolicy +metadata: + name: string +spec: + pairRef: string + mode: string + interval: string + snapshotRetention: integer +status: + ready: boolean + backendPolicyID: string + slotCount: integer + activeOpsRef: string + conditions: + - Condition +``` + +| Field | Description | Default | Validation | +| --- | --- | --- | --- | +| `apiVersion` _string_ | `storage.simplyblock.io/v1alpha1` | | | +| `kind` _string_ | `ReplicationPolicy` | | | +| `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | +| `spec` _[ReplicationPolicySpec](#replicationpolicyspec)_ | | | | +| `status` _[ReplicationPolicyStatus](#replicationpolicystatus)_ | | | | + + +#### ReplicationPolicySpec + + + +ReplicationPolicySpec defines the desired replication schedule and retention. + + + +_Appears in:_ +- [ReplicationPolicy](#replicationpolicy) + +_Example:_ + +```yaml +pairRef: string +mode: string +interval: string +snapshotRetention: integer +``` + +| Field | Description | Default | Validation | +| --- | --- | --- | --- | +| `pairRef` _string_ | PairRef is the name of the ReplicationPair that defines the source and target clusters.
Multiple ReplicationPolicies may reference the same pair with different schedules. | | Required: \{\}
| +| `mode` _string_ | Mode controls replication semantics.
failover: target is a DR standby; volumes are read-only on the target.
migration: planned online cutover to the target cluster. | failover | Enum: [failover migration]
Optional: \{\}
| +| `interval` _string_ | Interval is how often a replication snapshot is taken (e.g. "5m", "1h"). | 5m | Optional: \{\}
| +| `snapshotRetention` _integer_ | SnapshotRetention is the minimum number of snapshots to retain on the target. | 3 | Minimum: 2
Optional: \{\}
| + + +#### ReplicationPolicyStatus + + + +ReplicationPolicyStatus holds the observed state of a ReplicationPolicy. + + + +_Appears in:_ +- [ReplicationPolicy](#replicationpolicy) + +_Example:_ + +```yaml +ready: boolean +backendPolicyID: string +slotCount: integer +activeOpsRef: string +conditions: + - Condition +``` + +| Field | Description | Default | Validation | +| --- | --- | --- | --- | +| `ready` _boolean_ | Ready is true when the backend ReplicationPolicy has been created. | | Optional: \{\}
| +| `backendPolicyID` _string_ | BackendPolicyID is the UUID of the backend ReplicationPolicy resource. | | Optional: \{\}
| +| `slotCount` _integer_ | SlotCount is the number of ReplicationSlot CRs currently managed by this policy. | | Optional: \{\}
| +| `activeOpsRef` _string_ | ActiveOpsRef is the name of the currently running ReplicationOps CR.
Empty when no operation is in progress. | | Optional: \{\}
| +| `conditions` _[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#condition-v1-meta) array_ | Conditions holds standard Kubernetes condition types. | | Optional: \{\}
| + + +#### ReplicationSlot + + + +ReplicationSlot tracks the live replication state for a single PVC. +One ReplicationSlot is created per PVC by the PVCAnnotationWatcher controller when +a PVC references a ReplicationPolicy via annotation. It is owned by its PVC, so +deleting the PVC cascades deletion and triggers a backend detach via the slot finalizer. +The ReplicationSlot reconciler drives all backend calls: attach, monitor, cutover, +failover, and detach. + + + + + +_Example:_ + +```yaml +apiVersion: storage.simplyblock.io/v1alpha1 +kind: ReplicationSlot +metadata: + name: string +spec: + policyRef: string + pvcRef: string + volumeID: string +status: + state: string + direction: string + sourceLvolID: string + targetLvolID: string + targetNQN: string + lastReplicatedAt: Time + message: string + conditions: + - Condition +``` + +| Field | Description | Default | Validation | +| --- | --- | --- | --- | +| `apiVersion` _string_ | `storage.simplyblock.io/v1alpha1` | | | +| `kind` _string_ | `ReplicationSlot` | | | +| `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. | | | +| `spec` _[ReplicationSlotSpec](#replicationslotspec)_ | | | | +| `status` _[ReplicationSlotStatus](#replicationslotstatus)_ | | | | + + + + +#### ReplicationSlotSpec + + + +ReplicationSlotSpec defines the identity of a per-volume replication slot. + + + +_Appears in:_ +- [ReplicationSlot](#replicationslot) + +_Example:_ + +```yaml +policyRef: string +pvcRef: string +volumeID: string +``` + +| Field | Description | Default | Validation | +| --- | --- | --- | --- | +| `policyRef` _string_ | PolicyRef is the name of the ReplicationPolicy governing this slot. Immutable. | | Required: \{\}
| +| `pvcRef` _string_ | PVCRef is the name of the PVC being replicated. Immutable. | | Required: \{\}
| +| `volumeID` _string_ | VolumeID is the backend lvol UUID of the source volume. Immutable.
Format: "::" | | Required: \{\}
| + + + + +#### ReplicationSlotStatus + + + +ReplicationSlotStatus holds the observed state of a ReplicationSlot. + + + +_Appears in:_ +- [ReplicationSlot](#replicationslot) + +_Example:_ + +```yaml +state: string +direction: string +sourceLvolID: string +targetLvolID: string +targetNQN: string +lastReplicatedAt: Time +message: string +conditions: + - Condition +``` + +| Field | Description | Default | Validation | +| --- | --- | --- | --- | +| `state` _string_ | State is the current replication state for this slot. | | Enum: [attaching replicating cutover_pending cutover_done failed_over detaching error]
Optional: \{\}
| +| `direction` _string_ | Direction is which side of the replication relationship this cluster holds. | | Enum: [source target]
Optional: \{\}
| +| `sourceLvolID` _string_ | SourceLvolID is the UUID of the source volume on the source cluster. | | Optional: \{\}
| +| `targetLvolID` _string_ | TargetLvolID is the UUID of the replicated volume on the target cluster. | | Optional: \{\}
| +| `targetNQN` _string_ | TargetNQN is the NVMe NQN on the target cluster (populated after failover). | | Optional: \{\}
| +| `lastReplicatedAt` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#time-v1-meta)_ | LastReplicatedAt is the timestamp of the last successful replication snapshot. | | Optional: \{\}
| +| `message` _string_ | Message provides a human-readable description of the current state. | | Optional: \{\}
| +| `conditions` _[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#condition-v1-meta) array_ | Conditions holds standard Kubernetes condition types. | | Optional: \{\}
| #### StorageBackup @@ -1431,7 +1730,7 @@ filesystem: string | `fabric` _string_ | Fabric is the transport fabric (e.g. tcp). | tcp | | | `maxNamespacePerSubsys` _string_ | MaxNamespacePerSubsys limits namespaces per NVMf subsystem. | 1 | | | `tune2fsReservedBlocks` _string_ | Tune2fsReservedBlocks sets the ext4 reserved-blocks percentage. Left unset, the node
plugin skips tune2fs entirely and mkfs.ext4's own default reserve applies, matching a
StorageClass that omits tune2fs_reserved_blocks. A default of "0" here would not be a
no-op: it actively runs `tune2fs -m 0` on every volume, since the node plugin only skips
the call when the parameter is empty (see stageVolume in the CSI driver), not when it's
"0". | | | -| `filesystem` _string_ | Filesystem is the filesystem used to format logical volumes of this pool. | ext4 | Enum: [ext4 xfs]
| +| `filesystem` _string_ | Filesystem is the filesystem used to format logical volumes of this pool. | xfs | Enum: [ext4 xfs]
| #### StorageCluster @@ -1456,27 +1755,21 @@ spec: stripe: dataChunks: integer parityChunks: integer - haType: string - isSingleNode: boolean - strictNodeAntiAffinity: boolean - qpairCount: integer - blockSize: integer - pageSizeInBlocks: integer - maxQueueSize: integer - inflightIOThreshold: integer fabricType: string clientDataIfname: string - maxFaultTolerance: integer nvmfBasePort: integer rpcBasePort: integer snodeApiPort: integer + maxConcurrentWorkerRestarts: integer + maxSubsystemCount: integer + maxHugePagesSize: string + vcpuCount: integer warningThreshold: capacity: integer provisionedCapacity: integer criticalThreshold: capacity: integer provisionedCapacity: integer - clientQpairCount: integer backup: localEndpoint: '^https?://[a-zA-Z0-9.-]+(:[0-9]{1,5})?(/.*)?$' snapshotBackups: boolean @@ -1493,6 +1786,7 @@ spec: dataRealignment: enabled: boolean interval: Duration + minMoves: integer volumeAutoPlacement: enabled: boolean migrationEnabled: boolean @@ -1519,13 +1813,15 @@ status: nqn: string status: string rebalancing: boolean - pendingDataRealignment: boolean + volumeMoveGeneration: integer + realignedGeneration: integer lastDataRealignmentAt: Time erasureCodingScheme: string lastUpdated: Time created: Time configured: boolean maxFaultTolerance: integer + maxConcurrentWorkerRestarts: integer activeOpsRef: string rebalancingMetrics: avgDeviationPct: float @@ -1703,27 +1999,21 @@ enableNodeAffinity: boolean stripe: dataChunks: integer parityChunks: integer -haType: string -isSingleNode: boolean -strictNodeAntiAffinity: boolean -qpairCount: integer -blockSize: integer -pageSizeInBlocks: integer -maxQueueSize: integer -inflightIOThreshold: integer fabricType: string clientDataIfname: string -maxFaultTolerance: integer nvmfBasePort: integer rpcBasePort: integer snodeApiPort: integer +maxConcurrentWorkerRestarts: integer +maxSubsystemCount: integer +maxHugePagesSize: string +vcpuCount: integer warningThreshold: capacity: integer provisionedCapacity: integer criticalThreshold: capacity: integer provisionedCapacity: integer -clientQpairCount: integer backup: localEndpoint: '^https?://[a-zA-Z0-9.-]+(:[0-9]{1,5})?(/.*)?$' snapshotBackups: boolean @@ -1740,6 +2030,7 @@ volumeMigrationSettings: dataRealignment: enabled: boolean interval: Duration + minMoves: integer volumeAutoPlacement: enabled: boolean migrationEnabled: boolean @@ -1762,23 +2053,17 @@ enableFailureDomains: boolean | --- | --- | --- | --- | | `enableNodeAffinity` _boolean_ | EnableNodeAffinity enables node-affinity placement for storage components. | | | | `stripe` _[StripeSpec](#stripespec)_ | StripeSpec configures erasure-coding data/parity chunk counts. | | | -| `haType` _string_ | HAType defines the backend high-availability mode. | | | -| `isSingleNode` _boolean_ | IsSingleNode enables single-node cluster mode. | | | -| `strictNodeAntiAffinity` _boolean_ | StrictNodeAntiAffinity enforces strict anti-affinity between storage nodes. | | | -| `qpairCount` _integer_ | QpairCount defines the NVMe queue-pair count used by the cluster. | | | -| `blockSize` _integer_ | BlockSize defines the logical block size in bytes. | | | -| `pageSizeInBlocks` _integer_ | PageSizeInBlocks defines page size expressed in blocks. | | | -| `maxQueueSize` _integer_ | MaxQueueSize defines the maximum backend queue size. | | | -| `inflightIOThreshold` _integer_ | InflightIOThreshold defines the inflight I/O threshold. | | | | `fabricType` _string_ | FabricType defines the storage fabric type. | | | | `clientDataIfname` _string_ | ClientDataIfname defines the client data network interface. | | | -| `maxFaultTolerance` _integer_ | MaxFaultTolerance defines the maximum tolerated concurrent faults. | | | | `nvmfBasePort` _integer_ | NvmfBasePort defines the base NVMf service port. | | | | `rpcBasePort` _integer_ | RpcBasePort defines the base RPC service port. | | | | `snodeApiPort` _integer_ | SnodeApiPort defines the storage-node API port. | | | +| `maxConcurrentWorkerRestarts` _integer_ | MaxConcurrentWorkerRestarts is the maximum number of Kubernetes worker nodes the operator
may drain and restart simultaneously. The effective concurrency applied by the drain
coordinator is min(MaxConcurrentWorkerRestarts, MaxFaultTolerance).
Defaults to 1 when unset. | | Minimum: 1
Optional: \{\}
| +| `maxSubsystemCount` _integer_ | MaxSubsystemCount is the maximum number of NVMe-oF subsystems per storage
node. Applies to every storage node in the cluster. Required: it sizes huge
pages, and a node that receives no value fails config generation outright
rather than falling back to a default. | | Maximum: 75
Minimum: 10
Required: \{\}
| +| `maxHugePagesSize` _string_ | MaxHugePagesSize is the maximum allocatable size of huge pages on each
storage node (e.g. "100G", "1T"; a bare number is interpreted as GB). It is
a floor, not a cap: the effective huge-page allocation is the larger of this
value and the minimum the node's device and subsystem count requires. When
omitted the computed minimum is used. | | Optional: \{\}
| +| `vcpuCount` _integer_ | VCPUCount is the number of vCPUs allocated to SPDK on each storage node.
This is an explicit core count, not a percentage. Required: the core layout
it produces must match across the cluster, so it is stated rather than left
to a per-node heuristic. | | Minimum: 8
Required: \{\}
| | `warningThreshold` _[CapacityThresholdSpec](#capacitythresholdspec)_ | WarningThresholdSpec defines warning-level capacity thresholds. | | | | `criticalThreshold` _[CapacityThresholdSpec](#capacitythresholdspec)_ | CriticalThresholdSpec defines critical-level capacity thresholds. | | | -| `clientQpairCount` _integer_ | ClientQpairCount defines client-side queue-pair count. | | | | `backup` _[BackupSpec](#backupspec)_ | Backup specifies the specification for backup to S3 configuration | | | | `hashicorpVaultSettings` _[HashicorpVaultSettings](#hashicorpvaultsettings)_ | HashicorpVaultSettings configures the Vault endpoint used by the cluster for key storage. | | | | `volumeMigrationSettings` _[VolumeMigrationSettings](#volumemigrationsettings)_ | VolumeMigrationSettings controls volume migration for this cluster. | | Optional: \{\}
| @@ -1809,13 +2094,15 @@ storageNodes: integer nqn: string status: string rebalancing: boolean -pendingDataRealignment: boolean +volumeMoveGeneration: integer +realignedGeneration: integer lastDataRealignmentAt: Time erasureCodingScheme: string lastUpdated: Time created: Time configured: boolean maxFaultTolerance: integer +maxConcurrentWorkerRestarts: integer activeOpsRef: string rebalancingMetrics: avgDeviationPct: float @@ -1843,13 +2130,15 @@ rebalancingMetrics: | `nqn` _string_ | NQN is the cluster NVM subsystem qualified name. | | | | `status` _string_ | Status is the backend-reported lifecycle status. | | | | `rebalancing` _boolean_ | Rebalancing indicates whether cluster rebalancing is currently active. | | | -| `pendingDataRealignment` _boolean_ | PendingDataRealignment indicates that at least one volume has been moved since
the last successful control-plane data realignment, so a realignment is due on
the next DataRealignment.Interval tick. It is persisted so a pending realignment
survives an operator restart, and is cleared once a realignment completes
successfully. | | Optional: \{\}
| +| `volumeMoveGeneration` _integer_ | VolumeMoveGeneration counts completed volume moves. Every migration that reaches
Completed increments it, and nothing else writes it, so it only ever grows. | | Optional: \{\}
| +| `realignedGeneration` _integer_ | RealignedGeneration is the VolumeMoveGeneration that the last successfully
requested realignment covers. A realignment is outstanding while
VolumeMoveGeneration exceeds it.
This is recorded from the value read *before* the request is sent, because that
is what the realignment can actually account for: a migration completing while
the request is in flight raises VolumeMoveGeneration past it and so correctly
leaves another realignment outstanding, instead of being swallowed by the one
already running. | | Optional: \{\}
| | `lastDataRealignmentAt` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#time-v1-meta)_ | LastDataRealignmentAt is the time of the last successful control-plane data
realignment. It is used to space realignments by DataRealignment.Interval and to
avoid re-running at the end of an interval when nothing is pending. | | Optional: \{\}
| | `erasureCodingScheme` _string_ | ErasureCodingScheme is the active erasure-coding layout, for example "2x1". | | | | `lastUpdated` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#time-v1-meta)_ | LastUpdated is the last backend update timestamp.
FIXME: Unused for now (API update required?) | | | | `created` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#time-v1-meta)_ | Created is the backend creation timestamp.
FIXME: Unused for now (API update required?) | | | | `configured` _boolean_ | Configured indicates whether initial cluster setup completed. | | | | `maxFaultTolerance` _integer_ | MaxFaultTolerance is the backend-reported maximum number of nodes that can
be simultaneously offline (failed, drained, or restarted) without violating
the cluster's redundancy guarantees. | | | +| `maxConcurrentWorkerRestarts` _integer_ | MaxConcurrentWorkerRestarts is the effective concurrent-restart limit applied
by the drain coordinator: min(spec.MaxConcurrentWorkerRestarts, MaxFaultTolerance).
Defaults to 1. Exposed here so controllers and tooling can read a single
authoritative value without re-computing it. | | Optional: \{\}
| | `activeOpsRef` _string_ | ActiveOpsRef is the name of the currently active ClusterOps on this cluster.
Empty when no operation is in progress. | | Optional: \{\}
| | `rebalancingMetrics` _[RebalancingMetrics](#rebalancingmetrics)_ | RebalancingMetrics is updated by the auto-rebalancer each evaluation cycle. | | Optional: \{\}
| @@ -1880,11 +2169,8 @@ spec: nodeIndex: integer socketIndex: integer overrides: - maxSubsystemCount: integer - maxSize: string spdkImage: string spdkProxyImage: string - corePercentage: integer spdkSystemMemory: '^[0-9]+(G|GI|GB|GiB|M|MI|MB|MiB|g|gi|gb|gib|m|mi|mb|mib)?$' journalManager: count: integer @@ -2129,11 +2415,8 @@ _Appears in:_ _Example:_ ```yaml -maxSubsystemCount: integer -maxSize: string spdkImage: string spdkProxyImage: string -corePercentage: integer spdkSystemMemory: '^[0-9]+(G|GI|GB|GiB|M|MI|MB|MiB|g|gi|gb|gib|m|mi|mb|mib)?$' journalManager: count: integer @@ -2156,11 +2439,8 @@ expand: boolean | Field | Description | Default | Validation | | --- | --- | --- | --- | -| `maxSubsystemCount` _integer_ | MaxSubsystemCount overrides the maximum number of NVMe-oF subsystems for this node. | | Optional: \{\}
| -| `maxSize` _string_ | MaxSize overrides the maximum allocatable size of huge pages for this node. | | Optional: \{\}
| | `spdkImage` _string_ | SpdkImage overrides the SPDK image for this node (e.g. for phased rollouts). | | Optional: \{\}
| | `spdkProxyImage` _string_ | SpdkProxyImage overrides the SPDK proxy image for this node. | | Optional: \{\}
| -| `corePercentage` _integer_ | CorePercentage overrides the percentage of cores allocated to SPDK for this node (0-99). | | Optional: \{\}
| | `spdkSystemMemory` _string_ | SpdkSystemMemory overrides the SPDK huge-page memory allocation for this node
(e.g. "4G", "512M"). | | Pattern: `^[0-9]+(G\|GI\|GB\|GiB\|M\|MI\|MB\|MiB\|g\|gi\|gb\|gib\|m\|mi\|mb\|mib)?$`
Optional: \{\}
| | `journalManager` _[JournalManagerSpec](#journalmanagerspec)_ | JournalManagerSpec overrides journal manager tuning for this node. | | Optional: \{\}
| | `pcieAllowList` _string array_ | PcieAllowList overrides the list of PCI addresses allowed for use on this node. | | Optional: \{\}
| @@ -2252,16 +2532,13 @@ metadata: spec: clusterName: string clusterImage: '^($|(quay\.io/simplyblock-io|docker\.io/simplyblock|public\.ecr\.aws/simply-block)/[a-z0-9][a-z0-9._-]*:[a-zA-Z0-9][a-zA-Z0-9._-]*(@sha256:[a-f0-9]{64})?)$' - maxSubsystemCount: integer - maxSize: string spdkImage: '^($|(quay\.io/simplyblock-io|docker\.io/simplyblock|public\.ecr\.aws/simply-block)/[a-z0-9][a-z0-9._-]*:[a-zA-Z0-9][a-zA-Z0-9._-]*(@sha256:[a-f0-9]{64})?)$' spdkProxyImage: '^($|(quay\.io/simplyblock-io|docker\.io/simplyblock|public\.ecr\.aws/simply-block)/[a-z0-9][a-z0-9._-]*:[a-zA-Z0-9][a-zA-Z0-9._-]*(@sha256:[a-f0-9]{64})?)$' mgmtIfname: string - partitions: integer + enableJournalDevice: boolean journalManager: count: integer percentPerDevice: integer - corePercentage: integer pcieAllowList: - string pcieDenyList: @@ -2296,11 +2573,8 @@ spec: expand: boolean nodeConfigs: string: - maxSubsystemCount: integer - maxSize: string spdkImage: string spdkProxyImage: string - corePercentage: integer spdkSystemMemory: '^[0-9]+(G|GI|GB|GiB|M|MI|MB|MiB|g|gi|gb|gib|m|mi|mb|mib)?$' journalManager: count: integer @@ -2384,16 +2658,13 @@ _Example:_ ```yaml clusterName: string clusterImage: '^($|(quay\.io/simplyblock-io|docker\.io/simplyblock|public\.ecr\.aws/simply-block)/[a-z0-9][a-z0-9._-]*:[a-zA-Z0-9][a-zA-Z0-9._-]*(@sha256:[a-f0-9]{64})?)$' -maxSubsystemCount: integer -maxSize: string spdkImage: '^($|(quay\.io/simplyblock-io|docker\.io/simplyblock|public\.ecr\.aws/simply-block)/[a-z0-9][a-z0-9._-]*:[a-zA-Z0-9][a-zA-Z0-9._-]*(@sha256:[a-f0-9]{64})?)$' spdkProxyImage: '^($|(quay\.io/simplyblock-io|docker\.io/simplyblock|public\.ecr\.aws/simply-block)/[a-z0-9][a-z0-9._-]*:[a-zA-Z0-9][a-zA-Z0-9._-]*(@sha256:[a-f0-9]{64})?)$' mgmtIfname: string -partitions: integer +enableJournalDevice: boolean journalManager: count: integer percentPerDevice: integer -corePercentage: integer pcieAllowList: - string pcieDenyList: @@ -2428,11 +2699,8 @@ nodeFailureDomains: expand: boolean nodeConfigs: string: - maxSubsystemCount: integer - maxSize: string spdkImage: string spdkProxyImage: string - corePercentage: integer spdkSystemMemory: '^[0-9]+(G|GI|GB|GiB|M|MI|MB|MiB|g|gi|gb|gib|m|mi|mb|mib)?$' journalManager: count: integer @@ -2457,14 +2725,11 @@ nodeConfigs: | --- | --- | --- | --- | | `clusterName` _string_ | ClusterName is the target storage cluster name. | | | | `clusterImage` _string_ | ClusterImage is the container image used for storage-node workloads.
Must reference one of the trusted registries (quay.io/simplyblock-io, docker.io/simplyblock, public.ecr.aws/simply-block); digest pinning (@sha256:...) is recommended. | | Pattern: `^($\|(quay\.io/simplyblock-io\|docker\.io/simplyblock\|public\.ecr\.aws/simply-block)/[a-z0-9][a-z0-9._-]*:[a-zA-Z0-9][a-zA-Z0-9._-]*(@sha256:[a-f0-9]\{64\})?)$`
| -| `maxSubsystemCount` _integer_ | MaxSubsystemCount is the maximum number of NVMe-oF subsystems per node. | | | -| `maxSize` _string_ | MaxSize is the maximum allocatable size of huge pages. | | | | `spdkImage` _string_ | SpdkImage is the SPDK image reference used by node services.
Must reference one of the trusted registries (quay.io/simplyblock-io, docker.io/simplyblock, public.ecr.aws/simply-block); digest pinning (@sha256:...) is recommended. | | Pattern: `^($\|(quay\.io/simplyblock-io\|docker\.io/simplyblock\|public\.ecr\.aws/simply-block)/[a-z0-9][a-z0-9._-]*:[a-zA-Z0-9][a-zA-Z0-9._-]*(@sha256:[a-f0-9]\{64\})?)$`
| | `spdkProxyImage` _string_ | SpdkProxyImage is the SPDK proxy image reference used by node services.
Must reference one of the trusted registries (quay.io/simplyblock-io, docker.io/simplyblock, public.ecr.aws/simply-block); digest pinning (@sha256:...) is recommended. | | Pattern: `^($\|(quay\.io/simplyblock-io\|docker\.io/simplyblock\|public\.ecr\.aws/simply-block)/[a-z0-9][a-z0-9._-]*:[a-zA-Z0-9][a-zA-Z0-9._-]*(@sha256:[a-f0-9]\{64\})?)$`
| | `mgmtIfname` _string_ | MgmtIfname is the management interface name used by storage nodes. | | | -| `partitions` _integer_ | Partitions is the number of partitions created per backend storage device. | | | +| `enableJournalDevice` _boolean_ | EnableJournalDevice dedicates a whole NVMe device to the journal manager
instead of carving a journal partition out of every storage device. When
true the smallest device on the node becomes the journal device, and the
remaining devices are used whole; when false (the default) each device is
GPT-partitioned into a journal slice plus a storage slice. | | | | `journalManager` _[JournalManagerSpec](#journalmanagerspec)_ | JournalManagerSpec configures journal manager behavior. | | | -| `corePercentage` _integer_ | CorePercentage is the percentage of cores to be used for spdk (0-99). | | | | `pcieAllowList` _string array_ | PcieAllowList is the list of PCI addresses allowed for use. | | | | `pcieDenyList` _string array_ | PcieDenyList is the list of PCI addresses excluded from use. | | | | `pcieModel` _string_ | PcieModel filters devices by PCI model. | | | @@ -2580,11 +2845,8 @@ socketId: string nodeIndex: integer socketIndex: integer overrides: - maxSubsystemCount: integer - maxSize: string spdkImage: string spdkProxyImage: string - corePercentage: integer spdkSystemMemory: '^[0-9]+(G|GI|GB|GiB|M|MI|MB|MiB|g|gi|gb|gib|m|mi|mb|mib)?$' journalManager: count: integer @@ -3094,6 +3356,34 @@ tasks: | `tasks` _[TaskEntry](#taskentry) array_ | Tasks is the currently reported task list for the query scope. | | | +#### ValidationJob + + + +ValidationJob is one NVMe path-validation Job and the worker node it runs on. +The node is a consumer of some volume in the migrated subsystem — the volume named +in the spec, or one of its siblings sharing the same NVMe subsystem. + + + +_Appears in:_ +- [VolumeMigrationStatus](#volumemigrationstatus) + +_Example:_ + +```yaml +node: string +jobName: string +succeeded: boolean +``` + +| Field | Description | Default | Validation | +| --- | --- | --- | --- | +| `node` _string_ | Node is the Kubernetes node name the Job is pinned to. | | | +| `jobName` _string_ | JobName is the name of the Job object in the VolumeMigration's namespace. | | | +| `succeeded` _boolean_ | Succeeded records that this node's validation passed. It is kept because the
Job's own existence is not a reliable record: Jobs are reaped, and re-reading a
reaped Job would otherwise look like "never validated" and start it again. | | Optional: \{\}
| + + #### VolumeAutoPlacementSettings @@ -3174,9 +3464,9 @@ status: clusterUUID: string volumeUUID: string poolUUID: string + subsystemNQN: string sourceNodeUUID: string - snapsTotal: integer - snapsMigrated: integer + memberCount: integer errorMessage: string connections: - nqn: string @@ -3188,7 +3478,11 @@ status: ctrlLossTmo: integer fastIOFailTmo: integer keepAliveTmo: integer - validationJobName: string + validationJobs: + - node: string + jobName: string + succeeded: boolean + deferredSince: Time startedAt: Time completedAt: Time ``` @@ -3245,6 +3539,7 @@ rebalancerImage: string dataRealignment: enabled: boolean interval: Duration + minMoves: integer ``` | Field | Description | Default | Validation | @@ -3299,9 +3594,9 @@ migrationUUID: string clusterUUID: string volumeUUID: string poolUUID: string +subsystemNQN: string sourceNodeUUID: string -snapsTotal: integer -snapsMigrated: integer +memberCount: integer errorMessage: string connections: - nqn: string @@ -3313,7 +3608,11 @@ connections: ctrlLossTmo: integer fastIOFailTmo: integer keepAliveTmo: integer -validationJobName: string +validationJobs: + - node: string + jobName: string + succeeded: boolean +deferredSince: Time startedAt: Time completedAt: Time ``` @@ -3325,47 +3624,14 @@ completedAt: Time | `clusterUUID` _string_ | ClusterUUID is the storage cluster UUID resolved from the PV. | | | | `volumeUUID` _string_ | VolumeUUID is the logical volume UUID resolved from the PV's CSI volume handle. | | | | `poolUUID` _string_ | PoolUUID is the storage pool UUID that contains the volume. | | | +| `subsystemNQN` _string_ | SubsystemNQN is the NQN of the volume's NVMe subsystem, resolved from the
storage API when the migration is submitted. The migration is addressed by
it, and every volume sharing the subsystem moves with it. | | | | `sourceNodeUUID` _string_ | SourceNodeUUID is the storage node UUID where the volume resided before
migration, as reported by the storage API. | | | -| `snapsTotal` _integer_ | SnapsTotal is the total number of snapshots to migrate, as reported by the API. | | | -| `snapsMigrated` _integer_ | SnapsMigrated is the number of snapshots migrated so far. | | | +| `memberCount` _integer_ | MemberCount is the number of volumes (namespaces) in the migrated
subsystem, as reported by the storage API. More than one means the
migration moves sibling volumes along with this one. | | | | `errorMessage` _string_ | ErrorMessage holds the failure reason when Phase is Failed. | | | -| `connections` _[MigrationConnection](#migrationconnection) array_ | Connections holds the NVMe-oF connection parameters for the new target-side
paths returned by CreateMigration. Used during the Validating phase to
establish and verify the paths before calling ContinueMigration. | | | -| `validationJobName` _string_ | ValidationJobName is the name of the Job that runs `nvme connect` for each
connection path and validates ANA state before ContinueMigration is called.
Set during the Validating phase; cleared when the phase advances to Running. | | | +| `connections` _[MigrationConnection](#migrationconnection) array_ | Connections holds the NVMe-oF connection parameters for the new target-side
paths returned by CreateMigration. Used during the Validating phase to
establish and verify the paths before calling ContinueMigration, and again to
release them if the migration never cuts over.
These are the parameters the paths are actually connected with, not verbatim
what CreateMigration answered: ctrlLossTmo is replaced with the value every
path in this system uses, because a target path becomes the volume's data path
at cutover. The rest is passed through. | | | +| `validationJobs` _[ValidationJob](#validationjob) array_ | ValidationJobs are the Jobs that run `nvme connect` for each connection path
and validate ANA state before ContinueMigration is called — one per worker
node that consumes a volume of the migrated subsystem. A subsystem migrates
as a unit, so every consuming node must have the new paths before cutover;
all of these Jobs must succeed. Set during the Validating phase; cleared when
the phase advances to Running. | | | +| `deferredSince` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#time-v1-meta)_ | DeferredSince is when the storage API first refused to accept this migration
because the cluster was busy with work that ends on its own (a data realignment
or another node migration). While set, the migration is being retried and has
not started. It bounds the retrying: past a fixed window the migration fails
rather than waiting forever. Cleared once the migration is submitted. | | | | `startedAt` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#time-v1-meta)_ | StartedAt is the time the migration was submitted to the storage API. | | | | `completedAt` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#time-v1-meta)_ | CompletedAt is the time the migration finished (successfully or not). | | | -#### VolumeReplicationStatus - - - -VolumeReplicationStatus tracks the replication state of an individual volume - - - -_Appears in:_ -- [SnapshotReplicationStatus](#snapshotreplicationstatus) - -_Example:_ - -```yaml -volumeID: string -phase: string -lastSnapshotID: string -lastReplicationTime: Time -replicatedCount: integer -errors: - - timestamp: Time - message: string -``` - -| Field | Description | Default | Validation | -| --- | --- | --- | --- | -| `volumeID` _string_ | Volume ID | | | -| `phase` _string_ | Phase is the current replication phase for this volume. | | Enum: [Pending Running TriggeringTargetReplication WaitingForTargetReplication ReplicatingToSource WaitingForTargetDeletion Completed Failed Paused]
| -| `lastSnapshotID` _string_ | Last snapshot ID replicated for this volume | | | -| `lastReplicationTime` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.36/#time-v1-meta)_ | Timestamp of the last successful replication for this volume | | | -| `replicatedCount` _integer_ | Number of snapshots successfully replicated | | | -| `errors` _[ReplicationError](#replicationerror) array_ | Optional: list of errors encountered for this volume | | | - -