Moved volume migration and rebalancing to the new statemachine system

[noctarius]

This PR moves the previous state implementation of volume migration to the more stable statemachine implementation which screams loud when failiing.

[Copilot]

Pull request overview

This PR migrates volume migration and rebalancing workflows to the shared state-machine implementation, adding persisted phase deadlines, cleanup handling, and generation-based realignment tracking.

Changes:

• Adds the VolumeMigration lifecycle state machine and deadline persistence.
• Updates rebalancing cycles, metrics, realignment accounting, and controller tests.
• Updates APIs, CRDs, generated code, helpers, and state-machine utilities.

Reviewed changes

Copilot reviewed 46 out of 48 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
operator/internal/volumemigration/volumemoved_test.go	Updated as part of this pull request.
operator/internal/volumemigration/statemachine.go	Updated as part of this pull request.
operator/internal/volumemigration/statemachine_test.go	Updated as part of this pull request.
operator/internal/volumemigration/controller_test.go	Updated as part of this pull request.
operator/internal/volumemigration/controller_paths_test.go	Updated as part of this pull request.
operator/internal/volumemigration/controller_jobs_test.go	Updated as part of this pull request.
operator/internal/utils/names.go	Updated as part of this pull request.
operator/internal/utils/names_test.go	Updated as part of this pull request.
operator/internal/ctrltest/ctrltest.go	Updated as part of this pull request.
operator/internal/controller/volumerebalancer_metrics.go	Updated as part of this pull request.
operator/internal/controller/volumerebalancer_controller_test.go	Updated as part of this pull request.
operator/internal/controller/test_helpers_test.go	Updated as part of this pull request.
operator/internal/controller/storagenodeops_controller_unit_test.go	Updated as part of this pull request.
operator/internal/controller/storagenode_latency_controller.go	Updated as part of this pull request.
operator/internal/controller/storagenode_controller_unit_test.go	Updated as part of this pull request.
operator/internal/controller/storageclusterops_controller_unit_test.go	Updated as part of this pull request.
operator/internal/controller/storagebackupsync_controller_unit_test.go	Updated as part of this pull request.
operator/internal/controller/simplyblocktask_controller_unit_test.go	Updated as part of this pull request.
operator/internal/controller/simplyblockstoragepool_controller_unit_test.go	Updated as part of this pull request.
operator/internal/controller/simplyblockstoragenodeset_storagenode_unit_test.go	Updated as part of this pull request.
operator/internal/controller/simplyblockstoragenodeset_drain_unit_test.go	Updated as part of this pull request.
operator/internal/controller/simplyblockstoragenodeset_controller_unit_test.go	Updated as part of this pull request.
operator/internal/controller/simplyblockstoragecluster_controller_unit_test.go	Updated as part of this pull request.
operator/internal/controller/replicationslot_controller_unit_test.go	Updated as part of this pull request.
operator/internal/controller/replicationpolicy_controller_unit_test.go	Updated as part of this pull request.
operator/internal/controller/replicationpair_controller_unit_test.go	Updated as part of this pull request.
operator/internal/controller/replicationops_controller_unit_test.go	Updated as part of this pull request.
operator/internal/controller/pvcreplication_controller_unit_test.go	Updated as part of this pull request.
operator/internal/controller/persistentvolumeclaim_controller_unit_test.go	Updated as part of this pull request.
operator/internal/controller/nodedrain_controller_unit_test.go	Updated as part of this pull request.
operator/internal/controller/backuprestore_controller_test.go	Updated as part of this pull request.
operator/internal/controller/backuppolicy_controller_unit_test.go	Updated as part of this pull request.
operator/internal/autoplacement/metrics.go	Updated as part of this pull request.
operator/internal/autoplacement/cyclestatemachine.go	Updated as part of this pull request.
operator/internal/autoplacement/cyclestatemachine_test.go	Updated as part of this pull request.
operator/internal/autoplacement/controller.go	Updated as part of this pull request.
operator/internal/autoplacement/controller_realignment_test.go	Updated as part of this pull request.
operator/config/crd/bases/storage.simplyblock.io_volumemigrations.yaml	Updated as part of this pull request.
operator/cmd/main.go	Updated as part of this pull request.
operator/api/v1alpha1/zz_generated.deepcopy.go	Updated as part of this pull request.
operator/api/v1alpha1/volumemigration_types.go	Updated as part of this pull request.
helm-charts/charts/simplyblock-operator/crds/storage.simplyblock.io_volumemigrations.yaml	Updated as part of this pull request.
atlas-lib/statemachine/statemachine.go	Updated as part of this pull request.
atlas-lib/statemachine/statemachine_test.go	Updated as part of this pull request.
atlas-lib/statemachine/smtest/smtest.go	Updated as part of this pull request.
atlas-lib/statemachine/smtest/smtest_test.go	Updated as part of this pull request.

Files not reviewed (1)

• operator/api/v1alpha1/zz_generated.deepcopy.go: Generated file

Suppressed comments (11)

operator/internal/autoplacement/controller.go:255

• This return ignores realignRequeue, even though paced was calculated specifically to wake the controller at the earlier data-realignment deadline. With an evaluation interval longer than the realignment interval, a successful migration cycle can postpone the next realignment check until the full evaluation interval. Return nextRequeue(cycleStart, interval, realignRequeue) here (recomputed after the work).
  operator/internal/autoplacement/controller.go:204
• The pending-migration gate is backed only by in-memory MigrationState. After an operator restart, existing Validating/Running VolumeMigration CRs are not rehydrated, so this check passes and the rebalancer can create more migrations while those backend migrations are still active; the intended cluster-level serialization is lost. Rebuild pending state from non-terminal CRs or make this gate query them directly.
  operator/internal/autoplacement/cyclestatemachine.go:192
• The cooldown gauge is refreshed only in onCycleCompleted. Deferred, DryRun, and Failed cycles can be the only cycles after a volume's expiry, leaving the gauge at a stale nonzero value even though GetCooldownCountByCluster would return zero. Refresh the gauge in this common terminal hook whenever clusterUUID is known.

		rebalancerEvaluationTotal.WithLabelValues(c.cluster.Name, outcome).Inc()

operator/internal/volumemigration/controller.go:188

• This path handles every Pending reconcile but never checks the persisted phase deadline. If the operator is down past a deferred migration's deadline, it will still resolve the volume and call CreateMigration, then re-enter Validating and reset the timeout window. Check sm.TimeoutReached() before any backend side effects and fail the migration without submitting it.

This issue also appears on line 232 of the same file.
operator/internal/volumemigration/controller.go:714

• When the initial node lookup finds no consumers, this early return prevents the pre-cutover recheck from seeing a consumer that starts before TransitionTo(Running). That node can therefore miss path validation even though the migration is subsystem-wide. The late-node resolution should also run when ValidationJobs is empty.
  operator/internal/volumemigration/controller.go:238
• When every CreateMigration attempt is indeterminate, retryIndeterminateCreate eventually calls holdInPending → fail. At that point the migration UUID/NQN from the response were never recorded, cancelBackend is false, and onFailed performs no backend cleanup; a POST that actually succeeded can therefore leave an orphaned backend migration after the CR is Failed. On deadline, query/cancel the subsystem migration or persist enough identity to clean it up before terminalizing.
  operator/internal/volumemigration/controller.go:782
• The phase deadline is checked before polling, but job polling and validateLateNodes can run past it; TransitionTo does not reject an expired deadline. This line can therefore call ContinueMigration after the validation bound has elapsed. Re-check the deadline immediately before cutover and bound the transition hook's context to the remaining phase time.
  operator/internal/volumemigration/controller.go:164
• Abort is handled before pollMigration, so a backend migration that has already reached a terminal status while the CR still says Running is sent to CancelMigration. The API treats cancellation as an in-progress operation, so this can be rejected and retried forever instead of recording the already-known Completed/Failed result. Read and classify terminal backend status before attempting cancellation.
  operator/internal/volumemigration/controller.go:504
• The successful Jobs are added to Status.ValidationJobs only after the creation loop finishes. If a later Job creation fails, earlier Jobs already exist but remain untracked; timeout, abort, or failure cleanup then iterates an incomplete status list and can leave their paths connected. Record each successful Job before continuing, or make cleanup discover all Jobs owned by this migration.
  operator/internal/volumemigration/statemachine.go:353
• A failed CancelMigration is logged but the sentinel error is returned anyway. cutover interprets errMigrationStartFailed as a proven terminal start failure, sets releasePaths, and persists Failed; because cancelBackend remains false, no later reconcile retries the failed cancellation, leaving backend objects behind. Return a retryable error when cancellation does not succeed.
  operator/internal/volumemigration/statemachine.go:408
• A failed or cancelled backend response can have an empty error_message. This assignment then discards any local failure reason and persists an empty status.errorMessage, leaving users and the rebalancer with no explanation for a terminal failure. Only override p.failure when the backend message is non-empty, and provide a fallback when both are empty.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.