fix(storagenodeset): stop giving the shared storage-node ServiceAccount an ownerReference

[boddumanohar]

Summary

• If the simplyblock-storage-node-sa ServiceAccount is ever deleted and recreated, every already-running pod's mounted token stays bound to the old, now-nonexistent ServiceAccount UID. The API server correctly rejects those with 401 Unauthorized, and nothing self-heals until kubelet's next token refresh (default ~48min) — in practice a long, manual-intervention outage (had to force-delete pods to get fresh tokens).
• Root cause: reconcileRBAC gave the SA a controller: true ownerReference to a single StorageNodeSet. The SA is shared by every StorageNodeSet in the namespace, not owned 1:1 by any one of them, so whichever one reconciled last became the sole owner GC cares about. Deleting that StorageNodeSet — including as an ordinary part of cluster expansion, adding one and then removing it — let kube-controller-manager's garbage collector cascade-delete the SA out from under every other StorageNodeSet's already-running pods.
• Fix: drop the ownerReference entirely and mark the SA (and, for consistency, the ClusterRole/ClusterRoleBinding, which never carried one) with a plain simplyblock.io/managed-by label instead. Labels carry no GC semantics, so the SA is now only ever removed by an explicit kubectl delete — never as fallout from any single StorageNodeSet's lifecycle.

Root cause (confirmed, reproduced)

1. reconcileRBAC builds a fresh, ownerless ServiceAccount struct on every reconcile and unconditionally overwrites its ownerReferences to point at whichever StorageNodeSet is currently reconciling. With more than one StorageNodeSet sharing a namespace, ownership of the single shared SA just flip-flops to whichever reconciled most recently — there is no coordination between them.
2. Deterministic trigger, no race required: add a second StorageNodeSet (e.g. for cluster expansion) — it reconciles almost immediately and becomes the SA's current owner — then delete it. Once its finalizer is removed and the object is actually gone, the garbage collector sees the SA's sole ownerReferences entry pointing at a UID that no longer exists and deletes the SA. It has no way to know any other StorageNodeSet still depends on it, since only the deleted one was ever recorded as owner.
3. Separately confirmed the underlying GC mechanism is close to instantaneous, live on a real cluster, with a disposable ConfigMap/ServiceAccount pair carrying the same ownerReference shape: recreating the parent object under the same name still got the orphaned child deleted by GC in well under a second — even patching the child's ownerReferences to the new UID in the very next command was already too late. ownerReferences match by UID, not name, so GC doesn't care that an object with the same name exists again.
4. Blast radius: the control-plane→storage-node HTTP channel (webappapi/tasks-runner calling the storage-node's Flask API) is TLS-authenticated and stays reachable — this SA/token is unrelated to it. What breaks is the storage-node pod's own outbound calls to the Kubernetes API server (simplyblock_web/api/internal/storage_node/kubernetes.py in sbcli, using the pod's own SA token): creating/deleting the SPDK device-prep Jobs, creating/deleting the spdk-proxy pod, and polling Node cordon/MachineConfigPool status. Once the SA token is invalid those calls return 401, so every request that tries to actually start, stop, or health-check a node's SPDK backend fails at that point — effectively freezing node lifecycle operations across the whole cluster, since every storage-node pod shares this one SA.
5. There is no reconcile fast enough to win this race reactively once triggered — the actual fix has to be structural: stop making the SA's lifecycle GC's responsibility at all. With no ownerReference pointing at any StorageNodeSet, GC has nothing to cascade-delete here, so neither deleting a StorageNodeSet nor delete-recreating one can ever take the SA down with it again, by construction rather than by winning a race.

What still isn't identified: whether anything besides deliberate cluster-expansion delete/recreate triggers this in practice (e.g. deploy tooling bouncing CRs during a CRD/schema migration). That's a separate investigation — this PR closes the hole regardless of what triggers it.

Test plan

• go build ./... passes
• go test ./... passes (full suite, including envtest-backed controller tests)
• golangci-lint run ./internal/utils/... ./internal/controller/... --new-from-rev=5aa711c (branch base) — 0 new issues
• Updated TestStorageNodeSetReconcileServiceAccountHasOwnerReference → TestStorageNodeSetReconcileServiceAccountIsLabeledNotOwned, asserting the SA carries no ownerReference and does carry the simplyblock.io/managed-by label
• Reproduced the underlying GC-orphan mechanism live on a real cluster with a synthetic ConfigMap/ServiceAccount pair mirroring reconcileRBAC's ownerReference shape — confirms the SA-recreation scenario this PR fixes is real and near-deterministic, not hypothetical
• Full live end-to-end verification against this exact change (add a second StorageNodeSet in the same namespace, delete it, confirm the shared SA survives and every other StorageNodeSet's running pods are unaffected) — not yet re-verified against this standalone PR

Deployment note

This change does not touch the DaemonSet's pod template — no rolling restart of storage-node pods is triggered by this fix.

🤖 Generated with Claude Code

https://claude.ai/code/session_015pjXzhB8JQ6ySSm1vvwZq7