docs: add Talos node upgrade and DRBD recovery notes

plugins/cozystack/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "cozystack",
-  "version": "1.18.0",
+  "version": "1.18.1",
   "description": "Cozystack platform skills bundle. Start with cozystack:wizard — it begins with a free-form 'tell me about your setup and goal' question, parses hints, then asks Talos / Ubuntu / Existing, builds a chain, and dispatches downstream skills via a cluster config directory the operator picks (every artifact lives there: inventory.yml, kubeconfig, .state.yaml, cozystack-platform-package.yaml — operator manages git on their own; optional sops opt-in encrypts secret files in-tree). Skills, invoked as cozystack:<name>: wizard (orchestrator + 3-route dispatcher + Phase 4.5 active research + auto-dispatches debug on any failed_at), talos-bootstrap (Talos node prep via talm — Talos-1.12-aware maintenance probe, NAT-provider cert-SAN guardrail before first talm apply, multidoc machine-config with per-node VIP-link IPv4 stubs, etcd bootstrap, kubeconfig fetch, cozystack-tuned shape verification with Phase 11.5 auto-upgrade), talos-reset (cloud-provider terminate+relaunch helper for OCI/AWS/GCP/Hetzner when nodes are unrecoverable from inside; preserves block volumes + secondary VNICs + NSG memberships), ubuntu-bootstrap (wraps cozystack/ansible-cozystack — OS prep + k3s install in one go), cluster-install (Cozystack on a ready cluster — node-readiness, ZFS pool provisioning via privileged DaemonSet on Talos with hostNetwork, extractedprism for kube-apiserver HA, OCI-tag-normalized cozy-installer chart, Platform Package, inline tenants/root ingress patch + LINSTOR pool registration during watch loop with combined HRs-Ready + pools-registered gate, Phase 8.6 default StorageClasses for v1.3.x, Phase 9.1 end-to-end reachability probe), debug (investigate a stuck or broken install — gathers symptoms, classifies operator error / config drift / upstream bug / not-yet-supported, applies fixes or workarounds, drafts upstream issues on approval; never opens PRs or files silently), cluster-upgrade (v1.x patch/minor upgrade with release-notes analysis), package-deploy (dev-loop deploy of a single package with ExternalArtifact support), package-bump (single-package version bump with changelog adaptation), external-app-create (scaffold a new external-apps package), dev-ui-bootstrap (bootstrap a UI dev sandbox — Playwright + Vite dev server for cozystack-ui pointed at a chosen kubeconfig; worktree, @playwright/test, Chromium, playwright.config.ts, dev:e2e / test:e2e scripts, kubectl proxy + Vite on a free port), comment-audit (audit a diff's code comments for design-doc prose — rationale, product motivation, before/after narrative, incident retelling — and report cuts, tightens, keeps and arguments duplicated across sites; reports only, modifies nothing), comment-trim (apply the cuts — delete the prose, tighten verbose-but-legitimate blocks, keep every comment that documents a real mechanism, then prove mechanically with the bundled comments-only.py that no code moved and a grep-based pointer-invariant check that no pointer sits on top of the prose it replaces; leaves the result in the working tree, does not commit or push). comment-audit and comment-trim are repository-agnostic and usable on any codebase. All skills match the operator's natural language detected from conversation context — code identifiers, commands, file paths, and GitHub-public text stay canonical. All skills follow the same gate-and-confirm discipline: read-only lookups run freely; any mutation needs explicit per-step approval.",
   "author": {
     "name": "Cozystack",

plugins/cozystack/skills/cluster-upgrade/SKILL.md

@@ -146,6 +146,12 @@ Show user: result (success/partial/failed), before→after version, HR/Package t
 
 **Details, caveats, when not to roll back:** read `references/rollback.md`.
 
+## Talos node upgrades
+
+Upgrading Talos or Kubernetes on the nodes is a separate rolling operation, not part of the `helm upgrade`. On DRBD-backed clusters the Talos version decides the DRBD version, and Talos 1.14 no longer autoloads `drbd_transport_tcp`.
+
+**Before upgrading any Talos node:** read `references/talos-node-upgrade.md` (version path, machine-config changes to apply first, per-node gotchas, gate between node reboots).
+
 ## Known failure modes
 
 High-blast-radius stuck states — stuck helm `uninstalling`, Kamaji datastore cert mismatch, `MissingRollbackTarget`, orphan HRs from removed apps, `cozy-system` accidentally deleted, etc.
@@ -162,6 +168,8 @@ High-blast-radius stuck states — stuck helm `uninstalling`, Kamaji datastore c
 | HR `UninstallFailed, failed to delete release` | Stuck helm history (known-failures #1) |
 | TCP `INSTALLED VERSION` diverges from `VERSION` | Kamaji upgrade stuck (known-failures #4) |
 | `cozy-system` namespace gone | Missing `helm.sh/resource-policy=keep` (known-failures #7); restore from backup |
+| `linstor-controller` CrashLoopBackOff, `LAYER_DRBD_VOLUMES could not be restored` | Interrupted resource deletion left orphan layer rows (known-failures #8) |
+| After a Talos 1.14 node upgrade: `Failure: (172) Failed to create transport` | `drbd_transport_tcp` not loaded (`references/talos-node-upgrade.md`) |
 
 ## Common mistakes
 
@@ -174,7 +182,7 @@ High-blast-radius stuck states — stuck helm `uninstalling`, Kamaji datastore c
 
 ## References
 
-- Skill files: `references/release-notes-analysis.md`, `references/preflight-checks.md`, `references/post-upgrade-checks.md`, `references/rollback.md`, `references/known-failures.md`
+- Skill files: `references/release-notes-analysis.md`, `references/preflight-checks.md`, `references/post-upgrade-checks.md`, `references/rollback.md`, `references/known-failures.md`, `references/talos-node-upgrade.md`
 - Upstream (pick the version matching your target from the docs site version selector): `https://cozystack.io/docs/<vX.Y>/operations/cluster/upgrade/`
 - Troubleshooting checklist: `https://cozystack.io/docs/<vX.Y>/operations/troubleshooting/`
 - Releases: `https://github.com/cozystack/cozystack/releases`

plugins/cozystack/skills/cluster-upgrade/references/known-failures.md

@@ -253,6 +253,49 @@ Restore from backup. There is no clean in-cluster recovery for a deleted `cozy-s
 3. Re-apply the Platform Package from rescue.yaml (manual review required; CRD schemas may have moved).
 4. Expect tenant disruption; communicate to users.
 
+## 8. LINSTOR controller crash-loops after an interrupted resource deletion
+
+### Symptom
+
+`linstor-controller` in `CrashLoopBackOff` (often after a node reboot or a controller restart during a node upgrade), with this on startup:
+
+```text
+Database entry of table LAYER_DRBD_VOLUMES could not be restored
+```
+
+The detailed error report is written inside the controller container and is lost on every restart, so `linstor error-reports list` cannot show it.
+
+### Root cause
+
+With the Kubernetes CRD database backend (the Cozystack default), LINSTOR stores each table row as a separate `*.internal.linstor.linbit.com` custom resource, and a multi-row change is not transactional. A resource deletion interrupted halfway (controller killed, node rebooted) can remove the `resources` row while leaving its layer rows behind. On the next start the controller cannot attach those layer rows to a resource and refuses to load the database.
+
+### Recovery
+
+```bash
+# 1. Back up every LINSTOR internal CR before touching anything
+for crd in $(kubectl --context $CTX get crd --output name | grep internal.linstor.linbit.com); do
+  kubectl --context $CTX get "${crd#*/}" --output yaml > "backup-${crd#*/}.yaml"
+done
+
+# 2. Find orphaned layer resources: rows in layerresourceids whose
+#    (node_name, resource_name, snapshot_name) has no matching row in resources
+kubectl --context $CTX get resources.internal.linstor.linbit.com --output json \
+  | jq -r '.items[].spec | "\(.node_name)/\(.resource_name)"' | sort -u > resources.txt
+kubectl --context $CTX get layerresourceids.internal.linstor.linbit.com --output json \
+  | jq -r '.items[] | select((.spec.snapshot_name // "") == "") | "\(.spec.node_name)/\(.spec.resource_name) \(.spec.layer_resource_id) \(.metadata.name)"' \
+  | awk 'NR==FNR { have[$1]=1; next } !($1 in have)' resources.txt -
+# Rows with a non-empty snapshot_name belong to snapshots; check those against snapshots.internal.linstor.linbit.com instead.
+
+# 3. For each orphaned layer_resource_id, collect the rows that reference it in
+#    layerdrbdresources, layerdrbdvolumes and layerstoragevolumes (field layer_resource_id)
+
+# 4. Delete exactly those CRs (orphaned layerresourceids + the collected layer rows), then restart the controller
+kubectl --context $CTX delete <kind>.internal.linstor.linbit.com <name>
+kubectl --context $CTX rollout restart --namespace cozy-linstor deployment/linstor-controller
+```
+
+Field names in the CR `spec` are the lower-cased database column names; confirm them on one object (`kubectl --context $CTX get layerresourceids.internal.linstor.linbit.com --output yaml | head -40`) before scripting. Every deletion is a stop gate: show the list and wait for approval. After the controller starts, run `linstor resource list --faulty` and let `linstor:recover` handle the replicas of the affected resource.
+
 ## Diagnostic quick reference
 
 | Question | Command |

plugins/cozystack/skills/cluster-upgrade/references/talos-node-upgrade.md

@@ -0,0 +1,116 @@
+# Talos Node Upgrades Under Cozystack
+
+Read this before upgrading Talos (or Kubernetes via machine config) on nodes of a running Cozystack cluster. The Cozystack `helm upgrade` in the main skill does not touch Talos; node upgrades are a separate, per-node rolling operation, and on DRBD-backed clusters most of the risk sits in the storage layer, not in Talos itself.
+
+## Pick the target version (DRBD first)
+
+The DRBD kernel module ships inside the `siderolabs/drbd` extension, so the Talos version decides the DRBD version. Check the extension before picking a Talos target:
+
+| Talos | DRBD in extension | Notes |
+| --- | --- | --- |
+| 1.12.x | 9.2.16 | Bitmap race, see `linstor:recover` known upstream bugs |
+| 1.13.0 | 9.3.1 | kTLS page-refcount bug ([siderolabs/talos#13316](https://github.com/siderolabs/talos/issues/13316), [LINBIT/drbd#134](https://github.com/LINBIT/drbd/issues/134)), fixed in 9.3.3 |
+| 1.13.1–1.13.6 | 9.3.2 | Same kTLS bug; sender soft-lockup after a peer reboot (below) |
+| 1.13.7–1.13.10 | 9.3.3 | kTLS fixed, no soft-lockup fix |
+| 1.14.0–1.14.1 | 9.3.3 | No soft-lockup fix; needs the module change below |
+| 1.14.2+ | 9.3.4 | Soft-lockup and two-phase-commit fixes; needs the module change below |
+
+Verify the exact value for your target in `DRBD_DRIVER_VERSION` of `Pkgfile` at the matching tag of `siderolabs/extensions`.
+
+**DRBD 9.3.2 soft-lockup.** After a peer reboots, the DRBD sender thread loops on `dtt_send_page ... sent=-32` (EPIPE) and pins a CPU. The kernel reports `rcu: INFO: rcu_preempt detected stalls on CPUs/tasks`, and everything that waits for an RCU grace period (runc, mounts, `accept`, `drbdsetup`) goes into D state with wchan `synchronize_rcu_expedited` / `__wait_rcu_gp`. Two-phase commits time out (`Two-phase commit ... timeout`, `rv = -21` / `-23`), `drbdsetup down` hangs, image pulls hang on layer extraction, and the node may reboot itself or need a hard reset. The [DRBD 9.3.4 ChangeLog](https://github.com/LINBIT/drbd/blob/drbd-9.3.4/ChangeLog) lists "Fix soft lockups: the sender thread pinning a CPU while its connection is down", many two-phase-commit fixes, a fix that ends a hanging `drbdadm disconnect`, and "Fix two nodes ending UpToDate with different data after a reconnect".
+
+Recommendation for DRBD-backed clusters: skip 1.13.x and go 1.12 → 1.14.2+ directly. Talos 1.14 accepts upgrades from 1.12.0 (`MinimumHostUpgradeVersion` in `pkg/machinery/compatibility/talos114`), even though the Talos docs still recommend one minor at a time. A cluster already on 1.13.x should move to 1.14.2+ rather than to a later 1.13 patch.
+
+## Machine-config changes before the first node upgrade
+
+Apply these to every node before upgrading any of them. All three apply live, without a reboot.
+
+### 1. Load `drbd_transport_tcp` explicitly (required for Talos 1.14)
+
+Talos 1.14 builds the kernel with an empty modprobe path and a static usermode helper ([siderolabs/talos@e317d4b](https://github.com/siderolabs/talos/commit/e317d4b47239695f87a67e9b9484bb9aa4812fe7), from [siderolabs/pkgs#1565](https://github.com/siderolabs/pkgs/pull/1565)). The kernel can no longer load extension modules on demand through `request_module()`. DRBD used to autoload `drbd_transport_tcp` on the first `new-peer`; on 1.14 every LINSTOR adjust on the upgraded node fails with:
+
+```text
+drbdsetup new-peer ... Failure: (172) Failed to create transport (drbd_transport_xxx module missing?)
+```
+
+All resources on that node stay `Connecting` / `Unknown`, and the LINSTOR HA controller taints peers with `drbd.linbit.com/lost-quorum`. Tracked in [siderolabs/talos#14501](https://github.com/siderolabs/talos/issues/14501); the talm preset fix is [cozystack/talm#252](https://github.com/cozystack/talm/pull/252).
+
+Detect:
+
+```bash
+talosctl --talosconfig "$CONFIG_DIR/talosconfig" --nodes "$NODE_IP" read /proc/sys/kernel/modprobe   # empty on 1.14, /sbin/modprobe on 1.13
+talosctl --talosconfig "$CONFIG_DIR/talosconfig" --nodes "$NODE_IP" read /proc/modules | grep drbd   # drbd present, drbd_transport_tcp missing
+```
+
+Fix, next to the existing `drbd` entry:
+
+```yaml
+machine:
+  kernel:
+    modules:
+      - name: drbd
+        parameters:
+          - usermode_helper=disabled
+      - name: drbd_transport_tcp
+```
+
+The same applies to any other module that used to arrive through `request_module()` (extension modules, in-tree `=m` modules such as `dm-thin-pool` or `dm-multipath`): list it in `machine.kernel.modules`.
+
+### 2. Disable kexec on bare metal
+
+A kexec reboot during `talosctl upgrade` can hang bare-metal nodes: one ends powered off, another frozen with no console output. Disable kexec so Talos falls back to a firmware reboot:
+
+```yaml
+machine:
+  sysctls:
+    kernel.kexec_load_disabled: "1"
+```
+
+Talos then logs `kexec support is disabled via sysctl` and reboots through firmware. `talosctl upgrade --reboot-mode powercycle` also bypasses kexec for a single upgrade.
+
+### 3. Pin the system disk
+
+On nodes that host Talos tenant VMs, VM disks exposed on the host as loop devices or zvols carry the guest's META / BOOT / STATE partitions and can be picked as the host's SystemDisk. Talos stopped matching loop devices in 1.12.8 and 1.14.0 ([siderolabs/talos@3bae01a](https://github.com/siderolabs/talos/commit/3bae01ac11cd64265f0aaaa9e2e7f83e39bd7d73)); zvols are not covered. Right before each node upgrade:
+
+```bash
+talosctl --talosconfig "$CONFIG_DIR/talosconfig" --nodes "$NODE_IP" get systemdisk
+talosctl --talosconfig "$CONFIG_DIR/talosconfig" --nodes "$NODE_IP" get disks   # match the serial of the real boot disk
+```
+
+Pin `machine.install.disk` to a `/dev/disk/by-id/...` path or use `machine.install.diskSelector.serial`: NVMe names (`nvme0n1` / `nvme1n1`) can swap between boots.
+
+## Per-node procedure gotchas
+
+| Situation | What happens | What to do |
+| --- | --- | --- |
+| `talosctl upgrade` (1.14 client) from a workstation behind an access proxy | The client cordons and drains through the kubeconfig endpoint, often an internal VIP that the workstation cannot reach: `error cordoning node ... Gateway Timeout` | Drain beforehand with `kubectl --context $CTX drain`, then pass `--drain=false` |
+| Legacy upgrade path (old node API) with `--reboot-mode force` | Rejected by the legacy path | Use `default` or `powercycle` |
+| Drain evicts the in-cluster agent of the access proxy the operator connects through | The running `kubectl drain` dies with an "agent is offline" error; a script keeps waiting on a drain that no longer runs | Reschedule the agent off the node before the drain, or drain over a path that does not go through that agent; verify the node is empty before rebooting |
+| Counting leftover `virt-launcher` pods before reboot | Completed launchers of already-migrated VMs stay on the node | Count only `Running` / `Pending` pods |
+| Talos 1.14 post-install reboot hangs | `ext-zfs-service` stays in `Stopping` after SIGKILL, stage stays `rebooting` | The installer already switched the boot slot. After about 3 minutes run `talosctl --nodes "$NODE_IP" reboot --mode force`; the node boots the new version |
+| Automation discards command output | A failed `talosctl upgrade` or `kubectl drain` leaves nothing to diagnose | Never redirect operational output to `/dev/null`; log each step to a file |
+
+## Gate between node reboots
+
+Do not reboot the next node until storage has settled. A second reboot while a replica is still `Outdated` / `Connecting` can take quorum away from a resource.
+
+```bash
+kubectl --context $CTX get nodes --output custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key' | grep lost-quorum   # empty
+kubectl --context $CTX exec --namespace cozy-linstor deploy/linstor-controller -- linstor resource list --faulty   # only SyncTarget, or empty
+```
+
+A `drbd.linbit.com/lost-quorum` NoSchedule taint also blocks pods with local PVs pinned to that node (for example tenant etcd members). If a resource loops in `Connecting` after the reboot, see `linstor:recover` ("missed finish" resync loop).
+
+## Kubernetes minor upgrades via machine config
+
+Changing the Kubernetes version in machine config (talm: `kubernetesVersion`) applies without a reboot, about 40 seconds per node. One `kube-controller-manager` restart with `bind: address already in use` on port 10257 during the static-pod swap is benign.
+
+Before moving to a new Kubernetes minor, check:
+
+- Cozystack's tested version: `hack/e2e-prepare-cluster.bats` in `cozystack/cozystack` pins e2e to a specific Kubernetes release and names the supported range.
+- Cilium's tested range for the shipped Cilium version (Cilium 1.19 lists Kubernetes 1.32–1.35).
+- KubeVirt on Kubernetes 1.36: VMI status updates loop on strict CRD format validation of the `checksum` fields ([kubevirt/kubevirt#17858](https://github.com/kubevirt/kubevirt/issues/17858)). KubeVirt 1.8.4 generates these fields as `int64`; older releases need the fix.
+
+## Adding worker nodes
+
+If `kube-controller-manager` runs with `--allocate-node-cidrs=false`, new nodes get no `spec.podCIDR`, and Cilium in IPAM mode `kubernetes` stays not ready with `required IPv4 PodCIDR not available`. Enabling `allocate-node-cidrs` is safe on a live cluster: the range allocator marks existing node CIDRs as used and does not reassign them.