diff --git a/plugins/cozystack/.claude-plugin/plugin.json b/plugins/cozystack/.claude-plugin/plugin.json index 76e1766..0bac141 100644 --- a/plugins/cozystack/.claude-plugin/plugin.json +++ b/plugins/cozystack/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "cozystack", - "version": "1.18.0", + "version": "1.18.1", "description": "Cozystack platform skills bundle. Start with cozystack:wizard — it begins with a free-form 'tell me about your setup and goal' question, parses hints, then asks Talos / Ubuntu / Existing, builds a chain, and dispatches downstream skills via a cluster config directory the operator picks (every artifact lives there: inventory.yml, kubeconfig, .state.yaml, cozystack-platform-package.yaml — operator manages git on their own; optional sops opt-in encrypts secret files in-tree). Skills, invoked as cozystack:: wizard (orchestrator + 3-route dispatcher + Phase 4.5 active research + auto-dispatches debug on any failed_at), talos-bootstrap (Talos node prep via talm — Talos-1.12-aware maintenance probe, NAT-provider cert-SAN guardrail before first talm apply, multidoc machine-config with per-node VIP-link IPv4 stubs, etcd bootstrap, kubeconfig fetch, cozystack-tuned shape verification with Phase 11.5 auto-upgrade), talos-reset (cloud-provider terminate+relaunch helper for OCI/AWS/GCP/Hetzner when nodes are unrecoverable from inside; preserves block volumes + secondary VNICs + NSG memberships), ubuntu-bootstrap (wraps cozystack/ansible-cozystack — OS prep + k3s install in one go), cluster-install (Cozystack on a ready cluster — node-readiness, ZFS pool provisioning via privileged DaemonSet on Talos with hostNetwork, extractedprism for kube-apiserver HA, OCI-tag-normalized cozy-installer chart, Platform Package, inline tenants/root ingress patch + LINSTOR pool registration during watch loop with combined HRs-Ready + pools-registered gate, Phase 8.6 default StorageClasses for v1.3.x, Phase 9.1 end-to-end reachability probe), debug (investigate a stuck or broken install — gathers symptoms, classifies operator error / config drift / upstream bug / not-yet-supported, applies fixes or workarounds, drafts upstream issues on approval; never opens PRs or files silently), cluster-upgrade (v1.x patch/minor upgrade with release-notes analysis), package-deploy (dev-loop deploy of a single package with ExternalArtifact support), package-bump (single-package version bump with changelog adaptation), external-app-create (scaffold a new external-apps package), dev-ui-bootstrap (bootstrap a UI dev sandbox — Playwright + Vite dev server for cozystack-ui pointed at a chosen kubeconfig; worktree, @playwright/test, Chromium, playwright.config.ts, dev:e2e / test:e2e scripts, kubectl proxy + Vite on a free port), comment-audit (audit a diff's code comments for design-doc prose — rationale, product motivation, before/after narrative, incident retelling — and report cuts, tightens, keeps and arguments duplicated across sites; reports only, modifies nothing), comment-trim (apply the cuts — delete the prose, tighten verbose-but-legitimate blocks, keep every comment that documents a real mechanism, then prove mechanically with the bundled comments-only.py that no code moved and a grep-based pointer-invariant check that no pointer sits on top of the prose it replaces; leaves the result in the working tree, does not commit or push). comment-audit and comment-trim are repository-agnostic and usable on any codebase. All skills match the operator's natural language detected from conversation context — code identifiers, commands, file paths, and GitHub-public text stay canonical. All skills follow the same gate-and-confirm discipline: read-only lookups run freely; any mutation needs explicit per-step approval.", "author": { "name": "Cozystack", diff --git a/plugins/cozystack/skills/cluster-upgrade/SKILL.md b/plugins/cozystack/skills/cluster-upgrade/SKILL.md index 863f48c..93db9b4 100644 --- a/plugins/cozystack/skills/cluster-upgrade/SKILL.md +++ b/plugins/cozystack/skills/cluster-upgrade/SKILL.md @@ -146,6 +146,12 @@ Show user: result (success/partial/failed), before→after version, HR/Package t **Details, caveats, when not to roll back:** read `references/rollback.md`. +## Talos node upgrades + +Upgrading Talos or Kubernetes on the nodes is a separate rolling operation, not part of the `helm upgrade`. On DRBD-backed clusters the Talos version decides the DRBD version, and Talos 1.14 no longer autoloads `drbd_transport_tcp`. + +**Before upgrading any Talos node:** read `references/talos-node-upgrade.md` (version path, machine-config changes to apply first, per-node gotchas, gate between node reboots). + ## Known failure modes High-blast-radius stuck states — stuck helm `uninstalling`, Kamaji datastore cert mismatch, `MissingRollbackTarget`, orphan HRs from removed apps, `cozy-system` accidentally deleted, etc. @@ -162,6 +168,8 @@ High-blast-radius stuck states — stuck helm `uninstalling`, Kamaji datastore c | HR `UninstallFailed, failed to delete release` | Stuck helm history (known-failures #1) | | TCP `INSTALLED VERSION` diverges from `VERSION` | Kamaji upgrade stuck (known-failures #4) | | `cozy-system` namespace gone | Missing `helm.sh/resource-policy=keep` (known-failures #7); restore from backup | +| `linstor-controller` CrashLoopBackOff, `LAYER_DRBD_VOLUMES could not be restored` | Interrupted resource deletion left orphan layer rows (known-failures #8) | +| After a Talos 1.14 node upgrade: `Failure: (172) Failed to create transport` | `drbd_transport_tcp` not loaded (`references/talos-node-upgrade.md`) | ## Common mistakes @@ -174,7 +182,7 @@ High-blast-radius stuck states — stuck helm `uninstalling`, Kamaji datastore c ## References -- Skill files: `references/release-notes-analysis.md`, `references/preflight-checks.md`, `references/post-upgrade-checks.md`, `references/rollback.md`, `references/known-failures.md` +- Skill files: `references/release-notes-analysis.md`, `references/preflight-checks.md`, `references/post-upgrade-checks.md`, `references/rollback.md`, `references/known-failures.md`, `references/talos-node-upgrade.md` - Upstream (pick the version matching your target from the docs site version selector): `https://cozystack.io/docs//operations/cluster/upgrade/` - Troubleshooting checklist: `https://cozystack.io/docs//operations/troubleshooting/` - Releases: `https://github.com/cozystack/cozystack/releases` diff --git a/plugins/cozystack/skills/cluster-upgrade/references/known-failures.md b/plugins/cozystack/skills/cluster-upgrade/references/known-failures.md index 675412a..da700ea 100644 --- a/plugins/cozystack/skills/cluster-upgrade/references/known-failures.md +++ b/plugins/cozystack/skills/cluster-upgrade/references/known-failures.md @@ -253,6 +253,49 @@ Restore from backup. There is no clean in-cluster recovery for a deleted `cozy-s 3. Re-apply the Platform Package from rescue.yaml (manual review required; CRD schemas may have moved). 4. Expect tenant disruption; communicate to users. +## 8. LINSTOR controller crash-loops after an interrupted resource deletion + +### Symptom + +`linstor-controller` in `CrashLoopBackOff` (often after a node reboot or a controller restart during a node upgrade), with this on startup: + +```text +Database entry of table LAYER_DRBD_VOLUMES could not be restored +``` + +The detailed error report is written inside the controller container and is lost on every restart, so `linstor error-reports list` cannot show it. + +### Root cause + +With the Kubernetes CRD database backend (the Cozystack default), LINSTOR stores each table row as a separate `*.internal.linstor.linbit.com` custom resource, and a multi-row change is not transactional. A resource deletion interrupted halfway (controller killed, node rebooted) can remove the `resources` row while leaving its layer rows behind. On the next start the controller cannot attach those layer rows to a resource and refuses to load the database. + +### Recovery + +```bash +# 1. Back up every LINSTOR internal CR before touching anything +for crd in $(kubectl --context $CTX get crd --output name | grep internal.linstor.linbit.com); do + kubectl --context $CTX get "${crd#*/}" --output yaml > "backup-${crd#*/}.yaml" +done + +# 2. Find orphaned layer resources: rows in layerresourceids whose +# (node_name, resource_name, snapshot_name) has no matching row in resources +kubectl --context $CTX get resources.internal.linstor.linbit.com --output json \ + | jq -r '.items[].spec | "\(.node_name)/\(.resource_name)"' | sort -u > resources.txt +kubectl --context $CTX get layerresourceids.internal.linstor.linbit.com --output json \ + | jq -r '.items[] | select((.spec.snapshot_name // "") == "") | "\(.spec.node_name)/\(.spec.resource_name) \(.spec.layer_resource_id) \(.metadata.name)"' \ + | awk 'NR==FNR { have[$1]=1; next } !($1 in have)' resources.txt - +# Rows with a non-empty snapshot_name belong to snapshots; check those against snapshots.internal.linstor.linbit.com instead. + +# 3. For each orphaned layer_resource_id, collect the rows that reference it in +# layerdrbdresources, layerdrbdvolumes and layerstoragevolumes (field layer_resource_id) + +# 4. Delete exactly those CRs (orphaned layerresourceids + the collected layer rows), then restart the controller +kubectl --context $CTX delete .internal.linstor.linbit.com +kubectl --context $CTX rollout restart --namespace cozy-linstor deployment/linstor-controller +``` + +Field names in the CR `spec` are the lower-cased database column names; confirm them on one object (`kubectl --context $CTX get layerresourceids.internal.linstor.linbit.com --output yaml | head -40`) before scripting. Every deletion is a stop gate: show the list and wait for approval. After the controller starts, run `linstor resource list --faulty` and let `linstor:recover` handle the replicas of the affected resource. + ## Diagnostic quick reference | Question | Command | diff --git a/plugins/cozystack/skills/cluster-upgrade/references/talos-node-upgrade.md b/plugins/cozystack/skills/cluster-upgrade/references/talos-node-upgrade.md new file mode 100644 index 0000000..b9c3894 --- /dev/null +++ b/plugins/cozystack/skills/cluster-upgrade/references/talos-node-upgrade.md @@ -0,0 +1,116 @@ +# Talos Node Upgrades Under Cozystack + +Read this before upgrading Talos (or Kubernetes via machine config) on nodes of a running Cozystack cluster. The Cozystack `helm upgrade` in the main skill does not touch Talos; node upgrades are a separate, per-node rolling operation, and on DRBD-backed clusters most of the risk sits in the storage layer, not in Talos itself. + +## Pick the target version (DRBD first) + +The DRBD kernel module ships inside the `siderolabs/drbd` extension, so the Talos version decides the DRBD version. Check the extension before picking a Talos target: + +| Talos | DRBD in extension | Notes | +| --- | --- | --- | +| 1.12.x | 9.2.16 | Bitmap race, see `linstor:recover` known upstream bugs | +| 1.13.0 | 9.3.1 | kTLS page-refcount bug ([siderolabs/talos#13316](https://github.com/siderolabs/talos/issues/13316), [LINBIT/drbd#134](https://github.com/LINBIT/drbd/issues/134)), fixed in 9.3.3 | +| 1.13.1–1.13.6 | 9.3.2 | Same kTLS bug; sender soft-lockup after a peer reboot (below) | +| 1.13.7–1.13.10 | 9.3.3 | kTLS fixed, no soft-lockup fix | +| 1.14.0–1.14.1 | 9.3.3 | No soft-lockup fix; needs the module change below | +| 1.14.2+ | 9.3.4 | Soft-lockup and two-phase-commit fixes; needs the module change below | + +Verify the exact value for your target in `DRBD_DRIVER_VERSION` of `Pkgfile` at the matching tag of `siderolabs/extensions`. + +**DRBD 9.3.2 soft-lockup.** After a peer reboots, the DRBD sender thread loops on `dtt_send_page ... sent=-32` (EPIPE) and pins a CPU. The kernel reports `rcu: INFO: rcu_preempt detected stalls on CPUs/tasks`, and everything that waits for an RCU grace period (runc, mounts, `accept`, `drbdsetup`) goes into D state with wchan `synchronize_rcu_expedited` / `__wait_rcu_gp`. Two-phase commits time out (`Two-phase commit ... timeout`, `rv = -21` / `-23`), `drbdsetup down` hangs, image pulls hang on layer extraction, and the node may reboot itself or need a hard reset. The [DRBD 9.3.4 ChangeLog](https://github.com/LINBIT/drbd/blob/drbd-9.3.4/ChangeLog) lists "Fix soft lockups: the sender thread pinning a CPU while its connection is down", many two-phase-commit fixes, a fix that ends a hanging `drbdadm disconnect`, and "Fix two nodes ending UpToDate with different data after a reconnect". + +Recommendation for DRBD-backed clusters: skip 1.13.x and go 1.12 → 1.14.2+ directly. Talos 1.14 accepts upgrades from 1.12.0 (`MinimumHostUpgradeVersion` in `pkg/machinery/compatibility/talos114`), even though the Talos docs still recommend one minor at a time. A cluster already on 1.13.x should move to 1.14.2+ rather than to a later 1.13 patch. + +## Machine-config changes before the first node upgrade + +Apply these to every node before upgrading any of them. All three apply live, without a reboot. + +### 1. Load `drbd_transport_tcp` explicitly (required for Talos 1.14) + +Talos 1.14 builds the kernel with an empty modprobe path and a static usermode helper ([siderolabs/talos@e317d4b](https://github.com/siderolabs/talos/commit/e317d4b47239695f87a67e9b9484bb9aa4812fe7), from [siderolabs/pkgs#1565](https://github.com/siderolabs/pkgs/pull/1565)). The kernel can no longer load extension modules on demand through `request_module()`. DRBD used to autoload `drbd_transport_tcp` on the first `new-peer`; on 1.14 every LINSTOR adjust on the upgraded node fails with: + +```text +drbdsetup new-peer ... Failure: (172) Failed to create transport (drbd_transport_xxx module missing?) +``` + +All resources on that node stay `Connecting` / `Unknown`, and the LINSTOR HA controller taints peers with `drbd.linbit.com/lost-quorum`. Tracked in [siderolabs/talos#14501](https://github.com/siderolabs/talos/issues/14501); the talm preset fix is [cozystack/talm#252](https://github.com/cozystack/talm/pull/252). + +Detect: + +```bash +talosctl --talosconfig "$CONFIG_DIR/talosconfig" --nodes "$NODE_IP" read /proc/sys/kernel/modprobe # empty on 1.14, /sbin/modprobe on 1.13 +talosctl --talosconfig "$CONFIG_DIR/talosconfig" --nodes "$NODE_IP" read /proc/modules | grep drbd # drbd present, drbd_transport_tcp missing +``` + +Fix, next to the existing `drbd` entry: + +```yaml +machine: + kernel: + modules: + - name: drbd + parameters: + - usermode_helper=disabled + - name: drbd_transport_tcp +``` + +The same applies to any other module that used to arrive through `request_module()` (extension modules, in-tree `=m` modules such as `dm-thin-pool` or `dm-multipath`): list it in `machine.kernel.modules`. + +### 2. Disable kexec on bare metal + +A kexec reboot during `talosctl upgrade` can hang bare-metal nodes: one ends powered off, another frozen with no console output. Disable kexec so Talos falls back to a firmware reboot: + +```yaml +machine: + sysctls: + kernel.kexec_load_disabled: "1" +``` + +Talos then logs `kexec support is disabled via sysctl` and reboots through firmware. `talosctl upgrade --reboot-mode powercycle` also bypasses kexec for a single upgrade. + +### 3. Pin the system disk + +On nodes that host Talos tenant VMs, VM disks exposed on the host as loop devices or zvols carry the guest's META / BOOT / STATE partitions and can be picked as the host's SystemDisk. Talos stopped matching loop devices in 1.12.8 and 1.14.0 ([siderolabs/talos@3bae01a](https://github.com/siderolabs/talos/commit/3bae01ac11cd64265f0aaaa9e2e7f83e39bd7d73)); zvols are not covered. Right before each node upgrade: + +```bash +talosctl --talosconfig "$CONFIG_DIR/talosconfig" --nodes "$NODE_IP" get systemdisk +talosctl --talosconfig "$CONFIG_DIR/talosconfig" --nodes "$NODE_IP" get disks # match the serial of the real boot disk +``` + +Pin `machine.install.disk` to a `/dev/disk/by-id/...` path or use `machine.install.diskSelector.serial`: NVMe names (`nvme0n1` / `nvme1n1`) can swap between boots. + +## Per-node procedure gotchas + +| Situation | What happens | What to do | +| --- | --- | --- | +| `talosctl upgrade` (1.14 client) from a workstation behind an access proxy | The client cordons and drains through the kubeconfig endpoint, often an internal VIP that the workstation cannot reach: `error cordoning node ... Gateway Timeout` | Drain beforehand with `kubectl --context $CTX drain`, then pass `--drain=false` | +| Legacy upgrade path (old node API) with `--reboot-mode force` | Rejected by the legacy path | Use `default` or `powercycle` | +| Drain evicts the in-cluster agent of the access proxy the operator connects through | The running `kubectl drain` dies with an "agent is offline" error; a script keeps waiting on a drain that no longer runs | Reschedule the agent off the node before the drain, or drain over a path that does not go through that agent; verify the node is empty before rebooting | +| Counting leftover `virt-launcher` pods before reboot | Completed launchers of already-migrated VMs stay on the node | Count only `Running` / `Pending` pods | +| Talos 1.14 post-install reboot hangs | `ext-zfs-service` stays in `Stopping` after SIGKILL, stage stays `rebooting` | The installer already switched the boot slot. After about 3 minutes run `talosctl --nodes "$NODE_IP" reboot --mode force`; the node boots the new version | +| Automation discards command output | A failed `talosctl upgrade` or `kubectl drain` leaves nothing to diagnose | Never redirect operational output to `/dev/null`; log each step to a file | + +## Gate between node reboots + +Do not reboot the next node until storage has settled. A second reboot while a replica is still `Outdated` / `Connecting` can take quorum away from a resource. + +```bash +kubectl --context $CTX get nodes --output custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key' | grep lost-quorum # empty +kubectl --context $CTX exec --namespace cozy-linstor deploy/linstor-controller -- linstor resource list --faulty # only SyncTarget, or empty +``` + +A `drbd.linbit.com/lost-quorum` NoSchedule taint also blocks pods with local PVs pinned to that node (for example tenant etcd members). If a resource loops in `Connecting` after the reboot, see `linstor:recover` ("missed finish" resync loop). + +## Kubernetes minor upgrades via machine config + +Changing the Kubernetes version in machine config (talm: `kubernetesVersion`) applies without a reboot, about 40 seconds per node. One `kube-controller-manager` restart with `bind: address already in use` on port 10257 during the static-pod swap is benign. + +Before moving to a new Kubernetes minor, check: + +- Cozystack's tested version: `hack/e2e-prepare-cluster.bats` in `cozystack/cozystack` pins e2e to a specific Kubernetes release and names the supported range. +- Cilium's tested range for the shipped Cilium version (Cilium 1.19 lists Kubernetes 1.32–1.35). +- KubeVirt on Kubernetes 1.36: VMI status updates loop on strict CRD format validation of the `checksum` fields ([kubevirt/kubevirt#17858](https://github.com/kubevirt/kubevirt/issues/17858)). KubeVirt 1.8.4 generates these fields as `int64`; older releases need the fix. + +## Adding worker nodes + +If `kube-controller-manager` runs with `--allocate-node-cidrs=false`, new nodes get no `spec.podCIDR`, and Cilium in IPAM mode `kubernetes` stays not ready with `required IPv4 PodCIDR not available`. Enabling `allocate-node-cidrs` is safe on a live cluster: the range allocator marks existing node CIDRs as used and does not reassign them. diff --git a/plugins/cozystack/skills/talos-bootstrap/SKILL.md b/plugins/cozystack/skills/talos-bootstrap/SKILL.md index b09fe66..62a8dbd 100644 --- a/plugins/cozystack/skills/talos-bootstrap/SKILL.md +++ b/plugins/cozystack/skills/talos-bootstrap/SKILL.md @@ -24,7 +24,7 @@ Work in reasoning mode. Use the phrasing `cozystack:talos-bootstrap`. Announce p 3. `~/git/github.com/cozystack/cozystack/packages/core/talos/images/talos/profiles/installer.yaml` default. 4. URL fallback `https://raw.githubusercontent.com/cozystack/cozystack//packages/core/talos/images/talos/profiles/installer.yaml` when no local clone is present (`` is the cozystack tag from `state.cozystack.installer_version` or the latest from `git ls-remote --tags https://github.com/cozystack/cozystack` if not set). - System extensions: drbd, zfs, openvswitch, plus firmware (amd-ucode / intel-ucode / intel-ice / etc.). - - Kernel modules in machine-config: drbd, zfs, spl, openvswitch, vfio_pci, vfio_iommu_type1. + - Kernel modules in machine-config: drbd, drbd_transport_tcp, zfs, spl, openvswitch, vfio_pci, vfio_iommu_type1. `drbd_transport_tcp` is mandatory from Talos 1.14: the kernel no longer loads extension modules on demand (see Phase 7). - LVM filter in `/etc/lvm/lvm.conf`: `global_filter = [ "r|^/dev/drbd.*|", "r|^/dev/dm-.*|", "r|^/dev/zd.*|" ]`. - talm preset: `cozystack` chart from `~/git/github.com/cozystack/talm/charts/cozystack`. - Verify before declaring success. A green Apply is not enough — `kubectl get nodes Ready`, talos extensions present, LVM filter present. @@ -402,7 +402,8 @@ If `state.cluster.vip.per_node[$node]` exists but the LinkConfig doc is missing, Present each `nodes/.yaml` for review. Things to spot before apply: - `machine.install.image` should be `ghcr.io/cozystack/cozystack/talos:`. The cozystack preset sets this; surface if missing. -- `machine.kernel.modules` should list drbd / zfs / spl / openvswitch / vfio_pci / vfio_iommu_type1. +- `machine.kernel.modules` should list drbd / drbd_transport_tcp / zfs / spl / openvswitch / vfio_pci / vfio_iommu_type1. Talos 1.14 builds the kernel with an empty modprobe path ([siderolabs/talos#14501](https://github.com/siderolabs/talos/issues/14501)), so DRBD can no longer autoload `drbd_transport_tcp` on the first peer connection and every `drbdsetup new-peer` fails with `Failure: (172) Failed to create transport (drbd_transport_xxx module missing?)`. If the preset does not list it yet ([cozystack/talm#252](https://github.com/cozystack/talm/pull/252)), add `- name: drbd_transport_tcp` after the `drbd` entry. The same applies to any other module the operator expects the kernel to load on demand. +- Bare metal: add `kernel.kexec_load_disabled: "1"` to `machine.sysctls`. A kexec reboot during a later `talosctl upgrade` can hang bare-metal nodes; with kexec disabled Talos reboots through firmware. - For CP nodes: `machine.type: controlplane`, optional `machine.network.interfaces[].vip` for HA. - `machine.install.disk` defaults to a heuristic — confirm it picks the right system disk on multi-disk nodes (operator can edit nodes/.yaml before apply). @@ -514,6 +515,8 @@ talosctl --talosconfig "$TALOSCONFIG" --nodes "$NODES" \ | grep -E '^Tag:' | sort -u ``` +On Talos 1.14+ also expect `drbd_transport_tcp` in `/proc/modules`; if it is missing, fix `machine.kernel.modules` (Phase 7) and re-apply, no reboot needed. + If any check fails, the skill **does not silently mark `failed_at`** — it tries to reconcile via Phase 11.5 auto-upgrade first (the most common cause of missing extensions is operator-booted-from-base-Talos-image instead of cozystack-tuned). Only after Phase 11.5 fails does the skill write `status.talos-bootstrap.failed_at`. `cluster-install` will refuse a Talos cluster missing extensions. ## Phase 11.5 — Auto-upgrade to cozystack-tuned image (when Phase 11 fails on extensions/drift) diff --git a/plugins/cozystack/skills/talos-bootstrap/references/manual-steps.md b/plugins/cozystack/skills/talos-bootstrap/references/manual-steps.md index c77b5fc..3d7f269 100644 --- a/plugins/cozystack/skills/talos-bootstrap/references/manual-steps.md +++ b/plugins/cozystack/skills/talos-bootstrap/references/manual-steps.md @@ -78,7 +78,8 @@ $EDITOR nodes/w1.yaml Things to verify (the `cozystack` preset sets sensible defaults, but verify anyway): - `machine.install.image` points at `ghcr.io/cozystack/cozystack/talos:`. -- `machine.kernel.modules` lists drbd / zfs / spl / openvswitch / vfio_pci / vfio_iommu_type1. +- `machine.kernel.modules` lists drbd / drbd_transport_tcp / zfs / spl / openvswitch / vfio_pci / vfio_iommu_type1. `drbd_transport_tcp` is required on Talos 1.14+, which no longer loads extension modules on demand. +- On bare metal, `machine.sysctls` sets `kernel.kexec_load_disabled: "1"` so later upgrades reboot through firmware instead of kexec. - `machine.files` contains `/etc/lvm/lvm.conf` overwrite with the cozystack global_filter. - For CP nodes: `machine.type: controlplane`, with optional `machine.network.interfaces[].vip` for HA. diff --git a/plugins/linstor/.claude-plugin/plugin.json b/plugins/linstor/.claude-plugin/plugin.json index d8f9742..801de79 100644 --- a/plugins/linstor/.claude-plugin/plugin.json +++ b/plugins/linstor/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "linstor", - "version": "1.0.0", + "version": "1.0.1", "description": "LINSTOR / DRBD operations bundle for Kubernetes. Skills, invoked as linstor:: recover (diagnose and recover broken DRBD resources — handles StandAlone, DELETING, Inconsistent, Diskless, quorum loss, bitmap errors, and other common failure modes). Useful on any Kubernetes cluster that runs piraeus-operator / LINSTOR, not just on Cozystack.", "author": { "name": "Cozystack", diff --git a/plugins/linstor/skills/recover/SKILL.md b/plugins/linstor/skills/recover/SKILL.md index fdf4e27..310d3ce 100644 --- a/plugins/linstor/skills/recover/SKILL.md +++ b/plugins/linstor/skills/recover/SKILL.md @@ -107,10 +107,12 @@ Is the resource in StandAlone? │ Is the resource in Connecting? ├─ YES: Check error-reports. May be TCP port mismatch, bitmap error, or peer not up. +│ ├─ If dmesg shows `missed-finish` / `I shall become SyncTarget, but I am primary!`: see "Missed-finish resync loop" below +│ └─ If `new-peer` fails with `Failure: (172) Failed to create transport`: `drbd_transport_tcp` is not loaded (Known Upstream Bugs #7) │ Is the resource Inconsistent or Outdated? ├─ YES: If it has a Connected UpToDate peer, resync should happen automatically. -│ └─ If stuck: `drbdadm disconnect ` then `drbdadm connect ` +│ ├─ If stuck: `drbdadm disconnect ` then `drbdadm connect ` │ └─ If really stuck: `drbdadm down ` then `drbdadm up ` (only if Unused!) │ Is the resource Diskless when it should be diskful? @@ -260,6 +262,30 @@ drbdadm down drbdadm up ``` +## Fix: Missed-finish resync loop (node rebooted mid-resync) + +A node rebooted in the middle of a resync; after it returns the pair loops in `Connecting` and never resyncs. dmesg on the two sides: + +```text +# Outdated (rebooted) side +uuid_compare()=source-use-bitmap by rule=sync-source-missed-finish +# UpToDate Primary side +... by rule=sync-target-peer-missed-finish +I shall become SyncTarget, but I am primary! +``` + +Both sides think the other one must be the target. The Outdated replica has no quorum, so the HA controller taints its node `drbd.linbit.com/lost-quorum` (NoSchedule), which also blocks pods with local PVs pinned to that node (for example tenant etcd members). Rebooting another node in this state can take quorum away from the resource entirely. + +`drbdadm invalidate` hangs waiting for a connection, and `connect --discard-my-data` can still run into two-phase-commit timeouts on DRBD 9.3.2. What works, on the **Outdated** node only (the replica there must be Secondary and unused), after confirming the Primary side is UpToDate (principle 4): + +```bash +# For each peer of the resource on the stale node: +drbdsetup disconnect --force +drbdsetup down +``` + +Then let LINSTOR recreate the replica on its next adjust and watch `linstor r l -r `: it should come back and resync from the Primary. Ask before running this: `down` drops the local replica's participation until LINSTOR recreates it. + ## Fix: Suspended I/O (quorum lost) ```bash @@ -364,3 +390,5 @@ Prioritize by presence of UpToDate replicas: resources with zero UpToDate copies 3. **Toggle-disk doesn't preserve TCP ports** — `removeLayerData` frees ports, `ensureStackDataExists` allocates different ones. Fix: PR #476. 4. **CSI can delete ResourceDefinition while PVC is Bound** — if all resources have FlagDelete, CSI removes RD. Fix: linstor-csi PR #429. 5. **TCP sysctl defaults under DRBD churn** — Linux kernel's default `tcp_orphan_retries` produces excessive orphan-socket retries under DRBD load. On most distributions the sysctl reads `0`, which the kernel internally substitutes with `8`; some distros (Ubuntu, Debian) also expose it as `8` directly. Talos inherits this default unchanged. Fix: set `tcp_orphan_retries=3`, `tcp_fin_timeout=30`, `netdev_max_backlog=5000`. +6. **DRBD 9.3.2 sender soft-lockup** — after a peer reboots, the sender thread loops on `dtt_send_page ... sent=-32` and pins a CPU; RCU stalls (`rcu_preempt detected stalls`) push runc, mounts and `drbdsetup` into D state, two-phase commits time out (`rv = -21` / `-23`), `drbdsetup down` hangs. Fixed in DRBD 9.3.4 ("Fix soft lockups: the sender thread pinning a CPU while its connection is down"). On Talos, 9.3.2 ships with 1.13.1–1.13.6 and 9.3.4 first ships with 1.14.2. +7. **Talos 1.14 does not autoload `drbd_transport_tcp`** — the kernel has an empty modprobe path, so `drbdsetup new-peer` fails with `Failure: (172) Failed to create transport (drbd_transport_xxx module missing?)` and every resource on the node stays Connecting. Fix: list `drbd_transport_tcp` in `machine.kernel.modules` (applies without reboot). siderolabs/talos#14501.