docs: add Talos node upgrade and DRBD recovery notes

[Aleksei Sviridkin (lexfrei)]

The skills had nothing on upgrading Talos under a running Cozystack cluster, and most of what broke for me on a bare-metal cluster with LINSTOR/DRBD storage was in that gap. The cluster-upgrade skill stops at helm upgrade. The nodes, and the DRBD version that comes with each Talos release, are left to the operator.

This adds what I hit during that upgrade:

• A new cluster-upgrade reference for Talos node upgrades. It maps each Talos release to the DRBD version in the drbd extension and recommends skipping 1.13.x, because the DRBD 9.3.2 sender soft-lockup is only fixed in 9.3.4, which first ships with Talos 1.14.2. It also covers the machine-config changes to make before the first node upgrade, per-node gotchas, and a storage check between node reboots.
• A new cluster-upgrade known failure for the LINSTOR controller crash-loop on orphan layer rows after an interrupted deletion.
• drbd_transport_tcp in the talos-bootstrap module lists, plus kexec disabled on bare metal. Talos 1.14 no longer loads extension modules on demand (v1.14: kernel no longer loads extension modules on demand (drbd_transport_tcp) siderolabs/talos#14501), and the preset fix is still open in fix(cozystack): load drbd_transport_tcp explicitly talm#252.
• The "missed finish" resync loop after a mid-resync reboot in linstor:recover, plus the two DRBD bugs above.

Version claims are checked against upstream sources. DRBD versions come from Pkgfile at each siderolabs/extensions tag. The rest comes from the DRBD 9.3.4 ChangeLog, MinimumHostUpgradeVersion in Talos 1.14 and the checksum format in the KubeVirt 1.8.4 CRDs.

tools/check-refs.sh passes. markdownlint only flags the repeated Symptom, Root cause and Recovery headings, and every entry in known-failures.md already uses them.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 7180ebd3-9a9f-432c-ae95-94d2633ae086

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[IvanHunters]

LGTM

A careful, well-sourced runbook. I treated every command as something an operator will paste during an incident and checked the ones that can lose data or that depend on a third-party contract. They hold up.

Commands I verified against their real contracts:

• drbdsetup disconnect <resource> <peer-node-id> --force and drbdsetup down <resource> (the missed-finish recovery) match drbdsetup's own usage (disconnect {resource} {peer_node_id} [--force], down {resource}), checked with drbd-utils in a container.
• linstor resource list --faulty / linstor r l -r <resource> are the right client invocations.
• The Talos version-path claim is exact: MinimumHostUpgradeVersion = semver.MustParse("1.12.0") really is in pkg/machinery/compatibility/talos114/talos114.go, so "1.14 accepts upgrades from 1.12.0" is correct, and MinimumKubernetesVersion there is 1.32.0, consistent with the Kubernetes-range guidance.
• Port 10257 is kube-controller-manager's secure port, so the "address already in use during static-pod swap is benign" note is right.

Data-loss surface, and why each procedure is safe as written:

• known-failures #8 (deleting LINSTOR internal CRs): backs up every *.internal.linstor.linbit.com object first, the delete commands are placeholders, and the text makes every deletion a stop gate that waits for approval. The orphan-detection jq/awk is sound for the stated root cause (a resources row removed while its layer rows remain), it filters snapshot rows out explicitly, and it tells the operator to confirm the lower-cased column field names on one object before scripting. Nothing auto-deletes.
• Missed-finish resync loop (drbdsetup down): correctly scoped to the Outdated/Secondary/unused replica only, after confirming the other side is UpToDate Primary, with an explicit "ask before running" and a note that down drops the replica until LINSTOR recreates it. That is the safe direction (drop the stale copy, keep the good one), not the dangerous one.
• Talos node upgrades are presented as a per-node rolling operation with a storage-settle gate (lost-quorum taint empty, linstor resource list --faulty clean) before the next reboot.

Cross-references all resolve: the new talos-node-upgrade.md, known-failures #8, Known Upstream Bugs #6/#7 and the two symptom-table rows line up, and the repo's cross-reference-validator CI is green on the reviewed head (as are jq-lint and DCO). Version bumps (cozystack 1.17.0->1.17.1, linstor 1.0.0->1.0.1) are appropriate patch bumps for additive notes.

One trivial nit inline on the decision-tree glyph; non-blocking.

[IvanHunters]

[NIT] Decision-tree uses two final-branch glyphs under one parent

Both new sub-items hang off the Connecting branch with └─; the first should be ├─ and only the last └─, otherwise the tree renders with two closing elbows in a row. Cosmetic only.

[Aleksei Sviridkin (lexfrei)]

Fixed in 940a52c. Also fixed the same glyph in the Inconsistent or Outdated branch.

[IvanHunters]

Verdict

LGTM

Re-review after the earlier approve was dismissed by the rebase and the new linstor commit. The one nit from last round is fixed and nothing else of substance changed.

Reconciliation with my earlier round

The decision-tree glyph nit at plugins/linstor/skills/recover/SKILL.md:110 is fixed: the first sub-item under the Connecting branch is now ├─ and only the last stays └─. The identical double-elbow one branch below, under Inconsistent or Outdated, was corrected in the same change even though I had not flagged it.

Measured against the revision I approved, the only content change across all eight files is those two glyphs plus the plugin version bumps (linstor 1.0.0 to 1.0.1, cozystack 1.17.1 to 1.18.1). The missed-finish recovery section, the DRBD 9.3.x bug notes, the Talos node-upgrade reference and the LINSTOR CRD-orphan recovery were all present at the revision I already reviewed; the larger diff against the base is the rebase onto the slim-variants merge, not new work.

Runbook safety

The destructive steps are safe when followed as written: drbdsetup disconnect <resource> <peer-node-id> --force and drbdsetup down <resource> in the missed-finish fix, and kubectl delete <kind>.internal.linstor.linbit.com <name> in the CRD-orphan recovery all carry placeholders that cannot run verbatim, explicit preconditions (Outdated and Secondary node only with the Primary confirmed UpToDate; orphan rows only, after a full CR backup), an ask-before-running stop gate, and a reversible outcome (LINSTOR recreates the replica). The drbdsetup subcommands, their positional peer_node_id, and --force match a drbd-utils help oracle.

Caveats

I did no cluster-side validation. This is a docs change, so the review is reading the runbooks and checking command syntax against an offline oracle, nothing more.