Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion plugins/cozystack/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "cozystack",
"version": "1.17.0",
"version": "1.18.0",
"description": "Cozystack platform skills bundle. Start with cozystack:wizard — it begins with a free-form 'tell me about your setup and goal' question, parses hints, then asks Talos / Ubuntu / Existing, builds a chain, and dispatches downstream skills via a cluster config directory the operator picks (every artifact lives there: inventory.yml, kubeconfig, .state.yaml, cozystack-platform-package.yaml — operator manages git on their own; optional sops opt-in encrypts secret files in-tree). Skills, invoked as cozystack:<name>: wizard (orchestrator + 3-route dispatcher + Phase 4.5 active research + auto-dispatches debug on any failed_at), talos-bootstrap (Talos node prep via talm — Talos-1.12-aware maintenance probe, NAT-provider cert-SAN guardrail before first talm apply, multidoc machine-config with per-node VIP-link IPv4 stubs, etcd bootstrap, kubeconfig fetch, cozystack-tuned shape verification with Phase 11.5 auto-upgrade), talos-reset (cloud-provider terminate+relaunch helper for OCI/AWS/GCP/Hetzner when nodes are unrecoverable from inside; preserves block volumes + secondary VNICs + NSG memberships), ubuntu-bootstrap (wraps cozystack/ansible-cozystack — OS prep + k3s install in one go), cluster-install (Cozystack on a ready cluster — node-readiness, ZFS pool provisioning via privileged DaemonSet on Talos with hostNetwork, extractedprism for kube-apiserver HA, OCI-tag-normalized cozy-installer chart, Platform Package, inline tenants/root ingress patch + LINSTOR pool registration during watch loop with combined HRs-Ready + pools-registered gate, Phase 8.6 default StorageClasses for v1.3.x, Phase 9.1 end-to-end reachability probe), debug (investigate a stuck or broken install — gathers symptoms, classifies operator error / config drift / upstream bug / not-yet-supported, applies fixes or workarounds, drafts upstream issues on approval; never opens PRs or files silently), cluster-upgrade (v1.x patch/minor upgrade with release-notes analysis), package-deploy (dev-loop deploy of a single package with ExternalArtifact support), package-bump (single-package version bump with changelog adaptation), external-app-create (scaffold a new external-apps package), dev-ui-bootstrap (bootstrap a UI dev sandbox — Playwright + Vite dev server for cozystack-ui pointed at a chosen kubeconfig; worktree, @playwright/test, Chromium, playwright.config.ts, dev:e2e / test:e2e scripts, kubectl proxy + Vite on a free port), comment-audit (audit a diff's code comments for design-doc prose — rationale, product motivation, before/after narrative, incident retelling — and report cuts, tightens, keeps and arguments duplicated across sites; reports only, modifies nothing), comment-trim (apply the cuts — delete the prose, tighten verbose-but-legitimate blocks, keep every comment that documents a real mechanism, then prove mechanically with the bundled comments-only.py that no code moved and a grep-based pointer-invariant check that no pointer sits on top of the prose it replaces; leaves the result in the working tree, does not commit or push). comment-audit and comment-trim are repository-agnostic and usable on any codebase. All skills match the operator's natural language detected from conversation context — code identifiers, commands, file paths, and GitHub-public text stay canonical. All skills follow the same gate-and-confirm discipline: read-only lookups run freely; any mutation needs explicit per-step approval.",
"author": {
"name": "Cozystack",
Expand Down
27 changes: 15 additions & 12 deletions plugins/cozystack/skills/cluster-install/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
name: cluster-install
description: Use when installing Cozystack on an existing Kubernetes cluster (kubeadm / k3s / RKE2 / managed). Discovers cluster facts, validates node readiness via kubectl debug, recommends an installer + platform variant, gathers values interactively, creates the ZFS pool on each storage node through kubectl debug (cozystack standardises on ZFS for LINSTOR — LVM / LVM-thin paths are not supported), installs extractedprism (per-node kube-apiserver HA proxy, generic variant default), installs the cozy-installer chart, applies the Platform Package (on current releases OIDC is enabled after convergence, not at install), patches the root Tenant host + ingress, waits until every HelmRelease is Ready, prints a NOTES-style access summary, and offers an issue-template handoff to cozystack/* on fatal failure. Not for Kubernetes bootstrap and not for upgrades — see `cozystack:cluster-upgrade` for that. Talos node-prep is out of scope — `cozystack:cluster-install` refuses if Talos nodes are missing cozystack-tuned extensions and points at `cozystack:talos-bootstrap`.
argument-hint: "[--config-dir=<path>] [--context=<kube-context>] [--installer-version=<vX.Y.Z>] [--installer-variant=<talos|generic|hosted>] [--platform-variant=<isp-full|isp-full-generic|isp-hosted|default>] [--no-extractedprism] [--api-host=<ip>] [--dry-run]"
argument-hint: "[--config-dir=<path>] [--context=<kube-context>] [--installer-version=<vX.Y.Z>] [--installer-variant=<talos|generic|hosted>] [--platform-variant=<isp-full|isp-full-generic|isp-hosted|isp-slim|isp-slim-generic|isp-hosted-slim|default>] [--no-extractedprism] [--api-host=<ip>] [--dry-run]"
---

# cozystack:cluster-install
Expand Down Expand Up @@ -75,7 +75,7 @@ Collect:
- Existing storage classes and the default.
- Existing LoadBalancer-class services.
- Control-plane node labels: `node-role.kubernetes.io/control-plane` key **and value** per node (see `references/node-checks.md`).
- **Storage discovery** (per node, via `kubectl debug node` in read-only mode — `lsblk`, `pvs`, `vgs`, `lvs --all`, `zpool list`, `command -v pvcreate vgcreate lvcreate lvs`; on Talos additionally `lsmod` for `drbd`/`zfs`/`openvswitch` and `/etc/lvm/lvm.conf` global_filter). Records: unmounted devices ≥ 50 GiB, existing VGs / LVs / zpools (will be reused as Phase 4 defaults), LVM-tools availability, Talos extension readiness. Skip on `isp-hosted`.
- **Storage discovery** (per node, via `kubectl debug node` in read-only mode — `lsblk`, `pvs`, `vgs`, `lvs --all`, `zpool list`, `command -v pvcreate vgcreate lvcreate lvs`; on Talos additionally `lsmod` for `drbd`/`zfs`/`openvswitch` and `/etc/lvm/lvm.conf` global_filter). Records: unmounted devices ≥ 50 GiB, existing VGs / LVs / zpools (will be reused as Phase 4 defaults), LVM-tools availability, Talos extension readiness. Skip on `isp-hosted` and `isp-hosted-slim`.

Apply variant recommendation logic from `references/variants.md`. Print:

Expand Down Expand Up @@ -106,17 +106,18 @@ recommended:
Hard refusals (don't move on, surface clearly):

- Cluster domain is not `cozy.local` → refuse, link to bootstrap docs for the user's distribution.
- Selected variant (recommended, overridden or passed with `--platform-variant`) is `isp-slim` / `isp-slim-generic` and any node has an empty `spec.podCIDR` (`kubectl --context $CTX get nodes --output jsonpath='{range .items[*]}{.metadata.name}={.spec.podCIDR}{"\n"}{end}'`) → refuse: Cilium takes pod ranges from that field there. Fix with `--pod-network-cidr` on kubeadm or `--allocate-node-cidrs` plus `--cluster-cidr` on kube-controller-manager, or pick the full variant.
- CNI conflict (non-hosted) → refuse, list pods to remove.
- Conflicting workloads (ingress-nginx, cert-manager, etc., non-hosted) → refuse, list workloads.
- kubectl can't reach the cluster → refuse, suggest `kubectl auth can-i '*' '*' --all-namespaces`.

## Phase 3 — Node readiness validation

Skip on `isp-hosted` (no node access required).
Skip on `isp-hosted` and `isp-hosted-slim` (no node access required).

For every other case, drive `kubectl debug node` per `references/node-checks.md`. For homogeneous clusters, check one sample CP node and one sample worker; warn that the rest are assumed identical. For heterogeneous, check every node.

**Talos-specific early-exit.** If Phase 2 detected Talos on any node, run the four Talos checks from `references/node-checks.md` (lsmod drbd / lsmod zfs / lsmod openvswitch / `/etc/lvm/lvm.conf` cozystack global_filter). If any of them fail on any Talos node, **STOP GATE 1 fails immediately** with this message:
**Talos-specific early-exit.** If Phase 2 detected Talos on any node, run the four Talos checks from `references/node-checks.md` (lsmod drbd / lsmod zfs / lsmod openvswitch / `/etc/lvm/lvm.conf` cozystack global_filter). On `isp-slim` drop the openvswitch check: there is no Kube-OVN. If any of them fail on any Talos node, **STOP GATE 1 fails immediately** with this message:

```text
Talos nodes detected without cozystack-tuned extensions.
Expand Down Expand Up @@ -224,12 +225,14 @@ options:
Slot legend (every slot the operator may want to edit):

1. **Bundles** (multiSelect, defaults from variant overlay):
- system (required for `isp-full*`; off for `isp-hosted`)
- system (on for every `isp-*` variant except `isp-hosted` up to v1.6.x; `isp-hosted*` render it with a noop networking Package)
- paas (databases / applications)
- iaas (Cluster API + VMs)
- iaas (Cluster API + VMs; refused on `isp-hosted*` and `isp-slim*` — do not offer it there)
- naas (Network as a Service)

2. **Storage** (only when `system` bundle is on — `isp-hosted` skips). Cozystack standardises on **ZFS** for LINSTOR pools; see `references/storage-backends.md`. No backend question — only device selection and pool layout per node.
On a slim variant, also collect **enabledPackages**: the packages to opt in, as full Package names (`cozystack.postgres-operator`, `cozystack.postgres-application`, …), each with its whole `dependsOn` chain (see `references/variants.md`). Default empty. Rendered into `bundles.enabledPackages` in the Platform Package.

2. **Storage** (only when `system` bundle is on — `isp-hosted` and `isp-hosted-slim` skip). Cozystack standardises on **ZFS** for LINSTOR pools; see `references/storage-backends.md`. No backend question — only device selection and pool layout per node.

a. **Per-node disk selection** — for every storage-providing node, show Phase 2's unmounted-device list and ask which device(s) to use. Default is the largest unmounted disk ≥ 50 GiB. If a node has multiple candidates or the operator wants a mirror / RAID-Z, prompt for a vdev layout:

Expand All @@ -255,8 +258,8 @@ Slot legend (every slot the operator may want to edit):

Refuse to proceed if any storage node lacks ZFS tooling (`zpool` / `zfs` binaries) — the Phase 3 storage discovery would have caught it, but re-verify here in case a fix was applied since. RHEL 10 family is not supported on the storage path; see `references/known-failures.md`.

3. **podCIDR / serviceCIDR / joinCIDR** — show detected as defaults. If user picks Other, validate format (CIDR notation, no overlap with host networks the cluster sees, joinCIDR ≠ podCIDR ≠ serviceCIDR).
4. **podGateway** — auto-derive as the first IP of podCIDR, confirm.
3. **podCIDR / serviceCIDR / joinCIDR** — skip on `isp-slim` / `isp-slim-generic` (Kube-OVN only; Cilium takes pod CIDRs from `node.spec.podCIDR`). Otherwise show detected as defaults. If user picks Other, validate format (CIDR notation, no overlap with host networks the cluster sees, joinCIDR ≠ podCIDR ≠ serviceCIDR).
4. **podGateway** — skip on `isp-slim*`. Otherwise auto-derive as the first IP of podCIDR, confirm.
5. **apiServerHost** — the address Cilium / KubeOVN / cozystack-operator dial to reach kube-apiserver. The skill picks this automatically by installer variant — **do not ask the operator** unless they passed `--api-host=<ip>`.

- **`talos` variant** — `localhost:7445` (KubePrism, built into Talos machine-config). Operator can't override; Cozystack's `values-isp-full.yaml` overlay hard-codes this for Cilium.
Expand All @@ -266,7 +269,7 @@ Slot legend (every slot the operator may want to edit):

The plan presentation in Phase 5 always shows which choice landed and how to flip it.
6. **LB / external IPs** —
- Mode: `externalIPs` (recommended for now; deprecated upstream in k8s v1.36) vs `loadBalancer` (Cilium L2/BGP or external cloud LB; needs more wiring).
- Mode: `externalIPs` (recommended for now; deprecated upstream in k8s v1.36) vs `loadBalancer` (Cilium L2/BGP or external cloud LB; needs more wiring). On `isp-slim*` MetalLB is off unless `cozystack.metallb` is opted in: `loadBalancer` mode needs an admin-created `CiliumLoadBalancerIPPool` + `CiliumL2AnnouncementPolicy` (the platform enables Cilium L2 announcements on slim).
- Pool: resolved from `state.cozystack_intake.external_ips` (wizard Phase 4). The wizard already asked the operator's strategy — `internal` / `external` / `explicit` — and the reason. Here the skill **validates the strategy against live `Node.status.addresses`** and refuses to silently override:

```bash
Expand Down Expand Up @@ -355,7 +358,7 @@ Slot legend (every slot the operator may want to edit):

10. **exposedServices** (multiSelect): `api`, `dashboard`, `vm-exportproxy`, `cdi-uploadproxy`. Default `api,dashboard`.

11. **KubeOVN `MASTER_NODES`** — branch on Phase 2 finding:
11. **KubeOVN `MASTER_NODES`** — skip on `isp-slim*` and `isp-hosted*` (no Kube-OVN). Otherwise branch on Phase 2 finding:
- If CP label value matches variant expectation on at least one node → default empty (let lookup work).
- Otherwise → pre-fill comma-separated INTERNAL-IPs of CP nodes from Phase 2 and explain why the lookup would fail.

Expand Down Expand Up @@ -750,7 +753,7 @@ kubectl --context $CTX get hr --all-namespaces \
--output jsonpath='{range .items[?(@.status.conditions[?(@.type=="Ready" && @.status!="True")])]}{.metadata.namespace}/{.metadata.name} {end}'
```

On the full `system`-bundle path you may also want the root tenant's `etcd`/`monitoring`/`seaweedfs` services (this is what cozystack's own `hack/e2e-install-cozystack.bats` patches): extend the patch to `{"spec":{"ingress":true,"host":"${HOST}","monitoring":true,"etcd":true,"seaweedfs":true}}` when those were selected in Phase 4. Leave them at their defaults otherwise.
On the full `system`-bundle path you may also want the root tenant's `etcd`/`monitoring`/`seaweedfs` services (this is what cozystack's own `hack/e2e-install-cozystack.bats` patches): extend the patch to `{"spec":{"ingress":true,"host":"${HOST}","monitoring":true,"etcd":true,"seaweedfs":true}}` when those were selected in Phase 4. Leave them at their defaults otherwise. On `isp-slim*` / `isp-hosted-slim` turn on only the switches whose application chain is in `enabledPackages` (`cozystack.etcd-application`, `cozystack.monitoring-application`, `cozystack.seaweedfs-application` and their dependencies); otherwise the Tenant's release never becomes Ready.

```text
HelmRelease $NS/$NAME has been Failing for $T minutes.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,12 @@ Typical counts for orientation (not gospel):

| Variant | Approx HR count |
| ----------- | ----------- |
| `isp-full` | 40–50 |
| `isp-full-generic` | 40–50 |
| `isp-hosted` | 15–20 |
| `isp-full` | ~90, plus tenant-root releases |
| `isp-full-generic` | ~90, plus tenant-root releases |
| `isp-hosted` | ~60 on v1.7.0+, plus tenant-root releases |
| `isp-slim` | ~27 before opt-ins, plus tenant-root releases |
| `isp-slim-generic` | ~27 before opt-ins, plus tenant-root releases |
| `isp-hosted-slim` | ~18 before opt-ins, plus tenant-root releases |
| `default` | 1–3 |

## Reading a stuck HR
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ lsmod | awk '{print $1}' | grep -Ex 'overlay|br_netfilter|nf_conntrack|ip_tables

Required: `overlay`, `br_netfilter`, `nf_conntrack`, `ip_tables`, `iptable_nat`.

For Kube-OVN (non-hosted): `openvswitch`, `geneve` (autoloaded by ovs userspace — soft check).
For Kube-OVN (`isp-full*` only; not needed on hosted or slim variants): `openvswitch`, `geneve` (autoloaded by ovs userspace — soft check).

For DRBD storage (non-hosted): either `drbd` already in `lsmod` or a buildable kernel headers tree (`/usr/src/linux-headers-$(uname -r)`) so piraeus-operator can compile in-cluster. Ubuntu Secure Boot needs `drbd-dkms` pre-installed and signed — `lsmod | grep drbd` must succeed before install.

Expand Down Expand Up @@ -139,14 +139,14 @@ Skip the whole storage section on hosted variant.

### Talos cozystack-tuned image — Phase 3 gate (Talos only)

If Phase 2 detected Talos on any node, run these four checks. **All four must pass on every Talos node**, otherwise Phase 3 STOP GATE 1 fails with `cozystack:cluster-install` refusing to continue and pointing at `/cozystack:talos-bootstrap`.
If Phase 2 detected Talos on any node, run these four checks. **All four must pass on every Talos node** (on `isp-slim` the openvswitch check is dropped, there is no Kube-OVN), otherwise Phase 3 STOP GATE 1 fails with `cozystack:cluster-install` refusing to continue and pointing at `/cozystack:talos-bootstrap`.

```sh
# Kernel modules (cozystack-tuned image ships these as extensions; vanilla Talos does not)
lsmod | awk '{print $1}' | grep -Ex 'drbd|zfs|openvswitch'
```

All three names must appear in the output.
All three names must appear in the output (`drbd` and `zfs` on `isp-slim`).

```sh
# LVM filter — cozystack-tuned machine-config writes this verbatim
Expand Down Expand Up @@ -181,7 +181,8 @@ KubeOVN's chart looks nodes up by a key=value pair from `MASTER_NODES_LABEL`. **
| ----------- | ----------- |
| `isp-full` (Talos) | `node-role.kubernetes.io/control-plane=""` (empty value — Talos default) |
| `isp-full-generic` (k3s / kubeadm / RKE2) | `node-role.kubernetes.io/control-plane=true` (literal string `true`) |
| `isp-hosted` | not used — KubeOVN is not deployed |
| `isp-hosted`, `isp-hosted-slim` | not used — KubeOVN is not deployed |
| `isp-slim`, `isp-slim-generic` | not used — networking is Cilium alone |
| `default` | not used unless the operator hand-rolls KubeOVN |

The lookup compares the value byte-for-byte. If it doesn't match exactly, the chart `fail`s with:
Expand Down Expand Up @@ -214,7 +215,7 @@ If the value does **not** match what the variant expects, surface two recovery p

2. **Pin explicit IPs via `MASTER_NODES`** — bypasses the lookup entirely. Collect the CP nodes' internal IPs (`kubectl get nodes -o wide`, INTERNAL-IP column) and set `networking.kubeovn.MASTER_NODES` to the comma-separated list in the values collected in Phase 4. This is the safest choice for kubeadm.

Gate: do not apply the Platform Package until **either** the label matches the variant's expected value on at least one node, **or** `MASTER_NODES` is set explicitly in the collected values. Skip this gate on `isp-hosted`.
Gate: do not apply the Platform Package until **either** the label matches the variant's expected value on at least one node, **or** `MASTER_NODES` is set explicitly in the collected values. Skip this gate on `isp-hosted`, `isp-hosted-slim`, `isp-slim` and `isp-slim-generic`.

### CNI conflict (cluster-wide)

Expand Down
Loading
Loading