Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
114 changes: 114 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,37 @@ jobs:
- name: Unit tests
run: go test -race -count=1 ./...

- name: Portable check
run: |
go build -o bin/k3helper ./cmd/k3helper
scripts/portable-check.sh ./bin/k3helper

# macOS is a first-class target — it is what most operators drive a cluster
# from — and nothing else in CI runs there. Darwin differs where it matters
# for this binary: its own temp directory layout, a different libc for the
# cgo-free build, and BSD userland for anything shelled out to.
macos:
name: macOS build + tests
runs-on: macos-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-go@v5
with:
go-version: ${{ env.GO_VERSION }}
cache: true

- name: go vet
run: go vet ./...

- name: Unit tests
run: go test -race -count=1 ./...

- name: Portable check
run: |
go build -o bin/k3helper ./cmd/k3helper
scripts/portable-check.sh ./bin/k3helper

build:
name: Cross-compile
runs-on: ubuntu-latest
Expand All @@ -57,12 +88,95 @@ jobs:
- name: Build all platforms
run: make build-all

# The Windows assets must carry .exe or Windows will not execute them,
# and that suffix is applied in a shell loop rather than by the Go
# toolchain — so assert it rather than trusting the loop.
- name: Windows binaries carry .exe and are PE executables
run: |
for arch in amd64 arm64; do
f="bin/k3helper-windows-$arch.exe"
[ -f "$f" ] || { echo "::error::$f was not built"; exit 1; }
case "$(file -b "$f")" in
*"PE32"*|*"MS Windows"*) echo "✓ $f" ;;
*) echo "::error::$f is not a Windows executable: $(file -b "$f")"; exit 1 ;;
esac
done

- uses: actions/upload-artifact@v4
with:
name: k3helper-binaries
path: bin/
retention-days: 7

# Windows is not just another cross-compile target here. The kubeconfig
# transport exists partly because Windows has no /bin/sh: it runs kubectl by
# argument rather than through a shell, and it stages manifests in the OS
# temp directory rather than /tmp. None of that is exercised by building on
# Linux — it has to actually run.
windows:
name: Windows build + tests
runs-on: windows-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-go@v5
with:
go-version: ${{ env.GO_VERSION }}
cache: true

- name: go vet
run: go vet ./...

# -race needs a C toolchain; windows-latest has one, but the point of
# this job is the platform-specific paths rather than the race detector,
# which the Linux job already covers.
- name: Unit tests
run: go test -count=1 ./...

- name: Build the .exe
run: go build -o k3helper.exe ./cmd/k3helper

- name: The .exe actually runs
run: |
.\k3helper.exe version
.\k3helper.exe --help

# The same portable check Linux and macOS run, so the three platforms are
# held to one standard rather than to whatever each job happened to test.
- name: Portable check
shell: bash
run: scripts/portable-check.sh ./k3helper.exe

# The portable check above runs under the runner's Git Bash. This one
# deliberately does not: it drives the .exe from PowerShell, with native
# Windows paths and no POSIX shell anywhere in the picture, which is the
# environment a client actually double-clicks into. Native paths are also
# where a path check tends to quietly pass while being wrong.
- name: Runs from a bare directory under PowerShell
shell: pwsh
run: |
New-Item -ItemType Directory -Force -Path C:\portable | Out-Null
Copy-Item k3helper.exe C:\portable\
Set-Location C:\portable

& .\k3helper.exe init --kubeconfig C:\portable\missing.yaml --cluster demo
if ($LASTEXITCODE -eq 0) { throw "init accepted a kubeconfig that does not exist" }
if (Test-Path targets.yaml) { throw "init wrote a targets file after failing" }

"apiVersion: v1" | Out-File -Encoding ascii C:\portable\kube.yaml
& .\k3helper.exe init --kubeconfig C:\portable\kube.yaml --cluster demo
if ($LASTEXITCODE -ne 0) { throw "init failed on a valid kubeconfig with a native path" }
Get-Content targets.yaml

& .\k3helper.exe ctx
if ($LASTEXITCODE -ne 0) { throw "ctx failed" }

- uses: actions/upload-artifact@v4
with:
name: k3helper-windows-exe
path: k3helper.exe
retention-days: 7

# Sandbox E2E on the container driver — three privileged systemd containers
# standing in for three Linux hosts, which is what a GitHub-hosted runner can
# actually provide.
Expand Down
20 changes: 20 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,26 @@ jobs:
*) echo "::error::binary reports '$got', expected ${{ steps.tag.outputs.version }}"; exit 1 ;;
esac

# Every asset install.sh can ask for has to be in the release, and the
# Windows ones have to carry .exe or Windows will not execute them. A
# missing or misnamed asset only surfaces when somebody's download 404s.
- name: Every platform asset is present and listed in checksums.txt
run: |
for asset in \
k3helper-linux-amd64 k3helper-linux-arm64 \
k3helper-darwin-amd64 k3helper-darwin-arm64 \
k3helper-windows-amd64.exe k3helper-windows-arm64.exe
do
[ -f "dist/$asset" ] || { echo "::error::dist/$asset missing"; exit 1; }
grep -q " $asset\$" dist/checksums.txt \
|| { echo "::error::$asset not listed in checksums.txt"; exit 1; }
echo "✓ $asset"
done
case "$(file -b dist/k3helper-windows-amd64.exe)" in
*"PE32"*|*"MS Windows"*) ;;
*) echo "::error::the Windows asset is not a Windows executable"; exit 1 ;;
esac

# Pull the section for this version out of CHANGELOG.md. Falling back to
# generated notes would quietly ship a commit list as the release notes,
# so a missing section is an error instead.
Expand Down
77 changes: 77 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,83 @@ Notable changes per release. The release workflow publishes the section
matching the tag it is building, so this file is the source of the release
notes on GitHub.

## v0.5.0

Clusters you cannot SSH into. k3helper now reaches a cluster either by its
nodes or by a kubeconfig, which is what a managed cluster (EKS, GKE, AKS,
Rancher, anything else that hands you credentials and keeps the machines)
actually gives you.

### Install

```bash
curl -fsSL https://raw.githubusercontent.com/solutionforest/k3helper/main/install.sh | sh
```

No breaking changes from v0.4.0. Existing targets files keep working exactly
as they did.

### Added

- **`kubeconfig:` clusters.** A targets file entry with `kubeconfig:` (and an
optional `kube_context:`) instead of `nodes:` drives a local `kubectl`
against the API server. `k3helper init --kubeconfig ~/.kube/config
--kube-context prod` writes one.
- **Everything that works through the API server works there**: `doctor`'s
cluster and workload signatures, `deploy`, `verify --dry-run-server`, `gen`,
and the TUI including logs, describe and port-forward.
- **`k3helper ctx` gained a REACHED column**, so `0 nodes` reads as "reached by
kubeconfig" rather than as a broken file. One file can mix both kinds.
- **Windows binaries, released.** `k3helper-windows-amd64.exe` and
`k3helper-windows-arm64.exe` are built by `make release`, listed in
`checksums.txt`, and published with every tag. One file, no installer,
nothing written outside the folder you put it in. The kubeconfig transport
runs kubectl by argument rather than through a shell, so it needs no
`/bin/sh`.
- **`make portable-check`**, and CI runs it on Linux, macOS and Windows.
"Single portable binary" is a claim about the runtime rather than the build,
so it is checked by running the binary from a directory it has never seen:
it must start, explain a missing targets file, refuse a kubeconfig that is
not there without writing anything, complete the kubeconfig and SSH init
flows, generate and verify YAML with no cluster and no network, and leave
nothing behind in `$HOME`.
- **A macOS CI job.** macOS is what most operators drive a cluster from and
nothing in CI ran there before; it now gets vet, race tests and the portable
check on every push, the same as Linux.

### Fixed

- **`doctor` now catches a crash loop it happens to sample mid-restart.** A
container in backoff only reads as CrashLoopBackOff while it is waiting
between attempts; the moment it starts again the pod is Running with no
reason attached, and the diagnosis saw nothing worse than an unready pod.
Restart counts were already parsed and thrown away — they are now kept, and a
pod that is not ready after three or more restarts is called a crash loop
whichever half of the cycle we caught. This is why the fault matrix could
catch that fault or miss it depending on timing.

### Changed

- **`doctor` separates scope notes from faults.** A finding that describes what
could *not* be looked at — incomplete evidence, or a missing host layer — is
printed under "Notes on what was looked at" and no longer sets the exit code.
A healthy managed cluster exits 0 instead of 2.
- **`check` and the TUI build their host check list from one place**
(`check.HostChecks`), so a check added to one cannot go missing from the
other.

### Not available on a kubeconfig cluster

By nature, not by omission — a kubeconfig reaches the API server, not the
machines behind it:

- Host-layer checks: disk, memory, swap, cgroups, systemd units, container
runtime, clock skew. `doctor` reports the gap rather than implying those are
fine; `check` says it has nothing to check.
- `vm setup` and `registry apply`. Both write files on the nodes, and both
refuse with an explanation rather than doing half the job.
- Certificate expiry, which reads `k3s certificate check` on the node.

## v0.4.0

Private registries: declared once, applied to every node, and diagnosed
Expand Down
33 changes: 23 additions & 10 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
BINARY := k3helper
VERSION := 0.4.0
VERSION := 0.5.0
VERPKG := github.com/solutionforest/k3helper/internal/cli.version
LDFLAGS := -ldflags "-X $(VERPKG)=$(VERSION)"
# -s -w strips the symbol table and DWARF: ~25% smaller downloads, and Go
# panics keep their function names because the runtime carries its own tables.
RELFLAGS := -ldflags "-s -w -X $(VERPKG)=$(VERSION)"
PLATFORMS := linux/amd64 linux/arm64 darwin/arm64 darwin/amd64
PLATFORMS := linux/amd64 linux/arm64 darwin/arm64 darwin/amd64 windows/amd64 windows/arm64
# Windows will not execute a downloaded file without the extension, so those
# two assets carry .exe. Everything else stays extensionless; install.sh builds
# the asset name the same way.
DIST := dist
TARGETS := test/sandbox/targets.sandbox.yaml
SSH_KEY := test/sandbox/ssh/id_ed25519
Expand All @@ -14,7 +17,7 @@ SSH_OPTS := -i $(SSH_KEY) -o StrictHostKeyChecking=no -o UserKnownHostsFile=/d
# Sandbox nodes are OrbStack Linux VMs (NOT Docker containers — OrbStack
# containers share the macOS kernel and kubelet PLEG kills pods falsely).

.PHONY: help build build-all test test-integration clean e2e e2e-fast e2e-quick \
.PHONY: help build build-all test test-integration portable-check clean e2e e2e-fast e2e-quick \
sandbox-up sandbox-down sandbox-reset sandbox-verify sandbox-ssh \
bootstrap check doctor fault-clean fault-list fault-check-all release release-upload bundle

Expand Down Expand Up @@ -42,21 +45,30 @@ help:
@echo " make fault-list list faults + the signature each should trigger"
@echo " make fault-<name> inject one fault (e.g. make fault-oom)"
@echo " make fault-check-all inject every fault, assert doctor catches each"
@echo " make portable-check run the built binary from a bare directory"
@echo " make fault-clean undo injected faults"

build:
go build $(LDFLAGS) -o bin/$(BINARY) ./cmd/k3helper

build-all:
@for platform in linux/amd64 linux/arm64 darwin/arm64 darwin/amd64; do \
GOOS=$${platform%/*} GOARCH=$${platform#*/} go build $(LDFLAGS) \
-o bin/$(BINARY)-$${platform%/*}-$${platform#*/} ./cmd/k3helper; \
echo "✓ bin/$(BINARY)-$${platform%/*}-$${platform#*/}"; \
@for platform in $(PLATFORMS); do \
os=$${platform%/*}; arch=$${platform#*/}; ext=""; \
[ "$$os" = windows ] && ext=".exe"; \
GOOS=$$os GOARCH=$$arch go build $(LDFLAGS) \
-o bin/$(BINARY)-$$os-$$arch$$ext ./cmd/k3helper || exit 1; \
echo "✓ bin/$(BINARY)-$$os-$$arch$$ext"; \
done

test:
go test ./...

# "Single portable binary" is a runtime claim, so it is checked by running the
# binary from a directory it has never seen. CI runs this on Linux, macOS and
# Windows against the same script.
portable-check: build
@scripts/portable-check.sh ./bin/$(BINARY)

test-integration:
go test -tags=integration ./...

Expand All @@ -79,9 +91,10 @@ e2e-quick:
release:
@rm -rf $(DIST) && mkdir -p $(DIST)
@for platform in $(PLATFORMS); do \
os=$${platform%/*}; arch=$${platform#*/}; \
GOOS=$$os GOARCH=$$arch go build $(RELFLAGS) -o $(DIST)/$(BINARY)-$$os-$$arch ./cmd/k3helper || exit 1; \
echo "✓ $(DIST)/$(BINARY)-$$os-$$arch"; \
os=$${platform%/*}; arch=$${platform#*/}; ext=""; \
[ "$$os" = windows ] && ext=".exe"; \
GOOS=$$os GOARCH=$$arch go build $(RELFLAGS) -o $(DIST)/$(BINARY)-$$os-$$arch$$ext ./cmd/k3helper || exit 1; \
echo "✓ $(DIST)/$(BINARY)-$$os-$$arch$$ext"; \
done
@cd $(DIST) && (command -v sha256sum >/dev/null && sha256sum $(BINARY)-* || shasum -a 256 $(BINARY)-*) > checksums.txt
@cp install.sh $(DIST)/install.sh
Expand Down
40 changes: 40 additions & 0 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -395,10 +395,50 @@ scripts/bundle.sh
11. **Linux CI sandbox** — the container driver runs the full E2E. ✅
12. **Private registries** — targets-file declaration, `registry apply`,
pull-secret generation, node-side check, three diagnosis signatures. ✅
13. **Kubeconfig transport** — clusters reached through their API server
instead of by SSH, so managed Kubernetes is in scope. ✅

Remaining: nothing from the original scope. Open items are new asks, tracked
under "Open questions" below.

### 8. Kubeconfig transport (#13)

**Why.** Every layer below the cluster assumed SSH into a node and a kubectl
living there. That covers k3s and kubeadm and excludes every managed cluster,
which is what most clients run: EKS/GKE/AKS hand out a kubeconfig and keep the
machines. The cluster layer never needed the machines — only the transport did.

**Shape.** A `Targets` entry carries either `nodes:` or `kubeconfig:`, and
`Targets.Mode()` says which. `internal/transport` is the one place that opens
connections; `internal/kubectl` is the local transport.

**Why the local transport parses instead of shelling out.** The codebase builds
shell command strings. `internal/ssh/local.go` runs them through `/bin/sh`,
which Windows does not have — and the portable `.exe` is a deliverable. So
`kubectl.Local.Run` asserts the command is one of its own kubectl invocations,
strips the trailing redirection, splits the arguments honouring the single
quoting `kube.shellQuote` produces, and execs kubectl directly. Anything else —
`df`, `systemctl`, `free`, the k3s probes — is refused with exit 127. That
refusal is the feature: host probes fail where they are written, with a reason,
instead of somewhere further down.

**Layers.** `check.HostChecks` is the single host-check list both `check` and
the TUI build from. `Evidence.HostLayerUnavailable` is deliberately neither a
`ProbeError` nor an `UnreachableNode`: both of those mean "we tried and failed",
which is a fault, while this means "there was never anything to try", which is
how managed clusters work. Conflating them reports every healthy EKS cluster as
degraded. `Diagnosis.Informational()` keeps those notes out of the exit code.

**Refusals.** `vm setup` and `registry apply` need machines. They stop with an
explanation rather than doing half the job — a registry configured on some
nodes and not others is the exact failure the cluster-level registry block
exists to prevent.

**Proof.** Unit tests parse the real command strings from `gather.go` and
`deploy.go`; `exec_test.go` runs the whole path against a stub kubectl on PATH;
the integration suite reads the sandbox twice, once over SSH and once through
its kubeconfig, and asserts the two agree.

## Research — similar tools

| Tool | What it does | Covers our reqs | Gap |
Expand Down
Loading
Loading