feat(etcd): add etcd snapshot backups to S3

[jamie-stinson]

Adds topf etcd backup create and topf etcd backup list for backing up etcd
to S3-compatible storage (AWS, MinIO, Ceph, R2), configured in topf.yaml:

backups:
  s3:
    endpoint: minio.example.com:9000
    bucket: talos-backups

The block is nested under backups so other storage backends can be added
later. Credentials can be set inline (SOPS-encryptable like other fields) or
come from the usual chain (env vars, shared credentials file, IAM etc...).

create streams a snapshot from the first reachable control-plane node
(respects --nodes-filter, skips maintenance mode), verifies it offline before
uploading, then uploads the snapshot and a small .meta.yaml manifest with size,
checksum, etcd revision and key count.

I would also like to create a PR at some point for restore functionality.

[clementnuss]

hi @jamie-stinson

thanks for your contribution, but we will not be merging this PR. we didn't have any contribution guidelines so far (and sorry for that, you couldn't know), but we have now (#122), where we explain that topf aims to remain minimal (and therefore maintainable), and we will not re-implement talosctl commands if topf doesn't bring substantial added value (like gathering config files, automating commands for multiple nodes, doing health checks, etc).

an etcd snapshot is something requiring only few bash commands to be completed, and the backup towards S3 is better handled by a software that's specifically done for that (e.g. s5cmd, my favorite, or rclone, aws cli, etc) than including go libraries in topf.

even better the snapshot should be backed up properly (and encrypted) with kopia (but that's a user choice).

here are a few lines which do just that:

export CP_NODE=$(topf nodes -o yaml | yq '.[] | select(.node.role == "control-plane") | .node.host' | head -1)
topf talosconfig > ./talosconfig && export TALOSCONFIG=$(pwd)/talosconfig;

talosctl etcd snapshot -n $CP_NODE /tmp/etcd-snapshot.db
kopia snapshot create /tmp/etcd-snapshot.db

I hope you see my point: it's a bit like the UNIX philosophy, each binary does one thing (well) and you combine them together

---

[EDIT]: for the restore, what's needed is something like

topf apply --auto-bootstrap=false
talosctl bootstrap -n $CP_NODE --recover-from <path/to/backup.db>

[jamie-stinson]

Hey @clementnuss, no problem understood. For some extra context the reason I built this is I found that talosctl only gets you part of the way there as you have shown. You still have to work out which control plane node is actually
reachable, confirm the snapshot you just took is even restorable/valid, encrypt it (optionally) and then find a way to ship it to some form of remote storage like s3 and then theres no easy way to list + prune snapshots you've created

But I appreciate if this is not the direction the project wants to go, thanks for taking a look :)