Add external minion onboarding script and runbook for salt-minion-vcf

salt-minion-vcf/.gitignore

@@ -3,6 +3,8 @@
 .DS_Store
 dist/
 build/
+__pycache__/
+*.pyc
 
 # Real pillar data (credentials) - only *.sls.example templates are tracked.
 pillar/*.sls

salt-minion-vcf/docs/runbook.md

@@ -0,0 +1,228 @@
+# Runbook: Onboarding an External Minion and Connecting It to VCF Infrastructure
+
+This runbook covers two separate procedures:
+
+1. **Bring up a `salt-minion-vcf` instance and trust it against a VCF
+   Operations-managed Salt master** (Part 1) - using
+   [`scripts/onboarding/vcf-ops-onboard.py`](../scripts/onboarding/vcf-ops-onboard.py).
+2. **Give that minion the credentials it needs to actually operate against
+   VCF components** (vCenter, NSX, SDDC Manager, ESXi, VCFA, VCF Installer,
+   VCF Operations) via Salt Pillar (Part 2).
+
+These are independent: a minion can be connected to the master (Part 1)
+before it has any pillar data configured (Part 2) - it just can't run any
+`saltext.vcf` operations against a real target until Part 2 is done.
+
+---
+
+## Part 1 - Bring up the minion and connect it to the Salt master
+
+### Prerequisites
+
+- The `salt-minion-vcf` image built locally or available in a registry you
+  can pull from (`docker build -t salt-minion-vcf:0.1.0 .` from the repo
+  root - see the top-level [`README.md`](../README.md#quick-start) if this
+  hasn't been done yet).
+- `docker` on PATH (Docker mode), or `helm` + `kubectl` on PATH (Kubernetes mode).
+- Network access from wherever you run the script to your VCF Operations
+  instance's Suite API, and from the minion's host/cluster to the Salt
+  master (`SALT_MASTER_PORT`/`4506`, `SALT_PUBLISH_PORT`/`4505`).
+- Credentials for a VCF Operations user with the Salt Management view/manage
+  privileges, and the resource UUID of the VCF instance whose master you
+  want to attach to.
+
+### Procedure
+
+Run the onboarding script:
+
+```bash
+python3 scripts/onboarding/vcf-ops-onboard.py \
+  --ops-host vcfops.example.com \
+  --ops-user admin \
+  --vcf-instance-id <vcf-instance-resource-id> \
+  --deployment docker    # or: kubernetes
+```
+
+Everything not passed as a flag is prompted for interactively, with a
+review/confirm summary shown before anything is actually started. The script
+handles, in order:
+
+1. Logs in to VCF Operations.
+2. Resolves the Salt master governing the given VCF instance.
+3. Computes the master's identity fingerprint (`master_finger`).
+4. Starts the minion (`docker run`, or `helm upgrade --install`), passing it
+   the master FQDN, `master_finger`, and a freshly generated minion ID. The
+   minion generates its own RSA keypair locally on first start - the
+   private key never leaves it, and VCF Operations credentials never reach it.
+5. Reads back the minion's public key.
+6. Registers that key as trusted with the master.
+7. Waits until the master has actually accepted the connection.
+
+Use `--dry-run` first if you want to preview every command and API call
+without executing anything. See `--help` for the full flag list, or
+[`scripts/onboarding/README.md`](../scripts/onboarding/README.md) for a
+complete walkthrough of every option.
+
+### Verification
+
+From the Salt master:
+
+```bash
+salt-key -L                    # minion should be under "Accepted Keys"
+salt '<minion-id>' test.ping   # should return True
+```
+
+From the minion side (Docker):
+
+```bash
+docker exec salt-minion-vcf salt-call --local test.version
+docker logs salt-minion-vcf | grep "Minion is ready to receive requests"
+```
+
+(Kubernetes: substitute `kubectl exec -n <namespace> <pod> --` /
+`kubectl logs -n <namespace> <pod>`.)
+
+### Troubleshooting
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `CERTIFICATE_VERIFY_FAILED: self signed certificate` on login | VCF Operations uses a self-signed/internal CA cert | Pass `--insecure` |
+| `pull access denied for salt-minion-vcf` | Image not built locally yet - Docker tried to pull it from Docker Hub | `docker build -t salt-minion-vcf:0.1.0 .` from the repo root first, or point `--image` at wherever you built/pushed it |
+| `container name already in use` on retry | A previous failed attempt left a stopped container behind | The script now detects this and offers to remove it automatically |
+| `[CRITICAL] Unable to securely set the permissions of "/etc/salt/pki/minion"` / `PermissionError: Permission denied: '/etc/salt/pki/minion/tmp...'` | The PKI volume value was a host path (bind mount), not a named Docker volume - the container runs as non-root uid `10000`, and a bind-mounted host directory doesn't inherit the image's baked-in ownership | Use a plain volume name (e.g. `salt-minion-vcf-pki`, the default) instead of an absolute path. If you specifically need a host path, `chown -R 10000:10000` it first |
+| Minion key is accepted on the master (`salt-key -L` shows it), but the onboarding script (or the image's own `HEALTHCHECK`/`readinessProbe`) never reports it connected | `status.master`'s answer depends on `master_alive_interval` being configured on the minion, which the entrypoint doesn't set by default - it can under-report even once genuinely connected | The onboarding script also checks the minion's logs for `Minion is ready to receive requests` as a fallback, which doesn't have this gap. If you're checking manually, use that log line or `salt '<minion-id>' test.ping` from the master instead of relying on `status.master` alone |
+
+---
+
+## Part 2 - Pillar data for connecting to VCF components
+
+`saltext.vcf` reads all target credentials from Salt Pillar under
+`saltext.vcf.<target>`. There is **no way to pass these credentials through
+the onboarding script or through `SALT_MASTER`/`SALT_MINION_ID`-style
+environment variables** - they must be supplied as pillar data, by design
+(see [`docs/security.md`](security.md)).
+
+### Supported targets
+
+| Target key | Component | Example file |
+|---|---|---|
+| `vcenter` | vCenter Server (REST + SOAP/pyVmomi) | [`pillar/vcenter.sls.example`](../pillar/vcenter.sls.example) |
+| `nsx` | NSX Manager (Policy API) | [`pillar/nsx.sls.example`](../pillar/nsx.sls.example) |
+| `sddc_manager` | SDDC Manager | [`pillar/sddc_manager.sls.example`](../pillar/sddc_manager.sls.example) |
+| `esxi` | Standalone/unmanaged ESXi hosts only - a host already joined to vCenter uses the `vcenter` block instead (its REST session API is blocked once managed) | [`pillar/esxi.sls.example`](../pillar/esxi.sls.example) |
+| `vcfa` | VCF Automation (Aria Automation) | [`pillar/vcfa.sls.example`](../pillar/vcfa.sls.example) |
+| `vcf_installer` | VCF Installer (Day-0 bringup, formerly Cloud Builder) | [`pillar/vcf_installer.sls.example`](../pillar/vcf_installer.sls.example) |
+| `vcf_ops` | VCF Operations (Suite API) | [`pillar/vcf_ops.sls.example`](../pillar/vcf_ops.sls.example) |
+
+Each file follows the same shape - copy it, rename it (drop `.example`), and
+fill in real values:
+
+```yaml
+saltext.vcf:
+  vcenter:
+    host: mgmt-vc.example.test
+    username: administrator@vsphere.local
+    password: secret
+    verify_ssl: false
+```
+
+**Never commit the real `*.sls` files** - only `*.sls.example` is tracked;
+the rest are gitignored.
+
+### Which path applies depends on how you'll run VCF operations
+
+This is the detail most likely to cause confusion - pick the path that
+matches how you intend to trigger `saltext.vcf` calls against this minion.
+
+#### Path 1 - Locally inside the container (`salt-call --local`)
+
+Use this if scripts inside the container/Pod call `saltext.vcf` directly, or
+for ad-hoc testing. The minion always has `pillar_roots` pointed at its own
+local pillar directory; `top.sls` is auto-generated to match `'*'` against
+whatever `*.sls` files are present.
+
+**Docker** - bind-mount the directory at container start:
+
+```bash
+docker run -d --name salt-minion-vcf \
+  -e SALT_MASTER=<host> \
+  -v salt-minion-vcf-pki:/etc/salt/pki/minion \
+  -v "$(pwd)/pillar:/etc/salt/pillar" \
+  salt-minion-vcf:0.1.0
+```
+
+Or push files into an already-running container (no restart needed -
+`salt-call --local` recompiles pillar from disk on every call):
+
+```bash
+./scripts/pillar-push.sh salt-minion-vcf pillar/vcenter.sls
+```
+
+**Kubernetes** - create a Secret containing your `*.sls` files plus a
+`top.sls` matching `'*'` (a Pod only ever runs one minion ID, so a wildcard
+is always sufficient here):
+
+```bash
+cat > top.sls <<'EOF'
+base:
+  '*':
+    - vcenter
+EOF
+kubectl create secret generic salt-minion-vcf-pillar \
+  --from-file=top.sls \
+  --from-file=vcenter.sls=pillar/vcenter.sls
+helm upgrade --install vcf-executor ./helm/salt-minion-vcf \
+  --set salt.master=<host> \
+  --set pillar.secretName=salt-minion-vcf-pillar
+```
+
+To update without restarting the Pod, update the Secret object itself -
+kubelet re-syncs the mounted volume automatically (typically within ~60-90s).
+
+**Verify:**
+
+```bash
+docker exec salt-minion-vcf salt-call --local pillar.items
+docker exec salt-minion-vcf salt-call --local vcf_vcenter_vm.list_
+```
+
+#### Path 2 - Dispatched from the Salt Master (`salt '<minion-id>' ...`)
+
+This is the intended production model: VCF Operations/RaaS dispatches jobs
+to the minion from the master. **Jobs run this way are compiled using the
+Master's own `pillar_roots` - anything mounted into this container (Path 1)
+is invisible to them.** The customer's Salt master admin needs pillar data
+on the master side (e.g. `/srv/pillar`), targeted by this minion's ID - see
+[`pillar/master-top.sls.example`](../pillar/master-top.sls.example):
+
+```yaml
+# /srv/pillar/top.sls on the customer's Salt Master
+base:
+  '<minion-id>':
+    - vcenter
+```
+
+using the identical `saltext.vcf.<target>` structure as the `pillar/*.sls.example`
+files in this repo. This is outside this repo's control (it's the master
+admin's own `pillar_roots`); for production, prefer an `ext_pillar` backed by
+a secrets manager (e.g. Vault) over plain files in `/srv/pillar`.
+
+**Verify (run from the master, not `salt-call --local`):**
+
+```bash
+salt '<minion-id>' pillar.items
+salt '<minion-id>' test.ping
+```
+
+If you need both models at once (local ad-hoc testing *and* master-dispatched
+production jobs), configure Path 1 and Path 2 independently with the same
+values - they don't conflict, since each is scoped to a different pillar_roots.
+
+### Security reminders
+
+See [`docs/security.md`](security.md) for the full list. The two most
+relevant here:
+
+- Never put VCF credentials in a Kubernetes ConfigMap - use a Secret.
+- Only `*.sls.example` files are tracked in git; never force-add or commit
+  a real `*.sls` file.

salt-minion-vcf/scripts/onboarding/README.md

@@ -0,0 +1,115 @@
+# VCF Operations Onboarding Script
+
+`vcf-ops-onboard.py` is an interactive tool that brings up a `salt-minion-vcf`
+instance (Docker **or** Kubernetes/Helm) and registers it as a trusted minion
+against a Salt master managed by VMware VCF Operations - without the
+minion's private key ever leaving the minion, and without VCF Operations
+credentials ever reaching the minion itself.
+
+## What it does
+
+```text
+1. Log in to VCF Operations
+2. Resolve the Salt master for a given VCF instance
+3. Compute the master's identity fingerprint (master_finger)
+4. Start the minion (docker run, or helm upgrade --install), with a freshly
+   generated minion ID - the minion generates its own RSA keypair locally
+5. Read back the minion's public key (never the private key)
+6. Trust that key with the master
+7. Poll until the master accepts the connection
+```
+
+This mirrors the manual flow documented in the top-level
+[`README.md`](../../README.md#salt-master-registration), just automated and
+without a human needing to run `salt-key -a` by hand - trust is established
+via the VCF Operations API instead.
+
+Steps 4-7 can be repeated for multiple minions in one session without
+re-entering VCF Operations credentials or re-resolving the master.
+
+## Interactive features
+
+- **Input validation**: the VCF instance ID is checked against a UUID format
+  and re-prompted if invalid; deployment type is a numbered menu, not free text.
+- **Review before acting**: a summary of every setting (minion ID, image,
+  container/release name, target master, ...) is shown before the minion is
+  started, and again before its key is trusted - nothing consequential runs
+  without an explicit confirmation.
+- **Live progress**: waiting for the minion to generate its keypair and for
+  the master to accept the connection shows an animated spinner with a
+  countdown (falls back to periodic plain-text lines if output isn't a TTY,
+  e.g. when redirected to a file).
+- **Onboard multiple minions in one session**: after each successful
+  onboarding you're asked whether to onboard another against the same
+  master - container/volume/release names are auto-suggested with a `-2`,
+  `-3`, ... suffix so they don't collide with the previous minion.
+- **`-y`/`--yes`** skips all confirmations for scripted/CI use, and
+  **`--dry-run`** previews every command and API call without executing
+  anything.
+
+## Logging
+
+Every run writes a full step-by-step audit log to
+`vcf-ops-onboard-<timestamp>.log` in the current directory (override the path
+with `--log-file`). It captures every prompt, shell command, and API call/
+response - passwords and auth tokens are never written to it. Pass
+`-v`/`--verbose` to also mirror that detail live on the console.
+
+## Requirements
+
+- Python 3.8+ (standard library only - no `pip install` needed)
+- `docker` on PATH (Docker mode), or `helm` + `kubectl` on PATH (Kubernetes mode)
+- Network access from wherever you run this script to your VCF Operations
+  instance's Suite API
+
+## Usage
+
+Fully interactive - just run it and answer the prompts:
+
+```bash
+python3 scripts/onboarding/vcf-ops-onboard.py
+```
+
+Or supply anything up front via flags (anything omitted is still prompted for):
+
+```bash
+# Docker
+python3 scripts/onboarding/vcf-ops-onboard.py \
+  --ops-host vcfops.example.com \
+  --ops-user admin \
+  --vcf-instance-id <vcf-instance-resource-id> \
+  --deployment docker \
+  --image salt-minion-vcf:0.1.0
+
+# Kubernetes / Helm (run from the salt-minion-vcf repo root, so
+# --chart-path's default of ./helm/salt-minion-vcf resolves correctly)
+python3 scripts/onboarding/vcf-ops-onboard.py \
+  --ops-host vcfops.example.com \
+  --ops-user admin \
+  --vcf-instance-id <vcf-instance-resource-id> \
+  --deployment kubernetes \
+  --namespace vcf-salt \
+  --release-name vcf-executor
+```
+
+See `--help` for the full flag list (container/release naming, image
+repository/tag, connect timeout, `--log-file`/`-v` for audit logging,
+`--dry-run` to preview every command and API call without executing
+anything, `-y` to skip confirmation prompts).
+
+## Things to validate in your own environment
+
+- **VCF Operations auth flow**: the script logs in via
+  `POST /suite-api/api/auth/token/acquire` and sends
+  `Authorization: OpsToken <token>` on subsequent calls - the same pattern
+  used by other existing tooling against this backend. If your deployment
+  fronts VCF Operations with SSO/CSP instead, adjust `OpsClient.login()`.
+- **`master_finger` algorithm**: defaults to `sha256` (matches the Salt
+  version this image bundles). Override with `--master-finger-algo md5` if
+  your Salt master needs the legacy default.
+
+## Known limitation
+
+There is currently no API to *revoke* a trusted key (deregistration), so this
+script only covers onboarding. To remove a minion, use your master's own
+key-management tooling directly for now.