make operator own AppRole secret

[travisbcotton]

Description

When authMethod: appRole, the operator previously owned only half of the
Vault AppRole credential. It created the Vault policy, the AppRole, and threaded
the RoleID into the VSO VaultAuth CR — but it never created the Kubernetes
Secret (named by spec.platform.vault.appRoleSecretRef) that VSO reads the
SecretID from.
That split ownership left a bootstrap/lifecycle gap:

• Fresh installs hung. VSO logins returned 403 permission denied, so no
  VaultStaticSecret ever materialized and the control plane stalled with
  BucketReady=False / DatabaseReady=False.
• Namespace delete/recreate broke. The Vault-side AppRole survived while the
  Kubernetes Secret was lost, requiring a manual vault write .../secret-id +
  kubectl create secret workaround.
  This PR makes the operator own the complete AppRole credential lifecycle so
  fresh installs and namespace recreation are deterministic without any
  out-of-band steps.

What changed

Operator now provisions the AppRole SecretID

internal/reconcilers/vault.go — a new ensureAppRoleSecretID step runs in the
appRole branch of the Vault reconciler, immediately after EnsureAppRole:

1. Generates a secret_id via the existing vault.Client.GenerateSecretID.
2. Materializes the Kubernetes Secret VSO reads, storing the SecretID under the
   key id (VSO's required key name) in the per-cluster namespace.
3. Uses server-side apply under the operator's field manager, consistent with
   the other VSO resources.
   The RoleID continues to be supplied directly in VaultAuth.spec.appRole.roleId,
   so this Secret carries only the SecretID — a single, deterministic ownership
   model.

RoleID-bound regeneration (self-healing, no churn)

The stored SecretID is bound to the RoleID it was minted against via a new
annotation, openchami.org/vault-approle-role-id. The regeneration decision is
three-way:

Secret state	Action
Absent, or empty/missing id	Generate
Non-empty id bound to the current RoleID	Preserve (no VSO churn)
Non-empty id bound to a different RoleID, or no binding annotation	Regenerate + rebind

• A valid SecretID minted against the current RoleID is preserved across
  reconciles — the operator never rotates a working credential out from under
  VSO. SecretIDTTL="0" means it stays valid indefinitely.
• The "K8s Secret survives, but Vault/AppRole was recreated → new RoleID → stale
  403" case now self-heals on the next reconcile.
• A hand-created / pre-upgrade Secret with no binding annotation is treated as
  unbound and adopted: the operator mints a SecretID of known provenance and
  stamps the annotation.

Not covered: a manually revoked SecretID with an unchanged RoleID. That
would require active login-probing and is intentionally left as a separate
concern.

Ownership / cleanup of the operator-created Secret

• The Secret now carries the standard managed labels
  (openchami.org/managed-by=operator, app.kubernetes.io/name=vault-approle),
  matching the tokensmith_bootstrap.go convention.
• No owner reference is set (by design). The OpenCHAMIControlPlane is
  namespaced and typically lives in a different namespace than the
  openchami-<cluster> control-plane namespace, and Kubernetes forbids
  cross-namespace owner references. Cleanup already happens correctly: on CP
  deletion the controller tears down the entire openchami-<cluster> namespace,
  which garbage-collects this Secret. This rationale is captured in the function
  godoc.

New Vault permission requirement

The operator's Vault identity now needs create/update on
auth/approle/role/<role>/secret-id. Documented in the install guide (see
below).

Documentation

• docs/install-production.md §6.3 (new) — the operator's own Vault policy,
  explicitly including the new auth/approle/role/<role>/secret-id capability.
• docs/install-production.md §8 (rewritten) — manual SecretID staging is
  now "the operator provisions this for you"; §8.1 retains the legacy /
  air-gapped flow, including how to stamp the binding annotation for a fully
  admin-managed SecretID.
• hack/local-dev/seed-vault.sh — dropped the manual kubectl create secret
  step (the operator does it now); prints an explanatory note instead.
• test/fixtures/production-controlplane.yaml.example and
  internal/admin/init.go — stale "create out-of-band" comments updated.

Tests

Added/updated in internal/reconcilers/vault_test.go (all passing):

• TestVaultReconciler_AppRoleSecretIDBootstrap — fresh install generates the
  SecretID, creates the Secret with key id, and stamps the RoleID annotation +
  managed-by label. Exactly one GenerateSecretID call.
• TestVaultReconciler_AppRoleSecretIDIdempotent — a SecretID bound to the
  current RoleID is preserved; GenerateSecretID not called.
• TestVaultReconciler_AppRoleSecretIDRegeneratesWhenEmpty — empty id
  triggers regeneration (namespace-recreation case).
• TestVaultReconciler_AppRoleSecretIDRegeneratesOnRoleIDMismatch (new) — stale
  RoleID binding → regenerate + rebind annotation (Vault/AppRole recreation).
• TestVaultReconciler_AppRoleSecretIDAdoptsUnboundSecret (new) — unbound
  hand-created Secret → regenerate + stamp annotation.
• TestVaultReconciler_AppRoleVaultAuthWiring — VaultAuth carries the live
  RoleID and references the SecretRef by name.
  make test (including the envtest controller suite) is green.

Reconcile flow (issue #54's requested model)

ensure service policy
        ↓
ensure service AppRole  →  obtain RoleID
        ↓
generate SecretID (if missing / empty / RoleID-mismatch)
        ↓
create/adopt appRoleSecretRef Secret  (key `id`, bound to RoleID)
        ↓
configure VaultAuth  (RoleID + SecretRef)
        ↓
VSO authenticates  →  VaultStaticSecrets materialize

Backward compatibility / upgrade notes

• Existing clusters where an admin pre-staged the SecretID Secret: on the next reconcile the operator adopts it, replacing id with an operator-minted SecretID and stamping the binding annotation. VSO picks up the change on its next refresh.
• Operators must be granted create/update on auth/approle/role//secret-id (see docs/install-production.md §6.3) for the appRole flow.

Fixes #54

Checklist

• My code follows the style guidelines of this project
• I have added/updated comments where needed
• I have added tests that prove my fix is effective or my feature works
• I have run make test (or equivalent) locally and all tests pass
• I have updated the relevant documentation (CLI examples, man pages, README, other docs, etc.)
• DCO Sign-off: All commits are signed off (git commit -s) with my real name and email
• REUSE Compliance:
  • Each new/modified source file has SPDX copyright and license headers
  • Any non-commentable files include a <filename>.license sidecar
  • All referenced licenses are present in the LICENSES/ directory

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Dependency update
• Build system/CI

---

• I agree to follow the Code of Conduct.