Skip to content

CCF-18200: remove default credentials from the OpenShift offering - #248

Open
vavarshn wants to merge 4 commits into
mainfrom
vavarshn/ccf-18200-no-default-creds
Open

vavarshn wants to merge 4 commits into
mainfrom
vavarshn/ccf-18200-no-default-creds

Conversation

@vavarshn

@vavarshn vavarshn commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Implements CCF-18200 (PSB SEC-AUT-DEFROOT-3 "No default credentials") for the OpenShift deployment.

  • Image: stop generating and copying a self-signed key into every operator image (Dockerfile, Dockerfile.debug). Nothing read /certs; webhook/metrics certs come from cert-manager.
  • Operator: remove the unused SECA Secret with API_TOKEN: "replace-me". SECA stays registered, so CRDs are unchanged for k0s/EKS.
  • Object-store credentials: the two upload helpers no longer fall back to minioadmin/minioadmin. The OpenShift installer and air-gap path reject missing, placeholder and factory-default credentials before any cluster change; k0s fails early on the same rules.
  • Splunk JWT signing: Splunk now signs with a per-installation RSA key and random passphrase in <standaloneName>-jwt-signing instead of the built-in server.pem with passphrase password. Port 8089, HEC and KV store are unchanged.
    • Existing Secrets are validated before reuse.
    • A lost Secret on a configured install requires restore (or ROTATE_JWT_SIGNER=true).
    • A signer fingerprint in the Standalone extraEnv makes the Splunk Operator restart Splunk whenever the key changes.
    • After readiness, a fresh JWT is verified against the key, using the admin Secret the pod mounts.
  • Docs: credential requirements and error messages, and how to back up, restore and rotate the signing Secret.

Behaviour changes to note

  • upload_to_minio.sh and upload_to_seaweedfs_upload_only.sh now require OBJECT_STORE_ACCESS_KEY/OBJECT_STORE_SECRET_KEY and refuse minioadmin/minioadmin. The OpenShift and k0s installers already pass credentials; k0s now fails at preflight instead of at upload.
  • Upgrading an existing OpenShift install generates a new signing key and restarts Splunk once; previously issued tokens stop validating, so users sign in again.
  • delete/clean-all now also remove the signing Secret.

Test plan

  • tools/ai-tier-cluster-setup/test_openshift_with_stack.sh: 217 passed
  • tools/ai-tier-cluster-setup/test_k0s_cluster_with_stack.sh: 258 passed
  • go build ./..., go test ./pkg/ai/features/... ./internal/webhook/...
  • OpenShift: fresh install, re-run, and upgrade from main; Splunk becomes Ready and the installer's JWT check passes
  • OpenShift: ROTATE_JWT_SIGNER=true rotation, including an interrupted run followed by a plain re-run
  • OpenShift: AI Assistant onboarding, SAIA/SLIM accept new tokens (also after upgrade), management 8089, KV store and HEC healthy
  • Air-gap install rejects minioadmin/minioadmin before patching the cluster
  • Rebuilt operator image contains no /certs

🤖 Generated with Claude Code

vvarshney-splunk and others added 4 commits October 14, 2026 11:04
… (CCF-18200)

SEC-AUT-DEFROOT-3 forbids shipped default credentials. Remove the private
key baked into every operator image, unregister the unreleased SECA feature
and its "replace-me" token, and drop the minioadmin fallbacks from the upload
helpers. The installer and air-gap wrapper now reject missing, placeholder and
factory-default object-store credentials before any cluster change.

Splunk now signs JWTs with a per-installation RSA key and random passphrase
instead of the built-in server.pem with passphrase "password". Existing
signing Secrets are validated before reuse, a lost Secret on a configured
install requires restoration (or explicit ROTATE_JWT_SIGNER=true, which
restarts Splunk), and a freshly issued JWT is verified against the key.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…dentials

Unregistering SECA changed the shared CRDs and operator for k0s and EKS. The
default credential was only the unused "replace-me" Secret, which stays
removed, so restore SECA's registration, CRD enum and sample as on main.

k0s staging uses the shared upload helpers, which now refuse missing and
minioadmin/minioadmin credentials. Check the same rules in k0s preflight and
before staging so k0s fails before the Hugging Face download instead of at
upload.

Co-Authored-By: Claude Code <noreply@anthropic.com>
- Drop become from the Splunk pre-task (the pod cannot escalate) and skip it
  when the signing Secret is not mounted, so an interrupted upgrade cannot
  crash-loop Splunk.
- Restart Splunk via a signer fingerprint in the Standalone extraEnv instead
  of a one-shot pod delete, so a failed rotation still converges on rerun;
  readiness waits for the pod carrying the current fingerprint.
- ROTATE_JWT_SIGNER=true now replaces an existing Secret as documented, and
  oc errors reading the Standalone stop the run instead of looking like a
  fresh install.
- Check credentials before the air-gap path changes the cluster, remove the
  unreachable duplicate checks, delete the signing Secret on uninstall, and
  give JWT verification clear errors for login and non-JSON responses.
- Fix the Secret backup/restore docs and the upload helper usage docs.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…able Standalone lookups

JWT verification now reads the admin password from the versioned Secret the
Splunk pod actually mounts (mnt-splunk-secrets), so a rotated -v2 password is
used instead of a stale -v1. An unparseable Standalone response now stops the
run instead of being treated as a fresh install. Tests cover both, plus
readiness waiting for the pod that carries the current signer fingerprint.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-14T18:13:53.904548Z c607204 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@vavarshn vavarshn changed the title fix(security): remove default credentials from the OpenShift offering (CCF-18200) CCF-18200: remove default credentials from the OpenShift offering Oct 14, 2026
@vavarshn
vavarshn force-pushed the vavarshn/ccf-18200-no-default-creds branch from c607204 to 85915b6 Compare October 14, 2026 18:53

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants