Skip to content

docs(ops): bring the cutover runbook in line with the live cluster - #172

Merged
themightychris merged 5 commits into
developfrom
docs/cutover-runbook-reality
Sep 10, 2026
Merged

docs(ops): bring the cutover runbook in line with the live cluster#172
themightychris merged 5 commits into
developfrom
docs/cutover-runbook-reality

Conversation

@themightychris

@themightychris themightychris commented Sep 10, 2026

Copy link
Copy Markdown
Member

Summary

Docs-only. Brings docs/operations/cutover.md and the operator docs it leans on into line with what actually exists as of 2026-09-10, so the person running cutover isn't reading instructions written before the live cluster, the release flow, or the first data refresh existed. Plan: plans/cutover-runbook-reality.md.

What changed

  • Environments. Production is cfp-live-cluster / ns codeforphilly-ng / next.codeforphilly.org (alongside laddr in code-for-philly); sandbox is cfp-sandbox-cluster / ns codeforphilly-rewrite-sandbox / next-v2.codeforphilly.org. cfp-prod-cluster and codeforphilly-rewrite-staging.k8s.phl.io are gone from every doc; there is no separate staging.
  • Cutover = one commit, no DNS. T-0 and rollback are rewritten around moving the apex/www listeners + HTTPRoute hostnames from _gateways/code-for-philly.yaml to _gateways/codeforphilly-ng.yaml and flipping CFP_SITE_HOST; rollback is git revert. next.codeforphilly.org stays as a permanent alias. TTL-lowering and DNS-propagation steps removed.
  • Release + GitOps flow. develop → Release: vX.Y.Z → tag → container-publish.yml; cluster repos pin via .holo/sources/codeforphilly-ng.toml + images[].newTag, applied through releases/k8s-manifests → deploys/k8s-manifests. "Manual today" language removed; the :sandbox build is an emergency escape hatch only.
  • Private storage is a filesystem PVC in both environments. S3/GCS "production" instructions in legacy-credentials-import.md, the bucket checklist item, and the deploy.md bucket-provisioning section are replaced/qualified (S3 stays a supported, unused backend).
  • Secrets. Three SealedSecrets in cfp-live-cluster/codeforphilly-ng.secrets/ (codeforphilly-secrets, codeforphilly-saml carried over from laddr's saml2, codeforphilly-data-deploy-key), kubeseal --merge-into recipe, GitHub OAuth org app "Code for Philly" with no callback edit at cutover.
  • SAML. Stable SAML_ENTITY_ID; SSO-optional first, Slack "Test configuration" against the next.* metadata, update only the SSO URL at T-0; rollback note says a bad SAML flip is fixed in Slack, not by moving the hostname.
  • Data refresh. Data repo has no main; pipeline is import → merge legacy-import into published (thousands of deleted-by-us / modified-by-them conflicts for pruned spam, resolved by taking the import) → prune → push; notify-deployments.yml hot-reloads sandbox + prod and its prod URL must move to the apex at cutover. First refresh 2026-09-09/10: 36,254 → 22,625.
  • Legacy credentials. Export is emergence-site.people (Username, Email, Password) via the Habitat mysql client inside the laddr pod; load is kubectl cp + rollout restart. First prod load 2026-09-10: 21,761.
  • Passwords. T+90 mailout and T+180 purge sections deleted (per account-migration.md legacy password sign-in persists indefinitely); dangling #cutover-window-policy link removed; announcement templates no longer tell members their password is going away.
  • Monitoring. Kept honest: nothing external is wired; the T+1h window is watched by hand.
  • Also: sandbox-deploy.md now says GitOps is the path; namespace fixes (-n codeforphilly, codeforphilly-staging); codeforphilly-rewrite repo-name mentions in ops docs; .claude/CLAUDE.md deploy section.

Verified against local clones

cfp-live-cluster (pins, gateways, secret names/keys, kustomization patch), cfp-sandbox-cluster, codeforphilly-data-published (notify-deployments.yml), CodeForPhilly/ops tf/dns, and the Build k8s-manifests workflow.

Not verifiable from the repos (encoded as stated by the operator)

Slack SSO behavior (enterprise-only test workspaces, "SSO optional" fallback), GitHub OAuth app names and the subdomain-redirect behavior, the MySQL schema/table/column names and Habitat client path, the merge-conflict shape, and the import/load counts.

Spec drift noticed, not fixed here (docs-only PR)

  • specs/architecture.md: STORAGE_BACKEND "s3 in production"; entrypoint "git reset --hard origin/main"; overlays/staging/ + overlays/production/.
  • specs/behaviors/private-storage.md: "s3 backend (production)" and the bucket-versioning requirement, while both deployed environments run filesystem on a PVC with no versioning.
  • cutover-mailout.ts template says "Accounts unclaimed for one year may be retired" — no spec backing.

Recorded in the plan's Follow-ups.

Validation

  • npm run lint clean (no code touched).
  • Greps for cfp-prod-cluster, staging.k8s, cutover-window-policy, T+180, RESEND, -n codeforphilly return nothing under docs/ and .claude/.

🤖 Generated with Claude Code

https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR

themightychris and others added 5 commits September 10, 2026 02:06
@themightychris @claude
chore(plans): mark cutover-runbook-reality in-progress
ace1246 
Docs-only plan: bring the cutover runbook and the operator docs it
leans on into line with the environments that exist as of 2026-09-10
(cfp-live-cluster, GitOps release pins, filesystem private storage,
the published-branch refresh pipeline, the Slack SSO plan).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR
@themightychris @claude
docs(ops): rewrite cutover runbook around the gateway hostname move
9ebc021 
The runbook still described a DNS flip with a lowered TTL, a staging
host that never existed, an S3 bucket, a data-repo `main` branch, and a
T+90/T+180 claim window + password purge. None of that matches what is
deployed:

- Production runs in cfp-live-cluster (ns codeforphilly-ng) at
  next.codeforphilly.org, alongside laddr in code-for-philly. The apex
  and *.codeforphilly.org already resolve to that cluster's gateway, so
  T-0 is one commit moving the apex/www listeners and HTTPRoute
  hostnames from _gateways/code-for-philly.yaml to
  _gateways/codeforphilly-ng.yaml and flipping CFP_SITE_HOST. Rollback
  is a revert of that commit.
- The data repo has no `main`; the refresh is import -> merge
  legacy-import into published (thousands of deleted-by-us conflicts
  for pruned spam, resolved by taking the import) -> prune -> push,
  which hot-reloads sandbox and prod. First run: 36,254 -> 22,625.
- Slack SAML can only be tested in the real workspace, so the plan is
  SSO-optional first, Test configuration against the next.* metadata,
  then update only the SSO URL at cutover. A bad SAML flip is fixed in
  Slack, not by moving the hostname back.
- Legacy password sign-in persists indefinitely per
  account-migration.md; the T+90 mailout and T+180 purge sections are
  gone, and the announcement templates no longer tell members their
  password is going away.
- Monitoring: nothing external exists yet; the T+1h window says so
  instead of pointing at UptimeRobot.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR
@themightychris @claude
docs(ops): align deploy, secrets and runbook docs with live-cluster G…
e2636b6 
…itOps

Image publishing is no longer manual and production is not a future
`cfp-prod-cluster`: container-publish.yml pushes :vX.Y.Z on every
release tag, and each cluster repo (cfp-sandbox-cluster,
cfp-live-cluster) pins the app by .holo/sources/codeforphilly-ng.toml
plus images[].newTag in codeforphilly-ng/app/kustomization.yaml, applied
through the releases/k8s-manifests -> deploys/k8s-manifests PR. The
hand-built :sandbox image is kept only as an emergency escape hatch.

Private storage is a filesystem PVC in both environments; the S3
sections are marked supported-but-unused rather than "production".
Secrets are the three SealedSecrets in
cfp-live-cluster/codeforphilly-ng.secrets/ (codeforphilly-secrets,
codeforphilly-saml carried over from laddr's saml2, the deploy key),
edited with kubeseal --merge-into. Namespaces, the hot-reload secret
name, the pod-clone volume type and the monitoring status note are
corrected to match.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR
@themightychris @claude
docs(ops): document the real laddr credentials export and PVC load
ad174d5 
The export command pointed at a `laddr` database and a `users` table
with Handle/PasswordHash columns that don't exist. The live database is
`emergence-site` (the `codeforphilly` schema on the same server is a
stale 2024 copy), table `people`, columns Username/Email/Password, and
the export runs inside the laddr pod with the Habitat-packaged mysql
client against the service's client.cnf.

The production load is kubectl cp onto the pod's private-storage PVC
followed by a rollout restart, same as sandbox; the GCS/S3 upload
section is reduced to a note for the unused s3 backend. Records the
first prod load (2026-09-10, 21,761 profiles/credentials) and warns
that repeating it resets rehashed credentials now that sign-ins are
live at next.codeforphilly.org.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR
@themightychris @claude
chore(plans): mark cutover-runbook-reality done (PR #172)
5a41631 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR
@themightychris
themightychris merged commit fc0ab5d into develop Sep 10, 2026
1 check passed
@themightychris
themightychris deleted the docs/cutover-runbook-reality branch September 10, 2026 06:15
@themightychris themightychris mentioned this pull request Sep 10, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@themightychris