From ace1246e9a158038e0297a9687689754d3fa3d7b Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 10 Sep 2026 02:06:05 -0400 Subject: [PATCH 1/5] chore(plans): mark cutover-runbook-reality in-progress Docs-only plan: bring the cutover runbook and the operator docs it leans on into line with the environments that exist as of 2026-09-10 (cfp-live-cluster, GitOps release pins, filesystem private storage, the published-branch refresh pipeline, the Slack SSO plan). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR --- plans/cutover-runbook-reality.md | 114 +++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 plans/cutover-runbook-reality.md diff --git a/plans/cutover-runbook-reality.md b/plans/cutover-runbook-reality.md new file mode 100644 index 0000000..9784cc9 --- /dev/null +++ b/plans/cutover-runbook-reality.md @@ -0,0 +1,114 @@ +--- +status: in-progress +depends: + - cutover-prep + - release-flow + - postmark-notifier + - saml-self-host +specs: + - specs/behaviors/account-migration.md + - specs/behaviors/private-storage.md + - specs/behaviors/storage.md + - specs/api/saml.md +issues: [] +--- + +# Plan: Cutover runbook reality check + +## Scope + +Docs only. Bring `docs/operations/cutover.md` and the operator docs it leans +on (`cutover-rollback.md`, `deploy.md`, `runbook.md`, `secrets.md`, +`legacy-credentials-import.md`, `sandbox-deploy.md`, `monitoring.md`, +`releases.md`, `cutover-announcement.md`, plus the deploy section of +`.claude/CLAUDE.md`) into line with the environments that exist as of +2026-09-10, so the person running cutover is not reading instructions written +before the live cluster, the release flow, or the first data refresh existed. + +Out of scope: code changes; spec changes. Spec drift noticed along the way is +recorded under Follow-ups rather than fixed here. + +## Implements + +No new spec behavior. This plan reconciles operator docs with: + +- [behaviors/account-migration.md](../specs/behaviors/account-migration.md) — + legacy password sign-in persists indefinitely; no dated claim window or + credential purge exists, so the runbook's T+90 / T+180 sections go. +- [behaviors/private-storage.md](../specs/behaviors/private-storage.md) — + production runs the `filesystem` backend on a PVC; S3 stays a supported but + unused backend. +- [behaviors/storage.md](../specs/behaviors/storage.md) — the runtime branch is + `published` (the data repo has no `main`); hot reload via + `notify-deployments.yml`. +- [api/saml.md](../specs/api/saml.md) — stable `SAML_ENTITY_ID`, SSO endpoint + URLs follow `CFP_SITE_HOST`. + +## Approach + +Facts to encode, each verified against the sibling GitOps / DNS / data clones +on 2026-09-09/10: + +- **Environments.** Production is `cfp-live-cluster`, namespace + `codeforphilly-ng`, pre-cutover host `next.codeforphilly.org`, alongside + legacy laddr in `code-for-philly`. Sandbox is `cfp-sandbox-cluster`, + namespace `codeforphilly-rewrite-sandbox`, host `next-v2.codeforphilly.org`. + There is no separate staging; `cfp-prod-cluster` and + `codeforphilly-rewrite-staging.k8s.phl.io` never existed under those names. +- **Release + GitOps flow.** `develop` → `Release: vX.Y.Z` PR → tag → + `container-publish.yml` pushes `ghcr.io/codeforphilly/codeforphilly-ng:vX.Y.Z`. + Each cluster repo pins the app by `.holo/sources/codeforphilly-ng.toml` + (`ref`) and `images[].newTag` in `codeforphilly-ng/app/kustomization.yaml`, + bumped together. Merge to `main` → "Build k8s-manifests" → + `releases/k8s-manifests` → bot PR into `deploys/k8s-manifests` → merge applies. + The manual `:sandbox` image push is an escape hatch, not the path. +- **Cutover is a hostname move, not a DNS change.** `codeforphilly.org`, + `*.codeforphilly.org` and `*.live.k8s.phl.io` already resolve to the live + cluster's Envoy gateway (managed in `CodeForPhilly/ops` `tf/dns`). T-0 is one + commit in `cfp-live-cluster` moving the apex + `www` listeners and HTTPRoute + hostnames from `_gateways/code-for-philly.yaml` to + `_gateways/codeforphilly-ng.yaml` and flipping the `CFP_SITE_HOST` patch. + Rollback is `git revert`. +- **Private storage** is a filesystem PVC in prod (`/app/private-storage`). +- **Secrets** live in `cfp-live-cluster/codeforphilly-ng.secrets/` as three + SealedSecrets: `codeforphilly-secrets`, `codeforphilly-saml` (carried over + from laddr's `saml2`), `codeforphilly-data-deploy-key`. +- **Slack SAML plan**: SSO optional first, "Test configuration" against the + `next.codeforphilly.org` metadata, then update only the SSO URL at cutover. +- **Data refresh pipeline**: import → merge `legacy-import` into `published` + (thousands of deleted-by-us / modified-by-them conflicts for previously + pruned spam, resolved by taking the import's version) → prune-spam → push, + which hot-reloads sandbox and prod. First full refresh 2026-09-09/10: + 36,254 imported, 22,625 after prune. +- **Legacy credentials**: export from `emergence-site`.`people` inside the + laddr pod with the Habitat mysql client; load via `kubectl cp` + + `rollout restart`. First prod load 2026-09-10: 21,761 profiles/credentials. +- **Monitoring**: nothing external exists yet; say so rather than delete the + checklist. + +## Validation + +- [ ] `grep -rn 'cfp-prod-cluster\|staging.k8s\|cutover-window-policy\|T+180' docs/ .claude/CLAUDE.md` returns nothing. +- [ ] No doc names `main` as a data-repo branch; runtime branch is `published` everywhere. +- [ ] No doc gives S3/GCS as the production private-storage path; the filesystem PVC is. +- [ ] `cutover.md` T-0 and rollback sections describe the gateway-listener commit and its revert; no DNS/TTL steps remain. +- [ ] `legacy-credentials-import.md` export command targets `emergence-site`.`people` via the Habitat client. +- [ ] `npm run lint` clean. + +## Risks / unknowns + +- **Claims not verifiable from the repos** (Slack admin behavior, GitHub OAuth + app names, MySQL schema details, import counts) are encoded as stated by the + operator who ran them; flagged in the PR body. +- **Reconcile against production** — the runtime image ships `dist/` only (no + `tsx`, no `scripts/`), so `script:reconcile` cannot run inside the prod pod, + and the private store must not be copied to a laptop. Left as a documented + gap rather than a prescribed workaround. + +## Notes + +(Populated at closeout.) + +## Follow-ups + +(Populated at closeout.) From 9ebc0216319d717bef2e7a1ca41e937ed7a58657 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 10 Sep 2026 02:06:06 -0400 Subject: [PATCH 2/5] docs(ops): rewrite cutover runbook around the gateway hostname move The runbook still described a DNS flip with a lowered TTL, a staging host that never existed, an S3 bucket, a data-repo `main` branch, and a T+90/T+180 claim window + password purge. None of that matches what is deployed: - Production runs in cfp-live-cluster (ns codeforphilly-ng) at next.codeforphilly.org, alongside laddr in code-for-philly. The apex and *.codeforphilly.org already resolve to that cluster's gateway, so T-0 is one commit moving the apex/www listeners and HTTPRoute hostnames from _gateways/code-for-philly.yaml to _gateways/codeforphilly-ng.yaml and flipping CFP_SITE_HOST. Rollback is a revert of that commit. - The data repo has no `main`; the refresh is import -> merge legacy-import into published (thousands of deleted-by-us conflicts for pruned spam, resolved by taking the import) -> prune -> push, which hot-reloads sandbox and prod. First run: 36,254 -> 22,625. - Slack SAML can only be tested in the real workspace, so the plan is SSO-optional first, Test configuration against the next.* metadata, then update only the SSO URL at cutover. A bad SAML flip is fixed in Slack, not by moving the hostname back. - Legacy password sign-in persists indefinitely per account-migration.md; the T+90 mailout and T+180 purge sections are gone, and the announcement templates no longer tell members their password is going away. - Monitoring: nothing external exists yet; the T+1h window says so instead of pointing at UptimeRobot. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR --- docs/operations/cutover-announcement.md | 69 ++-- docs/operations/cutover-rollback.md | 137 ++++--- docs/operations/cutover.md | 479 +++++++++++++++--------- 3 files changed, 419 insertions(+), 266 deletions(-) diff --git a/docs/operations/cutover-announcement.md b/docs/operations/cutover-announcement.md index 1adf51b..4f4aa43 100644 --- a/docs/operations/cutover-announcement.md +++ b/docs/operations/cutover-announcement.md @@ -20,16 +20,17 @@ same accounts. The active cutover is about an hour. Expect ~5 minutes of What changes for you: - :white_check_mark: All your URLs keep working (legacy redirects in place) - :white_check_mark: Your Slack identity is preserved automatically -- :arrows_counterclockwise: First sign-in after cutover routes you through - GitHub OAuth + a one-click "claim your account" flow -- :no_entry_sign: Username/password sign-in is going away — GitHub is the - primary login from cutover forward +- :white_check_mark: Your existing username + password keep working +- :arrows_counterclockwise: You can also "Sign in with GitHub" — if your + GitHub email matches your CFP email your account links automatically; + otherwise a one-click claim flow does it. We'd love you to link GitHub, + but there's no deadline What we need from you BEFORE cutover: - Hold off on edits to your profile / projects starting {{ freeze_date }} (we're freezing writes to make the migration clean) -- Make sure you remember which GitHub account is associated with your CFP - email — your first sign-in needs to come from that GitHub account +- If you plan to sign in with GitHub, check which GitHub account uses your + CFP email — matching emails link automatically If you have questions: drop them in this thread or DM @{{ cutover_lead_slack }}. ``` @@ -46,15 +47,16 @@ On {{ cutover_date_long }} we're moving codeforphilly.org to a new platform. What's changing - The site is being rebuilt on a modern stack — same look, same URLs, same projects. -- Sign-in is moving to GitHub OAuth. Your password is no longer needed. +- Your existing username and password keep working. You can also sign in + with GitHub and link the two. - Your Slack identity is preserved automatically. What you need to do - Nothing right now — but please don't edit your profile or projects between {{ freeze_date }} and {{ cutover_date_short }}. -- After cutover, sign in via the new "Sign in with GitHub" button. If - your CFP email matches your GitHub email, your account is claimed - automatically. If not, follow the on-screen claim flow. +- After cutover, sign in as usual, or try the new "Sign in with GitHub" + button. If your CFP email matches your GitHub email, your account is + linked automatically. If not, follow the on-screen claim flow. When it happens - {{ cutover_date_long }} starting at {{ cutover_time }}. @@ -65,10 +67,14 @@ Questions? Reply to this email or ping us in Slack. — Code for Philly ``` -## T-0: maintenance page +## T-0: maintenance page (optional) -The static page served from the legacy site while DNS propagates. Plain HTML; -no JavaScript needed. +A static page for the legacy site during the hostname move. Cutover is a +gateway-listener change with no DNS propagation +([cutover.md → T-0](cutover.md#t-0-cutover)), so the switch is effectively +instant per hostname and this page is optional — use it if you want a +visible "hold on" while the final data delta runs. Plain HTML; no +JavaScript needed. ```html @@ -120,10 +126,18 @@ https://github.com/CodeForPhilly/codeforphilly-ng/issues or just reply here. ``` -## T+90 days: unclaimed-account reminder +## Unclaimed-account reminder (not scheduled) -Sent automatically by `apps/api/scripts/cutover-mailout.ts`. The template is -in code — see `buildEmailBody()` in that file. Reproduced here for review: +`apps/api/scripts/cutover-mailout.ts` can email members whose laddr account +has not yet been linked to GitHub. **It is not on the cutover timeline.** +Per [account-migration.md](../../specs/behaviors/account-migration.md#sunset-deferred) +legacy password sign-in has no deadline, so there is nothing to remind +people *of*; the only nudge is the "Connect GitHub" banner on `/account`. +If a future spec change sets a sunset date, this template is the starting +point — until then, don't send it. The template is in code — see +`buildEmailBody()` in that file. Reproduced here for review (note the last +paragraph about retiring accounts has no backing in the spec and would need +one before any send): Subject: `Action needed: claim your Code for Philly account` @@ -144,29 +158,6 @@ unclaimed for one year may be retired. — Code for Philly ``` -## T+180 days: password-credential drain notice - -A short Slack post in `#announcements` and an info-only email to remaining -unclaimed Persons. Honest about the irreversible step. - -### Slack - -``` -:warning: Six months post-cutover - -It's been six months since we moved to the new platform. We're cleaning up -the remaining legacy password records — about {{ count }} accounts. - -If you signed in via GitHub at any point: you're already migrated, this -doesn't affect you. - -If you have a laddr account you haven't logged into yet: please log in -in the next 14 days. After that, you'll need to contact a staff member -to verify your identity before reclaiming the account. - -Questions: ping @{{ cutover_lead_slack }}. -``` - ## Localization We don't send in any language other than English at v1. If we ever expand diff --git a/docs/operations/cutover-rollback.md b/docs/operations/cutover-rollback.md index 5fc0a0c..5f035f6 100644 --- a/docs/operations/cutover-rollback.md +++ b/docs/operations/cutover-rollback.md @@ -7,15 +7,21 @@ you want this in muscle memory, not on a screen you're scrolling through. > Companion: [cutover.md](cutover.md) (runbook), > [cutover-announcement.md](cutover-announcement.md) (Slack + email templates). +## The short version + +Cutover is one commit in `cfp-live-cluster` (gateway listeners + HTTPRoute +hostnames + `CFP_SITE_HOST`). **Rollback is `git revert` of that commit**, +pushed through the same GitOps path. No DNS is involved in either direction. + ## When to roll back The criterion is "is the rewrite serving users worse than legacy was, and will it stay that way longer than 15 minutes?" Concrete triggers: - `/api/health` flapping or stuck 503 with no clear single-fix path +- The apex or `www` serving an invalid certificate that cert-manager isn't + resolving within a few minutes - OAuth callback failure rate > 50% for more than 5 minutes -- Slack SAML failure rate > 50% for more than 5 minutes (Slack is critical - to brigade ops — this is a bigger deal than the OAuth callback) - A data corruption that's already in the gitsheets commit history and hard to surgically revert - Any unrecoverable cluster issue that isn't a five-minute fix (e.g. PVC @@ -23,18 +29,25 @@ will it stay that way longer than 15 minutes?" Concrete triggers: **Not rollback triggers:** a single failing route, a cosmetic UI bug, a known-not-blocking warning during boot, OAuth callback failures for a -single GitHub account that's been deactivated. +single GitHub account that's been deactivated, **or a Slack SSO failure** — +SSO is set to optional before cutover, members fall back to email magic +links, and the fix is correcting the SSO URL in Slack (see below), not +moving the hostname back. ## Point of no return The rollback is only safe **before the first real-user mutation lands on -the rewrite**. After a user signs in for the first time on the rewrite a +the rewrite via the apex**. After a user signs in for the first time a `PrivateProfile` is written; after they edit anything a gitsheets commit is created. Those records exist only on the new side. Rolling back would either lose them or require a manual re-import after the next attempt. +(The rewrite has been live at `next.codeforphilly.org` since before +cutover, so some divergence from laddr already exists. The point-of-no-return +question is specifically about writes that arrive *because* the apex moved.) + In practice this means: the rollback window is the first ~15 minutes -after DNS flip. After that, fix forward. +after the cutover PR applies. After that, fix forward. The cutover lead announces "point of no return crossed" in the cutover Slack thread when the first user-created record (not the importer's @@ -42,39 +55,57 @@ records) lands. After that, this document is informational only. ## Rollback procedure -The 4-step sequence. Cutover lead at the keyboard; engineering second -reads the steps back. +Cutover lead at the keyboard; engineering second reads the steps back. -### 1. DNS flip back to legacy +### 1. Revert the cutover commit ```bash -# Whatever your DNS provider is, flip codeforphilly.org back to the -# legacy ingress / load balancer. +cd ~/Repositories/cfp-live-cluster +git fetch origin +git checkout -b revert/cutover origin/main +git revert --mainline 1 # or the single commit sha if it was a plain commit +git push -u origin revert/cutover +gh-axi pr create --base main --title "Revert cutover: apex back to laddr" ``` -TTL was lowered to 60s a week before cutover, so propagation completes -in under two minutes for most resolvers. Verify with `dig +short codeforphilly.org` -from a fresh resolver. +Merge it. "Build k8s-manifests" projects to `releases/k8s-manifests`, the +bot opens the PR into `deploys/k8s-manifests`, merge that. The apex and +`www` listeners return to `_gateways/code-for-philly.yaml`, the rewrite's +`CFP_SITE_HOST` goes back to `next.codeforphilly.org`, and the rewrite pod +rolls once more. + +**Emergency shortcut** if the GitOps round-trip is too slow: revert +directly on `deploys/k8s-manifests` (see [runbook.md → Last-resort +recovery](runbook.md#3-last-resort-recovery)) and reconcile `main` +afterwards. + +Verify: `curl -sSI https://codeforphilly.org/` serves laddr; +`https://next.codeforphilly.org/` still serves the rewrite. -### 2. Re-enable legacy writes +### 2. Put Slack SSO back + +If the SSO URL in Slack was already updated to the apex, the Slack +workspace admin changes it back to +`https://next.codeforphilly.org/api/saml/slack/sso`. Because SSO is +optional, nothing here is urgent — members without a working SSO path use +email magic links until it's fixed. Do **not** move the hostname to fix a +SAML problem. + +### 3. Re-enable legacy writes If you flipped the legacy site to read-only at T-7, undo that flag now. The legacy DB has not been touched during the migration window (the importer -only reads from `?format=json` endpoints). Writes resume from the same state -as just-before-freeze. - -### 3. Take down the rewrite ingress +only reads from `?format=json` endpoints and the credentials export is +read-only). Writes resume from the same state as just-before-freeze. -```bash -# Scale the rewrite Deployment to zero so no traffic can hit it even if -# a stale DNS resolver still points there. -kubectl -n codeforphilly scale deploy/codeforphilly --replicas=0 -``` +### 4. Point the hot-reload workflow back -Or just remove the Ingress resource. The Deployment can stay running for -future debugging; what matters is no public traffic reaches it. +If you already switched the prod URL in the data repo's +`notify-deployments.yml` to the apex, switch it back to +`https://next.codeforphilly.org/api/_internal/reload-data` so `published` +pushes keep reaching the rewrite pod. -### 4. Post the rollback notice +### 5. Post the rollback notice Slack `#announcements`: @@ -103,9 +134,10 @@ Once users are back on legacy and the pressure is off: 1. Capture the failure in writing — what error, what frequency, in what logs, what was happening on the user side. -2. Pull `kubectl logs` for the dead pod (`--previous` if it CrashLoopBacked). -3. Pull the reconciliation report from T-1 and the dry-run report from - T-3 — compare against what landed on the production side. +2. Pull `kubectl -n codeforphilly-ng logs deploy/codeforphilly --previous` + for the dead pod (if it CrashLoopBacked). +3. Pull the dry-run report from T-3 and the refresh counts from T-1 — + compare against what landed on the production side. 4. File an issue documenting what we know vs. what we don't. The point is to have a tight description of the failure mode before @@ -119,44 +151,47 @@ modifications: - T-7 announce step is shorter — refer to the original. "We're trying again on {{ new_date }}; same plan, no further changes from your side." -- T-3 dry-run **must** explicitly reproduce the failure mode that triggered +- T-3 rehearsal **must** explicitly reproduce the failure mode that triggered the rollback and confirm the fix. - The cutover thread includes a "what we learned and changed" entry above the new timeline. ## After a partial-write rollback (rare and ugly) -The procedure above assumes nobody wrote anything to the rewrite before -rollback. If that assumption was wrong (you crossed point-of-no-return -and *still* had to roll back because something worse happened): +The procedure above assumes nobody wrote anything to the rewrite via the +apex before rollback. If that assumption was wrong (you crossed +point-of-no-return and *still* had to roll back because something worse +happened): -1. **Don't panic.** The rewrite's data is still in `git` history — nothing - is silently deleted. +1. **Don't panic.** The rewrite's public data is still in `git` history — + nothing is silently deleted. 2. Identify the time window in which user writes happened on the rewrite. - Pull every commit on the data repo between import-commit and the - rollback moment. Same for the private-storage bucket's versioned - `.jsonl` history. + Pull every commit on the data repo's `published` branch between the + refresh commit and the rollback moment. The private store is `.jsonl` + on the pod's PVC (`/app/private-storage`) — it has no versioning, so + snapshot it (`kubectl cp` out of the pod, kept in-cluster or on an + operator machine only as long as needed) before anything else touches it. 3. Export those records as a JSON diff. Send to the legacy site's owner to manually replay if they're irreplaceable, or — much more commonly — email the affected users explaining their write didn't land and asking them to redo it after the second cutover attempt. -4. Hard-reset the rewrite's data repo to the post-import commit. Re-deploy - when ready. +4. Hard-reset the rewrite's `published` branch to the post-refresh commit. + Re-deploy when ready. -This case is operationally painful but not data-loss; the records are in -git history forever. The cost is reconciliation effort. +This case is operationally painful but not data-loss on the public side; +the records are in git history forever. The cost is reconciliation effort. ## What rollback does NOT cover -- **Slack workspace SAML disconnection.** If you flipped the Slack SSO - config over to the new IdP *before* validating, you may need to ask - the Slack workspace owner to swap back. This is why the runbook says - to pre-stage but not activate the SAML connection at T-7. +- **Slack SSO.** Reverting the gateway does nothing for SAML. A wrong SSO + URL or a bad assertion is fixed in Slack's SAML settings; a member who + can't sign in via SSO uses the email magic link because SSO is optional. + If the workspace was mistakenly set to SSO *required*, the admin sets it + back to optional — that is the actual lockout fix. - **GitHub OAuth app suspension by GitHub.** If GitHub deactivated the - OAuth app (rate limit, ToS issue), rolling back the DNS doesn't help + OAuth app (rate limit, ToS issue), moving the hostname back doesn't help — legacy doesn't use OAuth. Contact GitHub support; legacy will work unaffected because legacy uses passwords. -- **DNS resolver caching beyond TTL.** Some corporate networks ignore - DNS TTLs and cache for hours. Users behind such a network will see - the rewrite for longer than the 60s flip implies. Rare; document the - user-side workaround (open a private tab / change DNS). +- **Writes already made at `next.codeforphilly.org`.** The rewrite was + live there before cutover and stays live after a rollback. Anything + members did there is real and stays. diff --git a/docs/operations/cutover.md b/docs/operations/cutover.md index ecd5bf7..a20c9a2 100644 --- a/docs/operations/cutover.md +++ b/docs/operations/cutover.md @@ -1,22 +1,46 @@ # Cutover runbook The sequenced playbook for moving codeforphilly.org from legacy laddr to the -rewrite. Mirrors the timeline in [plans/cutover-prep.md](../../plans/cutover-prep.md) -and the policy in [specs/behaviors/account-migration.md#cutover-window-policy](../../specs/behaviors/account-migration.md#cutover-window-policy). +rewrite. Grew out of [plans/cutover-prep.md](../../plans/cutover-prep.md); +the account-side policy is [specs/behaviors/account-migration.md](../../specs/behaviors/account-migration.md). > Companion docs: [cutover-announcement.md](cutover-announcement.md) (Slack + > email templates) and [cutover-rollback.md](cutover-rollback.md) (what to do > if it goes sideways). For routine ops see [deploy.md](deploy.md), -> [runbook.md](runbook.md), [secrets.md](secrets.md). +> [runbook.md](runbook.md), [secrets.md](secrets.md), +> [legacy-credentials-import.md](legacy-credentials-import.md). + +## Where things run + +| | GitOps repo | Namespace | Host | +|---|---|---|---| +| **Production** (the rewrite) | [`cfp-live-cluster`](https://github.com/CodeForPhilly/cfp-live-cluster) | `codeforphilly-ng` | `https://next.codeforphilly.org` before cutover; `https://codeforphilly.org` after | +| **Legacy laddr** | `cfp-live-cluster` | `code-for-philly` | `https://codeforphilly.org` before cutover; `https://codeforphilly.live.k8s.phl.io` throughout | +| **Sandbox** | [`cfp-sandbox-cluster`](https://github.com/CodeForPhilly/cfp-sandbox-cluster) | `codeforphilly-rewrite-sandbox` | `https://next-v2.codeforphilly.org` | + +There is no separate staging environment; the sandbox plays that role. The +production pod is already running and serving real data at +`next.codeforphilly.org` — cutover changes which hostnames reach it, nothing +about the pod itself. + +**Cutover needs no DNS change.** DNS for `codeforphilly.org` is managed in +[`CodeForPhilly/ops`](https://github.com/CodeForPhilly/ops) (`tf/dns`, +OpenTofu, Cloud DNS project `openphl-1177`). `codeforphilly.org`, +`*.codeforphilly.org` and `*.live.k8s.phl.io` all already resolve to the +live cluster's Envoy gateway. Which app answers for a hostname is decided +by the Gateway listeners in `cfp-live-cluster/_gateways/`, so T-0 is a +single commit in that repo. There is no TTL to lower and no propagation to +wait on. ## Roles | Role | Responsibility | |------|----------------| -| **Cutover lead** | Owns the timeline; the only person who runs the actual T-0 commands. | +| **Cutover lead** | Owns the timeline; the only person who merges the T-0 commit. | | **Comms lead** | Owns Slack + email announcements; on standby for member questions. | -| **Engineering second** | Runs reconciliation + dry-run; co-pilot for cutover lead at T-0. | -| **On-call** | Watches `/api/health` and the alerts channel during T-0 → T+24h. | +| **Engineering second** | Runs the data refresh + rehearsal; co-pilot for cutover lead at T-0. | +| **Slack workspace admin** | Owns the SAML SSO settings in Slack (SSO optional, Test configuration, SSO URL update). | +| **On-call** | Watches pod logs and `/api/health` during T-0 → T+24h. | These can be the same person at small org scale, but the role assignments should be explicit in the cutover Slack post. @@ -25,44 +49,62 @@ should be explicit in the cutover Slack post. | When | What | Reversible? | |------|------|-------------| -| T-7 days | Announce; freeze write workflow on legacy site | Yes | -| T-3 days | Final staging-rehearsal `cutover-dry-run.ts` | Yes | -| T-1 day | Final import from live laddr JSON; verify counts | Yes | -| T-0 | DNS flip, maintenance page comes down | **Point of no return** when first new sign-in lands | -| T+1h | Active monitoring; smoke-test public flows | Yes (rollback) | -| T+24h | Post-cutover all-clear in Slack | Yes (rollback) | -| T+90d | Reminder mailout to unclaimed Persons | Information-only | -| T+180d | Delete unclaimed `LegacyPasswordCredential` rows | Information-only | +| T-7 days | Announce; freeze write workflow on legacy site; Slack SSO to "optional" | Yes | +| T-3 days | Rehearsal: dry-run against sandbox, smoke prod at `next.codeforphilly.org`, Slack "Test configuration" | Yes | +| T-1 day | Data refresh (import → merge → prune → push); credentials load; verify counts | Yes | +| T-0 | Gateway hostname move + `CFP_SITE_HOST` flip (one commit); Slack SSO URL update | **Point of no return** when first new record lands | +| T+1h | Active monitoring; smoke-test public flows | Yes (revert) | +| T+24h | Post-cutover all-clear in Slack | Yes (revert) | +| T+7d | Reconciliation check | Information-only | + +There is deliberately **no** T+90 claim-window mailout and **no** T+180 +password purge. Per [account-migration.md](../../specs/behaviors/account-migration.md#sunset-deferred), +legacy password sign-in persists indefinitely with a link-GitHub nag on +`/account`; any future sunset is undated and needs its own spec change first. ## T-7 days: announce + freeze 1. Post the cutover announcement from [cutover-announcement.md](cutover-announcement.md) to `#announcements` and email all members via Postmark. -2. Lower DNS TTL on `codeforphilly.org` to 60s. Verify with `dig`. -3. **Freeze legacy writes.** Either put a banner on the legacy site asking +2. **Freeze legacy writes.** Either put a banner on the legacy site asking members to hold off on edits, or flip a feature flag making it read-only. - The point is to reduce the size of the "delta" between the T-3 dump and - T-0 dump. -4. Notify cluster operators that you'll be moving production traffic next week. -5. Confirm the GitHub OAuth app's production redirect URI is registered. -6. Confirm the Slack SAML connection is **pre-staged** (Slack admin has the - new IdP metadata; not active yet). -7. Confirm sealed-secrets are in the production cluster — see - [secrets.md](secrets.md). Missing secrets at T-0 is the most common - single failure mode. + The point is to reduce the size of the "delta" between the T-1 refresh + and the T-0 refresh. +3. Notify cluster operators that you'll be moving the apex hostname next week. +4. **GitHub OAuth.** The production app is the org-owned OAuth app + **"Code for Philly"** (the laddr one was renamed "Code for Philly + (legacy)"). Its callback is registered on the apex; GitHub accepts + redirects to subdomains of the registered callback, which is why + `next.codeforphilly.org` already works against it. **No edit is needed + at cutover.** Confirm the app still exists and its client ID matches the + one in `codeforphilly-secrets`. +5. **Slack SAML — lockout insurance first.** Slack SSO cannot be tested in a + throwaway workspace (SAML is enterprise-only), so the real workspace is + the only test bed. Have the Slack workspace admin set SSO to + **"optional"** now. Members can then always fall back to email magic + links, so a wrong SAML setting at any later step is an inconvenience, + not a lockout. +6. Confirm the three SealedSecrets are present in + `cfp-live-cluster/codeforphilly-ng.secrets/` and decrypted in the + `codeforphilly-ng` namespace — see [secrets.md](secrets.md). Missing + secrets at T-0 is the most common single failure mode. -## T-3 days: full staging rehearsal + ```bash + kubectl -n codeforphilly-ng get secret codeforphilly-secrets codeforphilly-saml codeforphilly-data-deploy-key + ``` + +## T-3 days: rehearsal -The rehearsal must run end-to-end against `codeforphilly-rewrite-staging.k8s.phl.io` -and produce a passing report. +There is no staging cluster to rehearse the hostname move itself on; what +you can rehearse is everything that happens *behind* the hostname. -1. Run the dry-run script against the live laddr site: +1. Run the dry-run script with the sandbox as target: ```bash npm run -w apps/api script:cutover-dry-run -- \ --source-host=codeforphilly.org \ --data-repo=/scratch/dry-run-data \ - --target=https://codeforphilly-rewrite-staging.k8s.phl.io \ + --target=https://next-v2.codeforphilly.org \ --json=/scratch/dry-run-T3.json ``` @@ -70,44 +112,59 @@ and produce a passing report. - `stages.import` must be `true`. - `stages.countDiff` must be `true` (every sheet's imported count is within tolerance of the server's reported `total`). - `stages.smoke` must be `true` (all probes return 2xx/3xx). -3. Manually verify Slack SAML continuity for a test laddr user. - This is the highest-stakes single check. See - [specs/api/saml.md](../../specs/api/saml.md): a user's `slackSamlNameId` - must equal their pre-cutover Slack NameID byte-for-byte. -4. File any anomalies, schedule a re-run before T-0 if anything fails. +3. Repeat the smoke probes by hand against production at + `https://next.codeforphilly.org` — `/api/health/ready`, + `/api/people/`, `/api/projects/`, + `/projects?ID=` (must 301 to the slug URL), a GitHub sign-in. +4. **Slack SAML.** In Slack's SSO settings, point the IdP metadata at + `https://next.codeforphilly.org/api/saml/slack/metadata` and use Slack's + **"Test configuration"** against it. The IdP entity ID it reports must be + `https://codeforphilly.org/api/saml/slack/metadata` — that value is + `SAML_ENTITY_ID`'s default and does not follow the host, so it is the + same before and after cutover ([specs/api/saml.md](../../specs/api/saml.md#idp-identity-and-hosts)). + The signing cert is the one carried over from laddr's `saml2` secret, so + Slack keeps trusting the same cert. A test user's NameID must equal + their pre-cutover Slack NameID byte-for-byte. +5. File any anomalies, schedule a re-run before T-0 if anything fails. If the dry-run reports unexpectedly low imported counts for any sheet, **stop**. Either the laddr JSON shape drifted (a new field broke Zod validation) or the importer needs an update. Don't proceed to T-0 with silently-dropped data. -## T-1 day: production migration +## T-1 day: data refresh + credentials + +The data repo (`CodeForPhilly/codeforphilly-data`) has **no `main`**. The +runtime-served branch is **`published`**; `legacy-import` holds the raw +importer snapshots. The refresh pipeline is always: + +**import → merge `legacy-import` into `published` → prune-spam → push** -The production import is a snapshot commit on the `legacy-import` branch of -the production data repo. Private data (emails, password hashes) is **not** -populated by this importer — that is a separate one-shot run of the -credentials importer; see -[legacy-credentials-import.md](./legacy-credentials-import.md) for the -CSV format, run command, and deploy steps (sandbox PVC or production -S3-compat bucket). +Pushing `published` fires the data repo's +`.github/workflows/notify-deployments.yml`, which POSTs the hot-reload +webhook on sandbox *and* prod; both pods rebuild in-memory state in place +with no restart ([runbook.md#hot-reload-webhook](runbook.md#hot-reload-webhook)). -1. **Bare-clone** the production data repo locally — the importer matches - the running pod's invariant ([storage.md → "The data clone is bare"](../../specs/behaviors/storage.md)) +The first full refresh through this pipeline ran 2026-09-09/10: **36,254** +person records imported, **22,625** left on `published` after prune. Use +those as the order-of-magnitude sanity check for your counts. + +### Public data + +1. **Bare-clone** the data repo — the importer matches the running pod's + invariant ([storage.md → "The data clone is bare"](../../specs/behaviors/storage.md)) and `openPublicStore` rejects a non-bare path: ```bash - git clone --bare git@github.com:CodeForPhilly/codeforphilly-data.git /scratch/codeforphilly-data + git clone --bare git@github.com:CodeForPhilly/codeforphilly-data.git /scratch/codeforphilly-data.git ``` - For one-off imports the convention is to clone into `/tmp` and clean up - after pushing — keeps your working data-repo clone clean. - -2. Run the importer against the production target — **with `--dry-run` first**: +2. Run the importer — **with `--dry-run` first**: ```bash npm run -w apps/api script:import-laddr -- \ --source-host=codeforphilly.org \ - --data-repo=/scratch/codeforphilly-data \ + --data-repo=/scratch/codeforphilly-data.git \ --branch=legacy-import \ --dry-run ``` @@ -116,104 +173,184 @@ S3-compat bucket). namespaces, and skipped HTTP-only buzz URLs are expected; zod errors are not. 4. Run the importer **without `--dry-run`**. This creates one snapshot - commit on the `legacy-import` branch. -5. Push the `legacy-import` branch to the production GitHub remote. -6. Merge `legacy-import` into `main` (operator step — review the diff in a - PR, resolve any path-template conflicts, then merge). -7. **Prune confident-spam** from the runtime branch before it goes live. The - merge in step 6 re-adds the full raw import (spam included); the deployed pod - cannot hold the unpruned set in memory, so this step is **mandatory** after - every import/merge. See + commit on `legacy-import`. Push it. +5. **Merge `legacy-import` into `published`.** This needs a working tree + (clone from your bare clone). Expect **thousands of conflicts** of the + form `deleted by us / modified by them`: every previously-pruned spam + person is absent on `published` and present (possibly modified) in the + fresh snapshot. Resolve them all by **taking the import's version** — + the prune step in the next step removes them again: + + ```bash + git clone /scratch/codeforphilly-data.git /scratch/codeforphilly-data-wt + cd /scratch/codeforphilly-data-wt + git checkout published + git merge origin/legacy-import || true + git diff --name-only --diff-filter=U | xargs git checkout --theirs -- + git add -A people/ # the conflicts are person records + git commit --no-edit + git push origin published:published # back to the bare clone only — don't push to GitHub yet + ``` + + Review `git status` before the `git add`; if conflicts show up outside + `people/`, look at them individually rather than bulk-resolving. + +6. **Prune confident-spam** from `published` before it goes anywhere. The + merge re-added the full raw import (spam included); the deployed pod + cannot hold the unpruned set in memory, so this step is **mandatory** + after every import/merge. See [spam-detection.md → Applying spam decisions](./spam-detection.md#applying-spam-decisions--the-prune-step): ```bash npm run -w apps/api script:prune-spam -- \ - --data-repo=/scratch/codeforphilly-data \ + --data-repo=/scratch/codeforphilly-data.git \ --evaluations-ref=spam-detection \ --branch=published \ - --dry-run # review counts, then drop --dry-run and push the branch + --dry-run # review counts, then drop --dry-run ``` Newly-imported accounts with no spam verdict yet are kept (the rule only - removes *confident* spam), so an incomplete eval pass is safe — it just keeps - more people than strictly necessary. -8. Run reconciliation: + removes *confident* spam), so an incomplete eval pass is safe — it just + keeps more people than strictly necessary. - ```bash - npm run -w apps/api script:reconcile -- --json=/scratch/reconcile-T1.json - ``` +7. **Push `published` to GitHub.** `notify-deployments.yml` hot-reloads + sandbox and prod. Watch the workflow run go green for both targets, then + confirm `https://next.codeforphilly.org/api/people/` reflects a + record you know changed upstream. + +### Private data (credentials) + +Emails and password hashes are **not** in the public import. They come from +the laddr MySQL database via the credentials importer and are loaded onto +the production pod's private-storage PVC — full procedure in +[legacy-credentials-import.md](./legacy-credentials-import.md). In short: - Every counter should be zero in the orphan + inconsistent categories. - If anything is flagged, **stop** and investigate before T-0. +1. Export `Username, Email, Password` from `emergence-site`.`people` inside + the laddr pod (Habitat mysql client) to a quoted CSV. +2. Run `script:import-laddr-credentials` against the bare clone's + `published` branch to produce `profiles.jsonl` + `legacy-passwords.jsonl`. +3. `kubectl -n codeforphilly-ng cp` both files into the pod's + `/app/private-storage/`, then `kubectl -n codeforphilly-ng rollout + restart deploy/codeforphilly`. **Hot reload does not cover the private + store**; the restart is required. -9. Deploy the rewrite to production via the production GitOps repo (a - sibling to [`cfp-sandbox-cluster`](https://github.com/CodeForPhilly/cfp-sandbox-cluster) - — see [deploy.md](deploy.md)). The pod will boot against the - just-imported data + bucket but receive no public traffic yet (Gateway - hostname not pointed at the prod LoadBalancer yet). +The first production load ran 2026-09-10: **21,761** profiles/credentials. -10. Smoke-test the production hostname through `/etc/hosts` or via direct - cluster IP: hit `/api/health`, `/api/people/`, - `/api/projects/`. Don't yet flip DNS. +Re-running after real users have signed in **reverts their rehashed +(argon2id) credentials to the legacy hashes** — see the safety notes in +that doc before repeating the load at T-0. + +### Confirm the deployed version + +Production is pinned in `cfp-live-cluster` by two values that move +together: `.holo/sources/codeforphilly-ng.toml` (`ref = "refs/tags/vX.Y.Z"`) +and `images[].newTag` in `codeforphilly-ng/app/kustomization.yaml`. Confirm +they match the release you intend to cut over on, and that the sandbox has +run that same tag for a while. ## T-0: cutover At T-0 the team is on a call together. Cutover lead at the keyboard; -engineering second has the runbook open and reads checks back. +engineering second has the runbook open and reads checks back. Legacy stays +reachable at `https://codeforphilly.live.k8s.phl.io` throughout, so a +maintenance page is optional — the switch itself is atomic per hostname. -1. **0:00 — maintenance page.** Put a static maintenance page on - the legacy `codeforphilly.org`. (Legacy site can stay up under - the hood; we just don't want users hitting a half-state.) -2. **0:01 — final delta.** Re-run the importer against the live laddr site - into the same data-repo path. UUIDs are read-forward from the previous +1. **0:00 — final delta.** Re-run the T-1 public-data pipeline (import → + merge → prune → push). UUIDs are read-forward from the previous snapshot's tree, so the diff between this commit and the T-1 commit is - exactly the records that changed upstream since T-1. **Then merge into the - runtime branch and re-run the spam prune (step 7 above) before the pod - reloads** — the final-delta merge re-adds raw import records, so an unpruned - reload would re-bloat memory. -3. **0:05 — DNS flip.** Update the `codeforphilly.org` A/CNAME to point - at the rewrite's ingress. TTL was lowered to 60s a week ago, so - propagation completes in under two minutes for most resolvers. -4. **0:10 — verify.** From four different networks (lead's laptop, - engineering second's tethered phone, a VPN exit node, a CFP volunteer's - home network), confirm `codeforphilly.org` serves the rewrite. Test: + exactly the records that changed upstream since T-1. The push + hot-reloads prod. Only re-run the credentials load if nobody has signed + in at `next.codeforphilly.org` since T-1 (or you accept the rehash reset). + +2. **0:05 — the cutover commit.** One commit in `cfp-live-cluster`, on a + branch, PR'd into `main`: + + - In `_gateways/code-for-philly.yaml`: remove the `https-apex` + (`codeforphilly.org`) and `https-www` (`www.codeforphilly.org`) + listeners and drop those hostnames from the legacy HTTPRoute. Leave + the `codeforphilly.live.k8s.phl.io` listener so laddr stays reachable. + - In `_gateways/codeforphilly-ng.yaml`: add those two listeners (with + their `certificateRefs`) and hostnames to the rewrite's Gateway + + HTTPRoute. **Keep the existing `next.codeforphilly.org` listener** — + it is a permanent alias. + - In `codeforphilly-ng/app/kustomization.yaml`: change the + `CFP_SITE_HOST` patch value from `next.codeforphilly.org` to + `codeforphilly.org`. This drives the markdown renderer's + internal/external link split, canonical URLs in email, and the SAML + metadata's `SingleSignOnService` locations. + + Merge to `main`. The **"Build k8s-manifests"** workflow projects to + `releases/k8s-manifests`, a bot opens a PR into `deploys/k8s-manifests`, + and merging *that* PR applies. The ConfigMap change rolls the pod + (`Recreate` strategy) — expect a short readiness gap while it re-clones + and reloads. cert-manager issues certificates for the moved hostnames + into the `codeforphilly-ng` namespace; allow a minute or two for ACME. + +3. **0:10 — verify.** From several networks (lead's laptop, engineering + second's tethered phone, a volunteer's home network), confirm + `https://codeforphilly.org` and `https://www.codeforphilly.org` serve + the rewrite with valid certificates: - Anonymous browse of `/projects` and `/members` (read paths). - - Anonymous browse of a known legacy URL form like - `/projects?ID=10` — must 301 to the slug URL. - - A team member signs in via GitHub OAuth; if their account is in - the laddr dataset they'll see the claim flow. - - Slack SAML: a team member logs out of Slack and re-authenticates - via the new IdP. Verify their Slack identity is preserved. -5. **0:15 — remove maintenance page.** The legacy site can stay up - read-only forever; we just need it not to be the public landing page. + - A known legacy URL form like `/projects?ID=10` — must 301 to the slug URL. + - A team member signs in via GitHub OAuth; a member with a laddr + account signs in with their legacy password. + - `https://next.codeforphilly.org` still works (alias). + - `https://codeforphilly.live.k8s.phl.io` still serves laddr. + +4. **0:15 — Slack SSO URL.** The Slack workspace admin updates **only the + SSO URL** in Slack's SAML settings from + `https://next.codeforphilly.org/api/saml/slack/sso` to + `https://codeforphilly.org/api/saml/slack/sso` (or re-fetches the + metadata from the apex — the entity ID and cert are unchanged). A team + member signs out of Slack and back in via SSO. If it fails, SSO is + "optional" so nobody is locked out; fix the URL rather than rolling back + the gateway. + +5. **0:20 — hot-reload target.** In the data repo, switch the prod URL in + `.github/workflows/notify-deployments.yml` from + `https://next.codeforphilly.org/...` to + `https://codeforphilly.org/api/_internal/reload-data`. (The alias would + keep working, but the workflow should name the canonical host.) + +6. **0:25 — legacy site.** If you put up a maintenance banner at T-7, it + can stay; laddr is now reachable only at + `codeforphilly.live.k8s.phl.io` and can remain read-only indefinitely. **Point of no return:** the moment **a real user creates a new record** -on the rewrite (a project, an update, a buzz post, even just their first -sign-in writing a `PrivateProfile`), data starts to diverge between the -two systems. Up to that point, a rollback (DNS flip back, see -[cutover-rollback.md](cutover-rollback.md)) is harmless. After it, a -rollback would discard those new records. +on the rewrite at the apex (a project, an update, a buzz post, even just +their first sign-in writing a `PrivateProfile`), data starts to diverge +between the two systems. Up to that point, reverting the cutover commit +([cutover-rollback.md](cutover-rollback.md)) is harmless. After it, a +rollback would discard those new records. Note that people have been able +to sign in at `next.codeforphilly.org` since before cutover — the rewrite's +data is already live and diverging from laddr's; "point of no return" here +is about writes that arrive *because* the apex moved. ## T+1h: monitoring window -For the first hour after DNS flip, the cutover lead and engineering second -watch: +**Be honest about what exists:** as of this writing there are no external +uptime checks and no alerts webhook ([monitoring.md](monitoring.md) +describes the target state, not the current one). For the first hour after +the hostname move, the cutover lead and engineering second watch by hand: -- `kubectl -n codeforphilly logs deploy/codeforphilly -f` — error rates - (look for spikes in `ERROR` or `WARN`). -- `/api/health` and `/api/health/ready` via UptimeRobot / healthchecks.io. -- The cluster log aggregator (if wired up) — search for stack traces. -- The `#alerts` Slack channel — anything fired by the WARN+ webhook. +- `kubectl -n codeforphilly-ng logs deploy/codeforphilly -f` — look for + spikes in `ERROR` or `WARN`. +- `curl -sS https://codeforphilly.org/api/health` and + `.../api/health/ready` every few minutes (a shell loop is fine). +- `kubectl -n codeforphilly-ng get pods -w` — restarts or OOMKills. Specific things to watch: - **OAuth callback failure rate.** Should be ~0%. A 4xx spike here - indicates a misconfigured redirect URI or a GitHub OAuth app issue. -- **SAML assertion failure rate.** Should be ~0%. A spike here means - the Slack integration is rejecting our assertions — usually a NameID - mismatch. -- **Push daemon errors.** Surface as `git push failed` in logs; means - the data repo's deploy key is wrong or rate-limited. + indicates a GitHub OAuth app issue (the callback URL itself should not + need changing — see T-7). +- **SAML assertion failure rate.** Should be ~0% once the SSO URL is + updated. A spike means Slack is rejecting our assertions — usually a + NameID or SSO URL mismatch. Members still have magic-link fallback. +- **Push daemon errors.** Surface as `git push failed` in logs; means the + data repo's deploy key is wrong or rate-limited. +- **Hot-reload webhook.** The next `published` push should log a + `hot-reload` line on the prod pod via the apex URL. If any of these alarm: pause, triage, decide rollback vs forward-fix in under 15 minutes. @@ -224,8 +361,8 @@ If everything is stable through the first day: 1. Post a success message to `#announcements` from [cutover-announcement.md](cutover-announcement.md). -2. Add a note to the rewrite README that legacy laddr is decommissioned. -3. Schedule the T+90 mailout to run automatically (see below). +2. Add a note to the rewrite README that legacy laddr is decommissioned + (or read-only at `codeforphilly.live.k8s.phl.io`). ## T+7 days: reconciliation check @@ -237,74 +374,64 @@ npm run -w apps/api script:reconcile After a week of real-world use, any orphans or inconsistencies that show up are likely from a real bug (dual-write coordination, racing OAuth -callbacks). File issues; fix before T+90. - -Schedule reconciliation weekly via a CI cron from this point on. Run the -script against production via `kubectl exec` into the deployed pod, or -configure a separate cron Deployment. - -## T+90 days: unclaimed-account mailout - -The active claim window from [account-migration.md](../../specs/behaviors/account-migration.md#cutover-window-policy) -ends at T+90. Send the reminder mail to anyone whose laddr account is -still unclaimed: - -1. **Dry-run first** to inspect the recipient list: - - ```bash - npm run -w apps/api script:cutover-mailout -- --dry-run \ - --json=/scratch/mailout-dry.json - ``` - -2. Spot-check the recipient list — sample 5 entries, confirm each - matches an unclaimed legacy Person. -3. Send: - - ```bash - POSTMARK_SERVER_TOKEN=... npm run -w apps/api script:cutover-mailout -- --send - ``` - -4. Monitor the Postmark activity stream for bounces. Hard bounces are expected — - defunct email providers are exactly why these accounts are - unclaimed. - -## T+180 days: drain legacy passwords - -At T+180 the remaining `LegacyPasswordCredential` records are deleted. -This is irreversible — any user who hadn't claimed by then must go through -the staff-approval path from [account-migration.md](../../specs/behaviors/account-migration.md#c-staff-approval). - -Run reconciliation with the legacy-purge follow-up (TBD: a small script -that calls `privateStore.deleteLegacyPassword` for every credential whose -Person is still unclaimed and whose `importedAt` is older than 180 days). +callbacks). File issues. + +**Open gap:** the reconcile script needs both the public data and the +private store. The runtime image ships only `dist/` (no `tsx`, no +`scripts/`), so it cannot run inside the prod pod as-is, and production +private data must not be copied to a laptop +([private-storage.md](../../specs/behaviors/private-storage.md)). Until the +script ships in the image or gets an in-pod entrypoint, reconciliation in +production is a manual, in-cluster exercise; treat this as a follow-up. + +## Legacy password sign-in stays on + +Nothing in this runbook retires legacy credentials. Migrated members keep +signing in with their laddr password for as long as they like; the only +nudge is the "Connect GitHub" banner on `/account` +([account-migration.md → The nag](../../specs/behaviors/account-migration.md#the-nag-banner-on-account)). +The spec tracks a coverage metric for *future* sunset planning; when the +numbers justify it, a separate spec change sets a date. Do not schedule a +mailout or a purge from this doc. ## Pre-cutover checklist Before T-0 the cutover lead confirms each of these in writing in the cutover Slack thread: -- [ ] DNS TTL lowered to 60s at least 7 days ago -- [ ] Sealed-secrets present in the production cluster (every var in - [secrets.md inventory](secrets.md)) -- [ ] GitHub OAuth app production redirect URI registered -- [ ] Slack SAML metadata pre-staged with workspace admin -- [ ] Data-repo deploy key uploaded to the data-repo's deploy-keys page -- [ ] Production S3 bucket exists with versioning enabled -- [ ] Staging dry-run from T-3 reports all stages passing +- [ ] SealedSecrets present and decrypted in `codeforphilly-ng` + (`codeforphilly-secrets`, `codeforphilly-saml`, + `codeforphilly-data-deploy-key`) — see [secrets.md](secrets.md) +- [ ] GitHub OAuth app "Code for Philly" exists; client ID matches the sealed value +- [ ] Slack SSO set to **optional**; "Test configuration" passed against + the `next.codeforphilly.org` metadata +- [ ] Data-repo deploy key has write access to `codeforphilly-data` +- [ ] `notify-deployments.yml` run green for both sandbox and prod on the last `published` push +- [ ] Credentials loaded on the prod PVC; a known legacy user can sign in + with their password at `next.codeforphilly.org` +- [ ] Sandbox dry-run from T-3 reports all stages passing +- [ ] Production pin (`.holo/sources/codeforphilly-ng.toml` + + `codeforphilly-ng/app/kustomization.yaml`) is the intended release +- [ ] The cutover commit is drafted on a branch in `cfp-live-cluster` and + reviewed (listeners, HTTPRoute hostnames, `CFP_SITE_HOST`) - [ ] On-call schedule covers T-0 through T+24h -- [ ] Rollback procedure rehearsed (see [cutover-rollback.md](cutover-rollback.md)) +- [ ] Rollback procedure read end-to-end (see [cutover-rollback.md](cutover-rollback.md)) +- [ ] **Monitoring: none wired.** Acknowledged that the T+1h window is + watched by hand; external checks + alerts are a post-cutover follow-up + ([monitoring.md](monitoring.md)) ## Known unknowns -- **Single big import commit size.** The data repo will land thousands - of `.toml` files in one push. Confirm with the gitsheets transaction - API that this works at scale; chunk by entity type (one commit per - sheet) if not. The importer already commits per-sheet. +- **Certificate issuance on the moved listeners.** The apex and `www` TLS + Secrets are namespaced, so cert-manager mints new ones in + `codeforphilly-ng` after the move. Let's Encrypt rate limits are + generous for two names, but a wedged ACME challenge would leave the apex + serving a bad cert — that is a revert trigger. - **GitHub OAuth rate limits.** A surge of users signing in within a cutover window could brush against GitHub's per-app rate limit - (~5000/hr). Probably fine at CFP scale (~1240 members); worth - monitoring. + (~5000/hr). Probably fine at CFP scale; worth monitoring. - **Newsletter email bounces.** First OAuth sign-in refreshes `PrivateProfile.email` to GitHub primary. Users who never sign in keep their pre-cutover email and may bounce on newsletter sends. Acceptable; cleaned up by standard bounce handling. +- **Reconcile in production** — see the open gap under T+7 days. From e2636b6681af511f02838a493b2d03f289f42182 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 10 Sep 2026 02:06:09 -0400 Subject: [PATCH 3/5] docs(ops): align deploy, secrets and runbook docs with live-cluster GitOps Image publishing is no longer manual and production is not a future `cfp-prod-cluster`: container-publish.yml pushes :vX.Y.Z on every release tag, and each cluster repo (cfp-sandbox-cluster, cfp-live-cluster) pins the app by .holo/sources/codeforphilly-ng.toml plus images[].newTag in codeforphilly-ng/app/kustomization.yaml, applied through the releases/k8s-manifests -> deploys/k8s-manifests PR. The hand-built :sandbox image is kept only as an emergency escape hatch. Private storage is a filesystem PVC in both environments; the S3 sections are marked supported-but-unused rather than "production". Secrets are the three SealedSecrets in cfp-live-cluster/codeforphilly-ng.secrets/ (codeforphilly-secrets, codeforphilly-saml carried over from laddr's saml2, the deploy key), edited with kubeseal --merge-into. Namespaces, the hot-reload secret name, the pod-clone volume type and the monitoring status note are corrected to match. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR --- .claude/CLAUDE.md | 29 ++-- docs/operations/deploy.md | 222 ++++++++++++++++---------- docs/operations/monitoring.md | 38 +++-- docs/operations/releases.md | 14 +- docs/operations/runbook.md | 142 ++++++++++------- docs/operations/sandbox-deploy.md | 163 ++++++++++++------- docs/operations/secrets.md | 252 +++++++++++++++++++----------- docs/operations/spam-detection.md | 10 +- 8 files changed, 555 insertions(+), 315 deletions(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 5d49f83..8a5627b 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -7,7 +7,7 @@ A modernization of [laddr](https://github.com/CodeForPhilly/laddr) (the platform These auto-trigger by topic — you don't load them manually. Mentioned here so you know what's already covered and don't duplicate. | Skill | Triggers on | What it covers | -|---|---|---| +| --- | --- | --- | | [`specops`](./skills/specops/SKILL.md) | `specs/`, `plans/`, "spec", "closeout commit", new features | Spec-driven workflow (specs are source of truth), plans-as-micro-DAG protocol, closeout commit ritual, follow-ups taxonomy, spec-drift auditor | | [`backend-fastify`](./skills/backend-fastify/SKILL.md) | New routes, services, plugins, env vars | Fastify 5 patterns, plugin ordering, `@fastify/env` validation, error handling | | [`frontend-shadcn`](./skills/frontend-shadcn/SKILL.md) | New screens, components, routing, styling | Vite + React 19 + shadcn/ui + Tailwind v4 + React Router v7 patterns | @@ -64,19 +64,23 @@ codeforphilly-data ─── Public data store. Branches: │ merge target). Hot-reload webhook │ fires on push. │ -cfp-sandbox-cluster ─── GitOps repo (hologit-projected) that pulls - │ this repo's `deploy/kustomize/` upstream - │ and applies it via Kustomize. See +cfp-sandbox-cluster ─── GitOps repos (hologit-projected) that pin a +cfp-live-cluster │ release tag of this repo, project its + │ `deploy/kustomize/base/` and apply it via + │ Kustomize. Sandbox = next-v2.codeforphilly.org + │ (ns codeforphilly-rewrite-sandbox); prod = + │ next.codeforphilly.org → codeforphilly.org at + │ cutover (ns codeforphilly-ng). See │ `docs/operations/deploy.md`. ``` -Operator docs in [`docs/operations/`](../docs/operations/): `deploy.md` for the cluster topology, `sandbox-deploy.md` for the manual procedure, `runbook.md` for incident response (including the hot-reload webhook). +Operator docs in [`docs/operations/`](../docs/operations/): `deploy.md` for the cluster topology + GitOps flow, `releases.md` for cutting a release, `sandbox-deploy.md` for the sandbox specifics, `runbook.md` for incident response (including the hot-reload webhook), `cutover.md` for the production hostname move. ## Stack - **Backend** — Fastify 5.x + TypeScript. Single replica, in-process write mutex. - **Public storage** — [gitsheets](https://github.com/JarvusInnovations/gitsheets) (TOML records in a git repo). Public-by-design — civic transparency. No persistent OLTP. See [specs/behaviors/storage.md](../specs/behaviors/storage.md). -- **Private storage** — S3-compatible bucket holding `.jsonl` files (private profiles + legacy password hashes). Boot-load + in-memory; PUT on mutation. See [specs/behaviors/private-storage.md](../specs/behaviors/private-storage.md). Real production private data never lands on a dev machine. +- **Private storage** — `.jsonl` files (private profiles + legacy password hashes) on a filesystem PVC in every deployed environment (`STORAGE_BACKEND=filesystem`); an S3-compatible backend exists but is unused. Boot-load + in-memory; rewrite on mutation. See [specs/behaviors/private-storage.md](../specs/behaviors/private-storage.md). Real production private data never lands on a dev machine. - **Schemas** — Zod in `packages/shared`, consumed by both web and api, validating records in both stores. - **Full-text search** — in-memory SQLite FTS5 (or MiniSearch fallback), rebuilt at boot from gitsheets state. - **Auth** — GitHub OAuth as the sole primary identity provider; stateless JWT sessions. We're also the SAML IdP for codeforphilly.slack.com. See [specs/api/auth.md](../specs/api/auth.md), [specs/api/saml.md](../specs/api/saml.md). @@ -135,13 +139,13 @@ npm run -w apps/web dev # web only ## Deploying -GitOps. This repo publishes a Docker image to GHCR; the [`cfp-sandbox-cluster`](https://github.com/CodeForPhilly/cfp-sandbox-cluster) repo pulls our [`deploy/kustomize/`](../deploy/kustomize/) upstream via hologit and applies it with Kustomize. Production will follow the same pattern under a `cfp-prod-cluster` repo. +GitOps. Releases publish a Docker image to GHCR; two cluster repos — [`cfp-sandbox-cluster`](https://github.com/CodeForPhilly/cfp-sandbox-cluster) (sandbox) and [`cfp-live-cluster`](https://github.com/CodeForPhilly/cfp-live-cluster) (production, alongside legacy laddr) — each pin a release tag, project our [`deploy/kustomize/base/`](../deploy/kustomize/base/) via hologit and apply it with Kustomize. Typical change flow: -1. **Merge to `main`** — CI builds + tests; nothing deploys yet. -2. **Publish image** (currently manual) — `docker build --platform=linux/amd64 -t ghcr.io/codeforphilly/codeforphilly-ng:sandbox . && docker push …`. Apple-silicon dev machines must set the platform flag — cluster nodes are amd64. -3. **GitOps pickup** — `cfp-sandbox-cluster` projects from our `deploy/kustomize/`; on its own merge, applies via `kubectl apply -k`. +1. **Merge to `develop`** — CI builds + tests; pushing `develop` opens/updates a `Release: vX.Y.Z` PR into `main` (see `docs/operations/releases.md` and the `release-flow` skill). +2. **Publish image** — merging the Release PR tags `vX.Y.Z`; `container-publish.yml` builds and pushes `ghcr.io/codeforphilly/codeforphilly-ng:vX.Y.Z` (+ `:latest`). No manual `docker push`. +3. **GitOps pickup** — in the cluster repo, bump `.holo/sources/codeforphilly-ng.toml` (`ref = "refs/tags/vX.Y.Z"`) and `images[].newTag` in `codeforphilly-ng/app/kustomization.yaml` together; merge to `main`; "Build k8s-manifests" projects to `releases/k8s-manifests`; a bot PR into `deploys/k8s-manifests` applies on merge. Sandbox first, then live. 4. **Pod boot** — single replica, `Recreate` strategy. Container entrypoint bare-clones the data repo on every pod start (the data volume is `emptyDir`, so first boot = every fresh pod). Node boots: env → store load → **reconcile** (ff/replay/escape-hatch against `origin/`) → **push daemon** → routes → SPA. `/api/health/ready` returns 200 once stores are loaded *and* reconciled. 5. **Live data updates** — independent of app deploy. Pushes to `published` trigger the [hot-reload webhook](../docs/operations/runbook.md#hot-reload-webhook); the pod rebuilds in-memory state in place, no restart. @@ -149,9 +153,10 @@ Constraints worth knowing before touching anything deploy-shaped: - **Single replica is non-negotiable** — writes flow through an in-process mutex; horizontal scale would lose write serialization. - **One image for API + SPA** — Fastify serves the built `apps/web/dist` as fallthrough. No separate web container. -- **Sealed secrets are cluster-bound** — sandbox-sealed secrets can't decrypt in prod. Plain values live in `.env` locally; cluster values live in the sandbox-cluster repo's `codeforphilly-ng.secrets/`. +- **Sealed secrets are cluster-bound** — sandbox-sealed secrets can't decrypt in prod. Plain values live in `.env` locally; cluster values live in each cluster repo's `codeforphilly-ng.secrets/`. +- **Cutover is a hostname move, not a DNS change** — the apex already resolves to the live cluster's gateway; T-0 is one commit in `cfp-live-cluster` moving listeners from `_gateways/code-for-philly.yaml` to `_gateways/codeforphilly-ng.yaml`. -Full operator docs: [`docs/operations/`](../docs/operations/) — `deploy.md` (boot sequence + env table), `sandbox-deploy.md` (manual bring-up), `secrets.md` (the secret contract), `runbook.md` (incident response + hot-reload webhook), `cutover.md` (production switch plan). +Full operator docs: [`docs/operations/`](../docs/operations/) — `deploy.md` (environments, GitOps flow, boot sequence, env table), `releases.md` (cutting a release), `sandbox-deploy.md` (sandbox specifics + manual escape hatch), `secrets.md` (the secret contract), `runbook.md` (incident response + hot-reload webhook), `cutover.md` (production hostname move), `legacy-credentials-import.md` (private-store seeding). ## Source control diff --git a/docs/operations/deploy.md b/docs/operations/deploy.md index 4945c72..3f78fb7 100644 --- a/docs/operations/deploy.md +++ b/docs/operations/deploy.md @@ -5,22 +5,36 @@ container. The authoritative architectural contract is [specs/architecture.md](../../specs/architecture.md#deploy); this document is the runbook that implements it. -> See also: [sandbox-deploy.md](sandbox-deploy.md) for the manual sandbox -> bring-up procedure, [secrets.md](secrets.md) for the secret contract, -> [runbook.md](runbook.md) for incident response. +> See also: [releases.md](releases.md) for how an image gets published, +> [sandbox-deploy.md](sandbox-deploy.md) for the sandbox cluster specifics +> and the manual escape hatch, [secrets.md](secrets.md) for the secret +> contract, [runbook.md](runbook.md) for incident response, +> [cutover.md](cutover.md) for the production hostname move. + +## Environments + +| | GitOps repo | Namespace | Host | Pinned at | +|---|---|---|---|---| +| **Production** | [`cfp-live-cluster`](https://github.com/CodeForPhilly/cfp-live-cluster) | `codeforphilly-ng` | `next.codeforphilly.org` (pre-cutover); `codeforphilly.org` after | `v0.2.0` as of 2026-09-10 | +| **Sandbox** | [`cfp-sandbox-cluster`](https://github.com/CodeForPhilly/cfp-sandbox-cluster) | `codeforphilly-rewrite-sandbox` | `next-v2.codeforphilly.org` | same tag, bumped ahead of prod | + +Legacy laddr runs in the live cluster too, in namespace `code-for-philly`. +There is no separate staging environment — the sandbox plays that role. ## TL;DR — anatomy ``` +----------------------+ -| GitHub Actions CI | ci.yml (build + test on PR / main) +| GitHub Actions | ci.yml (build + test on PR / develop) +| | release-*.yml (develop -> "Release: vX.Y.Z" PR -> tag) +| | container-publish.yml (on tag push) +----------+-----------+ - | docker build / push (manual today) + | docker build / push v +----------------------+ -| GHCR image | ghcr.io/codeforphilly/codeforphilly-ng: +| GHCR image | ghcr.io/codeforphilly/codeforphilly-ng:vX.Y.Z (+ :latest) +----------+-----------+ - | kubectl apply -k (via GitOps below) + | cluster repo bumps its pin -> "Build k8s-manifests" -> deploy PR -> apply v +----------------------+ | k8s Deployment | 1 replica, Recreate strategy, PVC + Secrets + ConfigMap @@ -37,36 +51,60 @@ The image holds **both** the API and the built SPA. There is no separate web container. The single replica is a hard architectural constraint ([specs/architecture.md](../../specs/architecture.md#process-model)). -## Manifests +## Release → image -Kustomize base + per-environment overlays at -[`deploy/kustomize/`](../../deploy/kustomize/). The base lives in -`deploy/kustomize/base/`; environment overlays under -`deploy/kustomize/overlays//`. +Image publishing is automated; nobody `docker push`es a release. The flow is +in [releases.md](releases.md): -The base ships everything the cluster needs in any environment: -`Deployment`, `Service`, `ConfigMap`, `PersistentVolumeClaim`s, `Gateway` + -`HTTPRoute` (per-env hostname patched in the overlay), `ServiceAccount`. -Sealed `Secret`s live only in overlays (sealed against the target cluster's -sealed-secrets controller). +1. Work merges into `develop`; pushing `develop` opens/updates a + `Release: vX.Y.Z` PR into `main`. +2. Merging that PR tags `vX.Y.Z`; the tag push triggers + `.github/workflows/container-publish.yml`, which builds and pushes + `ghcr.io/codeforphilly/codeforphilly-ng:vX.Y.Z` and `:latest`. -Cluster-level deploys are driven by the -[`cfp-sandbox-cluster`](https://github.com/CodeForPhilly/cfp-sandbox-cluster) -GitOps repo, which pulls the workload from this repo's main branch, composes -its own per-cluster Gateway/HTTPRoute (under `_gateways/codeforphilly-ng.yaml`) -and SealedSecrets (under `codeforphilly-ng.secrets/`), and applies on merge. -Production stand-up will follow the same pattern under a `cfp-prod-cluster` -repo. +The old hand-built `:sandbox` tag still exists as an emergency escape hatch +([sandbox-deploy.md](sandbox-deploy.md#emergency-escape-hatch-manual-image--apply)) +but is not how anything gets deployed. -For a one-shot manual apply (useful pre-GitOps or for an offline cluster): +## Image → cluster (GitOps) -```bash -kubectl apply -k deploy/kustomize/overlays/sandbox -``` +Both cluster repos work the same way: + +1. **Pin the release.** Two values in the cluster repo must move together: + - `.holo/sources/codeforphilly-ng.toml` — `ref = "refs/tags/vX.Y.Z"`. + hologit projects `deploy/kustomize/base/` from this repo at that tag. + - `codeforphilly-ng/app/kustomization.yaml` — `images[].newTag: vX.Y.Z`. +2. **Merge to `main`** in the cluster repo. The **"Build k8s-manifests"** + workflow projects the holobranch to `releases/k8s-manifests`, and a bot + opens a PR from there into `deploys/k8s-manifests`. +3. **Merge the deploy PR.** That applies the manifests. The `Recreate` + strategy swaps the pod; expect a short readiness gap while the new pod + re-clones the data repo and reloads state. + +Bump the sandbox first, let it soak, then bump prod. + +Per-cluster concerns live in the cluster repo, not here: the Gateway + +HTTPRoute (`_gateways/codeforphilly-ng.yaml`), the SealedSecrets +(`codeforphilly-ng.secrets/`), and the per-environment patches in +`codeforphilly-ng/app/kustomization.yaml` (`CFP_SITE_HOST`, +`imagePullPolicy`, the extra `codeforphilly-saml` `envFrom` in prod). + +## Manifests in this repo + +Kustomize base at [`deploy/kustomize/base/`](../../deploy/kustomize/base/): +`Deployment`, `Service`, `ConfigMap`, the private-storage +`PersistentVolumeClaim`, `Gateway` + `HTTPRoute` templates (hostname +placeholder), `ServiceAccount`. This is what the cluster repos project. + +[`deploy/kustomize/overlays/sandbox/`](../../deploy/kustomize/overlays/sandbox/) +is the pre-GitOps manual overlay (namespace, sealed secrets, hostname +patches). It is only used by the escape hatch in +[sandbox-deploy.md](sandbox-deploy.md); the sandbox cluster repo carries its +own equivalents. ## Image -### Build +### Build (local) ```bash docker build --platform=linux/amd64 \ @@ -77,9 +115,13 @@ Three stages — `deps` (full install), `build` (compile shared, api, web — in that order, since web/api consume shared's compiled output), `runtime` (alpine + git + ca-certificates + tini). Final image runs as `node` (uid 1000) per the `securityContext` in `deploy/kustomize/base/deployment.yaml`. +The runtime stage ships only `dist/` output and production deps — no +`tsx`, no `apps/api/scripts/` — so operator scripts can't be run inside a +pod. `--platform=linux/amd64` is required on Apple Silicon hosts — the cluster -nodes are amd64 and won't pull an arm64-only manifest. +nodes are amd64 and won't pull an arm64-only manifest. CI runners are amd64 +already, so `container-publish.yml` has no platform flag. ### Run (local smoke test) @@ -154,6 +196,9 @@ object DB via plumbing (`update-ref`, `merge-tree --write-tree`, When `CFP_DATA_REMOTE` is unset (typical local dev), the reconcile plugin skips reconciliation entirely. +The same path runs mid-life when the hot-reload webhook fires — see +[runbook.md#hot-reload-webhook](runbook.md#hot-reload-webhook). + ## Probes - **Liveness** — `GET /api/health` every 30s. The pod is killed only after @@ -182,33 +227,35 @@ Implications: default), the entrypoint relies on `GIT_SSH_COMMAND` (set in the ConfigMap) pointing at the mounted private key. Rotation: replace the SealedSecret, restart the pod. See - [secrets.md](secrets.md#data-repo-deploy-key) and the rotation - procedure in [sandbox-deploy.md](sandbox-deploy.md#rotating-the-deploy-key). - -## Bucket provisioning (production) - -Production uses an S3-compatible bucket for private storage -([specs/behaviors/private-storage.md](../../specs/behaviors/private-storage.md)). -The bucket is provisioned out-of-band and the manifests consume its -credentials via a SealedSecret. - -Recommended provider: **Cloudflare R2** (zero egress, pennies per month, -S3-compatible API). Backblaze B2 or AWS S3 also work. MinIO inside the -cluster is acceptable for cost reasons but trades operational simplicity -for storage simplicity. - -Required bucket configuration: - -- **Versioning enabled.** Hard requirement per - [private-storage.md](../../specs/behaviors/private-storage.md#bucket-requirements). -- **Lifecycle rule** deleting non-current versions after 365 days. -- **IAM policy** scoped to the bucket only — `s3:GetObject`, `s3:PutObject`, - `s3:ListBucket`, `s3:GetObjectVersion`. No cross-bucket access; no console - access for the service principal. -- **Endpoint URL** → `S3_ENDPOINT` (ConfigMap). -- **Bucket name** → `S3_BUCKET`. -- **Region** → `S3_REGION`. -- **Access keys** → `S3_ACCESS_KEY_ID` + `S3_SECRET_ACCESS_KEY` (Secret). + [secrets.md](secrets.md#data-repo-deploy-key). + +## Private storage + +Both sandbox and production run `STORAGE_BACKEND=filesystem` with +`CFP_PRIVATE_STORAGE_PATH=/app/private-storage` mounted from the +`codeforphilly-private` PersistentVolumeClaim +(`deploy/kustomize/base/pvc-private.yaml`, `linode-block-storage-retain`, +1Gi). The two `.jsonl` files there (`profiles.jsonl`, +`legacy-passwords.jsonl`) are loaded into memory at boot and rewritten +atomically on every private-side mutation. + +Implications: + +- **The PVC is the private store.** It survives pod restarts and image + bumps. The hot-reload webhook does **not** reload it — a pod restart is + the only way to pick up files placed there by hand + ([legacy-credentials-import.md](legacy-credentials-import.md)). +- **No versioning.** Unlike an object store there is no automatic history + of previous `.jsonl` versions. Snapshot the files (`kubectl cp` out of + the pod, kept in-cluster or on an operator machine only as long as + needed) before any manual operation that rewrites them. +- **Backups** are whatever the storage class's retain policy gives you. + There is no scheduled backup of the PVC today. + +The `s3` backend ([specs/behaviors/private-storage.md](../../specs/behaviors/private-storage.md#backends)) +remains fully supported and is the path if the private store ever needs to +outlive a single cluster, but **no environment uses it today**. The +`S3_*` variables below are documented for completeness only. ## Environment variables (reference) @@ -219,38 +266,47 @@ comments. Production pod gets these mounted: |----------|--------|-------| | `NODE_ENV` | ConfigMap | `production` | | `PORT` | ConfigMap | `3001` | -| `HOST` | ConfigMap | `0.0.0.0` | +| `NODE_OPTIONS` | ConfigMap | `--max-old-space-size=512` — see the comment in `configmap.yaml` for how that was sized | | `CFP_DATA_REPO_PATH` | ConfigMap | `/app/data` — bare gitdir, backed by an `emptyDir`; re-cloned on every pod boot | -| `CFP_DATA_REMOTE` | Secret | git URL (ssh in prod) | -| `CFP_DATA_BRANCH` | ConfigMap | e.g. `fixture` / `main` | -| `CFP_DATA_RELOAD_SECRET` | **Secret** | Shared bearer-token for the hot-reload webhook; when unset the `/api/_internal/reload-data` endpoint returns 503. See [runbook.md](runbook.md#hot-reload-webhook). | +| `CFP_DATA_REMOTE` | **Secret** (`codeforphilly-secrets`) | git URL (ssh in sandbox + prod) | +| `CFP_DATA_BRANCH` | ConfigMap | `published` in sandbox + prod (the data repo has no `main`). `fixture` / `legacy-import` for debug pods | +| `CFP_DATA_RELOAD_SECRET` | **Secret** (`codeforphilly-secrets`) | Shared bearer token for the hot-reload webhook; when unset the `/api/_internal/reload-data` endpoint returns 503. Must match the `CFP_DATA_RELOAD_SECRET` Actions secret in the data repo. See [runbook.md](runbook.md#hot-reload-webhook). | | `CFP_WEB_DIST_PATH` | ConfigMap | `/app/apps/web/dist` | -| `CFP_SITE_HOST` | ConfigMap | Public-facing host (`codeforphilly.org` base, `next-v2.codeforphilly.org` sandbox). Drives the markdown renderer's external-link transform — anchors with a different host get `target="_blank" rel="noopener nofollow"` — and the SAML IdP metadata's `SingleSignOnService` endpoint URLs. | -| `SAML_ENTITY_ID` | ConfigMap | Optional. Stable SAML IdP entity ID / assertion `Issuer` (default `https://codeforphilly.org/api/saml/slack/metadata`). Leave unset everywhere Slack should keep trusting the production IdP identity — it deliberately does **not** follow `CFP_SITE_HOST`, so flipping the host at cutover doesn't require editing Slack's SAML config. Only set it when standing up a separate IdP registration (e.g. a sandbox pointed at a test workspace). See [specs/api/saml.md](../../specs/api/saml.md#idp-identity-and-hosts). | -| `POSTMARK_SERVER_TOKEN` | **Secret** | Postmark server API token for outbound notifications. When unset, the email notifier falls back to a no-op LoggingNotifier — convenient for dev + tests but means no real emails go out. | +| `CFP_SITE_HOST` | ConfigMap (patched per cluster) | Public-facing host: `next.codeforphilly.org` in prod until cutover, `codeforphilly.org` after; `next-v2.codeforphilly.org` in sandbox. Drives the markdown renderer's external-link transform, canonical URLs in email, and the SAML IdP metadata's `SingleSignOnService` endpoint URLs. | +| `SAML_ENTITY_ID` | ConfigMap | Optional. Stable SAML IdP entity ID / assertion `Issuer` (default `https://codeforphilly.org/api/saml/slack/metadata`). Leave unset everywhere Slack should keep trusting the production IdP identity — it deliberately does **not** follow `CFP_SITE_HOST`, so flipping the host at cutover doesn't require re-registering with Slack. Only set it when standing up a separate IdP registration. See [specs/api/saml.md](../../specs/api/saml.md#idp-identity-and-hosts). | +| `POSTMARK_SERVER_TOKEN` | **Secret** (`codeforphilly-secrets`) | Postmark server API token for outbound notifications. When unset, the email notifier falls back to a no-op LoggingNotifier — convenient for dev + tests but means no real emails go out. | | `POSTMARK_MESSAGE_STREAM` | ConfigMap | Postmark message stream for outbound mail (default `outbound`). Must exist on the server the token belongs to. | -| `CFP_NOTIFICATION_FROM` | ConfigMap | RFC 5322 sender address for outbound notifications (default `"Code for Philly "`). Sender domain must be a verified Postmark sender signature (already true for `codeforphilly.org` via the legacy site) before flipping `POSTMARK_SERVER_TOKEN` on. | -| `STORAGE_BACKEND` | ConfigMap | `s3` (prod) / `filesystem` (sandbox) | -| `CFP_PRIVATE_STORAGE_PATH` | ConfigMap | `/app/private-storage` (when filesystem) | -| `S3_ENDPOINT` / `S3_BUCKET` / `S3_REGION` | ConfigMap | Bucket addressing | -| `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` | **Secret** | Bucket credentials | -| `GITHUB_OAUTH_CLIENT_ID` | **Secret** | OAuth app client ID | -| `GITHUB_OAUTH_CLIENT_SECRET` | **Secret** | OAuth app client secret | -| `CFP_JWT_SIGNING_KEY` | **Secret** | HS256 key (`openssl rand -base64 64`) | -| `SAML_PRIVATE_KEY` / `SAML_CERTIFICATE` | **Secret** | Slack IdP cert chain | +| `CFP_NOTIFICATION_FROM` | ConfigMap | RFC 5322 sender address for outbound notifications (default `"Code for Philly "`). Sender domain must be a verified Postmark sender signature (already true for `codeforphilly.org` via the legacy site). | +| `STORAGE_BACKEND` | ConfigMap | `filesystem` in sandbox **and** prod. `s3` is supported but unused. | +| `CFP_PRIVATE_STORAGE_PATH` | ConfigMap | `/app/private-storage` (the PVC mount) | +| `S3_ENDPOINT` / `S3_BUCKET` / `S3_REGION` | — | Only when `STORAGE_BACKEND=s3`. Not set anywhere today. | +| `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` | — | Only when `STORAGE_BACKEND=s3`. Not set anywhere today. | +| `GITHUB_OAUTH_CLIENT_ID` | **Secret** (`codeforphilly-secrets`) | OAuth app client ID (not sensitive, but it ships alongside the secret for one-place rotation) | +| `GITHUB_OAUTH_CLIENT_SECRET` | **Secret** (`codeforphilly-secrets`) | OAuth app client secret | +| `CFP_JWT_SIGNING_KEY` | **Secret** (`codeforphilly-secrets`) | HS256 key (`openssl rand -base64 64`) | +| `SAML_PRIVATE_KEY` / `SAML_CERTIFICATE` | **Secret** (`codeforphilly-saml`, prod) | Slack IdP key pair, carried over verbatim from laddr's `saml2` secret so Slack keeps trusting the same cert. Mounted via an extra `envFrom` patch in the live cluster repo; marked `optional` so a sandbox without it still boots. | | `SLACK_TEAM_HOST` | ConfigMap | Slack workspace host (default `codeforphilly.slack.com`). ACS URL, NameID `NameQualifier`, `/chat` + `/launch` redirect target. Never our own IdP identity. | | `GIT_SSH_COMMAND` | ConfigMap | Wires `ssh` to the mounted deploy key | +| `GIT_AUTHOR_NAME` / `GIT_AUTHOR_EMAIL` | ConfigMap | Committer identity for API-authored data commits | ## Rollback -Two distinct rollback flavors: - -- **Pod / image rollback** — change the image tag in the GitOps repo's - `images:` override (or, for an out-of-band hotfix, `kubectl set image - deployment/codeforphilly ...`). The deployment's `Recreate` strategy - serializes the swap; a few seconds of `503` on the readiness probe is - expected while the new pod boots. +Three distinct rollback flavors: + +- **Image / version rollback** — move the two pins in the cluster repo + (`.holo/sources/codeforphilly-ng.toml` `ref` and + `codeforphilly-ng/app/kustomization.yaml` `newTag`) back to the previous + release tag and run it through the deploy PR. For an out-of-band hotfix, + `kubectl -n set image deploy/codeforphilly + codeforphilly=ghcr.io/codeforphilly/codeforphilly-ng:` + and reconcile the repo afterwards. The `Recreate` strategy serializes the + swap; a few seconds of `503` on the readiness probe is expected while the + new pod boots. +- **Hostname rollback** (cutover only) — `git revert` the cutover commit in + `cfp-live-cluster`. See [cutover-rollback.md](cutover-rollback.md). - **Data rollback** — `git revert` (or `git push --force-with-lease` after - a careful local rebase) on the data repo. The next pod-boot entrypoint - reconciliation will pick up the change. Don't conflate the two: rolling - the image back does not undo data writes the API has already pushed. + a careful local rebase) on the data repo's `published` branch. The push + fires the hot-reload webhook, so the running pod picks it up without a + restart. Don't conflate the flavors: rolling the image back does not undo + data writes the API has already pushed, and neither touches the private + store on the PVC. diff --git a/docs/operations/monitoring.md b/docs/operations/monitoring.md index ecb1479..48d1869 100644 --- a/docs/operations/monitoring.md +++ b/docs/operations/monitoring.md @@ -1,6 +1,15 @@ # Post-cutover monitoring -The minimum-viable monitoring wired up before cutover. The bar isn't +> **Status (2026-09-10): none of the external monitoring below exists yet.** +> There are no uptime checks, no log→Slack webhook, no `#alerts` integration. +> The only things that are real today are the Kubernetes liveness/readiness +> probes in `deploy/kustomize/base/deployment.yaml` and `kubectl logs`. The +> cutover runbook's T+1h window is watched by hand for that reason +> ([cutover.md → T+1h](cutover.md#t1h-monitoring-window)). Treat the rest of +> this document as the target to build toward, not a description of what's +> running. + +The minimum-viable monitoring we want in place. The bar isn't "observable in detail" — it's "we know within 60 seconds if the site is down, and within 5 minutes if an error rate is climbing." @@ -39,7 +48,7 @@ Configure two monitors: | Monitor | URL | Interval | Alert when | |---------|-----|----------|-----------| -| codeforphilly.org liveness | `https://codeforphilly.org/api/health` | 1 min | 2 consecutive failures (≈ 2 min) | +| codeforphilly.org liveness | `https://codeforphilly.org/api/health` (`https://next.codeforphilly.org/...` until cutover) | 1 min | 2 consecutive failures (≈ 2 min) | | codeforphilly.org readiness | `https://codeforphilly.org/api/health/ready` | 5 min | 1 failure (≈ 5 min) | Both alarm via the UptimeRobot → Slack integration to the `#alerts` channel. @@ -60,14 +69,15 @@ The two layers are complementary: - External monitors: "is the site reachable from the public internet?" A pod can pass k8s probes but the public hostname can still be unreachable -(ingress misconfigured, cert-manager wedged, DNS broken). The external +(gateway misconfigured, cert-manager wedged, DNS broken). The external monitor catches those. ## 3. Log aggregation Pino logs go to stdout from the API process. The cluster's log aggregator (whatever is configured — at minimum `kubectl logs` works; ideally a -shipper to a hosted log store like BetterStack or Grafana Loki) collects +shipper to a hosted log store like BetterStack or Grafana Loki — the live +cluster does run Grafana, so Loki is the natural first look) collects them. Required log levels in production: @@ -104,12 +114,12 @@ The fix is two-layer: 1. In the API's push job, emit `info` on success and `error` on failure. The Pino error gets webhooked to Slack like any other. -2. A daily check: "is the data repo's `origin/main` HEAD within 24h of the - API pod's local HEAD?" Tooling TBD; could be a small `kubectl exec` +2. A daily check: "is the data repo's `origin/published` HEAD within 24h of + the API pod's local HEAD?" Tooling TBD; could be a small `kubectl exec` cron or a server-side check exposed at `/api/health/push-daemon`. -For cutover-prep we ship the level-1 layer (push errors are already -loggable). The daily check is deferred to a post-cutover follow-up. +Layer 1 exists (push errors are logged). The daily check is deferred to a +post-cutover follow-up. ## What we are deliberately NOT monitoring at v1 @@ -126,20 +136,22 @@ loggable). The daily check is deferred to a post-cutover follow-up. These are deferred to when we have a specific need, not pre-emptively. -## Pre-cutover monitoring checklist +## Monitoring checklist -The cutover lead confirms before T-0: +None of these are done as of 2026-09-10. Cutover proceeds without them, +watched by hand; ticking them off is the first post-cutover ops task. - [ ] UptimeRobot account exists; two monitors above are configured - [ ] UptimeRobot → `#alerts` Slack integration is fired by a test alarm -- [ ] k8s liveness + readiness probes are present in `deploy/kustomize/base/deployment.yaml` +- [x] k8s liveness + readiness probes are present in `deploy/kustomize/base/deployment.yaml` - [ ] Log webhook → `#alerts` integration fires on a test `WARN` line -- [ ] On-call rotation is set in PagerDuty / Slack handoff doc +- [ ] On-call rotation is set in a Slack handoff doc - [ ] At least one team member can reach `#alerts` outside business hours The "test alarm" step is non-negotiable: untested alerts have a well-documented tendency to silently not fire when they're needed most. -Trigger each one once in staging and confirm a Slack message arrived. +Trigger each one once against the sandbox and confirm a Slack message +arrived. ## Post-cutover: when to add more diff --git a/docs/operations/releases.md b/docs/operations/releases.md index f005317..ec7860e 100644 --- a/docs/operations/releases.md +++ b/docs/operations/releases.md @@ -30,9 +30,12 @@ feature branch ──▶ develop ──(push)──▶ "Release: vX.Y.Z" PR into 4. **Merge the Release PR.** `release-publish.yml` creates the `vX.Y.Z` tag and GitHub release. The tag push triggers `container-publish.yml`, which builds and pushes `ghcr.io/codeforphilly/codeforphilly-ng:vX.Y.Z` and `:latest`. -5. **Deploy.** The cluster picks up the image per - [deploy.md](./deploy.md) (the published `:vX.Y.Z` / `:latest` tags replace - the previously-manual `:sandbox` build for versioned releases). +5. **Deploy.** In each cluster repo (`cfp-sandbox-cluster` first, then + `cfp-live-cluster`), bump `.holo/sources/codeforphilly-ng.toml`'s `ref` to + `refs/tags/vX.Y.Z` **and** `images[].newTag` in + `codeforphilly-ng/app/kustomization.yaml` in one commit, merge to `main`, + then merge the bot's PR into `deploys/k8s-manifests`. Details in + [deploy.md](./deploy.md#image--cluster-gitops). ## Prerequisites (one-time) @@ -55,7 +58,8 @@ you want a different baseline, edit the Release PR title before merging. ## Notes - The manual `docker build --platform=linux/amd64 … :sandbox` path - ([sandbox-deploy.md](./sandbox-deploy.md)) still works for ad-hoc iteration; - versioned releases now go through `container-publish` instead. + ([sandbox-deploy.md](./sandbox-deploy.md#emergency-escape-hatch-manual-image--apply)) + is an emergency escape hatch only; every deployed version goes through + `container-publish` and a cluster-repo pin bump. - CI runners and cluster nodes are both amd64, so `container-publish` needs no `--platform` flag (that's only for local Apple-silicon builds). diff --git a/docs/operations/runbook.md b/docs/operations/runbook.md index 253b8de..c5f8ffb 100644 --- a/docs/operations/runbook.md +++ b/docs/operations/runbook.md @@ -1,9 +1,20 @@ # Runbook -On-call playbooks for the codeforphilly-rewrite production service. +On-call playbooks for the codeforphilly-ng production service. -> See also: [monitoring.md](monitoring.md) for the alarm signals that page -> on-call, [cutover.md](cutover.md) for cutover-specific procedures. +> See also: [monitoring.md](monitoring.md) for the alarm signals we *want* +> (none are wired yet), [cutover.md](cutover.md) for cutover-specific +> procedures, [deploy.md](deploy.md) for the GitOps flow. + +## Where to point kubectl + +| | Cluster repo | Namespace | +| --- | --- | --- | +| Production | `cfp-live-cluster` | `codeforphilly-ng` | +| Sandbox | `cfp-sandbox-cluster` | `codeforphilly-rewrite-sandbox` | + +Commands below use `-n codeforphilly-ng`; substitute the sandbox namespace +as needed. The Deployment is `deploy/codeforphilly` in both. ## "API won't boot" @@ -13,24 +24,25 @@ restarting, no `/api/health` response. ### 1. Read the logs first ```bash -kubectl -n codeforphilly logs deploy/codeforphilly --previous -kubectl -n codeforphilly logs deploy/codeforphilly --tail=200 +kubectl -n codeforphilly-ng logs deploy/codeforphilly --previous +kubectl -n codeforphilly-ng logs deploy/codeforphilly --tail=200 ``` -Look for one of the four common boot failures: +Look for one of the common boot failures: | Log line excerpt | Cause | Fix | -|------------------|-------|-----| -| `[entrypoint] ERROR: CFP_DATA_REMOTE is unset` | The Secret containing `CFP_DATA_REMOTE` isn't reaching the pod. | Check `kubectl get secret codeforphilly-secrets -o yaml`; verify the SealedSecret in the GitOps repo decrypted successfully (look at the sealed-secrets controller logs). | +| ------------------ | ------- | ----- | +| `[entrypoint] ERROR: CFP_DATA_REMOTE is unset` | The Secret containing `CFP_DATA_REMOTE` isn't reaching the pod. | Check `kubectl -n codeforphilly-ng get secret codeforphilly-secrets -o yaml`; verify the SealedSecret in the GitOps repo decrypted successfully (look at the sealed-secrets controller logs). | | `fatal: could not read Username for 'https://...'` or `Permission denied (publickey)` | Bad/missing data-repo credentials. | Verify the `codeforphilly-data-deploy-key` Secret holds a valid `id_ed25519` whose public key has push access to the data repo. See [secrets.md](secrets.md#data-repo-deploy-key). | | `Failed to open public gitsheets store` | Bare clone corrupt or missing `.gitsheets/` configs. | Exec into the pod, inspect `/app/data/refs/`, `/app/data/objects/`, and verify `.gitsheets/` exists in HEAD via `git --git-dir=/app/data show HEAD:.gitsheets`. Recovery: restart the pod — `data` is an `emptyDir`, so a fresh pod re-clones from `CFP_DATA_REMOTE` automatically. | -| `Failed to load private store (s3)` | Bucket creds wrong, bucket gone, or network ACL blocks egress. | Confirm `S3_*` env in the ConfigMap + Secret. From the pod, `curl $S3_ENDPOINT` to confirm reachability. | +| `Failed to load private store (filesystem)` | The `codeforphilly-private` PVC didn't mount, is read-only, or the `.jsonl` files are malformed. | `kubectl -n codeforphilly-ng describe pod` for volume events; exec in and `ls -la /app/private-storage`. A malformed line points at a bad manual load — see [legacy-credentials-import.md](legacy-credentials-import.md). | | `environment variable ... is required` | A required env (`CFP_DATA_REPO_PATH`, `STORAGE_BACKEND`, `CFP_JWT_SIGNING_KEY`) is missing. | Manifest regression. Compare against `deploy/kustomize/base/configmap.yaml` + the GitOps repo's SealedSecret. | +| OOMKilled (no log line; `kubectl describe pod` shows it) | In-memory record set too big for `NODE_OPTIONS`/`resources.limits`. Usually an unpruned `published` push. | Check whether the last `published` push skipped the prune step ([spam-detection.md](spam-detection.md)). Re-prune and push; the hot-reload will pick it up once the pod is up. | ### 2. Drop into the pod (if it stays up long enough) ```bash -kubectl -n codeforphilly debug -it deploy/codeforphilly \ +kubectl -n codeforphilly-ng debug -it deploy/codeforphilly \ --image=alpine --target=api -- sh ``` @@ -42,12 +54,12 @@ From inside: ls -la /app/data git --git-dir=/app/data show HEAD:.gitsheets -# Are env vars present? -env | grep -E '^(CFP_|S3_|STORAGE_|GITHUB_)' | sort +# Is the private store there? +ls -la /app/private-storage +wc -l /app/private-storage/*.jsonl -# Can we reach the bucket? -apk add --no-cache curl -curl -v "$S3_ENDPOINT" +# Are env vars present? +env | grep -E '^(CFP_|STORAGE_|GITHUB_|SAML_|POSTMARK_)' | sort # Can we reach the data remote? git ls-remote "$CFP_DATA_REMOTE" 2>&1 | head @@ -59,24 +71,27 @@ If the cluster state is unrecoverable but the data remote is intact: ```bash # Revert the most recent GitOps deploy (the cluster repo's deploy PR is a -# normal merge commit on `deploys/k8s-manifests`) -gh -R CodeForPhilly/cfp-sandbox-cluster pr list --base deploys/k8s-manifests --state merged -git -C ~/Repositories/cfp-sandbox-cluster revert --mainline 1 -git -C ~/Repositories/cfp-sandbox-cluster push origin deploys/k8s-manifests - -# Or pin to a previous image by editing the GitOps repo's -# .holo/branches/k8s-manifests/codeforphilly-ng/app/manifests.toml's image -# tag, committing on a hotfix branch, and merging through the deploy PR. +# normal merge commit on `deploys/k8s-manifests`). Use cfp-sandbox-cluster +# for the sandbox. +gh-axi -R CodeForPhilly/cfp-live-cluster pr list --base deploys/k8s-manifests --state merged +git -C ~/Repositories/cfp-live-cluster revert --mainline 1 +git -C ~/Repositories/cfp-live-cluster push origin deploys/k8s-manifests + +# Or pin to a previous release: move BOTH +# .holo/sources/codeforphilly-ng.toml (ref = "refs/tags/vX.Y.Z") +# codeforphilly-ng/app/kustomization.yaml (images[].newTag) +# back to the last known-good tag on a branch, PR into main, merge the +# resulting deploy PR. # Out-of-band hotfix (bypasses GitOps — fix the repo afterward): -kubectl -n codeforphilly-rewrite-sandbox set image \ +kubectl -n codeforphilly-ng set image \ deploy/codeforphilly codeforphilly=ghcr.io/codeforphilly/codeforphilly-ng: ``` The bare data clone lives in an `emptyDir` — re-cloned from the git remote on -every pod boot. Pod restart is the recovery primitive; there's no PVC to -delete. (A `codeforphilly-private` PVC still exists for the S3-fallback -private store; only `codeforphilly-data` was retired.) +every pod boot. Pod restart is the recovery primitive; there's no data PVC to +delete. The `codeforphilly-private` PVC **is** the private store — never +delete it as part of a recovery. ## "Readiness flapping / 503 spikes" @@ -86,7 +101,8 @@ likely means: - Liveness probe (`/api/health`) failed and k8s is restarting the pod. Look at the previous logs. -- Memory pressure → OOMKilled. Bump `resources.limits.memory` or lower +- Memory pressure → OOMKilled. Check for an unpruned `published` push first + (see the boot table above); only then bump `resources.limits.memory` / `NODE_OPTIONS=--max-old-space-size`. ## "Mutations succeed in UI but don't appear on GitHub" @@ -97,16 +113,22 @@ Push daemon failure. Check logs for git push errors. Common causes: - Remote branch protection rejecting the push. - Network egress blocked. -The local working tree continues to accept writes — it's only the -asynchronous mirror to GitHub that's broken. Once fixed, the daemon will -push the backlog. +The in-memory state and the pod's bare clone continue to accept writes — +it's only the asynchronous mirror to GitHub that's broken. Once fixed, the +daemon pushes the backlog. **Don't restart the pod to "fix" this** until the +backlog has been pushed or fetched out (below): the clone is an `emptyDir` +and unpushed commits die with the pod. ## Hot-reload webhook The API exposes `POST /api/_internal/reload-data` so that a push to -`CFP_DATA_BRANCH` (typically `published`) propagates to the running pod -without rolling it. The `codeforphilly-data` repo's -`notify-deployments.yml` workflow calls this endpoint on every push. +`CFP_DATA_BRANCH` (`published`) propagates to the running pod without +rolling it. The `codeforphilly-data` repo's +`.github/workflows/notify-deployments.yml` calls this endpoint on every push +to `published`, once per target: sandbox (`next-v2.codeforphilly.org`) and +prod (`next.codeforphilly.org` before cutover; the apex after — the +workflow's prod URL must be switched at cutover, see +[cutover.md](cutover.md#t-0-cutover)). See [specs/behaviors/storage.md#hot-reload](../../specs/behaviors/storage.md#hot-reload) for the authoritative contract. Operationally: @@ -114,16 +136,20 @@ for the authoritative contract. Operationally: - **Configured by** the `CFP_DATA_RELOAD_SECRET` env variable. When unset, the endpoint is still registered but returns 503 — the workflow's `curl` will exit non-zero and the operator must - investigate. The secret value lives in the GitOps repo's - `cfp-sandbox-cluster/codeforphilly-ng.secrets/` (or production - equivalent) as a sealed Secret — not in the app repo. + investigate. The value lives in each cluster repo's + `codeforphilly-ng.secrets/codeforphilly-secrets.yaml` (sealed) and as the + `CFP_DATA_RELOAD_SECRET` Actions secret on the data repo — all three must + agree. +- **Public data only.** The webhook reloads the gitsheets state + FTS. It + does **not** reload the private store; files placed on the PVC need a + `rollout restart`. - **Trigger manually** for debugging: ```bash - SECRET=$(kubectl -n codeforphilly-rewrite-sandbox get secret \ - codeforphilly-ng -o jsonpath='{.data.CFP_DATA_RELOAD_SECRET}' | base64 -d) + SECRET=$(kubectl -n codeforphilly-ng get secret \ + codeforphilly-secrets -o jsonpath='{.data.CFP_DATA_RELOAD_SECRET}' | base64 -d) - curl -sS -X POST https://next-v2.codeforphilly.org/api/_internal/reload-data \ + curl -sS -X POST https://next.codeforphilly.org/api/_internal/reload-data \ -H "Authorization: Bearer $SECRET" \ -H "Content-Type: application/json" \ -d '{}' | jq @@ -145,16 +171,17 @@ for the authoritative contract. Operationally: in an undefined state — restart the pod: ```bash - kubectl -n codeforphilly-rewrite-sandbox rollout restart deploy/codeforphilly + kubectl -n codeforphilly-ng rollout restart deploy/codeforphilly ``` - **Outside-the-pod observability** — every reload logs an info line with the outcome + commits; failures log error. Search the pod logs - for `hot-reload` to audit the most recent firings. + for `hot-reload` to audit the most recent firings. The data repo's + Actions run for `notify-deployments.yml` shows the HTTP status per target. ## Fetch from the pod's data clone -The pod's working tree lives on a PVC at `/app/data` inside the container and may briefly hold commits the push daemon hasn't shipped to GitHub yet (or that got escape-hatched onto a `conflicts/*` branch — those *are* always pushed, see [`apps/api/src/store/reconcile.ts`](../../apps/api/src/store/reconcile.ts)). When you want to inspect the pod's view of the data repo without exposing any network ports, add it as a git remote via the `ext::` transport. +The pod's bare clone lives in an `emptyDir` at `/app/data` and may briefly hold commits the push daemon hasn't shipped to GitHub yet (or that got escape-hatched onto a `conflicts/*` branch — those *are* always pushed, see [`apps/api/src/store/reconcile.ts`](../../apps/api/src/store/reconcile.ts)). When you want to inspect the pod's view of the data repo without exposing any network ports, add it as a git remote via the `ext::` transport. ```bash cd /path/to/codeforphilly-data @@ -163,17 +190,17 @@ git config protocol.ext.allow always git remote add pod 'ext::sh /path/to/codeforphilly-ng/scripts/git-pod-uploadpack.sh' git fetch pod -git log --oneline pod/published..pod/published # whatever you're chasing +git log --oneline origin/published..pod/published # unpushed commits, if any ``` -The helper script resolves the current pod by label selector, so it survives restarts. Override via env if your setup differs: +The helper script resolves the current pod by label selector, so it survives restarts. Override via env for the cluster you're targeting: -| Var | Default | -|---|---| -| `CFP_POD_KUBECONFIG` | `~/.kube/cfp-sandbox-cluster-kubeconfig.yaml` | -| `CFP_POD_NAMESPACE` | `codeforphilly-rewrite-sandbox` | -| `CFP_POD_SELECTOR` | `app.kubernetes.io/name=codeforphilly` | -| `CFP_POD_DATA_PATH` | `/app/data` | +| Var | Default | Production | +| --- | --- | --- | +| `CFP_POD_KUBECONFIG` | `~/.kube/cfp-sandbox-cluster-kubeconfig.yaml` | the live cluster kubeconfig | +| `CFP_POD_NAMESPACE` | `codeforphilly-rewrite-sandbox` | `codeforphilly-ng` | +| `CFP_POD_SELECTOR` | `app.kubernetes.io/name=codeforphilly` | same | +| `CFP_POD_DATA_PATH` | `/app/data` | same | **Read-only by design** — `git upload-pack` only serves fetch; pushing back to the pod would bypass gitsheets + in-memory state and fight the push daemon. Pull what you need to your local clone, reason about it there, then push to `origin` if appropriate. @@ -181,14 +208,17 @@ The helper script resolves the current pod by label selector, so it survives res ```bash # Watch a deploy -kubectl -n codeforphilly rollout status deploy/codeforphilly +kubectl -n codeforphilly-ng rollout status deploy/codeforphilly # Last 10 GitOps deploys (merge commits on deploys/k8s-manifests) -gh -R CodeForPhilly/cfp-sandbox-cluster pr list --base deploys/k8s-manifests --state merged --limit 10 +gh-axi -R CodeForPhilly/cfp-live-cluster pr list --base deploys/k8s-manifests --state merged --limit 10 + +# Which release is running? +kubectl -n codeforphilly-ng get deploy codeforphilly -o jsonpath='{.spec.template.spec.containers[0].image}' # Pod resource use -kubectl -n codeforphilly top pod +kubectl -n codeforphilly-ng top pod -# Force a config reload -kubectl -n codeforphilly rollout restart deploy/codeforphilly +# Force a config / secret reload (env is read at pod start) +kubectl -n codeforphilly-ng rollout restart deploy/codeforphilly ``` diff --git a/docs/operations/sandbox-deploy.md b/docs/operations/sandbox-deploy.md index 8997230..51d61ad 100644 --- a/docs/operations/sandbox-deploy.md +++ b/docs/operations/sandbox-deploy.md @@ -1,101 +1,156 @@ -# Manual sandbox deploy +# Sandbox deploy -This is the manual procedure for iterating on a deploy to the **CfP sandbox cluster** (Linode LKE, k8s.phl.io). GitOps wiring is a follow-up; this doc is the source of truth until that lands. +The **CfP sandbox cluster** (Linode LKE, `k8s.phl.io`) hosts the rewrite's +sandbox at . It plays the "staging" role: +every release is bumped here first and soaks before production. + +**Deploys are GitOps.** The +[`cfp-sandbox-cluster`](https://github.com/CodeForPhilly/cfp-sandbox-cluster) +repo pins a release of this repo and applies it; the manual +`docker build … :sandbox && kubectl apply -k` procedure this doc used to +describe is now an emergency escape hatch only (bottom of this page). See +[deploy.md](deploy.md) for the cross-environment picture and +[releases.md](releases.md) for how an image gets published. ## Cluster - **Kubeconfig:** `~/.kube/cfp-sandbox-cluster-kubeconfig.yaml` -- **Gateway:** Envoy Gateway (`gatewayClassName: eg`), wildcard DNS for `*.sandbox.k8s.phl.io` → `139.144.241.4`. The sandbox app is reachable at `next-v2.codeforphilly.org` via a CNAME to `sandbox.k8s.phl.io`. +- **Namespace:** `codeforphilly-rewrite-sandbox` +- **Gateway:** Envoy Gateway (`gatewayClassName: eg`). `*.sandbox.k8s.phl.io` + points at the sandbox LB; `next-v2.codeforphilly.org` is a CNAME onto it, + managed in `CodeForPhilly/ops` (`tf/dns`). - **Storage class:** `linode-block-storage-retain` (default) - **Sealed-secrets:** controller in `sealed-secrets` namespace -- **cert-manager:** `letsencrypt-staging` + `letsencrypt-prod` ClusterIssuers. Sandbox uses prod; per-overlay can override to staging for high-churn iteration (prod rate-limits to 50 certs/week per registered domain). +- **cert-manager:** `letsencrypt-staging` + `letsencrypt-prod` ClusterIssuers. + Sandbox uses prod; switch the Gateway annotation to staging for high-churn + iteration (prod rate-limits to 50 certs/week per registered domain). ## Data repo -The app reads its gitsheets data from a private GitHub repo cloned at boot: - -- **Repo:** `git@github.com:CodeForPhilly/codeforphilly-data.git` (private during cutover prep) -- **Branches** — each is an independent data scenario: - - `fixture` (default) — hand-/import-curated test data, used by sandbox - - `empty` — sheet configs only, no records - - `snapshot` — anonymized snapshot of prod (auto-produced post-cutover) -- A read-only **SSH deploy key** mounted into the pod authenticates the entrypoint's clone. +The app reads its gitsheets data from `git@github.com:CodeForPhilly/codeforphilly-data.git` +(private), bare-cloned at boot. Branches — each is an independent data scenario: -## One-shot deploy steps (manual, while iterating) +- `published` (default) — runtime-served in sandbox **and** prod; the merge + target of `legacy-import`, pruned of confident spam, plus whatever the + running APIs write. A push here hot-reloads both pods. +- `legacy-import` — raw importer snapshots, spam included. +- `fixture` — small hand-curated test data. +- `empty` — sheet configs only, no records. -```bash -export KUBECONFIG=~/.kube/cfp-sandbox-cluster-kubeconfig.yaml +A **write** SSH deploy key mounted into the pod authenticates the entrypoint's +clone and the push daemon. -# 1. Build + push the image -# --platform=linux/amd64 is required when building on Apple Silicon — the -# Linode LKE nodes are amd64 and won't pull an arm64-only manifest. -docker build --platform=linux/amd64 -t ghcr.io/codeforphilly/codeforphilly-ng:sandbox . -# NOTE: requires `write:packages` scope on your GitHub token. -# If `docker push` says "token does not match expected scopes": -# gh auth refresh -s write:packages -docker push ghcr.io/codeforphilly/codeforphilly-ng:sandbox +## Deploying a release (the normal path) -# 2. Apply manifests (creates namespace, sealed-secrets, PVCs, deployment, service, ingress) -kubectl apply -k deploy/kustomize/overlays/sandbox +In `cfp-sandbox-cluster`, on a branch: -# 3. Watch the rollout -kubectl -n codeforphilly-rewrite-sandbox rollout status deploy/codeforphilly -kubectl -n codeforphilly-rewrite-sandbox logs -f deploy/codeforphilly -``` +1. Set `ref = "refs/tags/vX.Y.Z"` in `.holo/sources/codeforphilly-ng.toml`. +2. Set `images[].newTag: vX.Y.Z` in `codeforphilly-ng/app/kustomization.yaml`. +3. Commit both together (`chore(codeforphilly-ng): bump to vX.Y.Z`), PR into + `main`, merge. +4. "Build k8s-manifests" projects the holobranch to `releases/k8s-manifests`; + a bot opens a PR into `deploys/k8s-manifests`. Merge it — that applies. +5. Watch it land: -After the first successful rollout, the app is live at: + ```bash + export KUBECONFIG=~/.kube/cfp-sandbox-cluster-kubeconfig.yaml + kubectl -n codeforphilly-rewrite-sandbox rollout status deploy/codeforphilly + kubectl -n codeforphilly-rewrite-sandbox logs -f deploy/codeforphilly + curl -sS https://next-v2.codeforphilly.org/api/health/ready + ``` -- +Environment-specific bits (Gateway + HTTPRoute for `next-v2.codeforphilly.org`, +the `CFP_SITE_HOST` patch, SealedSecrets) live in the cluster repo under +`_gateways/codeforphilly-ng.yaml`, `codeforphilly-ng/app/kustomization.yaml` +and `codeforphilly-ng.secrets/`. Change them there, not in this repo's +`deploy/kustomize/overlays/sandbox/`. ## Image visibility -The Docker image is built from this repo and pushed to `ghcr.io/codeforphilly/codeforphilly-ng`. For the cluster to pull without an `imagePullSecret`, the package must be **public** on GHCR. After the first push: - -1. Visit -2. Under "Danger Zone" → "Change package visibility" → Public - -Until that's done, the deployment will sit in `ImagePullBackOff` with `403 Forbidden`. +The image is built by `container-publish.yml` on every release tag and pushed +to `ghcr.io/codeforphilly/codeforphilly-ng`. For the cluster to pull without +an `imagePullSecret`, the package must be **public** on GHCR: + +→ "Change package visibility" → Public. If it ever flips back, the deployment +sits in `ImagePullBackOff` with `403 Forbidden`. ## Rotating the deploy key -The SSH deploy key currently in the cluster was generated locally and added to the data repo via `gh repo deploy-key add`. To rotate: +The public half is registered on the data repo's deploy-keys page; the +private half is sealed in `cfp-sandbox-cluster/codeforphilly-ng.secrets/codeforphilly-data-deploy-key.yaml`. ```bash +export KUBECONFIG=~/.kube/cfp-sandbox-cluster-kubeconfig.yaml ssh-keygen -t ed25519 -f /tmp/cfp-deploy-keys/codeforphilly-data-sandbox-rotated -N "" -C "cfp-sandbox-rotated" -gh repo deploy-key add /tmp/cfp-deploy-keys/codeforphilly-data-sandbox-rotated.pub \ - --repo CodeForPhilly/codeforphilly-data \ +gh-axi repo deploy-key add /tmp/cfp-deploy-keys/codeforphilly-data-sandbox-rotated.pub \ + --repo CodeForPhilly/codeforphilly-data --allow-write \ --title "cfp-sandbox cluster (rotated $(date +%Y-%m-%d))" -# Then re-seal the secret and re-apply kubectl create secret generic codeforphilly-data-deploy-key \ --namespace codeforphilly-rewrite-sandbox \ --from-file=id_ed25519=/tmp/cfp-deploy-keys/codeforphilly-data-sandbox-rotated \ --dry-run=client -o yaml \ | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets -o yaml \ - > deploy/kustomize/overlays/sandbox/sealed-secret-deploy-key.yaml -kubectl apply -k deploy/kustomize/overlays/sandbox + > ~/Repositories/cfp-sandbox-cluster/codeforphilly-ng.secrets/codeforphilly-data-deploy-key.yaml +# Commit in the cluster repo, PR, merge the deploy PR, then: +kubectl -n codeforphilly-rewrite-sandbox rollout restart deploy/codeforphilly # Delete the old deploy key from GitHub after the rotation lands cleanly. ``` -## Rotating the JWT signing key +## Rotating the JWT signing key (or any one key in `codeforphilly-secrets`) + +Use `kubeseal --merge-into` so the other keys stay untouched — full recipe in +[secrets.md](secrets.md#adding-or-changing-one-key-in-an-existing-sealedsecret). ```bash -JWT_KEY=$(openssl rand -base64 48) +export KUBECONFIG=~/.kube/cfp-sandbox-cluster-kubeconfig.yaml kubectl create secret generic codeforphilly-secrets \ --namespace codeforphilly-rewrite-sandbox \ - --from-literal=CFP_JWT_SIGNING_KEY="$JWT_KEY" \ - --from-literal=CFP_DATA_REMOTE="git@github.com:CodeForPhilly/codeforphilly-data.git" \ + --from-literal=CFP_JWT_SIGNING_KEY="$(openssl rand -base64 48)" \ --dry-run=client -o yaml \ - | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets -o yaml \ - > deploy/kustomize/overlays/sandbox/sealed-secret-env.yaml -kubectl apply -k deploy/kustomize/overlays/sandbox + | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets --format yaml \ + --merge-into ~/Repositories/cfp-sandbox-cluster/codeforphilly-ng.secrets/codeforphilly-secrets.yaml +# Commit, PR, merge the deploy PR, rollout restart. # Rotating the JWT signing key invalidates every issued session — users will # need to re-auth. Acceptable in sandbox; coordinate before doing this in prod. ``` ## Switching data branches -The active branch is set in `deploy/kustomize/base/configmap.yaml` via `CFP_DATA_BRANCH`. To swap: +`CFP_DATA_BRANCH` defaults to `published` in `deploy/kustomize/base/configmap.yaml`. +To point the sandbox at another branch, add a ConfigMap patch in +`cfp-sandbox-cluster/codeforphilly-ng/app/kustomization.yaml` (same shape as +the `CFP_SITE_HOST` patch), merge through the deploy PR, then +`kubectl -n codeforphilly-rewrite-sandbox rollout restart deploy/codeforphilly` +— the entrypoint re-clones against the new branch on the fresh `emptyDir`. + +## Emergency escape hatch: manual image + apply + +Only when the release pipeline or the GitOps repo is itself broken and you +need to get a fix onto the sandbox *now*. Fix the repos afterwards so the +next GitOps apply doesn't revert you. + +```bash +export KUBECONFIG=~/.kube/cfp-sandbox-cluster-kubeconfig.yaml + +# 1. Build + push a throwaway tag. --platform=linux/amd64 is required on +# Apple Silicon — the LKE nodes are amd64. Needs `write:packages` on your +# GitHub token (`gh auth refresh -s write:packages` if push is refused). +docker build --platform=linux/amd64 -t ghcr.io/codeforphilly/codeforphilly-ng:sandbox . +docker push ghcr.io/codeforphilly/codeforphilly-ng:sandbox + +# 2a. Point the running Deployment at it without touching manifests... +kubectl -n codeforphilly-rewrite-sandbox set image \ + deploy/codeforphilly codeforphilly=ghcr.io/codeforphilly/codeforphilly-ng:sandbox + +# 2b. ...or apply this repo's manual overlay wholesale (namespace, sealed +# secrets, PVC, deployment, service, gateway). Its sealed secrets are +# the sandbox ones — they only decrypt on that cluster. +kubectl apply -k deploy/kustomize/overlays/sandbox + +# 3. Watch +kubectl -n codeforphilly-rewrite-sandbox rollout status deploy/codeforphilly +``` -1. Edit the ConfigMap (or add an overlay patch) -2. `kubectl apply -k deploy/kustomize/overlays/sandbox` -3. `kubectl -n codeforphilly-rewrite-sandbox rollout restart deploy/codeforphilly` — entrypoint re-clones the working tree against the new branch +The `:sandbox` tag is mutable and `imagePullPolicy: Always` in the base +Deployment, so a pod restart picks up a re-push. Never do this against +production. diff --git a/docs/operations/secrets.md b/docs/operations/secrets.md index e236000..a614b09 100644 --- a/docs/operations/secrets.md +++ b/docs/operations/secrets.md @@ -1,6 +1,6 @@ # Secret management -Every secret consumed by codeforphilly-rewrite at runtime, how to generate it, +Every secret consumed by codeforphilly-ng at runtime, how to generate it, how it gets into the cluster, and how to rotate it. > See [deploy.md](deploy.md) for how the Deployment consumes these. See @@ -12,8 +12,8 @@ how it gets into the cluster, and how to rotate it. 1. **Never in the image.** Secrets are mounted at run-time. The Dockerfile carries zero credentials. 2. **Never in git.** Use [sealed-secrets](https://github.com/bitnami-labs/sealed-secrets) - (cluster default) or a SOPS-encrypted file. The plaintext only exists on - the machines that generated it. + (cluster default). The plaintext only exists on the machines that + generated it. 3. **Scoped.** Each secret is granted the minimum surface needed (per-namespace, per-environment). No "infra" secret used for multiple purposes. @@ -22,18 +22,51 @@ how it gets into the cluster, and how to rotate it. ## Where they live in the cluster -The Deployment consumes secrets from two Secret objects, both materialized -by the sealed-secrets controller from `SealedSecret` resources committed in -the GitOps repo (`cfp-sandbox-cluster/codeforphilly-ng.secrets/`): +SealedSecrets are committed in the GitOps repo for each cluster and +materialized by that cluster's sealed-secrets controller. Sealed values are +cluster-bound: a Secret sealed for the sandbox cannot decrypt in the live +cluster. + +| Cluster | Repo path | Namespace | +| --- | --- | --- | +| Production | `cfp-live-cluster/codeforphilly-ng.secrets/` | `codeforphilly-ng` | +| Sandbox | `cfp-sandbox-cluster/codeforphilly-ng.secrets/` | `codeforphilly-rewrite-sandbox` | + +Production has three Secret objects: | Secret name | Mount mechanism | Holds | -|-------------|-----------------|-------| -| `codeforphilly-secrets` | `envFrom: secretRef` (entire Secret becomes env) | All env-var secrets | -| `codeforphilly-data-deploy-key` | Volume-mounted, one file | SSH private key for the data repo | +| ------------- | ----------------- | ------- | +| `codeforphilly-secrets` | `envFrom: secretRef` (entire Secret becomes env) | `CFP_JWT_SIGNING_KEY`, `CFP_DATA_REMOTE`, `CFP_DATA_RELOAD_SECRET`, `GITHUB_OAUTH_CLIENT_ID`, `GITHUB_OAUTH_CLIENT_SECRET`, `POSTMARK_SERVER_TOKEN` | +| `codeforphilly-saml` | `envFrom: secretRef` (added by a patch in the live cluster repo, `optional: true`) | `SAML_PRIVATE_KEY`, `SAML_CERTIFICATE` — carried over verbatim from laddr's `saml2` secret | +| `codeforphilly-data-deploy-key` | Volume-mounted, one file (`id_ed25519`) | SSH private key for the data repo | + +The sandbox has `codeforphilly-secrets` and `codeforphilly-data-deploy-key` +(no SAML — the sandbox is not registered with Slack). + +The first two Secret names are referenced from +`deploy/kustomize/base/deployment.yaml` (`codeforphilly-secrets`, the deploy +key volume) and from the live cluster repo's kustomization patch +(`codeforphilly-saml`); changing them means touching those manifests. + +### Adding or changing one key in an existing SealedSecret + +Use `kubeseal --merge-into` so you only ever handle the key you're changing +and don't have to re-supply every other value: -The Secret names are referenced directly from -`deploy/kustomize/base/deployment.yaml`; changing them means touching the -manifest. +```bash +# against the target cluster's kubeconfig +kubectl create secret generic codeforphilly-secrets \ + --namespace codeforphilly-ng \ + --from-literal=POSTMARK_SERVER_TOKEN="$NEW_TOKEN" \ + --dry-run=client -o yaml \ + | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets \ + --format yaml --merge-into codeforphilly-ng.secrets/codeforphilly-secrets.yaml +``` + +Commit the updated sealed YAML on a branch in the cluster repo, PR into +`main`, merge the resulting deploy PR. Then `kubectl -n codeforphilly-ng +rollout restart deploy/codeforphilly` — env-mounted Secrets are read at pod +start, not live. ## Inventory @@ -50,53 +83,63 @@ HS256 key for stateless session JWTs. - **Rotation impact:** every active session is invalidated. Users have to sign in again. Plan rotations during low-traffic windows; do not rotate during launches. -- **Rotation procedure:** generate new value → update the sealed-secret → - `kubectl rollout restart deployment/codeforphilly` → users re-auth. +- **Rotation procedure:** generate new value → `--merge-into` the + sealed-secret → merge → `kubectl rollout restart` → users re-auth. - **Cadence:** every 12 months, plus immediately on suspected leak. -### `GITHUB_OAUTH_CLIENT_SECRET` - -Client secret for the GitHub OAuth app. **One app per environment** — a -separate app for staging and production, each with its own callback URL. - -- **Generate:** Rotate via the GitHub OAuth app settings page - (`https://github.com/settings/developers` → app → "Generate a new client - secret"). GitHub never reveals the old secret again. -- **Companion config:** `GITHUB_OAUTH_CLIENT_ID` is non-secret and lives in - the ConfigMap (`publicEnv.GITHUB_OAUTH_CLIENT_ID`). +### `GITHUB_OAUTH_CLIENT_ID` / `GITHUB_OAUTH_CLIENT_SECRET` + +Credentials for the GitHub OAuth app. Production uses the **org-owned +"Code for Philly"** app (the laddr app was renamed "Code for Philly +(legacy)"). Its callback URL is registered on the apex, +`https://codeforphilly.org/...`; GitHub accepts redirect URIs on subdomains +of the registered callback, so `next.codeforphilly.org` works against the +same app and **nothing changes at cutover**. Use a separate app for the +sandbox. + +- **Generate:** GitHub → Organizations → CodeForPhilly → Settings → + Developer settings → OAuth Apps → "Code for Philly" → "Generate a new + client secret". GitHub never reveals the old secret again. +- **Client ID:** not sensitive, but it lives in `codeforphilly-secrets` + next to the secret so the pair rotates in one place. - **Rotation impact:** in-flight OAuth callbacks fail. Existing sessions are unaffected (the secret is only used during the OAuth handshake). -- **Rotation procedure:** issue new secret in GitHub → update sealed-secret → - `kubectl rollout restart`. +- **Rotation procedure:** issue new secret in GitHub → `--merge-into` → + merge → `kubectl rollout restart`. - **Cadence:** every 12 months, plus immediately on suspected leak. -### `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` +### `CFP_DATA_RELOAD_SECRET` -Credentials for the private-storage bucket. The IAM policy attached to -these credentials must be scoped to the single bucket per -[deploy.md](deploy.md#bucket-provisioning). +Bearer token gating `POST /api/_internal/reload-data` (the hot-reload +webhook). The same value must be set as the `CFP_DATA_RELOAD_SECRET` +Actions secret in `CodeForPhilly/codeforphilly-data`, which +`notify-deployments.yml` sends to both sandbox and prod — so today one +value is shared across both environments. -- **Generate:** Provider console (R2 → API tokens; B2 → application keys; - AWS → IAM access key). -- **Rotation impact:** PUTs and GETs to the private store fail until the - pod is restarted with new keys. Newsletter signups and account claims - return `private_store_unavailable` (5xx). -- **Rotation procedure:** - 1. Provision a *second* access key in the bucket provider. - 2. Update the sealed-secret with the new value. - 3. `kubectl rollout restart deployment/codeforphilly` — pod boots with - new keys. - 4. Verify `/api/health/ready` returns 200. - 5. Revoke the old access key in the bucket provider. -- **Cadence:** every 6 months, plus immediately on suspected leak. +- **Generate:** `openssl rand -hex 32`. +- **Rotation impact:** until both sides match, every `published` push + fails its notify step (401) and pods serve stale data until restarted. +- **Rotation procedure:** `--merge-into` both cluster repos → merge → + restart both pods → update the data repo's Actions secret → push a + no-op to `published` (or re-run the last workflow) to confirm 200s. +- **Cadence:** every 12 months, plus immediately on suspected leak. + +### `CFP_DATA_REMOTE` + +The data repo's git URL (`git@github.com:CodeForPhilly/codeforphilly-data.git`). +Not itself sensitive — it lives in `codeforphilly-secrets` because the +entrypoint reads it from the same env block. Changing it takes effect on +the next pod start (fresh `emptyDir`, fresh clone). ### `SAML_PRIVATE_KEY` / `SAML_CERTIFICATE` -PEM-encoded cert chain that signs SAML assertions for the Slack IdP -integration ([specs/api/saml.md](../../specs/api/saml.md)). +PEM-encoded key pair that signs SAML assertions for the Slack IdP +integration ([specs/api/saml.md](../../specs/api/saml.md)). In production +these are **the laddr values**, copied from the legacy `saml2` Secret in +the `code-for-philly` namespace, so Slack's existing trust in that cert +carries straight over. Don't regenerate them as part of cutover. -- **Generate:** the openssl recipe documented in the legacy repo at - `laddr/docs/operations/update-saml2-certificate.md`: +- **Generate (only for a deliberate rotation):** ```bash openssl req -x509 -newkey rsa:2048 -days 1095 -nodes \ @@ -107,7 +150,8 @@ integration ([specs/api/saml.md](../../specs/api/saml.md)). - **Rotation impact:** Slack stops trusting assertions until its IdP config is updated with the new cert. **Do not rotate without coordinating with - the Slack workspace admin.** + the Slack workspace admin.** With SSO set to "optional" a botched + rotation is recoverable via email magic links. - **Not a secret, but paired:** `SAML_ENTITY_ID` (ConfigMap, optional) is the IdP identity Slack stores alongside this cert. It defaults to `https://codeforphilly.org/api/saml/slack/metadata` and must stay stable @@ -116,12 +160,13 @@ integration ([specs/api/saml.md](../../specs/api/saml.md)). - **Rotation procedure:** 1. Generate new key + cert. 2. Upload the *new cert* to Slack as a secondary signing cert. - 3. Update the sealed-secret with the new key + cert. - 4. `kubectl rollout restart`. - 5. Test SAML SSO from a clean browser. + 3. `--merge-into codeforphilly-ng.secrets/codeforphilly-saml.yaml` with + the new key + cert. + 4. Merge → `kubectl rollout restart`. + 5. Test SAML SSO from a clean browser (Slack "Test configuration"). 6. Once verified, remove the old cert from Slack. -- **Cadence:** every 36 months (cert expiry), plus immediately on - suspected leak. +- **Cadence:** on cert expiry (check the `notAfter` of the carried-over + cert — laddr issued it), plus immediately on suspected leak. ### `POSTMARK_SERVER_TOKEN` @@ -142,74 +187,106 @@ for local dev but means real users get no outbound mail in production. that exists on the chosen server. - **Rotation impact:** none in-flight (no in-flight email state on our end); next outbound mail uses the new token. -- **Rotation procedure:** create new token in Postmark → update - sealed-secret → `kubectl rollout restart` → delete the old token in - Postmark. +- **Rotation procedure:** create new token in Postmark → `--merge-into` → + merge → `kubectl rollout restart` → delete the old token in Postmark. - **Cadence:** every 12 months, plus immediately on suspected leak. ### Data-repo deploy key SSH ed25519 private key with **write** access to the `codeforphilly-data` repo on GitHub. Mounted as a file at `/etc/cfp-data-deploy-key/id_ed25519`; -the entrypoint sets `GIT_SSH_COMMAND` to use it. +the ConfigMap's `GIT_SSH_COMMAND` points ssh at it. - **Generate:** ```bash - ssh-keygen -t ed25519 -f cfp-data-deploy -C "codeforphilly k8s deploy" + ssh-keygen -t ed25519 -f cfp-data-deploy -N "" -C "codeforphilly-ng deploy" ``` Add the *public* key (`cfp-data-deploy.pub`) to the data repo's - Settings → Deploy keys with "Allow write access" checked. -- **Rotation impact:** push daemon will fail to push commits until the new - key is mounted. Reads/writes to the in-memory state continue; the - inability to push surfaces as a backlog of unpushed commits on the PVC. + Settings → Deploy keys with "Allow write access" checked + (`gh-axi repo deploy-key add ... --allow-write`). One key per cluster. +- **Rotation impact:** the entrypoint's clone fails on the next pod start + and the push daemon fails to push until the new key is mounted. Reads + and writes to in-memory state continue; unpushed commits sit in the + pod's bare clone (and are lost if the pod is recreated before the push + succeeds — see [runbook.md](runbook.md#fetch-from-the-pods-data-clone)). - **Rotation procedure:** 1. Generate new keypair. 2. Add new public key to the data repo (alongside the existing one). - 3. Update the sealed-secret with the new private key. - 4. `kubectl rollout restart`. + 3. Re-seal `codeforphilly-data-deploy-key` with the new private key + (`kubectl create secret generic ... --from-file=id_ed25519=... | kubeseal ...`, + replacing the file — a single-key Secret doesn't need `--merge-into`). + 4. Merge → `kubectl rollout restart`. 5. Verify a test mutation reaches the remote. 6. Remove the old deploy key from the data repo. - **Cadence:** every 12 months, plus immediately on team turnover or suspected leak. +### `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` (unused) + +Credentials for an S3-compatible private-storage bucket. **No environment +uses the `s3` backend today** — sandbox and production both run +`STORAGE_BACKEND=filesystem` on a PVC ([deploy.md](deploy.md#private-storage)). +If that ever changes: provision the bucket with versioning on, scope an IAM +policy to that single bucket, add the two keys to `codeforphilly-secrets` +and `S3_ENDPOINT` / `S3_BUCKET` / `S3_REGION` to the ConfigMap, and +migrate the two `.jsonl` files off the PVC. Rotation would follow the +two-key overlap pattern (provision second key → swap → verify → revoke). + ## Bootstrapping a new environment -First-time set up of `codeforphilly-staging` or `codeforphilly`: +First-time set up of a new namespace (e.g. a second sandbox). Run against +the target cluster's kubeconfig so `kubeseal` fetches that cluster's public +key. ```bash -# 1. Create namespace -kubectl create namespace codeforphilly-staging +NS=codeforphilly-ng # or codeforphilly-rewrite-sandbox +OUT=~/Repositories/cfp-live-cluster/codeforphilly-ng.secrets + +# 1. Namespace (in the cluster repo's namespace.yaml normally; kubectl for a scratch env) +kubectl create namespace "$NS" -# 2. Generate all secret values locally +# 2. Generate values locally +mkdir -p .secrets openssl rand -base64 64 > .secrets/jwt -ssh-keygen -t ed25519 -f .secrets/deploy -N "" -# ... GitHub OAuth secret from GitHub UI, S3 keys from R2 console ... +openssl rand -hex 32 > .secrets/reload +ssh-keygen -t ed25519 -f .secrets/deploy -N "" -C "codeforphilly-ng $NS deploy" +# ... GitHub OAuth client id + secret from the GitHub UI, Postmark token from Postmark ... -# 3. Build the Secret manifests +# 3. Seal kubectl create secret generic codeforphilly-secrets \ - --namespace codeforphilly-staging \ + --namespace "$NS" \ --from-literal=CFP_JWT_SIGNING_KEY="$(cat .secrets/jwt)" \ + --from-literal=CFP_DATA_RELOAD_SECRET="$(cat .secrets/reload)" \ + --from-literal=CFP_DATA_REMOTE="git@github.com:CodeForPhilly/codeforphilly-data.git" \ + --from-literal=GITHUB_OAUTH_CLIENT_ID="$GH_CLIENT_ID" \ --from-literal=GITHUB_OAUTH_CLIENT_SECRET="$GH_SECRET" \ - --from-literal=S3_ACCESS_KEY_ID="$S3_ID" \ - --from-literal=S3_SECRET_ACCESS_KEY="$S3_KEY" \ - --from-literal=SAML_PRIVATE_KEY="$(cat .secrets/saml-private.pem)" \ - --from-literal=SAML_CERTIFICATE="$(cat .secrets/saml-certificate.pem)" \ + --from-literal=POSTMARK_SERVER_TOKEN="$POSTMARK_TOKEN" \ --dry-run=client -o yaml \ - | kubeseal --format yaml > deploy/secrets/staging-secrets.sealed.yaml + | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets --format yaml \ + > "$OUT/codeforphilly-secrets.yaml" kubectl create secret generic codeforphilly-data-deploy-key \ - --namespace codeforphilly-staging \ + --namespace "$NS" \ --from-file=id_ed25519=.secrets/deploy \ --dry-run=client -o yaml \ - | kubeseal --format yaml > deploy/secrets/staging-deploy-key.sealed.yaml - -# 4. Commit the sealed YAMLs into the GitOps repo -# (cfp-sandbox-cluster/codeforphilly-ng.secrets/), open a PR. -# The deploy workflow applies them on merge. + | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets --format yaml \ + > "$OUT/codeforphilly-data-deploy-key.yaml" + +# Production only — carry the laddr SAML key pair over rather than minting a new one: +kubectl -n code-for-philly get secret saml2 -o yaml # inspect key names, then +kubectl create secret generic codeforphilly-saml \ + --namespace "$NS" \ + --from-literal=SAML_PRIVATE_KEY="$(kubectl -n code-for-philly get secret saml2 -o jsonpath='{.data.}' | base64 -d)" \ + --from-literal=SAML_CERTIFICATE="$(kubectl -n code-for-philly get secret saml2 -o jsonpath='{.data.}' | base64 -d)" \ + --dry-run=client -o yaml \ + | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets --format yaml \ + > "$OUT/codeforphilly-saml.yaml" -# 5. Wipe plaintext +# 4. Commit the sealed YAMLs in the GitOps repo, PR into main, merge the deploy PR. +# 5. Add the deploy public key to the data repo; set CFP_DATA_RELOAD_SECRET in the data repo's Actions secrets. +# 6. Wipe plaintext shred -u .secrets/* ``` @@ -221,9 +298,10 @@ the sealed-secrets controller in the matching cluster. Listed because operators ask: - `GITHUB_OAUTH_CLIENT_ID` — public by design; GitHub exposes it during - every OAuth flow. -- `S3_ENDPOINT` / `S3_BUCKET` / `S3_REGION` — public addressing info. + every OAuth flow. (Stored in the Secret anyway, for one-place rotation.) - `CFP_DATA_REMOTE` — URL form. The *access* (deploy key) is secret; the - URL itself isn't. + URL itself isn't. (Also stored in the Secret; see above.) - `SAML_CERTIFICATE` (the public cert) is technically published to Slack anyway, but we keep it next to the private key for atomic rotation. +- `SAML_ENTITY_ID`, `CFP_SITE_HOST`, `SLACK_TEAM_HOST`, `CFP_DATA_BRANCH` — + ConfigMap values. diff --git a/docs/operations/spam-detection.md b/docs/operations/spam-detection.md index 7cec414..4e00cc9 100644 --- a/docs/operations/spam-detection.md +++ b/docs/operations/spam-detection.md @@ -13,7 +13,7 @@ All work — scripts, sheet configs, and evaluation records — currently sits o Four sheets back the system, all defined under `.gitsheets/` in `codeforphilly-data`: | Sheet | Path template | Purpose | -|---|---|---| +| --- | --- | --- | | `person-evaluations` | `${{ personSlug }}/${{ evaluator }}` | One verdict per (person, evaluator). Multiple evaluators coexist per person. | | `slack-presence` | `${{ personSlug }}` | Per-person Slack snapshot: channel membership, message count, recent messages, spam-message aggregates. | | `slack-channels` | `${{ id }}` | Workspace channel catalog (id, name, member count, is-default, etc.). | @@ -159,7 +159,7 @@ npm run evaluate-heuristic npm run evaluate-llm # 6. Apply the verdicts — prune confident-spam from `published` (see below). -# Run from the codeforphilly-rewrite repo against a bare clone, then push. +# Run from the codeforphilly-ng repo against a bare clone, then push. # THIS STEP IS MANDATORY after any import/merge — see "Applying spam decisions". ``` @@ -173,10 +173,10 @@ When source records get updated (e.g., a previously-empty profile gets a new bio Verdicts are advisory until the **prune** step applies them. Prune is not a read-path filter (the runtime loader stays spam-unaware); it **removes confident-spam people from `published`** so the deployed app never loads them into memory or shows them. This is what keeps the in-memory footprint within the node budget — see [specs/behaviors/spam-exclusion.md](../../specs/behaviors/spam-exclusion.md) for the full contract. -The tool is `apps/api/scripts/prune-spam.ts` in the **`codeforphilly-rewrite`** repo (not the data repo). Run it against a bare clone of the data repo that carries both `published` and `spam-detection`, dry-run first, then push: +The tool is `apps/api/scripts/prune-spam.ts` in the **`codeforphilly-ng`** repo (not the data repo). Run it against a bare clone of the data repo that carries both `published` and `spam-detection`, dry-run first, then push: ```bash -# From the codeforphilly-rewrite repo +# From the codeforphilly-ng repo npm run -w apps/api script:prune-spam -- \ --data-repo=/path/to/codeforphilly-data.git \ --evaluations-ref=spam-detection \ @@ -230,7 +230,7 @@ npm run profile -- forager For the initial run on the full historical corpus (~31,470 people, ~119k Slack messages): | Step | Cost | -|---|---| +| --- | --- | | `fetch-slack-presence` | $0 (Slack API is free) | | `evaluate-slack-messages` (Pass B) | ~$13 | | `evaluate-heuristic` | $0 | From ad174d54f48f913646140259591c9709f8bbea62 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 10 Sep 2026 02:06:10 -0400 Subject: [PATCH 4/5] docs(ops): document the real laddr credentials export and PVC load The export command pointed at a `laddr` database and a `users` table with Handle/PasswordHash columns that don't exist. The live database is `emergence-site` (the `codeforphilly` schema on the same server is a stale 2024 copy), table `people`, columns Username/Email/Password, and the export runs inside the laddr pod with the Habitat-packaged mysql client against the service's client.cnf. The production load is kubectl cp onto the pod's private-storage PVC followed by a rollout restart, same as sandbox; the GCS/S3 upload section is reduced to a note for the unused s3 backend. Records the first prod load (2026-09-10, 21,761 profiles/credentials) and warns that repeating it resets rehashed credentials now that sign-ins are live at next.codeforphilly.org. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR --- docs/operations/legacy-credentials-import.md | 166 +++++++++++-------- 1 file changed, 101 insertions(+), 65 deletions(-) diff --git a/docs/operations/legacy-credentials-import.md b/docs/operations/legacy-credentials-import.md index 66947e4..a05fe33 100644 --- a/docs/operations/legacy-credentials-import.md +++ b/docs/operations/legacy-credentials-import.md @@ -1,8 +1,8 @@ # Legacy credentials import How to seed `profiles.jsonl` (PrivateProfile) + `legacy-passwords.jsonl` -(LegacyPasswordCredential) into a deployment's private store from a -laddr MySQL export. +(LegacyPasswordCredential) into a deployment's private store from the +laddr MySQL database. The public-side importer ([`script:import-laddr`](../../apps/api/scripts/import-laddr.ts)) deliberately handles only public data — laddr's public JSON API doesn't @@ -16,12 +16,16 @@ rehashed credentials with the originals. The script is re-runnable shape-wise (it always produces a complete replacement), but in practice it's run once per env unless something went wrong with the first run. +The first production load ran **2026-09-10**: 21,761 profiles/credentials +landed on the prod PVC. + ## What you need ### CSV export from laddr -A CSV with header `Username,Email,Password`, one row per active laddr -user. Each cell is double-quoted; embedded quotes use `""` escape. +A CSV with header `Username,Email,Password`, one row per laddr user with +both a username and an email. Each cell is double-quoted; embedded quotes +use `""` escape. ```csv "Username","Email","Password" @@ -29,42 +33,74 @@ user. Each cell is double-quoted; embedded quotes use `""` escape. "hhutch","hunter.hutchinson@gmail.com","7746451fd8c30b5b4068cd45fa7b1052cef54068" ``` -- **Username** — laddr `User.Handle` (= the slug that became `Person.slug`). +- **Username** — laddr `people.Username` (= the handle that became `Person.slug`). - **Email** — required; rows with empty Email are skipped (warned). -- **Password** — the raw `PasswordHash` column from laddr's `users` table. - Mixed algorithms in the wild: older users are unsalted SHA-1 (40 lowercase - hex chars, no prefix); newer users are bcrypt (`$2a$ / $2b$ / $2y$`). - Empty Password is allowed — emits a PrivateProfile but no - LegacyPasswordCredential; user will have to use the password-reset - flow if they ever want one. The runtime verifier handles all three - formats; see [specs/behaviors/password-hash-rotation.md](../../specs/behaviors/password-hash-rotation.md). +- **Password** — the raw `people.Password` column. Mixed algorithms in the + wild: older users are unsalted SHA-1 (40 lowercase hex chars, no prefix); + newer users are bcrypt (`$2a$ / $2b$ / $2y$`). Empty Password is allowed — + emits a PrivateProfile but no LegacyPasswordCredential; user will have + to use the password-reset flow if they ever want one. The runtime + verifier handles all three formats; see + [specs/behaviors/password-hash-rotation.md](../../specs/behaviors/password-hash-rotation.md). + +#### Where the data actually is + +The live laddr site runs in the live cluster, namespace `code-for-philly`, +as an Emergence/Habitat pod with MySQL alongside. The database is +**`emergence-site`**. The same server also has a `codeforphilly` schema — +that is a **stale 2024 copy**; don't export from it. The table is `people` +(not `users`), columns `Username`, `Email`, `Password`. -Produce the file with: +#### Produce the file + +Run the export *inside* the laddr pod with the Habitat-packaged mysql +client, reading the socket credentials from the service's client config: ```bash -mysql -h -u -p laddr \ - -B -e "SELECT Handle AS Username, Email, PasswordHash AS Password - FROM users WHERE Email IS NOT NULL AND PasswordHash IS NOT NULL" \ - | sed 's/"/\\\\"/g' | awk -F'\t' 'BEGIN{print "\"Username\",\"Email\",\"Password\""} NR>1 {printf "\"%s\",\"%s\",\"%s\"\n", $1, $2, $3}' \ - > .scratch/legacy-logins-export.csv +# from your machine, against the live cluster kubeconfig +POD=$(kubectl -n code-for-philly get pods -l app.kubernetes.io/name=code-for-philly -o jsonpath='{.items[0].metadata.name}') +# (adjust the label selector if it doesn't match; `kubectl -n code-for-philly get pods` and pick the site pod) + +kubectl -n code-for-philly exec "$POD" -- sh -c ' + M=$(ls -d /hab/pkgs/core/mysql/*/*/bin/mysql | tail -1); + $M --defaults-extra-file=/hab/svc/mysql/config/client.cnf emergence-site -N -B -e \ + "SELECT Username, Email, Password FROM people + WHERE Email IS NOT NULL AND Email<>'"'"''"'"' AND Username IS NOT NULL AND Username<>'"'"''"'"'" +' > .scratch/legacy-logins-export.tsv + +# TSV -> quoted CSV in the shape the importer wants +awk -F'\t' ' + BEGIN { print "\"Username\",\"Email\",\"Password\"" } + { for (i = 1; i <= 3; i++) gsub(/"/, "\"\"", $i); + printf "\"%s\",\"%s\",\"%s\"\n", $1, $2, $3 } +' .scratch/legacy-logins-export.tsv > .scratch/legacy-logins-export.csv + +wc -l .scratch/legacy-logins-export.csv # expect ~21.8k rows + header ``` -(Or use whatever extract tooling you prefer — the importer just needs -the CSV shape above. Land the file in `.scratch/` which is gitignored.) +If the nested quoting in the `exec` line fights you, `kubectl exec -it` into +the pod, run the two-line `M=…; $M …` command there redirecting to +`/tmp/export.tsv`, then `kubectl cp` it out. Land the file in `.scratch/`, +which is gitignored. ### A bare clone of `codeforphilly-data` The importer reads Person records (for the `slug → personId` map) via the same `openPublicStore` interface the runtime uses, which requires a -bare clone. If your dev sibling clone is a working tree, make a +bare clone on the **`published`** branch (the runtime branch — the data +repo has no `main`). If your dev sibling clone is a working tree, make a side-clone for the importer: ```bash git clone --bare --branch published \ - ~/Repositories/codeforphilly-data \ + git@github.com:CodeForPhilly/codeforphilly-data.git \ /tmp/codeforphilly-data-bare-published.git ``` +Make sure it's at the tip you just pushed in the public-data refresh +([cutover.md → T-1](cutover.md#t-1-day-data-refresh--credentials)) — +a Person that isn't on `published` can't be matched. + ## Run ```bash @@ -88,63 +124,62 @@ directory into the workspace and relative paths resolve from there. The report prints input row count, write counts, and a breakdown of skip reasons (no-username / no-email / no-person-match / deleted-person -/ duplicate-person). Expect a ~95% match rate on a healthy laddr corpus -— the missing 5% are typically deleted laddr users that didn't survive -the public-data import. +/ duplicate-person). A large `no-person-match` count is expected when +`published` has been spam-pruned — the pruned people are exactly the ones +that shouldn't get a credential. + +## Deploy (FilesystemPrivateStore on a PVC — sandbox and production) + +Both environments use `STORAGE_BACKEND=filesystem` with +`/app/private-storage` mounted from the `codeforphilly-private` +PersistentVolumeClaim ([deploy.md → Private storage](deploy.md#private-storage)). +The procedure is identical; only the namespace differs: + +| | Namespace | Health URL | +| --- | --- | --- | +| Production | `codeforphilly-ng` | `https://next.codeforphilly.org/api/health/ready` (apex after cutover) | +| Sandbox | `codeforphilly-rewrite-sandbox` | `https://next-v2.codeforphilly.org/api/health/ready` | + +1. **Snapshot what's there** (skip on a first load into an empty PVC): -## Deploy to sandbox (FilesystemPrivateStore on a PVC) + ```bash + NS=codeforphilly-ng + POD=$(kubectl -n "$NS" get pods -l app.kubernetes.io/name=codeforphilly -o jsonpath='{.items[0].metadata.name}') + kubectl -n "$NS" exec "$POD" -- sh -c 'cd /app/private-storage && for f in *.jsonl; do cp "$f" "$f.bak-$(date +%Y%m%dT%H%M%S)"; done; ls -la' + ``` -The sandbox uses `STORAGE_BACKEND=filesystem` with `/app/private-storage` -mounted to a PersistentVolumeClaim (see `deploy/kustomize/base/`). + The PVC has no versioning; the `.bak` copy on the volume is your undo. -1. **Copy the files into the pod:** +2. **Copy the files into the pod:** ```bash - POD=$(kubectl -n codeforphilly-rewrite-sandbox get pods -o jsonpath='{.items[0].metadata.name}') - kubectl -n codeforphilly-rewrite-sandbox cp \ + kubectl -n "$NS" cp \ .scratch/private-import/profiles.jsonl "$POD:/app/private-storage/profiles.jsonl" - kubectl -n codeforphilly-rewrite-sandbox cp \ + kubectl -n "$NS" cp \ .scratch/private-import/legacy-passwords.jsonl "$POD:/app/private-storage/legacy-passwords.jsonl" ``` -2. **Restart the pod** to reload the private store into memory. The +3. **Restart the pod** to reload the private store into memory. The `POST /api/_internal/reload-data` webhook **does not** cover the private store — it only reloads public + FTS. A full pod restart is the supported path. ```bash - kubectl -n codeforphilly-rewrite-sandbox rollout restart deployment + kubectl -n "$NS" rollout restart deploy/codeforphilly + kubectl -n "$NS" rollout status deploy/codeforphilly ``` -3. **Verify** the new pod sees the credentials: - - ```bash - curl -sS https://next-v2.codeforphilly.org/api/health/ready # waits until private store is loaded - ``` +4. **Verify** the new pod sees the credentials: `/api/health/ready` + returns 200, and a known legacy user can sign in with their laddr + password. -## Deploy to production (S3PrivateStore on GCS or S3) - -Production uses `STORAGE_BACKEND=s3` with `S3_ENDPOINT` pointing at -either GCS's XML API (`https://storage.googleapis.com`) or an actual -S3 bucket. Upload the two files to the bucket at the configured -`keyPrefix` (default empty): - -```bash -# GCS via gsutil -gsutil cp .scratch/private-import/profiles.jsonl \ - gs:///profiles.jsonl -gsutil cp .scratch/private-import/legacy-passwords.jsonl \ - gs:///legacy-passwords.jsonl - -# OR: any S3 endpoint via awscli -aws s3 cp .scratch/private-import/profiles.jsonl \ - s3:///profiles.jsonl --endpoint-url -aws s3 cp .scratch/private-import/legacy-passwords.jsonl \ - s3:///legacy-passwords.jsonl --endpoint-url -``` +### If the private store is ever moved to S3 -Then restart the prod pod the same way (private-store reload is -not in the hot-reload path; see above). +The `s3` backend ([specs/behaviors/private-storage.md](../../specs/behaviors/private-storage.md#backends)) +is supported but **not in use anywhere today**. If a deployment is ever +switched to it, the equivalent of step 2 is uploading the two files to the +bucket root (or the configured `keyPrefix`) with `aws s3 cp +--endpoint-url "$S3_ENDPOINT"`, then restarting the pod as above. ## Safety notes @@ -158,9 +193,10 @@ not in the hot-reload path; see above). - **Re-deploying overwrites the runtime files in full.** If users have already signed in and their credentials have been rehashed to argon2id (via the in-app rehash-on-login flow), a re-deploy would - revert those to the SHA-1/bcrypt originals. After cutover, only - re-run + re-deploy if you've confirmed nobody has signed in (or - you're OK with the rotation reset). + revert those to the SHA-1/bcrypt originals. Production has been + accepting sign-ins at `next.codeforphilly.org` since the first load, so + **do not repeat the prod load casually** — only if you've confirmed + nobody has signed in since, or you're OK with the rotation reset. - **PII risk.** The generated JSONL files contain every legacy user's email and password hash. Treat them as you would the source MySQL dump — never paste into chat, never check into git, delete from @@ -170,4 +206,4 @@ not in the hot-reload path; see above). - [specs/behaviors/private-storage.md](../../specs/behaviors/private-storage.md) — what these files store and the rules around them. - [specs/behaviors/password-hash-rotation.md](../../specs/behaviors/password-hash-rotation.md) — how the verifier handles SHA-1 / bcrypt / argon2id and rehashes on login. -- [docs/operations/cutover.md](./cutover.md) — full cutover sequence; this importer fits between the public-data import and the DNS flip. +- [docs/operations/cutover.md](./cutover.md) — full cutover sequence; this importer runs at T-1, after the public-data refresh and before the gateway hostname move. From 5a416313d988ba77bc670fb1a870431449127a69 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 10 Sep 2026 02:11:31 -0400 Subject: [PATCH 5/5] chore(plans): mark cutover-runbook-reality done (PR #172) Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR --- plans/cutover-runbook-reality.md | 57 +++++++++++++++++++++++++++----- 1 file changed, 48 insertions(+), 9 deletions(-) diff --git a/plans/cutover-runbook-reality.md b/plans/cutover-runbook-reality.md index 9784cc9..a31060d 100644 --- a/plans/cutover-runbook-reality.md +++ b/plans/cutover-runbook-reality.md @@ -1,5 +1,5 @@ --- -status: in-progress +status: done depends: - cutover-prep - release-flow @@ -11,6 +11,7 @@ specs: - specs/behaviors/storage.md - specs/api/saml.md issues: [] +pr: 172 --- # Plan: Cutover runbook reality check @@ -88,12 +89,12 @@ on 2026-09-09/10: ## Validation -- [ ] `grep -rn 'cfp-prod-cluster\|staging.k8s\|cutover-window-policy\|T+180' docs/ .claude/CLAUDE.md` returns nothing. -- [ ] No doc names `main` as a data-repo branch; runtime branch is `published` everywhere. -- [ ] No doc gives S3/GCS as the production private-storage path; the filesystem PVC is. -- [ ] `cutover.md` T-0 and rollback sections describe the gateway-listener commit and its revert; no DNS/TTL steps remain. -- [ ] `legacy-credentials-import.md` export command targets `emergence-site`.`people` via the Habitat client. -- [ ] `npm run lint` clean. +- [x] `grep -rn 'cfp-prod-cluster\|staging.k8s\|cutover-window-policy\|T+180' docs/ .claude/CLAUDE.md` returns nothing. +- [x] No doc names `main` as a data-repo branch; runtime branch is `published` everywhere. +- [x] No doc gives S3/GCS as the production private-storage path; the filesystem PVC is. +- [x] `cutover.md` T-0 and rollback sections describe the gateway-listener commit and its revert; no DNS/TTL steps remain. +- [x] `legacy-credentials-import.md` export command targets `emergence-site`.`people` via the Habitat client. +- [x] `npm run lint` clean. ## Risks / unknowns @@ -107,8 +108,46 @@ on 2026-09-09/10: ## Notes -(Populated at closeout.) +- **Verified from sibling clones, not from memory.** The cluster pins, + gateway listener names (`https-apex`, `https-www`, `https-subdomain` on + `code-for-philly`; `https-next` on `codeforphilly-ng`), the three + SealedSecret names and their keys, the `CFP_SITE_HOST` patch, the + `Build k8s-manifests` workflow, `notify-deployments.yml`, and the + `tf/dns` records were all read from local clones of `cfp-live-cluster`, + `cfp-sandbox-cluster`, `codeforphilly-data-published` and `CodeForPhilly/ops`. + Anything about Slack admin UI, GitHub OAuth app naming, the MySQL schema, + or import counts is as reported by the operator who ran it. +- **The runtime image can't run operator scripts.** `Dockerfile` prunes dev + deps and copies only `dist/`, so `script:reconcile` has no in-pod path and + the private store can't be copied out to run it locally. `cutover.md` + names this as an open gap under T+7 instead of prescribing a workaround. +- **`deploy/kustomize/overlays/sandbox/` is now legacy.** The sandbox cluster + repo carries its own namespace, secrets and hostname patches; the overlay + in this repo only backs the manual escape hatch. Left in place; removing it + is a separate decision. +- **The announcement templates were also wrong on passwords.** They told + members password sign-in "is going away"; fixed alongside the T+90/T+180 + deletions since a member reading them would be misled the same way. +- **`HOST` was dropped from the env table** — it isn't in + `deploy/kustomize/base/configmap.yaml`; Fastify's default binding is what + runs. ## Follow-ups -(Populated at closeout.) +- Spec drift in `specs/architecture.md` (`STORAGE_BACKEND` "`s3` in + production", entrypoint "`git reset --hard origin/main`", + `overlays/staging/` + `overlays/production/`) and + `specs/behaviors/private-storage.md` ("`s3` backend (production)", bucket + versioning as a production requirement). Both deployed environments run + `filesystem` on a PVC with no versioning. Needs its own spec PR; not + touched here because this plan is docs-only. +- `apps/api/scripts/cutover-mailout.ts` template says "Accounts unclaimed + for one year may be retired" — no spec backs that. Either the sunset spec + gets written first or the line goes. +- Ship `script:reconcile` (or an equivalent) somewhere it can reach the + production private store — in the image, or as an in-cluster Job — so the + T+7 reconciliation check in `cutover.md` is runnable. +- Monitoring: none of `docs/operations/monitoring.md` is wired. First + post-cutover ops task; the checklist there is the spec for it. +- Consider deleting `deploy/kustomize/overlays/sandbox/` once nobody needs + the manual escape hatch.