From ace1246e9a158038e0297a9687689754d3fa3d7b Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 10 Sep 2026 02:06:05 -0400 Subject: [PATCH 1/8] chore(plans): mark cutover-runbook-reality in-progress Docs-only plan: bring the cutover runbook and the operator docs it leans on into line with the environments that exist as of 2026-09-10 (cfp-live-cluster, GitOps release pins, filesystem private storage, the published-branch refresh pipeline, the Slack SSO plan). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR --- plans/cutover-runbook-reality.md | 114 +++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 plans/cutover-runbook-reality.md diff --git a/plans/cutover-runbook-reality.md b/plans/cutover-runbook-reality.md new file mode 100644 index 0000000..9784cc9 --- /dev/null +++ b/plans/cutover-runbook-reality.md @@ -0,0 +1,114 @@ +--- +status: in-progress +depends: + - cutover-prep + - release-flow + - postmark-notifier + - saml-self-host +specs: + - specs/behaviors/account-migration.md + - specs/behaviors/private-storage.md + - specs/behaviors/storage.md + - specs/api/saml.md +issues: [] +--- + +# Plan: Cutover runbook reality check + +## Scope + +Docs only. Bring `docs/operations/cutover.md` and the operator docs it leans +on (`cutover-rollback.md`, `deploy.md`, `runbook.md`, `secrets.md`, +`legacy-credentials-import.md`, `sandbox-deploy.md`, `monitoring.md`, +`releases.md`, `cutover-announcement.md`, plus the deploy section of +`.claude/CLAUDE.md`) into line with the environments that exist as of +2026-09-10, so the person running cutover is not reading instructions written +before the live cluster, the release flow, or the first data refresh existed. + +Out of scope: code changes; spec changes. Spec drift noticed along the way is +recorded under Follow-ups rather than fixed here. + +## Implements + +No new spec behavior. This plan reconciles operator docs with: + +- [behaviors/account-migration.md](../specs/behaviors/account-migration.md) — + legacy password sign-in persists indefinitely; no dated claim window or + credential purge exists, so the runbook's T+90 / T+180 sections go. +- [behaviors/private-storage.md](../specs/behaviors/private-storage.md) — + production runs the `filesystem` backend on a PVC; S3 stays a supported but + unused backend. +- [behaviors/storage.md](../specs/behaviors/storage.md) — the runtime branch is + `published` (the data repo has no `main`); hot reload via + `notify-deployments.yml`. +- [api/saml.md](../specs/api/saml.md) — stable `SAML_ENTITY_ID`, SSO endpoint + URLs follow `CFP_SITE_HOST`. + +## Approach + +Facts to encode, each verified against the sibling GitOps / DNS / data clones +on 2026-09-09/10: + +- **Environments.** Production is `cfp-live-cluster`, namespace + `codeforphilly-ng`, pre-cutover host `next.codeforphilly.org`, alongside + legacy laddr in `code-for-philly`. Sandbox is `cfp-sandbox-cluster`, + namespace `codeforphilly-rewrite-sandbox`, host `next-v2.codeforphilly.org`. + There is no separate staging; `cfp-prod-cluster` and + `codeforphilly-rewrite-staging.k8s.phl.io` never existed under those names. +- **Release + GitOps flow.** `develop` → `Release: vX.Y.Z` PR → tag → + `container-publish.yml` pushes `ghcr.io/codeforphilly/codeforphilly-ng:vX.Y.Z`. + Each cluster repo pins the app by `.holo/sources/codeforphilly-ng.toml` + (`ref`) and `images[].newTag` in `codeforphilly-ng/app/kustomization.yaml`, + bumped together. Merge to `main` → "Build k8s-manifests" → + `releases/k8s-manifests` → bot PR into `deploys/k8s-manifests` → merge applies. + The manual `:sandbox` image push is an escape hatch, not the path. +- **Cutover is a hostname move, not a DNS change.** `codeforphilly.org`, + `*.codeforphilly.org` and `*.live.k8s.phl.io` already resolve to the live + cluster's Envoy gateway (managed in `CodeForPhilly/ops` `tf/dns`). T-0 is one + commit in `cfp-live-cluster` moving the apex + `www` listeners and HTTPRoute + hostnames from `_gateways/code-for-philly.yaml` to + `_gateways/codeforphilly-ng.yaml` and flipping the `CFP_SITE_HOST` patch. + Rollback is `git revert`. +- **Private storage** is a filesystem PVC in prod (`/app/private-storage`). +- **Secrets** live in `cfp-live-cluster/codeforphilly-ng.secrets/` as three + SealedSecrets: `codeforphilly-secrets`, `codeforphilly-saml` (carried over + from laddr's `saml2`), `codeforphilly-data-deploy-key`. +- **Slack SAML plan**: SSO optional first, "Test configuration" against the + `next.codeforphilly.org` metadata, then update only the SSO URL at cutover. +- **Data refresh pipeline**: import → merge `legacy-import` into `published` + (thousands of deleted-by-us / modified-by-them conflicts for previously + pruned spam, resolved by taking the import's version) → prune-spam → push, + which hot-reloads sandbox and prod. First full refresh 2026-09-09/10: + 36,254 imported, 22,625 after prune. +- **Legacy credentials**: export from `emergence-site`.`people` inside the + laddr pod with the Habitat mysql client; load via `kubectl cp` + + `rollout restart`. First prod load 2026-09-10: 21,761 profiles/credentials. +- **Monitoring**: nothing external exists yet; say so rather than delete the + checklist. + +## Validation + +- [ ] `grep -rn 'cfp-prod-cluster\|staging.k8s\|cutover-window-policy\|T+180' docs/ .claude/CLAUDE.md` returns nothing. +- [ ] No doc names `main` as a data-repo branch; runtime branch is `published` everywhere. +- [ ] No doc gives S3/GCS as the production private-storage path; the filesystem PVC is. +- [ ] `cutover.md` T-0 and rollback sections describe the gateway-listener commit and its revert; no DNS/TTL steps remain. +- [ ] `legacy-credentials-import.md` export command targets `emergence-site`.`people` via the Habitat client. +- [ ] `npm run lint` clean. + +## Risks / unknowns + +- **Claims not verifiable from the repos** (Slack admin behavior, GitHub OAuth + app names, MySQL schema details, import counts) are encoded as stated by the + operator who ran them; flagged in the PR body. +- **Reconcile against production** — the runtime image ships `dist/` only (no + `tsx`, no `scripts/`), so `script:reconcile` cannot run inside the prod pod, + and the private store must not be copied to a laptop. Left as a documented + gap rather than a prescribed workaround. + +## Notes + +(Populated at closeout.) + +## Follow-ups + +(Populated at closeout.) From 9ebc0216319d717bef2e7a1ca41e937ed7a58657 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 10 Sep 2026 02:06:06 -0400 Subject: [PATCH 2/8] docs(ops): rewrite cutover runbook around the gateway hostname move The runbook still described a DNS flip with a lowered TTL, a staging host that never existed, an S3 bucket, a data-repo `main` branch, and a T+90/T+180 claim window + password purge. None of that matches what is deployed: - Production runs in cfp-live-cluster (ns codeforphilly-ng) at next.codeforphilly.org, alongside laddr in code-for-philly. The apex and *.codeforphilly.org already resolve to that cluster's gateway, so T-0 is one commit moving the apex/www listeners and HTTPRoute hostnames from _gateways/code-for-philly.yaml to _gateways/codeforphilly-ng.yaml and flipping CFP_SITE_HOST. Rollback is a revert of that commit. - The data repo has no `main`; the refresh is import -> merge legacy-import into published (thousands of deleted-by-us conflicts for pruned spam, resolved by taking the import) -> prune -> push, which hot-reloads sandbox and prod. First run: 36,254 -> 22,625. - Slack SAML can only be tested in the real workspace, so the plan is SSO-optional first, Test configuration against the next.* metadata, then update only the SSO URL at cutover. A bad SAML flip is fixed in Slack, not by moving the hostname back. - Legacy password sign-in persists indefinitely per account-migration.md; the T+90 mailout and T+180 purge sections are gone, and the announcement templates no longer tell members their password is going away. - Monitoring: nothing external exists yet; the T+1h window says so instead of pointing at UptimeRobot. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR --- docs/operations/cutover-announcement.md | 69 ++-- docs/operations/cutover-rollback.md | 137 ++++--- docs/operations/cutover.md | 479 +++++++++++++++--------- 3 files changed, 419 insertions(+), 266 deletions(-) diff --git a/docs/operations/cutover-announcement.md b/docs/operations/cutover-announcement.md index 1adf51b..4f4aa43 100644 --- a/docs/operations/cutover-announcement.md +++ b/docs/operations/cutover-announcement.md @@ -20,16 +20,17 @@ same accounts. The active cutover is about an hour. Expect ~5 minutes of What changes for you: - :white_check_mark: All your URLs keep working (legacy redirects in place) - :white_check_mark: Your Slack identity is preserved automatically -- :arrows_counterclockwise: First sign-in after cutover routes you through - GitHub OAuth + a one-click "claim your account" flow -- :no_entry_sign: Username/password sign-in is going away — GitHub is the - primary login from cutover forward +- :white_check_mark: Your existing username + password keep working +- :arrows_counterclockwise: You can also "Sign in with GitHub" — if your + GitHub email matches your CFP email your account links automatically; + otherwise a one-click claim flow does it. We'd love you to link GitHub, + but there's no deadline What we need from you BEFORE cutover: - Hold off on edits to your profile / projects starting {{ freeze_date }} (we're freezing writes to make the migration clean) -- Make sure you remember which GitHub account is associated with your CFP - email — your first sign-in needs to come from that GitHub account +- If you plan to sign in with GitHub, check which GitHub account uses your + CFP email — matching emails link automatically If you have questions: drop them in this thread or DM @{{ cutover_lead_slack }}. ``` @@ -46,15 +47,16 @@ On {{ cutover_date_long }} we're moving codeforphilly.org to a new platform. What's changing - The site is being rebuilt on a modern stack — same look, same URLs, same projects. -- Sign-in is moving to GitHub OAuth. Your password is no longer needed. +- Your existing username and password keep working. You can also sign in + with GitHub and link the two. - Your Slack identity is preserved automatically. What you need to do - Nothing right now — but please don't edit your profile or projects between {{ freeze_date }} and {{ cutover_date_short }}. -- After cutover, sign in via the new "Sign in with GitHub" button. If - your CFP email matches your GitHub email, your account is claimed - automatically. If not, follow the on-screen claim flow. +- After cutover, sign in as usual, or try the new "Sign in with GitHub" + button. If your CFP email matches your GitHub email, your account is + linked automatically. If not, follow the on-screen claim flow. When it happens - {{ cutover_date_long }} starting at {{ cutover_time }}. @@ -65,10 +67,14 @@ Questions? Reply to this email or ping us in Slack. — Code for Philly ``` -## T-0: maintenance page +## T-0: maintenance page (optional) -The static page served from the legacy site while DNS propagates. Plain HTML; -no JavaScript needed. +A static page for the legacy site during the hostname move. Cutover is a +gateway-listener change with no DNS propagation +([cutover.md → T-0](cutover.md#t-0-cutover)), so the switch is effectively +instant per hostname and this page is optional — use it if you want a +visible "hold on" while the final data delta runs. Plain HTML; no +JavaScript needed. ```html @@ -120,10 +126,18 @@ https://github.com/CodeForPhilly/codeforphilly-ng/issues or just reply here. ``` -## T+90 days: unclaimed-account reminder +## Unclaimed-account reminder (not scheduled) -Sent automatically by `apps/api/scripts/cutover-mailout.ts`. The template is -in code — see `buildEmailBody()` in that file. Reproduced here for review: +`apps/api/scripts/cutover-mailout.ts` can email members whose laddr account +has not yet been linked to GitHub. **It is not on the cutover timeline.** +Per [account-migration.md](../../specs/behaviors/account-migration.md#sunset-deferred) +legacy password sign-in has no deadline, so there is nothing to remind +people *of*; the only nudge is the "Connect GitHub" banner on `/account`. +If a future spec change sets a sunset date, this template is the starting +point — until then, don't send it. The template is in code — see +`buildEmailBody()` in that file. Reproduced here for review (note the last +paragraph about retiring accounts has no backing in the spec and would need +one before any send): Subject: `Action needed: claim your Code for Philly account` @@ -144,29 +158,6 @@ unclaimed for one year may be retired. — Code for Philly ``` -## T+180 days: password-credential drain notice - -A short Slack post in `#announcements` and an info-only email to remaining -unclaimed Persons. Honest about the irreversible step. - -### Slack - -``` -:warning: Six months post-cutover - -It's been six months since we moved to the new platform. We're cleaning up -the remaining legacy password records — about {{ count }} accounts. - -If you signed in via GitHub at any point: you're already migrated, this -doesn't affect you. - -If you have a laddr account you haven't logged into yet: please log in -in the next 14 days. After that, you'll need to contact a staff member -to verify your identity before reclaiming the account. - -Questions: ping @{{ cutover_lead_slack }}. -``` - ## Localization We don't send in any language other than English at v1. If we ever expand diff --git a/docs/operations/cutover-rollback.md b/docs/operations/cutover-rollback.md index 5fc0a0c..5f035f6 100644 --- a/docs/operations/cutover-rollback.md +++ b/docs/operations/cutover-rollback.md @@ -7,15 +7,21 @@ you want this in muscle memory, not on a screen you're scrolling through. > Companion: [cutover.md](cutover.md) (runbook), > [cutover-announcement.md](cutover-announcement.md) (Slack + email templates). +## The short version + +Cutover is one commit in `cfp-live-cluster` (gateway listeners + HTTPRoute +hostnames + `CFP_SITE_HOST`). **Rollback is `git revert` of that commit**, +pushed through the same GitOps path. No DNS is involved in either direction. + ## When to roll back The criterion is "is the rewrite serving users worse than legacy was, and will it stay that way longer than 15 minutes?" Concrete triggers: - `/api/health` flapping or stuck 503 with no clear single-fix path +- The apex or `www` serving an invalid certificate that cert-manager isn't + resolving within a few minutes - OAuth callback failure rate > 50% for more than 5 minutes -- Slack SAML failure rate > 50% for more than 5 minutes (Slack is critical - to brigade ops — this is a bigger deal than the OAuth callback) - A data corruption that's already in the gitsheets commit history and hard to surgically revert - Any unrecoverable cluster issue that isn't a five-minute fix (e.g. PVC @@ -23,18 +29,25 @@ will it stay that way longer than 15 minutes?" Concrete triggers: **Not rollback triggers:** a single failing route, a cosmetic UI bug, a known-not-blocking warning during boot, OAuth callback failures for a -single GitHub account that's been deactivated. +single GitHub account that's been deactivated, **or a Slack SSO failure** — +SSO is set to optional before cutover, members fall back to email magic +links, and the fix is correcting the SSO URL in Slack (see below), not +moving the hostname back. ## Point of no return The rollback is only safe **before the first real-user mutation lands on -the rewrite**. After a user signs in for the first time on the rewrite a +the rewrite via the apex**. After a user signs in for the first time a `PrivateProfile` is written; after they edit anything a gitsheets commit is created. Those records exist only on the new side. Rolling back would either lose them or require a manual re-import after the next attempt. +(The rewrite has been live at `next.codeforphilly.org` since before +cutover, so some divergence from laddr already exists. The point-of-no-return +question is specifically about writes that arrive *because* the apex moved.) + In practice this means: the rollback window is the first ~15 minutes -after DNS flip. After that, fix forward. +after the cutover PR applies. After that, fix forward. The cutover lead announces "point of no return crossed" in the cutover Slack thread when the first user-created record (not the importer's @@ -42,39 +55,57 @@ records) lands. After that, this document is informational only. ## Rollback procedure -The 4-step sequence. Cutover lead at the keyboard; engineering second -reads the steps back. +Cutover lead at the keyboard; engineering second reads the steps back. -### 1. DNS flip back to legacy +### 1. Revert the cutover commit ```bash -# Whatever your DNS provider is, flip codeforphilly.org back to the -# legacy ingress / load balancer. +cd ~/Repositories/cfp-live-cluster +git fetch origin +git checkout -b revert/cutover origin/main +git revert --mainline 1 # or the single commit sha if it was a plain commit +git push -u origin revert/cutover +gh-axi pr create --base main --title "Revert cutover: apex back to laddr" ``` -TTL was lowered to 60s a week before cutover, so propagation completes -in under two minutes for most resolvers. Verify with `dig +short codeforphilly.org` -from a fresh resolver. +Merge it. "Build k8s-manifests" projects to `releases/k8s-manifests`, the +bot opens the PR into `deploys/k8s-manifests`, merge that. The apex and +`www` listeners return to `_gateways/code-for-philly.yaml`, the rewrite's +`CFP_SITE_HOST` goes back to `next.codeforphilly.org`, and the rewrite pod +rolls once more. + +**Emergency shortcut** if the GitOps round-trip is too slow: revert +directly on `deploys/k8s-manifests` (see [runbook.md → Last-resort +recovery](runbook.md#3-last-resort-recovery)) and reconcile `main` +afterwards. + +Verify: `curl -sSI https://codeforphilly.org/` serves laddr; +`https://next.codeforphilly.org/` still serves the rewrite. -### 2. Re-enable legacy writes +### 2. Put Slack SSO back + +If the SSO URL in Slack was already updated to the apex, the Slack +workspace admin changes it back to +`https://next.codeforphilly.org/api/saml/slack/sso`. Because SSO is +optional, nothing here is urgent — members without a working SSO path use +email magic links until it's fixed. Do **not** move the hostname to fix a +SAML problem. + +### 3. Re-enable legacy writes If you flipped the legacy site to read-only at T-7, undo that flag now. The legacy DB has not been touched during the migration window (the importer -only reads from `?format=json` endpoints). Writes resume from the same state -as just-before-freeze. - -### 3. Take down the rewrite ingress +only reads from `?format=json` endpoints and the credentials export is +read-only). Writes resume from the same state as just-before-freeze. -```bash -# Scale the rewrite Deployment to zero so no traffic can hit it even if -# a stale DNS resolver still points there. -kubectl -n codeforphilly scale deploy/codeforphilly --replicas=0 -``` +### 4. Point the hot-reload workflow back -Or just remove the Ingress resource. The Deployment can stay running for -future debugging; what matters is no public traffic reaches it. +If you already switched the prod URL in the data repo's +`notify-deployments.yml` to the apex, switch it back to +`https://next.codeforphilly.org/api/_internal/reload-data` so `published` +pushes keep reaching the rewrite pod. -### 4. Post the rollback notice +### 5. Post the rollback notice Slack `#announcements`: @@ -103,9 +134,10 @@ Once users are back on legacy and the pressure is off: 1. Capture the failure in writing — what error, what frequency, in what logs, what was happening on the user side. -2. Pull `kubectl logs` for the dead pod (`--previous` if it CrashLoopBacked). -3. Pull the reconciliation report from T-1 and the dry-run report from - T-3 — compare against what landed on the production side. +2. Pull `kubectl -n codeforphilly-ng logs deploy/codeforphilly --previous` + for the dead pod (if it CrashLoopBacked). +3. Pull the dry-run report from T-3 and the refresh counts from T-1 — + compare against what landed on the production side. 4. File an issue documenting what we know vs. what we don't. The point is to have a tight description of the failure mode before @@ -119,44 +151,47 @@ modifications: - T-7 announce step is shorter — refer to the original. "We're trying again on {{ new_date }}; same plan, no further changes from your side." -- T-3 dry-run **must** explicitly reproduce the failure mode that triggered +- T-3 rehearsal **must** explicitly reproduce the failure mode that triggered the rollback and confirm the fix. - The cutover thread includes a "what we learned and changed" entry above the new timeline. ## After a partial-write rollback (rare and ugly) -The procedure above assumes nobody wrote anything to the rewrite before -rollback. If that assumption was wrong (you crossed point-of-no-return -and *still* had to roll back because something worse happened): +The procedure above assumes nobody wrote anything to the rewrite via the +apex before rollback. If that assumption was wrong (you crossed +point-of-no-return and *still* had to roll back because something worse +happened): -1. **Don't panic.** The rewrite's data is still in `git` history — nothing - is silently deleted. +1. **Don't panic.** The rewrite's public data is still in `git` history — + nothing is silently deleted. 2. Identify the time window in which user writes happened on the rewrite. - Pull every commit on the data repo between import-commit and the - rollback moment. Same for the private-storage bucket's versioned - `.jsonl` history. + Pull every commit on the data repo's `published` branch between the + refresh commit and the rollback moment. The private store is `.jsonl` + on the pod's PVC (`/app/private-storage`) — it has no versioning, so + snapshot it (`kubectl cp` out of the pod, kept in-cluster or on an + operator machine only as long as needed) before anything else touches it. 3. Export those records as a JSON diff. Send to the legacy site's owner to manually replay if they're irreplaceable, or — much more commonly — email the affected users explaining their write didn't land and asking them to redo it after the second cutover attempt. -4. Hard-reset the rewrite's data repo to the post-import commit. Re-deploy - when ready. +4. Hard-reset the rewrite's `published` branch to the post-refresh commit. + Re-deploy when ready. -This case is operationally painful but not data-loss; the records are in -git history forever. The cost is reconciliation effort. +This case is operationally painful but not data-loss on the public side; +the records are in git history forever. The cost is reconciliation effort. ## What rollback does NOT cover -- **Slack workspace SAML disconnection.** If you flipped the Slack SSO - config over to the new IdP *before* validating, you may need to ask - the Slack workspace owner to swap back. This is why the runbook says - to pre-stage but not activate the SAML connection at T-7. +- **Slack SSO.** Reverting the gateway does nothing for SAML. A wrong SSO + URL or a bad assertion is fixed in Slack's SAML settings; a member who + can't sign in via SSO uses the email magic link because SSO is optional. + If the workspace was mistakenly set to SSO *required*, the admin sets it + back to optional — that is the actual lockout fix. - **GitHub OAuth app suspension by GitHub.** If GitHub deactivated the - OAuth app (rate limit, ToS issue), rolling back the DNS doesn't help + OAuth app (rate limit, ToS issue), moving the hostname back doesn't help — legacy doesn't use OAuth. Contact GitHub support; legacy will work unaffected because legacy uses passwords. -- **DNS resolver caching beyond TTL.** Some corporate networks ignore - DNS TTLs and cache for hours. Users behind such a network will see - the rewrite for longer than the 60s flip implies. Rare; document the - user-side workaround (open a private tab / change DNS). +- **Writes already made at `next.codeforphilly.org`.** The rewrite was + live there before cutover and stays live after a rollback. Anything + members did there is real and stays. diff --git a/docs/operations/cutover.md b/docs/operations/cutover.md index ecd5bf7..a20c9a2 100644 --- a/docs/operations/cutover.md +++ b/docs/operations/cutover.md @@ -1,22 +1,46 @@ # Cutover runbook The sequenced playbook for moving codeforphilly.org from legacy laddr to the -rewrite. Mirrors the timeline in [plans/cutover-prep.md](../../plans/cutover-prep.md) -and the policy in [specs/behaviors/account-migration.md#cutover-window-policy](../../specs/behaviors/account-migration.md#cutover-window-policy). +rewrite. Grew out of [plans/cutover-prep.md](../../plans/cutover-prep.md); +the account-side policy is [specs/behaviors/account-migration.md](../../specs/behaviors/account-migration.md). > Companion docs: [cutover-announcement.md](cutover-announcement.md) (Slack + > email templates) and [cutover-rollback.md](cutover-rollback.md) (what to do > if it goes sideways). For routine ops see [deploy.md](deploy.md), -> [runbook.md](runbook.md), [secrets.md](secrets.md). +> [runbook.md](runbook.md), [secrets.md](secrets.md), +> [legacy-credentials-import.md](legacy-credentials-import.md). + +## Where things run + +| | GitOps repo | Namespace | Host | +|---|---|---|---| +| **Production** (the rewrite) | [`cfp-live-cluster`](https://github.com/CodeForPhilly/cfp-live-cluster) | `codeforphilly-ng` | `https://next.codeforphilly.org` before cutover; `https://codeforphilly.org` after | +| **Legacy laddr** | `cfp-live-cluster` | `code-for-philly` | `https://codeforphilly.org` before cutover; `https://codeforphilly.live.k8s.phl.io` throughout | +| **Sandbox** | [`cfp-sandbox-cluster`](https://github.com/CodeForPhilly/cfp-sandbox-cluster) | `codeforphilly-rewrite-sandbox` | `https://next-v2.codeforphilly.org` | + +There is no separate staging environment; the sandbox plays that role. The +production pod is already running and serving real data at +`next.codeforphilly.org` — cutover changes which hostnames reach it, nothing +about the pod itself. + +**Cutover needs no DNS change.** DNS for `codeforphilly.org` is managed in +[`CodeForPhilly/ops`](https://github.com/CodeForPhilly/ops) (`tf/dns`, +OpenTofu, Cloud DNS project `openphl-1177`). `codeforphilly.org`, +`*.codeforphilly.org` and `*.live.k8s.phl.io` all already resolve to the +live cluster's Envoy gateway. Which app answers for a hostname is decided +by the Gateway listeners in `cfp-live-cluster/_gateways/`, so T-0 is a +single commit in that repo. There is no TTL to lower and no propagation to +wait on. ## Roles | Role | Responsibility | |------|----------------| -| **Cutover lead** | Owns the timeline; the only person who runs the actual T-0 commands. | +| **Cutover lead** | Owns the timeline; the only person who merges the T-0 commit. | | **Comms lead** | Owns Slack + email announcements; on standby for member questions. | -| **Engineering second** | Runs reconciliation + dry-run; co-pilot for cutover lead at T-0. | -| **On-call** | Watches `/api/health` and the alerts channel during T-0 → T+24h. | +| **Engineering second** | Runs the data refresh + rehearsal; co-pilot for cutover lead at T-0. | +| **Slack workspace admin** | Owns the SAML SSO settings in Slack (SSO optional, Test configuration, SSO URL update). | +| **On-call** | Watches pod logs and `/api/health` during T-0 → T+24h. | These can be the same person at small org scale, but the role assignments should be explicit in the cutover Slack post. @@ -25,44 +49,62 @@ should be explicit in the cutover Slack post. | When | What | Reversible? | |------|------|-------------| -| T-7 days | Announce; freeze write workflow on legacy site | Yes | -| T-3 days | Final staging-rehearsal `cutover-dry-run.ts` | Yes | -| T-1 day | Final import from live laddr JSON; verify counts | Yes | -| T-0 | DNS flip, maintenance page comes down | **Point of no return** when first new sign-in lands | -| T+1h | Active monitoring; smoke-test public flows | Yes (rollback) | -| T+24h | Post-cutover all-clear in Slack | Yes (rollback) | -| T+90d | Reminder mailout to unclaimed Persons | Information-only | -| T+180d | Delete unclaimed `LegacyPasswordCredential` rows | Information-only | +| T-7 days | Announce; freeze write workflow on legacy site; Slack SSO to "optional" | Yes | +| T-3 days | Rehearsal: dry-run against sandbox, smoke prod at `next.codeforphilly.org`, Slack "Test configuration" | Yes | +| T-1 day | Data refresh (import → merge → prune → push); credentials load; verify counts | Yes | +| T-0 | Gateway hostname move + `CFP_SITE_HOST` flip (one commit); Slack SSO URL update | **Point of no return** when first new record lands | +| T+1h | Active monitoring; smoke-test public flows | Yes (revert) | +| T+24h | Post-cutover all-clear in Slack | Yes (revert) | +| T+7d | Reconciliation check | Information-only | + +There is deliberately **no** T+90 claim-window mailout and **no** T+180 +password purge. Per [account-migration.md](../../specs/behaviors/account-migration.md#sunset-deferred), +legacy password sign-in persists indefinitely with a link-GitHub nag on +`/account`; any future sunset is undated and needs its own spec change first. ## T-7 days: announce + freeze 1. Post the cutover announcement from [cutover-announcement.md](cutover-announcement.md) to `#announcements` and email all members via Postmark. -2. Lower DNS TTL on `codeforphilly.org` to 60s. Verify with `dig`. -3. **Freeze legacy writes.** Either put a banner on the legacy site asking +2. **Freeze legacy writes.** Either put a banner on the legacy site asking members to hold off on edits, or flip a feature flag making it read-only. - The point is to reduce the size of the "delta" between the T-3 dump and - T-0 dump. -4. Notify cluster operators that you'll be moving production traffic next week. -5. Confirm the GitHub OAuth app's production redirect URI is registered. -6. Confirm the Slack SAML connection is **pre-staged** (Slack admin has the - new IdP metadata; not active yet). -7. Confirm sealed-secrets are in the production cluster — see - [secrets.md](secrets.md). Missing secrets at T-0 is the most common - single failure mode. + The point is to reduce the size of the "delta" between the T-1 refresh + and the T-0 refresh. +3. Notify cluster operators that you'll be moving the apex hostname next week. +4. **GitHub OAuth.** The production app is the org-owned OAuth app + **"Code for Philly"** (the laddr one was renamed "Code for Philly + (legacy)"). Its callback is registered on the apex; GitHub accepts + redirects to subdomains of the registered callback, which is why + `next.codeforphilly.org` already works against it. **No edit is needed + at cutover.** Confirm the app still exists and its client ID matches the + one in `codeforphilly-secrets`. +5. **Slack SAML — lockout insurance first.** Slack SSO cannot be tested in a + throwaway workspace (SAML is enterprise-only), so the real workspace is + the only test bed. Have the Slack workspace admin set SSO to + **"optional"** now. Members can then always fall back to email magic + links, so a wrong SAML setting at any later step is an inconvenience, + not a lockout. +6. Confirm the three SealedSecrets are present in + `cfp-live-cluster/codeforphilly-ng.secrets/` and decrypted in the + `codeforphilly-ng` namespace — see [secrets.md](secrets.md). Missing + secrets at T-0 is the most common single failure mode. -## T-3 days: full staging rehearsal + ```bash + kubectl -n codeforphilly-ng get secret codeforphilly-secrets codeforphilly-saml codeforphilly-data-deploy-key + ``` + +## T-3 days: rehearsal -The rehearsal must run end-to-end against `codeforphilly-rewrite-staging.k8s.phl.io` -and produce a passing report. +There is no staging cluster to rehearse the hostname move itself on; what +you can rehearse is everything that happens *behind* the hostname. -1. Run the dry-run script against the live laddr site: +1. Run the dry-run script with the sandbox as target: ```bash npm run -w apps/api script:cutover-dry-run -- \ --source-host=codeforphilly.org \ --data-repo=/scratch/dry-run-data \ - --target=https://codeforphilly-rewrite-staging.k8s.phl.io \ + --target=https://next-v2.codeforphilly.org \ --json=/scratch/dry-run-T3.json ``` @@ -70,44 +112,59 @@ and produce a passing report. - `stages.import` must be `true`. - `stages.countDiff` must be `true` (every sheet's imported count is within tolerance of the server's reported `total`). - `stages.smoke` must be `true` (all probes return 2xx/3xx). -3. Manually verify Slack SAML continuity for a test laddr user. - This is the highest-stakes single check. See - [specs/api/saml.md](../../specs/api/saml.md): a user's `slackSamlNameId` - must equal their pre-cutover Slack NameID byte-for-byte. -4. File any anomalies, schedule a re-run before T-0 if anything fails. +3. Repeat the smoke probes by hand against production at + `https://next.codeforphilly.org` — `/api/health/ready`, + `/api/people/`, `/api/projects/`, + `/projects?ID=` (must 301 to the slug URL), a GitHub sign-in. +4. **Slack SAML.** In Slack's SSO settings, point the IdP metadata at + `https://next.codeforphilly.org/api/saml/slack/metadata` and use Slack's + **"Test configuration"** against it. The IdP entity ID it reports must be + `https://codeforphilly.org/api/saml/slack/metadata` — that value is + `SAML_ENTITY_ID`'s default and does not follow the host, so it is the + same before and after cutover ([specs/api/saml.md](../../specs/api/saml.md#idp-identity-and-hosts)). + The signing cert is the one carried over from laddr's `saml2` secret, so + Slack keeps trusting the same cert. A test user's NameID must equal + their pre-cutover Slack NameID byte-for-byte. +5. File any anomalies, schedule a re-run before T-0 if anything fails. If the dry-run reports unexpectedly low imported counts for any sheet, **stop**. Either the laddr JSON shape drifted (a new field broke Zod validation) or the importer needs an update. Don't proceed to T-0 with silently-dropped data. -## T-1 day: production migration +## T-1 day: data refresh + credentials + +The data repo (`CodeForPhilly/codeforphilly-data`) has **no `main`**. The +runtime-served branch is **`published`**; `legacy-import` holds the raw +importer snapshots. The refresh pipeline is always: + +**import → merge `legacy-import` into `published` → prune-spam → push** -The production import is a snapshot commit on the `legacy-import` branch of -the production data repo. Private data (emails, password hashes) is **not** -populated by this importer — that is a separate one-shot run of the -credentials importer; see -[legacy-credentials-import.md](./legacy-credentials-import.md) for the -CSV format, run command, and deploy steps (sandbox PVC or production -S3-compat bucket). +Pushing `published` fires the data repo's +`.github/workflows/notify-deployments.yml`, which POSTs the hot-reload +webhook on sandbox *and* prod; both pods rebuild in-memory state in place +with no restart ([runbook.md#hot-reload-webhook](runbook.md#hot-reload-webhook)). -1. **Bare-clone** the production data repo locally — the importer matches - the running pod's invariant ([storage.md → "The data clone is bare"](../../specs/behaviors/storage.md)) +The first full refresh through this pipeline ran 2026-09-09/10: **36,254** +person records imported, **22,625** left on `published` after prune. Use +those as the order-of-magnitude sanity check for your counts. + +### Public data + +1. **Bare-clone** the data repo — the importer matches the running pod's + invariant ([storage.md → "The data clone is bare"](../../specs/behaviors/storage.md)) and `openPublicStore` rejects a non-bare path: ```bash - git clone --bare git@github.com:CodeForPhilly/codeforphilly-data.git /scratch/codeforphilly-data + git clone --bare git@github.com:CodeForPhilly/codeforphilly-data.git /scratch/codeforphilly-data.git ``` - For one-off imports the convention is to clone into `/tmp` and clean up - after pushing — keeps your working data-repo clone clean. - -2. Run the importer against the production target — **with `--dry-run` first**: +2. Run the importer — **with `--dry-run` first**: ```bash npm run -w apps/api script:import-laddr -- \ --source-host=codeforphilly.org \ - --data-repo=/scratch/codeforphilly-data \ + --data-repo=/scratch/codeforphilly-data.git \ --branch=legacy-import \ --dry-run ``` @@ -116,104 +173,184 @@ S3-compat bucket). namespaces, and skipped HTTP-only buzz URLs are expected; zod errors are not. 4. Run the importer **without `--dry-run`**. This creates one snapshot - commit on the `legacy-import` branch. -5. Push the `legacy-import` branch to the production GitHub remote. -6. Merge `legacy-import` into `main` (operator step — review the diff in a - PR, resolve any path-template conflicts, then merge). -7. **Prune confident-spam** from the runtime branch before it goes live. The - merge in step 6 re-adds the full raw import (spam included); the deployed pod - cannot hold the unpruned set in memory, so this step is **mandatory** after - every import/merge. See + commit on `legacy-import`. Push it. +5. **Merge `legacy-import` into `published`.** This needs a working tree + (clone from your bare clone). Expect **thousands of conflicts** of the + form `deleted by us / modified by them`: every previously-pruned spam + person is absent on `published` and present (possibly modified) in the + fresh snapshot. Resolve them all by **taking the import's version** — + the prune step in the next step removes them again: + + ```bash + git clone /scratch/codeforphilly-data.git /scratch/codeforphilly-data-wt + cd /scratch/codeforphilly-data-wt + git checkout published + git merge origin/legacy-import || true + git diff --name-only --diff-filter=U | xargs git checkout --theirs -- + git add -A people/ # the conflicts are person records + git commit --no-edit + git push origin published:published # back to the bare clone only — don't push to GitHub yet + ``` + + Review `git status` before the `git add`; if conflicts show up outside + `people/`, look at them individually rather than bulk-resolving. + +6. **Prune confident-spam** from `published` before it goes anywhere. The + merge re-added the full raw import (spam included); the deployed pod + cannot hold the unpruned set in memory, so this step is **mandatory** + after every import/merge. See [spam-detection.md → Applying spam decisions](./spam-detection.md#applying-spam-decisions--the-prune-step): ```bash npm run -w apps/api script:prune-spam -- \ - --data-repo=/scratch/codeforphilly-data \ + --data-repo=/scratch/codeforphilly-data.git \ --evaluations-ref=spam-detection \ --branch=published \ - --dry-run # review counts, then drop --dry-run and push the branch + --dry-run # review counts, then drop --dry-run ``` Newly-imported accounts with no spam verdict yet are kept (the rule only - removes *confident* spam), so an incomplete eval pass is safe — it just keeps - more people than strictly necessary. -8. Run reconciliation: + removes *confident* spam), so an incomplete eval pass is safe — it just + keeps more people than strictly necessary. - ```bash - npm run -w apps/api script:reconcile -- --json=/scratch/reconcile-T1.json - ``` +7. **Push `published` to GitHub.** `notify-deployments.yml` hot-reloads + sandbox and prod. Watch the workflow run go green for both targets, then + confirm `https://next.codeforphilly.org/api/people/` reflects a + record you know changed upstream. + +### Private data (credentials) + +Emails and password hashes are **not** in the public import. They come from +the laddr MySQL database via the credentials importer and are loaded onto +the production pod's private-storage PVC — full procedure in +[legacy-credentials-import.md](./legacy-credentials-import.md). In short: - Every counter should be zero in the orphan + inconsistent categories. - If anything is flagged, **stop** and investigate before T-0. +1. Export `Username, Email, Password` from `emergence-site`.`people` inside + the laddr pod (Habitat mysql client) to a quoted CSV. +2. Run `script:import-laddr-credentials` against the bare clone's + `published` branch to produce `profiles.jsonl` + `legacy-passwords.jsonl`. +3. `kubectl -n codeforphilly-ng cp` both files into the pod's + `/app/private-storage/`, then `kubectl -n codeforphilly-ng rollout + restart deploy/codeforphilly`. **Hot reload does not cover the private + store**; the restart is required. -9. Deploy the rewrite to production via the production GitOps repo (a - sibling to [`cfp-sandbox-cluster`](https://github.com/CodeForPhilly/cfp-sandbox-cluster) - — see [deploy.md](deploy.md)). The pod will boot against the - just-imported data + bucket but receive no public traffic yet (Gateway - hostname not pointed at the prod LoadBalancer yet). +The first production load ran 2026-09-10: **21,761** profiles/credentials. -10. Smoke-test the production hostname through `/etc/hosts` or via direct - cluster IP: hit `/api/health`, `/api/people/`, - `/api/projects/`. Don't yet flip DNS. +Re-running after real users have signed in **reverts their rehashed +(argon2id) credentials to the legacy hashes** — see the safety notes in +that doc before repeating the load at T-0. + +### Confirm the deployed version + +Production is pinned in `cfp-live-cluster` by two values that move +together: `.holo/sources/codeforphilly-ng.toml` (`ref = "refs/tags/vX.Y.Z"`) +and `images[].newTag` in `codeforphilly-ng/app/kustomization.yaml`. Confirm +they match the release you intend to cut over on, and that the sandbox has +run that same tag for a while. ## T-0: cutover At T-0 the team is on a call together. Cutover lead at the keyboard; -engineering second has the runbook open and reads checks back. +engineering second has the runbook open and reads checks back. Legacy stays +reachable at `https://codeforphilly.live.k8s.phl.io` throughout, so a +maintenance page is optional — the switch itself is atomic per hostname. -1. **0:00 — maintenance page.** Put a static maintenance page on - the legacy `codeforphilly.org`. (Legacy site can stay up under - the hood; we just don't want users hitting a half-state.) -2. **0:01 — final delta.** Re-run the importer against the live laddr site - into the same data-repo path. UUIDs are read-forward from the previous +1. **0:00 — final delta.** Re-run the T-1 public-data pipeline (import → + merge → prune → push). UUIDs are read-forward from the previous snapshot's tree, so the diff between this commit and the T-1 commit is - exactly the records that changed upstream since T-1. **Then merge into the - runtime branch and re-run the spam prune (step 7 above) before the pod - reloads** — the final-delta merge re-adds raw import records, so an unpruned - reload would re-bloat memory. -3. **0:05 — DNS flip.** Update the `codeforphilly.org` A/CNAME to point - at the rewrite's ingress. TTL was lowered to 60s a week ago, so - propagation completes in under two minutes for most resolvers. -4. **0:10 — verify.** From four different networks (lead's laptop, - engineering second's tethered phone, a VPN exit node, a CFP volunteer's - home network), confirm `codeforphilly.org` serves the rewrite. Test: + exactly the records that changed upstream since T-1. The push + hot-reloads prod. Only re-run the credentials load if nobody has signed + in at `next.codeforphilly.org` since T-1 (or you accept the rehash reset). + +2. **0:05 — the cutover commit.** One commit in `cfp-live-cluster`, on a + branch, PR'd into `main`: + + - In `_gateways/code-for-philly.yaml`: remove the `https-apex` + (`codeforphilly.org`) and `https-www` (`www.codeforphilly.org`) + listeners and drop those hostnames from the legacy HTTPRoute. Leave + the `codeforphilly.live.k8s.phl.io` listener so laddr stays reachable. + - In `_gateways/codeforphilly-ng.yaml`: add those two listeners (with + their `certificateRefs`) and hostnames to the rewrite's Gateway + + HTTPRoute. **Keep the existing `next.codeforphilly.org` listener** — + it is a permanent alias. + - In `codeforphilly-ng/app/kustomization.yaml`: change the + `CFP_SITE_HOST` patch value from `next.codeforphilly.org` to + `codeforphilly.org`. This drives the markdown renderer's + internal/external link split, canonical URLs in email, and the SAML + metadata's `SingleSignOnService` locations. + + Merge to `main`. The **"Build k8s-manifests"** workflow projects to + `releases/k8s-manifests`, a bot opens a PR into `deploys/k8s-manifests`, + and merging *that* PR applies. The ConfigMap change rolls the pod + (`Recreate` strategy) — expect a short readiness gap while it re-clones + and reloads. cert-manager issues certificates for the moved hostnames + into the `codeforphilly-ng` namespace; allow a minute or two for ACME. + +3. **0:10 — verify.** From several networks (lead's laptop, engineering + second's tethered phone, a volunteer's home network), confirm + `https://codeforphilly.org` and `https://www.codeforphilly.org` serve + the rewrite with valid certificates: - Anonymous browse of `/projects` and `/members` (read paths). - - Anonymous browse of a known legacy URL form like - `/projects?ID=10` — must 301 to the slug URL. - - A team member signs in via GitHub OAuth; if their account is in - the laddr dataset they'll see the claim flow. - - Slack SAML: a team member logs out of Slack and re-authenticates - via the new IdP. Verify their Slack identity is preserved. -5. **0:15 — remove maintenance page.** The legacy site can stay up - read-only forever; we just need it not to be the public landing page. + - A known legacy URL form like `/projects?ID=10` — must 301 to the slug URL. + - A team member signs in via GitHub OAuth; a member with a laddr + account signs in with their legacy password. + - `https://next.codeforphilly.org` still works (alias). + - `https://codeforphilly.live.k8s.phl.io` still serves laddr. + +4. **0:15 — Slack SSO URL.** The Slack workspace admin updates **only the + SSO URL** in Slack's SAML settings from + `https://next.codeforphilly.org/api/saml/slack/sso` to + `https://codeforphilly.org/api/saml/slack/sso` (or re-fetches the + metadata from the apex — the entity ID and cert are unchanged). A team + member signs out of Slack and back in via SSO. If it fails, SSO is + "optional" so nobody is locked out; fix the URL rather than rolling back + the gateway. + +5. **0:20 — hot-reload target.** In the data repo, switch the prod URL in + `.github/workflows/notify-deployments.yml` from + `https://next.codeforphilly.org/...` to + `https://codeforphilly.org/api/_internal/reload-data`. (The alias would + keep working, but the workflow should name the canonical host.) + +6. **0:25 — legacy site.** If you put up a maintenance banner at T-7, it + can stay; laddr is now reachable only at + `codeforphilly.live.k8s.phl.io` and can remain read-only indefinitely. **Point of no return:** the moment **a real user creates a new record** -on the rewrite (a project, an update, a buzz post, even just their first -sign-in writing a `PrivateProfile`), data starts to diverge between the -two systems. Up to that point, a rollback (DNS flip back, see -[cutover-rollback.md](cutover-rollback.md)) is harmless. After it, a -rollback would discard those new records. +on the rewrite at the apex (a project, an update, a buzz post, even just +their first sign-in writing a `PrivateProfile`), data starts to diverge +between the two systems. Up to that point, reverting the cutover commit +([cutover-rollback.md](cutover-rollback.md)) is harmless. After it, a +rollback would discard those new records. Note that people have been able +to sign in at `next.codeforphilly.org` since before cutover — the rewrite's +data is already live and diverging from laddr's; "point of no return" here +is about writes that arrive *because* the apex moved. ## T+1h: monitoring window -For the first hour after DNS flip, the cutover lead and engineering second -watch: +**Be honest about what exists:** as of this writing there are no external +uptime checks and no alerts webhook ([monitoring.md](monitoring.md) +describes the target state, not the current one). For the first hour after +the hostname move, the cutover lead and engineering second watch by hand: -- `kubectl -n codeforphilly logs deploy/codeforphilly -f` — error rates - (look for spikes in `ERROR` or `WARN`). -- `/api/health` and `/api/health/ready` via UptimeRobot / healthchecks.io. -- The cluster log aggregator (if wired up) — search for stack traces. -- The `#alerts` Slack channel — anything fired by the WARN+ webhook. +- `kubectl -n codeforphilly-ng logs deploy/codeforphilly -f` — look for + spikes in `ERROR` or `WARN`. +- `curl -sS https://codeforphilly.org/api/health` and + `.../api/health/ready` every few minutes (a shell loop is fine). +- `kubectl -n codeforphilly-ng get pods -w` — restarts or OOMKills. Specific things to watch: - **OAuth callback failure rate.** Should be ~0%. A 4xx spike here - indicates a misconfigured redirect URI or a GitHub OAuth app issue. -- **SAML assertion failure rate.** Should be ~0%. A spike here means - the Slack integration is rejecting our assertions — usually a NameID - mismatch. -- **Push daemon errors.** Surface as `git push failed` in logs; means - the data repo's deploy key is wrong or rate-limited. + indicates a GitHub OAuth app issue (the callback URL itself should not + need changing — see T-7). +- **SAML assertion failure rate.** Should be ~0% once the SSO URL is + updated. A spike means Slack is rejecting our assertions — usually a + NameID or SSO URL mismatch. Members still have magic-link fallback. +- **Push daemon errors.** Surface as `git push failed` in logs; means the + data repo's deploy key is wrong or rate-limited. +- **Hot-reload webhook.** The next `published` push should log a + `hot-reload` line on the prod pod via the apex URL. If any of these alarm: pause, triage, decide rollback vs forward-fix in under 15 minutes. @@ -224,8 +361,8 @@ If everything is stable through the first day: 1. Post a success message to `#announcements` from [cutover-announcement.md](cutover-announcement.md). -2. Add a note to the rewrite README that legacy laddr is decommissioned. -3. Schedule the T+90 mailout to run automatically (see below). +2. Add a note to the rewrite README that legacy laddr is decommissioned + (or read-only at `codeforphilly.live.k8s.phl.io`). ## T+7 days: reconciliation check @@ -237,74 +374,64 @@ npm run -w apps/api script:reconcile After a week of real-world use, any orphans or inconsistencies that show up are likely from a real bug (dual-write coordination, racing OAuth -callbacks). File issues; fix before T+90. - -Schedule reconciliation weekly via a CI cron from this point on. Run the -script against production via `kubectl exec` into the deployed pod, or -configure a separate cron Deployment. - -## T+90 days: unclaimed-account mailout - -The active claim window from [account-migration.md](../../specs/behaviors/account-migration.md#cutover-window-policy) -ends at T+90. Send the reminder mail to anyone whose laddr account is -still unclaimed: - -1. **Dry-run first** to inspect the recipient list: - - ```bash - npm run -w apps/api script:cutover-mailout -- --dry-run \ - --json=/scratch/mailout-dry.json - ``` - -2. Spot-check the recipient list — sample 5 entries, confirm each - matches an unclaimed legacy Person. -3. Send: - - ```bash - POSTMARK_SERVER_TOKEN=... npm run -w apps/api script:cutover-mailout -- --send - ``` - -4. Monitor the Postmark activity stream for bounces. Hard bounces are expected — - defunct email providers are exactly why these accounts are - unclaimed. - -## T+180 days: drain legacy passwords - -At T+180 the remaining `LegacyPasswordCredential` records are deleted. -This is irreversible — any user who hadn't claimed by then must go through -the staff-approval path from [account-migration.md](../../specs/behaviors/account-migration.md#c-staff-approval). - -Run reconciliation with the legacy-purge follow-up (TBD: a small script -that calls `privateStore.deleteLegacyPassword` for every credential whose -Person is still unclaimed and whose `importedAt` is older than 180 days). +callbacks). File issues. + +**Open gap:** the reconcile script needs both the public data and the +private store. The runtime image ships only `dist/` (no `tsx`, no +`scripts/`), so it cannot run inside the prod pod as-is, and production +private data must not be copied to a laptop +([private-storage.md](../../specs/behaviors/private-storage.md)). Until the +script ships in the image or gets an in-pod entrypoint, reconciliation in +production is a manual, in-cluster exercise; treat this as a follow-up. + +## Legacy password sign-in stays on + +Nothing in this runbook retires legacy credentials. Migrated members keep +signing in with their laddr password for as long as they like; the only +nudge is the "Connect GitHub" banner on `/account` +([account-migration.md → The nag](../../specs/behaviors/account-migration.md#the-nag-banner-on-account)). +The spec tracks a coverage metric for *future* sunset planning; when the +numbers justify it, a separate spec change sets a date. Do not schedule a +mailout or a purge from this doc. ## Pre-cutover checklist Before T-0 the cutover lead confirms each of these in writing in the cutover Slack thread: -- [ ] DNS TTL lowered to 60s at least 7 days ago -- [ ] Sealed-secrets present in the production cluster (every var in - [secrets.md inventory](secrets.md)) -- [ ] GitHub OAuth app production redirect URI registered -- [ ] Slack SAML metadata pre-staged with workspace admin -- [ ] Data-repo deploy key uploaded to the data-repo's deploy-keys page -- [ ] Production S3 bucket exists with versioning enabled -- [ ] Staging dry-run from T-3 reports all stages passing +- [ ] SealedSecrets present and decrypted in `codeforphilly-ng` + (`codeforphilly-secrets`, `codeforphilly-saml`, + `codeforphilly-data-deploy-key`) — see [secrets.md](secrets.md) +- [ ] GitHub OAuth app "Code for Philly" exists; client ID matches the sealed value +- [ ] Slack SSO set to **optional**; "Test configuration" passed against + the `next.codeforphilly.org` metadata +- [ ] Data-repo deploy key has write access to `codeforphilly-data` +- [ ] `notify-deployments.yml` run green for both sandbox and prod on the last `published` push +- [ ] Credentials loaded on the prod PVC; a known legacy user can sign in + with their password at `next.codeforphilly.org` +- [ ] Sandbox dry-run from T-3 reports all stages passing +- [ ] Production pin (`.holo/sources/codeforphilly-ng.toml` + + `codeforphilly-ng/app/kustomization.yaml`) is the intended release +- [ ] The cutover commit is drafted on a branch in `cfp-live-cluster` and + reviewed (listeners, HTTPRoute hostnames, `CFP_SITE_HOST`) - [ ] On-call schedule covers T-0 through T+24h -- [ ] Rollback procedure rehearsed (see [cutover-rollback.md](cutover-rollback.md)) +- [ ] Rollback procedure read end-to-end (see [cutover-rollback.md](cutover-rollback.md)) +- [ ] **Monitoring: none wired.** Acknowledged that the T+1h window is + watched by hand; external checks + alerts are a post-cutover follow-up + ([monitoring.md](monitoring.md)) ## Known unknowns -- **Single big import commit size.** The data repo will land thousands - of `.toml` files in one push. Confirm with the gitsheets transaction - API that this works at scale; chunk by entity type (one commit per - sheet) if not. The importer already commits per-sheet. +- **Certificate issuance on the moved listeners.** The apex and `www` TLS + Secrets are namespaced, so cert-manager mints new ones in + `codeforphilly-ng` after the move. Let's Encrypt rate limits are + generous for two names, but a wedged ACME challenge would leave the apex + serving a bad cert — that is a revert trigger. - **GitHub OAuth rate limits.** A surge of users signing in within a cutover window could brush against GitHub's per-app rate limit - (~5000/hr). Probably fine at CFP scale (~1240 members); worth - monitoring. + (~5000/hr). Probably fine at CFP scale; worth monitoring. - **Newsletter email bounces.** First OAuth sign-in refreshes `PrivateProfile.email` to GitHub primary. Users who never sign in keep their pre-cutover email and may bounce on newsletter sends. Acceptable; cleaned up by standard bounce handling. +- **Reconcile in production** — see the open gap under T+7 days. From e2636b6681af511f02838a493b2d03f289f42182 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 10 Sep 2026 02:06:09 -0400 Subject: [PATCH 3/8] docs(ops): align deploy, secrets and runbook docs with live-cluster GitOps Image publishing is no longer manual and production is not a future `cfp-prod-cluster`: container-publish.yml pushes :vX.Y.Z on every release tag, and each cluster repo (cfp-sandbox-cluster, cfp-live-cluster) pins the app by .holo/sources/codeforphilly-ng.toml plus images[].newTag in codeforphilly-ng/app/kustomization.yaml, applied through the releases/k8s-manifests -> deploys/k8s-manifests PR. The hand-built :sandbox image is kept only as an emergency escape hatch. Private storage is a filesystem PVC in both environments; the S3 sections are marked supported-but-unused rather than "production". Secrets are the three SealedSecrets in cfp-live-cluster/codeforphilly-ng.secrets/ (codeforphilly-secrets, codeforphilly-saml carried over from laddr's saml2, the deploy key), edited with kubeseal --merge-into. Namespaces, the hot-reload secret name, the pod-clone volume type and the monitoring status note are corrected to match. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR --- .claude/CLAUDE.md | 29 ++-- docs/operations/deploy.md | 222 ++++++++++++++++---------- docs/operations/monitoring.md | 38 +++-- docs/operations/releases.md | 14 +- docs/operations/runbook.md | 142 ++++++++++------- docs/operations/sandbox-deploy.md | 163 ++++++++++++------- docs/operations/secrets.md | 252 +++++++++++++++++++----------- docs/operations/spam-detection.md | 10 +- 8 files changed, 555 insertions(+), 315 deletions(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 5d49f83..8a5627b 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -7,7 +7,7 @@ A modernization of [laddr](https://github.com/CodeForPhilly/laddr) (the platform These auto-trigger by topic — you don't load them manually. Mentioned here so you know what's already covered and don't duplicate. | Skill | Triggers on | What it covers | -|---|---|---| +| --- | --- | --- | | [`specops`](./skills/specops/SKILL.md) | `specs/`, `plans/`, "spec", "closeout commit", new features | Spec-driven workflow (specs are source of truth), plans-as-micro-DAG protocol, closeout commit ritual, follow-ups taxonomy, spec-drift auditor | | [`backend-fastify`](./skills/backend-fastify/SKILL.md) | New routes, services, plugins, env vars | Fastify 5 patterns, plugin ordering, `@fastify/env` validation, error handling | | [`frontend-shadcn`](./skills/frontend-shadcn/SKILL.md) | New screens, components, routing, styling | Vite + React 19 + shadcn/ui + Tailwind v4 + React Router v7 patterns | @@ -64,19 +64,23 @@ codeforphilly-data ─── Public data store. Branches: │ merge target). Hot-reload webhook │ fires on push. │ -cfp-sandbox-cluster ─── GitOps repo (hologit-projected) that pulls - │ this repo's `deploy/kustomize/` upstream - │ and applies it via Kustomize. See +cfp-sandbox-cluster ─── GitOps repos (hologit-projected) that pin a +cfp-live-cluster │ release tag of this repo, project its + │ `deploy/kustomize/base/` and apply it via + │ Kustomize. Sandbox = next-v2.codeforphilly.org + │ (ns codeforphilly-rewrite-sandbox); prod = + │ next.codeforphilly.org → codeforphilly.org at + │ cutover (ns codeforphilly-ng). See │ `docs/operations/deploy.md`. ``` -Operator docs in [`docs/operations/`](../docs/operations/): `deploy.md` for the cluster topology, `sandbox-deploy.md` for the manual procedure, `runbook.md` for incident response (including the hot-reload webhook). +Operator docs in [`docs/operations/`](../docs/operations/): `deploy.md` for the cluster topology + GitOps flow, `releases.md` for cutting a release, `sandbox-deploy.md` for the sandbox specifics, `runbook.md` for incident response (including the hot-reload webhook), `cutover.md` for the production hostname move. ## Stack - **Backend** — Fastify 5.x + TypeScript. Single replica, in-process write mutex. - **Public storage** — [gitsheets](https://github.com/JarvusInnovations/gitsheets) (TOML records in a git repo). Public-by-design — civic transparency. No persistent OLTP. See [specs/behaviors/storage.md](../specs/behaviors/storage.md). -- **Private storage** — S3-compatible bucket holding `.jsonl` files (private profiles + legacy password hashes). Boot-load + in-memory; PUT on mutation. See [specs/behaviors/private-storage.md](../specs/behaviors/private-storage.md). Real production private data never lands on a dev machine. +- **Private storage** — `.jsonl` files (private profiles + legacy password hashes) on a filesystem PVC in every deployed environment (`STORAGE_BACKEND=filesystem`); an S3-compatible backend exists but is unused. Boot-load + in-memory; rewrite on mutation. See [specs/behaviors/private-storage.md](../specs/behaviors/private-storage.md). Real production private data never lands on a dev machine. - **Schemas** — Zod in `packages/shared`, consumed by both web and api, validating records in both stores. - **Full-text search** — in-memory SQLite FTS5 (or MiniSearch fallback), rebuilt at boot from gitsheets state. - **Auth** — GitHub OAuth as the sole primary identity provider; stateless JWT sessions. We're also the SAML IdP for codeforphilly.slack.com. See [specs/api/auth.md](../specs/api/auth.md), [specs/api/saml.md](../specs/api/saml.md). @@ -135,13 +139,13 @@ npm run -w apps/web dev # web only ## Deploying -GitOps. This repo publishes a Docker image to GHCR; the [`cfp-sandbox-cluster`](https://github.com/CodeForPhilly/cfp-sandbox-cluster) repo pulls our [`deploy/kustomize/`](../deploy/kustomize/) upstream via hologit and applies it with Kustomize. Production will follow the same pattern under a `cfp-prod-cluster` repo. +GitOps. Releases publish a Docker image to GHCR; two cluster repos — [`cfp-sandbox-cluster`](https://github.com/CodeForPhilly/cfp-sandbox-cluster) (sandbox) and [`cfp-live-cluster`](https://github.com/CodeForPhilly/cfp-live-cluster) (production, alongside legacy laddr) — each pin a release tag, project our [`deploy/kustomize/base/`](../deploy/kustomize/base/) via hologit and apply it with Kustomize. Typical change flow: -1. **Merge to `main`** — CI builds + tests; nothing deploys yet. -2. **Publish image** (currently manual) — `docker build --platform=linux/amd64 -t ghcr.io/codeforphilly/codeforphilly-ng:sandbox . && docker push …`. Apple-silicon dev machines must set the platform flag — cluster nodes are amd64. -3. **GitOps pickup** — `cfp-sandbox-cluster` projects from our `deploy/kustomize/`; on its own merge, applies via `kubectl apply -k`. +1. **Merge to `develop`** — CI builds + tests; pushing `develop` opens/updates a `Release: vX.Y.Z` PR into `main` (see `docs/operations/releases.md` and the `release-flow` skill). +2. **Publish image** — merging the Release PR tags `vX.Y.Z`; `container-publish.yml` builds and pushes `ghcr.io/codeforphilly/codeforphilly-ng:vX.Y.Z` (+ `:latest`). No manual `docker push`. +3. **GitOps pickup** — in the cluster repo, bump `.holo/sources/codeforphilly-ng.toml` (`ref = "refs/tags/vX.Y.Z"`) and `images[].newTag` in `codeforphilly-ng/app/kustomization.yaml` together; merge to `main`; "Build k8s-manifests" projects to `releases/k8s-manifests`; a bot PR into `deploys/k8s-manifests` applies on merge. Sandbox first, then live. 4. **Pod boot** — single replica, `Recreate` strategy. Container entrypoint bare-clones the data repo on every pod start (the data volume is `emptyDir`, so first boot = every fresh pod). Node boots: env → store load → **reconcile** (ff/replay/escape-hatch against `origin/`) → **push daemon** → routes → SPA. `/api/health/ready` returns 200 once stores are loaded *and* reconciled. 5. **Live data updates** — independent of app deploy. Pushes to `published` trigger the [hot-reload webhook](../docs/operations/runbook.md#hot-reload-webhook); the pod rebuilds in-memory state in place, no restart. @@ -149,9 +153,10 @@ Constraints worth knowing before touching anything deploy-shaped: - **Single replica is non-negotiable** — writes flow through an in-process mutex; horizontal scale would lose write serialization. - **One image for API + SPA** — Fastify serves the built `apps/web/dist` as fallthrough. No separate web container. -- **Sealed secrets are cluster-bound** — sandbox-sealed secrets can't decrypt in prod. Plain values live in `.env` locally; cluster values live in the sandbox-cluster repo's `codeforphilly-ng.secrets/`. +- **Sealed secrets are cluster-bound** — sandbox-sealed secrets can't decrypt in prod. Plain values live in `.env` locally; cluster values live in each cluster repo's `codeforphilly-ng.secrets/`. +- **Cutover is a hostname move, not a DNS change** — the apex already resolves to the live cluster's gateway; T-0 is one commit in `cfp-live-cluster` moving listeners from `_gateways/code-for-philly.yaml` to `_gateways/codeforphilly-ng.yaml`. -Full operator docs: [`docs/operations/`](../docs/operations/) — `deploy.md` (boot sequence + env table), `sandbox-deploy.md` (manual bring-up), `secrets.md` (the secret contract), `runbook.md` (incident response + hot-reload webhook), `cutover.md` (production switch plan). +Full operator docs: [`docs/operations/`](../docs/operations/) — `deploy.md` (environments, GitOps flow, boot sequence, env table), `releases.md` (cutting a release), `sandbox-deploy.md` (sandbox specifics + manual escape hatch), `secrets.md` (the secret contract), `runbook.md` (incident response + hot-reload webhook), `cutover.md` (production hostname move), `legacy-credentials-import.md` (private-store seeding). ## Source control diff --git a/docs/operations/deploy.md b/docs/operations/deploy.md index 4945c72..3f78fb7 100644 --- a/docs/operations/deploy.md +++ b/docs/operations/deploy.md @@ -5,22 +5,36 @@ container. The authoritative architectural contract is [specs/architecture.md](../../specs/architecture.md#deploy); this document is the runbook that implements it. -> See also: [sandbox-deploy.md](sandbox-deploy.md) for the manual sandbox -> bring-up procedure, [secrets.md](secrets.md) for the secret contract, -> [runbook.md](runbook.md) for incident response. +> See also: [releases.md](releases.md) for how an image gets published, +> [sandbox-deploy.md](sandbox-deploy.md) for the sandbox cluster specifics +> and the manual escape hatch, [secrets.md](secrets.md) for the secret +> contract, [runbook.md](runbook.md) for incident response, +> [cutover.md](cutover.md) for the production hostname move. + +## Environments + +| | GitOps repo | Namespace | Host | Pinned at | +|---|---|---|---|---| +| **Production** | [`cfp-live-cluster`](https://github.com/CodeForPhilly/cfp-live-cluster) | `codeforphilly-ng` | `next.codeforphilly.org` (pre-cutover); `codeforphilly.org` after | `v0.2.0` as of 2026-09-10 | +| **Sandbox** | [`cfp-sandbox-cluster`](https://github.com/CodeForPhilly/cfp-sandbox-cluster) | `codeforphilly-rewrite-sandbox` | `next-v2.codeforphilly.org` | same tag, bumped ahead of prod | + +Legacy laddr runs in the live cluster too, in namespace `code-for-philly`. +There is no separate staging environment — the sandbox plays that role. ## TL;DR — anatomy ``` +----------------------+ -| GitHub Actions CI | ci.yml (build + test on PR / main) +| GitHub Actions | ci.yml (build + test on PR / develop) +| | release-*.yml (develop -> "Release: vX.Y.Z" PR -> tag) +| | container-publish.yml (on tag push) +----------+-----------+ - | docker build / push (manual today) + | docker build / push v +----------------------+ -| GHCR image | ghcr.io/codeforphilly/codeforphilly-ng: +| GHCR image | ghcr.io/codeforphilly/codeforphilly-ng:vX.Y.Z (+ :latest) +----------+-----------+ - | kubectl apply -k (via GitOps below) + | cluster repo bumps its pin -> "Build k8s-manifests" -> deploy PR -> apply v +----------------------+ | k8s Deployment | 1 replica, Recreate strategy, PVC + Secrets + ConfigMap @@ -37,36 +51,60 @@ The image holds **both** the API and the built SPA. There is no separate web container. The single replica is a hard architectural constraint ([specs/architecture.md](../../specs/architecture.md#process-model)). -## Manifests +## Release → image -Kustomize base + per-environment overlays at -[`deploy/kustomize/`](../../deploy/kustomize/). The base lives in -`deploy/kustomize/base/`; environment overlays under -`deploy/kustomize/overlays//`. +Image publishing is automated; nobody `docker push`es a release. The flow is +in [releases.md](releases.md): -The base ships everything the cluster needs in any environment: -`Deployment`, `Service`, `ConfigMap`, `PersistentVolumeClaim`s, `Gateway` + -`HTTPRoute` (per-env hostname patched in the overlay), `ServiceAccount`. -Sealed `Secret`s live only in overlays (sealed against the target cluster's -sealed-secrets controller). +1. Work merges into `develop`; pushing `develop` opens/updates a + `Release: vX.Y.Z` PR into `main`. +2. Merging that PR tags `vX.Y.Z`; the tag push triggers + `.github/workflows/container-publish.yml`, which builds and pushes + `ghcr.io/codeforphilly/codeforphilly-ng:vX.Y.Z` and `:latest`. -Cluster-level deploys are driven by the -[`cfp-sandbox-cluster`](https://github.com/CodeForPhilly/cfp-sandbox-cluster) -GitOps repo, which pulls the workload from this repo's main branch, composes -its own per-cluster Gateway/HTTPRoute (under `_gateways/codeforphilly-ng.yaml`) -and SealedSecrets (under `codeforphilly-ng.secrets/`), and applies on merge. -Production stand-up will follow the same pattern under a `cfp-prod-cluster` -repo. +The old hand-built `:sandbox` tag still exists as an emergency escape hatch +([sandbox-deploy.md](sandbox-deploy.md#emergency-escape-hatch-manual-image--apply)) +but is not how anything gets deployed. -For a one-shot manual apply (useful pre-GitOps or for an offline cluster): +## Image → cluster (GitOps) -```bash -kubectl apply -k deploy/kustomize/overlays/sandbox -``` +Both cluster repos work the same way: + +1. **Pin the release.** Two values in the cluster repo must move together: + - `.holo/sources/codeforphilly-ng.toml` — `ref = "refs/tags/vX.Y.Z"`. + hologit projects `deploy/kustomize/base/` from this repo at that tag. + - `codeforphilly-ng/app/kustomization.yaml` — `images[].newTag: vX.Y.Z`. +2. **Merge to `main`** in the cluster repo. The **"Build k8s-manifests"** + workflow projects the holobranch to `releases/k8s-manifests`, and a bot + opens a PR from there into `deploys/k8s-manifests`. +3. **Merge the deploy PR.** That applies the manifests. The `Recreate` + strategy swaps the pod; expect a short readiness gap while the new pod + re-clones the data repo and reloads state. + +Bump the sandbox first, let it soak, then bump prod. + +Per-cluster concerns live in the cluster repo, not here: the Gateway + +HTTPRoute (`_gateways/codeforphilly-ng.yaml`), the SealedSecrets +(`codeforphilly-ng.secrets/`), and the per-environment patches in +`codeforphilly-ng/app/kustomization.yaml` (`CFP_SITE_HOST`, +`imagePullPolicy`, the extra `codeforphilly-saml` `envFrom` in prod). + +## Manifests in this repo + +Kustomize base at [`deploy/kustomize/base/`](../../deploy/kustomize/base/): +`Deployment`, `Service`, `ConfigMap`, the private-storage +`PersistentVolumeClaim`, `Gateway` + `HTTPRoute` templates (hostname +placeholder), `ServiceAccount`. This is what the cluster repos project. + +[`deploy/kustomize/overlays/sandbox/`](../../deploy/kustomize/overlays/sandbox/) +is the pre-GitOps manual overlay (namespace, sealed secrets, hostname +patches). It is only used by the escape hatch in +[sandbox-deploy.md](sandbox-deploy.md); the sandbox cluster repo carries its +own equivalents. ## Image -### Build +### Build (local) ```bash docker build --platform=linux/amd64 \ @@ -77,9 +115,13 @@ Three stages — `deps` (full install), `build` (compile shared, api, web — in that order, since web/api consume shared's compiled output), `runtime` (alpine + git + ca-certificates + tini). Final image runs as `node` (uid 1000) per the `securityContext` in `deploy/kustomize/base/deployment.yaml`. +The runtime stage ships only `dist/` output and production deps — no +`tsx`, no `apps/api/scripts/` — so operator scripts can't be run inside a +pod. `--platform=linux/amd64` is required on Apple Silicon hosts — the cluster -nodes are amd64 and won't pull an arm64-only manifest. +nodes are amd64 and won't pull an arm64-only manifest. CI runners are amd64 +already, so `container-publish.yml` has no platform flag. ### Run (local smoke test) @@ -154,6 +196,9 @@ object DB via plumbing (`update-ref`, `merge-tree --write-tree`, When `CFP_DATA_REMOTE` is unset (typical local dev), the reconcile plugin skips reconciliation entirely. +The same path runs mid-life when the hot-reload webhook fires — see +[runbook.md#hot-reload-webhook](runbook.md#hot-reload-webhook). + ## Probes - **Liveness** — `GET /api/health` every 30s. The pod is killed only after @@ -182,33 +227,35 @@ Implications: default), the entrypoint relies on `GIT_SSH_COMMAND` (set in the ConfigMap) pointing at the mounted private key. Rotation: replace the SealedSecret, restart the pod. See - [secrets.md](secrets.md#data-repo-deploy-key) and the rotation - procedure in [sandbox-deploy.md](sandbox-deploy.md#rotating-the-deploy-key). - -## Bucket provisioning (production) - -Production uses an S3-compatible bucket for private storage -([specs/behaviors/private-storage.md](../../specs/behaviors/private-storage.md)). -The bucket is provisioned out-of-band and the manifests consume its -credentials via a SealedSecret. - -Recommended provider: **Cloudflare R2** (zero egress, pennies per month, -S3-compatible API). Backblaze B2 or AWS S3 also work. MinIO inside the -cluster is acceptable for cost reasons but trades operational simplicity -for storage simplicity. - -Required bucket configuration: - -- **Versioning enabled.** Hard requirement per - [private-storage.md](../../specs/behaviors/private-storage.md#bucket-requirements). -- **Lifecycle rule** deleting non-current versions after 365 days. -- **IAM policy** scoped to the bucket only — `s3:GetObject`, `s3:PutObject`, - `s3:ListBucket`, `s3:GetObjectVersion`. No cross-bucket access; no console - access for the service principal. -- **Endpoint URL** → `S3_ENDPOINT` (ConfigMap). -- **Bucket name** → `S3_BUCKET`. -- **Region** → `S3_REGION`. -- **Access keys** → `S3_ACCESS_KEY_ID` + `S3_SECRET_ACCESS_KEY` (Secret). + [secrets.md](secrets.md#data-repo-deploy-key). + +## Private storage + +Both sandbox and production run `STORAGE_BACKEND=filesystem` with +`CFP_PRIVATE_STORAGE_PATH=/app/private-storage` mounted from the +`codeforphilly-private` PersistentVolumeClaim +(`deploy/kustomize/base/pvc-private.yaml`, `linode-block-storage-retain`, +1Gi). The two `.jsonl` files there (`profiles.jsonl`, +`legacy-passwords.jsonl`) are loaded into memory at boot and rewritten +atomically on every private-side mutation. + +Implications: + +- **The PVC is the private store.** It survives pod restarts and image + bumps. The hot-reload webhook does **not** reload it — a pod restart is + the only way to pick up files placed there by hand + ([legacy-credentials-import.md](legacy-credentials-import.md)). +- **No versioning.** Unlike an object store there is no automatic history + of previous `.jsonl` versions. Snapshot the files (`kubectl cp` out of + the pod, kept in-cluster or on an operator machine only as long as + needed) before any manual operation that rewrites them. +- **Backups** are whatever the storage class's retain policy gives you. + There is no scheduled backup of the PVC today. + +The `s3` backend ([specs/behaviors/private-storage.md](../../specs/behaviors/private-storage.md#backends)) +remains fully supported and is the path if the private store ever needs to +outlive a single cluster, but **no environment uses it today**. The +`S3_*` variables below are documented for completeness only. ## Environment variables (reference) @@ -219,38 +266,47 @@ comments. Production pod gets these mounted: |----------|--------|-------| | `NODE_ENV` | ConfigMap | `production` | | `PORT` | ConfigMap | `3001` | -| `HOST` | ConfigMap | `0.0.0.0` | +| `NODE_OPTIONS` | ConfigMap | `--max-old-space-size=512` — see the comment in `configmap.yaml` for how that was sized | | `CFP_DATA_REPO_PATH` | ConfigMap | `/app/data` — bare gitdir, backed by an `emptyDir`; re-cloned on every pod boot | -| `CFP_DATA_REMOTE` | Secret | git URL (ssh in prod) | -| `CFP_DATA_BRANCH` | ConfigMap | e.g. `fixture` / `main` | -| `CFP_DATA_RELOAD_SECRET` | **Secret** | Shared bearer-token for the hot-reload webhook; when unset the `/api/_internal/reload-data` endpoint returns 503. See [runbook.md](runbook.md#hot-reload-webhook). | +| `CFP_DATA_REMOTE` | **Secret** (`codeforphilly-secrets`) | git URL (ssh in sandbox + prod) | +| `CFP_DATA_BRANCH` | ConfigMap | `published` in sandbox + prod (the data repo has no `main`). `fixture` / `legacy-import` for debug pods | +| `CFP_DATA_RELOAD_SECRET` | **Secret** (`codeforphilly-secrets`) | Shared bearer token for the hot-reload webhook; when unset the `/api/_internal/reload-data` endpoint returns 503. Must match the `CFP_DATA_RELOAD_SECRET` Actions secret in the data repo. See [runbook.md](runbook.md#hot-reload-webhook). | | `CFP_WEB_DIST_PATH` | ConfigMap | `/app/apps/web/dist` | -| `CFP_SITE_HOST` | ConfigMap | Public-facing host (`codeforphilly.org` base, `next-v2.codeforphilly.org` sandbox). Drives the markdown renderer's external-link transform — anchors with a different host get `target="_blank" rel="noopener nofollow"` — and the SAML IdP metadata's `SingleSignOnService` endpoint URLs. | -| `SAML_ENTITY_ID` | ConfigMap | Optional. Stable SAML IdP entity ID / assertion `Issuer` (default `https://codeforphilly.org/api/saml/slack/metadata`). Leave unset everywhere Slack should keep trusting the production IdP identity — it deliberately does **not** follow `CFP_SITE_HOST`, so flipping the host at cutover doesn't require editing Slack's SAML config. Only set it when standing up a separate IdP registration (e.g. a sandbox pointed at a test workspace). See [specs/api/saml.md](../../specs/api/saml.md#idp-identity-and-hosts). | -| `POSTMARK_SERVER_TOKEN` | **Secret** | Postmark server API token for outbound notifications. When unset, the email notifier falls back to a no-op LoggingNotifier — convenient for dev + tests but means no real emails go out. | +| `CFP_SITE_HOST` | ConfigMap (patched per cluster) | Public-facing host: `next.codeforphilly.org` in prod until cutover, `codeforphilly.org` after; `next-v2.codeforphilly.org` in sandbox. Drives the markdown renderer's external-link transform, canonical URLs in email, and the SAML IdP metadata's `SingleSignOnService` endpoint URLs. | +| `SAML_ENTITY_ID` | ConfigMap | Optional. Stable SAML IdP entity ID / assertion `Issuer` (default `https://codeforphilly.org/api/saml/slack/metadata`). Leave unset everywhere Slack should keep trusting the production IdP identity — it deliberately does **not** follow `CFP_SITE_HOST`, so flipping the host at cutover doesn't require re-registering with Slack. Only set it when standing up a separate IdP registration. See [specs/api/saml.md](../../specs/api/saml.md#idp-identity-and-hosts). | +| `POSTMARK_SERVER_TOKEN` | **Secret** (`codeforphilly-secrets`) | Postmark server API token for outbound notifications. When unset, the email notifier falls back to a no-op LoggingNotifier — convenient for dev + tests but means no real emails go out. | | `POSTMARK_MESSAGE_STREAM` | ConfigMap | Postmark message stream for outbound mail (default `outbound`). Must exist on the server the token belongs to. | -| `CFP_NOTIFICATION_FROM` | ConfigMap | RFC 5322 sender address for outbound notifications (default `"Code for Philly "`). Sender domain must be a verified Postmark sender signature (already true for `codeforphilly.org` via the legacy site) before flipping `POSTMARK_SERVER_TOKEN` on. | -| `STORAGE_BACKEND` | ConfigMap | `s3` (prod) / `filesystem` (sandbox) | -| `CFP_PRIVATE_STORAGE_PATH` | ConfigMap | `/app/private-storage` (when filesystem) | -| `S3_ENDPOINT` / `S3_BUCKET` / `S3_REGION` | ConfigMap | Bucket addressing | -| `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` | **Secret** | Bucket credentials | -| `GITHUB_OAUTH_CLIENT_ID` | **Secret** | OAuth app client ID | -| `GITHUB_OAUTH_CLIENT_SECRET` | **Secret** | OAuth app client secret | -| `CFP_JWT_SIGNING_KEY` | **Secret** | HS256 key (`openssl rand -base64 64`) | -| `SAML_PRIVATE_KEY` / `SAML_CERTIFICATE` | **Secret** | Slack IdP cert chain | +| `CFP_NOTIFICATION_FROM` | ConfigMap | RFC 5322 sender address for outbound notifications (default `"Code for Philly "`). Sender domain must be a verified Postmark sender signature (already true for `codeforphilly.org` via the legacy site). | +| `STORAGE_BACKEND` | ConfigMap | `filesystem` in sandbox **and** prod. `s3` is supported but unused. | +| `CFP_PRIVATE_STORAGE_PATH` | ConfigMap | `/app/private-storage` (the PVC mount) | +| `S3_ENDPOINT` / `S3_BUCKET` / `S3_REGION` | — | Only when `STORAGE_BACKEND=s3`. Not set anywhere today. | +| `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` | — | Only when `STORAGE_BACKEND=s3`. Not set anywhere today. | +| `GITHUB_OAUTH_CLIENT_ID` | **Secret** (`codeforphilly-secrets`) | OAuth app client ID (not sensitive, but it ships alongside the secret for one-place rotation) | +| `GITHUB_OAUTH_CLIENT_SECRET` | **Secret** (`codeforphilly-secrets`) | OAuth app client secret | +| `CFP_JWT_SIGNING_KEY` | **Secret** (`codeforphilly-secrets`) | HS256 key (`openssl rand -base64 64`) | +| `SAML_PRIVATE_KEY` / `SAML_CERTIFICATE` | **Secret** (`codeforphilly-saml`, prod) | Slack IdP key pair, carried over verbatim from laddr's `saml2` secret so Slack keeps trusting the same cert. Mounted via an extra `envFrom` patch in the live cluster repo; marked `optional` so a sandbox without it still boots. | | `SLACK_TEAM_HOST` | ConfigMap | Slack workspace host (default `codeforphilly.slack.com`). ACS URL, NameID `NameQualifier`, `/chat` + `/launch` redirect target. Never our own IdP identity. | | `GIT_SSH_COMMAND` | ConfigMap | Wires `ssh` to the mounted deploy key | +| `GIT_AUTHOR_NAME` / `GIT_AUTHOR_EMAIL` | ConfigMap | Committer identity for API-authored data commits | ## Rollback -Two distinct rollback flavors: - -- **Pod / image rollback** — change the image tag in the GitOps repo's - `images:` override (or, for an out-of-band hotfix, `kubectl set image - deployment/codeforphilly ...`). The deployment's `Recreate` strategy - serializes the swap; a few seconds of `503` on the readiness probe is - expected while the new pod boots. +Three distinct rollback flavors: + +- **Image / version rollback** — move the two pins in the cluster repo + (`.holo/sources/codeforphilly-ng.toml` `ref` and + `codeforphilly-ng/app/kustomization.yaml` `newTag`) back to the previous + release tag and run it through the deploy PR. For an out-of-band hotfix, + `kubectl -n set image deploy/codeforphilly + codeforphilly=ghcr.io/codeforphilly/codeforphilly-ng:` + and reconcile the repo afterwards. The `Recreate` strategy serializes the + swap; a few seconds of `503` on the readiness probe is expected while the + new pod boots. +- **Hostname rollback** (cutover only) — `git revert` the cutover commit in + `cfp-live-cluster`. See [cutover-rollback.md](cutover-rollback.md). - **Data rollback** — `git revert` (or `git push --force-with-lease` after - a careful local rebase) on the data repo. The next pod-boot entrypoint - reconciliation will pick up the change. Don't conflate the two: rolling - the image back does not undo data writes the API has already pushed. + a careful local rebase) on the data repo's `published` branch. The push + fires the hot-reload webhook, so the running pod picks it up without a + restart. Don't conflate the flavors: rolling the image back does not undo + data writes the API has already pushed, and neither touches the private + store on the PVC. diff --git a/docs/operations/monitoring.md b/docs/operations/monitoring.md index ecb1479..48d1869 100644 --- a/docs/operations/monitoring.md +++ b/docs/operations/monitoring.md @@ -1,6 +1,15 @@ # Post-cutover monitoring -The minimum-viable monitoring wired up before cutover. The bar isn't +> **Status (2026-09-10): none of the external monitoring below exists yet.** +> There are no uptime checks, no log→Slack webhook, no `#alerts` integration. +> The only things that are real today are the Kubernetes liveness/readiness +> probes in `deploy/kustomize/base/deployment.yaml` and `kubectl logs`. The +> cutover runbook's T+1h window is watched by hand for that reason +> ([cutover.md → T+1h](cutover.md#t1h-monitoring-window)). Treat the rest of +> this document as the target to build toward, not a description of what's +> running. + +The minimum-viable monitoring we want in place. The bar isn't "observable in detail" — it's "we know within 60 seconds if the site is down, and within 5 minutes if an error rate is climbing." @@ -39,7 +48,7 @@ Configure two monitors: | Monitor | URL | Interval | Alert when | |---------|-----|----------|-----------| -| codeforphilly.org liveness | `https://codeforphilly.org/api/health` | 1 min | 2 consecutive failures (≈ 2 min) | +| codeforphilly.org liveness | `https://codeforphilly.org/api/health` (`https://next.codeforphilly.org/...` until cutover) | 1 min | 2 consecutive failures (≈ 2 min) | | codeforphilly.org readiness | `https://codeforphilly.org/api/health/ready` | 5 min | 1 failure (≈ 5 min) | Both alarm via the UptimeRobot → Slack integration to the `#alerts` channel. @@ -60,14 +69,15 @@ The two layers are complementary: - External monitors: "is the site reachable from the public internet?" A pod can pass k8s probes but the public hostname can still be unreachable -(ingress misconfigured, cert-manager wedged, DNS broken). The external +(gateway misconfigured, cert-manager wedged, DNS broken). The external monitor catches those. ## 3. Log aggregation Pino logs go to stdout from the API process. The cluster's log aggregator (whatever is configured — at minimum `kubectl logs` works; ideally a -shipper to a hosted log store like BetterStack or Grafana Loki) collects +shipper to a hosted log store like BetterStack or Grafana Loki — the live +cluster does run Grafana, so Loki is the natural first look) collects them. Required log levels in production: @@ -104,12 +114,12 @@ The fix is two-layer: 1. In the API's push job, emit `info` on success and `error` on failure. The Pino error gets webhooked to Slack like any other. -2. A daily check: "is the data repo's `origin/main` HEAD within 24h of the - API pod's local HEAD?" Tooling TBD; could be a small `kubectl exec` +2. A daily check: "is the data repo's `origin/published` HEAD within 24h of + the API pod's local HEAD?" Tooling TBD; could be a small `kubectl exec` cron or a server-side check exposed at `/api/health/push-daemon`. -For cutover-prep we ship the level-1 layer (push errors are already -loggable). The daily check is deferred to a post-cutover follow-up. +Layer 1 exists (push errors are logged). The daily check is deferred to a +post-cutover follow-up. ## What we are deliberately NOT monitoring at v1 @@ -126,20 +136,22 @@ loggable). The daily check is deferred to a post-cutover follow-up. These are deferred to when we have a specific need, not pre-emptively. -## Pre-cutover monitoring checklist +## Monitoring checklist -The cutover lead confirms before T-0: +None of these are done as of 2026-09-10. Cutover proceeds without them, +watched by hand; ticking them off is the first post-cutover ops task. - [ ] UptimeRobot account exists; two monitors above are configured - [ ] UptimeRobot → `#alerts` Slack integration is fired by a test alarm -- [ ] k8s liveness + readiness probes are present in `deploy/kustomize/base/deployment.yaml` +- [x] k8s liveness + readiness probes are present in `deploy/kustomize/base/deployment.yaml` - [ ] Log webhook → `#alerts` integration fires on a test `WARN` line -- [ ] On-call rotation is set in PagerDuty / Slack handoff doc +- [ ] On-call rotation is set in a Slack handoff doc - [ ] At least one team member can reach `#alerts` outside business hours The "test alarm" step is non-negotiable: untested alerts have a well-documented tendency to silently not fire when they're needed most. -Trigger each one once in staging and confirm a Slack message arrived. +Trigger each one once against the sandbox and confirm a Slack message +arrived. ## Post-cutover: when to add more diff --git a/docs/operations/releases.md b/docs/operations/releases.md index f005317..ec7860e 100644 --- a/docs/operations/releases.md +++ b/docs/operations/releases.md @@ -30,9 +30,12 @@ feature branch ──▶ develop ──(push)──▶ "Release: vX.Y.Z" PR into 4. **Merge the Release PR.** `release-publish.yml` creates the `vX.Y.Z` tag and GitHub release. The tag push triggers `container-publish.yml`, which builds and pushes `ghcr.io/codeforphilly/codeforphilly-ng:vX.Y.Z` and `:latest`. -5. **Deploy.** The cluster picks up the image per - [deploy.md](./deploy.md) (the published `:vX.Y.Z` / `:latest` tags replace - the previously-manual `:sandbox` build for versioned releases). +5. **Deploy.** In each cluster repo (`cfp-sandbox-cluster` first, then + `cfp-live-cluster`), bump `.holo/sources/codeforphilly-ng.toml`'s `ref` to + `refs/tags/vX.Y.Z` **and** `images[].newTag` in + `codeforphilly-ng/app/kustomization.yaml` in one commit, merge to `main`, + then merge the bot's PR into `deploys/k8s-manifests`. Details in + [deploy.md](./deploy.md#image--cluster-gitops). ## Prerequisites (one-time) @@ -55,7 +58,8 @@ you want a different baseline, edit the Release PR title before merging. ## Notes - The manual `docker build --platform=linux/amd64 … :sandbox` path - ([sandbox-deploy.md](./sandbox-deploy.md)) still works for ad-hoc iteration; - versioned releases now go through `container-publish` instead. + ([sandbox-deploy.md](./sandbox-deploy.md#emergency-escape-hatch-manual-image--apply)) + is an emergency escape hatch only; every deployed version goes through + `container-publish` and a cluster-repo pin bump. - CI runners and cluster nodes are both amd64, so `container-publish` needs no `--platform` flag (that's only for local Apple-silicon builds). diff --git a/docs/operations/runbook.md b/docs/operations/runbook.md index 253b8de..c5f8ffb 100644 --- a/docs/operations/runbook.md +++ b/docs/operations/runbook.md @@ -1,9 +1,20 @@ # Runbook -On-call playbooks for the codeforphilly-rewrite production service. +On-call playbooks for the codeforphilly-ng production service. -> See also: [monitoring.md](monitoring.md) for the alarm signals that page -> on-call, [cutover.md](cutover.md) for cutover-specific procedures. +> See also: [monitoring.md](monitoring.md) for the alarm signals we *want* +> (none are wired yet), [cutover.md](cutover.md) for cutover-specific +> procedures, [deploy.md](deploy.md) for the GitOps flow. + +## Where to point kubectl + +| | Cluster repo | Namespace | +| --- | --- | --- | +| Production | `cfp-live-cluster` | `codeforphilly-ng` | +| Sandbox | `cfp-sandbox-cluster` | `codeforphilly-rewrite-sandbox` | + +Commands below use `-n codeforphilly-ng`; substitute the sandbox namespace +as needed. The Deployment is `deploy/codeforphilly` in both. ## "API won't boot" @@ -13,24 +24,25 @@ restarting, no `/api/health` response. ### 1. Read the logs first ```bash -kubectl -n codeforphilly logs deploy/codeforphilly --previous -kubectl -n codeforphilly logs deploy/codeforphilly --tail=200 +kubectl -n codeforphilly-ng logs deploy/codeforphilly --previous +kubectl -n codeforphilly-ng logs deploy/codeforphilly --tail=200 ``` -Look for one of the four common boot failures: +Look for one of the common boot failures: | Log line excerpt | Cause | Fix | -|------------------|-------|-----| -| `[entrypoint] ERROR: CFP_DATA_REMOTE is unset` | The Secret containing `CFP_DATA_REMOTE` isn't reaching the pod. | Check `kubectl get secret codeforphilly-secrets -o yaml`; verify the SealedSecret in the GitOps repo decrypted successfully (look at the sealed-secrets controller logs). | +| ------------------ | ------- | ----- | +| `[entrypoint] ERROR: CFP_DATA_REMOTE is unset` | The Secret containing `CFP_DATA_REMOTE` isn't reaching the pod. | Check `kubectl -n codeforphilly-ng get secret codeforphilly-secrets -o yaml`; verify the SealedSecret in the GitOps repo decrypted successfully (look at the sealed-secrets controller logs). | | `fatal: could not read Username for 'https://...'` or `Permission denied (publickey)` | Bad/missing data-repo credentials. | Verify the `codeforphilly-data-deploy-key` Secret holds a valid `id_ed25519` whose public key has push access to the data repo. See [secrets.md](secrets.md#data-repo-deploy-key). | | `Failed to open public gitsheets store` | Bare clone corrupt or missing `.gitsheets/` configs. | Exec into the pod, inspect `/app/data/refs/`, `/app/data/objects/`, and verify `.gitsheets/` exists in HEAD via `git --git-dir=/app/data show HEAD:.gitsheets`. Recovery: restart the pod — `data` is an `emptyDir`, so a fresh pod re-clones from `CFP_DATA_REMOTE` automatically. | -| `Failed to load private store (s3)` | Bucket creds wrong, bucket gone, or network ACL blocks egress. | Confirm `S3_*` env in the ConfigMap + Secret. From the pod, `curl $S3_ENDPOINT` to confirm reachability. | +| `Failed to load private store (filesystem)` | The `codeforphilly-private` PVC didn't mount, is read-only, or the `.jsonl` files are malformed. | `kubectl -n codeforphilly-ng describe pod` for volume events; exec in and `ls -la /app/private-storage`. A malformed line points at a bad manual load — see [legacy-credentials-import.md](legacy-credentials-import.md). | | `environment variable ... is required` | A required env (`CFP_DATA_REPO_PATH`, `STORAGE_BACKEND`, `CFP_JWT_SIGNING_KEY`) is missing. | Manifest regression. Compare against `deploy/kustomize/base/configmap.yaml` + the GitOps repo's SealedSecret. | +| OOMKilled (no log line; `kubectl describe pod` shows it) | In-memory record set too big for `NODE_OPTIONS`/`resources.limits`. Usually an unpruned `published` push. | Check whether the last `published` push skipped the prune step ([spam-detection.md](spam-detection.md)). Re-prune and push; the hot-reload will pick it up once the pod is up. | ### 2. Drop into the pod (if it stays up long enough) ```bash -kubectl -n codeforphilly debug -it deploy/codeforphilly \ +kubectl -n codeforphilly-ng debug -it deploy/codeforphilly \ --image=alpine --target=api -- sh ``` @@ -42,12 +54,12 @@ From inside: ls -la /app/data git --git-dir=/app/data show HEAD:.gitsheets -# Are env vars present? -env | grep -E '^(CFP_|S3_|STORAGE_|GITHUB_)' | sort +# Is the private store there? +ls -la /app/private-storage +wc -l /app/private-storage/*.jsonl -# Can we reach the bucket? -apk add --no-cache curl -curl -v "$S3_ENDPOINT" +# Are env vars present? +env | grep -E '^(CFP_|STORAGE_|GITHUB_|SAML_|POSTMARK_)' | sort # Can we reach the data remote? git ls-remote "$CFP_DATA_REMOTE" 2>&1 | head @@ -59,24 +71,27 @@ If the cluster state is unrecoverable but the data remote is intact: ```bash # Revert the most recent GitOps deploy (the cluster repo's deploy PR is a -# normal merge commit on `deploys/k8s-manifests`) -gh -R CodeForPhilly/cfp-sandbox-cluster pr list --base deploys/k8s-manifests --state merged -git -C ~/Repositories/cfp-sandbox-cluster revert --mainline 1 -git -C ~/Repositories/cfp-sandbox-cluster push origin deploys/k8s-manifests - -# Or pin to a previous image by editing the GitOps repo's -# .holo/branches/k8s-manifests/codeforphilly-ng/app/manifests.toml's image -# tag, committing on a hotfix branch, and merging through the deploy PR. +# normal merge commit on `deploys/k8s-manifests`). Use cfp-sandbox-cluster +# for the sandbox. +gh-axi -R CodeForPhilly/cfp-live-cluster pr list --base deploys/k8s-manifests --state merged +git -C ~/Repositories/cfp-live-cluster revert --mainline 1 +git -C ~/Repositories/cfp-live-cluster push origin deploys/k8s-manifests + +# Or pin to a previous release: move BOTH +# .holo/sources/codeforphilly-ng.toml (ref = "refs/tags/vX.Y.Z") +# codeforphilly-ng/app/kustomization.yaml (images[].newTag) +# back to the last known-good tag on a branch, PR into main, merge the +# resulting deploy PR. # Out-of-band hotfix (bypasses GitOps — fix the repo afterward): -kubectl -n codeforphilly-rewrite-sandbox set image \ +kubectl -n codeforphilly-ng set image \ deploy/codeforphilly codeforphilly=ghcr.io/codeforphilly/codeforphilly-ng: ``` The bare data clone lives in an `emptyDir` — re-cloned from the git remote on -every pod boot. Pod restart is the recovery primitive; there's no PVC to -delete. (A `codeforphilly-private` PVC still exists for the S3-fallback -private store; only `codeforphilly-data` was retired.) +every pod boot. Pod restart is the recovery primitive; there's no data PVC to +delete. The `codeforphilly-private` PVC **is** the private store — never +delete it as part of a recovery. ## "Readiness flapping / 503 spikes" @@ -86,7 +101,8 @@ likely means: - Liveness probe (`/api/health`) failed and k8s is restarting the pod. Look at the previous logs. -- Memory pressure → OOMKilled. Bump `resources.limits.memory` or lower +- Memory pressure → OOMKilled. Check for an unpruned `published` push first + (see the boot table above); only then bump `resources.limits.memory` / `NODE_OPTIONS=--max-old-space-size`. ## "Mutations succeed in UI but don't appear on GitHub" @@ -97,16 +113,22 @@ Push daemon failure. Check logs for git push errors. Common causes: - Remote branch protection rejecting the push. - Network egress blocked. -The local working tree continues to accept writes — it's only the -asynchronous mirror to GitHub that's broken. Once fixed, the daemon will -push the backlog. +The in-memory state and the pod's bare clone continue to accept writes — +it's only the asynchronous mirror to GitHub that's broken. Once fixed, the +daemon pushes the backlog. **Don't restart the pod to "fix" this** until the +backlog has been pushed or fetched out (below): the clone is an `emptyDir` +and unpushed commits die with the pod. ## Hot-reload webhook The API exposes `POST /api/_internal/reload-data` so that a push to -`CFP_DATA_BRANCH` (typically `published`) propagates to the running pod -without rolling it. The `codeforphilly-data` repo's -`notify-deployments.yml` workflow calls this endpoint on every push. +`CFP_DATA_BRANCH` (`published`) propagates to the running pod without +rolling it. The `codeforphilly-data` repo's +`.github/workflows/notify-deployments.yml` calls this endpoint on every push +to `published`, once per target: sandbox (`next-v2.codeforphilly.org`) and +prod (`next.codeforphilly.org` before cutover; the apex after — the +workflow's prod URL must be switched at cutover, see +[cutover.md](cutover.md#t-0-cutover)). See [specs/behaviors/storage.md#hot-reload](../../specs/behaviors/storage.md#hot-reload) for the authoritative contract. Operationally: @@ -114,16 +136,20 @@ for the authoritative contract. Operationally: - **Configured by** the `CFP_DATA_RELOAD_SECRET` env variable. When unset, the endpoint is still registered but returns 503 — the workflow's `curl` will exit non-zero and the operator must - investigate. The secret value lives in the GitOps repo's - `cfp-sandbox-cluster/codeforphilly-ng.secrets/` (or production - equivalent) as a sealed Secret — not in the app repo. + investigate. The value lives in each cluster repo's + `codeforphilly-ng.secrets/codeforphilly-secrets.yaml` (sealed) and as the + `CFP_DATA_RELOAD_SECRET` Actions secret on the data repo — all three must + agree. +- **Public data only.** The webhook reloads the gitsheets state + FTS. It + does **not** reload the private store; files placed on the PVC need a + `rollout restart`. - **Trigger manually** for debugging: ```bash - SECRET=$(kubectl -n codeforphilly-rewrite-sandbox get secret \ - codeforphilly-ng -o jsonpath='{.data.CFP_DATA_RELOAD_SECRET}' | base64 -d) + SECRET=$(kubectl -n codeforphilly-ng get secret \ + codeforphilly-secrets -o jsonpath='{.data.CFP_DATA_RELOAD_SECRET}' | base64 -d) - curl -sS -X POST https://next-v2.codeforphilly.org/api/_internal/reload-data \ + curl -sS -X POST https://next.codeforphilly.org/api/_internal/reload-data \ -H "Authorization: Bearer $SECRET" \ -H "Content-Type: application/json" \ -d '{}' | jq @@ -145,16 +171,17 @@ for the authoritative contract. Operationally: in an undefined state — restart the pod: ```bash - kubectl -n codeforphilly-rewrite-sandbox rollout restart deploy/codeforphilly + kubectl -n codeforphilly-ng rollout restart deploy/codeforphilly ``` - **Outside-the-pod observability** — every reload logs an info line with the outcome + commits; failures log error. Search the pod logs - for `hot-reload` to audit the most recent firings. + for `hot-reload` to audit the most recent firings. The data repo's + Actions run for `notify-deployments.yml` shows the HTTP status per target. ## Fetch from the pod's data clone -The pod's working tree lives on a PVC at `/app/data` inside the container and may briefly hold commits the push daemon hasn't shipped to GitHub yet (or that got escape-hatched onto a `conflicts/*` branch — those *are* always pushed, see [`apps/api/src/store/reconcile.ts`](../../apps/api/src/store/reconcile.ts)). When you want to inspect the pod's view of the data repo without exposing any network ports, add it as a git remote via the `ext::` transport. +The pod's bare clone lives in an `emptyDir` at `/app/data` and may briefly hold commits the push daemon hasn't shipped to GitHub yet (or that got escape-hatched onto a `conflicts/*` branch — those *are* always pushed, see [`apps/api/src/store/reconcile.ts`](../../apps/api/src/store/reconcile.ts)). When you want to inspect the pod's view of the data repo without exposing any network ports, add it as a git remote via the `ext::` transport. ```bash cd /path/to/codeforphilly-data @@ -163,17 +190,17 @@ git config protocol.ext.allow always git remote add pod 'ext::sh /path/to/codeforphilly-ng/scripts/git-pod-uploadpack.sh' git fetch pod -git log --oneline pod/published..pod/published # whatever you're chasing +git log --oneline origin/published..pod/published # unpushed commits, if any ``` -The helper script resolves the current pod by label selector, so it survives restarts. Override via env if your setup differs: +The helper script resolves the current pod by label selector, so it survives restarts. Override via env for the cluster you're targeting: -| Var | Default | -|---|---| -| `CFP_POD_KUBECONFIG` | `~/.kube/cfp-sandbox-cluster-kubeconfig.yaml` | -| `CFP_POD_NAMESPACE` | `codeforphilly-rewrite-sandbox` | -| `CFP_POD_SELECTOR` | `app.kubernetes.io/name=codeforphilly` | -| `CFP_POD_DATA_PATH` | `/app/data` | +| Var | Default | Production | +| --- | --- | --- | +| `CFP_POD_KUBECONFIG` | `~/.kube/cfp-sandbox-cluster-kubeconfig.yaml` | the live cluster kubeconfig | +| `CFP_POD_NAMESPACE` | `codeforphilly-rewrite-sandbox` | `codeforphilly-ng` | +| `CFP_POD_SELECTOR` | `app.kubernetes.io/name=codeforphilly` | same | +| `CFP_POD_DATA_PATH` | `/app/data` | same | **Read-only by design** — `git upload-pack` only serves fetch; pushing back to the pod would bypass gitsheets + in-memory state and fight the push daemon. Pull what you need to your local clone, reason about it there, then push to `origin` if appropriate. @@ -181,14 +208,17 @@ The helper script resolves the current pod by label selector, so it survives res ```bash # Watch a deploy -kubectl -n codeforphilly rollout status deploy/codeforphilly +kubectl -n codeforphilly-ng rollout status deploy/codeforphilly # Last 10 GitOps deploys (merge commits on deploys/k8s-manifests) -gh -R CodeForPhilly/cfp-sandbox-cluster pr list --base deploys/k8s-manifests --state merged --limit 10 +gh-axi -R CodeForPhilly/cfp-live-cluster pr list --base deploys/k8s-manifests --state merged --limit 10 + +# Which release is running? +kubectl -n codeforphilly-ng get deploy codeforphilly -o jsonpath='{.spec.template.spec.containers[0].image}' # Pod resource use -kubectl -n codeforphilly top pod +kubectl -n codeforphilly-ng top pod -# Force a config reload -kubectl -n codeforphilly rollout restart deploy/codeforphilly +# Force a config / secret reload (env is read at pod start) +kubectl -n codeforphilly-ng rollout restart deploy/codeforphilly ``` diff --git a/docs/operations/sandbox-deploy.md b/docs/operations/sandbox-deploy.md index 8997230..51d61ad 100644 --- a/docs/operations/sandbox-deploy.md +++ b/docs/operations/sandbox-deploy.md @@ -1,101 +1,156 @@ -# Manual sandbox deploy +# Sandbox deploy -This is the manual procedure for iterating on a deploy to the **CfP sandbox cluster** (Linode LKE, k8s.phl.io). GitOps wiring is a follow-up; this doc is the source of truth until that lands. +The **CfP sandbox cluster** (Linode LKE, `k8s.phl.io`) hosts the rewrite's +sandbox at . It plays the "staging" role: +every release is bumped here first and soaks before production. + +**Deploys are GitOps.** The +[`cfp-sandbox-cluster`](https://github.com/CodeForPhilly/cfp-sandbox-cluster) +repo pins a release of this repo and applies it; the manual +`docker build … :sandbox && kubectl apply -k` procedure this doc used to +describe is now an emergency escape hatch only (bottom of this page). See +[deploy.md](deploy.md) for the cross-environment picture and +[releases.md](releases.md) for how an image gets published. ## Cluster - **Kubeconfig:** `~/.kube/cfp-sandbox-cluster-kubeconfig.yaml` -- **Gateway:** Envoy Gateway (`gatewayClassName: eg`), wildcard DNS for `*.sandbox.k8s.phl.io` → `139.144.241.4`. The sandbox app is reachable at `next-v2.codeforphilly.org` via a CNAME to `sandbox.k8s.phl.io`. +- **Namespace:** `codeforphilly-rewrite-sandbox` +- **Gateway:** Envoy Gateway (`gatewayClassName: eg`). `*.sandbox.k8s.phl.io` + points at the sandbox LB; `next-v2.codeforphilly.org` is a CNAME onto it, + managed in `CodeForPhilly/ops` (`tf/dns`). - **Storage class:** `linode-block-storage-retain` (default) - **Sealed-secrets:** controller in `sealed-secrets` namespace -- **cert-manager:** `letsencrypt-staging` + `letsencrypt-prod` ClusterIssuers. Sandbox uses prod; per-overlay can override to staging for high-churn iteration (prod rate-limits to 50 certs/week per registered domain). +- **cert-manager:** `letsencrypt-staging` + `letsencrypt-prod` ClusterIssuers. + Sandbox uses prod; switch the Gateway annotation to staging for high-churn + iteration (prod rate-limits to 50 certs/week per registered domain). ## Data repo -The app reads its gitsheets data from a private GitHub repo cloned at boot: - -- **Repo:** `git@github.com:CodeForPhilly/codeforphilly-data.git` (private during cutover prep) -- **Branches** — each is an independent data scenario: - - `fixture` (default) — hand-/import-curated test data, used by sandbox - - `empty` — sheet configs only, no records - - `snapshot` — anonymized snapshot of prod (auto-produced post-cutover) -- A read-only **SSH deploy key** mounted into the pod authenticates the entrypoint's clone. +The app reads its gitsheets data from `git@github.com:CodeForPhilly/codeforphilly-data.git` +(private), bare-cloned at boot. Branches — each is an independent data scenario: -## One-shot deploy steps (manual, while iterating) +- `published` (default) — runtime-served in sandbox **and** prod; the merge + target of `legacy-import`, pruned of confident spam, plus whatever the + running APIs write. A push here hot-reloads both pods. +- `legacy-import` — raw importer snapshots, spam included. +- `fixture` — small hand-curated test data. +- `empty` — sheet configs only, no records. -```bash -export KUBECONFIG=~/.kube/cfp-sandbox-cluster-kubeconfig.yaml +A **write** SSH deploy key mounted into the pod authenticates the entrypoint's +clone and the push daemon. -# 1. Build + push the image -# --platform=linux/amd64 is required when building on Apple Silicon — the -# Linode LKE nodes are amd64 and won't pull an arm64-only manifest. -docker build --platform=linux/amd64 -t ghcr.io/codeforphilly/codeforphilly-ng:sandbox . -# NOTE: requires `write:packages` scope on your GitHub token. -# If `docker push` says "token does not match expected scopes": -# gh auth refresh -s write:packages -docker push ghcr.io/codeforphilly/codeforphilly-ng:sandbox +## Deploying a release (the normal path) -# 2. Apply manifests (creates namespace, sealed-secrets, PVCs, deployment, service, ingress) -kubectl apply -k deploy/kustomize/overlays/sandbox +In `cfp-sandbox-cluster`, on a branch: -# 3. Watch the rollout -kubectl -n codeforphilly-rewrite-sandbox rollout status deploy/codeforphilly -kubectl -n codeforphilly-rewrite-sandbox logs -f deploy/codeforphilly -``` +1. Set `ref = "refs/tags/vX.Y.Z"` in `.holo/sources/codeforphilly-ng.toml`. +2. Set `images[].newTag: vX.Y.Z` in `codeforphilly-ng/app/kustomization.yaml`. +3. Commit both together (`chore(codeforphilly-ng): bump to vX.Y.Z`), PR into + `main`, merge. +4. "Build k8s-manifests" projects the holobranch to `releases/k8s-manifests`; + a bot opens a PR into `deploys/k8s-manifests`. Merge it — that applies. +5. Watch it land: -After the first successful rollout, the app is live at: + ```bash + export KUBECONFIG=~/.kube/cfp-sandbox-cluster-kubeconfig.yaml + kubectl -n codeforphilly-rewrite-sandbox rollout status deploy/codeforphilly + kubectl -n codeforphilly-rewrite-sandbox logs -f deploy/codeforphilly + curl -sS https://next-v2.codeforphilly.org/api/health/ready + ``` -- +Environment-specific bits (Gateway + HTTPRoute for `next-v2.codeforphilly.org`, +the `CFP_SITE_HOST` patch, SealedSecrets) live in the cluster repo under +`_gateways/codeforphilly-ng.yaml`, `codeforphilly-ng/app/kustomization.yaml` +and `codeforphilly-ng.secrets/`. Change them there, not in this repo's +`deploy/kustomize/overlays/sandbox/`. ## Image visibility -The Docker image is built from this repo and pushed to `ghcr.io/codeforphilly/codeforphilly-ng`. For the cluster to pull without an `imagePullSecret`, the package must be **public** on GHCR. After the first push: - -1. Visit -2. Under "Danger Zone" → "Change package visibility" → Public - -Until that's done, the deployment will sit in `ImagePullBackOff` with `403 Forbidden`. +The image is built by `container-publish.yml` on every release tag and pushed +to `ghcr.io/codeforphilly/codeforphilly-ng`. For the cluster to pull without +an `imagePullSecret`, the package must be **public** on GHCR: + +→ "Change package visibility" → Public. If it ever flips back, the deployment +sits in `ImagePullBackOff` with `403 Forbidden`. ## Rotating the deploy key -The SSH deploy key currently in the cluster was generated locally and added to the data repo via `gh repo deploy-key add`. To rotate: +The public half is registered on the data repo's deploy-keys page; the +private half is sealed in `cfp-sandbox-cluster/codeforphilly-ng.secrets/codeforphilly-data-deploy-key.yaml`. ```bash +export KUBECONFIG=~/.kube/cfp-sandbox-cluster-kubeconfig.yaml ssh-keygen -t ed25519 -f /tmp/cfp-deploy-keys/codeforphilly-data-sandbox-rotated -N "" -C "cfp-sandbox-rotated" -gh repo deploy-key add /tmp/cfp-deploy-keys/codeforphilly-data-sandbox-rotated.pub \ - --repo CodeForPhilly/codeforphilly-data \ +gh-axi repo deploy-key add /tmp/cfp-deploy-keys/codeforphilly-data-sandbox-rotated.pub \ + --repo CodeForPhilly/codeforphilly-data --allow-write \ --title "cfp-sandbox cluster (rotated $(date +%Y-%m-%d))" -# Then re-seal the secret and re-apply kubectl create secret generic codeforphilly-data-deploy-key \ --namespace codeforphilly-rewrite-sandbox \ --from-file=id_ed25519=/tmp/cfp-deploy-keys/codeforphilly-data-sandbox-rotated \ --dry-run=client -o yaml \ | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets -o yaml \ - > deploy/kustomize/overlays/sandbox/sealed-secret-deploy-key.yaml -kubectl apply -k deploy/kustomize/overlays/sandbox + > ~/Repositories/cfp-sandbox-cluster/codeforphilly-ng.secrets/codeforphilly-data-deploy-key.yaml +# Commit in the cluster repo, PR, merge the deploy PR, then: +kubectl -n codeforphilly-rewrite-sandbox rollout restart deploy/codeforphilly # Delete the old deploy key from GitHub after the rotation lands cleanly. ``` -## Rotating the JWT signing key +## Rotating the JWT signing key (or any one key in `codeforphilly-secrets`) + +Use `kubeseal --merge-into` so the other keys stay untouched — full recipe in +[secrets.md](secrets.md#adding-or-changing-one-key-in-an-existing-sealedsecret). ```bash -JWT_KEY=$(openssl rand -base64 48) +export KUBECONFIG=~/.kube/cfp-sandbox-cluster-kubeconfig.yaml kubectl create secret generic codeforphilly-secrets \ --namespace codeforphilly-rewrite-sandbox \ - --from-literal=CFP_JWT_SIGNING_KEY="$JWT_KEY" \ - --from-literal=CFP_DATA_REMOTE="git@github.com:CodeForPhilly/codeforphilly-data.git" \ + --from-literal=CFP_JWT_SIGNING_KEY="$(openssl rand -base64 48)" \ --dry-run=client -o yaml \ - | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets -o yaml \ - > deploy/kustomize/overlays/sandbox/sealed-secret-env.yaml -kubectl apply -k deploy/kustomize/overlays/sandbox + | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets --format yaml \ + --merge-into ~/Repositories/cfp-sandbox-cluster/codeforphilly-ng.secrets/codeforphilly-secrets.yaml +# Commit, PR, merge the deploy PR, rollout restart. # Rotating the JWT signing key invalidates every issued session — users will # need to re-auth. Acceptable in sandbox; coordinate before doing this in prod. ``` ## Switching data branches -The active branch is set in `deploy/kustomize/base/configmap.yaml` via `CFP_DATA_BRANCH`. To swap: +`CFP_DATA_BRANCH` defaults to `published` in `deploy/kustomize/base/configmap.yaml`. +To point the sandbox at another branch, add a ConfigMap patch in +`cfp-sandbox-cluster/codeforphilly-ng/app/kustomization.yaml` (same shape as +the `CFP_SITE_HOST` patch), merge through the deploy PR, then +`kubectl -n codeforphilly-rewrite-sandbox rollout restart deploy/codeforphilly` +— the entrypoint re-clones against the new branch on the fresh `emptyDir`. + +## Emergency escape hatch: manual image + apply + +Only when the release pipeline or the GitOps repo is itself broken and you +need to get a fix onto the sandbox *now*. Fix the repos afterwards so the +next GitOps apply doesn't revert you. + +```bash +export KUBECONFIG=~/.kube/cfp-sandbox-cluster-kubeconfig.yaml + +# 1. Build + push a throwaway tag. --platform=linux/amd64 is required on +# Apple Silicon — the LKE nodes are amd64. Needs `write:packages` on your +# GitHub token (`gh auth refresh -s write:packages` if push is refused). +docker build --platform=linux/amd64 -t ghcr.io/codeforphilly/codeforphilly-ng:sandbox . +docker push ghcr.io/codeforphilly/codeforphilly-ng:sandbox + +# 2a. Point the running Deployment at it without touching manifests... +kubectl -n codeforphilly-rewrite-sandbox set image \ + deploy/codeforphilly codeforphilly=ghcr.io/codeforphilly/codeforphilly-ng:sandbox + +# 2b. ...or apply this repo's manual overlay wholesale (namespace, sealed +# secrets, PVC, deployment, service, gateway). Its sealed secrets are +# the sandbox ones — they only decrypt on that cluster. +kubectl apply -k deploy/kustomize/overlays/sandbox + +# 3. Watch +kubectl -n codeforphilly-rewrite-sandbox rollout status deploy/codeforphilly +``` -1. Edit the ConfigMap (or add an overlay patch) -2. `kubectl apply -k deploy/kustomize/overlays/sandbox` -3. `kubectl -n codeforphilly-rewrite-sandbox rollout restart deploy/codeforphilly` — entrypoint re-clones the working tree against the new branch +The `:sandbox` tag is mutable and `imagePullPolicy: Always` in the base +Deployment, so a pod restart picks up a re-push. Never do this against +production. diff --git a/docs/operations/secrets.md b/docs/operations/secrets.md index e236000..a614b09 100644 --- a/docs/operations/secrets.md +++ b/docs/operations/secrets.md @@ -1,6 +1,6 @@ # Secret management -Every secret consumed by codeforphilly-rewrite at runtime, how to generate it, +Every secret consumed by codeforphilly-ng at runtime, how to generate it, how it gets into the cluster, and how to rotate it. > See [deploy.md](deploy.md) for how the Deployment consumes these. See @@ -12,8 +12,8 @@ how it gets into the cluster, and how to rotate it. 1. **Never in the image.** Secrets are mounted at run-time. The Dockerfile carries zero credentials. 2. **Never in git.** Use [sealed-secrets](https://github.com/bitnami-labs/sealed-secrets) - (cluster default) or a SOPS-encrypted file. The plaintext only exists on - the machines that generated it. + (cluster default). The plaintext only exists on the machines that + generated it. 3. **Scoped.** Each secret is granted the minimum surface needed (per-namespace, per-environment). No "infra" secret used for multiple purposes. @@ -22,18 +22,51 @@ how it gets into the cluster, and how to rotate it. ## Where they live in the cluster -The Deployment consumes secrets from two Secret objects, both materialized -by the sealed-secrets controller from `SealedSecret` resources committed in -the GitOps repo (`cfp-sandbox-cluster/codeforphilly-ng.secrets/`): +SealedSecrets are committed in the GitOps repo for each cluster and +materialized by that cluster's sealed-secrets controller. Sealed values are +cluster-bound: a Secret sealed for the sandbox cannot decrypt in the live +cluster. + +| Cluster | Repo path | Namespace | +| --- | --- | --- | +| Production | `cfp-live-cluster/codeforphilly-ng.secrets/` | `codeforphilly-ng` | +| Sandbox | `cfp-sandbox-cluster/codeforphilly-ng.secrets/` | `codeforphilly-rewrite-sandbox` | + +Production has three Secret objects: | Secret name | Mount mechanism | Holds | -|-------------|-----------------|-------| -| `codeforphilly-secrets` | `envFrom: secretRef` (entire Secret becomes env) | All env-var secrets | -| `codeforphilly-data-deploy-key` | Volume-mounted, one file | SSH private key for the data repo | +| ------------- | ----------------- | ------- | +| `codeforphilly-secrets` | `envFrom: secretRef` (entire Secret becomes env) | `CFP_JWT_SIGNING_KEY`, `CFP_DATA_REMOTE`, `CFP_DATA_RELOAD_SECRET`, `GITHUB_OAUTH_CLIENT_ID`, `GITHUB_OAUTH_CLIENT_SECRET`, `POSTMARK_SERVER_TOKEN` | +| `codeforphilly-saml` | `envFrom: secretRef` (added by a patch in the live cluster repo, `optional: true`) | `SAML_PRIVATE_KEY`, `SAML_CERTIFICATE` — carried over verbatim from laddr's `saml2` secret | +| `codeforphilly-data-deploy-key` | Volume-mounted, one file (`id_ed25519`) | SSH private key for the data repo | + +The sandbox has `codeforphilly-secrets` and `codeforphilly-data-deploy-key` +(no SAML — the sandbox is not registered with Slack). + +The first two Secret names are referenced from +`deploy/kustomize/base/deployment.yaml` (`codeforphilly-secrets`, the deploy +key volume) and from the live cluster repo's kustomization patch +(`codeforphilly-saml`); changing them means touching those manifests. + +### Adding or changing one key in an existing SealedSecret + +Use `kubeseal --merge-into` so you only ever handle the key you're changing +and don't have to re-supply every other value: -The Secret names are referenced directly from -`deploy/kustomize/base/deployment.yaml`; changing them means touching the -manifest. +```bash +# against the target cluster's kubeconfig +kubectl create secret generic codeforphilly-secrets \ + --namespace codeforphilly-ng \ + --from-literal=POSTMARK_SERVER_TOKEN="$NEW_TOKEN" \ + --dry-run=client -o yaml \ + | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets \ + --format yaml --merge-into codeforphilly-ng.secrets/codeforphilly-secrets.yaml +``` + +Commit the updated sealed YAML on a branch in the cluster repo, PR into +`main`, merge the resulting deploy PR. Then `kubectl -n codeforphilly-ng +rollout restart deploy/codeforphilly` — env-mounted Secrets are read at pod +start, not live. ## Inventory @@ -50,53 +83,63 @@ HS256 key for stateless session JWTs. - **Rotation impact:** every active session is invalidated. Users have to sign in again. Plan rotations during low-traffic windows; do not rotate during launches. -- **Rotation procedure:** generate new value → update the sealed-secret → - `kubectl rollout restart deployment/codeforphilly` → users re-auth. +- **Rotation procedure:** generate new value → `--merge-into` the + sealed-secret → merge → `kubectl rollout restart` → users re-auth. - **Cadence:** every 12 months, plus immediately on suspected leak. -### `GITHUB_OAUTH_CLIENT_SECRET` - -Client secret for the GitHub OAuth app. **One app per environment** — a -separate app for staging and production, each with its own callback URL. - -- **Generate:** Rotate via the GitHub OAuth app settings page - (`https://github.com/settings/developers` → app → "Generate a new client - secret"). GitHub never reveals the old secret again. -- **Companion config:** `GITHUB_OAUTH_CLIENT_ID` is non-secret and lives in - the ConfigMap (`publicEnv.GITHUB_OAUTH_CLIENT_ID`). +### `GITHUB_OAUTH_CLIENT_ID` / `GITHUB_OAUTH_CLIENT_SECRET` + +Credentials for the GitHub OAuth app. Production uses the **org-owned +"Code for Philly"** app (the laddr app was renamed "Code for Philly +(legacy)"). Its callback URL is registered on the apex, +`https://codeforphilly.org/...`; GitHub accepts redirect URIs on subdomains +of the registered callback, so `next.codeforphilly.org` works against the +same app and **nothing changes at cutover**. Use a separate app for the +sandbox. + +- **Generate:** GitHub → Organizations → CodeForPhilly → Settings → + Developer settings → OAuth Apps → "Code for Philly" → "Generate a new + client secret". GitHub never reveals the old secret again. +- **Client ID:** not sensitive, but it lives in `codeforphilly-secrets` + next to the secret so the pair rotates in one place. - **Rotation impact:** in-flight OAuth callbacks fail. Existing sessions are unaffected (the secret is only used during the OAuth handshake). -- **Rotation procedure:** issue new secret in GitHub → update sealed-secret → - `kubectl rollout restart`. +- **Rotation procedure:** issue new secret in GitHub → `--merge-into` → + merge → `kubectl rollout restart`. - **Cadence:** every 12 months, plus immediately on suspected leak. -### `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` +### `CFP_DATA_RELOAD_SECRET` -Credentials for the private-storage bucket. The IAM policy attached to -these credentials must be scoped to the single bucket per -[deploy.md](deploy.md#bucket-provisioning). +Bearer token gating `POST /api/_internal/reload-data` (the hot-reload +webhook). The same value must be set as the `CFP_DATA_RELOAD_SECRET` +Actions secret in `CodeForPhilly/codeforphilly-data`, which +`notify-deployments.yml` sends to both sandbox and prod — so today one +value is shared across both environments. -- **Generate:** Provider console (R2 → API tokens; B2 → application keys; - AWS → IAM access key). -- **Rotation impact:** PUTs and GETs to the private store fail until the - pod is restarted with new keys. Newsletter signups and account claims - return `private_store_unavailable` (5xx). -- **Rotation procedure:** - 1. Provision a *second* access key in the bucket provider. - 2. Update the sealed-secret with the new value. - 3. `kubectl rollout restart deployment/codeforphilly` — pod boots with - new keys. - 4. Verify `/api/health/ready` returns 200. - 5. Revoke the old access key in the bucket provider. -- **Cadence:** every 6 months, plus immediately on suspected leak. +- **Generate:** `openssl rand -hex 32`. +- **Rotation impact:** until both sides match, every `published` push + fails its notify step (401) and pods serve stale data until restarted. +- **Rotation procedure:** `--merge-into` both cluster repos → merge → + restart both pods → update the data repo's Actions secret → push a + no-op to `published` (or re-run the last workflow) to confirm 200s. +- **Cadence:** every 12 months, plus immediately on suspected leak. + +### `CFP_DATA_REMOTE` + +The data repo's git URL (`git@github.com:CodeForPhilly/codeforphilly-data.git`). +Not itself sensitive — it lives in `codeforphilly-secrets` because the +entrypoint reads it from the same env block. Changing it takes effect on +the next pod start (fresh `emptyDir`, fresh clone). ### `SAML_PRIVATE_KEY` / `SAML_CERTIFICATE` -PEM-encoded cert chain that signs SAML assertions for the Slack IdP -integration ([specs/api/saml.md](../../specs/api/saml.md)). +PEM-encoded key pair that signs SAML assertions for the Slack IdP +integration ([specs/api/saml.md](../../specs/api/saml.md)). In production +these are **the laddr values**, copied from the legacy `saml2` Secret in +the `code-for-philly` namespace, so Slack's existing trust in that cert +carries straight over. Don't regenerate them as part of cutover. -- **Generate:** the openssl recipe documented in the legacy repo at - `laddr/docs/operations/update-saml2-certificate.md`: +- **Generate (only for a deliberate rotation):** ```bash openssl req -x509 -newkey rsa:2048 -days 1095 -nodes \ @@ -107,7 +150,8 @@ integration ([specs/api/saml.md](../../specs/api/saml.md)). - **Rotation impact:** Slack stops trusting assertions until its IdP config is updated with the new cert. **Do not rotate without coordinating with - the Slack workspace admin.** + the Slack workspace admin.** With SSO set to "optional" a botched + rotation is recoverable via email magic links. - **Not a secret, but paired:** `SAML_ENTITY_ID` (ConfigMap, optional) is the IdP identity Slack stores alongside this cert. It defaults to `https://codeforphilly.org/api/saml/slack/metadata` and must stay stable @@ -116,12 +160,13 @@ integration ([specs/api/saml.md](../../specs/api/saml.md)). - **Rotation procedure:** 1. Generate new key + cert. 2. Upload the *new cert* to Slack as a secondary signing cert. - 3. Update the sealed-secret with the new key + cert. - 4. `kubectl rollout restart`. - 5. Test SAML SSO from a clean browser. + 3. `--merge-into codeforphilly-ng.secrets/codeforphilly-saml.yaml` with + the new key + cert. + 4. Merge → `kubectl rollout restart`. + 5. Test SAML SSO from a clean browser (Slack "Test configuration"). 6. Once verified, remove the old cert from Slack. -- **Cadence:** every 36 months (cert expiry), plus immediately on - suspected leak. +- **Cadence:** on cert expiry (check the `notAfter` of the carried-over + cert — laddr issued it), plus immediately on suspected leak. ### `POSTMARK_SERVER_TOKEN` @@ -142,74 +187,106 @@ for local dev but means real users get no outbound mail in production. that exists on the chosen server. - **Rotation impact:** none in-flight (no in-flight email state on our end); next outbound mail uses the new token. -- **Rotation procedure:** create new token in Postmark → update - sealed-secret → `kubectl rollout restart` → delete the old token in - Postmark. +- **Rotation procedure:** create new token in Postmark → `--merge-into` → + merge → `kubectl rollout restart` → delete the old token in Postmark. - **Cadence:** every 12 months, plus immediately on suspected leak. ### Data-repo deploy key SSH ed25519 private key with **write** access to the `codeforphilly-data` repo on GitHub. Mounted as a file at `/etc/cfp-data-deploy-key/id_ed25519`; -the entrypoint sets `GIT_SSH_COMMAND` to use it. +the ConfigMap's `GIT_SSH_COMMAND` points ssh at it. - **Generate:** ```bash - ssh-keygen -t ed25519 -f cfp-data-deploy -C "codeforphilly k8s deploy" + ssh-keygen -t ed25519 -f cfp-data-deploy -N "" -C "codeforphilly-ng deploy" ``` Add the *public* key (`cfp-data-deploy.pub`) to the data repo's - Settings → Deploy keys with "Allow write access" checked. -- **Rotation impact:** push daemon will fail to push commits until the new - key is mounted. Reads/writes to the in-memory state continue; the - inability to push surfaces as a backlog of unpushed commits on the PVC. + Settings → Deploy keys with "Allow write access" checked + (`gh-axi repo deploy-key add ... --allow-write`). One key per cluster. +- **Rotation impact:** the entrypoint's clone fails on the next pod start + and the push daemon fails to push until the new key is mounted. Reads + and writes to in-memory state continue; unpushed commits sit in the + pod's bare clone (and are lost if the pod is recreated before the push + succeeds — see [runbook.md](runbook.md#fetch-from-the-pods-data-clone)). - **Rotation procedure:** 1. Generate new keypair. 2. Add new public key to the data repo (alongside the existing one). - 3. Update the sealed-secret with the new private key. - 4. `kubectl rollout restart`. + 3. Re-seal `codeforphilly-data-deploy-key` with the new private key + (`kubectl create secret generic ... --from-file=id_ed25519=... | kubeseal ...`, + replacing the file — a single-key Secret doesn't need `--merge-into`). + 4. Merge → `kubectl rollout restart`. 5. Verify a test mutation reaches the remote. 6. Remove the old deploy key from the data repo. - **Cadence:** every 12 months, plus immediately on team turnover or suspected leak. +### `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` (unused) + +Credentials for an S3-compatible private-storage bucket. **No environment +uses the `s3` backend today** — sandbox and production both run +`STORAGE_BACKEND=filesystem` on a PVC ([deploy.md](deploy.md#private-storage)). +If that ever changes: provision the bucket with versioning on, scope an IAM +policy to that single bucket, add the two keys to `codeforphilly-secrets` +and `S3_ENDPOINT` / `S3_BUCKET` / `S3_REGION` to the ConfigMap, and +migrate the two `.jsonl` files off the PVC. Rotation would follow the +two-key overlap pattern (provision second key → swap → verify → revoke). + ## Bootstrapping a new environment -First-time set up of `codeforphilly-staging` or `codeforphilly`: +First-time set up of a new namespace (e.g. a second sandbox). Run against +the target cluster's kubeconfig so `kubeseal` fetches that cluster's public +key. ```bash -# 1. Create namespace -kubectl create namespace codeforphilly-staging +NS=codeforphilly-ng # or codeforphilly-rewrite-sandbox +OUT=~/Repositories/cfp-live-cluster/codeforphilly-ng.secrets + +# 1. Namespace (in the cluster repo's namespace.yaml normally; kubectl for a scratch env) +kubectl create namespace "$NS" -# 2. Generate all secret values locally +# 2. Generate values locally +mkdir -p .secrets openssl rand -base64 64 > .secrets/jwt -ssh-keygen -t ed25519 -f .secrets/deploy -N "" -# ... GitHub OAuth secret from GitHub UI, S3 keys from R2 console ... +openssl rand -hex 32 > .secrets/reload +ssh-keygen -t ed25519 -f .secrets/deploy -N "" -C "codeforphilly-ng $NS deploy" +# ... GitHub OAuth client id + secret from the GitHub UI, Postmark token from Postmark ... -# 3. Build the Secret manifests +# 3. Seal kubectl create secret generic codeforphilly-secrets \ - --namespace codeforphilly-staging \ + --namespace "$NS" \ --from-literal=CFP_JWT_SIGNING_KEY="$(cat .secrets/jwt)" \ + --from-literal=CFP_DATA_RELOAD_SECRET="$(cat .secrets/reload)" \ + --from-literal=CFP_DATA_REMOTE="git@github.com:CodeForPhilly/codeforphilly-data.git" \ + --from-literal=GITHUB_OAUTH_CLIENT_ID="$GH_CLIENT_ID" \ --from-literal=GITHUB_OAUTH_CLIENT_SECRET="$GH_SECRET" \ - --from-literal=S3_ACCESS_KEY_ID="$S3_ID" \ - --from-literal=S3_SECRET_ACCESS_KEY="$S3_KEY" \ - --from-literal=SAML_PRIVATE_KEY="$(cat .secrets/saml-private.pem)" \ - --from-literal=SAML_CERTIFICATE="$(cat .secrets/saml-certificate.pem)" \ + --from-literal=POSTMARK_SERVER_TOKEN="$POSTMARK_TOKEN" \ --dry-run=client -o yaml \ - | kubeseal --format yaml > deploy/secrets/staging-secrets.sealed.yaml + | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets --format yaml \ + > "$OUT/codeforphilly-secrets.yaml" kubectl create secret generic codeforphilly-data-deploy-key \ - --namespace codeforphilly-staging \ + --namespace "$NS" \ --from-file=id_ed25519=.secrets/deploy \ --dry-run=client -o yaml \ - | kubeseal --format yaml > deploy/secrets/staging-deploy-key.sealed.yaml - -# 4. Commit the sealed YAMLs into the GitOps repo -# (cfp-sandbox-cluster/codeforphilly-ng.secrets/), open a PR. -# The deploy workflow applies them on merge. + | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets --format yaml \ + > "$OUT/codeforphilly-data-deploy-key.yaml" + +# Production only — carry the laddr SAML key pair over rather than minting a new one: +kubectl -n code-for-philly get secret saml2 -o yaml # inspect key names, then +kubectl create secret generic codeforphilly-saml \ + --namespace "$NS" \ + --from-literal=SAML_PRIVATE_KEY="$(kubectl -n code-for-philly get secret saml2 -o jsonpath='{.data.}' | base64 -d)" \ + --from-literal=SAML_CERTIFICATE="$(kubectl -n code-for-philly get secret saml2 -o jsonpath='{.data.}' | base64 -d)" \ + --dry-run=client -o yaml \ + | kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets --format yaml \ + > "$OUT/codeforphilly-saml.yaml" -# 5. Wipe plaintext +# 4. Commit the sealed YAMLs in the GitOps repo, PR into main, merge the deploy PR. +# 5. Add the deploy public key to the data repo; set CFP_DATA_RELOAD_SECRET in the data repo's Actions secrets. +# 6. Wipe plaintext shred -u .secrets/* ``` @@ -221,9 +298,10 @@ the sealed-secrets controller in the matching cluster. Listed because operators ask: - `GITHUB_OAUTH_CLIENT_ID` — public by design; GitHub exposes it during - every OAuth flow. -- `S3_ENDPOINT` / `S3_BUCKET` / `S3_REGION` — public addressing info. + every OAuth flow. (Stored in the Secret anyway, for one-place rotation.) - `CFP_DATA_REMOTE` — URL form. The *access* (deploy key) is secret; the - URL itself isn't. + URL itself isn't. (Also stored in the Secret; see above.) - `SAML_CERTIFICATE` (the public cert) is technically published to Slack anyway, but we keep it next to the private key for atomic rotation. +- `SAML_ENTITY_ID`, `CFP_SITE_HOST`, `SLACK_TEAM_HOST`, `CFP_DATA_BRANCH` — + ConfigMap values. diff --git a/docs/operations/spam-detection.md b/docs/operations/spam-detection.md index 7cec414..4e00cc9 100644 --- a/docs/operations/spam-detection.md +++ b/docs/operations/spam-detection.md @@ -13,7 +13,7 @@ All work — scripts, sheet configs, and evaluation records — currently sits o Four sheets back the system, all defined under `.gitsheets/` in `codeforphilly-data`: | Sheet | Path template | Purpose | -|---|---|---| +| --- | --- | --- | | `person-evaluations` | `${{ personSlug }}/${{ evaluator }}` | One verdict per (person, evaluator). Multiple evaluators coexist per person. | | `slack-presence` | `${{ personSlug }}` | Per-person Slack snapshot: channel membership, message count, recent messages, spam-message aggregates. | | `slack-channels` | `${{ id }}` | Workspace channel catalog (id, name, member count, is-default, etc.). | @@ -159,7 +159,7 @@ npm run evaluate-heuristic npm run evaluate-llm # 6. Apply the verdicts — prune confident-spam from `published` (see below). -# Run from the codeforphilly-rewrite repo against a bare clone, then push. +# Run from the codeforphilly-ng repo against a bare clone, then push. # THIS STEP IS MANDATORY after any import/merge — see "Applying spam decisions". ``` @@ -173,10 +173,10 @@ When source records get updated (e.g., a previously-empty profile gets a new bio Verdicts are advisory until the **prune** step applies them. Prune is not a read-path filter (the runtime loader stays spam-unaware); it **removes confident-spam people from `published`** so the deployed app never loads them into memory or shows them. This is what keeps the in-memory footprint within the node budget — see [specs/behaviors/spam-exclusion.md](../../specs/behaviors/spam-exclusion.md) for the full contract. -The tool is `apps/api/scripts/prune-spam.ts` in the **`codeforphilly-rewrite`** repo (not the data repo). Run it against a bare clone of the data repo that carries both `published` and `spam-detection`, dry-run first, then push: +The tool is `apps/api/scripts/prune-spam.ts` in the **`codeforphilly-ng`** repo (not the data repo). Run it against a bare clone of the data repo that carries both `published` and `spam-detection`, dry-run first, then push: ```bash -# From the codeforphilly-rewrite repo +# From the codeforphilly-ng repo npm run -w apps/api script:prune-spam -- \ --data-repo=/path/to/codeforphilly-data.git \ --evaluations-ref=spam-detection \ @@ -230,7 +230,7 @@ npm run profile -- forager For the initial run on the full historical corpus (~31,470 people, ~119k Slack messages): | Step | Cost | -|---|---| +| --- | --- | | `fetch-slack-presence` | $0 (Slack API is free) | | `evaluate-slack-messages` (Pass B) | ~$13 | | `evaluate-heuristic` | $0 | From ad174d54f48f913646140259591c9709f8bbea62 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 10 Sep 2026 02:06:10 -0400 Subject: [PATCH 4/8] docs(ops): document the real laddr credentials export and PVC load The export command pointed at a `laddr` database and a `users` table with Handle/PasswordHash columns that don't exist. The live database is `emergence-site` (the `codeforphilly` schema on the same server is a stale 2024 copy), table `people`, columns Username/Email/Password, and the export runs inside the laddr pod with the Habitat-packaged mysql client against the service's client.cnf. The production load is kubectl cp onto the pod's private-storage PVC followed by a rollout restart, same as sandbox; the GCS/S3 upload section is reduced to a note for the unused s3 backend. Records the first prod load (2026-09-10, 21,761 profiles/credentials) and warns that repeating it resets rehashed credentials now that sign-ins are live at next.codeforphilly.org. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR --- docs/operations/legacy-credentials-import.md | 166 +++++++++++-------- 1 file changed, 101 insertions(+), 65 deletions(-) diff --git a/docs/operations/legacy-credentials-import.md b/docs/operations/legacy-credentials-import.md index 66947e4..a05fe33 100644 --- a/docs/operations/legacy-credentials-import.md +++ b/docs/operations/legacy-credentials-import.md @@ -1,8 +1,8 @@ # Legacy credentials import How to seed `profiles.jsonl` (PrivateProfile) + `legacy-passwords.jsonl` -(LegacyPasswordCredential) into a deployment's private store from a -laddr MySQL export. +(LegacyPasswordCredential) into a deployment's private store from the +laddr MySQL database. The public-side importer ([`script:import-laddr`](../../apps/api/scripts/import-laddr.ts)) deliberately handles only public data — laddr's public JSON API doesn't @@ -16,12 +16,16 @@ rehashed credentials with the originals. The script is re-runnable shape-wise (it always produces a complete replacement), but in practice it's run once per env unless something went wrong with the first run. +The first production load ran **2026-09-10**: 21,761 profiles/credentials +landed on the prod PVC. + ## What you need ### CSV export from laddr -A CSV with header `Username,Email,Password`, one row per active laddr -user. Each cell is double-quoted; embedded quotes use `""` escape. +A CSV with header `Username,Email,Password`, one row per laddr user with +both a username and an email. Each cell is double-quoted; embedded quotes +use `""` escape. ```csv "Username","Email","Password" @@ -29,42 +33,74 @@ user. Each cell is double-quoted; embedded quotes use `""` escape. "hhutch","hunter.hutchinson@gmail.com","7746451fd8c30b5b4068cd45fa7b1052cef54068" ``` -- **Username** — laddr `User.Handle` (= the slug that became `Person.slug`). +- **Username** — laddr `people.Username` (= the handle that became `Person.slug`). - **Email** — required; rows with empty Email are skipped (warned). -- **Password** — the raw `PasswordHash` column from laddr's `users` table. - Mixed algorithms in the wild: older users are unsalted SHA-1 (40 lowercase - hex chars, no prefix); newer users are bcrypt (`$2a$ / $2b$ / $2y$`). - Empty Password is allowed — emits a PrivateProfile but no - LegacyPasswordCredential; user will have to use the password-reset - flow if they ever want one. The runtime verifier handles all three - formats; see [specs/behaviors/password-hash-rotation.md](../../specs/behaviors/password-hash-rotation.md). +- **Password** — the raw `people.Password` column. Mixed algorithms in the + wild: older users are unsalted SHA-1 (40 lowercase hex chars, no prefix); + newer users are bcrypt (`$2a$ / $2b$ / $2y$`). Empty Password is allowed — + emits a PrivateProfile but no LegacyPasswordCredential; user will have + to use the password-reset flow if they ever want one. The runtime + verifier handles all three formats; see + [specs/behaviors/password-hash-rotation.md](../../specs/behaviors/password-hash-rotation.md). + +#### Where the data actually is + +The live laddr site runs in the live cluster, namespace `code-for-philly`, +as an Emergence/Habitat pod with MySQL alongside. The database is +**`emergence-site`**. The same server also has a `codeforphilly` schema — +that is a **stale 2024 copy**; don't export from it. The table is `people` +(not `users`), columns `Username`, `Email`, `Password`. -Produce the file with: +#### Produce the file + +Run the export *inside* the laddr pod with the Habitat-packaged mysql +client, reading the socket credentials from the service's client config: ```bash -mysql -h -u -p laddr \ - -B -e "SELECT Handle AS Username, Email, PasswordHash AS Password - FROM users WHERE Email IS NOT NULL AND PasswordHash IS NOT NULL" \ - | sed 's/"/\\\\"/g' | awk -F'\t' 'BEGIN{print "\"Username\",\"Email\",\"Password\""} NR>1 {printf "\"%s\",\"%s\",\"%s\"\n", $1, $2, $3}' \ - > .scratch/legacy-logins-export.csv +# from your machine, against the live cluster kubeconfig +POD=$(kubectl -n code-for-philly get pods -l app.kubernetes.io/name=code-for-philly -o jsonpath='{.items[0].metadata.name}') +# (adjust the label selector if it doesn't match; `kubectl -n code-for-philly get pods` and pick the site pod) + +kubectl -n code-for-philly exec "$POD" -- sh -c ' + M=$(ls -d /hab/pkgs/core/mysql/*/*/bin/mysql | tail -1); + $M --defaults-extra-file=/hab/svc/mysql/config/client.cnf emergence-site -N -B -e \ + "SELECT Username, Email, Password FROM people + WHERE Email IS NOT NULL AND Email<>'"'"''"'"' AND Username IS NOT NULL AND Username<>'"'"''"'"'" +' > .scratch/legacy-logins-export.tsv + +# TSV -> quoted CSV in the shape the importer wants +awk -F'\t' ' + BEGIN { print "\"Username\",\"Email\",\"Password\"" } + { for (i = 1; i <= 3; i++) gsub(/"/, "\"\"", $i); + printf "\"%s\",\"%s\",\"%s\"\n", $1, $2, $3 } +' .scratch/legacy-logins-export.tsv > .scratch/legacy-logins-export.csv + +wc -l .scratch/legacy-logins-export.csv # expect ~21.8k rows + header ``` -(Or use whatever extract tooling you prefer — the importer just needs -the CSV shape above. Land the file in `.scratch/` which is gitignored.) +If the nested quoting in the `exec` line fights you, `kubectl exec -it` into +the pod, run the two-line `M=…; $M …` command there redirecting to +`/tmp/export.tsv`, then `kubectl cp` it out. Land the file in `.scratch/`, +which is gitignored. ### A bare clone of `codeforphilly-data` The importer reads Person records (for the `slug → personId` map) via the same `openPublicStore` interface the runtime uses, which requires a -bare clone. If your dev sibling clone is a working tree, make a +bare clone on the **`published`** branch (the runtime branch — the data +repo has no `main`). If your dev sibling clone is a working tree, make a side-clone for the importer: ```bash git clone --bare --branch published \ - ~/Repositories/codeforphilly-data \ + git@github.com:CodeForPhilly/codeforphilly-data.git \ /tmp/codeforphilly-data-bare-published.git ``` +Make sure it's at the tip you just pushed in the public-data refresh +([cutover.md → T-1](cutover.md#t-1-day-data-refresh--credentials)) — +a Person that isn't on `published` can't be matched. + ## Run ```bash @@ -88,63 +124,62 @@ directory into the workspace and relative paths resolve from there. The report prints input row count, write counts, and a breakdown of skip reasons (no-username / no-email / no-person-match / deleted-person -/ duplicate-person). Expect a ~95% match rate on a healthy laddr corpus -— the missing 5% are typically deleted laddr users that didn't survive -the public-data import. +/ duplicate-person). A large `no-person-match` count is expected when +`published` has been spam-pruned — the pruned people are exactly the ones +that shouldn't get a credential. + +## Deploy (FilesystemPrivateStore on a PVC — sandbox and production) + +Both environments use `STORAGE_BACKEND=filesystem` with +`/app/private-storage` mounted from the `codeforphilly-private` +PersistentVolumeClaim ([deploy.md → Private storage](deploy.md#private-storage)). +The procedure is identical; only the namespace differs: + +| | Namespace | Health URL | +| --- | --- | --- | +| Production | `codeforphilly-ng` | `https://next.codeforphilly.org/api/health/ready` (apex after cutover) | +| Sandbox | `codeforphilly-rewrite-sandbox` | `https://next-v2.codeforphilly.org/api/health/ready` | + +1. **Snapshot what's there** (skip on a first load into an empty PVC): -## Deploy to sandbox (FilesystemPrivateStore on a PVC) + ```bash + NS=codeforphilly-ng + POD=$(kubectl -n "$NS" get pods -l app.kubernetes.io/name=codeforphilly -o jsonpath='{.items[0].metadata.name}') + kubectl -n "$NS" exec "$POD" -- sh -c 'cd /app/private-storage && for f in *.jsonl; do cp "$f" "$f.bak-$(date +%Y%m%dT%H%M%S)"; done; ls -la' + ``` -The sandbox uses `STORAGE_BACKEND=filesystem` with `/app/private-storage` -mounted to a PersistentVolumeClaim (see `deploy/kustomize/base/`). + The PVC has no versioning; the `.bak` copy on the volume is your undo. -1. **Copy the files into the pod:** +2. **Copy the files into the pod:** ```bash - POD=$(kubectl -n codeforphilly-rewrite-sandbox get pods -o jsonpath='{.items[0].metadata.name}') - kubectl -n codeforphilly-rewrite-sandbox cp \ + kubectl -n "$NS" cp \ .scratch/private-import/profiles.jsonl "$POD:/app/private-storage/profiles.jsonl" - kubectl -n codeforphilly-rewrite-sandbox cp \ + kubectl -n "$NS" cp \ .scratch/private-import/legacy-passwords.jsonl "$POD:/app/private-storage/legacy-passwords.jsonl" ``` -2. **Restart the pod** to reload the private store into memory. The +3. **Restart the pod** to reload the private store into memory. The `POST /api/_internal/reload-data` webhook **does not** cover the private store — it only reloads public + FTS. A full pod restart is the supported path. ```bash - kubectl -n codeforphilly-rewrite-sandbox rollout restart deployment + kubectl -n "$NS" rollout restart deploy/codeforphilly + kubectl -n "$NS" rollout status deploy/codeforphilly ``` -3. **Verify** the new pod sees the credentials: - - ```bash - curl -sS https://next-v2.codeforphilly.org/api/health/ready # waits until private store is loaded - ``` +4. **Verify** the new pod sees the credentials: `/api/health/ready` + returns 200, and a known legacy user can sign in with their laddr + password. -## Deploy to production (S3PrivateStore on GCS or S3) - -Production uses `STORAGE_BACKEND=s3` with `S3_ENDPOINT` pointing at -either GCS's XML API (`https://storage.googleapis.com`) or an actual -S3 bucket. Upload the two files to the bucket at the configured -`keyPrefix` (default empty): - -```bash -# GCS via gsutil -gsutil cp .scratch/private-import/profiles.jsonl \ - gs:///profiles.jsonl -gsutil cp .scratch/private-import/legacy-passwords.jsonl \ - gs:///legacy-passwords.jsonl - -# OR: any S3 endpoint via awscli -aws s3 cp .scratch/private-import/profiles.jsonl \ - s3:///profiles.jsonl --endpoint-url -aws s3 cp .scratch/private-import/legacy-passwords.jsonl \ - s3:///legacy-passwords.jsonl --endpoint-url -``` +### If the private store is ever moved to S3 -Then restart the prod pod the same way (private-store reload is -not in the hot-reload path; see above). +The `s3` backend ([specs/behaviors/private-storage.md](../../specs/behaviors/private-storage.md#backends)) +is supported but **not in use anywhere today**. If a deployment is ever +switched to it, the equivalent of step 2 is uploading the two files to the +bucket root (or the configured `keyPrefix`) with `aws s3 cp +--endpoint-url "$S3_ENDPOINT"`, then restarting the pod as above. ## Safety notes @@ -158,9 +193,10 @@ not in the hot-reload path; see above). - **Re-deploying overwrites the runtime files in full.** If users have already signed in and their credentials have been rehashed to argon2id (via the in-app rehash-on-login flow), a re-deploy would - revert those to the SHA-1/bcrypt originals. After cutover, only - re-run + re-deploy if you've confirmed nobody has signed in (or - you're OK with the rotation reset). + revert those to the SHA-1/bcrypt originals. Production has been + accepting sign-ins at `next.codeforphilly.org` since the first load, so + **do not repeat the prod load casually** — only if you've confirmed + nobody has signed in since, or you're OK with the rotation reset. - **PII risk.** The generated JSONL files contain every legacy user's email and password hash. Treat them as you would the source MySQL dump — never paste into chat, never check into git, delete from @@ -170,4 +206,4 @@ not in the hot-reload path; see above). - [specs/behaviors/private-storage.md](../../specs/behaviors/private-storage.md) — what these files store and the rules around them. - [specs/behaviors/password-hash-rotation.md](../../specs/behaviors/password-hash-rotation.md) — how the verifier handles SHA-1 / bcrypt / argon2id and rehashes on login. -- [docs/operations/cutover.md](./cutover.md) — full cutover sequence; this importer fits between the public-data import and the DNS flip. +- [docs/operations/cutover.md](./cutover.md) — full cutover sequence; this importer runs at T-1, after the public-data refresh and before the gateway hostname move. From 5a416313d988ba77bc670fb1a870431449127a69 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 10 Sep 2026 02:11:31 -0400 Subject: [PATCH 5/8] chore(plans): mark cutover-runbook-reality done (PR #172) Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR --- plans/cutover-runbook-reality.md | 57 +++++++++++++++++++++++++++----- 1 file changed, 48 insertions(+), 9 deletions(-) diff --git a/plans/cutover-runbook-reality.md b/plans/cutover-runbook-reality.md index 9784cc9..a31060d 100644 --- a/plans/cutover-runbook-reality.md +++ b/plans/cutover-runbook-reality.md @@ -1,5 +1,5 @@ --- -status: in-progress +status: done depends: - cutover-prep - release-flow @@ -11,6 +11,7 @@ specs: - specs/behaviors/storage.md - specs/api/saml.md issues: [] +pr: 172 --- # Plan: Cutover runbook reality check @@ -88,12 +89,12 @@ on 2026-09-09/10: ## Validation -- [ ] `grep -rn 'cfp-prod-cluster\|staging.k8s\|cutover-window-policy\|T+180' docs/ .claude/CLAUDE.md` returns nothing. -- [ ] No doc names `main` as a data-repo branch; runtime branch is `published` everywhere. -- [ ] No doc gives S3/GCS as the production private-storage path; the filesystem PVC is. -- [ ] `cutover.md` T-0 and rollback sections describe the gateway-listener commit and its revert; no DNS/TTL steps remain. -- [ ] `legacy-credentials-import.md` export command targets `emergence-site`.`people` via the Habitat client. -- [ ] `npm run lint` clean. +- [x] `grep -rn 'cfp-prod-cluster\|staging.k8s\|cutover-window-policy\|T+180' docs/ .claude/CLAUDE.md` returns nothing. +- [x] No doc names `main` as a data-repo branch; runtime branch is `published` everywhere. +- [x] No doc gives S3/GCS as the production private-storage path; the filesystem PVC is. +- [x] `cutover.md` T-0 and rollback sections describe the gateway-listener commit and its revert; no DNS/TTL steps remain. +- [x] `legacy-credentials-import.md` export command targets `emergence-site`.`people` via the Habitat client. +- [x] `npm run lint` clean. ## Risks / unknowns @@ -107,8 +108,46 @@ on 2026-09-09/10: ## Notes -(Populated at closeout.) +- **Verified from sibling clones, not from memory.** The cluster pins, + gateway listener names (`https-apex`, `https-www`, `https-subdomain` on + `code-for-philly`; `https-next` on `codeforphilly-ng`), the three + SealedSecret names and their keys, the `CFP_SITE_HOST` patch, the + `Build k8s-manifests` workflow, `notify-deployments.yml`, and the + `tf/dns` records were all read from local clones of `cfp-live-cluster`, + `cfp-sandbox-cluster`, `codeforphilly-data-published` and `CodeForPhilly/ops`. + Anything about Slack admin UI, GitHub OAuth app naming, the MySQL schema, + or import counts is as reported by the operator who ran it. +- **The runtime image can't run operator scripts.** `Dockerfile` prunes dev + deps and copies only `dist/`, so `script:reconcile` has no in-pod path and + the private store can't be copied out to run it locally. `cutover.md` + names this as an open gap under T+7 instead of prescribing a workaround. +- **`deploy/kustomize/overlays/sandbox/` is now legacy.** The sandbox cluster + repo carries its own namespace, secrets and hostname patches; the overlay + in this repo only backs the manual escape hatch. Left in place; removing it + is a separate decision. +- **The announcement templates were also wrong on passwords.** They told + members password sign-in "is going away"; fixed alongside the T+90/T+180 + deletions since a member reading them would be misled the same way. +- **`HOST` was dropped from the env table** — it isn't in + `deploy/kustomize/base/configmap.yaml`; Fastify's default binding is what + runs. ## Follow-ups -(Populated at closeout.) +- Spec drift in `specs/architecture.md` (`STORAGE_BACKEND` "`s3` in + production", entrypoint "`git reset --hard origin/main`", + `overlays/staging/` + `overlays/production/`) and + `specs/behaviors/private-storage.md` ("`s3` backend (production)", bucket + versioning as a production requirement). Both deployed environments run + `filesystem` on a PVC with no versioning. Needs its own spec PR; not + touched here because this plan is docs-only. +- `apps/api/scripts/cutover-mailout.ts` template says "Accounts unclaimed + for one year may be retired" — no spec backs that. Either the sunset spec + gets written first or the line goes. +- Ship `script:reconcile` (or an equivalent) somewhere it can reach the + production private store — in the image, or as an in-cluster Job — so the + T+7 reconciliation check in `cutover.md` is runnable. +- Monitoring: none of `docs/operations/monitoring.md` is wired. First + post-cutover ops task; the checklist there is the spec for it. +- Consider deleting `deploy/kustomize/overlays/sandbox/` once nobody needs + the manual escape hatch. From b60d77aa01f5ccfbb9a855b0150d07622d34719c Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 10 Sep 2026 02:13:21 -0400 Subject: [PATCH 6/8] docs(specs): require AuthnStatement and accept Redirect-binding SSO The SAML assertion spec described Subject, Conditions and the attribute set but never an AuthnStatement, and the implementation followed it. The Web Browser SSO profile requires at least one, and Slack's SP config sends a RequestedAuthnContext, so an assertion without one is the most likely rejection when we test against the real workspace. Spec the statement: AuthnInstant = assertion issue time, a fresh SessionIndex, and a fixed AuthnContextClassRef of ...:ac:classes:Password. That class matches what the legacy Emergence SAML2 connector emitted (setAuthnContext(AC_PASSWORD)) against this same workspace, which is worth more than the marginally more precise PasswordProtectedTransport. Slack's live "Test configuration" also failed with a 404: it sends the AuthnRequest over HTTP-Redirect (GET with a DEFLATEd SAMLRequest in the query), and although our metadata advertises that binding at the /sso Location, the spec only defined POST. Add GET to the endpoints table and describe both bindings as one flow that differs only in transport. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR --- specs/api/saml.md | 46 +++++++++++++++++++++++++++++++++++++++------- 1 file changed, 39 insertions(+), 7 deletions(-) diff --git a/specs/api/saml.md b/specs/api/saml.md index f99b2f0..05d53e1 100644 --- a/specs/api/saml.md +++ b/specs/api/saml.md @@ -10,7 +10,8 @@ GitHub OAuth is how a member proves identity to the new site. SAML is how the si | ------ | ---- | ---- | ------- | | `GET` | `/api/saml/slack/metadata` | public | IdP metadata XML for Slack to consume | | `GET` | `/api/saml/slack/launch` | user | IdP-initiated SSO — site → Slack | -| `POST` | `/api/saml/slack/sso` | user | SP-initiated SSO callback — handles AuthnRequest from Slack | +| `GET` | `/api/saml/slack/sso` | user | SP-initiated SSO callback — AuthnRequest from Slack via the HTTP-Redirect binding | +| `POST` | `/api/saml/slack/sso` | user | SP-initiated SSO callback — AuthnRequest from Slack via the HTTP-POST binding | For the existing `/chat` redirect that Slack-launches members into channels, see [screens/chat.md](../screens/chat.md). The SAML endpoints live under `/api/saml/slack/*` because the v1 design leaves room for additional SAML SP integrations later. @@ -43,6 +44,25 @@ The attribute values come from: - `first_name` → `Person.firstName` - `last_name` → `Person.lastName` +### Authentication statement + +Every assertion carries exactly one `` — the Web Browser SSO profile (saml-profiles §4.1.4.2) requires at least one, and an assertion without it is a valid rejection reason for any SP. It sits between `` and `` (schema order: Subject, Conditions, then statements). The legacy connector emitted one via simplesamlphp's `Assertion` (`setSessionIndex(generateId())` + `setAuthnContext(SAML2_Constants::AC_PASSWORD)`); v1 preserves that shape: + +```text +AuthnStatement: + AuthnInstant the moment this assertion was issued + SessionIndex one per assertion; not the assertion ID + AuthnContext/ + AuthnContextClassRef urn:oasis:names:tc:SAML:2.0:ac:classes:Password fixed — see below +``` + +Rules: + +- **`AuthnInstant` is the assertion's issue time**, not the time the member's underlying session was established. We don't track the original sign-in instant in the JWT, and re-asserting "now" is what the legacy connector did. +- **`SessionIndex` is a fresh opaque identifier per assertion.** Slack never sends us a LogoutRequest, so nothing correlates on it; it exists to satisfy the profile. Use a fresh id rather than reusing the assertion ID so the two values stay independently meaningful. +- **The `AuthnContextClassRef` is the fixed value above.** It is *not* echoed back from the AuthnRequest's `RequestedAuthnContext` — a member proves identity to us via GitHub OAuth or the legacy password path, and we describe that once, the same way for every SP-initiated and IdP-initiated response. Slack's default `RequestedAuthnContext` is `PasswordProtectedTransport`; the legacy connector asserted `Password` against that same workspace for years without rejection, and matching the value existing Slack accounts were established under is worth more than the marginally more precise class. +- **`SessionNotOnOrAfter` is omitted.** The assertion's `Conditions/@NotOnOrAfter` already bounds the assertion; we make no claim about IdP-session lifetime. + ## GET /api/saml/slack/metadata Returns the IdP's SAML metadata XML, signed with the IdP cert. Slack consumes this once during admin setup; we generally don't re-fetch. @@ -111,30 +131,42 @@ The destination URL inside the Response includes the `redir` so Slack's POST end - `400 validation_failed` — bad `channel` format - `500 internal_error` with `error.code = "saml_signing_failed"` — IdP cert/key misconfiguration -## POST /api/saml/slack/sso +## GET | POST /api/saml/slack/sso **SP-initiated sign-in** — Slack received a request from a member who wants to sign in, sent us a SAML AuthnRequest. We complete authentication and return a SAML Response. -### Request body +The metadata advertises this one Location under both `SingleSignOnService` bindings, so the endpoint accepts the AuthnRequest either way. Slack uses **HTTP-Redirect** (`GET`) when a member starts sign-in from Slack and when an admin runs "Test configuration"; `POST` is accepted for the HTTP-POST binding. The two differ only in how `SAMLRequest` is transported; everything after decoding is one flow. + +### Request — HTTP-Redirect binding (`GET`) + +Query string (saml-bindings §3.4.4.1, `DEFLATE` encoding): + +| Param | Required | Notes | +| ----- | -------- | ----- | +| `SAMLRequest` | yes | raw-DEFLATEd, then base64-encoded, then URL-encoded SAML AuthnRequest XML | +| `RelayState` | no | opaque value Slack wants us to echo back | +| `SigAlg`, `Signature` | no | detached signature — ignored unless request signing is enabled (it isn't for Slack) | + +### Request — HTTP-POST binding (`POST`) `application/x-www-form-urlencoded`: | Field | Required | Notes | | ----- | -------- | ----- | -| `SAMLRequest` | yes | base64-encoded SAML AuthnRequest XML | +| `SAMLRequest` | yes | base64-encoded SAML AuthnRequest XML (no DEFLATE) | | `RelayState` | no | opaque value Slack wants us to echo back | ### Behavior -1. Decode + parse the AuthnRequest. Validate signature if Slack signs requests (configurable; usually no for Slack). +1. Decode + parse the AuthnRequest (inflate first for the Redirect binding). Validate signature if Slack signs requests (configurable; usually no for Slack). 2. Require a signed-in session. If not → store the AuthnRequest in a short-lived signed cookie, redirect to `/login?return=/api/saml/slack/sso?resume=1`. After login the user comes back here and the AuthnRequest replays from the cookie. 3. Resolve the AuthnRequest's `AssertionConsumerServiceURL` against Slack's documented ACS endpoint(s) — only Slack's ACS is accepted. 4. Build + sign a SAML Response as in `/launch`. -5. POST back to Slack's ACS via the auto-submitting form, including `RelayState`. +5. POST back to Slack's ACS via the auto-submitting form, including `RelayState`. The Response always goes back over HTTP-POST regardless of which binding carried the request — Slack's ACS only accepts POST. ### Errors -- `400 validation_failed` with code `saml_request_invalid` — malformed AuthnRequest or unrecognized ACS URL +- `400 validation_failed` with code `saml_request_invalid` — malformed AuthnRequest (including a Redirect-binding payload that fails to inflate) or unrecognized ACS URL - `401 unauthenticated` — no session (with resume-cookie flow as above) - `403 forbidden` with `error.code = "saml_not_permitted"` From 3d6bca2a452f3f92756063b196e920764275a8ba Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 10 Sep 2026 02:28:03 -0400 Subject: [PATCH 7/8] fix(api): emit AuthnStatement and accept Redirect-binding AuthnRequests Two gaps found while preparing to test the IdP against the live Slack workspace, both in the SP-initiated path. The assertion carried Subject, Conditions and AttributeStatement but no AuthnStatement, which the Web Browser SSO profile requires and which Slack's RequestedAuthnContext gives it a reason to check. Add one to the login response template between Conditions and AttributeStatement, with AuthnInstant = the assertion issue time, a fresh SessionIndex from the same id source as the Response/Assertion IDs, and the fixed class urn:oasis:names:tc:SAML:2.0:ac:classes:Password. That class is what the legacy Emergence SAML2 connector asserted against this workspace (setAuthnContext(AC_PASSWORD)), so existing Slack accounts see the same context they were established under. The placeholders ride the existing customTagReplacement path; samlify signs after that callback returns, so the statement lands inside the signed subtree. Slack's "Test configuration" then 404ed: Slack sends the AuthnRequest over HTTP-Redirect (GET with a DEFLATEd SAMLRequest in the query), and while the metadata advertised that binding at /sso, only POST was registered. Add GET /api/saml/slack/sso. Rather than call samlify's 'redirect' parser and teach the resume cookie about bindings, inflate at the edge back to the plain-base64 form the POST binding carries and run both through one handleSpInitiatedSso; samlify's redirect flow is that same inflate followed by the same parser, so nothing is lost and the cookie's samlRequest claim keeps a single shape. Fold spaces back to '+' in the query value, since a sender that leaves base64 '+' unescaped has it URL-decoded to a space. Tests cover the AuthnStatement (position, ClassRef, AuthnInstant <= IssueInstant, SessionIndex distinct from the assertion ID), verify the signature cryptographically against the metadata endpoint's cert and show that tampering with the ClassRef breaks it, and exercise the Redirect binding for signed-in, anonymous (through /sso/resume), bad ACS and non-DEFLATEd payloads. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR --- apps/api/src/routes/saml.ts | 299 +++++++++++++++++++++++------------- apps/api/src/saml/config.ts | 34 +++- apps/api/tests/saml.test.ts | 201 ++++++++++++++++++++++++ 3 files changed, 426 insertions(+), 108 deletions(-) diff --git a/apps/api/src/routes/saml.ts b/apps/api/src/routes/saml.ts index e565398..8331cb1 100644 --- a/apps/api/src/routes/saml.ts +++ b/apps/api/src/routes/saml.ts @@ -4,15 +4,17 @@ * Implements specs/api/saml.md: * GET /api/saml/slack/metadata * GET /api/saml/slack/launch - * POST /api/saml/slack/sso + * GET /api/saml/slack/sso (SP-initiated, HTTP-Redirect binding) + * POST /api/saml/slack/sso (SP-initiated, HTTP-POST binding) * GET /api/saml/slack/sso/resume (sign-in continuation) * * Cert + key load lazily — endpoints return 500 saml_signing_failed if the * environment is missing them. Routes are mounted regardless so the * metadata URL is always discoverable. */ -import type { FastifyInstance, FastifyRequest } from 'fastify'; +import type { FastifyInstance, FastifyReply, FastifyRequest } from 'fastify'; import { randomBytes } from 'node:crypto'; +import { inflateRawSync } from 'node:zlib'; import { ForbiddenError, UnauthenticatedError, ApiValidationError } from '../lib/errors.js'; import { errorResponse } from '../lib/response.js'; import { @@ -186,6 +188,11 @@ function buildAssertionUser(opts: { * - fill in NameQualifier / SPNameQualifier (samlify's default skips both) * - substitute the per-attribute placeholder tags built by samlify's * `attributeStatementBuilder` (e.g. `{attrEmail}`, `{attrUsername}`). + * - fill the AuthnStatement (AuthnInstant / SessionIndex / ClassRef) our + * template carries — samlify's default path blanks its own slot. + * + * samlify signs the assertion *after* this callback returns, so everything + * substituted here is covered by the signature. */ function buildCustomTagReplacement(opts: { readonly user: SlackAssertionUser; @@ -197,11 +204,15 @@ function buildCustomTagReplacement(opts: { return (template) => { const id = opts.generateID(); const assertionId = opts.generateID(); + // Fresh per assertion and distinct from the assertion ID — mirrors the + // legacy connector's `setSessionIndex(generateId())`. + const sessionIndex = opts.generateID(); const subs = buildResponseSubstitutions({ user: opts.user, slackTeamHost: opts.slackTeamHost, issuerEntityId: opts.issuerEntityId, inResponseTo: opts.inResponseTo, + sessionIndex, }); const fullSubs: Record = { ID: id, @@ -249,6 +260,149 @@ async function loadPersonAndProfile( return { person, profile }; } +/** + * Lift an HTTP-Redirect-binding `SAMLRequest` (saml-bindings §3.4.4.1: XML → + * raw DEFLATE → base64 → URL-encode) back to the plain-base64 form the + * HTTP-POST binding carries, so both bindings share one parse + resume path + * and the resume cookie's `samlRequest` claim keeps a single shape. + * + * samlify's own redirect flow is exactly this inflate followed by the same + * parser; doing the inflate here rather than calling the 'redirect' parser + * is what lets the cookie stay binding-agnostic. + * + * Fastify has already URL-decoded the query. A sender that leaves base64 `+` + * unescaped would have it decoded to a space, so fold spaces back — base64 + * never legitimately contains one. + */ +function inflateRedirectBindingRequest(deflatedB64: string): string { + const normalised = deflatedB64.replace(/ /g, '+'); + if (!normalised) { + throw new ApiValidationError('SAMLRequest is required', { SAMLRequest: 'required' }); + } + let xml: string; + try { + xml = inflateRawSync(Buffer.from(normalised, 'base64')).toString('utf8'); + } catch { + throw new ApiValidationError('Malformed SAMLRequest', { + SAMLRequest: 'inflate failed', + }); + } + return Buffer.from(xml, 'utf8').toString('base64'); +} + +/** + * SP-initiated SSO, binding-agnostic. Takes the AuthnRequest as plain base64 + * XML (the HTTP-POST wire form — the Redirect handler inflates into it first) + * and either issues the signed Response (signed-in) or parks the request in + * the resume cookie and bounces through /login (anonymous). + */ +async function handleSpInitiatedSso( + fastify: FastifyInstance, + request: FastifyRequest, + reply: FastifyReply, + input: { readonly samlRequestB64: string; readonly relayState: string }, +): Promise { + const cfg = fastify.config; + if (!cfg.SAML_PRIVATE_KEY || !cfg.SAML_CERTIFICATE) { + return reply.code(500).send( + errorResponse( + 'saml_signing_failed', + 'SAML IdP is not configured', + (request as FastifyRequest & { traceId?: string }).traceId, + ), + ); + } + + const { samlRequestB64, relayState } = input; + if (!samlRequestB64) { + throw new ApiValidationError('SAMLRequest is required', { + SAMLRequest: 'required', + }); + } + + const { entities } = getSamlContext(fastify); + + // Parse the AuthnRequest to extract its ID + AssertionConsumerServiceURL. + let parsed: Awaited>; + try { + parsed = await entities.idp.parseLoginRequest(entities.sp, 'post', { + body: { SAMLRequest: samlRequestB64 }, + }); + } catch (err) { + fastify.log.warn({ err }, 'SAML AuthnRequest parse failed'); + throw new ApiValidationError('Malformed SAMLRequest', { + SAMLRequest: 'parse failed', + }); + } + + const extract = parsed.extract as { + request?: { id?: string; assertionConsumerServiceUrl?: string }; + }; + const acsUrl = extract.request?.assertionConsumerServiceUrl ?? entities.acsUrl; + const requestId = extract.request?.id ?? ''; + + assertAcsAllowed(acsUrl, cfg.SLACK_TEAM_HOST); + + // Anonymous → stash the AuthnRequest in the resume cookie, redirect to /login. + if (!request.session.personId) { + const resumeToken = await signSamlResume( + { + samlRequest: samlRequestB64, + relayState, + acsUrl, + requestId, + }, + cfg.CFP_JWT_SIGNING_KEY, + ); + reply.setCookie(RESUME_COOKIE, resumeToken, { + httpOnly: true, + sameSite: 'lax', + secure: isSecure(cfg.NODE_ENV), + path: '/api/saml', + maxAge: RESUME_COOKIE_TTL_SECONDS, + }); + const resumeReturn = `${originBase(request)}/api/saml/slack/sso/resume`; + return reply.redirect(`/login?return=${encodeURIComponent(resumeReturn)}`); + } + + // Signed in — build the assertion immediately. + const { person, profile } = await loadPersonAndProfile(fastify, request.session.personId); + const user = buildAssertionUser({ person, profile }); + + const customTagReplacement = buildCustomTagReplacement({ + user, + slackTeamHost: cfg.SLACK_TEAM_HOST, + issuerEntityId: entities.entityId, + inResponseTo: requestId, + generateID: () => `_${cryptoRandomId()}`, + }); + + // The Response always returns over HTTP-POST regardless of which binding + // carried the request — Slack's ACS only accepts POST. + const bindingCtx = await entities.idp.createLoginResponse( + entities.sp, + { extract: parsed.extract }, + 'post', + {}, + { relayState, customTagReplacement }, + ); + + const samlResponse = bindingCtx.context; + const actionUrl = + 'entityEndpoint' in bindingCtx && typeof bindingCtx.entityEndpoint === 'string' + ? bindingCtx.entityEndpoint + : acsUrl; + const replyRelayState = 'relayState' in bindingCtx ? bindingCtx.relayState : relayState; + + return reply.header('Content-Type', 'text/html; charset=utf-8').send( + renderPostForm({ + actionUrl, + samlResponse, + relayState: replyRelayState ?? undefined, + }), + ); +} + // --------------------------------------------------------------------------- // Routes // --------------------------------------------------------------------------- @@ -368,15 +522,48 @@ export async function samlRoutes(fastify: FastifyInstance): Promise { ); // ------------------------------------------------------------------------- - // POST /api/saml/slack/sso — SP-initiated SSO + // GET | POST /api/saml/slack/sso — SP-initiated SSO + // + // One Location, two bindings (the metadata advertises both). The handlers + // below only differ in how they lift `SAMLRequest` off the wire; everything + // from parsing onward is `handleSpInitiatedSso`. Slack itself uses the + // Redirect binding (GET) — both for member-started sign-in and the admin + // "Test configuration" button. // ------------------------------------------------------------------------- + fastify.get( + '/api/saml/slack/sso', + { + schema: { + tags: ['saml'], + summary: 'SP-initiated Slack sign-in (AuthnRequest, HTTP-Redirect binding)', + querystring: { + type: 'object', + properties: { + SAMLRequest: { type: 'string' }, + RelayState: { type: 'string' }, + SigAlg: { type: 'string' }, + Signature: { type: 'string' }, + }, + required: ['SAMLRequest'], + }, + }, + }, + async (request, reply) => { + const query = request.query as { SAMLRequest?: string; RelayState?: string }; + return handleSpInitiatedSso(fastify, request, reply, { + samlRequestB64: inflateRedirectBindingRequest(query.SAMLRequest ?? ''), + relayState: query.RelayState ?? '', + }); + }, + ); + fastify.post( '/api/saml/slack/sso', { schema: { tags: ['saml'], - summary: 'SP-initiated Slack sign-in (AuthnRequest)', + summary: 'SP-initiated Slack sign-in (AuthnRequest, HTTP-POST binding)', body: { type: 'object', properties: { @@ -388,109 +575,11 @@ export async function samlRoutes(fastify: FastifyInstance): Promise { }, }, async (request, reply) => { - const cfg = fastify.config; - if (!cfg.SAML_PRIVATE_KEY || !cfg.SAML_CERTIFICATE) { - return reply.code(500).send( - errorResponse( - 'saml_signing_failed', - 'SAML IdP is not configured', - (request as FastifyRequest & { traceId?: string }).traceId, - ), - ); - } - const body = request.body as { SAMLRequest?: string; RelayState?: string }; - const samlRequestB64 = body.SAMLRequest ?? ''; - const relayState = body.RelayState ?? ''; - if (!samlRequestB64) { - throw new ApiValidationError('SAMLRequest is required', { - SAMLRequest: 'required', - }); - } - - const { entities } = getSamlContext(fastify); - - // Parse the AuthnRequest to extract its ID + AssertionConsumerServiceURL. - let parsed: Awaited>; - try { - parsed = await entities.idp.parseLoginRequest(entities.sp, 'post', { - body: { SAMLRequest: samlRequestB64 }, - }); - } catch (err) { - fastify.log.warn({ err }, 'SAML AuthnRequest parse failed'); - throw new ApiValidationError('Malformed SAMLRequest', { - SAMLRequest: 'parse failed', - }); - } - - const extract = parsed.extract as { - request?: { id?: string; assertionConsumerServiceUrl?: string }; - }; - const acsUrl = - extract.request?.assertionConsumerServiceUrl ?? entities.acsUrl; - const requestId = extract.request?.id ?? ''; - - assertAcsAllowed(acsUrl, cfg.SLACK_TEAM_HOST); - - // Anonymous → stash the AuthnRequest in the resume cookie, redirect to /login. - if (!request.session.personId) { - const resumeToken = await signSamlResume( - { - samlRequest: samlRequestB64, - relayState, - acsUrl, - requestId, - }, - cfg.CFP_JWT_SIGNING_KEY, - ); - reply.setCookie(RESUME_COOKIE, resumeToken, { - httpOnly: true, - sameSite: 'lax', - secure: isSecure(cfg.NODE_ENV), - path: '/api/saml', - maxAge: RESUME_COOKIE_TTL_SECONDS, - }); - const resumeReturn = `${originBase(request)}/api/saml/slack/sso/resume`; - return reply.redirect(`/login?return=${encodeURIComponent(resumeReturn)}`); - } - - // Signed in — build the assertion immediately. - const { person, profile } = await loadPersonAndProfile(fastify, request.session.personId); - const user = buildAssertionUser({ person, profile }); - - const customTagReplacement = buildCustomTagReplacement({ - user, - slackTeamHost: cfg.SLACK_TEAM_HOST, - issuerEntityId: entities.entityId, - inResponseTo: requestId, - generateID: () => `_${cryptoRandomId()}`, + return handleSpInitiatedSso(fastify, request, reply, { + samlRequestB64: body.SAMLRequest ?? '', + relayState: body.RelayState ?? '', }); - - const bindingCtx = await entities.idp.createLoginResponse( - entities.sp, - { extract: parsed.extract }, - 'post', - {}, - { relayState, customTagReplacement }, - ); - - const samlResponse = bindingCtx.context; - const actionUrl = - 'entityEndpoint' in bindingCtx && typeof bindingCtx.entityEndpoint === 'string' - ? bindingCtx.entityEndpoint - : acsUrl; - const replyRelayState = - 'relayState' in bindingCtx ? bindingCtx.relayState : relayState; - - return reply - .header('Content-Type', 'text/html; charset=utf-8') - .send( - renderPostForm({ - actionUrl, - samlResponse, - relayState: replyRelayState ?? undefined, - }), - ); }, ); diff --git a/apps/api/src/saml/config.ts b/apps/api/src/saml/config.ts index 7038188..84aee48 100644 --- a/apps/api/src/saml/config.ts +++ b/apps/api/src/saml/config.ts @@ -22,6 +22,15 @@ const { IdentityProvider, ServiceProvider, Constants, SamlLib, setSchemaValidato // `@authenio/samlify-xsd-schema-validator` java dependency. const NAMEID_FORMAT_PERSISTENT = 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent'; +/** + * The one `AuthnContextClassRef` every assertion carries, per + * specs/api/saml.md#authentication-statement. Fixed — never echoed back from + * the AuthnRequest's `RequestedAuthnContext`. Matches what the legacy + * Emergence SAML2 connector asserted (`SAML2\Constants::AC_PASSWORD`) against + * this same Slack workspace. + */ +export const SLACK_AUTHN_CONTEXT_CLASS_REF = 'urn:oasis:names:tc:SAML:2.0:ac:classes:Password'; + let schemaValidatorConfigured = false; function ensureSchemaValidator(): void { if (schemaValidatorConfigured) return; @@ -79,6 +88,12 @@ export interface BuildResponseSubstitutionsOptions { readonly slackTeamHost: string; readonly issuerEntityId: string; readonly inResponseTo: string; + /** + * Opaque per-assertion `AuthnStatement/@SessionIndex`. Generated by the + * caller alongside the Response/Assertion IDs so all three share one id + * source; it must differ from the assertion ID (see spec). + */ + readonly sessionIndex: string; /** RFC3339 timestamp (current time) — passed in so tests can pin it. */ readonly nowIso?: string; } @@ -113,6 +128,14 @@ export function buildResponseSubstitutions( SPNameQualifier: 'https://slack.com', NameID: opts.user.nameId, InResponseTo: opts.inResponseTo, + // AuthnStatement (specs/api/saml.md#authentication-statement). These are + // our own placeholders — samlify's built-in template has a bare + // `{AuthnStatement}` slot that the default (non-custom) path blanks out, + // so we carry the statement literally in our template and fill its + // attributes here. AuthnInstant is the assertion issue time by design. + AuthnInstant: now, + SessionIndex: opts.sessionIndex, + AuthnContextClassRef: SLACK_AUTHN_CONTEXT_CLASS_REF, // samlify's `attributeStatementBuilder` derives placeholder names from // each attribute's `valueTag` via `'attr' + camelCase + first-upper`, so // `valueTag: 'email'` → `{attrEmail}`, `valueTag: 'firstName'` → @@ -199,10 +222,15 @@ export function buildSlackSamlEntities(settings: SamlIdpSettings): SlackSamlEnti ], loginResponseTemplate: { // samlify substitutes {AttributeStatement} from the configured attribute - // list; the rest of the template comes from the library's built-in - // response template. + // list at IdP construction; every other placeholder is filled by our + // customTagReplacement callback (see buildResponseSubstitutions). + // + // Element order inside follows the schema (saml-core + // §2.3.3): Issuer, [Signature — inserted after Issuer by samlify], + // Subject, Conditions, then statements. AuthnStatement precedes + // AttributeStatement, mirroring the legacy connector's output. context: - '{Issuer}{Issuer}{NameID}{Audience}{AttributeStatement}', + '{Issuer}{Issuer}{NameID}{Audience}{AuthnContextClassRef}{AttributeStatement}', attributes: [ { name: 'User.Email', diff --git a/apps/api/tests/saml.test.ts b/apps/api/tests/saml.test.ts index c2ce141..1fdc524 100644 --- a/apps/api/tests/saml.test.ts +++ b/apps/api/tests/saml.test.ts @@ -8,7 +8,12 @@ * SAMLResponse carrying the expected NameID + attribute set * - GET /api/saml/slack/launch?channel=phlask → relayState carries channel * - GET /api/saml/slack/launch?channel= → 422 + * - Assertion carries an AuthnStatement (fixed ClassRef, AuthnInstant <= + * IssueInstant, fresh SessionIndex) — specs/api/saml.md#authentication-statement * - POST /api/saml/slack/sso (anonymous) → resume cookie + 302 to /login + * - GET /api/saml/slack/sso (HTTP-Redirect binding, DEFLATEd SAMLRequest) + * behaves exactly as POST for signed-in / signed-out / bad payload, and + * the anonymous path resumes through /sso/resume * - GET /api/saml/slack/sso/resume (signed-in, valid cookie) → POST form * - Metadata endpoint without SAML_PRIVATE_KEY → 500 saml_signing_failed */ @@ -16,7 +21,9 @@ import { afterAll, beforeAll, describe, expect, it } from 'vitest'; import { type FastifyInstance } from 'fastify'; import { writeFile } from 'node:fs/promises'; import { join } from 'node:path'; +import { deflateRawSync } from 'node:zlib'; import { DOMParser } from '@xmldom/xmldom'; +import * as samlify from 'samlify'; import { buildApp } from '../src/app.js'; import { mintSessionFor } from '../src/auth/issue.js'; @@ -57,6 +64,34 @@ function decodeSamlResponse(html: string): string { return Buffer.from(match![1]!, 'base64').toString('utf8'); } +/** Fixed AuthnContextClassRef per specs/api/saml.md#authentication-statement. */ +const AUTHN_CONTEXT_CLASS_REF = 'urn:oasis:names:tc:SAML:2.0:ac:classes:Password'; + +/** A minimal Slack-shaped AuthnRequest targeting the configured ACS. */ +function slackAuthnRequestXml(id: string, acsHost: string = SLACK_TEAM_HOST): string { + return ` +https://slack.com`; +} + +/** + * HTTP-Redirect binding encoding (saml-bindings §3.4.4.1): raw DEFLATE → + * base64 → URL-encode. Returns the ready-to-append query string. + */ +function redirectBindingQuery(xml: string, relayState?: string): string { + const deflated = deflateRawSync(Buffer.from(xml, 'utf8')).toString('base64'); + const params = new URLSearchParams({ SAMLRequest: deflated }); + if (relayState !== undefined) params.set('RelayState', relayState); + return params.toString(); +} + +function resumeCookieValue(res: { headers: Record }): string { + const cookies = res.headers['set-cookie']; + const list = Array.isArray(cookies) ? cookies : [String(cookies ?? '')]; + const hit = list.find((c) => c.startsWith('cfp_saml_resume=')); + expect(hit).toBeDefined(); + return hit!.split(';')[0]!.slice('cfp_saml_resume='.length); +} + async function seedPerson( repoDir: string, opts: { @@ -272,6 +307,172 @@ describe('SAML IdP — Slack', () => { expect((sigs?.length ?? 0)).toBeGreaterThan(0); }); + it('assertion carries an AuthnStatement with the fixed ClassRef and a sane AuthnInstant', async () => { + const { accessToken } = await mintSessionFor(personId, 'user', JWT_KEY); + const res = await app.inject({ + method: 'GET', + url: '/api/saml/slack/launch', + cookies: { cfp_session: accessToken }, + }); + expect(res.statusCode).toBe(200); + + const xml = decodeSamlResponse(res.body); + const doc = new DOMParser().parseFromString(xml, 'application/xml'); + const root = doc.documentElement!; + const assertion = root.getElementsByTagNameNS(ASSERTION_NS, 'Assertion')[0]!; + + // Exactly one AuthnStatement, placed after Conditions and before + // AttributeStatement (schema order, saml-core §2.3.3). + const authnStatements = assertion.getElementsByTagNameNS(ASSERTION_NS, 'AuthnStatement'); + expect(authnStatements.length).toBe(1); + const authn = authnStatements[0]!; + const childNames = Array.from(assertion.childNodes) + .filter((n) => n.nodeType === 1) + .map((n) => (n as Element).localName); + expect(childNames.indexOf('AuthnStatement')).toBeGreaterThan(childNames.indexOf('Conditions')); + expect(childNames.indexOf('AuthnStatement')).toBeLessThan( + childNames.indexOf('AttributeStatement'), + ); + + // Fixed ClassRef — not echoed from any RequestedAuthnContext. + const classRef = authn.getElementsByTagNameNS(ASSERTION_NS, 'AuthnContextClassRef')[0]; + expect(classRef?.textContent).toBe(AUTHN_CONTEXT_CLASS_REF); + + // AuthnInstant parses as an ISO date and is <= the assertion's IssueInstant. + const authnInstant = authn.getAttribute('AuthnInstant'); + const issueInstant = assertion.getAttribute('IssueInstant'); + expect(authnInstant).toMatch(/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?Z$/); + expect(Number.isNaN(Date.parse(authnInstant!))).toBe(false); + expect(Date.parse(authnInstant!)).toBeLessThanOrEqual(Date.parse(issueInstant!)); + + // SessionIndex is present, opaque, and not the assertion ID. + const sessionIndex = authn.getAttribute('SessionIndex'); + expect(sessionIndex).toMatch(/^_[0-9a-f]+$/); + expect(sessionIndex).not.toBe(assertion.getAttribute('ID')); + + // SessionNotOnOrAfter is omitted by design. + expect(authn.hasAttribute('SessionNotOnOrAfter')).toBe(false); + + // The statement sits inside the signed subtree: the enveloped Signature + // is a child of the Assertion and its Reference points at the Assertion ID. + const sig = assertion.getElementsByTagNameNS('http://www.w3.org/2000/09/xmldsig#', 'Signature')[0]; + expect(sig?.parentNode).toBe(assertion); + const ref = sig?.getElementsByTagNameNS('http://www.w3.org/2000/09/xmldsig#', 'Reference')[0]; + expect(ref?.getAttribute('URI')).toBe(`#${assertion.getAttribute('ID')}`); + }); + + it('assertion signature verifies against the metadata cert and covers the AuthnStatement', async () => { + const meta = await app.inject({ method: 'GET', url: '/api/saml/slack/metadata' }); + expect(meta.statusCode).toBe(200); + const idpMetadata = samlify.IdPMetadata(meta.body); + + const { accessToken } = await mintSessionFor(personId, 'user', JWT_KEY); + const res = await app.inject({ + method: 'GET', + url: '/api/saml/slack/launch', + cookies: { cfp_session: accessToken }, + }); + expect(res.statusCode).toBe(200); + const xml = decodeSamlResponse(res.body); + + // Signing happens after templating, so the substituted AuthnStatement is + // inside the signed subtree: the untouched Response verifies... + const [verified, signedAssertion] = samlify.SamlLib.verifySignature(xml, { + metadata: idpMetadata, + }); + expect(verified).toBe(true); + expect(signedAssertion).toContain(' { + const res = await app.inject({ + method: 'GET', + url: `/api/saml/slack/sso?${redirectBindingQuery(slackAuthnRequestXml('id-redirect-1'), 'opaque-redirect-state')}`, + }); + expect(res.statusCode).toBe(302); + expect(res.headers.location).toMatch(/^\/login\?return=/); + expect(resumeCookieValue(res)).not.toBe(''); + }); + + it('GET /api/saml/slack/sso (redirect binding, anonymous) → /sso/resume completes with InResponseTo + RelayState', async () => { + const start = await app.inject({ + method: 'GET', + url: `/api/saml/slack/sso?${redirectBindingQuery(slackAuthnRequestXml('id-redirect-2'), 'opaque-redirect-state')}`, + }); + expect(start.statusCode).toBe(302); + const resumeCookie = resumeCookieValue(start); + + const { accessToken } = await mintSessionFor(personId, 'user', JWT_KEY); + const res = await app.inject({ + method: 'GET', + url: '/api/saml/slack/sso/resume', + cookies: { cfp_session: accessToken, cfp_saml_resume: resumeCookie }, + }); + expect(res.statusCode).toBe(200); + expect(res.body).toContain(`action="https://${SLACK_TEAM_HOST}/sso/saml"`); + expect(res.body).toContain('name="RelayState" value="opaque-redirect-state"'); + + const xml = decodeSamlResponse(res.body); + const doc = new DOMParser().parseFromString(xml, 'application/xml'); + expect(doc.documentElement?.getAttribute('InResponseTo')).toBe('id-redirect-2'); + expect( + doc.documentElement?.getElementsByTagNameNS(ASSERTION_NS, 'AuthnStatement').length, + ).toBe(1); + }); + + it('GET /api/saml/slack/sso (redirect binding, signed-in) returns auto-submit form back to Slack ACS', async () => { + const { accessToken } = await mintSessionFor(personId, 'user', JWT_KEY); + const res = await app.inject({ + method: 'GET', + url: `/api/saml/slack/sso?${redirectBindingQuery(slackAuthnRequestXml('id-redirect-3'), 'opaque-redirect-state')}`, + cookies: { cfp_session: accessToken }, + }); + expect(res.statusCode).toBe(200); + expect(res.headers['content-type']).toMatch(/text\/html/); + expect(res.body).toContain(`action="https://${SLACK_TEAM_HOST}/sso/saml"`); + expect(res.body).toContain('name="RelayState" value="opaque-redirect-state"'); + + const xml = decodeSamlResponse(res.body); + const doc = new DOMParser().parseFromString(xml, 'application/xml'); + const root = doc.documentElement!; + expect(root.localName).toBe('Response'); + expect(root.getAttribute('InResponseTo')).toBe('id-redirect-3'); + expect(root.getElementsByTagNameNS(ASSERTION_NS, 'NameID')[0]?.textContent).toBe(slug); + }); + + it('GET /api/saml/slack/sso (redirect binding) with bad ACS URL → 422', async () => { + const { accessToken } = await mintSessionFor(personId, 'user', JWT_KEY); + const res = await app.inject({ + method: 'GET', + url: `/api/saml/slack/sso?${redirectBindingQuery(slackAuthnRequestXml('id-redirect-4', 'evil.example.com'))}`, + cookies: { cfp_session: accessToken }, + }); + expect(res.statusCode).toBe(422); + expect(res.json<{ error: { code: string } }>().error.code).toBe('validation_failed'); + }); + + it('GET /api/saml/slack/sso with a non-DEFLATEd (POST-style) SAMLRequest → 422', async () => { + // Plain base64 (no DEFLATE) is the POST binding's wire form; on the + // Redirect binding it must fail to inflate rather than be accepted. + const plainB64 = Buffer.from(slackAuthnRequestXml('id-redirect-5'), 'utf8').toString('base64'); + const res = await app.inject({ + method: 'GET', + url: `/api/saml/slack/sso?${new URLSearchParams({ SAMLRequest: plainB64 }).toString()}`, + }); + expect(res.statusCode).toBe(422); + expect(res.json<{ error: { code: string } }>().error.code).toBe('validation_failed'); + }); + it('GET /api/saml/slack/launch?channel=phlask carries channel as RelayState', async () => { const { accessToken } = await mintSessionFor(personId, 'user', JWT_KEY); const res = await app.inject({ From 78ab9be303e74fc6085c79bc19a9303e05625960 Mon Sep 17 00:00:00 2001 From: Chris Alfano Date: Thu, 10 Sep 2026 02:36:02 -0400 Subject: [PATCH 8/8] chore(plans): mark saml-authn-statement done (PR #174) Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015jmpurQpBHXD9yLFEGUriR --- plans/saml-authn-statement.md | 152 ++++++++++++++++++++++++++++++++++ 1 file changed, 152 insertions(+) create mode 100644 plans/saml-authn-statement.md diff --git a/plans/saml-authn-statement.md b/plans/saml-authn-statement.md new file mode 100644 index 0000000..92d40eb --- /dev/null +++ b/plans/saml-authn-statement.md @@ -0,0 +1,152 @@ +--- +status: done +depends: [saml-idp, saml-self-host] +specs: + - specs/api/saml.md +issues: [] +pr: 174 +--- + +# Plan: SAML AuthnStatement + Redirect-binding SSO + +## Scope + +Close the two conformance gaps that surfaced while preparing to test the IdP +against the real Slack workspace: + +1. The assertion emits no ``. The Web Browser SSO profile + requires at least one and Slack sends a `RequestedAuthnContext`, so this is + the most likely rejection on first contact. +2. Slack's "Test configuration" 404s: Slack delivers the AuthnRequest over the + HTTP-Redirect binding (`GET /api/saml/slack/sso?SAMLRequest=`), + which our metadata advertises but the API only registered as `POST`. + +In: spec both behaviours, implement, test. Out: any change to NameID / +attribute shape (frozen by [`saml-idp`](saml-idp.md)); echoing the SP's +`RequestedAuthnContext` (deliberately not — see spec); signed AuthnRequests +(Slack doesn't sign; `wantAuthnRequestsSigned` stays false); the still-open +end-to-end verification against the live workspace (#51). + +## Implements + +- [api/saml.md](../specs/api/saml.md) — `## Identity assertion` → + `### Authentication statement` (AuthnInstant, SessionIndex, fixed + `AuthnContextClassRef`, `SessionNotOnOrAfter` omitted); the endpoints table + and `## GET | POST /api/saml/slack/sso` (both bindings, one flow). + +## Approach + +- **Legacy check first.** Read the Emergence SAML2 connector laddr used + (`JarvusInnovations/emergence-saml2` → `Emergence\SAML2\Connector`) to see + what it emitted: `setSessionIndex(generateId())` and + `setAuthnContext(SAML2_Constants::AC_PASSWORD)` on a simplesamlphp + `Assertion`, whose `AuthnInstant` defaults to construction time and whose + `SessionNotOnOrAfter` is left null. `AC_PASSWORD` is + `urn:oasis:names:tc:SAML:2.0:ac:classes:Password` (confirmed in + simplesamlphp/saml2 `Constants.php`). Match that class rather than the + brief's initial `PasswordProtectedTransport` — continuity with what the + workspace already accepted beats the more precise label. +- **Template.** Add the AuthnStatement literally to `loginResponseTemplate.context` + in `apps/api/src/saml/config.ts`, after `` and before + `{AttributeStatement}`. samlify's built-in template has a bare + `{AuthnStatement}` slot that its default path blanks; we're on the + `customTagReplacement` path anyway, so we own the markup and fill three + placeholders of our own: `{AuthnInstant}`, `{SessionIndex}`, + `{AuthnContextClassRef}`. +- **Substitutions.** `buildResponseSubstitutions` gains a `sessionIndex` input + and emits the three tags; `AuthnInstant` reuses the same `now` as + `IssueInstant`. The route's `buildCustomTagReplacement` mints the + SessionIndex from the same `generateID` as the Response/Assertion IDs so it + is fresh and distinct. Signing happens after the callback returns, so the + statement is inside the signed subtree with no further work. +- **Redirect binding.** Register `GET /api/saml/slack/sso`. Instead of samlify's + `parseLoginRequest(sp, 'redirect', { query })` plus a binding-aware resume + cookie, inflate at the edge (`inflateRawSync` → re-base64) into the plain + form the POST binding carries, then run both methods through a shared + `handleSpInitiatedSso`. samlify's redirect flow is exactly that inflate + followed by the same parser (checked in `flow.js`), and the resume cookie's + `samlRequest` claim keeps one shape. Fold `' '` back to `'+'` in the query + value before decoding — a sender that leaves base64 `+` unescaped has it + URL-decoded to a space. +- **Tests** in `apps/api/tests/saml.test.ts`: AuthnStatement position, fixed + ClassRef, `AuthnInstant` ISO and `<= IssueInstant`, SessionIndex distinct + from the assertion ID, no `SessionNotOnOrAfter`; cryptographic signature + verification via `samlify.SamlLib.verifySignature` against the metadata + endpoint's cert, plus a tamper case; Redirect binding for signed-in, + anonymous → `/sso/resume`, bad ACS, and non-DEFLATEd payload. + +## Validation + +- [x] Every issued assertion contains exactly one `AuthnStatement`, after + `Conditions` and before `AttributeStatement` +- [x] `AuthnContextClassRef` is `urn:oasis:names:tc:SAML:2.0:ac:classes:Password` + on both IdP-initiated and SP-initiated responses +- [x] `AuthnInstant` parses as ISO-8601 and is `<=` the assertion `IssueInstant` +- [x] `SessionIndex` present, opaque, and not equal to the assertion `ID`; + `SessionNotOnOrAfter` absent +- [x] Assertion signature verifies against the metadata cert with the + AuthnStatement present; altering the ClassRef invalidates it +- [x] `GET /api/saml/slack/sso` with a DEFLATEd `SAMLRequest`: signed-in → 200 + auto-submit form with `InResponseTo` + `RelayState`; anonymous → resume + cookie + 302 `/login`, and `/sso/resume` completes the flow +- [x] `GET /api/saml/slack/sso` with a bad ACS → 422; with a non-DEFLATEd + payload → 422 +- [x] Existing POST-binding tests unchanged and green +- [x] `type-check` + `lint` clean; api suite green (saml.test.ts 18/18; full + api suite 440/440, no fixture-clone flake this run) +- [ ] Slack "Test configuration" passes against the live workspace + +## Risks / unknowns + +- **Slack rejecting `Password` vs its requested `PasswordProtectedTransport`.** + Slack's `RequestedAuthnContext` uses `Comparison="exact"` by default, but the + legacy connector asserted `Password` against this workspace for years, so the + check is evidently lenient. If the live test rejects on context, the fix is a + one-line constant change plus a spec edit — not a redesign. +- **Redirect-binding query decoding.** Fastify's query parser URL-decodes once; + a compliant sender percent-encodes `+`, `/`, `=` and we get the base64 back + intact. The space-fold covers the one common non-compliance. A sender that + double-encodes would still fail to inflate → 422 with a log line. +- **Fixture-clone flake (#171)** on the full api suite is unrelated; re-run the + file alone if it trips. + +## Notes + +- **Live-workspace criterion left unchecked.** Requires a Slack admin to run + "Test configuration" against a deployed build; the code fix for the 404 it + hit is here, but the pass itself can only be observed in sandbox/prod. Closes + out under #51 with the rest of the e2e verification. +- **`Password`, not `PasswordProtectedTransport`.** The brief opened with + `PasswordProtectedTransport` (Slack's default request). Reading the legacy + connector changed the call: laddr asserted `AC_PASSWORD` against this exact + workspace with no rejections, so matching it is the lower-risk choice and is + now the spec'd value. The class is a single exported constant + (`SLACK_AUTHN_CONTEXT_CLASS_REF`) if the live test disagrees. +- **Legacy `NotBefore` skew.** The legacy connector set `Conditions/@NotBefore` + to `time() - 30`; ours is `now`. Not changed here — samlify's own default is + `now`, there's no evidence Slack has trouble with it, and it's out of this + plan's scope. Worth remembering if the live test reports a `NotBefore` + clock-skew failure. +- **samlify template ownership.** samlify's built-in login-response template + has a `{AuthnStatement}` slot, but the default (non-custom) substitution + blanks it and we're on the `customTagReplacement` path regardless, so the + statement lives literally in our template with our own placeholder names. + Do not expect samlify to inject one for you. +- **Redirect-binding normalisation.** `inflateRedirectBindingRequest` folds + the GET wire form into the POST wire form at the edge; downstream code and + the resume cookie never learn which binding carried the request. samlify's + `'redirect'` parser was checked (`flow.js` → `inflateRawSync` then the same + parse) and deliberately not used so the cookie stays binding-agnostic. +- **Cryptographic signature check in tests.** `samlify.SamlLib.verifySignature` + with `IdPMetadata()` verifies against the exact cert + the metadata advertises and returns the signed subtree, which is a cheap way + to assert "X is inside the signature" — reusable for future assertion-shape + changes. + +## Follow-ups + +- Tracked as: live Slack "Test configuration" + member sign-in against the + deployed build — issue [#51](https://github.com/CodeForPhilly/codeforphilly-ng/issues/51) + (e2e verification against a real workspace) already covers it; this plan + adds the AuthnStatement/ClassRef and Redirect-binding outcomes to what that + run should confirm.