feat(helm): postgres control-plane profile — ADR 0010 slice 6 (E4)

docs/clickhouse-cutover-plan.md

@@ -102,30 +102,68 @@ live-verified). Do not do this while backend is DuckDB or while
 render** until both halves of the gate are set, so this phase cannot be
 executed accidentally.
 
-- [ ] Prerequisite: ADR 0010 slices 1–5 merged (`PostgresControlPlaneStore`
+- [x] Prerequisite: ADR 0010 slices 1–5 merged (`PostgresControlPlaneStore`
       exists, live-verified); slice 6 extends the values schema so
-      `controlPlane.store=postgres` renders.
-- [ ] `helm/agentflow/values.yaml` — set serving to ClickHouse (drop
-      `config.duckdbPath` from the request path; point at the ClickHouse service
-      via `CLICKHOUSE_HOST` etc. / `SERVING_BACKEND=clickhouse`) and set
-      `controlPlane.store=postgres` + DSN secret. `usageDbPath` retires with
-      ADR 0010 slice 4 (usage moves into the control-plane store).
-- [ ] Remove the request-path write PVC dependency; confirm pods are stateless
-      (with the control plane external there is no writable volume left on the
-      request path).
-- [ ] Enable `autoscaling` with sane `minReplicas`/`maxReplicas` and an HPA CPU
-      target; remove the `replicaCount: 1` hard-pin.
+      `controlPlane.store=postgres` renders. *(Done 2026-07-04, slice 6: the
+      `store` enum now admits `postgres`, `controlPlane.postgres.{existingSecret,dsnKey}`
+      carries the DSN, and `templates/deployment.yaml` wires
+      `AGENTFLOW_CONTROLPLANE_STORE` + `AGENTFLOW_CONTROLPLANE_PG_DSN` — E4.)*
+- [x] Chart profile — the scale profile is now expressible **as an overlay**,
+      not by editing the shipped `values.yaml` (whose default stays the
+      zero-dependency DuckDB/embedded demo): `k8s/staging/values-staging-scale.yaml.example`
+      sets `serving.backend=clickhouse` (+ `CLICKHOUSE_HOST` etc.),
+      `controlPlane.store=postgres` + DSN secret, and drops the DuckDB request
+      path. `usageDbPath` retires with ADR 0010 slice 4 (usage already in the
+      store). *(Done 2026-07-04, E4.)*
+- [x] Request-path PVC dependency removed in the scale profile: the overlay
+      sets `persistence.enabled=false`, so with the control plane external there
+      is no writable volume left on the request path (ADR 0007 "stateless pods"
+      becomes literally true). *(Done 2026-07-04, E4.)*
+- [x] `autoscaling` renders an HPA in the scale profile (min/max + CPU target);
+      the chart carries no `replicaCount: 1` hard-pin — the default is 1 but any
+      value is admissible once both gate halves are set. *(Done 2026-07-04, E4:
+      `test_full_scale_profile_admits_autoscaling_hpa`.)*
 - [x] Add a `values.schema.json` / comment note: `autoscaling` requires an
       external serving engine. *(Done 2026-07-02, strengthened beyond a note:
-      the schema pins `controlPlane.store` to `embedded` until the adapter
-      ships, and `templates/deployment.yaml` fails any multi-replica render
+      the schema pinned `controlPlane.store` to `embedded` until the adapter
+      shipped, and `templates/deployment.yaml` fails any multi-replica render
       unless `controlPlane.store=postgres` and `serving.backend=clickhouse` —
-      see ADR 0010.)*
-- [ ] Verify: `scripts/k8s_staging_up.sh` on kind, `k8s_smoke_test.sh` green,
-      `replicaCount: 2` schedules without PVC contention; **plus the ADR 0010
-      replica-correctness checks** — exactly one delivery per (webhook, event)
-      across two pods, one alert page per incident, a webhook registered via
-      either pod visible to both.
+      see ADR 0010. Slice 6 (2026-07-04) released the enum ratchet to
+      `[embedded, postgres]`.)*
+- [ ] **LIVE verify (kind, Docker — pending Mac/CI, E4 tail):**
+      `scripts/k8s_staging_up.sh` on kind with the scale overlay,
+      `k8s_smoke_test.sh` green, `replicaCount: 2` schedules without PVC
+      contention; **plus the ADR 0010 replica-correctness checks** — exactly one
+      delivery per (webhook, event) across two pods, one alert page per incident,
+      a webhook registered via either pod visible to both. See the recipe below.
+      *(Chart-side render verified locally 2026-07-04 via `helm template`/`lint`;
+      the two-real-pods run needs Docker, unavailable on the authoring host.)*
+
+### Phase 3 replica-correctness verify recipe (ADR 0010 slice 6)
+
+Bring up the scale stand (kind + in-cluster PostgreSQL + ClickHouse + Redis and
+their secrets), install the chart with the scale overlay, then:
+
+1. **>=2 pods on the postgres store + cross-pod registration visibility** —
+   automated by `scripts/k8s_replica_correctness_verify.sh`: asserts the
+   deployment runs ≥2 ready pods all carrying `AGENTFLOW_CONTROLPLANE_STORE=postgres`,
+   registers one webhook through the Service, and confirms it is visible on
+   every round-robin read (on the embedded YAML store a read served by the pod
+   that did not register would miss it — the sharpest split-brain, class 5).
+2. **Exactly-one delivery per (webhook, event)** — emit one pipeline event
+   matching a registered webhook's filter; read `GET /v1/webhooks/{id}/logs`;
+   assert exactly one delivery record for that `event_id` despite both pods
+   scanning the event (idempotent enqueue insert-win).
+3. **One alert page per incident** — configure an alert rule, drive one
+   triggering evaluation window; assert a single page and one `alert_history`
+   transition, not one per pod (per-rule `claim_alert_tick` single-flight).
+
+Checks 2–3 need event/alert emission plus a capture sink, so they are part of
+the live run; the **store-level guarantee** behind them (idempotent enqueue,
+single-flight tick, outbox↔dead-letter atomicity) is already live-verified by
+the slice-5 standalone-PostgreSQL probe suite (31/31,
+`docs/perf/control-plane-pg-verify-2026-07-03.md`) — Phase 3 adds only the
+two-real-pods topology proof.
 
 ## Phase 4 — NL→SQL (routes through GraceKelly)
 

docs/decisions/0010-control-plane-externalization-postgres.md

@@ -202,6 +202,19 @@ ClickHouse is consumed), and `psycopg` joins the optional dependencies.
    plan Phase 3 execution: kind staging at `replicaCount=2`, verifying
    exactly-one delivery per (webhook, event) with two pods, one alert page
    per incident, and a webhook registered through either pod visible to both.
+   Chart wiring executed 2026-07-04: the `store` enum released to
+   `[embedded, postgres]`, `controlPlane.postgres.{existingSecret,dsnKey}`
+   sources the DSN, and `templates/deployment.yaml` wires
+   `AGENTFLOW_CONTROLPLANE_STORE` + `AGENTFLOW_CONTROLPLANE_PG_DSN`; the scale
+   profile is an overlay (`k8s/staging/values-staging-scale.yaml.example`), not
+   a change to the zero-dependency default. Render side verified locally
+   (`helm template`/`lint` + `tests/unit/test_helm_values_contract.py`). The
+   **two-real-pods live run** (kind at `replicaCount=2` + the replica-correctness
+   checks above) needs Docker and is the deferred Mac/CI tail —
+   `scripts/k8s_replica_correctness_verify.sh` automates the pod-count and
+   cross-pod registration-visibility checks; the delivery/alert checks follow
+   the recipe in the cutover plan (their store-level guarantee is already
+   live-verified by slice 5's 31/31 PG probes).
 
 ## Consequences
 

docs/helm-deployment.md

@@ -106,14 +106,51 @@ If `ingress.enabled=true`, verify the configured host instead of using port-forw
 - Default `secrets.apiKeys.keys` is empty. Supply API-key config through `secrets.existingSecret` or through an environment-specific values file; do not reuse repository defaults as runtime credentials.
 - If `secrets.create=false`, `secrets.existingSecret` must name a Kubernetes Secret with `admin-key` and `api_keys.yaml`.
 - `config.tenants` is the source of truth for tenant routing and API version pinning.
-- `autoscaling.enabled=true` creates an HPA from `minReplicas` to `maxReplicas`, but persistent DuckDB storage is guarded to one writer replica. Rendering fails when `persistence.enabled=true` and the chart is configured for more than one API writer replica.
+- `autoscaling.enabled=true` creates an HPA from `minReplicas` to `maxReplicas`, but persistent DuckDB storage is guarded to one writer replica. Rendering fails when `persistence.enabled=true` and the chart is configured for more than one API writer replica. Multi-replica also requires the external control-plane store — see [Horizontal scaling](#horizontal-scaling-postgres-control-plane-profile).
 - `ingress.tls` accepts the standard Helm ingress TLS structure.
 - ConfigMap and Secret checksums are injected into the pod template, so `helm upgrade` rolls the deployment when mounted config changes.
 - DuckDB is still a stateful local file. If your storage class only supports `ReadWriteOnce`, start with `replicaCount: 1` until you validate your storage and concurrency model.
 - Optional DuckDB file encryption is runtime-configured with `AGENTFLOW_DUCKDB_ENCRYPTION_KEY` or `AGENTFLOW_DUCKDB_ENCRYPTION_KEY_FILE`; use `extraEnv` with a `secretKeyRef` to supply the key. The default remains unencrypted for backward compatibility.
 - DuckDB encryption is a local at-rest hardening option only. It is not a NIST, GDPR, HIPAA, SOC 2, or external-compliance attestation by itself.
 - Optional append-only audit export is runtime-configured with `AGENTFLOW_AUDIT_LOG_PATH`, which writes a hash-chained JSONL file in addition to DuckDB usage analytics. For externally immutable retention, operators still need object-lock or SIEM evidence outside this chart.
 
+## Horizontal scaling (postgres control-plane profile)
+
+The chart default is the single-replica, zero-dependency demo (DuckDB serving,
+embedded per-pod control plane). Scaling the API horizontally needs **both**
+halves of the ADR 0009/0010 gate, enforced at render time — the chart fails any
+multi-replica render unless both are set:
+
+1. **External serving engine** — `serving.backend=clickhouse` (+ `serving.clickhouse.host`
+   and a password `existingSecret`). ADR 0006/0007.
+2. **External control-plane store** — `controlPlane.store=postgres` with a DSN
+   in `controlPlane.postgres.existingSecret` (key `controlPlane.postgres.dsnKey`,
+   default `controlplane-pg-dsn`). ADR 0009/0010. This moves the webhook
+   queue/registrations, alert rules+history, outbox/dead-letter and usage out of
+   the per-pod DuckDB/YAML into one shared store, so N replicas no longer fork
+   that state (duplicate deliveries, split alert history).
+
+The chart ships **no** PostgreSQL and **no** ClickHouse service, exactly as it
+consumes an external ClickHouse: the operator provides both and the referenced
+secrets. The DSN carries a password, so — like the ClickHouse password — it is
+never inlined into values, only referenced via `existingSecret`.
+
+`k8s/staging/values-staging-scale.yaml.example` is a ready overlay:
+
+```bash
+helm upgrade --install agentflow ./helm/agentflow \
+  -f k8s/staging/values-staging.yaml \
+  -f k8s/staging/values-staging-scale.yaml \
+  --namespace agentflow
+```
+
+Replica-correctness (exactly-one delivery per (webhook, event), one alert page
+per incident, cross-pod registration visibility) is verified by
+`scripts/k8s_replica_correctness_verify.sh` plus the recipe in
+`docs/clickhouse-cutover-plan.md` (Phase 3). With the postgres store the render
+gate relaxes automatically; with `controlPlane.store=embedded` any `replicaCount
+> 1` (or `autoscaling.maxReplicas > 1`) render is refused.
+
 ## Contract Maintenance
 
 - `helm/agentflow/values.schema.json` is the chart contract for runtime values consumed from Helm.

helm/agentflow/templates/deployment.yaml

@@ -85,6 +85,20 @@ spec:
                   key: {{ .Values.serving.clickhouse.passwordKey | quote }}
             {{- end }}
             {{- end }}
+            # Control-plane store (ADR 0010). Set explicitly (mirrors
+            # SERVING_BACKEND) so the rendered manifest is self-documenting; the
+            # app treats an unset value as 'embedded'. The postgres profile also
+            # sources its DSN from an operator-provided secret — the chart ships
+            # no PostgreSQL service, exactly as it ships no ClickHouse.
+            - name: AGENTFLOW_CONTROLPLANE_STORE
+              value: {{ .Values.controlPlane.store | quote }}
+            {{- if eq .Values.controlPlane.store "postgres" }}
+            - name: AGENTFLOW_CONTROLPLANE_PG_DSN
+              valueFrom:
+                secretKeyRef:
+                  name: {{ required "controlPlane.postgres.existingSecret is required when controlPlane.store=postgres (it must hold the PostgreSQL DSN; the chart ships no PG service)" .Values.controlPlane.postgres.existingSecret | quote }}
+                  key: {{ .Values.controlPlane.postgres.dsnKey | quote }}
+            {{- end }}
             - name: DUCKDB_PATH
               value: {{ .Values.config.duckdbPath | quote }}
             - name: AGENTFLOW_USAGE_DB_PATH

helm/agentflow/values.schema.json

@@ -446,8 +446,25 @@
       "properties": {
         "store": {
           "type": "string",
-          "enum": ["embedded"],
-          "description": "Where control-plane state (webhook queue/log, alert rules+history, outbox, dead-letter, usage) lives. 'embedded' = per-pod DuckDB + config files, single replica only. 'postgres' joins this enum only when the PostgresControlPlaneStore adapter ships (ADR 0010 rollout) — the schema must not advertise a profile the app cannot run."
+          "enum": ["embedded", "postgres"],
+          "description": "Where control-plane state (webhook queue/log, alert rules+history, outbox, dead-letter, usage) lives. 'embedded' = per-pod DuckDB + config files, single replica only (the default profile). 'postgres' = the shared PostgresControlPlaneStore adapter (ADR 0010), required for multi-replica: it needs controlPlane.postgres.existingSecret to hold the DSN and serving.backend=clickhouse. The adapter shipped in ADR 0010 rollout slice 5 and the chart profile in slice 6, so the render gate now admits multi-replica once both halves are set."
+        },
+        "postgres": {
+          "type": "object",
+          "additionalProperties": false,
+          "required": ["existingSecret", "dsnKey"],
+          "description": "Scale-profile control-plane store connection (used only when store=postgres). The chart never ships a PostgreSQL service (mirroring how ClickHouse is consumed, ADR 0010): the operator provides the DSN in a Secret and references it here.",
+          "properties": {
+            "existingSecret": {
+              "type": "string",
+              "description": "Name of an operator-managed Secret holding the PostgreSQL DSN. Required when store=postgres; the DSN carries a password, so like the ClickHouse password it is never inlined into values."
+            },
+            "dsnKey": {
+              "type": "string",
+              "minLength": 1,
+              "description": "Key inside existingSecret whose value is the full PostgreSQL DSN (e.g. postgresql://user:pass@host:5432/agentflow). Wired to AGENTFLOW_CONTROLPLANE_PG_DSN."
+            }
+          }
         }
       }
     },

helm/agentflow/values.yaml

@@ -138,14 +138,21 @@ serving:
 # alert rules and history, outbox, dead-letter and usage accounting.
 # 'embedded' = per-pod DuckDB + config files — correct for the single-replica
 # profile, a split-brain at replicaCount > 1 (duplicate deliveries, forked
-# alert state). The PostgresControlPlaneStore adapter shipped with ADR 0010
-# rollout slice 5 (app-side: AGENTFLOW_CONTROLPLANE_STORE=postgres +
-# AGENTFLOW_CONTROLPLANE_PG_DSN); the chart profile for it (env + secret
-# wiring, schema enum extension) lands with rollout slice 6 — until then the
-# values schema pins store=embedded (fail-closed ratchet) and the chart
-# refuses any multi-replica render.
+# alert state). 'postgres' = the shared PostgresControlPlaneStore adapter
+# (ADR 0010): the app-side half shipped in rollout slice 5
+# (AGENTFLOW_CONTROLPLANE_STORE=postgres + AGENTFLOW_CONTROLPLANE_PG_DSN) and
+# the chart profile below in slice 6. To scale horizontally set
+# store: postgres + postgres.existingSecret (DSN) AND serving.backend:
+# clickhouse — templates/deployment.yaml fails any multi-replica render unless
+# BOTH halves of the gate are set, so the gate relaxes automatically and only
+# when the store that makes scaling correct exists. The chart never ships a
+# PostgreSQL service (mirroring ClickHouse) — the operator provides the DSN in
+# a Secret referenced by postgres.existingSecret / postgres.dsnKey.
 controlPlane:
   store: embedded
+  postgres:
+    existingSecret: ""
+    dsnKey: controlplane-pg-dsn
 
 config:
   duckdbPath: /data/agentflow.duckdb

k8s/staging/values-staging-scale.yaml.example

@@ -0,0 +1,53 @@
+# Scale-profile staging overlay (ADR 0010 rollout slice 6 / cutover plan
+# Phase 3). Layer this ON TOP of values-staging.yaml:
+#
+#   helm upgrade --install agentflow helm/agentflow \
+#     -f k8s/staging/values-staging.yaml \
+#     -f k8s/staging/values-staging-scale.yaml \
+#     --namespace agentflow
+#
+# It flips BOTH halves of the render gate (external serving engine + external
+# control-plane store) so replicaCount=2 is admissible, and exercises the
+# multi-replica correctness the ADR pins. The chart ships NO PostgreSQL and NO
+# ClickHouse service — the operator (or the kind bring-up script) provides both
+# and the secrets referenced below. This file is an .example: copy it to
+# values-staging-scale.yaml and fill in your real secret names. It carries no
+# credentials itself — only Secret *names* and hostnames — so it is safe to
+# commit (the DSN and ClickHouse password live in the referenced Secrets).
+#
+# Prerequisites in-cluster (see scripts/k8s_replica_correctness_verify.sh):
+#   - a PostgreSQL reachable at the DSN in secret `agentflow-controlplane-pg`
+#     (key controlplane-pg-dsn), schema initialised by the app on boot
+#   - a ClickHouse reachable at serving.clickhouse.host, password in secret
+#     `agentflow-clickhouse` (key clickhouse-password)
+#   - Redis (already stood up by k8s_staging_up.sh)
+
+# Two API pods behind the Service — the split-brain surface the control-plane
+# externalisation exists to make correct.
+replicaCount: 2
+
+# The DuckDB single-writer PVC must be gone on the request path: the control
+# plane now lives in PostgreSQL and serving reads go to ClickHouse, so the pod
+# is stateless (ADR 0007 "stateless pods" becomes literally true here).
+persistence:
+  enabled: false
+
+# External serving engine — the ADR 0006/0007 half of the gate.
+serving:
+  backend: clickhouse
+  clickhouse:
+    host: agentflow-clickhouse
+    port: 8123
+    database: agentflow
+    user: default
+    existingSecret: agentflow-clickhouse
+    passwordKey: clickhouse-password
+
+# External control-plane store — the ADR 0009/0010 half of the gate. The DSN
+# is operator-provided in a Secret (never inlined; it carries a password, same
+# posture as the ClickHouse password above).
+controlPlane:
+  store: postgres
+  postgres:
+    existingSecret: agentflow-controlplane-pg
+    dsnKey: controlplane-pg-dsn