diff --git a/docs/clickhouse-cutover-plan.md b/docs/clickhouse-cutover-plan.md index a691933a..65f09725 100644 --- a/docs/clickhouse-cutover-plan.md +++ b/docs/clickhouse-cutover-plan.md @@ -102,30 +102,68 @@ live-verified). Do not do this while backend is DuckDB or while render** until both halves of the gate are set, so this phase cannot be executed accidentally. -- [ ] Prerequisite: ADR 0010 slices 1–5 merged (`PostgresControlPlaneStore` +- [x] Prerequisite: ADR 0010 slices 1–5 merged (`PostgresControlPlaneStore` exists, live-verified); slice 6 extends the values schema so - `controlPlane.store=postgres` renders. -- [ ] `helm/agentflow/values.yaml` — set serving to ClickHouse (drop - `config.duckdbPath` from the request path; point at the ClickHouse service - via `CLICKHOUSE_HOST` etc. / `SERVING_BACKEND=clickhouse`) and set - `controlPlane.store=postgres` + DSN secret. `usageDbPath` retires with - ADR 0010 slice 4 (usage moves into the control-plane store). -- [ ] Remove the request-path write PVC dependency; confirm pods are stateless - (with the control plane external there is no writable volume left on the - request path). -- [ ] Enable `autoscaling` with sane `minReplicas`/`maxReplicas` and an HPA CPU - target; remove the `replicaCount: 1` hard-pin. + `controlPlane.store=postgres` renders. *(Done 2026-07-04, slice 6: the + `store` enum now admits `postgres`, `controlPlane.postgres.{existingSecret,dsnKey}` + carries the DSN, and `templates/deployment.yaml` wires + `AGENTFLOW_CONTROLPLANE_STORE` + `AGENTFLOW_CONTROLPLANE_PG_DSN` — E4.)* +- [x] Chart profile — the scale profile is now expressible **as an overlay**, + not by editing the shipped `values.yaml` (whose default stays the + zero-dependency DuckDB/embedded demo): `k8s/staging/values-staging-scale.yaml.example` + sets `serving.backend=clickhouse` (+ `CLICKHOUSE_HOST` etc.), + `controlPlane.store=postgres` + DSN secret, and drops the DuckDB request + path. `usageDbPath` retires with ADR 0010 slice 4 (usage already in the + store). *(Done 2026-07-04, E4.)* +- [x] Request-path PVC dependency removed in the scale profile: the overlay + sets `persistence.enabled=false`, so with the control plane external there + is no writable volume left on the request path (ADR 0007 "stateless pods" + becomes literally true). *(Done 2026-07-04, E4.)* +- [x] `autoscaling` renders an HPA in the scale profile (min/max + CPU target); + the chart carries no `replicaCount: 1` hard-pin — the default is 1 but any + value is admissible once both gate halves are set. *(Done 2026-07-04, E4: + `test_full_scale_profile_admits_autoscaling_hpa`.)* - [x] Add a `values.schema.json` / comment note: `autoscaling` requires an external serving engine. *(Done 2026-07-02, strengthened beyond a note: - the schema pins `controlPlane.store` to `embedded` until the adapter - ships, and `templates/deployment.yaml` fails any multi-replica render + the schema pinned `controlPlane.store` to `embedded` until the adapter + shipped, and `templates/deployment.yaml` fails any multi-replica render unless `controlPlane.store=postgres` and `serving.backend=clickhouse` — - see ADR 0010.)* -- [ ] Verify: `scripts/k8s_staging_up.sh` on kind, `k8s_smoke_test.sh` green, - `replicaCount: 2` schedules without PVC contention; **plus the ADR 0010 - replica-correctness checks** — exactly one delivery per (webhook, event) - across two pods, one alert page per incident, a webhook registered via - either pod visible to both. + see ADR 0010. Slice 6 (2026-07-04) released the enum ratchet to + `[embedded, postgres]`.)* +- [ ] **LIVE verify (kind, Docker — pending Mac/CI, E4 tail):** + `scripts/k8s_staging_up.sh` on kind with the scale overlay, + `k8s_smoke_test.sh` green, `replicaCount: 2` schedules without PVC + contention; **plus the ADR 0010 replica-correctness checks** — exactly one + delivery per (webhook, event) across two pods, one alert page per incident, + a webhook registered via either pod visible to both. See the recipe below. + *(Chart-side render verified locally 2026-07-04 via `helm template`/`lint`; + the two-real-pods run needs Docker, unavailable on the authoring host.)* + +### Phase 3 replica-correctness verify recipe (ADR 0010 slice 6) + +Bring up the scale stand (kind + in-cluster PostgreSQL + ClickHouse + Redis and +their secrets), install the chart with the scale overlay, then: + +1. **>=2 pods on the postgres store + cross-pod registration visibility** — + automated by `scripts/k8s_replica_correctness_verify.sh`: asserts the + deployment runs ≥2 ready pods all carrying `AGENTFLOW_CONTROLPLANE_STORE=postgres`, + registers one webhook through the Service, and confirms it is visible on + every round-robin read (on the embedded YAML store a read served by the pod + that did not register would miss it — the sharpest split-brain, class 5). +2. **Exactly-one delivery per (webhook, event)** — emit one pipeline event + matching a registered webhook's filter; read `GET /v1/webhooks/{id}/logs`; + assert exactly one delivery record for that `event_id` despite both pods + scanning the event (idempotent enqueue insert-win). +3. **One alert page per incident** — configure an alert rule, drive one + triggering evaluation window; assert a single page and one `alert_history` + transition, not one per pod (per-rule `claim_alert_tick` single-flight). + +Checks 2–3 need event/alert emission plus a capture sink, so they are part of +the live run; the **store-level guarantee** behind them (idempotent enqueue, +single-flight tick, outbox↔dead-letter atomicity) is already live-verified by +the slice-5 standalone-PostgreSQL probe suite (31/31, +`docs/perf/control-plane-pg-verify-2026-07-03.md`) — Phase 3 adds only the +two-real-pods topology proof. ## Phase 4 — NL→SQL (routes through GraceKelly) diff --git a/docs/decisions/0010-control-plane-externalization-postgres.md b/docs/decisions/0010-control-plane-externalization-postgres.md index 9a57eb5f..c17e9df2 100644 --- a/docs/decisions/0010-control-plane-externalization-postgres.md +++ b/docs/decisions/0010-control-plane-externalization-postgres.md @@ -202,6 +202,19 @@ ClickHouse is consumed), and `psycopg` joins the optional dependencies. plan Phase 3 execution: kind staging at `replicaCount=2`, verifying exactly-one delivery per (webhook, event) with two pods, one alert page per incident, and a webhook registered through either pod visible to both. + Chart wiring executed 2026-07-04: the `store` enum released to + `[embedded, postgres]`, `controlPlane.postgres.{existingSecret,dsnKey}` + sources the DSN, and `templates/deployment.yaml` wires + `AGENTFLOW_CONTROLPLANE_STORE` + `AGENTFLOW_CONTROLPLANE_PG_DSN`; the scale + profile is an overlay (`k8s/staging/values-staging-scale.yaml.example`), not + a change to the zero-dependency default. Render side verified locally + (`helm template`/`lint` + `tests/unit/test_helm_values_contract.py`). The + **two-real-pods live run** (kind at `replicaCount=2` + the replica-correctness + checks above) needs Docker and is the deferred Mac/CI tail — + `scripts/k8s_replica_correctness_verify.sh` automates the pod-count and + cross-pod registration-visibility checks; the delivery/alert checks follow + the recipe in the cutover plan (their store-level guarantee is already + live-verified by slice 5's 31/31 PG probes). ## Consequences diff --git a/docs/helm-deployment.md b/docs/helm-deployment.md index bbc2acee..b40431ff 100644 --- a/docs/helm-deployment.md +++ b/docs/helm-deployment.md @@ -106,7 +106,7 @@ If `ingress.enabled=true`, verify the configured host instead of using port-forw - Default `secrets.apiKeys.keys` is empty. Supply API-key config through `secrets.existingSecret` or through an environment-specific values file; do not reuse repository defaults as runtime credentials. - If `secrets.create=false`, `secrets.existingSecret` must name a Kubernetes Secret with `admin-key` and `api_keys.yaml`. - `config.tenants` is the source of truth for tenant routing and API version pinning. -- `autoscaling.enabled=true` creates an HPA from `minReplicas` to `maxReplicas`, but persistent DuckDB storage is guarded to one writer replica. Rendering fails when `persistence.enabled=true` and the chart is configured for more than one API writer replica. +- `autoscaling.enabled=true` creates an HPA from `minReplicas` to `maxReplicas`, but persistent DuckDB storage is guarded to one writer replica. Rendering fails when `persistence.enabled=true` and the chart is configured for more than one API writer replica. Multi-replica also requires the external control-plane store — see [Horizontal scaling](#horizontal-scaling-postgres-control-plane-profile). - `ingress.tls` accepts the standard Helm ingress TLS structure. - ConfigMap and Secret checksums are injected into the pod template, so `helm upgrade` rolls the deployment when mounted config changes. - DuckDB is still a stateful local file. If your storage class only supports `ReadWriteOnce`, start with `replicaCount: 1` until you validate your storage and concurrency model. @@ -114,6 +114,43 @@ If `ingress.enabled=true`, verify the configured host instead of using port-forw - DuckDB encryption is a local at-rest hardening option only. It is not a NIST, GDPR, HIPAA, SOC 2, or external-compliance attestation by itself. - Optional append-only audit export is runtime-configured with `AGENTFLOW_AUDIT_LOG_PATH`, which writes a hash-chained JSONL file in addition to DuckDB usage analytics. For externally immutable retention, operators still need object-lock or SIEM evidence outside this chart. +## Horizontal scaling (postgres control-plane profile) + +The chart default is the single-replica, zero-dependency demo (DuckDB serving, +embedded per-pod control plane). Scaling the API horizontally needs **both** +halves of the ADR 0009/0010 gate, enforced at render time — the chart fails any +multi-replica render unless both are set: + +1. **External serving engine** — `serving.backend=clickhouse` (+ `serving.clickhouse.host` + and a password `existingSecret`). ADR 0006/0007. +2. **External control-plane store** — `controlPlane.store=postgres` with a DSN + in `controlPlane.postgres.existingSecret` (key `controlPlane.postgres.dsnKey`, + default `controlplane-pg-dsn`). ADR 0009/0010. This moves the webhook + queue/registrations, alert rules+history, outbox/dead-letter and usage out of + the per-pod DuckDB/YAML into one shared store, so N replicas no longer fork + that state (duplicate deliveries, split alert history). + +The chart ships **no** PostgreSQL and **no** ClickHouse service, exactly as it +consumes an external ClickHouse: the operator provides both and the referenced +secrets. The DSN carries a password, so — like the ClickHouse password — it is +never inlined into values, only referenced via `existingSecret`. + +`k8s/staging/values-staging-scale.yaml.example` is a ready overlay: + +```bash +helm upgrade --install agentflow ./helm/agentflow \ + -f k8s/staging/values-staging.yaml \ + -f k8s/staging/values-staging-scale.yaml \ + --namespace agentflow +``` + +Replica-correctness (exactly-one delivery per (webhook, event), one alert page +per incident, cross-pod registration visibility) is verified by +`scripts/k8s_replica_correctness_verify.sh` plus the recipe in +`docs/clickhouse-cutover-plan.md` (Phase 3). With the postgres store the render +gate relaxes automatically; with `controlPlane.store=embedded` any `replicaCount +> 1` (or `autoscaling.maxReplicas > 1`) render is refused. + ## Contract Maintenance - `helm/agentflow/values.schema.json` is the chart contract for runtime values consumed from Helm. diff --git a/helm/agentflow/templates/deployment.yaml b/helm/agentflow/templates/deployment.yaml index 39c863a1..2c9b5460 100644 --- a/helm/agentflow/templates/deployment.yaml +++ b/helm/agentflow/templates/deployment.yaml @@ -85,6 +85,20 @@ spec: key: {{ .Values.serving.clickhouse.passwordKey | quote }} {{- end }} {{- end }} + # Control-plane store (ADR 0010). Set explicitly (mirrors + # SERVING_BACKEND) so the rendered manifest is self-documenting; the + # app treats an unset value as 'embedded'. The postgres profile also + # sources its DSN from an operator-provided secret — the chart ships + # no PostgreSQL service, exactly as it ships no ClickHouse. + - name: AGENTFLOW_CONTROLPLANE_STORE + value: {{ .Values.controlPlane.store | quote }} + {{- if eq .Values.controlPlane.store "postgres" }} + - name: AGENTFLOW_CONTROLPLANE_PG_DSN + valueFrom: + secretKeyRef: + name: {{ required "controlPlane.postgres.existingSecret is required when controlPlane.store=postgres (it must hold the PostgreSQL DSN; the chart ships no PG service)" .Values.controlPlane.postgres.existingSecret | quote }} + key: {{ .Values.controlPlane.postgres.dsnKey | quote }} + {{- end }} - name: DUCKDB_PATH value: {{ .Values.config.duckdbPath | quote }} - name: AGENTFLOW_USAGE_DB_PATH diff --git a/helm/agentflow/values.schema.json b/helm/agentflow/values.schema.json index c0cde87b..24869fd8 100644 --- a/helm/agentflow/values.schema.json +++ b/helm/agentflow/values.schema.json @@ -446,8 +446,25 @@ "properties": { "store": { "type": "string", - "enum": ["embedded"], - "description": "Where control-plane state (webhook queue/log, alert rules+history, outbox, dead-letter, usage) lives. 'embedded' = per-pod DuckDB + config files, single replica only. 'postgres' joins this enum only when the PostgresControlPlaneStore adapter ships (ADR 0010 rollout) — the schema must not advertise a profile the app cannot run." + "enum": ["embedded", "postgres"], + "description": "Where control-plane state (webhook queue/log, alert rules+history, outbox, dead-letter, usage) lives. 'embedded' = per-pod DuckDB + config files, single replica only (the default profile). 'postgres' = the shared PostgresControlPlaneStore adapter (ADR 0010), required for multi-replica: it needs controlPlane.postgres.existingSecret to hold the DSN and serving.backend=clickhouse. The adapter shipped in ADR 0010 rollout slice 5 and the chart profile in slice 6, so the render gate now admits multi-replica once both halves are set." + }, + "postgres": { + "type": "object", + "additionalProperties": false, + "required": ["existingSecret", "dsnKey"], + "description": "Scale-profile control-plane store connection (used only when store=postgres). The chart never ships a PostgreSQL service (mirroring how ClickHouse is consumed, ADR 0010): the operator provides the DSN in a Secret and references it here.", + "properties": { + "existingSecret": { + "type": "string", + "description": "Name of an operator-managed Secret holding the PostgreSQL DSN. Required when store=postgres; the DSN carries a password, so like the ClickHouse password it is never inlined into values." + }, + "dsnKey": { + "type": "string", + "minLength": 1, + "description": "Key inside existingSecret whose value is the full PostgreSQL DSN (e.g. postgresql://user:pass@host:5432/agentflow). Wired to AGENTFLOW_CONTROLPLANE_PG_DSN." + } + } } } }, diff --git a/helm/agentflow/values.yaml b/helm/agentflow/values.yaml index b254e47c..337c9e06 100644 --- a/helm/agentflow/values.yaml +++ b/helm/agentflow/values.yaml @@ -138,14 +138,21 @@ serving: # alert rules and history, outbox, dead-letter and usage accounting. # 'embedded' = per-pod DuckDB + config files — correct for the single-replica # profile, a split-brain at replicaCount > 1 (duplicate deliveries, forked -# alert state). The PostgresControlPlaneStore adapter shipped with ADR 0010 -# rollout slice 5 (app-side: AGENTFLOW_CONTROLPLANE_STORE=postgres + -# AGENTFLOW_CONTROLPLANE_PG_DSN); the chart profile for it (env + secret -# wiring, schema enum extension) lands with rollout slice 6 — until then the -# values schema pins store=embedded (fail-closed ratchet) and the chart -# refuses any multi-replica render. +# alert state). 'postgres' = the shared PostgresControlPlaneStore adapter +# (ADR 0010): the app-side half shipped in rollout slice 5 +# (AGENTFLOW_CONTROLPLANE_STORE=postgres + AGENTFLOW_CONTROLPLANE_PG_DSN) and +# the chart profile below in slice 6. To scale horizontally set +# store: postgres + postgres.existingSecret (DSN) AND serving.backend: +# clickhouse — templates/deployment.yaml fails any multi-replica render unless +# BOTH halves of the gate are set, so the gate relaxes automatically and only +# when the store that makes scaling correct exists. The chart never ships a +# PostgreSQL service (mirroring ClickHouse) — the operator provides the DSN in +# a Secret referenced by postgres.existingSecret / postgres.dsnKey. controlPlane: store: embedded + postgres: + existingSecret: "" + dsnKey: controlplane-pg-dsn config: duckdbPath: /data/agentflow.duckdb diff --git a/k8s/staging/values-staging-scale.yaml.example b/k8s/staging/values-staging-scale.yaml.example new file mode 100644 index 00000000..479041bc --- /dev/null +++ b/k8s/staging/values-staging-scale.yaml.example @@ -0,0 +1,53 @@ +# Scale-profile staging overlay (ADR 0010 rollout slice 6 / cutover plan +# Phase 3). Layer this ON TOP of values-staging.yaml: +# +# helm upgrade --install agentflow helm/agentflow \ +# -f k8s/staging/values-staging.yaml \ +# -f k8s/staging/values-staging-scale.yaml \ +# --namespace agentflow +# +# It flips BOTH halves of the render gate (external serving engine + external +# control-plane store) so replicaCount=2 is admissible, and exercises the +# multi-replica correctness the ADR pins. The chart ships NO PostgreSQL and NO +# ClickHouse service — the operator (or the kind bring-up script) provides both +# and the secrets referenced below. This file is an .example: copy it to +# values-staging-scale.yaml and fill in your real secret names. It carries no +# credentials itself — only Secret *names* and hostnames — so it is safe to +# commit (the DSN and ClickHouse password live in the referenced Secrets). +# +# Prerequisites in-cluster (see scripts/k8s_replica_correctness_verify.sh): +# - a PostgreSQL reachable at the DSN in secret `agentflow-controlplane-pg` +# (key controlplane-pg-dsn), schema initialised by the app on boot +# - a ClickHouse reachable at serving.clickhouse.host, password in secret +# `agentflow-clickhouse` (key clickhouse-password) +# - Redis (already stood up by k8s_staging_up.sh) + +# Two API pods behind the Service — the split-brain surface the control-plane +# externalisation exists to make correct. +replicaCount: 2 + +# The DuckDB single-writer PVC must be gone on the request path: the control +# plane now lives in PostgreSQL and serving reads go to ClickHouse, so the pod +# is stateless (ADR 0007 "stateless pods" becomes literally true here). +persistence: + enabled: false + +# External serving engine — the ADR 0006/0007 half of the gate. +serving: + backend: clickhouse + clickhouse: + host: agentflow-clickhouse + port: 8123 + database: agentflow + user: default + existingSecret: agentflow-clickhouse + passwordKey: clickhouse-password + +# External control-plane store — the ADR 0009/0010 half of the gate. The DSN +# is operator-provided in a Secret (never inlined; it carries a password, same +# posture as the ClickHouse password above). +controlPlane: + store: postgres + postgres: + existingSecret: agentflow-controlplane-pg + dsnKey: controlplane-pg-dsn diff --git a/scripts/k8s_replica_correctness_verify.sh b/scripts/k8s_replica_correctness_verify.sh new file mode 100644 index 00000000..5ddc7f4c --- /dev/null +++ b/scripts/k8s_replica_correctness_verify.sh @@ -0,0 +1,106 @@ +#!/usr/bin/env bash +# +# ADR 0010 rollout slice 6 / cutover plan Phase 3 — replica-correctness verify. +# +# Runs against an ALREADY-DEPLOYED scale-profile cluster (replicaCount>=2, +# controlPlane.store=postgres, serving.backend=clickhouse). Bring it up with: +# +# helm upgrade --install agentflow helm/agentflow \ +# -f k8s/staging/values-staging.yaml \ +# -f k8s/staging/values-staging-scale.yaml \ +# --namespace agentflow --wait +# +# after standing up in-cluster PostgreSQL + ClickHouse + Redis and their +# secrets (agentflow-controlplane-pg / agentflow-clickhouse). +# +# What this script automates (deterministic over the HTTP API): +# [Check 1] the deployment actually runs >=2 ready pods behind one Service, +# and the control-plane store env is 'postgres' on every pod; +# [Check 2] cross-pod registration visibility — a webhook registered through +# the Service is visible on repeated reads that round-robin the +# pods. This is the SHARPEST split-brain (ADR 0010 §Verified state +# inventory class 5): on the embedded YAML store it fails (pod A's +# registration is invisible on pod B); on postgres it holds. +# +# What it does NOT automate (documented recipe — see +# docs/clickhouse-cutover-plan.md Phase 3): +# - exactly-one delivery per (webhook, event) across two pods, and +# - one alert page per incident. +# Both need an emitted pipeline event / alert-triggering tick plus a capture +# sink. The STORE-level guarantee behind them (idempotent enqueue insert-win, +# single-flight alert claim, outbox<->dead-letter atomicity) is already +# live-verified by the slice-5 standalone-PG probe suite (31/31, +# docs/perf/control-plane-pg-verify-2026-07-03.md); Phase 3 only adds the +# two-real-pods topology proof on top. +# +# NOTE: authored on a Docker-less host and not yet executed end-to-end — it is +# the Mac/CI half of E4 (kind + Docker required). Syntax-checked (bash -n) only. +set -Eeuo pipefail + +BASE_URL="${BASE_URL:-http://127.0.0.1:8080}" +NAMESPACE="${NAMESPACE:-agentflow}" +RELEASE_NAME="${RELEASE_NAME:-agentflow}" +# A support-tenant key with webhook scope (matches k8s_smoke_test.sh defaults / +# values-staging.yaml fixtures). +API_KEY="${API_KEY:-af-prod-agent-ops-def456}" +MIN_REPLICAS="${MIN_REPLICAS:-2}" +# A public https URL that passes the egress guard but is never actually called +# (no matching event is emitted in this check). +WEBHOOK_URL="${WEBHOOK_URL:-https://example.com/agentflow-replica-verify}" +LIST_READS="${LIST_READS:-8}" + +fail() { echo "FAIL: $*" >&2; exit 1; } +pass() { echo "PASS: $*"; } + +# --- Check 1: >=2 ready pods, all on the postgres control-plane store --------- +echo "==> [Check 1] deployment runs >=${MIN_REPLICAS} ready pods on postgres store" +kubectl wait --namespace "$NAMESPACE" --for=condition=available \ + "deployment/$RELEASE_NAME" --timeout=180s + +ready=$(kubectl get deployment "$RELEASE_NAME" --namespace "$NAMESPACE" \ + -o jsonpath='{.status.readyReplicas}') +ready="${ready:-0}" +(( ready >= MIN_REPLICAS )) || fail "only ${ready} ready replica(s), need >= ${MIN_REPLICAS}" + +mapfile -t pods < <(kubectl get pods --namespace "$NAMESPACE" \ + -l "app.kubernetes.io/instance=$RELEASE_NAME" -o name) +(( ${#pods[@]} >= MIN_REPLICAS )) || fail "only ${#pods[@]} pod(s) found" + +for pod in "${pods[@]}"; do + store=$(kubectl get "$pod" --namespace "$NAMESPACE" \ + -o jsonpath='{.spec.containers[0].env[?(@.name=="AGENTFLOW_CONTROLPLANE_STORE")].value}') + [[ "$store" == "postgres" ]] || fail "$pod has AGENTFLOW_CONTROLPLANE_STORE=${store:-}, expected postgres" +done +pass "${ready} ready pods, all AGENTFLOW_CONTROLPLANE_STORE=postgres" + +# --- Check 2: cross-pod registration visibility ------------------------------- +echo "==> [Check 2] webhook registered via the Service is visible from every pod" +reg=$(curl -fsS -X POST \ + -H "X-API-Key: $API_KEY" -H "Content-Type: application/json" \ + -d "{\"url\":\"$WEBHOOK_URL\"}" \ + "$BASE_URL/v1/webhooks") +webhook_id=$(printf '%s' "$reg" | python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])') +[[ -n "$webhook_id" ]] || fail "registration returned no id: $reg" +echo " registered webhook_id=$webhook_id" + +cleanup() { + curl -fsS -X DELETE -H "X-API-Key: $API_KEY" \ + "$BASE_URL/v1/webhooks/$webhook_id" >/dev/null 2>&1 || true +} +trap cleanup EXIT + +# Round-robin the Service; on the embedded store a read served by the pod that +# did NOT register would miss the id. Postgres shares one registration table, +# so every read must see it. +misses=0 +for i in $(seq 1 "$LIST_READS"); do + listing=$(curl -fsS -H "X-API-Key: $API_KEY" "$BASE_URL/v1/webhooks") + if ! printf '%s' "$listing" | grep -q "$webhook_id"; then + misses=$((misses + 1)) + echo " read #$i did NOT see $webhook_id" + fi +done +(( misses == 0 )) || fail "webhook invisible on ${misses}/${LIST_READS} reads — split-brain (embedded store?)" +pass "webhook visible on all ${LIST_READS} round-robin reads across pods" + +echo "==> replica-correctness verify OK (Checks 1-2 automated; see cutover plan Phase 3 for the delivery/alert recipe)" diff --git a/tests/unit/test_helm_values_contract.py b/tests/unit/test_helm_values_contract.py index 9502e892..3a018cc5 100644 --- a/tests/unit/test_helm_values_contract.py +++ b/tests/unit/test_helm_values_contract.py @@ -178,19 +178,28 @@ def test_serving_clickhouse_render_wires_env_and_requires_host(): assert "serving.clickhouse.host is required" in _combined_output(missing_host) -def test_chart_pins_embedded_control_plane_store(): +def test_chart_control_plane_store_defaults_embedded_and_admits_postgres(): """ADR 0009/0010: the control plane (webhook queue/log, alert rules+history, - outbox, dead-letter, usage) is embedded per-pod state. The schema enum is a - fail-closed ratchet — 'postgres' joins it only when the - PostgresControlPlaneStore adapter actually ships, so the chart can never - advertise a profile the app cannot run.""" + outbox, dead-letter, usage) defaults to embedded per-pod state. The schema + enum was a fail-closed ratchet — 'postgres' joined it only once the + PostgresControlPlaneStore adapter and its chart profile shipped (rollout + slices 5–6). The default value stays 'embedded' (the zero-dependency demo), + but 'postgres' is now a first-class, schema-advertised profile.""" values = _load_yaml(CHART_PATH / "values.yaml") assert values["controlPlane"]["store"] == "embedded" + # The scale-profile DSN is operator-provided (never inlined) — same posture + # as the ClickHouse password: empty existingSecret by default. + assert values["controlPlane"]["postgres"]["existingSecret"] == "" + assert values["controlPlane"]["postgres"]["dsnKey"] schema = json.loads((CHART_PATH / "values.schema.json").read_text(encoding="utf-8")) assert "controlPlane" in schema["required"] - store = schema["properties"]["controlPlane"]["properties"]["store"] - assert store["enum"] == ["embedded"] + control_plane = schema["properties"]["controlPlane"]["properties"] + assert control_plane["store"]["enum"] == ["embedded", "postgres"] + postgres = control_plane["postgres"] + assert postgres["additionalProperties"] is False + assert "existingSecret" in postgres["required"] + assert "dsnKey" in postgres["required"] def test_helm_template_rejects_multi_replica_with_embedded_control_plane(): @@ -236,3 +245,116 @@ def test_helm_template_single_replica_renders_with_embedded_control_plane(): result = _run_helm_template() output = _combined_output(result) assert result.returncode == 0, output + + +def _scale_profile_args(*extra: str) -> list[str]: + """The minimal set of overrides that satisfies BOTH halves of the ADR + 0010 render gate (external serving engine + external control-plane store), + so multi-replica / autoscaling is admissible.""" + return [ + "--set", + "persistence.enabled=false", + "--set", + "serving.backend=clickhouse", + "--set", + "serving.clickhouse.host=clickhouse.data.svc", + "--set", + "serving.clickhouse.existingSecret=agentflow-clickhouse", + "--set", + "controlPlane.store=postgres", + "--set", + "controlPlane.postgres.existingSecret=agentflow-controlplane-pg", + *extra, + ] + + +def test_embedded_profile_sets_store_env_and_omits_dsn(): + """The default profile still boots embedded: the store env is set + explicitly (self-documenting manifest) and no PG DSN env is wired.""" + result = _run_helm_template() + output = _combined_output(result) + assert result.returncode == 0, output + assert "AGENTFLOW_CONTROLPLANE_STORE" in output + assert 'value: "embedded"' in output + assert "AGENTFLOW_CONTROLPLANE_PG_DSN" not in output + + +def test_postgres_control_plane_profile_wires_store_env_and_dsn_secret(): + """ADR 0010 slice 6: the postgres profile wires AGENTFLOW_CONTROLPLANE_STORE + and sources AGENTFLOW_CONTROLPLANE_PG_DSN from the operator-provided secret + (never inlined, mirroring the ClickHouse password).""" + result = _run_helm_template(*_scale_profile_args("--set", "replicaCount=2")) + output = _combined_output(result) + assert result.returncode == 0, output + assert 'value: "postgres"' in output + assert "AGENTFLOW_CONTROLPLANE_PG_DSN" in output + assert 'name: "agentflow-controlplane-pg"' in output + assert 'key: "controlplane-pg-dsn"' in output + + +def test_postgres_control_plane_requires_dsn_secret(): + """store=postgres without a DSN secret fails the render (a silent fallback + to embedded would re-open the split-brain the gate prevents).""" + result = _run_helm_template( + "--set", + "persistence.enabled=false", + "--set", + "serving.backend=clickhouse", + "--set", + "serving.clickhouse.host=clickhouse.data.svc", + "--set", + "serving.clickhouse.existingSecret=agentflow-clickhouse", + "--set", + "controlPlane.store=postgres", + "--set", + "replicaCount=2", + ) + output = _combined_output(result) + assert result.returncode != 0, output + assert "controlPlane.postgres.existingSecret is required" in output + + +def test_full_scale_profile_admits_multi_replica(): + """The render gate relaxes automatically once BOTH halves are set: a + replicaCount=2 render succeeds and schedules two pods.""" + result = _run_helm_template(*_scale_profile_args("--set", "replicaCount=2")) + output = _combined_output(result) + assert result.returncode == 0, output + assert "replicas: 2" in output + + +def test_full_scale_profile_admits_autoscaling_hpa(): + """Phase 3: autoscaling.maxReplicas>1 renders an HPA once the scale profile + (external serving + external control-plane store) is set.""" + result = _run_helm_template( + *_scale_profile_args( + "--set", + "autoscaling.enabled=true", + "--set", + "autoscaling.minReplicas=2", + "--set", + "autoscaling.maxReplicas=4", + ) + ) + output = _combined_output(result) + assert result.returncode == 0, output + assert "kind: HorizontalPodAutoscaler" in output + assert "maxReplicas: 4" in output + + +def test_postgres_store_still_gated_without_clickhouse_backend(): + """The control-plane half alone does not open the gate — multi-replica on + the duckdb backend must still fail (ADR 0007 engine half unmet).""" + result = _run_helm_template( + "--set", + "persistence.enabled=false", + "--set", + "replicaCount=2", + "--set", + "controlPlane.store=postgres", + "--set", + "controlPlane.postgres.existingSecret=agentflow-controlplane-pg", + ) + output = _combined_output(result) + assert result.returncode != 0, output + assert "Multi-replica requires BOTH" in output