diff --git a/design-proposals/external-database-exposure/README.md b/design-proposals/external-database-exposure/README.md index f8bb52d..10ad01d 100644 --- a/design-proposals/external-database-exposure/README.md +++ b/design-proposals/external-database-exposure/README.md @@ -16,7 +16,7 @@ This is the design-proposal artifact required by `cozystack/cozystack#2816`, and ## Scope and related proposals -- **Depends on:** `design-proposals/unified-tls-pki` — provides the `-ca-cert` key-free trust anchor that external clients use to verify the endpoint. This proposal does not re-specify it. It is a companion submission under the same epic, on its own branch; the path resolves once both proposals merge. The dependency is **per-engine, not blanket**: an engine is exposable here only once its `ca.crt` is actually delivered under that contract, and the path differs by engine — redis self-publishes a key-free `-ca-cert` through its forked operator, while postgres and mongodb obtain theirs through the extraction controller in `unified-tls-pki`. So SNI exposure for a given engine is gated on that engine's `unified-tls-pki` convergence, not merely on the contract existing. +- **Depends on:** `design-proposals/unified-tls-pki` — provides the `.tenant-ca` key-free trust anchor that external clients use to verify the endpoint. This proposal does not re-specify it. It is a companion submission under the same epic, on its own branch; the path resolves once both proposals merge. The dependency is **per-engine, not blanket**: an engine is exposable here only once its `ca.crt` is actually delivered under that contract, and the path differs by engine — redis self-publishes a key-free `-ca-cert` through its forked operator, while postgres and mongodb obtain theirs through the extraction controller in `unified-tls-pki`. So SNI exposure for a given engine is gated on that engine's `unified-tls-pki` convergence, not merely on the contract existing. - **Related:** `design-proposals/structured-external-exposure` (community pull request #29) — replaces the chart-level `external` boolean with a structured, additive `expose` list riding `ExposureClass` / `ServiceExposure` (`cozystack/cozystack#3081`), and lists Gateway/SNI consolidation as the forward-compatible future path. The two proposals meet at the trigger surface: once `expose` lands, "expose this database via SNI-passthrough" is naturally one `expose` entry (an exposure class or scope backed by the tenant Gateway) rather than a new chart toggle, and the per-release route rendering naturally belongs to that orchestration layer. This proposal defines the Gateway-side mechanics either trigger drives; it does not depend on `expose` landing first. - **Umbrella:** `cozystack/cozystack#2811`. This proposal covers WS4 (`cozystack/cozystack#2815`, SNI exposure) and WS5 (`cozystack/cozystack#2816`, end-to-end TLS). - **Referenced, not designed here:** WS6 east-west / in-cluster CNI encryption (`cozystack/cozystack#2977`, PR `cozystack/cozystack#2984`). It is complementary defense-in-depth for pod-to-pod traffic and is explicitly out of scope (see Non-goals). @@ -136,7 +136,7 @@ One subtlety the "single certificate" framing hides: an external client doing `s WS5 adds nothing beyond two hooks that already exist: -1. **Trust-anchor delivery** — the `-ca-cert` `ca.crt`-only object from `unified-tls-pki`, projected to the tenant. The external client verifies against that `ca.crt`. +1. **Trust-anchor delivery** — the `.tenant-ca` `ca.crt`-only object from `unified-tls-pki`, projected to the tenant. The external client verifies against that `ca.crt`. 2. **SAN coverage** — the chart already injects the external hostname into the operator-issued certificate (postgres `serverAltDNSNames` in `db.yaml`, gated on TLS and external). Under this proposal the hook injects **both** name shapes — `..` and `.` — so the certificate is valid for the subdomain scheme shipping now and the flat scheme arriving with Cilium 1.20 (§6). Stated plainly, where this does **not** work: diff --git a/design-proposals/unified-tls-pki/README.md b/design-proposals/unified-tls-pki/README.md index 9d1c1bf..3ad97da 100644 --- a/design-proposals/unified-tls-pki/README.md +++ b/design-proposals/unified-tls-pki/README.md @@ -20,7 +20,7 @@ This proposal is the umbrella design for the work tracked by epic `cozystack/coz - **Edge, open:** `cozystack/cozystack#2990` (propagate the operator wildcard to per-tenant termination points — the PR implementing issue `cozystack/cozystack#2820`). - **Workstreams (issues):** `cozystack/cozystack#2812` and `cozystack/cozystack#2400` (closed, edge wildcard); `cozystack/cozystack#2814` (converge per-app TLS and close the trust-anchor delivery gap — the first consumer of this contract); `cozystack/cozystack#2815` (external DB exposure via Gateway TLS-passthrough); `cozystack/cozystack#2816` (end-to-end TLS for databases); `cozystack/cozystack#2977` (opt-in east-west encryption). Throughout this document a `cozystack/cozystack#NNNN` reference is an issue unless called out as a PR. - **Per-app TLS series (open):** `cozystack/cozystack#2729` (redis), `cozystack/cozystack#2692` (mongodb), `cozystack/cozystack#2683` (rabbitmq), `cozystack/cozystack#2682` (opensearch), `cozystack/cozystack#2680` (mariadb). These are the pull requests that should land *after* this contract is accepted, not before; several currently propose handing tenants a key-bearing Secret, which this contract exists to correct (see Security). -- **Related (consumers of this contract):** `design-proposals/external-database-exposure` — the sibling proposal under the same epic; its end-to-end TLS story has external clients validate against the `-ca-cert` object defined here. `design-proposals/structured-external-exposure` (community pull request #29) — the structured `expose` model whose "external implies TLS" rule likewise points clients at this trust anchor. Both consume this proposal's output and define no PKI of their own. +- **Related (consumers of this contract):** `design-proposals/external-database-exposure` — the sibling proposal under the same epic; its end-to-end TLS story has external clients validate against the `.tenant-ca` object defined here. `design-proposals/structured-external-exposure` (community pull request #29) — the structured `expose` model whose "external implies TLS" rule likewise points clients at this trust anchor. Both consume this proposal's output and define no PKI of their own. All repository paths below refer to the `cozystack/cozystack` repository; paths attributed to an open PR (for example the wildcard-secret reconciler in PR `cozystack/cozystack#2990`) are not yet on `main`. @@ -134,7 +134,11 @@ The takeaway: only **kafka** (native) and **redis** (forked operator) are at the ### 4. The uniform consume contract -Every engine, regardless of who owns its CA, exposes its trust anchor through one canonical object: a Secret named `-ca-cert`, containing only `ca.crt`, stamped with `internal.cozystack.io/tenantresource: "true"`. The platform already has the building block — `cozy-lib.tls.caCertSecret` renders exactly this object and fails closed if the input contains a private key. +Every engine, regardless of who owns its CA, exposes its trust anchor through one canonical object: a Secret named `.tenant-ca`, containing only `ca.crt`, stamped with `internal.cozystack.io/tenant-ca: "true"` and made tenant-visible through the lineage webhook's `tenantresource` verdict. + +The name took three revisions, and the reason is worth recording because it is the same mistake twice. `-ca-cert` was rejected because Percona PSMDB already creates a Secret of that exact name and puts a private key in it — a collision across ENGINES. Its replacement `-tenant-ca` was justified as "claimed by no operator this platform ships", and that was false: it collides across RELEASES. For an application `foo` the projection is `postgres-foo-tenant-ca`; for a sibling application `foo-tenant`, CloudNativePG's own CA is `postgres-foo-tenant` + `-ca` — the same string. Both names are legal and several Postgres instances in one namespace is the normal model. Nor is it merely a guard problem: if the projection lands first, CNPG rejects the key-free Secret (`missing ca.key secret data`) and the sibling's PKI never completes, so the sibling application never comes up — silently, blaming a Secret its owner never created. That is a regression, since before this contract existed the name was unclaimed and both applications worked, and no guard can refuse it, because the controller that wrote first was legitimate. + +The canonical name is therefore `.tenant-ca`, and it is collision-free by CHARACTER CLASS rather than by survey. Application names are validated as DNS-1035 labels (`[a-z]([-a-z0-9]*[a-z0-9])?`), so a dot is structurally impossible in one; every release prefix is dot-free; and Secret names are DNS-1123 subdomains, where a dot is legal. So `` is dot-free by construction and can never equal a dotted name — for any application name, any operator, and any suffix an operator may add in future. That is a proof. "No operator claims it today" is a guess with an expiry date, and it expired twice. The platform already has the building block — `cozy-lib.tls.caCertSecret` renders exactly this object and fails closed if the input contains a private key. This is where the label-filtered projection matters. Because `tenantsecrets` delivers the whole Secret `Data`, the helper's fail-closed guard is not a nicety — it is the boundary that keeps a server or CA private key out of a tenant's hands. Kafka's `-clients-ca-cert` is the shape to match: a CA certificate, no key, readable by the tenant. @@ -149,18 +153,18 @@ The alternative to one controller is to fork each upstream operator the way redi The controller is deliberately **engine-agnostic**: it does not branch on cert-manager-vs-operator-owned, and it does not key off Secret names. Its contract is three pieces, only the middle of which is new code. -**(a) An explicit source-selection label, engine-agnostic, not a name convention.** The owner of the CA-bearing Secret stamps `internal.cozystack.io/publish-ca-cert: "true"`, with an optional `internal.cozystack.io/publish-ca-cert-key` annotation naming the key to lift (default `ca.crt`). For the cert-manager-minting charts this rides `Certificate.spec.secretTemplate.labels`, so even the asynchronously-created CA Secret carries the marker from the moment cert-manager writes it. The label, not the name, is the contract — which matters because the CA-bearing Secret names are non-uniform (`-ca`, `-ca-tls`, `-http-ca`, `-ca-cert`) and the name `-ca-cert` is overloaded: key-free for the redis fork, key-bearing for PSMDB. A name convention would mis-handle one of them; the label plus a content check does not. +**(a) An explicit source-selection label, engine-agnostic, not a name convention.** The owner of the CA-bearing Secret stamps `internal.cozystack.io/publish-ca-cert: "true"`, with an optional `internal.cozystack.io/publish-ca-cert-key` annotation naming the key to lift (default `ca.crt`). It also stamps `internal.cozystack.io/publish-ca-cert-release` with the release name: implementation showed the release cannot be inferred on the label leg, because the platform runs cert-manager with `enableCertificateOwnerRef: false`, so a cert-manager-issued Secret carries no ownerReference back to its owning application and the lineage graph cannot resolve which release it belongs to. The release label supplies that link explicitly. For the cert-manager-minting charts all three markers ride `Certificate.spec.secretTemplate.labels`, so even the asynchronously-created CA Secret carries them from the moment cert-manager writes it. The label, not the name, is the contract — which matters because the CA-bearing Secret names are non-uniform (`-ca`, `-ca-tls`, `-http-ca`, `-ca-cert`) and the name `-ca-cert` is overloaded: key-free for the redis fork, key-bearing for PSMDB. A name convention would mis-handle one of them; the label plus a content check does not. -**(b) A small extraction controller.** Its watch/upsert skeleton can follow the wildcard-secret reconciler from PR `cozystack/cozystack#2990` (`internal/controller/wildcardsecret/reconciler.go`), but it carries none of that reconciler's copy-marking or prune logic — lineage provides those (part (c)) — and, being **intra-namespace**, it can do something that reconciler cannot. For each label-selected source Secret it upserts a `type: Opaque` Secret named `-ca-cert` containing **only** `ca.crt`, re-copying on every source change so a CA rotation propagates without a chart re-render. It does four security-load-bearing things and nothing more: +**(b) A small extraction controller.** Its watch/upsert skeleton can follow the wildcard-secret reconciler from PR `cozystack/cozystack#2990` (`internal/controller/wildcardsecret/reconciler.go`), but it carries none of that reconciler's copy-marking or prune logic — lineage provides those (part (c)) — and, being **intra-namespace**, it can do something that reconciler cannot. For each selected source Secret it upserts a `type: Opaque` Secret named `.tenant-ca` containing **only** `ca.crt`, re-copying on every source change so a CA rotation propagates without a chart re-render. It does four security-load-bearing things and nothing more: - **Key off the label and the content, never the name.** Select sources by the `publish-ca-cert` label; before writing, verify the lifted value is a CA certificate and carries no `-----BEGIN … PRIVATE KEY-----` header. This is what lets one controller serve a key-free redis source it should ignore and a key-bearing PSMDB source it must strip, both named `-ca-cert`-adjacent, without confusion. - **Tolerate operator-created, asynchronous sources.** The source may be created by the CNPG or PSMDB operator after the chart renders (not just by cert-manager). The controller waits on the watch event for the labeled source; it does not assume the chart owns the Secret and does not error or busy-loop before the source exists. - **Sanitize at write time, not just render time.** The `cozy-lib.tls.caCertSecret` helper's fail-closed guard runs at *chart-render* time; this controller writes at *runtime*, so it must itself copy only the single `ca.crt` key (an explicit whitelist) and re-assert the no-private-key check on every write. It never copies the whole `Data`. -- **Owner-ref the projected Secret to the application instance CR**, resolved from the `app.kubernetes.io/instance` label already on the source Secret. This is where intra-namespace beats the `cozystack/cozystack#2990` reconciler, which tracks its replicas by a management label plus a back-reference annotation *because* a cross-namespace replica cannot carry a valid `ownerReference`. Our projection is same-namespace, so a real owner-reference is valid: Kubernetes garbage-collects the `-ca-cert` on app deletion for free, and the controller needs no prune logic of its own. (It does not owner-ref the *source* `-ca`: cert-manager and the DB operators do not own their output Secrets, so a walk from the source would dead-end before the app.) +- **Owner-ref the projected Secret to the release's `HelmRelease`.** An earlier revision of this document said to owner-ref the application instance CR; that is not implementable. The `apps.cozystack.io` kinds are virtual — served by the aggregated apiserver, and a tenant application is *stored* as a `HelmRelease` whose `spec.values` is the application spec verbatim — so there is no application object in etcd and therefore no UID to reference. The `HelmRelease` is the only real object in the chain, and it is enough: deleting the application through the aggregated API deletes the `HelmRelease`, which garbage-collects the projection with it. This is where intra-namespace beats the `cozystack/cozystack#2990` reconciler, which tracks its replicas by a management label plus a back-reference annotation *because* a cross-namespace replica cannot carry a valid `ownerReference`. Our projection is same-namespace, so a real owner-reference is valid and the controller needs no prune logic of its own. `blockOwnerDeletion` stays false: a trust anchor must never delay the teardown of the application it belongs to. (It does not owner-ref the *source* `-ca`: cert-manager and the DB operators do not own their output Secrets, so a walk from the source would dead-end before the app.) **(c) Marking stays in `spec.secrets`, by label not by name.** Because `ApplicationDefinition.spec.secrets.include` accepts a label selector (`internal/lineagecontrollerwebhook/matcher.go`), one generic entry — `matchLabels: {internal.cozystack.io/tenant-ca: "true"}`, stamped by the controller on every projected Secret — covers every engine with no per-release `resourceName` templating. The lineage admission webhook then does the rest: on admission it walks the projected Secret's `ownerReferences` to the owning application and authoritatively stamps `internal.cozystack.io/tenantresource` to `true` (and to `false` should the Secret ever stop matching). -What this *reuses* rather than rebuilds: the label selector in `ApplicationDefinition.spec.secrets` (`internal/lineagecontrollerwebhook/matcher.go`), the lineage webhook's owner-reference walk and authoritative `tenantresource` stamping (`internal/lineagecontrollerwebhook/webhook.go`, `pkg/lineage/lineage.go`), the private-key guard in `cozy-lib.tls.caCertSecret` (`packages/library/cozy-lib/templates/_tls.tpl`), and native Kubernetes garbage collection via the owner reference. The irreducibly new work is the extraction step itself — read one key, write a key-free copy, re-copy on rotation — the same job an operator does natively on the engines that already ship a key-free CA object (kafka's `-clients-ca-cert`, the redis fork's `-ca-cert`). +What this *reuses* rather than rebuilds: the label selector in `ApplicationDefinition.spec.secrets` (`internal/lineagecontrollerwebhook/matcher.go`), the lineage webhook's owner-reference walk and authoritative `tenantresource` stamping (`internal/lineagecontrollerwebhook/webhook.go`, `pkg/lineage/lineage.go`), the private-key guard in `cozy-lib.tls.caCertSecret` (`packages/library/cozy-lib/templates/_tls.tpl`), and native Kubernetes garbage collection via the owner reference on the `HelmRelease`. The irreducibly new work is the extraction step itself — read one key, write a key-free copy, re-copy on rotation — the same job an operator does natively on the engines that already ship a key-free CA object (kafka's `-clients-ca-cert`, the redis fork's `-ca-cert`) — those two still need the projection, because their CA lands under an engine-specific name rather than the canonical one; for them the extraction step is a straight copy, with nothing to strip. ### 6. Per-engine application order @@ -208,7 +212,7 @@ Two consequences for the controller. First, it writes at runtime, after chart-re ## Open questions - **Labelling operator-created sources.** The cert-manager charts stamp the source label via `Certificate.spec.secretTemplate.labels`. Operators that create the CA Secret themselves (CNPG `-ca`, PSMDB `-ca-cert`) may not let the chart label their output. For those, the controller needs either an operator that supports output labels or a small per-engine source-name configuration as a fallback — this is the one place the engine-agnostic label is not yet sufficient, and it should be resolved before postgres/mongodb convergence. -- The exact namespace convention for the `-ca-cert` object (per-release in the app namespace is assumed here) and the final label/annotation names. This proposal uses `internal.cozystack.io/publish-ca-cert` on the source and `internal.cozystack.io/tenant-ca` on the projected copy, matching the `internal.cozystack.io/` convention of the existing `tenantresource` and `managed-by-cozystack` markers. +- The exact namespace convention for the `.tenant-ca` object (per-release in the app namespace is assumed here) and the final label/annotation names. This proposal uses `internal.cozystack.io/publish-ca-cert` and `internal.cozystack.io/publish-ca-cert-release` on the source and `internal.cozystack.io/tenant-ca` on the projected copy, matching the `internal.cozystack.io/` convention of the existing `tenantresource` and `managed-by-cozystack` markers. - How this intersects with per-tenant wildcard propagation (issue `cozystack/cozystack#2820`, implemented by PR `cozystack/cozystack#2990`). That is a *cross-namespace* replication problem; the extraction controller here is *intra-namespace, per-release*, so the two do not share a mechanism — but they should share the management-label and foreign-collision conventions. Note that `clustersecret-operator` is packaged in the repo but is not wired into a default bundle, so it is not an installed primitive this design can assume. ## Alternatives considered