docs: document gateway certificate issuance model and downstream solver

[ecv]

Problem

NSO owns the gateway certificate issuance model — the two paths a Gateway-listener cert can take, the enableDownstreamCertificateSolver / clusterIssuerMap config that selects them, and the failure modes an operator hits when a cert gets stuck. Today none of this is documented in this repo. The knowledge only existed as tribal context in the infra deployment, which is how infra#3410 hid for months: a consumer of NSO reading only the operator docs cannot tell how downstream solving is supposed to behave, so they can't tell a wiring bug from expected behavior.

This is operator-level knowledge — true for anyone running NSO with downstream certificate solving, not just Datum's Karmada deployment. It belongs here, not in infra. Infra should only document its own wiring (how it satisfies the prerequisites across Karmada) and link back to this.

What to document

Suggested home: docs/ (alongside runbooks/), or extend the gateway/certificate config reference.

1. The two issuance paths for a tenant Gateway listener cert:

   • Hub / control-plane issue + propagate — cert-manager issues where the operator runs; the resulting Secret is distributed to the dataplane out of band.
   • Downstream (member) solve — with enableDownstreamCertificateSolver: true, the cert is issued in place on the member via HTTP01, using the ClusterIssuer named by clusterIssuerMap (e.g. auto → <issuer>).
   • How to tell which path a live cert took (backing Secret provenance).

2. Config reference — EnableDownstreamCertificateSolver and clusterIssuerMap semantics (internal/config/config.go): what turns on, what the operator expects to already exist on the member (a resolvable ClusterIssuer, solver HTTPRoute/HTTPRouteFilter carrying meta.datumapis.com/http01-solver), and what is the deployer's responsibility to provide vs. what NSO creates.

3. Failure modes (operator-generic):

   • Referenced "ClusterIssuer" not found on the member → the issuer named by clusterIssuerMap is absent where the cert is being solved. NSO assumes it exists; providing it is the deployer's job.
   • HTTP01 challenge stuck pending with self-check wrong status code '404' → challenge presented but the ACME probe never reaches the gateway; typically the hostname's DNS does not point at the gateway. Not an issuer problem.

4. Automatic re-issuance (feat: add automatic re-issuance of failed custom hostname certificates #259) — document the recovery behavior next to the model: failed custom-hostname Certificates are deleted past retryInterval to GC the whole CertificateRequest/Order/Challenge/solver-route chain and recreate, bounded by maxRetries, count tracked on the downstream Gateway. Critically: it assumes the member can issue — it retries against a missing issuer or off-gateway DNS, it does not fix either.

Why now

infra is trimming its cert docs (infra#3420) to cover only Datum-specific Karmada wiring and link here for the model. Without this doc, that link has no target and the operator-level knowledge stays undocumented.

Related

• feat: add automatic re-issuance of failed custom hostname certificates #259 — automatic re-issuance of failed custom-hostname certs (behavior to document)
• Expired/abandoned customer certs can freeze an entire edge — health-gate cert propagation + isolate per-cert failures #212 — listener cert health-gating (adjacent cert-failure path)
• infra#3410 / infra#3413 — the propagation bug the missing model let hide

[ecv]

Gap surfaced concretely during the ab.dk / AB1889 investigation (datum-cloud/infra#3407): an investigator reading the config chased a missing ClusterIssuer/datum-gateway-http on the edge members as the suspected cert blocker — because clusterIssuerMap: auto → datum-gateway-http is the most visible issuer reference in the control-plane config. @scotwells caught it: the downstream per-tenant gateway cert is not issued by that ClusterIssuer.

The doc as scoped above still risks the same conflation — bullet 1's "Downstream (member) solve … using the ClusterIssuer named by clusterIssuerMap" describes only one issuer kind. Please have the model call out issuer kind explicitly:

• The downstream per-tenant Gateway listener cert is issued by a namespaced Issuer, one per Gateway, living in the tenant namespace. (In Datum's deployment this Issuer is generated by a Kyverno ClusterPolicy, per-datum-gateway-issuer — that generation is Datum wiring, but that NSO's downstream solve resolves an issuer per-gateway/namespaced rather than a single shared ClusterIssuer is the operator-level model, with a real design reason: cert-manager's HTTP01 solver must attach its solver route to the same Gateway it references, so a single shared ClusterIssuer pointing at a separate verifier gateway causes duplicate-hostname conflicts programming Envoy.)
• clusterIssuerMap: auto → <ClusterIssuer> is a separate opt-in path (TLS option certificate-issuer: auto), not the default per-tenant path.

Making "which issuer kind serves which path" explicit is what would have prevented the ab.dk misread. Same distinction also explains the bold-fest split we saw (*-https-hostname-0 Ready via the namespaced Issuer; *-default-https NotReady via the ClusterIssuer path) — two issuer kinds, not a propagation mystery.